Skip to content
Cello
Esc
↑↓navigate↵open⌘Jpreview
On this page

What Cello guarantees

Cello makes two promises. Exactly one node serves a cell at a time, so two machines never write the same database. Cello does not acknowledge a write until the write survives a failure, so an acknowledged write is never lost. Both promises require a bucket with working conditional writes and ranged reads, and a supervisor that restarts the process.

What the bucket must provide

  • A conditional create: the write must fail when the object exists.
  • A conditional overwrite: the write must fail when the object changed after the read.
  • Read-after-write consistency: a read after a successful write must return that write.
  • Ranged reads: a read must return the requested byte range and the bytes from that range.
  • For epoch GC (CELLO_LTX_RETENTION_SECS) only, list-after-write consistency: a listing after a successful write must include the written object. Amazon S3, Cloudflare R2, Google Cloud Storage, and Azure Blob Storage provide it. A Tigris Global or Dual-region bucket provides it only in the region of the write, so do not enable epoch GC on one when fleet nodes are in more than one region.

The qualified stores are Amazon S3, Cloudflare R2, Tigris, Google Cloud Storage, and Azure Blob Storage. The release tests run against R2; the S3 path uses the same client and headers. Azure was qualified on 2026-08-18, single-node, under an account key, a VM managed identity, and an AKS workload identity. A managed identity on Azure App Service or Azure Container Apps does not work; see limitations.

Backblaze B2, Hetzner Object Storage, and DigitalOcean Spaces do not implement the required conditional writes, so two nodes can own one cell on them. A store that accepts the conditional headers but ignores the condition fails late and silently, so run the storage test.

MinIO (the community edition) passes the storage test but is not qualified for production. RELEASE.2025-09-06T17-38-46Z answers the conditional create of an absent object with NoSuchKey, so the first deploy fails (denoland/celld#162). Use RELEASE.2025-09-07T16-13-09Z or later.

A gs:// bucket uses the Cloud Storage XML API with x-goog-if-generation-match and OAuth credentials, because Cloud Storage does not apply If-Match to a PUT. For an az:// bucket, the NAME is the container.

The storage test

cello diagnose sends four conditional writes to the bucket:

ok bucket conditional write (create, reject-create, update, reject-stale)

Two of the four must fail; if the store accepts either, the command exits with an error that names the store. Use cello diagnose --read-only with a credential that cannot write.

Each node runs the same writes before it serves, then reads a range of a second object and verifies the range and bytes. A node cannot disable this test. It makes at most three attempts, with new objects, when an operation fails without a clear cause; then the node starts with a warning, because a temporary outage can end. The node stops immediately when a required conditional write or ranged read is unsupported, when the store ignores a condition or the Range header, or when it returns a wrong range or wrong bytes.

A process that stops mid-test can leave a small object under probe/; Cello never reads it.

Cello reserves probe/, cells/, nodes/, node-cells/, fleet/, deploy/, deploy-blobs/, log/, wake/, and telemetry/, and deletes objects under some of them. An application must not write under any of them.

An operator must not delete, expire, or overwrite an object under these prefixes, except under probe/ and telemetry/. Cello reads these objects to fence and restore cells, and Cello deletes the objects that it no longer needs. A bucket lifecycle rule therefore must not select an object under another reserved prefix. With a key prefix, each reserved prefix follows the key prefix, for example PREFIX/telemetry/.

The supervisor

Run Cello under a supervisor that restarts the process, such as systemd, Docker with a restart policy, or Kubernetes. A node that loses its lease fences itself and exits, and without a restart the fleet loses that capacity. The supervisor must restart without an attempt limit and wait at least one lease lifetime between attempts.

The mechanism

The ownership record

Each cell has one ownership record in the bucket. It names the owner node’s session and carries a fencing epoch. A node acquires a cell with a conditional create when no record exists, or a compare-and-swap on an existing record, so two nodes cannot acquire the same cell. Every activation, a takeover or a local wake, advances the epoch, so an epoch never has two writers.

The record is cells/<cell>/own.json. Cello never deletes it, and it must stay in the bucket for the life of the cell. A missing record next to LTX under cells/<cell>/ltx/ means that the record was lost, for example to a lifecycle rule. Cello then refuses to acquire the cell, because a new record at epoch 1 would start it with an empty database. Each request that must read the record, to acquire the cell or to find its owner, fails, and the node writes cello ownership record lost to stderr. The node then refuses the cell for 10 seconds without a bucket request, so it writes the line at most once in that interval. A node that still serves the cell keeps serving it and keeps writing LTX until it stops. That node cannot acknowledge a write that a bucket proof covers, because the record no longer names it, so the write fails and the node writes cello output gate to stderr. With CELLO_DURABILITY=bucket, every write takes a bucket proof. Cello detects the loss only from LTX under cells/<cell>/ltx/. A new cell can keep its first writes in a node bundle under log/ until the tiering path drains them, so a loss before that drain starts the cell empty.

To restore the cell, first stop each node that served the cell with SIGTERM, not with POST /shutdown?handoff=preserve. A preserved restart resumes its cells without a read of the record, so the node keeps serving the cell. No peer can acquire the cell, so its handoff cannot complete: the stop can take the whole CELLO_SHUTDOWN_TOTAL_MS and log shutdown_handoff_stalled.

A stopped process can still hold acknowledged writes in its node log, outside the per-cell prefixes, so wait until that log is recovered. A new process with the same CELLO_NODE recovers it during startup, so the log is recovered when that process reports healthy. Without the same CELLO_NODE, a running node with CELLO_DURABILITY=fleet, the default, recovers it after the lease of the stopped process expires. In that case, wait until nodes/<node>.json of the stopped process is absent, has no log object, or shows "state": "sealed" in it. A stop that logs node-log close: sealed epoch has already sealed its log.

Then write a released record, which names no node, at the newest epoch: the numerically highest e<epoch> prefix under cells/<cell>/ltx/. For a newest epoch of 7, the record is {"node": "", "epoch": 7}. After the refusal interval ends, the next request acquires the cell at the next epoch and restores it from the bucket. A released record skips the recovery of the node log of the previous owner. A record that is written before these logs are recovered can therefore lose acknowledged writes, and a node that still serves the cell becomes a second writer.

The epoch prefix

The replicator copies each cell’s SQLite data to cells/<cell>/ltx/e<epoch>/ with unconditional PUTs. The epoch in the key is the fence: a node that lost ownership can keep writing, but only into a superseded prefix, and a restore selects the current lineage. The tiering path can first combine segments from many cells into a node bundle and drain each segment into its per-cell prefix later. A bundle is deleted only when the per-cell prefixes cover every segment in it. A failed compaction retries after 30 seconds, backing off to at most 300 seconds.

An eviction proves the cell’s writes durable, publishes one snapshot of the closed database or falls back to its L0 chain, and keeps the local copy until the bucket can restore the cell. A cell whose restorable chain was born in this epoch, is fully per-cell covered, and fits in a short upload journal skips the snapshot by policy and hands off through that L0 chain directly. the bucket can restore the cell. When this residency itself uploaded into the epoch prefix and got a success response, that response is the proof: a successor finds the objects through a listing, as every restore does, which relies on the list-after-write consistency described above. When the residency’s upload journal names keys but that self-uploaded mark is absent — recovery uploads and compaction output journal their keys without setting it — the guard reads the newest journaled key instead of listing: a HEAD on the exact key the upload wrote, which needs only the read-after-write consistency described above, falling back to a listing on a miss or an error. A residency with an empty journal, such as one that resumed a local copy without uploading, and an eviction whose fallback finds no journaled key, list the prefix instead. The capture before an eviction skips the time-based checkpoint, whose sealing write would upload one more object just before the snapshot that covers it. After the snapshot publishes, a background task deletes the epoch’s per-cell L0 and L1 objects the snapshot supersedes (max txid at or below the snapshot’s), addressed from the keys this residency uploaded, with no listing: one single-key DELETE per key at bounded concurrency, abandoning the rest on stop. The deleted objects are in no restore plan — the planner anchors on the snapshot and only extends past it — so the epoch still opens with its snapshot and still ends at the same cut whenever a DELETE lands, including after a successor activates at its own new epoch and prefix. Failed or abandoned deletes leave garbage for epoch GC and never fail the eviction. An eviction without a snapshot deletes nothing.

The acknowledgement rule (RPO=0)

A gate holds each response until a durability proof covers every write it can reveal: a write response; a read-only response when the object has an uncovered committed write; an error response, because a thrown message can carry a value the handler read; an R2 mutation, so it cannot change the application bucket before the source write is durable; a raw TCP connect, write, TLS upgrade, or shutdown; and each chunk of a streamed body. A client therefore cannot act on a value that a crash can still lose.

After a bucket proof, Cello reads the ownership record once and acknowledges only if it still names this node at this epoch. Concurrent writes on one cell share that read: a write whose proof lands while a read is in flight joins it when its position is at or below the position of the write that issued the read, and issues its own read at once otherwise, so sharing never delays a write. Positions strictly increase with every committing write, so a joiner committed before the issuer did and the issuer’s completed proof captured it. A write at a position a verified proof already covers releases without a new read at all. The read is always issued after a proof that captured the writes it answers completed, so a takeover that orphans the prefix fails the gates it touches instead of acknowledging into them. A partitioned node can commit locally and replicate into its superseded prefix, but it does not acknowledge. The check reads the record instead of a clock, so a paused process or a skewed clock cannot pass it.

In bucket posture the node groups writes when the operator opts in: with CELLO_BUCKET_BUNDLES=1, once per flush round it writes one shared bundle object (log/<node>/<generation>/bundle/) carrying every dirty cell’s new rows, and that single PUT is the bucket proof for every row it carries — instead of one upload per write. The default uploads per-cell as before; the ownership read above still answers every acknowledgement, and CELLO_BUCKET_BUNDLE_LINGER_MS holds a round open only while more than one cell is dirty, so a lone dirty cell never waits.

A fleet proof needs no such read. The owner sends each write to one or two other nodes, its followers (together, the ensemble), and every follower must fsync it. A takeover seals the prior node-log session before it restores, so a stale owner cannot complete another fleet proof.

An unfinished SQL write cursor can return rows before SQLite commits. Outside an explicit transaction, the application must consume those rows before output or storage.sync(); Cello rejects output while a write cursor is unfinished.

The ensemble needs two nodes

A node never counts itself as a follower, and one follower is enough, therefore a fleet needs two running Cello nodes before any node can complete a fleet proof. CELLO_DURABILITY=fleet is the default, so a one-node fleet requests the fleet posture and does not get it.

A node recruits up to two followers, so a fleet of three or more nodes holds three copies of an acknowledged write. The ensemble keeps acknowledging while one follower remains. A node with one follower recruits a second one when another node becomes available. Until then, a write that is not yet in the bucket is only on the owner and on that one follower.

A node without an ensemble stays correct. It acknowledges each write on a bucket proof instead, at the cost of latency: an object store round trip is much slower than a follower fsync.

A node with no eligible peer slows its recruitment scan: the wait between scans doubles from one second up to the maintenance tick (thirty seconds) while every scan finds nobody. Recruitment is advisory — every write above already waits for a bucket proof — so the slower scan only delays fleet posture; it never weakens the acknowledgement rule. The tradeoff: a node that starts next to a lone node is recruited by it within about thirty seconds instead of about one (the new node recruits the old one itself at once).

The takeover recovery gate

In fleet mode, Cello can acknowledge a write before its bucket upload completes. Each process session therefore creates a conditional node-log record before its first fleet-durable acknowledgement.

A cold activation checks the prior owner’s log records before it reads the bucket. An absent record proves that the session never acknowledged past the bucket; a sealed record proves that recovery completed. An open or recovering record forces recovery before the restore: compare-and-swap the record to fence it, seal the reachable followers, upload their retained segments and bundles into the per-cell prefixes, and mark the record sealed.

A bucket-posture node that writes shared bundle proofs creates a node-log record of a bucket-session flavour before its first bundle-credited acknowledgement, for the same reason: once bundles exist, “no record” no longer proves the bucket complete, because acknowledged rows may live only in log/<node>/<generation>/bundle/. The record is recovered like any other — fence it, fold every retained bundle of the dead session into the per-cell prefixes, seal it — with no follower leg: there is no ensemble to seal or tail, and a bucket pass never declares bounded loss, it only drains bundles. A sealed record therefore means every acknowledged row is per-cell: the seal is refused while any retained bundle row sits outside the per-cell layout, so a takeover reading per-cell prefixes can never miss an acknowledged row.

A cell can stop while its node session stays open; its next activation gathers any acknowledged tail outside its per-cell prefix. Once the log epoch is active (before its first fleet proof), at least one current follower must return its complete retained range, or the activation fails and keeps the recovery requirement.

Recovery has these limits:

  • A follower’s HTTP error does not prove its data is absent, even when its lease has expired, so persistent errors can block recovery and startup. The one exception is a member answering as a different session (409 WrongTarget): that member is gone, and past the witness grace the pass proceeds to the loss record instead of wedging.
  • A missing lease with surviving session objects is not proof of completeness: the seal declares the bounded loss loudly instead.
  • A restarted follower cannot certify a range with a damaged batch or a gap until valid data covers it.
  • An older node’s entries-only tail (CLT1) proves neither completeness nor loss, so recovery or startup can stall during a mixed-version update.
  • A deleted final batch is undetectable if no later batch or persisted end records its range.
  • A torn batch with an unacknowledged write looks like damage after an acknowledgement, so the loss record can report an uncertified range when no acknowledged write is missing.

A restarting node serves authenticated follower seal and tail requests before its own predecessor recovery completes, so nodes that restart together can recover acknowledged writes from surviving follower disks — but only under its own session id. A process that restarted with a fresh id rejects its predecessor’s requests, which is why the id must be stable per disk (above). It accepts application requests and new follower appends only after startup completes. Follower tails stream under a progress timeout rather than a whole-request one: a large tail on a slow link completes instead of failing at ten seconds, while a stalled one still fails so recovery refuses to seal.

Recovery of a large dead node can take minutes. It reads retained bundles in windows of at most 512 MiB and uploads and releases each window before the next, so memory does not grow with the session; a cell with rows in several windows receives one object per window. A failed or timed-out attempt does not fail waiting requests: the cell retries with a backoff (CELLO_RECOVERY_RETRY_MS, default 1000), and the requests fail with a resolve error only after CELLO_RECOVERY_RETRIES (default 240) attempts.

Bucket-posture bundle proofs

Bundles are written only while every live lease reads format 3. A node writes a bundle proof only while no live peer could mistake the record’s absence for completeness: a single node’s own format-3 lease opens the gate, any live older lease holds it closed, and the node uploads per-cell while it does. The gate is re-checked per flush, not per process.

Downgrade safety rests on recovery, not on refusal. An older binary that takes over from a bundling node decodes the bucket-session record (unknown fields ignored) and recovers it like any open record: it fences the record, folds the whole bundle prefix into the per-cell prefixes, never declares loss, and seals it. No path treats the record as complete while bundles may exist, so neither a rollout step nor a rollback loses an acknowledged row. No binary refuses to boot on downgrade; the refusal would have to run in the old binary, which cannot know the new format.

An eviction keeps the sealed-means-per-cell promise for bundled cells: the evicting node uploads one per-cell snapshot from its complete local database, covering every bundle-credited row, before it releases the cell. The guard proves durability through that snapshot and refuses the eviction when the per-cell layout (snapshot, or the L0 fallback it verifies with one HEAD per journaled key) does not cover every acknowledged row, so the folded per-cell layout a successor restores from is complete.

Epoch-chain restore

A restore chains the epoch prefixes that contain LTX data, from the newest down to an epoch that opened with a whole-database snapshot. An epoch that paged in continues its predecessor from the cut it paged from, and a predecessor that does not end exactly at that cut is not part of the chain. A legacy e<epoch>.seal.json object does not limit the chain.

A paged cell reads no chain up front. It opens over a sparse local file, reads each page from the objects on first use, and fills the rest in the background; a filled cell reads only its local file. A chain smaller than CELLO_LTX_PAGED_MIN_MB is downloaded whole.

Epoch GC

When CELLO_LTX_RETENTION_SECS is positive, the owner of a cell deletes the epoch prefixes that no restore reads, in both CELLO_DURABILITY modes:

  1. It builds the chain over every epoch prefix and continues only when the newest epoch is its own. Its first object is then in the listing, so a later owner restores from a base at the same epoch or higher.
  2. A paged cell waits until its local file is complete.
  3. It continues only when the ownership record names this node at this epoch.
  4. It writes retired.json with the base and deletes the prefixes below it, keeping its own epoch, the one before, and each epoch younger than the configured time.

The order matters: otherwise a fenced owner could delete its successor’s base. A late delete is safe because an epoch below the base never rejoins a chain. This relies on list-after-write consistency.

A fenced node can append an unacknowledged tail to an older prefix. After a snapshot successor, a later restore can expose that tail; this does not violate the contract, because a missing acknowledgement does not prove a write absent. After a paged successor, the chain clips the older prefix at the cut, and the takeover recovery gate seals the prior node-log session before the successor restores, so no write past the cut can be acknowledged.

Self-fencing

Each node holds a lease in the bucket with an expiry, renewed after one third of the lifetime (CELLO_TTL_MS, default 10000 ms). A failed renewal is retried while the published expiry has not passed. Each attempt is bounded so one hung store request cannot eat the whole remaining authority: a renewal CAS, and the readback that follows a contested renewal, each run at most half the remaining authority (250 ms to 2.5 s), and the core retries on what is left. A timeout is therefore never the fence — only the published expiry is. A rejected CAS is either a lost race (HTTP 412), which reads back and retries with a fresh ETag, or a transient store conflict (HTTP 409), which retries twice with the same guard before falling back to the readback. The store target logs a rejected outcome with a conflict flag, so the two are distinguishable without packet captures.

Fencing is decided by each node alone, so a store stall longer than the TTL fences every node in the same window, and every restart rejoins at once. A longer TTL narrows that window but slows takeover after a real crash; there is no separate blip-decoupling knob. What the restarted fleet does next is decided by recovery under the witness grace below.

When the expiry passes, the node fences itself: it stops each active cell and fails every incomplete request. It also fences at once when its lease record is gone or no longer matches what it published. The fence writes nothing; peers already read the lease as dead or replaced and acquire the cells through the ownership records. Cello checks the published expiry on every routed request, so a request is safe even before the fence runs.

An ingress checks a cached remote route against the observed lease deadline on each new request and rereads ownership at the deadline, even if the old owner holds connections open. Recovery still needs the bucket and the required durable data. A draining ingress can forward to a live remote owner but refuses new ownership of an unowned cell or one with an expired owner. The ingress does not replay a request already sent to the old owner, because the handler can have committed a write; the caller can cancel it.

A fenced node logs a line starting with SELF-FENCE: and exits with code 3. Other internal failures share the prefix and code; the line names the cause: node_lease_watchdog_fence (expired), node_lease_record_missing_fence, or node_lease_record_mismatch_fence (which names no author, because the node cannot prove who wrote the record). The fenced state is terminal: only a restart returns the node, through the normal cold-activation path. The testing page shows the kill tests for this path.

RUST_LOG=cello=info,store=debug logs a node_lease_read or node_lease_write event for each store request against the node’s own lease record, with an outcome of found, missing, applied, rejected, or error. An error carries the store failure in an error field; a found carries the record’s generation; a rejected write carries a conflict flag distinguishing a transient store conflict from a lost race. The target costs nothing at other filters.

Stable node identities

Every process session answers peer requests under its node id, and recovery addresses members by the id in their lease. An explicit CELLO_NODE is that id. Without one, the node pins a generated id to node-id in its data directory (beside the follower peerlog) on first boot and reuses it across restarts; an explicit value is recorded there too. A pin that cannot be read at all (I/O error, permissions, a non-file at the pin path) is left alone: the boot starts with an ephemeral id and logs loudly instead of minting over the retained fragments. The pin and the disks it recovers must share a fate: keep the data directory on persistent per-node disk. On an ephemeral disk the pin dies with the container and each restart is a stranger, which is correct only because there is nothing to recover.

A process that answers a seal request for a different session rejects it with 409 WrongTarget and a stale-route marker. Recovery treats that answer as proof the old member is gone: the member stays undecided while its lease is live or inside the witness grace, then the pass proceeds — to a witness, or to the loss record. The log line names the old id, so an operator can restart that address with its old id on the same disk and serve the tail. Never run two processes on one data directory: both would claim the pin and split the session’s fate.

The witness grace

A member whose lease merely lapsed — a restart, a fleet-wide power cycle — is not conclusively gone: its fsync’d fragments boot back in seconds, and declaring loss against boot order would discard acknowledged writes sitting intact on disk. Recovery therefore applies a grace of three TTLs (at least 20 seconds) to member fate: a member that is unreachable, or answered by a stranger, stays undecided until its lease has been dead past the grace, and only then does the pass proceed — to a witness, or to the loss record. While any member is undecided, recovery refuses to seal and says so loudly (refusing to seal while member state remains unverified); a node whose own predecessor is that session waits with it instead of starting. Do not “resolve” this by deleting lease or log records: a deleted lease with surviving session objects declares bounded loss (below); it unblocks nothing faster.

The loss record

When no complete true witness survives for an active log that may hold fleet acknowledgements, recovery writes a loss record beside the log and seals: log/<session>.e<epoch>.loss.json, or log/<node>.loss.json when the lease and record are gone but session objects survive. “Loss is a record, never a prompt”: the pass never asks the operator, and it never silently proceeds. The record names the dead session, its epoch, who declared it, and what was unrecoverable — acknowledged writes in the final flush window may be missing.

After a declared loss the new log epoch restarts the sequence space, so seqs of written-off acknowledged writes can reappear carrying different rows. A client that stored a seq from an acknowledged write can later read a different row under it. Match acknowledged writes by content across a declared loss, never by sequence, and treat the loss record as the marker that the lineage turned over. A returning follower disk still holds its written-off rows, but nothing rejoins them automatically; reconcile them by content against the record.

Alarm discovery and the wake format

SQLite stores the alarm deadline, consumption, retry state, and installation identity. An alarm hint only makes Cello read SQLite; it never authorizes a handler to run.

Each committed installation has an object under wake/entries/, identified by the ownership epoch and a persistent SQLite sequence. An alarm response waits for its publication PUT and the output proof; an update within the same minute also needs a new PUT.

Cleanup publishes a retirement record under wake/retired/ with conditional writes. The record needs a durability proof, a current owner, and, if an alarm remains armed, a confirmed replacement publication. Cleanup deletes only older identities or a proven consumed one, so an old DELETE cannot remove a later installation and the bucket needs no conditional DELETE. Deletes of obsolete publications do not block an alarm response.

Each cleanup pass lists at most 128 objects and processes at most eight cells concurrently, continuing the listing on the next pass and restarting after the last page. A failed DELETE or a late PUT can need another full scan. The interval defaults to 60 seconds; CELLO_WAKER_TICK_MS sets it and the due-scan interval. A large inventory can take many intervals, and each extra node can repeat the same reads and deletes. Each due-scan pass also wakes for the next not-yet-due alarm minute when it lands before the next tick, so a persisted alarm fires on schedule after a restart instead of waiting out the 60 s scan.

A firing alarm that outlives the operation deadline (CELLO_OPERATION_DEADLINE_MS, default 15000 ms) is still running, not failed: the first expiry only marks the firing, and the running copy settles the alarm when it answers, so the first expiry never dispatches a duplicate. A second expiry with the op still outstanding retires the op and re-fires, preserving at-least-once — but the retired handler’s executor task is not cancelled, so the re-fired copy can overlap a handler that is still running. A firing alarm holds its cell out of worker routing until the copy settles or is retired.

wake/format.json selects format 2; wake/waker.json holds the advisory lease for the waker role. A new node initializes an empty fleet or upgrades a stopped v0.4.1 fleet automatically. An unsupported format, or the old wake-format.json, wake-v2/, or wake-retired-v2/ names, prevents startup. Application objects outside the reserved namespaces do not.

Start a fleet with this format

An empty fleet initializes this format automatically. An upgrade from v0.4.1 keeps the bucket and node data directories but needs a stopped fleet, because v0.4.1 cannot read the new discovery entries.

  1. Stop application traffic and deployment writers. Stop every old node and its supervisor, then wait for every node lease to expire.
  2. Back up the bucket and node data. Keep the node names, peer addresses, and data directories. A follower disk can hold acknowledged writes that the bucket does not yet hold.
  3. Prevent the old binaries from restarting or writing to the bucket: revoke their credentials or access. The format marker cannot stop them.
  4. Start the new binary on every node with the same configuration, data, and addresses. Wait for the fleet to become healthy, then resume traffic.

The nodes can start together. A live node lease blocks the migration, and the operator must keep old writers from returning.

The migration preserves databases, node logs, ownership records, deployments, and application objects. It creates a discovery seed for each stored cell, in pages of at most 128 entries. A seed makes recovery derive the installation identity from SQLite without changing the deadline or retry state, and it is removed only after durable recovery, so an interrupted migration cannot discard an alarm. Recovery can load cells with future alarms while it processes seeds.

If a node stops mid-migration, another starting node resumes it. No node serves until the full inventory succeeds, so alarms can run late. Later starts do not migrate again.

To roll back, restore the full stopped-fleet backup; never start an old binary against the upgraded fleet. The restore loses writes made after the backup, so preserve those separately.

Restarts and ownership transfers preserve alarm history. Each restored writer takes a new epoch and keeps the stored sequence and consumed state, so an old mutation cannot target a later installation.

Was this page helpful?