---
title: Fleet and durability
description: Ownership, replication, takeover, staged activation, and bucket-posture group commit over upstream denoland/celld.
---

These patches change how the fleet agrees on who serves a cell, how versions take over serving, and how bucket-posture writes batch. The guarantees themselves are unchanged; see [what Cello guarantees](/concepts/durability). Each entry names its fork patch number.

## Release a version without serving it yet

See [staged versions](/guides/staged-versions) for the workflow:

- `0039` — Staged activation: `cello deploy --no-activate` uploads a version without moving any pointer; `cello activate SCRIPT=VERSION…` serves a multi-script generation with one atomic fleet-pointer write (service bindings resolve from the pointer's component map, queue attachments publish at activate time); `cello discard` removes versions no pointer names.
- `0048` — `cello invoke` sends one HTTP request to a staged script version on one node, served by that version's code with its declared bindings against the real data — so a migration endpoint runs before the activation that would serve it. `POST /invoke` on the internal listener requires the operator token plus the fleet HMAC; a shadow generation serves staged pins overlaid on the live map in fresh pools and retires after the request. Nothing serving moves. Needs every node on this release (older nodes have no `/invoke` route).

## Batch bucket-posture writes (opt-in)

Writes in bucket posture can share one bundle object per flush round instead of one upload per write. It is **off by default**: an unset `CELLO_BUCKET_BUNDLES` disables grouping and only an explicit `1` enables it (`0` still disables; anything else stops startup). `CELLO_BUCKET_BUNDLE_LINGER_MS` (default 0, cap 1000) holds a round open to batch more, but only while more than one cell is dirty — a lone cell never waits.

- `0059` — Layer 1: the sans-I/O core. `BUCKET_FORMAT` becomes 3; a node writes shared bundle proofs only while every live lease reads format 3, judged per feature against its own minimum so the bump never closes paged epochs on a mixed fleet.
- `0060` — Layer 2: the executor. A bucket node with an open gate publishes its bucket-session record before the first bundle-credited ack and credits through the same watermarks as per-cell uploads. Event-driven flush, capped rounds, fleet path untouched, bucket posture only (`CELLO_DURABILITY=bucket`).
- `0061` — The eviction guard proves a journaled epoch with a HEAD on the newest journaled key instead of a prefix LIST (one Class-B HEAD replaces one Class-A LIST per warm guard).
- `0062` — Layer 1 follow-up: the per-round fence credit predicate plus the output-gate settlement it feeds, proven by a two-arm check (session record after the PUT, per-cell fence read after the proof) with proof-then-read ordering.
- `0063` — Layer 3: the successor fold. A bucket session's acked rows are restorable after death, restart, or quiet handoff; a live donor session's retained uncovered bundle rows fold before the restore.
- `0064` — Layer 4: drain, GC, and graceful close for bucket sessions. Eviction writes one per-cell snapshot from the complete local DB; `gc_bundles` deletes only fully-covered bundles with single-key `DeleteObject`s; graceful close seals only when every acked row is per-cell.
- `0070` / `0071` / `0075` — Rollout, the hot-cell double-write fix, and the final default-off. `0071` also quiets idle bundle GC: once a full-prefix scan leaves zero bundles and no bundle PUT was attempted since, GC skips its LIST until the next write — an idle bucket node costs one discovery LIST and then nothing.

Measured at `DELAY_MS=100`, release builds, 1 node: the 200-agent workload drops from 1934 to 1140 Class-A requests with grouping on, but write p50 rises from 210 to 445 ms; a hot cell with 16 writers rises from 235 to 432 ms p50 with ~45% fewer turns. Enable `CELLO_BUCKET_BUNDLES=1` for request-cost-sensitive workloads that tolerate about one extra round trip per durable write. A `.10` binary recovers a bucket a bundling `.11` wrote (whole-prefix fold, no loss declared), so a downgrade is safe. Do not enable production grouping until every live lease reads format 3 (every node on the release that publishes it).

## Adopt ported coordination fixes

`0025`–`0034` port correctness fixes from [meln1k/celld](https://github.com/meln1k/celld) (Apache-2.0, same author line kept with a `Ported-from` trailer; simulator and docs files not ported), and `0035` follows them up:

- `0025` — A newer ownership release retires the in-flight one it supersedes (debug `indexed cell op has no owning slot` panic and release `cell_ops` leak fixed).
- `0026` — Fence settles startup confirmation reads (debug `confirming cell left the owner read in Fenced` panic on the halting event fixed).
- `0027` — The alarm observer carries `WakeCoverage`, and wake maintenance reports `Confirmed` once the flusher covers the current installation, so a restored alarm no longer pins its cell resident forever.
- `0028` — A reset settles an alarm handler still running without a barrier as a failed handler (re-armed, op retired, wake entry kept).
- `0029` — The near-alarm eviction refusal applies only when the node has room to spare, so a full node sheds instead of parking a waiter.
- `0030` — A drain counts every handoff against `max_releases`, its pump reads the live count, wake hints and alarm timers wait out a balancing move, fence clears the handoff mark, and preserve settles a confirming read.
- `0031` — A reset drops the cell's sockets with its runtime, and a hibernatable socket's events wait for the restarted runtime on a quiescing cell (swap deadlock fixed).
- `0032` — A takeover from an expired lease that never opened a log waits for recovery, which CAS-seals the absent log so a stale session cannot open one through a renewal.
- `0033` / `0034` — 51 behavior-neutral code probes counting the core's rare branches, then the rule they enable: a cell whose last restore failed keeps its owner record — drain and balancer skip it — so no peer restores it from the bucket alone and loses the acknowledged tail.
- `0035` — Wake-coverage follow-ups: the startup alarm observation lands under the registry lock after registration, coverage compares alarm installations instead of snapshots, and a coverage-only confirmation no longer cancels a non-drain eviction.

## Hand over dead owners without losing writes

See [run a fleet](/guides/run-fleet) for the operations these patches harden:

- `0009` — `cello deploy` treats a lost content-tag race as success: when tagging the built image fails with `AlreadyExists`, the tag is re-inspected and the deploy continues if it names the same content.
- `0088` — A stale-target 409 proves a member is gone: only a 409 `WrongTarget` carrying the peer-auth gate's stale-route marker counts; every other seal failure stays inconclusive and retries ([upstream #273](https://github.com/denoland/celld/issues/273)).
- `0089` — The renewal lease readback is bounded to the remaining authority (`authority_expires_ms`), so a stalled store cannot wedge every renewal until the fence timer ([upstream #264](https://github.com/denoland/celld/issues/264)).
- `0090` — The first alarm-deadline expiry only marks the firing; the running copy settles the alarm, so a slow handler is never doubled. A second expiry with the op still outstanding retires and re-fires, keeping alarms at-least-once ([upstream #256](https://github.com/denoland/celld/issues/256)).
- `0091` — Follower tails stream under a 10 s progress timeout with a 5 s connect budget, and the transfer beats the recovery claim while streaming, so a healthy-but-slow tail holds its claim ([upstream #250](https://github.com/denoland/celld/issues/250)).
- `0092` — The node-session id is pinned in a `node-id` file beside the follower fragments (tmp, fsync, rename, dir fsync), so a restart reuses its id instead of stranding leases behind `WrongTarget`. An explicit `CELLO_NODE` wins outright and is recorded; a corrupt pin warns and regenerates. Set `CELLO_WATCH` to a persistent directory to keep the id — the PID-scoped default temp dir mints a new one every boot ([upstream #249](https://github.com/denoland/celld/issues/249)).
- `0093` — Transient CAS conflicts retry twice with the same guard before paying a readback round trip ([upstream #239](https://github.com/denoland/celld/issues/239)).
- `0094` — An absent session record counts as closed: the fragments are garbage no new epoch will adopt, and the sweep declares the loss in a `log/<node>.loss.json` record — bounded and loud instead of silent ([upstream #245](https://github.com/denoland/celld/issues/245)).
- `0095` — A degraded ensemble whose membership still matches its record resumes without an epoch step, up to 3 consecutive resumes before stepping anyway so a healed member can rejoin ([upstream #268](https://github.com/denoland/celld/issues/268)).
- `0096` — The periodic scan sleeps until the next upcoming alarm minute, not the next 60 s tick ([upstream #308](https://github.com/denoland/celld/issues/308)).
- `0097` — Docs only: guarantees cover the pinned node-session id, witness grace, the loss record, same-guard conflict retries, the authority-bounded readback, the mark-first alarm deadline, and wake-for-the-upcoming-minute timing ([upstream #244](https://github.com/denoland/celld/issues/244)).
- `0103` — Fragments of record-less sessions are collected once absence is terminal (no lease at all, or a superseding generation); a live logless lease's fragments are kept ([upstream #259](https://github.com/denoland/celld/issues/259)).
- `0105` — After an unclean restart the boot prunes live-named epochs (only a clean reload resumes one) and replays the take-once boot list for ownership confirmation ([upstream #290](https://github.com/denoland/celld/issues/290)).
- `0126` — Transient bucket write conflicts (`OperationAborted`) retry with backoff instead of surfacing a lost-race error; the match reads the tagged service code, never the object key ([upstream #283](https://github.com/denoland/celld/issues/283)).
- `0129` — Dead-owner takeovers route with fleet capacity: a zero-capacity node hands the cell to a peer instead of piling on ([upstream #263](https://github.com/denoland/celld/issues/263)).
