Skip to content
Cello
Esc
↑↓navigate↵open⌘Jpreview
On this page

Fleet and durability

Ownership, replication, takeover, staged activation, and bucket-posture group commit over upstream denoland/celld.

Patches to the fleet core: lease ownership and takeover, staged deployment activation, request invocation against staged versions, and the opt-in bucket-posture group commit. Includes the ported coordination-core fixes (0025–0035). Each entry names its fork patch number. The guarantees themselves are unchanged; see What Cello guarantees.

Staged activation and invocation

  • 0039 — Staged activation: cello deploy --no-activate uploads a version without moving any pointer; cello activate SCRIPT=VERSION… serves a multi-script generation with one atomic fleet-pointer write (service bindings resolve from the pointer’s component map, queue attachments publish at activate time); cello discard removes versions no pointer names. See Invoke a staged version.
  • 0048 — cello invoke sends one HTTP request to a staged script version on one node, served by that version’s code with its declared bindings against the real data — so a migration endpoint runs before the activation that would serve it. POST /invoke on the internal listener requires the operator token plus the fleet HMAC; a shadow generation serves staged pins overlaid on the live map in fresh pools and retires after the request. Nothing serving moves. Needs every node on this release (older nodes have no /invoke route).

Bucket-posture group commit (opt-in)

Writes in bucket posture can share one bundle object per flush round instead of one upload per write. It is off by default: an unset CELLO_BUCKET_BUNDLES disables grouping and only an explicit 1 enables it (0 still disables; anything else stops startup). CELLO_BUCKET_BUNDLE_LINGER_MS (default 0, cap 1000) holds a round open to batch more, but only while more than one cell is dirty — a lone cell never waits.

  • 0059 — Layer 1: the sans-I/O core. BUCKET_FORMAT becomes 3; a node writes shared bundle proofs only while every live lease reads format 3, judged per feature against its own minimum so the bump never closes paged epochs on a mixed fleet.
  • 0060 — Layer 2: the executor. A bucket node with an open gate publishes its bucket-session record before the first bundle-credited ack and credits through the same watermarks as per-cell uploads. Event-driven flush, capped rounds, fleet path untouched, bucket posture only (CELLO_DURABILITY=bucket).
  • 0061 — The eviction guard proves a journaled epoch with a HEAD on the newest journaled key instead of a prefix LIST (one Class-B HEAD replaces one Class-A LIST per warm guard).
  • 0062 — Layer 1 follow-up: the per-round fence credit predicate plus the output-gate settlement it feeds, proven by a two-arm check (session record after the PUT, per-cell fence read after the proof) with proof-then-read ordering.
  • 0063 — Layer 3: the successor fold. A bucket session’s acked rows are restorable after death, restart, or quiet handoff; a live donor session’s retained uncovered bundle rows fold before the restore.
  • 0064 — Layer 4: drain, GC, and graceful close for bucket sessions. Eviction writes one per-cell snapshot from the complete local DB; gc_bundles deletes only fully-covered bundles with single-key DeleteObjects; graceful close seals only when every acked row is per-cell.
  • 0070 / 0071 / 0075 — Rollout, the hot-cell double-write fix, and the final default-off. 0071 also quiets idle bundle GC: once a full-prefix scan leaves zero bundles and no bundle PUT was attempted since, GC skips its LIST until the next write — an idle bucket node costs one discovery LIST and then nothing.

Measured at DELAY_MS=100, release builds, 1 node: the 200-agent workload drops from 1934 to 1140 Class-A requests with grouping on, but write p50 rises from 210 to 445 ms; a hot cell with 16 writers rises from 235 to 432 ms p50 with ~45% fewer turns. Enable CELLO_BUCKET_BUNDLES=1 for request-cost-sensitive workloads that tolerate about one extra round trip per durable write. A .10 binary recovers a bucket a bundling .11 wrote (whole-prefix fold, no loss declared), so a downgrade is safe. Layers pending in .11 shipped in .12–.13 form; do not enable production grouping until every live lease reads format 3 (every node on the release that publishes it).

Ported coordination-core fixes

0025–0034 port correctness fixes from meln1k/celld (Apache-2.0, same author line kept with a Ported-from trailer; simulator and docs files not ported), and 0035 follows them up:

  • 0025 — A newer ownership release retires the in-flight one it supersedes (debug indexed cell op N has no owning slot panic and release cell_ops leak fixed).
  • 0026 — Fence settles startup confirmation reads (debug confirming cell left the owner read in Fenced panic on the halting event fixed).
  • 0027 — The alarm observer carries WakeCoverage, and wake maintenance reports Confirmed once the flusher covers the current installation, so a restored alarm no longer pins its cell resident forever.
  • 0028 — A reset settles an alarm handler still running without a barrier as a failed handler (re-armed, op retired, wake entry kept).
  • 0029 — The near-alarm eviction refusal applies only when the node has room to spare, so a full node sheds instead of parking a waiter.
  • 0030 — A drain counts every handoff against max_releases, its pump reads the live count, wake hints and alarm timers wait out a balancing move, fence clears the handoff mark, and preserve settles a confirming read.
  • 0031 — A reset drops the cell’s sockets with its runtime, and a hibernatable socket’s events wait for the restarted runtime on a quiescing cell (swap deadlock fixed).
  • 0032 — A takeover from an expired lease that never opened a log waits for recovery, which CAS-seals the absent log so a stale session cannot open one through a renewal.
  • 0033 / 0034 — 51 behavior-neutral code probes counting the core’s rare branches, then the rule they enable: a cell whose last restore failed keeps its owner record — drain and balancer skip it — so no peer restores it from the bucket alone and loses the acknowledged tail.
  • 0035 — Wake-coverage follow-ups: the startup alarm observation lands under the registry lock after registration, coverage compares alarm installations instead of snapshots, and a coverage-only confirmation no longer cancels a non-drain eviction.

Fleet operations and takeover

  • 0009 — cello deploy treats a lost content-tag race as success: when tagging the built image fails with AlreadyExists, the tag is re-inspected and the deploy continues if it names the same content.
  • 0088 — A stale-target 409 proves a member is gone: only a 409 WrongTarget carrying the peer-auth gate’s stale-route marker counts; every other seal failure stays inconclusive and retries (upstream #273).
  • 0089 — The renewal lease readback is bounded to the remaining authority (authority_expires_ms), so a stalled store cannot wedge every renewal until the fence timer (upstream #264).
  • 0090 — The first alarm-deadline expiry only marks the firing; the running copy settles the alarm, so a slow handler is never doubled. A second expiry with the op still outstanding retires and re-fires, keeping alarms at-least-once (upstream #256).
  • 0091 — Follower tails stream under a 10 s progress timeout with a 5 s connect budget, and the transfer beats the recovery claim while streaming, so a healthy-but-slow tail holds its claim (upstream #250).
  • 0092 — The node-session id is pinned in a node-id file beside the follower fragments (tmp, fsync, rename, dir fsync), so a restart reuses its id instead of stranding leases behind WrongTarget. An explicit CELLO_NODE wins outright and is recorded; a corrupt pin warns and regenerates. Set CELLO_WATCH to a persistent directory to keep the id — the PID-scoped default temp dir mints a new one every boot (upstream #249).
  • 0093 — Transient CAS conflicts retry twice with the same guard before paying a readback round trip (upstream #239).
  • 0094 — An absent session record counts as closed: the fragments are garbage no new epoch will adopt, and the sweep declares the loss in a log/<node>.loss.json record — bounded and loud instead of silent (upstream #245).
  • 0095 — A degraded ensemble whose membership still matches its record resumes without an epoch step, up to 3 consecutive resumes before stepping anyway so a healed member can rejoin (upstream #268).
  • 0096 — The periodic scan sleeps until the next upcoming alarm minute, not the next 60 s tick (upstream #308).
  • 0097 — Docs only: guarantees cover the pinned node-session id, witness grace, the loss record, same-guard conflict retries, the authority-bounded readback, the mark-first alarm deadline, and wake-for-the-upcoming-minute timing (upstream #244).
  • 0103 — Fragments of record-less sessions are collected once absence is terminal (no lease at all, or a superseding generation); a live logless lease’s fragments are kept (upstream #259).
  • 0105 — After an unclean restart the boot prunes live-named epochs (only a clean reload resumes one) and replays the take-once boot list for ownership confirmation (upstream #290).
  • 0126 — Transient bucket write conflicts (OperationAborted) retry with backoff instead of surfacing a lost-race error; the match reads the tagged service code, never the object key (upstream #283).
  • 0129 — Dead-owner takeovers route with fleet capacity: a zero-capacity node hands the cell to a peer instead of piling on (upstream #263).

Was this page helpful?