Fleet and durability
Ownership, replication, takeover, staged activation, and bucket-posture group commit over upstream denoland/celld.
Patches to the fleet core: lease ownership and takeover, staged
deployment activation, request invocation against staged versions, and
the opt-in bucket-posture group commit. Includes the ported
coordination-core fixes (0025–0035). Each entry names its fork patch
number. The guarantees themselves are unchanged; see
What Cello guarantees.
Staged activation and invocation
0039— Staged activation:cello deploy --no-activateuploads a version without moving any pointer;cello activate SCRIPT=VERSION…serves a multi-script generation with one atomic fleet-pointer write (service bindings resolve from the pointer’s component map, queue attachments publish at activate time);cello discardremoves versions no pointer names. See Invoke a staged version.0048—cello invokesends one HTTP request to a staged script version on one node, served by that version’s code with its declared bindings against the real data — so a migration endpoint runs before the activation that would serve it.POST /invokeon the internal listener requires the operator token plus the fleet HMAC; a shadow generation serves staged pins overlaid on the live map in fresh pools and retires after the request. Nothing serving moves. Needs every node on this release (older nodes have no/invokeroute).
Bucket-posture group commit (opt-in)
Writes in bucket posture can share one bundle object per flush round
instead of one upload per write. It is off by default: an unset
CELLO_BUCKET_BUNDLES disables grouping and only an explicit 1
enables it (0 still disables; anything else stops startup).
CELLO_BUCKET_BUNDLE_LINGER_MS (default 0, cap 1000) holds a round
open to batch more, but only while more than one cell is dirty — a lone
cell never waits.
0059— Layer 1: the sans-I/O core.BUCKET_FORMATbecomes 3; a node writes shared bundle proofs only while every live lease reads format 3, judged per feature against its own minimum so the bump never closes paged epochs on a mixed fleet.0060— Layer 2: the executor. A bucket node with an open gate publishes its bucket-session record before the first bundle-credited ack and credits through the same watermarks as per-cell uploads. Event-driven flush, capped rounds, fleet path untouched, bucket posture only (CELLO_DURABILITY=bucket).0061— The eviction guard proves a journaled epoch with a HEAD on the newest journaled key instead of a prefix LIST (one Class-B HEAD replaces one Class-A LIST per warm guard).0062— Layer 1 follow-up: the per-round fence credit predicate plus the output-gate settlement it feeds, proven by a two-arm check (session record after the PUT, per-cell fence read after the proof) with proof-then-read ordering.0063— Layer 3: the successor fold. A bucket session’s acked rows are restorable after death, restart, or quiet handoff; a live donor session’s retained uncovered bundle rows fold before the restore.0064— Layer 4: drain, GC, and graceful close for bucket sessions. Eviction writes one per-cell snapshot from the complete local DB;gc_bundlesdeletes only fully-covered bundles with single-keyDeleteObjects; graceful close seals only when every acked row is per-cell.0070/0071/0075— Rollout, the hot-cell double-write fix, and the final default-off.0071also quiets idle bundle GC: once a full-prefix scan leaves zero bundles and no bundle PUT was attempted since, GC skips its LIST until the next write — an idle bucket node costs one discovery LIST and then nothing.
Measured at DELAY_MS=100, release builds, 1 node: the 200-agent
workload drops from 1934 to 1140 Class-A requests with grouping on, but
write p50 rises from 210 to 445 ms; a hot cell with 16 writers rises
from 235 to 432 ms p50 with ~45% fewer turns. Enable
CELLO_BUCKET_BUNDLES=1 for request-cost-sensitive workloads that
tolerate about one extra round trip per durable write. A .10 binary
recovers a bucket a bundling .11 wrote (whole-prefix fold, no loss
declared), so a downgrade is safe. Layers pending in .11 shipped in
.12–.13 form; do not enable production grouping until every live
lease reads format 3 (every node on the release that publishes it).
Ported coordination-core fixes
0025–0034 port correctness fixes from
meln1k/celld (Apache-2.0, same
author line kept with a Ported-from trailer; simulator and docs
files not ported), and 0035 follows them up:
0025— A newer ownership release retires the in-flight one it supersedes (debugindexed cell op N has no owning slotpanic and releasecell_opsleak fixed).0026— Fence settles startup confirmation reads (debugconfirming cell left the owner read in Fencedpanic on the halting event fixed).0027— The alarm observer carriesWakeCoverage, and wake maintenance reportsConfirmedonce the flusher covers the current installation, so a restored alarm no longer pins its cell resident forever.0028— A reset settles an alarm handler still running without a barrier as a failed handler (re-armed, op retired, wake entry kept).0029— The near-alarm eviction refusal applies only when the node has room to spare, so a full node sheds instead of parking a waiter.0030— A drain counts every handoff againstmax_releases, its pump reads the live count, wake hints and alarm timers wait out a balancing move, fence clears the handoff mark, and preserve settles a confirming read.0031— A reset drops the cell’s sockets with its runtime, and a hibernatable socket’s events wait for the restarted runtime on a quiescing cell (swap deadlock fixed).0032— A takeover from an expired lease that never opened a log waits for recovery, which CAS-seals the absent log so a stale session cannot open one through a renewal.0033/0034— 51 behavior-neutral code probes counting the core’s rare branches, then the rule they enable: a cell whose last restore failed keeps its owner record — drain and balancer skip it — so no peer restores it from the bucket alone and loses the acknowledged tail.0035— Wake-coverage follow-ups: the startup alarm observation lands under the registry lock after registration, coverage compares alarm installations instead of snapshots, and a coverage-only confirmation no longer cancels a non-drain eviction.
Fleet operations and takeover
0009—cello deploytreats a lost content-tag race as success: when tagging the built image fails withAlreadyExists, the tag is re-inspected and the deploy continues if it names the same content.0088— A stale-target 409 proves a member is gone: only a 409WrongTargetcarrying the peer-auth gate’s stale-route marker counts; every other seal failure stays inconclusive and retries (upstream #273).0089— The renewal lease readback is bounded to the remaining authority (authority_expires_ms), so a stalled store cannot wedge every renewal until the fence timer (upstream #264).0090— The first alarm-deadline expiry only marks the firing; the running copy settles the alarm, so a slow handler is never doubled. A second expiry with the op still outstanding retires and re-fires, keeping alarms at-least-once (upstream #256).0091— Follower tails stream under a 10 s progress timeout with a 5 s connect budget, and the transfer beats the recovery claim while streaming, so a healthy-but-slow tail holds its claim (upstream #250).0092— The node-session id is pinned in anode-idfile beside the follower fragments (tmp, fsync, rename, dir fsync), so a restart reuses its id instead of stranding leases behindWrongTarget. An explicitCELLO_NODEwins outright and is recorded; a corrupt pin warns and regenerates. SetCELLO_WATCHto a persistent directory to keep the id — the PID-scoped default temp dir mints a new one every boot (upstream #249).0093— Transient CAS conflicts retry twice with the same guard before paying a readback round trip (upstream #239).0094— An absent session record counts as closed: the fragments are garbage no new epoch will adopt, and the sweep declares the loss in alog/<node>.loss.jsonrecord — bounded and loud instead of silent (upstream #245).0095— A degraded ensemble whose membership still matches its record resumes without an epoch step, up to 3 consecutive resumes before stepping anyway so a healed member can rejoin (upstream #268).0096— The periodic scan sleeps until the next upcoming alarm minute, not the next 60 s tick (upstream #308).0097— Docs only: guarantees cover the pinned node-session id, witness grace, the loss record, same-guard conflict retries, the authority-bounded readback, the mark-first alarm deadline, and wake-for-the-upcoming-minute timing (upstream #244).0103— Fragments of record-less sessions are collected once absence is terminal (no lease at all, or a superseding generation); a live logless lease’s fragments are kept (upstream #259).0105— After an unclean restart the boot prunes live-named epochs (only a clean reload resumes one) and replays the take-once boot list for ownership confirmation (upstream #290).0126— Transient bucket write conflicts (OperationAborted) retry with backoff instead of surfacing a lost-race error; the match reads the tagged service code, never the object key (upstream #283).0129— Dead-owner takeovers route with fleet capacity: a zero-capacity node hands the cell to a peer instead of piling on (upstream #263).