---
title: Testing
description: How Cello tests your code the same way on both runtimes, and how it checks its own coordination.
---

Cello makes three promises:

- An acknowledged write is durable.
- A cell has one writer at a time.
- Code written for Cloudflare Workers and Durable Objects runs the same on
  Cello, within the supported APIs ([compatibility](/reference/cloudflare-compat)
  lists what BroadcastChannel and the other gaps do instead).

You test the API contract by running the same program twice, the
coordination protocol by checking every reachable state, and the full system
by breaking live fleets.

## Run the same program on two runtimes

Conformance runs your Workers and Durable Objects programs on workerd,
Cloudflare's production runtime, and on Cello, and the outputs must match.
Workerd supplies the reference output, so the expected behavior never
depends on Cello.

Each new API adds fixtures to the corpus. Cello also ports the workerd
Durable Objects contract tests, its web-platform globals tests, and the
upstream Web Platform Tests. Before a release, replay storage, SQL, alarm,
stream, WebSocket, and lifecycle scenarios through the full `cello` binary
in each deployment mode.

## Check every reachable state

The coordination protocol is specified in TLA+. Heyang Zhou wrote the
specifications against upstream denoland/celld. His model checking found
four bugs and a split-brain that lost an acknowledged write. All are fixed.
None had surfaced in review or testing.

At a small configuration, the checker visits every reachable state. The
model grants a linearizable object store and perfect shared clocks, so a
violation needs no clock skew or storage anomaly. The invariants are one
writer for each epoch and no lost acknowledged write. The checker also
verifies that the epoch in the key stops a stale owner's late writes from
losing an acknowledged write.

Each configuration pins an expected verdict, and most verdicts are
failures: each models a past protocol bug or a deliberately broken checker,
and must produce its counterexample.

A redesign of the write-acknowledgment fence was checked before it was
built. The check found an eight-state counterexample against the old
version: a dormant cell resumes at its old epoch while its release is in
flight, acknowledges a write, and the following takeover restores without
it. The fix shipped, and the counterexample stays as a pinned failure.

The checker also removed code. Cello once sealed a cell's durable history
at restore. The verdicts showed that the seal only prevented the return of
an unacknowledged write, which Cello does not promise, and that it could
turn a recoverable ordering error into permanent data loss.

The specifications are updated by hand with protocol changes. They do not
run in continuous integration. A separate record tracks what the model does
not yet describe, including guarantees weaker than the code's.

## Drive the protocol with hostile schedules

The dangerous bugs live in coordination: a crash during an ownership
handoff, a lease renewal that races a takeover, an alarm against a
partially restored cell. These windows are too narrow and too rare to wait
for.

The coordination protocol is therefore a
[pure decision core](https://github.com/ryuzcorp/cello/tree/main/crates/logic)
with no I/O. The clock, the randomness, and the object store are
interfaces that a simulator drives. The simulated store injects latency,
compare-and-swap races, and lost responses. The clocks drift, and a node
can crash at each await point. Scripted adversaries play the cells, such as
a handler that never returns or a write stream that stops halfway. V8 is
not deterministic, so it stays out of the simulation.

A seeded scheduler drives each run, so a failure reproduces exactly. The
checks cover safety (two writers in one epoch, a lost acknowledged write,
an expired lease that returns) and liveness (each armed alarm fires,
ownership settles on one node after a crash). A property must survive tens
of thousands of seeds, and the core protocols have run through millions of
schedules. Deliberately broken protocol variants verify that the checkers
catch the faults.

## Break live fleets

Simulation cannot see real S3 tail latency, real kernel and filesystem
behavior, or V8 under memory pressure. A permanent fleet lab runs standard
VMs from standard providers against a real bucket. Workloads rotate: chat
rooms with many WebSocket connections, working sets that shift across tens
of thousands of checksummed cells, deployment cutovers under load, and
nodes filled to the memory limit. The lab qualifies each release. Each
run's configuration, verification sweeps, node journals, kernel logs, and
phase timings are archived, including failed runs.

Faults land between verification passes. A pass fetches each cell through
different nodes and compares the status, the body, and the full message
ledger. A cell can be briefly unavailable while ownership moves, but its
committed state must stay complete and a live node must serve it again.

The scenarios:

- `SIGKILL` a node mid-write-stream and delete its local database, so
  recovery comes only from the bucket. Every acknowledged write returns,
  because the output gate held each response until the write was durable.
- Freeze an owner, write to its cells through other nodes, and unfreeze
  it. The node sees that its lease moved and refuses to serve the old
  state. Each write lands exactly once.
- Cut a node off from the bucket. It fences itself.
- Throttle the bucket to 429 on every request. The engine slows to the
  store's rate and does not amplify the throttle.
- Stop a full host at the provider level mid-workload. Its cells move to
  other nodes, and the returning host rejoins with no duplicate residency.

Across every run of every scenario, the verification sweeps show zero body
faults, zero status faults, and zero lost messages.

## Read the numbers we trust

- **The epoch fence holds under contention.** Five hundred concurrent
  claimants made 5,500 attempts on the same cells: one writer for each
  epoch, zero violations.
- **A warm resident request is local.** A request to a resident cell does
  zero bucket operations and returns in p50 ~1.1 ms and p99 ~7 ms (a
  fixed-host measurement). Only a cold activation touches object storage.
- **A durable write waits for a durability proof.** A single node proves
  each write through the bucket, so one storage round trip is the minimum.
  A fleet of two or more nodes can instead prove a write when each
  follower holds it on disk. A lab fleet measured about 600 ms for a
  bucket proof and about 25 ms for a fleet proof. The bucket upload races
  every fleet proof, so a slow follower cannot make a write slower than a
  bucket proof. Concurrent writes to one cell share one upload.
- **A restore is normal work.** Placement treats the restore of an
  inactive cell as ordinary work. The measured restore times come from the
  retired external replicator, so this page gives no number until a fleet
  run measures `cello-ltx`.
- **Ten small nodes held real scale.** Ten nodes, each with 4 vCPU and 8
  GB, held 10,000 resident cells and 20,000 concurrent WebSocket
  connections. With two of the ten nodes stopped, every cell was available
  again on another node in ~11 s at the tail (with reserve headroom).

## Mind the known failure edge

A fleet at its resident limit has no space for a lost node's cells, so
losing multiple nodes degrades service. Recovery is tested with and
without reserve capacity to measure this limit.

Report a schedule that breaks a guarantee, an untested fault, or a
measurement that you cannot reproduce in the
[issue tracker](https://github.com/ryuzcorp/cello/issues).
