Operations
How to configure storage, deploy, develop, and run a Cello fleet. For a new machine, start at Install.
Configure object storage
The s3://, gs://, and az:// scheme names are case-insensitive, and
Cello ignores leading and trailing whitespace in CELLO_BUCKET.
For an S3-compatible bucket, Cello uses the standard AWS credential chain, including an EC2 instance role and Amazon EKS Pod Identity. For Cloudflare R2, create a bucket and an S3 API token with access to it, and set:
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=auto
export S3_ENDPOINT=https://ACCOUNT_ID.r2.cloudflarestorage.com
export CELLO_BUCKET=s3://YOUR-BUCKET
For Google Cloud Storage, Cello uses Application Default Credentials. Run
gcloud auth application-default login, or point
GOOGLE_APPLICATION_CREDENTIALS at a service-account key. Then set
CELLO_BUCKET=gs://YOUR-BUCKET. On Compute Engine, the default instance
scope permits only storage reads, so create the instance with the
cloud-platform scope.
For Azure Blob Storage, the bucket name is the container:
export AZURE_STORAGE_ACCOUNT_NAME=YOUR-ACCOUNT
export AZURE_STORAGE_ACCOUNT_KEY=...
export CELLO_BUCKET=az://YOUR-CONTAINER
You must configure exactly one Azure credential family: an account key, a
managed identity, or a workload identity. A system-assigned managed
identity needs only the account name. A user-assigned managed identity
needs exactly one of AZURE_CLIENT_ID, AZURE_OBJECT_ID, and
AZURE_MSI_RESOURCE_ID. An identity needs the
Storage Blob Data Contributor
role or equal permissions. A managed identity works on an Azure VM and an
AKS node, but not on App Service or Container Apps; use a workload
identity or an account key there. The AKS workload identity environment
(AZURE_AUTHORITY_HOST, AZURE_CLIENT_ID, AZURE_TENANT_ID,
AZURE_FEDERATED_TOKEN_FILE) must use the public authority host,
https://login.microsoftonline.com. Cello rejects every other recognized
Azure configuration variable, such as another credential source or an
endpoint override. For local development against Azurite, set
AZURE_STORAGE_USE_EMULATOR=true. Cello does not qualify Azurite for a
fleet.
A gs:// or az:// bucket takes no S3_ENDPOINT and no AWS_*
credentials, and Cello ignores the storage region.
The bucket credentials give full control of the fleet, so keep them safe.
A bucket value can add a key prefix, s3://YOUR-BUCKET/PREFIX, so two
fleets can share one bucket. Without a prefix, the objects stay at the
bucket root.
The store must provide conditional writes, exact ranged reads, and read-after-write consistency. Amazon S3, Cloudflare R2, Google Cloud Storage, Tigris, and Azure Blob Storage qualify; Backblaze B2, Hetzner, and DigitalOcean Spaces do not. MinIO (community edition) passes the storage test but is not qualified for production; do not use RELEASE.2025-09-06T17-38-46Z, which rejects the first deploy (denoland/celld#162). See what Cello guarantees.
Deploy an application
Install esbuild on PATH if the project contains Worker code. Then run
cello deploy in a Wrangler project. On Linux, esbuild runs under
Landlock with reads confined to the project directory and its read roots, so an import,
input source map, or tsconfig.json extends outside it fails like a
missing file (whether an outside directory exists can still show in the
error). It must be the self-contained release binary, not a wrapper
script. A deploy reads its project only inside the config’s directory;
--read-root DIR (repeatable) names further directories that esbuild’s
inputs and containers[].image Dockerfiles with their build contexts may
come from, such as a monorepo root whose node_modules sit above the
config. Only the operator sets it, on the command line. A Dockerfile
builds with its own directory as the context, so one placed directly at a
read root sends that whole root to the builder. A kernel without Landlock runs it unconfined, with a warning:
bundled files are still checked against the project directory.
git clone https://github.com/ryuzcorp/cello
cd cello/examples/counter
cello deploy . \
--bucket "$CELLO_BUCKET" \
--endpoint "$S3_ENDPOINT" \
--region "$AWS_REGION"
cello deploy accepts module Workers, Durable Object bindings, service
bindings, variables, cron triggers, D1 databases, KV namespaces, Queues,
R2 buckets, Workflows, WebAssembly modules, and static assets. An unknown
Wrangler configuration key stops the deploy. See
Cloudflare compatibility for the complete boundary.
A node verifies the SHA-256 digest of each module before it builds a
deployment. Pass --sourcemap PATH to also write the Worker’s source
map to a local file for stack-trace symbolication: the esbuild bundle’s
external map, or the no_bundle entry’s sibling <entry>.map. The
deployed bytes are identical with or without it, and asset-only and
Python projects, and no_bundle entries without a sibling map, warn
and write nothing. A sibling map never fails the deploy: one that is a
symlink, not a regular file, larger than 64 MiB, outside the project
root, or not a v3 source map is skipped with a warning.
A running node does not restart for a new deployment. Each node reads
deploy/current.json every CELLO_DEPLOY_POLL_S seconds (default 30) and
adopts a new deployment in place. POST /reload on the internal listener
adopts it now, and also rebuilds an unchanged deployment. A request that
started on the previous deployment finishes on it. A deployment that does
not build leaves the current one serving, and the node reports the error
in its log and in the /reload response.
An inactive Durable Object runs the new deployment at its next activation.
A resident one moves when no request, alarm handler, or regular WebSocket
is active, and it keeps its storage and its hibernatable WebSockets. A
request that arrives during the move waits for the new code. An outbound
call from a request that already runs in the object does not wait, so that
request can finish on the old code. After
CELLO_DEPLOY_MAX_AGE_S seconds (default 60; 0 forces at once), Cello
forces the move: it cancels the running work and closes regular WebSockets
with code 1012, as Cloudflare does. During the move, a request on one
deployment can call a Durable Object on the other, so two adjacent
versions must accept each other’s calls. /state reports the deployment
of each resident object.
Develop an application locally
cello dev [PROJECT_DIR_OR_CONFIG]
The command starts a local object store, deploys the application, and runs
one node. It does not need Docker or a cloud bucket. The Worker listener
is http://127.0.0.1:9876 by default.
| flag | effect |
|---|---|
--port PORT |
The Worker listener port |
--host ADDR |
The Worker listener interface. The operator listener stays on loopback |
--logs |
Show the node warning and information logs |
--clean |
Delete .cello/dev before the start |
--state DIR |
Keep local state in DIR instead of PROJECT/.cello/dev (CELLO_DEV_STATE sets the same) |
--watch-ignore GLOB |
Ignore one more project-relative glob. Repeatable. Not valid with --no-watch |
--no-watch |
Disable automatic builds and restarts |
Set NO_COLOR to disable color, or FORCE_COLOR to enable it when the
output is not a terminal. NO_COLOR takes priority.
The command reads .dev.vars beside the Wrangler configuration, as
wrangler dev does. Without .dev.vars, it reads .env and then
.env.local, which overrides .env. Each NAME=value entry becomes a
Worker variable and overrides vars. A line can start with export, and
a quoted value can span lines, for example a PEM key. Comments after a
value and variable references are not supported. In .env files, the
command skips, with a warning, an entry that is not a valid binding name
or that collides with another binding; in .dev.vars such an entry stops
the build. cello deploy does not read these files. Add .dev.vars,
.env, .env.local, and .cello/ to .gitignore.
The command keeps the local state in PROJECT/.cello/dev (--state DIR or
CELLO_DEV_STATE moves it elsewhere, so backups and search tools skip it),
across restarts and configuration changes. Cello does not migrate stored
state, so a stored value from an earlier configuration can make the
application fail with an error that looks unrelated. Use --clean to start
from an empty state. --clean deletes nothing else under .cello, and over
an explicit --state directory it deletes only what Cello created: a missing
directory is already clean, and a foreign one is refused with its contents
intact. cello d1 --local resolves the same directory, so it follows the
override; pass the same --state DIR (or CELLO_DEV_STATE) when the node
runs with an explicit state directory.
The command watches the project directory and rebuilds on a source,
configuration, or dotenv change. The current application serves during the
build, and a failed build does not replace it. The watcher ignores
.cello, .git, .wrangler, node_modules, and target at each depth,
and it does not watch files outside the project.
The local object store is not available to a fleet node or an operator subcommand; they need a supported cloud bucket.
Operate D1, KV, and R2
These commands use the fleet bucket. cello d1 and cello kv reach the
database cell through a node.
cello d1 migrations apply ledger --bucket "$CELLO_BUCKET"
cello kv bulk put sessions wrangler-export.json --bucket "$CELLO_BUCKET"
The migrations_dir value must be a relative path inside the project,
without a .. component. The KV bulk commands use the Wrangler file
format. cello kv bulk get streams; a named output file changes only
after the export completes, and a failed stdout export leaves an
incomplete JSON array.
cello r2 reads and writes the objects of an r2_buckets binding under
the reserved r2/<bucket_name>/ prefix. It needs no running node, so a
release pipeline can publish an artifact before it deploys.
cello r2 put assets app.zip --path dist/app.zip \
--content-type application/zip \
--metadata '{"release":"1.2.3"}' \
--bucket "$CELLO_BUCKET"
The first argument is the bucket_name, not the binding name. --metadata
becomes customMetadata, and the content flags become httpMetadata. An
object that another tool writes into the prefix is also readable, without
cacheExpiry or checksums. cello r2 get streams the body to stdout, and
cello r2 head prints the stored record; with --json, the
http and custom fields hold httpMetadata and customMetadata.
cello kv list and cello r2 list print at most 1000 keys and, on stderr,
the flag that continues the listing: --after KEY for kv, --cursor CURSOR
for r2. Pass --all for every key. cello kv list --json prints one JSON
object per key. cello r2 list takes --prefix and --delimiter; with
--page it prints one page object — objects with httpMetadata and
customMetadata, plus delimitedPrefixes, truncated and cursor — a
Worker’s list() page without version, checksums and storageClass.
Every Cello command writes data to stdout and messages to stderr. A closed stdout pipe is a successful stop.
Invoke a staged version
A version uploaded with cello deploy --no-activate serves no traffic
until cello activate names it. cello invoke sends one HTTP request to
a staged version on one node of the fleet, served by that version’s code
with its declared bindings against the real data, so a migration endpoint
runs before the activation that would serve it:
cello invoke app=VERSION --path /migrate --method POST --bucket "$CELLO_BUCKET"
The invoke moves no pointer and changes nothing serving. Only the operator can invoke: the internal-listener route requires the fleet signature plus the operator token when one is set. See Invoke a staged version.
cello versions [SCRIPT] lists the deployed versions from one LIST of the
deployment manifests, marking the serving generation current from the
fleet-wide pointer (top-level version plus staged components); an optional
SCRIPT narrows the rows. Rollback and prune stay cello activate and
cello discard. Serving an old version against live cell state is out of
scope: there is no preview.
Start a node
For local development, the default listener is sufficient:
cello \
--bucket "$CELLO_BUCKET" \
--endpoint "$S3_ENDPOINT" \
--region "$AWS_REGION"
For a fleet node, bind the public and internal listeners separately:
cello \
--bucket "$CELLO_BUCKET" \
--endpoint "$S3_ENDPOINT" \
--region "$AWS_REGION" \
--listen 0.0.0.0:8080 \
--internal-listen 10.0.0.12:8081 \
--advertise node-a.internal:8081
--advertise requires an explicit --internal-listen, and so does a
non-loopback --listen. You must route the advertised address to the
internal listener, not to the public listener; Cello cannot verify this.
Add nodes
Start each node with the same bucket settings and its own reachable
--advertise address. The nodes find each other through leases in the
bucket; there is no join command.
Peer traffic has no TLS and cell fetch and RPC traffic is not signed, and
without CELLO_OPERATOR_TOKEN the internal listener has an
unauthenticated operator API. Put the advertised addresses on a trusted
private network or an encrypted overlay such as WireGuard or Tailscale,
and never expose the internal listener to the internet. See
security.
The node that receives a request activates a new cell, so a load balancer
must include a new node in its rotation. A node at its residency cap or
under memory pressure hands the activation to the least-loaded peer. A
pressure eviction releases the ownership record
(CELLO_PRESSURE_OWNERSHIP=release, the default), so the cell can move.
An idle eviction keeps the record, so the cell wakes on its own node. An
idle eviction must stop the runtime within CELLO_OPERATION_DEADLINE_MS,
or the cell stays resident until the next idle period.
A new node also receives idle cells. Every node reads a shared fleet
sample, fleet/capacity-v1.json, every CELLO_REBALANCE_INTERVAL_MS
(default 5000; 0 disables balancing). Each node gets a share of the
owned cells in proportion to CELLO_PLACEMENT_WEIGHT (default: the CPU
count). The most loaded node hands at most 32 idle cells at a time to the
node furthest below its target, and a receiver fills to 2% below its
target. Only a hibernated cell moves; its hibernatable WebSockets close
with code 1012 so that the clients reconnect. A fleet without
CELLO_IDLE_EVICT_S therefore balances only the cells that hibernate on
their own. A draining node and a node with a cold-activation backlog
(restoring) receive no cells. Nothing moves while a node reports a lease
without a weight, so a rolling upgrade to this version completes first.
POST /rebalance/pause on any node pauses balancing for the fleet, and
POST /rebalance/resume on the same node resumes it.
Feed an autoscaler
Cello does not scale itself. An external system starts and stops the nodes, and Cello hands the cells off.
Each node writes its lease to nodes/<node>.json in the bucket. The lease
lives CELLO_TTL_MS (default 10000) and renews at one third of that. Its
load block, dated by sampled_ms, contains owned_cells,
placement_weight, resident_cells, host_websockets, rss_bytes,
in_use_bytes, cpu_percent_x100, open_fds, pressured,
memory_headroom, shed_cells, and restoring. owned_cells includes
dormant cells; a restarted node omits it until it has read its ownership
records. resident_cells counts only cells in memory. The memory and CPU
values are sampled every second.
GET /state on the internal listener reports the live counters. Its
node_load object is the lease load block.
| field | meaning |
|---|---|
capacity_waiting |
Activations queued behind the residency cap. Positive means add a node |
activation_waiting, restoring |
Cold activations that wait for or hold a permit |
owned_cells, occupied, shedding |
Ownership, residency, and pressure shedding |
handed_off, rebalanced, rebalance_failed |
Cells given to peers, the part balancing moved, and the balancing moves no peer took |
remote_route_refreshes |
Cached routes that expired and started a new lookup. Normal lease renewal increases it |
allocator |
Rust allocator bytes: allocated_bytes, resident_bytes, mapped_bytes, retained_bytes. V8 heaps are not included |
libc_malloc |
Linux only: in_use_bytes and free_bytes of the C allocator that SQLite and V8 use |
deployment.isolates, deployment.draining |
Per Worker script (cells, stateless, services): live, live_empty, retiring, freed, heap_bytes, external_bytes |
A live_empty count that persists for more than 30 seconds means that
isolate maintenance does not run. A persistent retiring count means that
a turn or a request still holds the heap.
A positive capacity_waiting or a pressured lease is the signal to add
a node. Scale down only when every remaining node reports
memory_headroom and a small restoring backlog. A drain into a full
fleet leaves the cells dormant on the survivors, and each later activation
must shed a resident cell.
/.well-known/cello/health on the public listener is a boolean. It
reports 503 during a drain and before a joining node settles.
Shut down and roll out a node
Cello shuts down gracefully on SIGTERM or SIGINT. The health path reports
503, new public requests receive 503 with the connection closed, and
accepted requests finish. The node hands its cells off in batches: it
cancels running alarms (the successor retries them), proves the data
durable, publishes a final snapshot, and releases each ownership record to
a peer. The peer keeps the cell dormant until a request arrives. A
database larger than the durability deadline can upload (80 MiB at the
default 10 seconds) hands off without the snapshot. Busy cells move first.
POST /shutdown on the internal listener starts the same handoff.
| variable | default | effect |
|---|---|---|
CELLO_SHUTDOWN_TOTAL_MS |
40000 | The complete stop bound. The drain-token wait is 3/4 of it (30000) and the handoff no-progress interval is 5/8 (25000) |
CELLO_RELEASES |
128 | Concurrent complete handoffs |
CELLO_ACTIVATIONS |
see table | Demand-driven restores and startup work |
CELLO_READY_FLEET_GATE_MS |
120000 | The first-readiness gate deadline; 0 disables the gate |
You must set the orchestrator stop grace (systemd TimeoutStopSec,
Kubernetes terminationGracePeriodSeconds) above
CELLO_SHUTDOWN_TOTAL_MS, or SIGKILL can interrupt the handoff. Cello
rejects the removed CELLO_SHUTDOWN_DRAIN_MS and
CELLO_DRAIN_TOKEN_WAIT_MS at startup.
Concurrent stops do not flood the survivors: a draining node claims a fleet drain token, so nodes hand off one at a time. A node that cannot claim the token within its wait proceeds without it.
A new process does not report healthy until the fleet settles: no node
drains, every node is below its memory low watermark, the restore backlogs
are small, and ownership is balanced. At CELLO_READY_FLEET_GATE_MS,
Cello logs ready_gate_expired once, but readiness stays closed, so set
an orchestrator rollout deadline. After the first healthy response, fleet
state does not remove readiness.
A cut handoff can leave the log of the stopped process for its replacement to recover. Another process can replace a recovery that does not respond for 30 seconds.
A remote call to a cell that is changing owner waits for the new owner,
for at most CELLO_OPERATION_DEADLINE_MS (default 15000). Cello retries
only an attempt that provably did not start. An application must keep one
stable operation ID when it retries an ambiguous fetch, RPC, D1, or
service operation. A WebSocket client must reconnect.
To roll out a new version, use the rolling update of your orchestrator: stop each node with SIGTERM, wait for its replacement to report healthy, and then move to the next node. Cello paces the handoffs and readiness itself.
POST /shutdown?handoff=preserve prepares a same-node restart that keeps
the ownership records. It cancels cold activations and uploads each
stopped database first. If an upload fails, the next process uses normal
recovery. The internal operator API is alpha and can change in any
release, so keep operator tooling and Cello releases together.
Upgrade notes
- celld / Cello 0.1.x to v0.2.0: stop all. Stop every old node, then
start v0.2.0 on the same bucket: 0.2.0 reads data a celld node wrote
(new writes use the
cellospellings) but speaks a new peer protocol, so a fleet must not mix versions. Rename everyCELLD_*variable toCELLO_*; the node refuses to start otherwise. See Compatibility. - v0.2.1 to v0.3.0: rolling. The default durability changes from
buckettofleet. Do not start a v0.2.x binary after a node runs v0.3.0 unless its shutdown log containsnode-log close: sealed epoch. Otherwise, the downgrade can lose acknowledged writes. - v0.3.0 to v0.4.0: stop all. The peer protocol changes, and a v0.3.0 node cannot read new large KV values, so a mixed fleet can make committed KV values unavailable.
- v0.4.0 to v0.4.1: rolling. Paged restore starts one lease lifetime after the last v0.4.0 node stops. Do not start a v0.4.0 binary after that: it cannot restore a paged cell.
- v0.5.1 to v0.6.0: stop all with
fleetdurability. A v0.6.0 node refuses to start against a v0.5.1 follower. Abucketdurability fleet can roll. - v0.6.0 to v0.6.1: rolling. Until every node runs v0.6.1, do not set
CELLO_LTX_RETENTION_SECS, do not deploy a Python Worker, and do not raiseCELLO_MAX_ASSET_FILE_BYTESabove 25 MiB. A rollback to v0.6.0 can roll after you unsetCELLO_LTX_RETENTION_SECSon every node. - v0.6.1 to v0.6.2: rolling. Until every node runs v0.6.2, a handler
failure on the owner of a forwarded Durable Object fetch can reach the
caller as a 500 response. A v0.6.2 pair rejects the caller’s
stub.fetch()instead, as a local owner does. A rollback to v0.6.1 can also roll. - v0.6.2 to v0.6.3: rolling. A v0.6.3 node no longer trusts the emSign ECC Root CA - G3 certificate or the eight compromised eMudhra intermediate CA certificates for outbound TLS, so upgrade each node promptly. Until every node runs v0.6.3, a v0.6.2 node can still acquire a cell whose ownership record was lost and start the cell with an empty database. A rollback to v0.6.2 can also roll.
Diagnose a fleet
cello diagnose reads the node leases and probes each live peer. It does
not take a lease or change ownership. Use --peer NODE_ID, repeatable, to
probe only some nodes.
cello diagnose \
--bucket "$CELLO_BUCKET" \
--endpoint "$S3_ENDPOINT" \
--region "$AWS_REGION"
The report shows expired leases, bad advertised addresses, unreachable
peers, authentication failures, and protocol mismatches. Each node line
shows restoring, the cold activations in progress. During a rolling
update, wait for restoring=0 on every node before you restart the next.
List Durable Objects
cello cell list \
--bucket "$CELLO_BUCKET" \
--endpoint "$S3_ENDPOINT" \
--region "$AWS_REGION"
Each line is a Class:ID scope. Give a class name to list only that
class, and pass --json for one JSON object per line. An instance appears
after its first event; an ID that is only derived does not appear. D1
databases, KV namespaces, and Workflows are cells in reserved classes whose
names start with __. The --json output marks them "reserved": true:
cello cell list --all --json --bucket "$CELLO_BUCKET" |
jq -r 'select(.reserved | not) | .scope'
The command stops at 1000 instances and prints the continuation on stderr:
1000 cells shown; more exist. Continue with --after Room:d99d9174b25e46310694dd931b47fbde70a7460bb7b210b546060651ea2ff6e0
Pass that --after SCOPE for the next page, --limit N for a different
page size, or --all for everything (one request per 1000, with progress
on stderr). The listing is not a snapshot, but a sequence of --after
commands lists each existing instance once.
cello cell gc --dry-run takes the same options and prints, for each cell,
the epoch prefixes that epoch GC can delete, their bytes, and the restore
base. It writes nothing, and it exits with an error at the end if it could
not read a cell.
cello cell gc --dry-run --bucket "$CELLO_BUCKET" --grace-secs 3600
The real deletion can come later or not at all. Only a node with a
positive CELLO_LTX_RETENTION_SECS deletes, with that grace instead of
--grace-secs. A cell deletes only while it is active, after its
activation has a write and, if paged, after its local file is complete.
With CELLO_LTX_COMPACTION=0, a cell on a fleet node waits for a handoff
snapshot. One pass deletes at most 64 epochs of a cell.
Hot-cell overload
Cello admits at most 64 concurrent fetch events for one Durable Object
(CELLO_MAX_CELL_REQUESTS). A Queue broker admits 256 concurrent producer
calls, commits at most 64 in one transaction, and overlaps up to four
transactions.
Over the limit, Cello returns 503 with Retry-After: 1 and
x-cello-overload: cell, and does not start the event. The Queue response
body is {"error":"cell admission refused"}, and a caught Queue producer
error contains cell overload: admission refused. A load generator must
count these responses as rejected work. Cello logs cell_overload_refused
with the cell scope, node, region, in-flight count, and limit.
Environment variables
For the full list, including advanced tuning switches, run cello -h. An
unset variable selects its default. A Boolean variable accepts only 0 or
1. Cello exits at startup on an invalid value.
| variable | purpose |
|---|---|
CELLO_BUCKET |
The fleet bucket, and an optional key prefix. The same as --bucket |
S3_ENDPOINT |
The S3-compatible endpoint. The same as --endpoint |
AWS_REGION, AWS_DEFAULT_REGION |
The storage region |
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN |
Explicit AWS credentials. The standard AWS credential chain is also available |
GOOGLE_APPLICATION_CREDENTIALS, GOOGLE_SERVICE_ACCOUNT_KEY |
Google credentials for a gs:// bucket. Application Default Credentials are also available |
AZURE_STORAGE_ACCOUNT_NAME |
The storage account for an az:// bucket |
AZURE_STORAGE_ACCOUNT_KEY |
The storage account key. Do not combine it with an identity selector |
AZURE_AUTHORITY_HOST, AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE |
The AKS workload identity environment. The authority must be the public Azure host |
AZURE_STORAGE_USE_EMULATOR |
Set to true to develop against Azurite |
CELLO_ADDR |
The public Worker listener. The same as --listen |
CELLO_INTERNAL_ADDR |
The peer and operator listener. The same as --internal-listen |
CELLO_ADVERTISE |
The internal address that peers can reach. The same as --advertise |
CELLO_UNSAFE_PUBLIC_ADVERTISE |
Set to 1 to permit a literal public IP in CELLO_ADVERTISE. It does not resolve a DNS name or restrict the internal listener |
CELLO_OPERATOR_TOKEN |
A bearer token the whole fleet shares. When set, every internal-listener route requires Authorization: Bearer <token> (compared in constant time) and answers 401 without it; cello d1, cello kv, cello queue, cello diagnose, and cello invoke read it from the environment and send it. Unset: the internal listener stays open. Set but empty: the node refuses to start. See security |
CELLO_NODE |
An explicit node-session ID: 1 to 128 ASCII letters, numbers, dots, dashes, or underscores, but not . or .. |
CELLO_WATCH |
The local work directory for SQLite and replication |
CELLO_ESBUILD |
The path of the esbuild executable: the self-contained release binary, since on Linux it runs with reads confined to the project and its read roots |
CELLO_ACTIVATIONS |
Concurrent cold-cell activations (default: 8 per CPU, at least 16, at most 128) |
CELLO_DEPLOY_POLL_S |
The deployment pointer poll interval in seconds (default: 30) |
CELLO_DEPLOY_MAX_AGE_S |
Seconds before Cello forces a resident Durable Object onto a new deployment (default: 60; 0 forces at once) |
CELLO_CACHE_SCOPE |
Which cache entries a Worker sees: script (default) isolates each script’s entries, deployment shares one caches.default and one set of caches.open(name) namespaces across every script of the serving deployment, like one Cloudflare zone. See cache and security |
CELLO_OPERATION_DEADLINE_MS |
The deadline for a non-restore operation (default: 15000) |
CELLO_MAX_CELL_REQUESTS |
Concurrent fetch events for one Durable Object (default: 64) |
CELLO_MAX_CELLS_PER_ISOLATE |
Cells packed into one isolate (default: 32). Lower it for heavyweight cells such as agents |
CELLO_BACKGROUND_BUDGET_S |
Post-response background budget in seconds (default: 30, Cloudflare’s waitUntil limit). Past it the driver ends leftover timers and waitUntil without failing the answered request, and the abort pass releases the isolate |
CELLO_MAX_REQUEST_BODY_BYTES |
The body limit for a public Worker request or a direct Durable Object request (default: 1 GiB) |
CELLO_TMP_MAX_BYTES |
The per-request /tmp size limit in bytes (default: 67108864, 64 MiB). Any write, append, or stream chunk that would push a request past it fails with ENOSPC; an overwrite frees the replaced bytes |
CELLO_MAX_RESIDENT_CELLS |
The hard limit for resident cells, enforced at admission |
CELLO_IDLE_EVICT_S |
Seconds after which an idle cell leaves memory and hibernates (unset: only pressure or the residency cap removes an idle cell) |
CELLO_PLACEMENT_WEIGHT |
The ownership share of this node, relative to the other nodes (default: the CPU count) |
CELLO_REBALANCE_INTERVAL_MS |
The fleet sample interval and maximum age (default: 5000; 0 disables balancing) |
CELLO_MAX_RSS_MB |
The memory threshold for pressure shedding, on the greater of the allocator-adjusted RSS and the active cgroup working set (default: 80% of the available memory; 0 disables the threshold and the absolute cap) |
CELLO_MEMORY_SCOPE |
Which cgroup the pressure sample and limit read: container (default) measures the cgroupfs root, self the process’s own cgroup v2 leaf, falling back to the root with a warning on cgroup v1 or where the leaf is unreadable. With self and an unset CELLO_MAX_RSS_MB, the threshold defaults to 80% of the leaf’s memory.max |
CELLO_C_TRIM_MB |
Freed C-heap growth, in MiB, that earns another malloc_trim on the sampler tick (default: 8). Lower returns glibc free pages sooner after eviction batches |
CELLO_DURABILITY |
fleet (default): the node sends each write to one or two other nodes and answers when they hold it on disk or the bucket upload finishes. A single node has no peer, so every write waits for the bucket. bucket: always wait for the bucket |
CELLO_BUCKET_BUNDLES |
0 (default): in bucket posture, upload per-cell L0 objects. Set 1 to group each flush round’s dirty rows into shared bundle proofs (log/<node>/<generation>/bundle/). Bundling additionally needs every live lease to read format 3 (see guarantees); any other value stops startup |
CELLO_BUCKET_BUNDLE_LINGER_MS |
Hold a bundle round open this many milliseconds after the first dirty wake so concurrent writes share one PUT (default: 0; values above 1000 clamp to 1000). The window applies only while more than one cell is dirty: a lone dirty cell flushes at once, so the linger costs latency only when it also saves PUTs. Measured on 200 agents × 6 writes (Class A PUT/POST/LIST): 1934 with bundling off, 1120–1136 with bundling on (two runs), 982–986 with 200 |
CELLO_LOG_PIPELINE |
Fleet log rounds in flight (default: 4) |
CELLO_LOG_HEDGE_MS |
The wait before a second copy of a slow log append. The default is adaptive: 4 times the slowest recent append, at least 250 ms. 0 disables the copy |
CELLO_LTX_TRUNCATE_PAGES |
The WAL size, in pages, at which Cello truncates the WAL at the next checkpoint (default: 128, 512 KiB). A database larger than 4 MiB waits until the WAL is larger than the database. 0 disables it |
CELLO_SQL_STATEMENT_MS |
The per-statement SQL time bound in milliseconds (unset: unbounded, the historical behavior). A timed-out statement fails with a catchable timeout error and rolls back |
CELLO_SQLITE_CACHE_KB |
The SQLite page-cache budget per resident cell connection, in KiB (default: 2000, the stock ceiling). Lower bounds per-cell glibc retention on dense nodes; a working set past the cap faults pages back instead |
CELLO_LTX_RETENTION_SECS |
Unset or 0 (default): Cello deletes no epoch prefix. A positive value enables epoch GC: the owner deletes the prefixes below the restore base, but keeps its own epoch, the one before it, and each epoch younger than this many seconds. A pass runs at most every 5 minutes. With CELLO_LTX_HYDRATE_MBPS=0, a paged cell never qualifies. Epoch GC skips facet streams and needs list-after-write consistency |
CELLO_LTX_COMPACTION |
1 (default) creates additive L1 objects, so a takeover reads tens of objects instead of thousands. Set 0 on every node of a mixed fleet until all nodes read v0.5.2 block objects |
CELLO_LTX_COMPACTION_MIN_TXIDS |
The TXID distance that queues an L1 compaction (default: 256) |
CELLO_LTX_COMPACTION_MIN_MB |
The L0 MiB that queue an L1 compaction (default: 32, at most 64) |
CELLO_LTX_COMPACTIONS |
Concurrent L1 compactions per node (default: 2) |
CELLO_LTX_PAGED |
1 (default): a takeover of a chain of at least CELLO_LTX_PAGED_MIN_MB reads pages from the bucket on first use instead of a full download. Until the background fill completes, a page fault blocks the cell’s isolate, so a large query can take minutes. Set 0 to download every chain. Set 0 on every node of a mixed fleet with a node before v0.4.1, which permanently refuses a paged cell. The paged_gate log event reports when paging turns on or off |
CELLO_LTX_PAGED_MIN_MB |
The chain size, in MiB, from which a restore pages (default: 256; 0 pages every chain) |
CELLO_LTX_HYDRATE_MBPS |
The background fill rate of a paged cell, in MiB per second, one cell at a time per node (default: 16; 0 keeps the cell sparse) |
CELLO_LTX_DURABILITY_TIMEOUT_SECS |
The budget for one durability proof, and for the final snapshot of a handoff (default: 10). A queued write waits at most six budgets. A slow store can need more |
CELLO_TOKIO_THREADS |
Host Tokio worker threads (default: the CPU count). The host_runtime log event at startup reports worker_count |
RUST_LOG |
The runtime log filter |
An L1 compaction merges at most 256 source objects within a 64 MiB buffer. A larger source object spills to temporary files in the cell’s local LTX directory, so the node needs free disk for the source and the output.
These settings are removed. Remove them from the environment, including an empty value or the former default:
| removed setting | current behavior |
|---|---|
CELLO_OUTPUT_GATE |
Cello always waits for the durability proof before it acknowledges a write |
CELLO_SHUTDOWN_DRAIN_MS, CELLO_DRAIN_TOKEN_WAIT_MS |
Rejected at startup. Use CELLO_SHUTDOWN_TOTAL_MS |
CELLO_OTEL_SINK |
Set CELLO_OTEL=1 for the fleet bucket or set CELLO_OTEL to the collector base URL for OTLP |
CELLO_AI_BINDING, CELLO_AI_URL |
The AI adapter is removed. Call the provider from application code and remove the AI binding |
CELLO_CLOUD_RESTART_ON_DEPLOY |
A managed deployment adopts the new code in place. Credential rotation can still restart the process |
CELLO_STORAGE_PROBE |
A node always checks the storage contract. A violation prevents startup; an ambiguous transport error produces a warning |
CELLO_EVICTIONS |
A node runs at most four concurrent evictions |
CELLO_LOG_CAPTURE_WORKERS |
A node uses at most eight log capture workers |
CELLO_REBALANCE_BATCH_CELLS |
A balancing batch moves at most 32 idle cells |
CELLO_PRESENCE_SHADOW |
A managed node sends its serving status and owned-cell count. Use cello diagnose --read-only to check leases |
CELLO_LOG_BUNDLE |
Fleet durability always bundles uploads |
CELLO_QUEUE_PRODUCER_GROUP_MS |
A Queue owner groups producer calls with a 4 ms timer |
CELLO_LOG_GROUP_COMMIT_MS |
A node waits 1 ms before a fleet log capture when Queue writes are pending |
CELLO_PACED_HANDOFF |
A node always attempts the handoff within CELLO_SHUTDOWN_TOTAL_MS |