Security
Cello v0.6.3 is a beta release. It is not safe for hostile multi-tenant use. Only the latest release receives security fixes.
Security boundary
One fleet runs one application. Cello trusts the application code, the fleet nodes, and the operators. Application code can use its configured bindings and can consume shared node resources. Do not run code from mutually distrusting tenants in one fleet.
Application code cannot reach the engine’s host functions. An internal script
receives them as function parameters, and globalThis carries no
__-prefixed property. A Worker Loader worker with globalOutbound: null
therefore reaches the host only through the capabilities in its env.
Cello depends on two external boundaries: a trusted private network protects the internal listener, and object storage credentials control the fleet.
Runtime cell isolation
Cells of the shared runtime classes — D1 databases, KV namespaces, and Queue
brokers — serve every script in the fleet, so they run in a dedicated runtime
isolate pool whose isolates evaluate no tenant bundle: the harness plus a
fixed inert entry module. No script’s top level shares a realm with them, and
the pool holds no tenant bindings, vars, services, loaders, or schedule —
only the deployment-wide queue catalog, the node’s routing identity, the
generation tag, and the union of every declaring script’s compatibility
switches. Workflow
cells and the cron cell stay in their own script’s pool: they execute that
script’s run() and scheduled() handlers with its bindings, so isolating
them from their own script would isolate them from the code they run.
Per-cell placement and ownership are unchanged.
Cache scope and co-hosted scripts
The Workers Cache API is per script by default: no script can read,
overwrite, or purge another script’s entries. CELLO_CACHE_SCOPE=deployment
lifts that boundary for scripts of one deployment, like one Cloudflare
zone: every script of the serving deployment shares one caches.default
and one set of caches.open(name) namespaces, so a co-hosted script can
read entries holding another script’s session or personalized bodies,
overwrite them, and purge them. Turn it on only when every script of the
deployment is mutually trusted. The scope is an operator setting read
from the node’s process environment at startup; no binding, Worker var,
or JS API can change it, and an unknown value refuses the boot rather
than silently sharing. Sharing stays bounded: a different deployment or
fleet still sees nothing, the runtime pool and dynamically loaded
workers keep their own entries, and each scope carries its own size cap
and recency (see cache).
Separate the listeners
Cello opens two HTTP listeners and terminates TLS on neither.
| Listener | Serves | Required protection |
|---|---|---|
--listen (public) |
The deployed Worker | Terminate TLS and authenticate users in a proxy or the application. |
--internal-listen (default 127.0.0.1:0, a free loopback port) |
The peer protocol and the operator API | Restrict access to trusted operators and fleet nodes. Never expose it to the public internet. Use an encrypted overlay such as WireGuard or Tailscale when the network does not provide confidentiality. |
--advertise gives peers the internal address. Cello rejects an explicit
--advertise or a non-loopback --listen without an explicit
--internal-listen. Cello cannot verify a hostname or a translated port, so
you must route the advertised address to the internal listener.
The public listener reserves only /.well-known/cello/health: 200 with
{"ok":true} when healthy, 503 otherwise. The Worker owns every other public
path, including /health. The internal listener returns 404 for an unknown
path, so an operator request cannot become an application request.
The internal listener has three request groups, and all three require the trusted private network:
- Most operator routes have no request authentication, unless
CELLO_OPERATOR_TOKENis set (see below). /peer/tunnelopens a tunnel for cell fetch, RPC, and WebSocket calls. The establishment request carries the fleet HMAC, a clock limit, and replay protection. The calls inside the tunnel are plain, unsigned HTTP.- The peer-control and reserved-cell routes sign each request with the fleet HMAC, a clock limit, and replay protection.
Every path to a runtime class therefore demands the fleet secret. The HMAC does not authenticate tunnel bytes after establishment and does not encrypt traffic, so it does not replace the private network.
Guard the internal listener with a token
Set CELLO_OPERATOR_TOKEN to the same value on every node of the fleet,
and every internal-listener route requires
Authorization: Bearer <token>, compared in constant time, and answers
401 without it. There are no exempt routes: unknown paths answer 401 too,
and liveness and readiness probes use the public listener’s
/.well-known/cello/health, which the token does not guard. A node with
the variable set but empty refuses to start; an unset variable is the
open listener above.
Every in-tree client of the internal listener reads the variable from its
own environment and sends the token: node-to-node peer forwarding,
handoff, abort, the log tier, and the signed probe, as well as
cello d1, cello kv, cello queue, cello diagnose, cello invoke,
and the cello dev supervisor’s shutdown. A caller outside the fleet — a deployment
script that posts /reload, a monitor that reads /state — must send
the header itself. Roll the token out only after every node runs a Cello
that knows the variable: a node that never sends the header is refused
by the nodes that require it, so a mixed fleet would split its peer
traffic.
A node holds one token and tunnels between nodes do not re-authenticate, so rotating the token means restarting the fleet with the new value: either every node at once, or node by node while accepting that nodes on different tokens refuse each other’s peer traffic until the last one restarts.
The token authenticates the caller; it does not encrypt the traffic. Without TLS on the internal listener the token travels in cleartext, so a network observer still sees it. Keep the trusted private network or the encrypted overlay: the token shrinks what that network must prove, it does not replace it.
Use the internal operator API
The operator API is an alpha interface. A release can change its paths or response formats. Without the token, these routes do not authenticate the caller:
/statereports node state./cell/<SCOPE>resolves or activates a cell./evict/<SCOPE>tries to evict a resident cell and reports the result./do/<ID>sends a direct request to an ordinary Durable Object.POST /shutdownstarts a graceful ownership handoff.handoff=preserveprepares a same-node reload.
/do/<ID> refuses every reserved runtime class, such as D1, Workflows, KV,
and Queues, because their protocols can access application data. Use the
HMAC-authenticated /runtime/<SCOPE> route for these classes.
POST /invoke serves one request from staged, not yet activated, script
versions (cello invoke), so a migration endpoint runs before the
activation that would serve it. Like /runtime/, it requires the
per-request fleet HMAC plus the operator token when one is set; without
the token it still requires the HMAC, which a tenant cannot mint. No
public route and no peer tunnel reaches a non-activated version: the
public listener has no such path, and tunnels carry cell calls, never
this route. The invoke writes nothing to the bucket, moves no pointer,
and serves staged code in fresh isolate pools that share nothing with
live traffic and are retired after the request; cells resolve from the
live generation, so bindings reach the real data. It runs to completion
server-side even if the caller disconnects, so stopping the wait cannot
leave a half-applied migration, and it answers only after waitUntil
settles. See Invoke a staged version.
/peer/probe returns a signed diagnostic response. Do not call the other
reserved peer paths directly.
Read an eviction result
An accepted eviction waits for the operation to finish, and a refused one returns at once. Concurrent callers can join one eviction, so the success count does not equal the stop count.
The Rust method AppHandle::evict returns Result<EvictSuccess, EvictError>.
Evicted confirms a completed runtime stop, and AlreadyAbsent confirms
settled local absence. A caller that needs a completed eviction must check for
Evicted. Over HTTP, both return 200 with {"ok":true}. kind() and
reason() on the error match the HTTP error body. A later request can
reactivate the cell before the response arrives.
A missing cell or a settled Inactive, Dormant, or Remote cell is locally
absent. A pending activation, stop, or ownership transfer prevents that
result. The request does not evict a remote runtime. A dormant cell keeps its
ownership and its hibernated host sockets. The node refuses during
preservation, reload, or local inventory confirmation, or when it lacks
authority.
An error body has this form:
{"ok":false,"error":{"kind":"refused","reason":"cell_active"}}
| Status | Kind | Reason | Cause |
|---|---|---|---|
| 409 | refused |
cell_active |
The cell has active work or a socket that requires a runtime. |
| 409 | refused |
cell_transitioning |
The cell has another lifecycle transition. |
| 409 | refused |
alarm_imminent |
The alarm residency policy retains the cell. |
| 409 | refused |
alarm_uncovered |
The alarm coverage is unconfirmed, or a firing alarm blocks eviction during pressure shedding. |
| 503 | refused |
node_unavailable |
The node cannot accept the eviction. |
| 503 | refused |
eviction_limit |
The node has reached its eviction concurrency limit. |
| 409 | cancelled |
new_activity |
A new request cancels an accepted eviction. |
| 409 | cancelled |
alarm_activity |
An alarm observation or firing cancels an accepted eviction. |
| 409 | cancelled |
node_fenced |
The node loses its authority during the eviction. |
| 503 | failed |
actor_unavailable |
The request cannot reach the Actor. |
| 500 | failed |
reply_lost |
A delivered request loses its reply, so its outcome is unknown. |
| 500 | failed |
durability_failed |
The durability verification fails. |
| 500 | failed |
durability_timeout |
The durability verification exceeds its operation deadline. |
| 500 | failed |
runtime_stop_failed |
The runtime stop reports a failure. |
A malformed scope returns 400. An error does not prove that the runtime is still resident. The runtime stop has no overall timeout, so a stop that never returns keeps the request pending.
Set the forwarded-header policy
Cello ignores X-Forwarded-Host and X-Forwarded-Proto by default. Set
--trust-forwarded-headers or CELLO_TRUST_FORWARDED_HEADERS=1 only when a
trusted proxy replaces both headers. Cello uses the last value in each header,
so an earlier client value cannot override the proxy value.
Cello takes the path and query from the request target and ignores the scheme
and authority of an absolute-form target. Without a trusted proxy, the Host
header sets the hostname in request.url. Cello accepts a hostname, an IPv4
address, or a bracketed IPv6 address, with an optional port. It rejects
malformed and noncanonical values and falls back to cello.local.
The hostname is still client-controlled. Do not use an unchecked hostname for an authorization decision. Use a trusted proxy, or check the hostname against a list in the Worker.
Limit request bodies
The public listener and /do/<ID> limit a request body to 1 GiB by default.
Set CELLO_MAX_REQUEST_BODY_BYTES to a smaller positive value to lower it.
Cello returns 413 for a declared oversized body and when a Worker reads past
the limit.
For a method other than GET or HEAD, /do/<ID> streams a body of unknown
length or of at least 1 MiB. It collects a smaller body before dispatch.
Restrict Worker egress
By default (CELLO_EGRESS=open) Worker code may dial any address with
fetch(), connect() sockets, and outbound WebSockets.
CELLO_EGRESS=public refuses dials to every non-public address:
loopback, private (RFC 1918), link-local, carrier-grade NAT (RFC 6598),
IPv6 unique-local, unspecified, multicast, reserved (240.0.0.0/4),
the IPv4-mapped and transition-embedded (NAT64, 6to4, Teredo,
IPv4-compatible) forms of those, and the cloud metadata address
(169.254.169.254).
CELLO_EGRESS_ALLOW lists exceptions as comma-separated CIDRs (any
port) or host:port pairs (that host on that port only), for example
an object store endpoint on a private address:
CELLO_EGRESS=public CELLO_EGRESS_ALLOW=10.89.0.7:9000,10.89.0.0/24
A hostname entry resolves on every check, so a moved endpoint is
re-allowed without a restart. A host:port entry is port-scoped on
every hop: the initial request and each redirect hop match the entry
only on its port, against the hop’s full URL. A redirect to a bare IP
literal matches only IP, CIDR, and IP:port entries — never a hostname
entry, which cannot be resolved in that path — so allowlist redirect
targets in one of those forms. The node refuses to boot when either
variable is invalid.
Enforcement happens on the resolved address at connect time: hostnames
are resolved and filtered before the dial, every address is judged, and
redirects re-check on every hop, so DNS rebinding and redirects cannot
bypass the policy. In public mode tenant clients never use a proxy
(HTTP_PROXY/HTTPS_PROXY/ALL_PROXY are ignored): through a proxy
the target would resolve with the proxy’s DNS and redirect hops would
expose only the proxy’s address, so a proxied fetch cannot be policed.
Cello’s own traffic — the object store, peers, control, telemetry, and
container-engine clients — never goes through these checks and keeps
its proxy handling. Dials to container endpoints this node published
via getTcpPort() are exempt, and that exemption is node-global: any
Worker on the node may dial a published container endpoint by its
address and port, the same reachability the nft gateway granted. A
refused fetch() rejects with a TypeError whose message starts
fetch failed: egress policy refused ...; a refused connect()
rejects the socket’s opened/closed promises, and a refused
WebSocket fires error and closes with 1006, each naming the refused
address and reason. Every refusal also logs an egress_refused
warning with the target’s scheme, host, and port, the refused address,
and the reason — never the full URL, whose userinfo or query may carry
secrets.
Protect the fleet bucket
The fleet bucket is the root of authority. It stores the deployments, the cell state, the ownership and node leases, and the shared peer-authentication secret. A holder of the bucket credentials controls the fleet. Give each credential access to one fleet bucket only, and rotate it after a suspected disclosure.
Cell ownership
Each cell is a SQLite database with one writer. An ownership epoch fences each cell, so a node that loses its lease cannot modify the current state. This fencing protects storage consistency. It does not isolate hostile applications. See what Cello guarantees and limitations.