Skip to content
Cello
Esc
↑↓navigate↵open⌘Jpreview
On this page

Security

Cello v0.6.3 is a beta release. It is not safe for hostile multi-tenant use. Only the latest release receives security fixes.

Security boundary

One fleet runs one application. Cello trusts the application code, the fleet nodes, and the operators. Application code can use its configured bindings and can consume shared node resources. Do not run code from mutually distrusting tenants in one fleet.

Application code cannot reach the engine’s host functions. An internal script receives them as function parameters, and globalThis carries no __-prefixed property. A Worker Loader worker with globalOutbound: null therefore reaches the host only through the capabilities in its env.

Cello depends on two external boundaries: a trusted private network protects the internal listener, and object storage credentials control the fleet.

Runtime cell isolation

Cells of the shared runtime classes — D1 databases, KV namespaces, and Queue brokers — serve every script in the fleet, so they run in a dedicated runtime isolate pool whose isolates evaluate no tenant bundle: the harness plus a fixed inert entry module. No script’s top level shares a realm with them, and the pool holds no tenant bindings, vars, services, loaders, or schedule — only the deployment-wide queue catalog, the node’s routing identity, the generation tag, and the union of every declaring script’s compatibility switches. Workflow cells and the cron cell stay in their own script’s pool: they execute that script’s run() and scheduled() handlers with its bindings, so isolating them from their own script would isolate them from the code they run. Per-cell placement and ownership are unchanged.

Cache scope and co-hosted scripts

The Workers Cache API is per script by default: no script can read, overwrite, or purge another script’s entries. CELLO_CACHE_SCOPE=deployment lifts that boundary for scripts of one deployment, like one Cloudflare zone: every script of the serving deployment shares one caches.default and one set of caches.open(name) namespaces, so a co-hosted script can read entries holding another script’s session or personalized bodies, overwrite them, and purge them. Turn it on only when every script of the deployment is mutually trusted. The scope is an operator setting read from the node’s process environment at startup; no binding, Worker var, or JS API can change it, and an unknown value refuses the boot rather than silently sharing. Sharing stays bounded: a different deployment or fleet still sees nothing, the runtime pool and dynamically loaded workers keep their own entries, and each scope carries its own size cap and recency (see cache).

Separate the listeners

Cello opens two HTTP listeners and terminates TLS on neither.

Listener Serves Required protection
--listen (public) The deployed Worker Terminate TLS and authenticate users in a proxy or the application.
--internal-listen (default 127.0.0.1:0, a free loopback port) The peer protocol and the operator API Restrict access to trusted operators and fleet nodes. Never expose it to the public internet. Use an encrypted overlay such as WireGuard or Tailscale when the network does not provide confidentiality.

--advertise gives peers the internal address. Cello rejects an explicit --advertise or a non-loopback --listen without an explicit --internal-listen. Cello cannot verify a hostname or a translated port, so you must route the advertised address to the internal listener.

The public listener reserves only /.well-known/cello/health: 200 with {"ok":true} when healthy, 503 otherwise. The Worker owns every other public path, including /health. The internal listener returns 404 for an unknown path, so an operator request cannot become an application request.

The internal listener has three request groups, and all three require the trusted private network:

  • Most operator routes have no request authentication, unless CELLO_OPERATOR_TOKEN is set (see below).
  • /peer/tunnel opens a tunnel for cell fetch, RPC, and WebSocket calls. The establishment request carries the fleet HMAC, a clock limit, and replay protection. The calls inside the tunnel are plain, unsigned HTTP.
  • The peer-control and reserved-cell routes sign each request with the fleet HMAC, a clock limit, and replay protection.

Every path to a runtime class therefore demands the fleet secret. The HMAC does not authenticate tunnel bytes after establishment and does not encrypt traffic, so it does not replace the private network.

Guard the internal listener with a token

Set CELLO_OPERATOR_TOKEN to the same value on every node of the fleet, and every internal-listener route requires Authorization: Bearer <token>, compared in constant time, and answers 401 without it. There are no exempt routes: unknown paths answer 401 too, and liveness and readiness probes use the public listener’s /.well-known/cello/health, which the token does not guard. A node with the variable set but empty refuses to start; an unset variable is the open listener above.

Every in-tree client of the internal listener reads the variable from its own environment and sends the token: node-to-node peer forwarding, handoff, abort, the log tier, and the signed probe, as well as cello d1, cello kv, cello queue, cello diagnose, cello invoke, and the cello dev supervisor’s shutdown. A caller outside the fleet — a deployment script that posts /reload, a monitor that reads /state — must send the header itself. Roll the token out only after every node runs a Cello that knows the variable: a node that never sends the header is refused by the nodes that require it, so a mixed fleet would split its peer traffic.

A node holds one token and tunnels between nodes do not re-authenticate, so rotating the token means restarting the fleet with the new value: either every node at once, or node by node while accepting that nodes on different tokens refuse each other’s peer traffic until the last one restarts.

The token authenticates the caller; it does not encrypt the traffic. Without TLS on the internal listener the token travels in cleartext, so a network observer still sees it. Keep the trusted private network or the encrypted overlay: the token shrinks what that network must prove, it does not replace it.

Use the internal operator API

The operator API is an alpha interface. A release can change its paths or response formats. Without the token, these routes do not authenticate the caller:

  • /state reports node state.
  • /cell/<SCOPE> resolves or activates a cell.
  • /evict/<SCOPE> tries to evict a resident cell and reports the result.
  • /do/<ID> sends a direct request to an ordinary Durable Object.
  • POST /shutdown starts a graceful ownership handoff. handoff=preserve prepares a same-node reload.

/do/<ID> refuses every reserved runtime class, such as D1, Workflows, KV, and Queues, because their protocols can access application data. Use the HMAC-authenticated /runtime/<SCOPE> route for these classes.

POST /invoke serves one request from staged, not yet activated, script versions (cello invoke), so a migration endpoint runs before the activation that would serve it. Like /runtime/, it requires the per-request fleet HMAC plus the operator token when one is set; without the token it still requires the HMAC, which a tenant cannot mint. No public route and no peer tunnel reaches a non-activated version: the public listener has no such path, and tunnels carry cell calls, never this route. The invoke writes nothing to the bucket, moves no pointer, and serves staged code in fresh isolate pools that share nothing with live traffic and are retired after the request; cells resolve from the live generation, so bindings reach the real data. It runs to completion server-side even if the caller disconnects, so stopping the wait cannot leave a half-applied migration, and it answers only after waitUntil settles. See Invoke a staged version.

/peer/probe returns a signed diagnostic response. Do not call the other reserved peer paths directly.

Read an eviction result

An accepted eviction waits for the operation to finish, and a refused one returns at once. Concurrent callers can join one eviction, so the success count does not equal the stop count.

The Rust method AppHandle::evict returns Result<EvictSuccess, EvictError>. Evicted confirms a completed runtime stop, and AlreadyAbsent confirms settled local absence. A caller that needs a completed eviction must check for Evicted. Over HTTP, both return 200 with {"ok":true}. kind() and reason() on the error match the HTTP error body. A later request can reactivate the cell before the response arrives.

A missing cell or a settled Inactive, Dormant, or Remote cell is locally absent. A pending activation, stop, or ownership transfer prevents that result. The request does not evict a remote runtime. A dormant cell keeps its ownership and its hibernated host sockets. The node refuses during preservation, reload, or local inventory confirmation, or when it lacks authority.

An error body has this form:

{"ok":false,"error":{"kind":"refused","reason":"cell_active"}}
Status Kind Reason Cause
409 refused cell_active The cell has active work or a socket that requires a runtime.
409 refused cell_transitioning The cell has another lifecycle transition.
409 refused alarm_imminent The alarm residency policy retains the cell.
409 refused alarm_uncovered The alarm coverage is unconfirmed, or a firing alarm blocks eviction during pressure shedding.
503 refused node_unavailable The node cannot accept the eviction.
503 refused eviction_limit The node has reached its eviction concurrency limit.
409 cancelled new_activity A new request cancels an accepted eviction.
409 cancelled alarm_activity An alarm observation or firing cancels an accepted eviction.
409 cancelled node_fenced The node loses its authority during the eviction.
503 failed actor_unavailable The request cannot reach the Actor.
500 failed reply_lost A delivered request loses its reply, so its outcome is unknown.
500 failed durability_failed The durability verification fails.
500 failed durability_timeout The durability verification exceeds its operation deadline.
500 failed runtime_stop_failed The runtime stop reports a failure.

A malformed scope returns 400. An error does not prove that the runtime is still resident. The runtime stop has no overall timeout, so a stop that never returns keeps the request pending.

Set the forwarded-header policy

Cello ignores X-Forwarded-Host and X-Forwarded-Proto by default. Set --trust-forwarded-headers or CELLO_TRUST_FORWARDED_HEADERS=1 only when a trusted proxy replaces both headers. Cello uses the last value in each header, so an earlier client value cannot override the proxy value.

Cello takes the path and query from the request target and ignores the scheme and authority of an absolute-form target. Without a trusted proxy, the Host header sets the hostname in request.url. Cello accepts a hostname, an IPv4 address, or a bracketed IPv6 address, with an optional port. It rejects malformed and noncanonical values and falls back to cello.local.

The hostname is still client-controlled. Do not use an unchecked hostname for an authorization decision. Use a trusted proxy, or check the hostname against a list in the Worker.

Limit request bodies

The public listener and /do/<ID> limit a request body to 1 GiB by default. Set CELLO_MAX_REQUEST_BODY_BYTES to a smaller positive value to lower it. Cello returns 413 for a declared oversized body and when a Worker reads past the limit.

For a method other than GET or HEAD, /do/<ID> streams a body of unknown length or of at least 1 MiB. It collects a smaller body before dispatch.

Restrict Worker egress

By default (CELLO_EGRESS=open) Worker code may dial any address with fetch(), connect() sockets, and outbound WebSockets. CELLO_EGRESS=public refuses dials to every non-public address: loopback, private (RFC 1918), link-local, carrier-grade NAT (RFC 6598), IPv6 unique-local, unspecified, multicast, reserved (240.0.0.0/4), the IPv4-mapped and transition-embedded (NAT64, 6to4, Teredo, IPv4-compatible) forms of those, and the cloud metadata address (169.254.169.254).

CELLO_EGRESS_ALLOW lists exceptions as comma-separated CIDRs (any port) or host:port pairs (that host on that port only), for example an object store endpoint on a private address:

CELLO_EGRESS=public CELLO_EGRESS_ALLOW=10.89.0.7:9000,10.89.0.0/24

A hostname entry resolves on every check, so a moved endpoint is re-allowed without a restart. A host:port entry is port-scoped on every hop: the initial request and each redirect hop match the entry only on its port, against the hop’s full URL. A redirect to a bare IP literal matches only IP, CIDR, and IP:port entries — never a hostname entry, which cannot be resolved in that path — so allowlist redirect targets in one of those forms. The node refuses to boot when either variable is invalid.

Enforcement happens on the resolved address at connect time: hostnames are resolved and filtered before the dial, every address is judged, and redirects re-check on every hop, so DNS rebinding and redirects cannot bypass the policy. In public mode tenant clients never use a proxy (HTTP_PROXY/HTTPS_PROXY/ALL_PROXY are ignored): through a proxy the target would resolve with the proxy’s DNS and redirect hops would expose only the proxy’s address, so a proxied fetch cannot be policed. Cello’s own traffic — the object store, peers, control, telemetry, and container-engine clients — never goes through these checks and keeps its proxy handling. Dials to container endpoints this node published via getTcpPort() are exempt, and that exemption is node-global: any Worker on the node may dial a published container endpoint by its address and port, the same reachability the nft gateway granted. A refused fetch() rejects with a TypeError whose message starts fetch failed: egress policy refused ...; a refused connect() rejects the socket’s opened/closed promises, and a refused WebSocket fires error and closes with 1006, each naming the refused address and reason. Every refusal also logs an egress_refused warning with the target’s scheme, host, and port, the refused address, and the reason — never the full URL, whose userinfo or query may carry secrets.

Protect the fleet bucket

The fleet bucket is the root of authority. It stores the deployments, the cell state, the ownership and node leases, and the shared peer-authentication secret. A holder of the bucket credentials controls the fleet. Give each credential access to one fleet bucket only, and rotate it after a suspected disclosure.

Cell ownership

Each cell is a SQLite database with one writer. An ownership epoch fences each cell, so a node that loses its lease cannot modify the current state. This fencing protects storage consistency. It does not isolate hostile applications. See what Cello guarantees and limitations.

Was this page helpful?