The Guest Never Gets a Port

The hard part of building an edge gateway to run other people's code isn't the sandboxing. Wasmer will give you a WASI sandbox in about four lines, and you can hand it a directory and go to lunch.

The hard part is that once you're actually hosting untrusted workloads, you owe someone an invoice, and the number on that invoice comes from a counter in your metrics endpoint. Everything else is negotiation with the runtime library.

pk-edge is the gateway. It's a Rust workspace, about 4,500 lines of source across 37 files, and it has existed for three commits. This is what I learned building it.

One socket on the host

The single most important line in the whole codebase is in src/capabilities/network.rs:

pub const GUEST_HTTP_PORT: u16 = 8080;

That reads like a constant and is not one. A guest running in http mode binds 127.0.0.1:8080 inside a LoopbackNetworking provider an in-process virtual network stack. It is not a host TCP port. No host listener ever exists for the guest.

Only pk-edge binds a host socket, on 0.0.0.0:8080 by default, and it reverse-proxies inward over a virtual TcpSocketHalfRx/TcpSocketHalfTx pair. The guest cannot see a host interface at all, because there isn't one from its point of view.

This is worth being pedantic about, because the obvious cheaper implementation is to just bind the guest on a random high port and route to it. That version has a name and it is a denial of service: every app you host consumes a port, the host's ephemeral range is shared with everything else on the box, and now your application count is bounded by the kernel rather than by your billing model.

Deny by default is a function, not a config value

Guest egress to host services is controlled by a per-app allowlist, and it is deny-by-default in the strict sense an empty allowlist produces None rather than an empty ruleset:

pub fn ruleset_string(&self) -> Option<String> {
    if self.loopback_ports.is_empty() {
        return None;
    }
    // ...
}

None means no host networking bridge gets built at all. That's different from "build a bridge that permits nothing," and the difference matters: one of those is a sandbox with a rule in it, and the other is a sandbox that was never given the capability.

The ruleset that does get built is a string, which is unglamorous and very easy to audit:

ipv4: allow = { 127.0.0.1 }:{ 5432 }/in, ipv4: allow = { 127.0.0.1 }:{ 5432 }/out,
ipv6: allow = { ::1 }:{ 5432 }/in, ipv6: allow = { ::1 }:{ 5432 }/out

Both families, both directions. A guest that wants to reach Postgres declares loopback_ports.allow: [5432] and gets exactly that. A guest that wants to reach the metadata service, or bind a listener the outside world can see, has no vocabulary for expressing it.

The whole crate runs under [lints.rust] unsafe_code = "forbid", which is a runtime dependency's problem in disguise and not mine. Wasmer itself is pinned through [patch.crates-io] across nine crates to a single upstream revision, because I need the virtual networking APIs that hadn't shipped in a release. Forcing the whole dependency graph back onto crates.io releases would have meant giving up the only part of the design I actually care about.

Suspended is not an outage

The app lifecycle has three states and they do not map onto HTTP naively:

state	Sandbox	Response
active	Spawns and serves	Whatever the guest returns
suspended	No sandbox	503, content-negotiated status page
disabled	No sandbox	403, content-negotiated status page

The distinction is billing. A suspended app isn't serving and isn't charging, so 503 is honest it's temporarily unavailable and will come back. A disabled app is a decision, and 403 says so. If you collapse both into 503, your support inbox fills with people asking why their site is down when the real answer is that they didn't pay.

Unknown app IDs get a hardcoded negotiated 404 from assets/404.json or assets/404.html. Content negotiation is a real q-value parser in status_pages.rs rather than a contains("text/html") check, because a browser sending text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 is the common case and it should get HTML.

Note that suspending drops the warm instance rather than parking it. Reactivation should give you a clean spawn, not the memory footprint of whatever the app had accumulated before you stopped paying for it.

Warm instances get drained, not killed

Instances are cached per app_id with two independent clocks, and the config refuses to start if they're set in a way that makes no sense:

if instance_hard_max_age < instance_idle_ttl {
    bail!("INSTANCE_HARD_MAX_AGE_SECS ({:?}) must be >= INSTANCE_IDLE_TTL_SECS ({:?})", ...);
}

A sliding idle TTL of 300 seconds parks quiet apps, and a hard max age of 3600 seconds guarantees every instance is replaced at least hourly regardless of traffic. The hard age exists for a reason that has nothing to do with resources: it bounds how long a customer's code runs in one process, which means a slow memory leak or an accumulating global is a billing problem rather than an indefinite one.

The part I like most is that replacement never kills a running request. Each slot holds a current instance and a Vec of retiring ones, and in-flight requests are tracked by an RAII ActiveGuard that decrements an AtomicUsize on drop. The reaper runs every 30 seconds and closes an instance only when active_requests() == 0 and it's dead, retiring, or idle. A request that started at second 3599 of an instance's life gets to finish.

Generation IDs are ULIDs, stamped at construction, which makes "which process served this request" answerable after the fact. That turns out to matter more than I expected when you're the one explaining a customer's 502.

The health check must not be billable

This is the requirement I'd write into the first paragraph of any spec for a metered gateway:

pk_edge_requests_total{app_id, mode, status_class}

That series is the invoice. Which makes its definition a product decision wearing a metrics costume.

So /health and /metrics are deliberately uninstrumented, and the code says so at both sites:

// Intentionally uninstrumented so probes do not inflate app request counters.

A load balancer checking this gateway eight times a second across three availability zones generates twenty-four probes a second, or roughly sixty million requests a month. Those requests would then sit in the same counter as your customer's traffic, and the resulting invoice has a monitoring surcharge nobody agreed to.

And that surcharge is unfixable in the worst way, because the number is technically correct. There is no way to explain to a customer that the gateway billed them for their own uptime check without admitting the gateway is measuring the wrong thing.

The same reasoning drove the rest of the metric set. Spawns and purges are labelled by reason (cold, hard_age, timeout_respawn, and on the purge side idle, error, hard_age), so "why is my app slow" has an answer before anyone opens a debugger. Handler timeouts and config fetches get their own counters because a guest that is wedged, a control service that is down, and an app that was simply suspended all present to the customer as the same 502, and none of those are the same bug.

/metrics is unauthenticated, and the README's advice is to restrict it at the network layer rather than bolt a token check onto it. If you're already scraping it, you've already solved that problem; if you haven't, you have no reason to have the endpoint.

Two ways to run code, one of them weirdly nice

http mode is what most people expect a reverse proxy to a server the guest runs, with hop-by-hop headers stripped.

handler mode is a warm process speaking NDJSON on its own stdin and stdout:

{"method":"handle","input":{...}}
{"ok":true,"output":{...}}

No ports, no HTTP server inside the sandbox, no header parsing. The host validates input and output against a JSON Schema pulled from a separately-fetched event type, enforces a timeout clamped by HANDLER_TIMEOUT_MAX_MS, and serialises invokes with an invoke_lock because the pipes are shared and a half-written request would be someone else's problem.

Timeout semantics follow from the same billing logic as suspension. A handler that exceeds its event-type timeout returns 504 and its instance is invalidated a guest wedged mid-request is not warm, it's broken, and leaving it in the cache means every subsequent request inherits the wedge.

Testing against real things

The e2e suite that actually runs a guest is #[ignore]d, needs MinIO and a built WASI module, and spawns the binary as a child process. That last part isn't incidental:

The pk-edge binary is spawned as a child process because the S3 client only reads AWS_* from the environment, and this crate forbids unsafe (which std::env::set_var requires on edition 2024).

A crate-level lint forbidding unsafe quietly costs you your ability to mutate the environment in-process, and the correct response is to spawn yourself rather than to add an allow. The companion suite drives the real Axum router through the real instance cache using a stub control client that rejects every app before a sandbox would spawn, so the common paths get tested with neither Wasmer nor S3 present. wiremock covers the event-type fetches.

There's also pk-dns in the workspace, which makes the whole company's DNS resolution a question answered by a customer's own sandbox. I'll write about that separately because it deserves it.

The lesson

If I'd written this gateway in the other order build the sandbox, bolt on counters, discover later that the health check was billable the mistake would have been expensive to undo, because by then the counter would be an artifact that customers had been quoted.

The trust boundary came first. Virtual networking because guests must not hold host sockets, deny-by-default egress because the alternative is auditing capabilities after the fact, hard max age because unbounded process lifetime is unbounded liability. Every one of those decisions constrained the accounting for free.

If you find yourself designing a metered gateway, decide what a request is before you decide how to count one. The billing counter is the last thing you write and the only thing you cannot quietly change later.