# Why every shedix machine is a whole VM, not a pod > Why shedix gives each machine its own VM instead of a container on a shared cluster, what that gives an agent, and what it costs in boot time and density. *2026-11-02 · agents, remote-dev, security* In October 2024, Gitpod said it was leaving Kubernetes, after six years of running developer environments on it for about 1.5 million users. Christian Weichel and Alejandro de Brito Fontes [explained why](https://ona.com/stories/we-are-leaving-kubernetes): Kubernetes "is built to run well controlled application workloads, not unruly development environments." A coding agent's machine is about as unruly as development environments get. The agent installs whatever it likes, runs Docker, fills disks, and sits idle between bursts of work. So shedix doesn't run machines as pods. Each machine is one virtual machine on Google Compute Engine, managed by a single Cloudflare Worker. This post covers the whole design, what it gives an agent, and what it costs. ## What Gitpod learned Their post is worth reading in full. Three of its problems shaped our design: - **Storage.** Local disks tied a workspace to one node and were costly to back up. Network disks attached and detached unpredictably, with limits per node and per zone. - **Startup.** Workspace images could pass 10 GB, so they built pre-pulls, lazy pulls and pools of placeholder workspaces to hide the wait. - **Isolation.** People wanted root. Giving it to them on a shared node took user namespaces, shifted file ownership, a masked `/proc`, FUSE through eBPF filters and a facade over runc so Docker would work. They tried microVMs too (Firecracker, Cloud Hypervisor and QEMU) and found better isolation, but also overhead and gaps of their own. ## Our design in one picture ``` browser, CLI, agents │ ▼ Cloudflare Worker ──────> Compute Engine: one VM per machine, (page, API, CLI) from a prebuilt image, with its own disk │ ▼ Postgres: owners, history, tokens ``` - The Worker serves the page, the API and the CLI, and calls Compute Engine's REST API with a token it signs itself. There are no servers of ours to run. - Each machine is one VM with its own persistent disk, booted from a prebuilt image for its stack, or from the OS as Google ships it for a Plain machine. - Stopping a machine ends its compute charge and keeps its disk. Deleting it deletes the disk. - Machines sit on a network of their own, which drops traffic between machines in both directions. SSH comes in with keys, and outbound traffic is open. - Compute Engine is the source of truth for each machine's state. Postgres keeps what the cloud doesn't: who owns what, each machine's history, tokens and keys. ## What a whole VM gives an agent Two of Gitpod's problems turn into things we don't have to build. **Storage:** a disk is created with its VM, kept through every stop and deleted with it. We never move a disk from one machine to another. Copying a machine is Google's work: one copy is a disk clone, which took 6 seconds, then 13 seconds to create the machine, about 20 seconds in all. **Isolation:** root is `sudo` on a VM that holds nothing of yours, not even a cloud service account, so there are no cloud credentials on it to find. The kernel, systemd and every process belong to that machine. Docker runs as it does on any server, so nothing needs `--privileged`. On a machine with one of our stacks, you can check from its terminal: ```bash ps -p 1 -o comm= # systemd: the machine boots like any server uname -r # its own kernel, not a host's docker run --rm hello-world # no --privileged, no socket borrowed from a host ``` For an agent, that means a machine it can treat like a server. It can install system packages, run Postgres and Redis in Docker (their images pulled in 9.6 and 4.8 seconds on a machine), drive Chromium with Playwright, leave services running in tmux, and find its files where it left them after a stop. ## What it costs us **Boot time.** Startup is the problem we pay for. Installing tools at boot took about 5 minutes. Baking them into an image for each stack brought a new machine down to about 60 to 110 seconds: Web 80 s, Python 59–113 s, Go & Rust 80 s. Fly.io made the opposite trade with Sprites. Thomas Ptacek [describes](https://fly.io/blog/design-and-implementation/) pools of empty, pre-staged Sprites on each server, with storage kept in object storage behind a local cache, and Kurt Mackey [puts creation](https://fly.io/blog/code-and-let-live/) at one to two seconds. For us, faster starts would come from suspending machines instead of stopping them, and from a pool of machines kept ready. Neither is built. New disks add a cost of their own: a fresh disk is still being read from its image for a while after boot. A first-boot step that renamed the machine's user took 15.6 seconds, 13 of them spent walking a home folder on such a disk. Rewritten to skip the walk, it takes 2 to 5 seconds. **Density.** A cluster can pack idle environments tightly and overcommit memory. A VM gets its memory whether it uses it or not, and a stopped machine still holds a disk, which costs us money while nobody uses it. Inside a machine, compressed swap stretches memory. With it and a warm package cache, an S ran a project's two installs side by side in 179 seconds; before, they had to run one after the other and took 322. **Addresses.** Each machine has its own public IPv4 address, and addresses cost money and run out. exe.dev took a different route: David Crawshaw [explains](https://blog.exe.dev/ssh-host-header) how it shares IPv4 addresses, each serving one owner's VMs, and picks the VM from the user's SSH key and the address the connection arrived on. We weighed a shared path of our own, with outbound traffic through shared addresses and one SSH entry point, and turned it down because one person's abuse could get a shared address blocked for everyone. ## Running it with no cluster There's no control plane beyond the Worker and a schedule: - **Reports, not connections.** Machines report their state through Compute Engine's guest attributes, each stamped with a time, and the API ignores anything older than the machine's last start. When the Worker needs a machine to act, such as writing its memory to disk before a copy, it sets a metadata value and waits for the answer in a guest attribute. - **A five-minute schedule** stops machines unused for an hour and finishes copies that were waiting for a snapshot. - **The cloud's limits set the rules.** Google allows one clone of a disk every 30 seconds and refuses to clone a disk that was just made. So several copies come from one snapshot instead, and are ready 3¼ to 5½ minutes after the call. - **One interface for providers.** Creating, starting, stopping and copying go through one interface, so a warm pool, suspend or another cloud would be a new implementation, not a rewrite. The first limits we hit weren't in this code. They were the cloud project's quotas: public addresses, CPUs and disk. Every machine takes an address and a disk of its own, so quotas decide how many machines can run long before the Worker does. ## When a pod or a microVM is the better choice If you need thousands of short-lived sandboxes, starts in a second, or untrusted code packed densely onto hardware you run, VMs like ours are the wrong tool. Firecracker-style microVMs or a well-run cluster will serve you better. If you want a machine an agent can live on for days, with root, Docker and a disk that stays, a whole VM is the simpler answer, and that's the trade we made. For how the boundaries compare, see [container, microVM or full VM](/blog/container-microvm-or-vm-for-coding-agents). The [security page](/security) lists what's isolated, and [pricing](/pricing) shows each size by the hour.