Why our machines take 80 seconds to boot, and where the time goes

A new shedix machine is ready one to two minutes after you ask for it. When we first timed the three stacks, on 1 November, the web stack took 80 seconds, Python between 59 and 113, and Go & Rust 80. Some sandbox services start in about a second. This post covers where our time goes, what we cut, and what we haven't built.

Our numbers

Every shedix machine is a whole VM on Google Compute Engine, booted from an image we bake for its stack. The times we've recorded:

These are a handful of runs on different days and images, and our notes don't always record the machine size. They don't line up neatly either: a copy, which boots the same services as a new machine, was ready sooner than a restart of a machine whose disk already existed.

"Ready" means something specific. For our stacks, it's when the machine has reported this boot's address for its web terminal. A Plain machine is ready when its startup script reports that it's up, which is less work.

Where the seconds go

Between the API call and "ready", five things happen, roughly in order:

1. Google makes the VM. It passes through Compute Engine's PROVISIONING and STAGING states while Google allocates resources and prepares it, then enters RUNNING as it boots (instance lifecycle). Its boot disk is made from the newest image in the stack's image family.
2. Linux boots for the first time. The image is made generic when it's baked, so each machine creates its own SSH host keys and machine ID, and cloud-init runs as if on a first boot. New machines also rename the image's user after their person.
3. Our services start: the web terminal, VS Code in the browser, a small server for files and ports, three small helpers (agent state, activity, and a disk flush for copies), and three Cloudflare tunnels.
4. The tunnels get addresses. The terminal's tunnel writes its address, stamped with the time, to a guest attribute, retrying if the write fails.
5. Our API notices. A create call with ?wait= checks every 3 seconds, ignores reports older than this boot, and answers once the terminal's address is there.

We haven't timed these steps separately, so we can't say how the 80 seconds divide. To find out, we'd time Google's state changes, run systemd-analyze critical-chain on a fresh machine, log when each tunnel gets its address, and allow for up to 3 seconds of polling.

One wait is our choice: readiness waits for the web terminal even when the caller is an agent that only uses SSH. In a 12-minute setup session we traced, that wait was one of the delays.

Baking an image instead of installing at boot

The first version installed its tools when a machine booted: system packages, Node, the web terminal, Claude Code and Codex. That took about five minutes. Now a script boots a temporary builder VM from Ubuntu 24.04 with our bake script as its startup script, waits for it to report "done" through a guest attribute, stops the builder, saves its disk as a new image in the stack's family, and deletes the builder. A build takes six to eight minutes, and new machines use the new image as soon as it exists.

The bake also removes boot-time work: snaps, the apt timers that would compete for the CPU at boot, and Docker starting before anything needs it. Its last lines make the disk generic, as systemd's machine-id documentation asks of images used on many machines:

truncate -s 0 /etc/machine-id
rm -f /var/lib/dbus/machine-id
cloud-init clean --logs

The price is freshness. We rebuild images by hand, and the web image needs it about monthly, because it carries a warm npm cache: about 1.4 GB of common packages as they were on bake day. It speeds up the work after boot: with compressed swap and quieter npm settings, it took a typical Next.js install on an M from 77 seconds to 37.

A fresh disk is slow to walk

Since 2 November, a new machine's first boot renames the image's user, dev, after its person. The first version used usermod, and the step took 15.6 seconds; a trace showed usermod going through the whole home folder after the rename, which took 13 of them. The rewrite edits the account files directly, as usermod would, and moves the home folder with one rename on the same disk. It now takes 2–5 seconds, once; restarts and copies skip it.

Our notes put those 13 seconds down to a disk whose data was still arriving from its image. That's how AWS volumes made from an image work: Jacob Gillespie of Depot explains that each block comes from S3 the first time it's read, and saw a first boot spend 5 seconds on the kernel and 5 on systemd loading unit files. Google's docs say clones and disks restored from snapshots reach full performance without pre-warming, and the pages we read don't cover disks made from images. We haven't repeated the walk on a warmed disk, so it's our best explanation, not a proven one. Either way, a first boot shouldn't touch every file.

What didn't help

We measured three changes on throwaway machines on 1 November and left them out. These timed the work after boot:

• SSD persistent disks: installing two services at once on an M took 148 s, against 142 s on the balanced disks we use, for about 70% more per GB.
• Premium networking: a 45 MB upload took 45–108 s, against 53–56 s on the Standard tier. No steady gain, and its outbound traffic costs more.
• Docker images pulled ahead of time: Postgres 16 pulled in 9.6 s and Redis 7 in 4.8 s, and only an exact tag match would save that.

How microVMs start in a second

The fastest numbers come from skipping the boot. Aitor Mato and Laith Weinberger of Browser Use resume each browser's Firecracker VM from a snapshot taken just before Chromium launches. Their VM cold start is under 400 ms, and creating a browser through their API takes 825 ms at p50 and 1.35 s at p99. As they put it, "Resuming a VM is much faster than booting one."

Thomas Ptacek's post on Sprites explains why a Sprite appears in a second or two: every Sprite starts from the same standard container, so Fly.io keeps pools of empty ones standing by. He notes that creating an ordinary Fly Machine can take over a minute.

Depot took EC2 from about 40 seconds to under 5 by booting each instance once to warm its volume, stopping it, and starting it again when a job arrives. They also removed cloud-init, which slowed both boots and restarts on their image.

Each moves the slow part to before anyone asks: into a snapshot, or into a pool kept ready.

What would make ours faster

Two changes would matter most. Neither is built.

• Suspend instead of stop. Our machines sleep after an hour unused, and today sleep is a full stop. Compute Engine's suspend moves a VM's memory to storage and back, so processes survive; Google suggests it for developer workstations that take long to get ready. A machine can stay suspended for up to 60 days, and its memory is still charged meanwhile. We haven't measured how fast our images resume.
• A warm pool for new machines, booted from the image and waiting to be claimed. Our stopped machines take 47 seconds to start, so a pool of stopped ones wouldn't come near Depot's five seconds. It would have to hold running or suspended machines, which cost money while they wait, and the per-person setup done at first boot would move to the moment of claiming.

A smaller idea we haven't tried: answer agents once SSH works. First, though, we'd time the five steps above, and find out why our numbers vary as much as they do.

If a minute or two matters for how you work, our post on what an always-on agent machine costs covers when to stop a machine and when to delete it.