How we rebuilt floo's build system on Firecracker in a weekend

floo is built for internal software: the tools, dashboards, and AI systems a company runs on, built by its own people and their agents. That kind of software changes all the time. Someone asks for a new column, an agent pushes the fix, and everyone wants to see it live. Every one of those changes waits on a build.

Last week that wait cost us. We were onboarding a technical team that had used other deployment platforms before. They liked what floo gave them beyond hosting: sign-in, managed databases, and an API their agents could work through. But their first deploy took about 6 minutes, and they were used to rebuilds in about 30 seconds. We were adding value in the hard places and missing an easy one.

This is how we fixed it, mostly over one weekend.

Where did the time go?

We timed every stage of that team's first deploy. It took 6m10s:

• Build, 4m47s. npm ci took 57 seconds, next build 76 seconds, and saving the build cache 51 seconds. Starting BuildKit took 12 seconds. Uploading source, pushing the image, and waiting in Cloud Build's queue took the rest.
• Creating the service, 15 seconds.
• Routing, 67 seconds. On a first deploy we waited for Google's IAM to let our gateway reach the new service.

We shipped the quick fixes first. Public apps now skip the first-deploy IAM wait. BuildKit comes from our own registry mirror instead of Docker Hub. Builds moved to 8-core machines.

Why wasn't a bigger machine enough?

Bigger machines made the build itself faster, but Cloud Build's 8-core machines waited 41 to 61 seconds in a queue before a build even started. We had hit the same wall a month earlier and moved to the default 2-core machine to avoid it. That fixed the wait and made heavy Next.js builds slow.

So the choice was a fast machine that started late or a slow machine that started on time. A managed build service gave us no third option, because we did not control the machine. That made the decision for us.

What did we build instead?

On Friday we decided to run our own build infrastructure. That meant dropping one of our own rules, "Cloud Run only, no VMs," because build speed matters more to the product than that rule did.

The design:

• One always-on build host. An 8 vCPU machine with nested virtualization, in its own private network with no public IP.
• A fresh Firecracker micro-VM for every build. Firecracker boots a VM in about a second, so a build never waits for a machine. Each VM has its own kernel, memory, and disk, runs under Firecracker's jailer, and is destroyed when the build ends.
• Per-app, short-lived credentials. For each build, floo mints a 15-minute token for that one app's build identity. That token is the only credential that enters the VM. The host's own identity cannot read customer source or push to customer registries.
• Closed networking. A build VM can reach the internet and package registries. It cannot reach the host agent, the cloud metadata server, other builds, or floo's private network.
• A cache only that app can see. Each app gets its own cache disk on the host, mounted only for that app's builds.
• Trust the registry, not the report. The host reports an image digest, but floo deploys only after confirming that digest exists in the app's own registry with this deploy's tag.
• Cloud Build as the fallback. If the host does not pick up a build within 5 seconds, or fails on our side, the build goes to Cloud Build as before, and we never bill the failed attempt.

How did it come together in a weekend?

Friday was the scope and a hand-run prototype. Over Saturday and Sunday we built both halves. On the floo side: a job queue the host claims from, heartbeats so a silent host fails over, billing from the VM's real start and stop times, and the fallback. On the host: the agent, the micro-VM scripts, and the Terraform for the host, its network, and its permissions. The floo side merged Sunday, the host merged Monday after review, and we timed it that night.

Most of the code was written with coding agents, and every change went through automated review before merging. That review earned its keep. It caught a way to slip control characters into a build setting that could have pushed the build's token into its log, and a way a build VM could copy a host file into the logs a customer sees. Both were fixed before any customer build ran on the host.

How much faster is it?

We timed a Next.js app with about 1,000 packages, slightly heavier than our customer's, on the same setup with Cloud Build and with the build host. Times are from push to live.

• First deploy: 7m09s on Cloud Build, 3m35s on the floo build host.
• One-line change: about 3m58s on Cloud Build, 2m14s on the floo build host.

That is about half the time on a first deploy and about 44% less on a follow-up.

What is left in those numbers is mostly the build itself, not waiting on floo. On the one-line change, about 91 seconds of the 2m14s is next build. The queue is gone, the micro-VM boots in about a second, and the deploy starts when the build finishes. The remaining fight is the framework doing its work, which is why the cache work below matters.

What's next?

We are not at our target yet. We want a first deploy under 2 minutes and a follow-up under 60 seconds. Most of what is left is next build itself, which still takes about 91 seconds on a small change. Next up:

• Keep the Next.js build cache between builds, on the app's own cache disk, so a small change only recompiles what changed.
• Share a read-only npm cache filled by floo, so first installs skip most downloads.
• Set up each app's build resources when the app is created, not on its first build.
• Start the first build when a repo is connected, so the first deploy is often ready before anyone asks for it.

The first item is the one that closes the gap this started with. The team that pushed us was used to rebuilds in about 30 seconds, and that is what a warm framework cache buys you. Until ours persists between builds, we will not match it, and we would rather say that plainly than round 2m14s down.

Why we care this much about build speed

For internal software, the build is the feedback loop. An agent that pushes a fix waits for the deploy to go live before it can check its work. A person who asked for a change waits to see it. Shorter builds mean more turns of that loop per day for the people and agents building on floo.

Running our own build machines gives us control over that loop. We pick the hardware, keep the caches warm, and stop waiting on someone else's queue. The build host is now live in production for the team that pushed us to build it, and other customers come next.

Getting started

If you use an agent, install floo from getfloo.com/agents.md and tell it what to build. This is the build system it ships on now.

Originally published on X.