Lessons learned from scaling Chromium on bare metal

I previously was CTO of Clever. Part of my job as CTO was managing our AWS bill. And in the back of my mind, I always thought: man, if there were some escape from the death grip that AWS had on us, I’d jump on that. Because once you have a big enough AWS bill, there really aren’t huge levers you can pull. You can probably pull some levers that give you 10, 20, maybe even 30% savings. But 90%? No chance. That feeling never really went away. So when we started Kernel and I found myself staring at another compute‑heavy workload (Chromium) it was déjà vu. The second time around, I was even more convinced it shouldn’t be this expensive or inefficient, so I started exploring a bunch of ways we could run thousands of Chromium instances on bare metal.

First attempt: Docker containers (no K8s)

The first thing we did was the obvious thing: run Chromium in Docker and keep a warm pool. At the beginning, when there were maybe fewer than ten users playing with the API, this was fine. It was the easiest way to get something working and see what people actually wanted from “cloud browsers.”

But as you can imagine, running hundreds of hot chromium containers that are just sitting there waiting to be used is expensive. On top of that, it became clear from early design partners that many use cases required that these Chromium instances run for a very long period of time. Sometimes even for a couple days. Sometimes even indefinitely. This only multiplies the cost of running warm pools. Not to mention, containers share the host kernel by design. A single container escape means host access and full tenant exposure.

Second attempt: (Unikraft) Unikernels

In parallel, I’d been tinkering with Unikernels. Unikernels seemed to promise it all. At least in the pure form, you get: one process, single address space, app linked directly with a minimal kernel. In theory, you get tiny images, fast boot, and less overhead.​ Almost perfect for our use case. The obvious mismatch is that Chromium is far from a single‑process program. It spins up multiple processes and on top of that we run additional services alongside it. Still, the combination of strong isolation and millisecond-level cold starts was intriguing. More importantly, this approach would let us leverage memory snapshots, allowing us to capture the entire in-RAM state of a running browser instance and save it to disk. This would make it extremely cost-efficient to maintain instances over long periods of time.

I wanted to make this work, so I started talking to the Unikraft folks to see how far we could bend things. What we ended up with is something that stretches the strict definition of “Unikernel,” but works extremely well in practice. Unikraft, the Unikernel Development Kit, is more or less a fork of Firecracker that follows the principles of Unikernels. The important part is that it lets you run multi-process general purpose workloads while keeping all the isolation and performance benefits of Unikernel. With their help, we put Chromium on a Unikernel.

The way it works today is: when we spin up a browser, we pay the full cost of starting Chromium (think 5-10 seconds), and then we immediately put it into standby, basically snapshot RAM and save it to disk. From the outside, it feels like we have a huge warm pool of browsers ready to go. Under the hood, most of them are RAM snapshots sitting on disk, not consuming CPU or memory. When a request comes in, we read that snapshot off NVMe, wire it up to the network, and hand it back to you. The request is effectively decoupled from the cost of starting Chromium. And NVMe is fast enough that this takes on the order of 30 ms.

On top of that, this enables us to not charge you for idle time. Instead we put your browser VM into standby if the CDP connection goes idle. The trick is to not break the connection. To handle that, we built an ingress proxy layer that holds onto the CDP connection while the VM is asleep. The VM disappears, the proxy doesn’t. When we wake the VM back up, it resumes with the same state, the same CDP connection, and from your point of view nothing weird happened. This is super useful when you're waiting on an LLM response to take further action or waiting for a human-in-the-loop step.

At this point, we had strong isolation, super fast cold starts, and memory snapshots, so we’re done, right? Unfortunately, Unikraft was almost the perfect solution for us, but it’s missing live VM migrations and device passthrough (for GPU support). VM migrations is on Unikraft’s roadmap, but it’s not something we can treat as a solved problem today. The result is that these memory snapshots we rely on are effectively pinned to a specific host. That’s not great. If the host dies, the disk fails, or power disappears mid-session, the state goes with it. That's completely at odds with what users expect from a browser. Getting device passthrough was going to be even trickier. As mentioned earlier, Unikraft is a fork of Firecracker, and that hits a hard ceiling for GPU support. It has no device passthrough at all and explicitly has no plans on supporting it.

Third attempt: Hypeman (QEMU)

To get GPU support and live VM migrations, we had to build a brand new data plane while preserving the strong isolation, fast cold starts, and memory snapshots of the previous implementation. The question then became which virtualization technology to use. The options came down to two: Cloud Hypervisor, QEMU. We started with Cloud Hypervisor, but found the ecosystem and our early experiments with vGPUs didn’t get us as far as QEMU.

Today, our GPU-accelerated browsers run on a QEMU-based data plane. We open sourced this data plane and called it Hypeman. This allows us to do a bunch of cool things like get 60 fps (6x faster than our previous implementation), and, most importantly, play Doom, compiled to WASM, on our cloud browsers.

Probably don’t do as I do

At Clever, the ceiling on efficiency was essentially fixed. Infrastructure was something we consumed, not something we could fundamentally change ourselves. For many teams, letting someone else handle it is the right move. But for us, it turned out to be almost always better to own more and more of our infrastructure stack. That ownership has really started to pay off in how much performance and efficiency we have been able to squeeze out.