On training VLM browser agents with open source tools

The most valuable AI infrastructure has historically been open source, from our team’s roots in cloud browsers and hypervisors to today’s AI frameworks and eval stacks. We believe that making agent training as accessible as possible will accelerate the maturity of AI, benefiting researchers, and developers alike.

That’s why we joined forces with Fireworks AI, a fast inference and fine-tuning platform, to make training VLM browser agents with open source tools as easy as possible.

When product launches align

Late last year, our friends at Fireworks open sourced Eval Protocol, which lets you do reinforcement fine-tuning on existing agents across any language, container, or framework. Shortly after, they launched Fireworks RFT, a service for simplifying reinforcement learning.

Around the same time, we introduced Kernel Browser Pools: pools of reserved browsers with the logins, cookies, and extensions your agents depend on. This turns out to be a big deal for training, because VLM browser agents need consistent, parallel environments—real browsers they can actually interact with.

Used together, Eval Protocol and Browser Pools give you a straightforward way to train VLM agents to perform computer use tasks with reinforcement learning.

What this repo gives you

In this open source repo, we’re publishing the three things you need to evaluate and fine-tune VLM browser agents today: Fireworks RFT and Eval Protocol for reinforcement fine-tuning, Kernel for cloud browsers, and Fireworks for serving your VLM models. We hope you enjoy it. Happy training.