How Effective AI builds agents that ingest 10M+ regulatory filings with Kernel

Effective AI is an agent platform purpose-built for insurance, one of the most regulated industries in the U.S. Every state sets its own requirements for rate filings, policy forms, and actuarial justification, and those requirements shift constantly. The platform gives insurance teams AI agents that research filings, analyze competitor pricing, and build structured models of the products carriers sell. They’re trusted by Skyward Specialty Insurance, ERGO-NEXT Insurance, Monoline, and many more.

Browser infrastructure is a critical layer in Effective’s stack. The platform uses it in two ways: first, to power always-on data collection agents that ingest regulatory filings, rate manuals, circulars, and bulletins from public government websites across all 50 states, building the structured knowledge base that Effective’s AI agents reason over. Second, to automate workflows on their customers’ on-prem systems: policy administration, claims management, and underwriting platforms that sit behind corporate firewalls and have never been automatable because there are no APIs.

Problem: Regulated data scattered across hundreds of sources with no API

Before Effective, insurance companies maintained internal checklists and standard operating procedures (SOPs) for every new program launch, pricing update, or state expansion. Dedicated employees manually tracked regulatory circulars published by state Departments of Insurance, maintained internal databases, and cross-referenced requirements that changed without warning. For a large insurance company launching a new product in a new state, the work goes beyond just understanding filing requirements. Teams need to research how competitors price similar products, analyze the regulatory regime’s stance on rating variables, trace form revision histories, and build actuarial justification, a process that could take multiple weeks per state. The data is all public, but it’s published in dozens of different formats across hundreds of government websites: sometimes as web pages, sometimes as PDFs, sometimes as obscure document formats. There are no centralized APIs to access the state regulatory databases.

And the regulatory data problem is only half the picture. Insurance companies run many critical business operations on on-prem legacy systems that are 10 to 15 years old, behind tight corporate firewalls, with no modern APIs. The only interface is a browser, because these systems were built for humans.

The team had already been running headless browsers on their own GCP infrastructure to tackle both problems, but they knew the volume of browsers needed to launch would be significant, and maintaining browser infrastructure was not where they wanted to spend engineering time.

“The amount of browsers we want to be launching would be quite a lot. And this is not an infrastructure we had any intention of spending our time in maintaining. Every hour we spend managing a custom solution is an hour we’re not helping our customers get insurance products to market.”

Kunal Singhal, Co-founder & CTO, Effective AI

Solution: Selecting Kernel for massive scale and bot anti-detection

Effective’s Co-founder & CTO, Kunal Singhal, discovered Kernel through social media and spent time reading about our technical architecture, which gave them two things their self-hosted setup couldn’t. First, the ability to run 24/7 data collection with rolling batch processes distributed across thousands of browsers in parallel. Second, IP masking and stealth capabilities for accessing third-party services.

Beyond scale and bot anti-detection, Kernel’s authenticated browser sessions would let their agents connect to customer on-prem systems through reverse tunnels. Effective’s agents could open a browser, wait for a user to log in, and then use that authenticated session to automate tasks on legacy systems without building bespoke API integrations.

Within days of testing things out, the team decided to fully integrate Kernel.

“Just reading about Kernel felt like a lot of good technical decisions are going to be made there, so we decided to compound on that instead of trying to host it in house.”

Kunal Singhal, Co-founder & CTO, Effective AI

Results: 10M+ regulatory documents processed and a new way to work

Today, Effective has ingested 10 million regulatory documents using Kernel-powered browser agents running around the clock, and that number grows continuously. This data feeds Effective’s AI agents, which don’t just retrieve documents but analyze them: extracting rate tables, decomposing rating algorithms, comparing policy form language across the states, and building structured product models that insurance teams work from daily. What used to take insurance teams weeks of manual research, such as understanding how a competitor prices commercial general liability across a dozen states, is now a question answered in minutes, grounded in primary-source filing data.

The interactive automation capability has been equally impactful. What started as a simple proof of concept has spread to hundreds of users inside a single customer organization. People who previously needed IT approval and months of integration work can now prototype and ship automations immediately on systems that were never automatable before, because those systems were only ever exposed to humans through browsers.

“What used to be a multi-month collaboration project with IT is now a question answered in minutes, or a quick browser automation. The flexibility you get with browser agents is enormous.”

Kunal Singhal, Co-founder & CTO, Effective AI

If you’re building AI agents that need to interact with legacy systems, government websites, or any service without a modern API, browser infrastructure is the bridge. Let us handle it so you can focus on the problem you’re actually solving.