Rehearse a year of agent behaviour in 22 minutes

The bugs that take an agent fleet down are not the ones a unit test finds. A login token for one of the model providers quietly expires. A rate limit hits on a busy morning. Someone edits a rule in the knowledge base and an agent that used to refuse a question starts answering it. All of these only show up when the real model runs against the real prompt with live data, which in most setups means they show up in front of a customer, weeks after the change that caused them.

At KURK (kurk.life) we got tired of learning in production, so we built a rehearsal room.

An isolated copy of the live setup

The simulator is a second profile on the same machine. It clones the production agent templates and the knowledge base into its own folder and never touches the live one. Inside that copy we can break things on purpose.

A scenario is a small json file that says what to change and what to expect. The changes come in two kinds. Data changes: a revenue spike in the sales export, a day of missing rows, a customer cohort renamed. Knowledge changes: a directive file deleted, or a rule reversed so the metric the agents were told to lead with becomes the one they must not quote. Then the scenario lists the questions to ask each agent, and for each reply what it must contain, must not contain and must cite.

The part that matters is that the agents answer for real. The simulator dispatches the actual models, with the actual prompts and the thinking settings production uses. No mocks. A mocked model returns whatever you told it to return, which is exactly why a mocked test sails straight through a dead token or a rate limit. When we went live the real-model run caught an expired provider token within hours, and pointed at the exact profile that was failing. The mocked version of that test would have been green.

Three runs, pass on two

Models are not deterministic. On any single run roughly one assertion in eight fails and then passes on the next attempt, and if you treat one run as truth you spend your week chasing ghosts. So every scenario runs three times and passes when at least two of the three pass. That threshold cleanly separates a real regression from noise, and the ones that keep landing on exactly two out of three go on a watchlist so we know which tests are flaky rather than broken.

A full sweep is eight scenarios, three runs each. It takes about 22 minutes of wall clock and costs a few dollars in model calls. Everything in it, the spike, the missing data, the rename, the deleted rule, the reversed rule, would take a year or more to happen naturally in production, and you would meet each one for the first time live. That is where the title comes from. A year of surprises, rehearsed before lunch.

What it found in the first week

Building it took about two days. In the first hours after it went live it caught the expired token, two flake patterns in agents that had looked stable, and a flaw in one of our own scenarios. That last one is worth spelling out. The scenario renamed a cohort in one file, but the same name appeared in five places across the knowledge base, so the agents saw four old names and one new one and, sensibly, went with the majority. A realistic rename touches every copy, and a scenario that only mutates one file tests nothing. We rewrote it.

The failover chain gets the same treatment. Every agent has a primary model and a fallback on a different provider, because a fallback on the same provider shares the outage. But a failover path you have never exercised is a guess, so it has to be exercised with the real models too, in the same sandbox, or you will not know it works until the day you need it.

We build the simulator second, after the knowledge base is versioned and the daily behaviour tests are running, because it borrows both. Then every change to a rule, a prompt, a model setting or a scenario goes through a sweep before it ships. Twenty two minutes is a cheap price for not finding out from a customer.