Ten thousand conversations. No two the same.
The user side is a model too. It gets a character and something it’s trying to get done, it follows up on whatever your agent just said, and it stops when it’s satisfied or when it gives up. A scenario isn’t a script that replays. It’s a starting position, and ten thousand runs of it are ten thousand different conversations.
enterprise Free in your cluster while you build. An agreement when it carries real traffic. See exactly where the line falls →
One scenario is run repeatedly. Each run takes a different number of turns and ends a different way: some resolved in two or three, one escalated after six, one abandoned after eight.
Same starting position every time. How long it takes and how it ends are up to the conversation. That’s where you find the run where somebody won’t take the first answer.
You can make them harder.
How much a simulated user pushes is something you set. Polite and specific, or vague and impatient, or someone working quite hard to talk your agent out of the rules it was given. Same scenario, more pressure. The difficult ones are the ones worth keeping in the suite.
Does it hold at Monday-morning volume?
Ten thousand cases through the real tools: a night’s invoice backlog, a claims queue, a support line at 9am. Not a synthetic HTTP flood.
Does it decide the same way, ten thousand times?
The same cases you already wrote, run wide instead of once and scored the same way, so the one-in-a-thousand judgementjudgment call shows up here.
What a scored run leaves behind.
Every run is scored, so running wide leaves a labelledlabeled record of your own situations judged by your own standards. Generated inside your boundary.
Nobody else has your awkward cases.
Every run leaves a scored record: what the simulated user was trying to get done, what your agent did about it, and whether your own assertions and judges accepted the result. Run that wide and you accumulate a labelledlabeled corpus of your own situations, judged to your own standard, inside your own boundary.
Most teams don’t need to fine-tune anything, and we’ve argued that at length. But if you decide you do, the expensive part was never the training. It’s having enough correctly labelledlabeled examples of your actual work, and that’s the part you were generating anyway.
Long before anyone mentions training, the same corpus is a regression suite, a benchmark for whatever model ships next, and the evidence for what your agent has been doing.
A fleet run costs tokens, so spend is tracked by agent, provider and workspace as it happens, and a workspace can carry a daily or monthly budget with an action when it’s exceeded. A limit that holds.
And nothing forces a run to use the expensive model. Point it at a local or self-hosted endpoint for the volume work, and keep the frontier model for the runs where its judgementjudgment is the thing being measured.