Everybody ships AI agents. Prove yours actually work.
PromptArena runs your agent against simulated users that hit its real tools, policies, and memory — including adversarial personas that phone your voice agents and score whether it holds the line: verify before acting, never approve what it shouldn't, escalate to a human when it should. Wire it into your deploy gate so nothing ships unproven.
promptarena-webui.png to
/public/atlas/screens/.
The text-eval tools are good. But what if your agent uses voice?
| PromptArena | Promptfoo | LangSmith | DeepEval | |
|---|---|---|---|---|
| Text eval, red-team scenarios, tool-call assertions | ✓ | ✓ | ✓ | ✓ |
| Runs in CI without live provider keys | ✓ | ✓ | partial | ✓ |
| Voice self-play over a realtime agent | ✓ | partial | — | — |
| Realtime duplex audio + session recording | ✓ | — | — | — |
| Realtime voice providers (Cartesia, ElevenLabs, OpenAI Realtime) | ✓ | partial | — | — |
On text eval these tools are mature and we are not ahead. The row that matters is the last three: driving a persona through a live duplex call.
The most mature open-source eval harness there is, and the one to reach for if your agent is text-only. Bigger community, more scorers, more integrations. On pure eval it beats us.
Already building on LangChain? Its tracing and dataset tooling integrate with effectively zero effort, and the trajectory evaluators cover tool-call assertions well.
Pytest-native and Apache 2.0. If you want agent evals that behave like unit tests inside an existing Python suite, start there rather than here.
An employee pushing an out-of-policy expense, a caller impersonating a vendor to reroute a payment, someone who won't take no for an answer. They hit your agent over real audio, push on it, and get scored on whether it held policy — not on whether the transcript looked reasonable.
Red-team scenarios probe the agent for policy bypass and unsafe actions, scored in CI on every PR like any other run.
Cartesia, ElevenLabs, OpenAI Realtime, Gemini, or a local mock. One config runs the same suite across all of them, so you can hear what changing provider does to your pass rate before you commit to one.
Assert the agent verified the vendor, escalated to a human, and never released a payment it shouldn't. tools_called and tools_not_called, mock or live.
Barge-in, overlapping speech, and dead air are where voice agents actually fall over — and none of it survives into a transcript. PromptArena runs real duplex audio and records the session, so you can listen to the failure.
init writes a Claude Code / Codex skill and AGENTS.md, so your coding agent already knows PromptArena conventions.
From clone to a scored adversarial run.
Every template runs green out of the box on a mock provider, so you see a real arena match before you spend a token. Swap in a live provider when you're ready; the variation in pass-rate is the test.
# install $ npm i -g @altairalabs/promptarena # scaffold a project: personas, scenarios, tools, providers $ promptarena init refund-demo --template voice-refund-demo $ cd refund-demo # run the suite against a mock provider (no keys) $ promptarena run --provider mock-duplex # drive the live web UI: results dashboard + start new runs $ promptarena serve --open # web UI at localhost:8080
A pack that passed the arena is ready for production.
PromptArena compiles to a portable, vendor-neutral pack. Omnia runs it in production: governed, audited, on your own infrastructure. Same artifact, all the way through.
