# Omnia vs Promptfoo

> PromptArena runs every simulated user and attacker on providers you choose, tests duplex voice, and Omnia turns the result into a release decision. Promptfoo, now part of OpenAI, runs its simulated users and several attacks on hosted inference by default.

Category: Testing. Checked against Promptfoo, now part of OpenAI’s published material on 2026-10-22. Canonical page: https://www.altairalabs.ai/vs/promptfoo/

## Already red-teaming with Promptfoo? PromptArena does the same job on whichever providers you choose.

Then check where the traffic goes. Promptfoo, now part of OpenAI and still MIT licensed, runs its simulated users and several multi-turn attacks on hosted inference by default. PromptArena is the open test harness under Omnia.

## When it matters: A bank red-teams its payments agent before launch.

**On Omnia:** The simulated customers and the attacker trying to reroute a payment run on the model provider the bank already approved, against the agent’s real tools, inside the bank’s own boundary. The transcripts stay there, and the agent ships only if the suite passes. (Omnia docs: https://promptarena.altairalabs.ai/arena/explanation/scenario-design/)

**On Promptfoo:** By default, the simulated user runs on Promptfoo’s hosted conversation models. Several multi-turn attack strategies use remote inference in the community edition. (source: https://www.promptfoo.dev/docs/providers/simulated-user/)

## Every simulated user runs where you say

Simulated users, attackers and judges run on whichever provider you configure: Claude, GPT, Gemini, Bedrock, Vertex, or a model you host. (Omnia docs: https://promptarena.altairalabs.ai/arena/explanation/scenario-design/)

Promptfoo: Hosted conversation models by default for simulated users; remote inference for GOAT, Goblin and Hydra in the community edition. (source: https://www.promptfoo.dev/docs/red-team/strategies/)

## Voice, as a phone call

PromptArena’s voice tests hold a two-way audio conversation with the agent, so the simulated caller can cut in while the agent is speaking. Each session is recorded. (Omnia docs: https://promptarena.altairalabs.ai/arena/how-to/voice/setup-voice-testing/)

Promptfoo: The audio strategy converts prompt text into speech and encodes it, and requires remote generation. (source: https://www.promptfoo.dev/docs/red-team/strategies/audio/)

## Red-team results gate the release

Omnia runs the agent as well as the tests, so a passing suite starts the rollout. If live metrics drop, the version is rolled back and the production failure is added to the suite. *(enterprise)* (Omnia docs: https://promptarena.altairalabs.ai/arena/how-to/interfaces/run-in-ci/)

Promptfoo: A CI flag fails the build on test failures. Promptfoo doesn’t run agents, so what happens after the test is up to you. (source: https://www.promptfoo.dev/docs/integrations/ci-cd/)

## At a glance

| Topic | Omnia | Omnia docs | Promptfoo | Promptfoo docs | Source |
|---|---|---|---|---|---|
| Open source | PromptArena, Apache 2.0. | https://promptarena.altairalabs.ai/arena/explanation/scenario-design/ | MIT. | documented | https://github.com/promptfoo/promptfoo |
| Red-team strategies | Adversarial callers probing for policy bypass, against real tools. | https://promptarena.altairalabs.ai/arena/how-to/voice/voice-red-team/ | A broad library, including Crescendo and GOAT. | documented | https://www.promptfoo.dev/docs/red-team/strategies/ |
| Simulated users on your own provider | Always. | https://promptarena.altairalabs.ai/arena/explanation/scenario-design/ | Hosted by default. | partly documented | https://www.promptfoo.dev/docs/providers/simulated-user/ |
| Duplex voice testing | Barge-in, overlapping speech and dead air. | https://promptarena.altairalabs.ai/arena/how-to/voice/setup-voice-testing/ | Text converted to audio. | partly documented | https://www.promptfoo.dev/docs/red-team/strategies/audio/ |
| CI gate | Built into the release. | https://promptarena.altairalabs.ai/arena/how-to/interfaces/run-in-ci/ | Fail the build on errors. | documented | https://www.promptfoo.dev/docs/integrations/ci-cd/ |
| Self-hosted server for production | Omnia, installed with Helm. | https://omnia.altairalabs.ai/how-to/workspaces/manage-workspaces/ | Not recommended for production; no built-in SSO. | not documented | https://www.promptfoo.dev/docs/usage/self-hosting/ |
| Runs and rolls out the agent | Yes, with automatic rollback. | https://omnia.altairalabs.ai/tutorials/progressive-rollouts/ | Testing only. | not documented | https://github.com/promptfoo/promptfoo |

## Questions

**Is Promptfoo still open source?**

Its README says it is now part of OpenAI and remains open source under the MIT licence.

**Is PromptArena free?**

Yes. PromptArena is Apache 2.0 and installs with npm. The paid parts are Omnia’s enterprise features in production.