{"slug":"promptfoo","url":"https://www.altairalabs.ai/vs/promptfoo/","markdown":"https://www.altairalabs.ai/vs/promptfoo.md","name":"Promptfoo","vendor":"Promptfoo, now part of OpenAI","category":"Testing","checked":"2026-10-22","summary":"PromptArena runs every simulated user and attacker on providers you choose, tests duplex voice, and Omnia turns the result into a release decision. Promptfoo, now part of OpenAI, runs its simulated users and several attacks on hosted inference by default.","headline":"Already red-teaming with Promptfoo? PromptArena does the same job on whichever providers you choose.","scene":{"setup":"A bank red-teams its payments agent before launch.","omnia":"The simulated customers and the attacker trying to reroute a payment run on the model provider the bank already approved, against the agent’s real tools, inside the bank’s own boundary. The transcripts stay there, and the agent ships only if the suite passes.","omniaCaps":["selfplay","adversarial","tool-assertions","deploy-gate"],"them":"By default, the simulated user runs on Promptfoo’s hosted conversation models. Several multi-turn attack strategies use remote inference in the community edition.","source":"https://www.promptfoo.dev/docs/providers/simulated-user/","omniaEvidence":"https://promptarena.altairalabs.ai/arena/explanation/scenario-design/"},"differences":[{"title":"Every simulated user runs where you say","body":"Simulated users, attackers and judges run on whichever provider you configure: Claude, GPT, Gemini, Bedrock, Vertex, or a model you host.","caps":["selfplay","adversarial","native-interfaces","openai-compatible"],"them":"Hosted conversation models by default for simulated users; remote inference for GOAT, Goblin and Hydra in the community edition.","source":"https://www.promptfoo.dev/docs/red-team/strategies/","omniaEnterprise":false,"omniaEvidence":"https://promptarena.altairalabs.ai/arena/explanation/scenario-design/"},{"title":"Voice, as a phone call","body":"PromptArena’s voice tests hold a two-way audio conversation with the agent, so the simulated caller can cut in while the agent is speaking. Each session is recorded.","caps":["voice-testing","duplex-voice"],"them":"The audio strategy converts prompt text into speech and encodes it, and requires remote generation.","source":"https://www.promptfoo.dev/docs/red-team/strategies/audio/","omniaEnterprise":false,"omniaEvidence":"https://promptarena.altairalabs.ai/arena/how-to/voice/setup-voice-testing/"},{"title":"Red-team results gate the release","body":"Omnia runs the agent as well as the tests, so a passing suite starts the rollout. If live metrics drop, the version is rolled back and the production failure is added to the suite.","caps":["deploy-gate","progressive-rollout","prod-to-regression"],"them":"A CI flag fails the build on test failures. Promptfoo doesn’t run agents, so what happens after the test is up to you.","source":"https://www.promptfoo.dev/docs/integrations/ci-cd/","omniaEnterprise":true,"omniaEvidence":"https://promptarena.altairalabs.ai/arena/how-to/interfaces/run-in-ci/"}],"rows":[{"topic":"Open source","omnia":"PromptArena, Apache 2.0.","caps":["selfplay"],"them":"MIT.","coverage":"yes","source":"https://github.com/promptfoo/promptfoo","omniaEnterprise":false,"omniaEvidence":"https://promptarena.altairalabs.ai/arena/explanation/scenario-design/"},{"topic":"Red-team strategies","omnia":"Adversarial callers probing for policy bypass, against real tools.","caps":["adversarial"],"them":"A broad library, including Crescendo and GOAT.","coverage":"yes","source":"https://www.promptfoo.dev/docs/red-team/strategies/","omniaEnterprise":false,"omniaEvidence":"https://promptarena.altairalabs.ai/arena/how-to/voice/voice-red-team/"},{"topic":"Simulated users on your own provider","omnia":"Always.","caps":["selfplay"],"them":"Hosted by default.","coverage":"partly","source":"https://www.promptfoo.dev/docs/providers/simulated-user/","omniaEnterprise":false,"omniaEvidence":"https://promptarena.altairalabs.ai/arena/explanation/scenario-design/"},{"topic":"Duplex voice testing","omnia":"Barge-in, overlapping speech and dead air.","caps":["voice-testing"],"them":"Text converted to audio.","coverage":"partly","source":"https://www.promptfoo.dev/docs/red-team/strategies/audio/","omniaEnterprise":false,"omniaEvidence":"https://promptarena.altairalabs.ai/arena/how-to/voice/setup-voice-testing/"},{"topic":"CI gate","omnia":"Built into the release.","caps":["deploy-gate"],"them":"Fail the build on errors.","coverage":"yes","source":"https://www.promptfoo.dev/docs/integrations/ci-cd/","omniaEnterprise":false,"omniaEvidence":"https://promptarena.altairalabs.ai/arena/how-to/interfaces/run-in-ci/"},{"topic":"Self-hosted server for production","omnia":"Omnia, installed with Helm.","caps":["workspaces","oidc"],"them":"Not recommended for production; no built-in SSO.","coverage":"undocumented","source":"https://www.promptfoo.dev/docs/usage/self-hosting/","omniaEnterprise":false,"omniaEvidence":"https://omnia.altairalabs.ai/how-to/workspaces/manage-workspaces/"},{"topic":"Runs and rolls out the agent","omnia":"Yes, with automatic rollback.","caps":["progressive-rollout"],"them":"Testing only.","coverage":"undocumented","source":"https://github.com/promptfoo/promptfoo","omniaEnterprise":false,"omniaEvidence":"https://omnia.altairalabs.ai/tutorials/progressive-rollouts/"}],"faq":[{"q":"Is Promptfoo still open source?","a":"Its README says it is now part of OpenAI and remains open source under the MIT licence."},{"q":"Is PromptArena free?","a":"Yes. PromptArena is Apache 2.0 and installs with npm. The paid parts are Omnia’s enterprise features in production."}]}