LangSmith evaluates the new version. In Omnia, the evaluation is the release decision.
Omnia picks up where the experiment ends: the gate, the rollout and the rules on each tool call, all in your cluster. LangSmith traces and evaluates, self-hosted on its Enterprise plan, and gating a release on the result is a CI script you maintain.
That suite is also the release gate, so the version stays where it is. One that passes goes to part of the traffic first and is rolled back if its live metrics drop. docs
on langsmith The docs’ example gate is a script that exits non-zero below a threshold. For deployments, the rollback advice is to tag your images with version information. their docs
A release gate
Simulated users, hostile and voice callers among them, run the agent through its real tools. The gate is part of the platform rather than a script in your CI. docs
LangSmith: Multi-turn simulation through the open-source openevals package, run from pytest, Vitest or evaluate. The gate is your own script. source
Rollout and rollback
A new version gets traffic in steps, and each step waits for its metrics to hold. A conversation that started on the old version finishes there. If the metrics drop, the version is rolled back. docs enterprise
LangSmith: Deployments ship as revisions, and assistant versions can be promoted to active. Canary and automatic rollback aren’t documented. source
Policy as configuration
Tool policy is a CEL rule in the agent’s pack, checked against each call’s arguments and caller. If the broker can’t answer, the call is refused. Security can review the rule without reading your Python. docs enterprise
LangSmith: Human-in-the-loop middleware interrupts per tool, with an optional Python predicate on the call’s arguments. source
At a glance
| Topic | Omnia | LangSmith, per its docs |
|---|---|---|
| Runs in your Kubernetes cluster | yes Helm, in every edition. docs | documented Self-hosted on the Enterprise plan. source |
| Simulated users | yes Built in, adversarial included. docs | documented Via openevals. source |
| Voice testing | yes Barge-in, overlapping speech and dead air. docs | not documented Not documented. source |
| Eval-gated release | yes Built in. docs | partly Write your own CI script. source |
| Stepped rollout, auto rollback | yes Yes. docs | not documented Not documented. source |
| OpenTelemetry out to your collector | yes Generally available, with a documented span inventory. docs | partly Alpha on Agent Server. source |
| End-user identity to tools | yes Carried through the agent to every call. docs | partly Custom auth exposes the user to the graph. source |
Common questions
- Can I keep LangSmith for tracing and use Omnia to run agents?
- Yes. Omnia emits standard OpenTelemetry, which LangSmith ingests, and runs agents built in other frameworks under its own policy and testing.
- How is each priced?
- LangSmith charges per seat and per trace, with deployments metered per vCPU-hour; self-hosting is on the Enterprise plan. Omnia is free while you build, the core is free in production, and the enterprise licencelicense is one price per environment per year from £30,000, with nothing metered.
Put your own agent through the gate.
Install Omnia in your cluster and run it free while you build.
LangSmith claims are from LangChain’s published documentation, checked 22 October 2026. Spot something out of date? Tell us.