An agent can return a syntactically perfect answer with the wrong value inside it. Schema-valid is not correct — and a demo will not show you the difference.
An AI agent can return a syntactically perfect JSON object with the wrong number inside it. That is the exact failure mode identified by the Structured Output Benchmark, or SOB (arXiv:2604.25359, April 2026), which tested 21 models for both schema compliance and factual accuracy. Across all 21 models, the best value accuracy reached was 83.0% for text tasks and 67.2% for image tasks — despite near-perfect schema compliance.
That gap is the whole problem with treating structured output as a proxy for correct output. Schema validation confirms an agent filled in the right fields in the right format. It says nothing about whether the number, the recommendation, or the claim inside that format is actually true. A system that only checks for valid JSON will pass its own tests while being confidently wrong.
For a buyer evaluating an AI vendor, that distinction is exactly what a demo will not show. A live demo runs a handful of friendly inputs through the happy path. It does not run the edge cases that surface in the first month of production traffic, and it does not report an accuracy number at all — only a schema that happened to validate.
In TriStorm, Vstorm's delivery methodology, evaluation is a concrete gate — the stage where an agent earns the right to reach production, or does not.
Before any agent is built, the domain scenarios that will define correctness are identified — the real cases the evaluation set has to cover, not a happy-path demo script.
The stage where an agent earns the right to reach production, or does not. It produces a working prototype, an evaluation suite and guardrails, and a production-ready MVP path — with a go/no-go decision scored on the domain's real cases, not a demo script. If the agent cannot clear that gate, it does not move to production, regardless of how well it performs in a live demo.
Clearing the gate is a handoff, not the end. Monitoring and alerting exist because production traffic surfaces edge cases a pre-launch set does not anticipate. Ownership transfer and runbooks let the client's own team read those alerts and operate the system without Vstorm in the loop.
These are production results, evaluated against a domain-specific test set built during Proof of Value — not slide claims. Schmitt-Thompson Clinical Content needed a HIPAA-compliant agentic RAG system to support nurse triage decisions, validated across 329+ nurse-validated scenarios and 16 triage guidelines.
Every figure here is tied to the named test set it was measured against — the scenario count, the household count, the workflow success rate — so it can be checked, not just quoted.
Across 329+ nurse-validated scenarios, with a 95%+ Correct Recommended Disposition rate — a domain where a wrong recommendation is a patient-safety event, not a UX complaint.
Complex engineering workflows in 3 minutes, down from 2 hours, through multi-step validation rather than a single generation pass. Four months after launch, Synera closed a $40M Series B.
Serving 150,000+ households, with a 10x cut in error-processing time. A multi-agent architecture reduces hallucination risk: a narrower task is a task that can be evaluated well.
Not a promise that every agent is immune to hallucination — the discipline behind the numbers, applied case by case.
A structured go/no-go gate, built against real scenarios in the client's domain — not a demo script run once for a stakeholder audience. If the agent cannot clear the gate, it does not reach production, regardless of how well the demo performed.
Mixam's product advisor is constrained to real, live product options pulled via API from Mixam's own systems. It cannot recommend an unprintable option because it is never given one — a structural guardrail, not a prompt hoping the model behaves — and it refuses off-topic requests outright, reaching a 95.4% workflow success rate.
Monitoring and alerting exist because production traffic surfaces edge cases a pre-launch test set does not anticipate. Ownership transfer and runbooks let the client's team operate the system without Vstorm in the loop, and the evaluation suite becomes the baseline they keep using long after launch.
STCC's 0%, Synera's 0%, and the telecom's 98% automation rate are results earned against specific, named evaluation sets built for those domains — not a blanket guarantee that extends automatically to the next engagement. None of this is a claim that every agentic system is immune to hallucination by default.
"A rate without an evaluation set attached is not evidence. It is a marketing claim." — Vstorm perspective
What is repeatable is the discipline behind them: evaluate before it ships, constrain the agent to real, live data, and keep evaluating after launch. That is why each number above is presented with the evaluation set behind it — its scenario count, its household count, or its workflow success rate — rather than as a bare percentage.
Where to go deeper: the monitoring and evaluation infrastructure that keeps a gate like this in place after launch is covered on LLM Ops, and the mechanism-level case for what evaluation is guarding — reasoning over a fixed rule — is Part 2 of this series, why we bet on agentic AI over RPA.
Talk to Vstorm about a Proof of Value engagement: a working prototype, an evaluation suite and guardrails, before any large-scale commitment.
Book a free discovery call — we'll map one real process worth automating.