Shaping agentic AI
Agentic AI Foundation
Vstorm was the first agentic AI engineering company to join the Agentic AI Foundation and received its 2026 Open Agentic AI Innovation Award.
Read the announcementAI agent evaluation and observability
Know when your agent gets worse before your users do
Ship agent changes without your users finding the regressions. Tests from your real cases rerun on every change and show accuracy and cost per task before release.
50+ workflows agentified since 2017
-
42% → 98% raw LLM vs guideline-executing accuracy on the open 50-scenario benchmarkHIPAA-compliant guideline-executing triage — 42% to 98% on the open 50-scenario benchmark.
Read case study -
Up to 1 hr → ~2 min to create a workflowText-to-workflow agents building validated node graphs inside the platform.
Read case study -
~20% → ~40% conversion to a final saleThree-agent product advisor guiding customers through print-order configuration.
Read case study -
Every inquiry answered in English and Arabic, from ARIJ's own knowledge baseBilingual English/Arabic agent inside ARIJ's Moodle environment, answering only from ARIJ's own knowledge base.
-
Supply chain intelligence agents reducing manual coordination overhead.
-
AI agent implementation for a global automotive enterprise.
Why agents get worse unnoticed
Ten demo prompts will not predict the next real case.
Agents fail on what nobody wrote down: an odd invoice, a second language, a timeout.
- 01
Where it stallsA handful of prompts the developer wrote.
What we do Tests from your real cases and past failures.
- 02
Where it stallsChanges ship after a spot check.
What we do The suite reruns on every change.
- 03
Where it stallsA wrong answer, and no idea which step.
What we do Every step traced: tool calls, tokens, cost.
How we measure
Your real cases, tested and traced
From the first test case to a suite your team runs.
- 0103
Your cases, turned into tests
Your experts help us turn real tickets, documents and decisions into scored cases, known failures included.
- Suite built from your cases
- Scoring agreed with your experts
- A recorded baseline score
On an open 50-scenario benchmark: 42% for a raw LLM, 98% for the guideline-executing agent. - 0203
Traced on every run
Every wrong answer can be traced to the step, tool call and cost behind it.
- Regression runs in your CI
- Cost per task and model
- Input and answer drift checks
- 0303
Read and run by your team
Dashboards track pass rate per release, cost and failure types; each new failure becomes a case.
- Suites in your repository
- Alerts in your observability stack
- Training for your team
Engineering in the open
We help build what agentic AI runs on
Open standards, open-source tooling and published research, all public, so you can check our engineering before you hire us.
Open source
We build open-source agent tooling that other engineering teams install
10M+ PyPI downloads
4,000+ GitHub stars
33 public repositories
Framework contributors
When the framework breaks, we fix it at the source
We maintain our own open-source agent framework and contribute fixes to the frameworks our agents run on, through official partnerships with their teams. Breaking changes reach us before they reach your codebase.
Research
Our engineers and PhDs publish peer-reviewed research
Peer-reviewed papers on agent architectures, evaluation and AI security, and an open benchmark anyone can rerun against our results.
Track record
Frontier AI teams now build on what we shipped first
Our client agents were in production before most of today's agent frameworks had a stable release. You get a team that already knows what breaks between a promising demo and a system people rely on every day.
Case studies
Agents our clients run and measure
Engineering workflows built in minutes, validated before they run
The agent drafts each workflow and checks it in several steps before it runs, so engineers skip the manual setup.
Up to 1 hrto prepare a workflow by hand
~2 minto create one with the agent
A print-order advisor that lifted conversion to a final sale
Three agents and 15 tools walk customers through print-order configuration, across more than a billion product combinations.
~20% → ~40%conversion to a final sale
96%+web-to-print orders handled end to end
Clinical triage that executes the guideline, step by step
The model answers the assessment questions. The guideline decides the disposition, the same way every time.
42% → 98%accuracy, raw LLM vs guideline-executing
96%weighted F1 across 566 scenarios
Engagement models
Ways to start
Each path starts with a free call with a founder or senior AI engineer. A build goes ahead only after its Proof of Value meets the agreed threshold.
Start with one workflow
We map one workflow with your experts and agree a success threshold before any build.
Best whenYou want the success test agreed before you commit to a build.
Technology
The evaluation and tracing stack
Open standards first, then the tools your team already reads.
-
-
- OpenTelemetry
- Logfire
-
- Logfire
- Grafana
- Datadog
- Prometheus
FAQ
Questions about agent evaluation and observability
What is AI agent evaluation?
AI agent evaluation checks whether an agent gets your real cases right, how its results change between versions, and what each task costs. The cases come from your team's work, and the scoring rules are agreed with your experts.
What is AI agent observability?
AI agent observability means tracing every agent run step by step (tool calls, tokens, cost per task) with OpenTelemetry, so a wrong answer can be followed back to the step that caused it. Traces go to Logfire or the tool your team already uses.
What does AI agent evaluation and observability include?
It includes an evaluation suite built from your real cases, scored by rules agreed with your experts, with a baseline score. The suite reruns in your CI whenever a prompt, tool, model or retrieval setting changes. Every run is traced with OpenTelemetry, cost is counted per task, workflow and model, drift checks watch inputs and answers, and dashboards with alerts sit in the observability stack you already run. Suites and scoring code live in your repository, and we train the team that runs them.
Where do the test cases come from?
From your own work: tickets, documents, conversations and decisions your team has already made, with the right outcome recorded for each. We add edge cases and known failures on purpose, and new failures from daily use are added as they appear.
How do you score answers that need judgment?
With a written rubric agreed with your domain experts. Where a model grades answers against that rubric, your experts review a sample of its verdicts so the grader is checked against people who know the work. Answers that are simply right or wrong get exact checks instead.
Which tracing and monitoring tools do you use?
We instrument agents with OpenTelemetry, an open standard, and often send traces to Logfire. If your team already runs Datadog, Grafana (Tempo for traces, Prometheus for metrics) or another tool, the traces, metrics and dashboards go there instead.
Where does evaluation stop and LLMOps begin?
LLMOps covers releasing LLM systems: deployment pipelines, staged rollouts and model versions. This service covers what the agent does: whether it gets your cases right, which step failed and what a task costs. The evaluation suites built here are what an LLMOps release gate runs.
Can you add evaluation to an agent another team built?
Yes. We start from the code and the agent as they are, add tracing, build the suite from your cases and record a baseline.
Who owns the evaluation suites and dashboards?
You do. Suites, scoring code and dashboard definitions live in your repositories and your observability stack, with no license or platform fee to Vstorm. We train your team to add cases and read the results.
Evaluation and observability
See how your agent scores on your cases
Bring one agent and real cases. We check how it is tested and traced today.
From the blog
Reading on measuring agents
Agentic AI
One Dashboard, Full Visibility: Why We Use Logfire in Our Stack
When people ask us about Logfire, they usually think AI agents. Monitoring PydanticAI workflows, tracking token usage, debugging LLM calls.
Agentic AI
The hidden standard behind every reliable AI Agent
Reliable agents share one property that rarely appears in vendor decks. What we standardised across production deployments, and why it holds.
Agentic AI
Dynamic Tool Discovery: A Systematic Evaluation of 8 Search Strategies for AI Agents
51 tools cost 18,492 tokens per query. Eight discovery strategies — regex, BM25, RAG and more — evaluated on accuracy, latency and token cost.