AI agent evaluation and observability

Know when your agent gets worse before your users do

Ship agent changes without your users finding the regressions. Tests from your real cases rerun on every change and show accuracy and cost per task before release.

Book a free consultation See client results
30+ production deployments since 2017
Why agents get worse unnoticed

Ten demo prompts will not predict the next real case.

Agents fail on what nobody wrote down: an odd invoice, a second language, a timeout.

  1. 01

    Where it stallsA handful of prompts the developer wrote.

    What we do Tests from your real cases and past failures.

  2. 02

    Where it stallsChanges ship after a spot check.

    What we do The suite reruns on every change.

  3. 03

    Where it stallsA wrong answer, and no idea which step.

    What we do Every step traced: tool calls, tokens, cost.

How we measure

Your real cases, tested and traced

From the first test case to a suite your team runs.

  1. 0103

    Your cases, turned into tests

    Your experts help us turn real tickets, documents and decisions into scored cases, known failures included.

    • Suite built from your cases
    • Scoring agreed with your experts
    • A recorded baseline score
    On an open 50-scenario benchmark: 44% for a raw LLM, 98% for the guideline-executing agent.
  2. 0203

    Traced on every run

    Every wrong answer can be traced to the step, tool call and cost behind it.

    • Regression runs in your CI
    • Cost per task and model
    • Input and answer drift checks
  3. 0303

    Read and run by your team

    Dashboards track pass rate per release, cost and failure types; each new failure becomes a case.

    • Suites in your repository
    • Alerts in your observability stack
    • Training for your team
Engineering in the open

We help build what agentic AI runs on

Open standards, open-source tooling and published research, all public, so you can check our engineering before you hire us.

Shaping agentic AI

The first agentic AI engineering company to join the Agentic AI Foundation

The Linux Foundation formed it around Anthropic's MCP, OpenAI's AGENTS.md and Block's goose. AWS, Google and Microsoft are platinum members. In its own words: “Together, we're shaping the future of agentic AI.”

Open source

We build open-source agent tooling that other engineering teams install

9M+ PyPI downloads
3,900+ GitHub stars
31 public repositories

Framework contributors

When the framework breaks, we fix it at the source

We maintain our own open-source agent framework and contribute fixes to the frameworks our agents run on, through official partnerships with their teams. Breaking changes reach us before they reach your codebase.

Research

Our engineers and PhDs publish peer-reviewed research

Peer-reviewed papers on agent architectures, evaluation and AI security, and an open benchmark anyone can rerun against our results.

Track record

Frontier AI teams now build on what we shipped first

Our client agents were in production before most of today's agent frameworks had a stable release. You get a team that already knows what breaks between a promising demo and a system people rely on every day.

Case studies
All case studies
Engagement models

Ways to start

Each path starts with a free call with a founder or senior AI engineer. A build goes ahead only after its Proof of Value meets the agreed threshold.

Start with one workflow

We map one workflow with your experts and agree a success threshold before any build.

Best whenYou want the success test agreed before you commit to a build.

Book a free consultation
  1. AI agent development Best whenYou are about to build an agent.
  2. Deployment and infrastructure Best whenNow it has to run in your environment.
  3. Stalled pilot recovery Best whenThe agent exists but falls short on real work.
Technology

The evaluation and tracing stack

Open standards first, then the tools your team already reads.

  1. Model Same suite before any model switch
  2. Agent framework Instrumented where the runs happen
  3. Tracing Open standard, sent where you choose
    • OpenTelemetry
    • Logfire
  4. Dashboards and alerts In the stack your team reads
    • Logfire
    • Grafana
    • Datadog
    • Prometheus

How we choose the stack

FAQ

Questions about agent evaluation and observability

What is AI agent evaluation? +
AI agent evaluation checks whether an agent gets your real cases right, how its results change between versions, and what each task costs. The cases come from your team's work, and the scoring rules are agreed with your experts.
What is AI agent observability? +
AI agent observability means tracing every agent run step by step (tool calls, tokens, cost per task) with OpenTelemetry, so a wrong answer can be followed back to the step that caused it. Traces go to Logfire or the tool your team already uses.
What does AI agent evaluation and observability include? +
It includes an evaluation suite built from your real cases, scored by rules agreed with your experts, with a baseline score. The suite reruns in your CI whenever a prompt, tool, model or retrieval setting changes. Every run is traced with OpenTelemetry, cost is counted per task, workflow and model, drift checks watch inputs and answers, and dashboards with alerts sit in the observability stack you already run. Suites and scoring code live in your repository, and we train the team that runs them.
Where do the test cases come from? +
From your own work: tickets, documents, conversations and decisions your team has already made, with the right outcome recorded for each. We add edge cases and known failures on purpose, and new failures from daily use are added as they appear.
How do you score answers that need judgment? +
With a written rubric agreed with your domain experts. Where a model grades answers against that rubric, your experts review a sample of its verdicts so the grader is checked against people who know the work. Answers that are simply right or wrong get exact checks instead.
Which tracing and monitoring tools do you use? +
We instrument agents with OpenTelemetry, an open standard, and often send traces to Logfire. If your team already runs Datadog, Grafana (Tempo for traces, Prometheus for metrics) or another tool, the traces, metrics and dashboards go there instead.
Where does evaluation stop and LLMOps begin? +
LLMOps covers releasing LLM systems: deployment pipelines, staged rollouts and model versions. This service covers what the agent does: whether it gets your cases right, which step failed and what a task costs. The evaluation suites built here are what an LLMOps release gate runs.
Can you add evaluation to an agent another team built? +
Yes. We start from the code and the agent as they are, add tracing, build the suite from your cases and record a baseline.
Who owns the evaluation suites and dashboards? +
You do. Suites, scoring code and dashboard definitions live in your repositories and your observability stack, with no license or platform fee to Vstorm. We train your team to add cases and read the results.
Evaluation and observability

See how your agent scores on your cases

Bring one agent and real cases. We check how it is tested and traced today.

Book a free consultationRead case studies
From the blog

Reading on measuring agents

All articles
Agentic AI

One Dashboard, Full Visibility: Why We Use Logfire in Our Stack

When people ask us about Logfire, they usually think AI agents. Monitoring PydanticAI workflows, tracking token usage, debugging LLM calls.
Agentic AI

The hidden standard behind every reliable AI Agent

Reliable agents share one property that rarely appears in vendor decks. What we standardised across production deployments, and why it holds.
Agentic AI

Dynamic Tool Discovery: A Systematic Evaluation of 8 Search Strategies for AI Agents

51 tools cost 18,492 tokens per query. Eight discovery strategies — regex, BM25, RAG and more — evaluated on accuracy, latency and token cost.