Engineering a zero-hallucination agentic RAG system for clinical triage guidelines.

The challenge

Logic-based guidelines in a language-model world

STCC recognised that content built over decades and validated against expert clinical benchmarks had to be made machine-accessible. Competitors building AI-native tools would fill the gap, and public LLMs were already at risk of absorbing proprietary guideline content through uncontrolled data exposure. STCC came to Vstorm for a controlled, validated, HIPAA-compliant agentic system that could operate at clinical accuracy without any chance of hallucination.

The first challenge was not architecture. It was data. STCC guidelines are not written in natural language. They are structured as decision trees: sequential yes/no questions arranged in descending order of urgency, each linked to a recommended disposition such as “Go to ED Now” or “Home Care.”

That format is efficient for human practitioners and a specific failure risk for large language models. Standard LLMs are trained to interpret natural prose. When presented with branching conditionals, tabular urgency tiers, or Boolean qualifiers, they can misread the logic and produce confident but incorrect outputs. In a clinical setting, a misread is not an edge case to tolerate. It is a patient-safety risk.

The engineering brief was therefore precise: treat the guidelines as the exclusive source of truth, introduce no external inference, and report clearly when the system cannot find an answer rather than generating one.

How it works

A four-stage pipeline, then a database instead of a PDF

Vstorm designed a four-stage agentic pipeline to handle the full scenario-to-disposition workflow. The pipeline runs on the PydanticAI framework, with Logfire providing observability across every stage. Every decision is traceable. Every step is auditable.

Stage 1 extracts patient age and gender so the applicable guideline set is filtered before any clinical reasoning begins. Stage 2 identifies the primary and secondary reasons for the visit, so retrieval receives clean inputs rather than raw prose. Stage 3 matches that reason against a pre-indexed guideline vector store using retrieval-augmented generation, then classifies the most appropriate guideline — the stage where selection errors concentrate on clinically ambiguous presentations. Stage 4 evaluates Triage Assessment Questions concurrently across disposition levels. Multi-part TAQs are parsed with Boolean logic so a partial match cannot produce a disposition.

In earlier Proof of Value builds, guideline content reached retrieval as a pre-parsed PDF. That was fast to prototype and introduced structural noise: fragmented chunks broke the logical continuity of individual guidelines, particularly around multi-step TAQ sequences. The team moved to a relational database. Guidelines were indexed as structured records with explicit relationships between questions, urgency tiers, and disposition outcomes. Database retrieval outperformed PDF delivery in both disposition accuracy and operational scalability across the full test suite.

Model selection across three GPT versions

The team evaluated three GPT model versions across Proof of Value builds v0.3.x through v9.1: GPT-4.1, GPT-5, and GPT-5.1. All models ran with HIPAA compliance configurations, a non-negotiable requirement given the sensitivity of patient scenario data.

GPT-5 and GPT-5.1 consistently outperformed GPT-4.1 across the guideline test suite. The gap was most pronounced in clinically complex scenarios that require multi-step reasoning across branching TAQ sequences. GPT-5.1 was selected as the production model, offering the strongest balance of disposition accuracy and reasoning reliability within the compliance constraints.

Testing and validation: 329 scenarios, 16 guidelines

Clinical scenarios used to evaluate the system were developed through a 14-step validation process managed jointly by STCC and Vstorm. STCC Nurse Editors drafted scenarios for each guideline. Scenarios were revised or discarded where editorial agreement could not be reached. Remaining scenarios went to a panel of five expert telehealth triage nurses. Only scenarios where the panel's aggregate Correct Recommended Disposition (CRD) rate reached 95% or higher were retained and reviewed by the Senior Medical Editor.

The final test set comprised 329 validated clinical scenarios across 16 adult after-hours telehealth guidelines. Chatbot disposition accuracy was measured against this expert-validated benchmark — not against theoretical correctness, but against the standard established by practising clinicians.

Results

What the 329-scenario suite showed

Full-scenario disposition accuracy was strong across the majority of guidelines tested. Thirteen of 16 guidelines demonstrated chatbot accuracy within five percentage points of their respective benchmarks. Two guidelines exceeded the benchmark: Abdominal Pain — Male (100% vs. 96.2%) and Headache (100% vs. 97.4%).

Three guidelines underperformed the benchmark: Neurologic Deficit (85.0% vs. 96.0%), Cough — Acute Productive (84.0% vs. 98.3%), and Urinary Symptoms (84.0% vs. 98.4%).

Post-analysis identified two primary failure modes. The most frequent was no positive TAQ identified: the system correctly selected the guideline but failed to match any TAQ to the clinical scenario, producing an incomplete disposition. The second was incorrect guideline selection on overlapping or ambiguous chief complaints, where two or more guidelines were plausible matches at retrieval.

Both failure modes are deterministic. They do not arise from hallucination or fabricated reasoning. They come from retrieval and matching gaps that targeted engineering can close. The testing also confirmed zero hallucination events across the full scenario set. When the system could not identify an answer, it reported the absence of a match and provided traceable source references, or the explicit absence of them.

What is still in flight

The system remains in active development. TAQ matching logic is being extended to handle scenarios where no positive TAQ is present: a condition the current pipeline flags but does not yet resolve. Guideline selection is being improved for ambiguous chief-complaint presentations through enhanced retrieval ranking and disambiguation logic.

In parallel, Vstorm is developing a locally deployable model variant designed to run on client-owned infrastructure. That addresses healthcare environments with stricter data residency constraints, where patient history and treatment records cannot leave the organisation's perimeter. The local model is being engineered with reinforced anti-hallucination constraints and explicit data boundary controls, so a higher degree of patient context can be passed into the system safely.