Engineering a zero-hallucination agentic RAG system for clinical triage guidelines

The first challenge was not architecture, it was data. STCC guidelines are not written in natural language. They are structured as decision trees: sequential yes/no questions arranged in descending order of urgency, each linked to a recommended disposition level such as “Go to ED Now” or “Home Care.”

The engineering problem: logic-based content in a language model world

This format is efficient for human practitioners but presents a specific failure risk for large language models. Standard LLMs are trained to interpret natural prose. When presented with logic-indicator syntax, such as branching conditionals, tabular urgency tiers, or Boolean qualifiers, they can misread the logic and produce confident but incorrect outputs. In a clinical setting, a misread is not an edge case to tolerate. It is a patient safety risk.

The engineering brief was therefore precise: build a system that treats the guidelines as the exclusive source of truth, introduces no external inference, and reports clearly when it cannot find an answer rather than generating one.

Architecture: a four-stage pipeline

Our team at Vstorm designed a four-stage agentic pipeline to handle the full scenario-to-disposition workflow.

• Stage 1 — Patient context extraction. A dedicated LLM agent extracts patient age and gender from the incoming scenario. This information filters the applicable guideline set before any clinical reasoning begins, narrowing the retrieval space immediately.
• Stage 2 — Reason for visit identification. A second dedicated agent identifies the primary and secondary reasons for the visit. This structured extraction step provides clean, unambiguous inputs for the retrieval stage rather than passing raw prose directly to the vector store.
• Stage 3 — Guideline selection via RAG. The extracted reason for visit is matched against a pre-indexed guideline vector store using retrieval-augmented generation. An LLM then classifies the most appropriate guideline from the retrieved candidates. This stage is where guideline selection errors are most likely to occur, particularly in clinically ambiguous presentations where multiple guidelines are plausible.
• Stage 4 — Parallel TAQ evaluation. Triage Assessment Questions (TAQs) are evaluated concurrently across all disposition levels using parallel LLM calls. Multi-part TAQs, which contain compound conditions, are parsed using Boolean logic to prevent partial matches from producing incorrect dispositions. The system outputs a recommended disposition, a SOAP-formatted clinical note, and an optional LLM-generated rationale.

The entire pipeline is built on the PydanticAI framework as the agent environment, with Logfire providing observability across every stage. Every decision is traceable. Every step is auditable.

Retrieval method: why we moved from PDF to database

In earlier proof-of-concept builds, guideline content was delivered to the retrieval layer via pre-parsed PDF. This approach was fast to prototype but introduced structural noise: PDF parsing produced fragmented chunks that broke the logical continuity of individual guidelines, particularly around multi-step TAQ sequences.

Our engineering team moved to a relational database retrieval model. Guidelines were indexed as structured records with explicit relationships between questions, urgency tiers, and disposition outcomes. The results confirmed the decision: database retrieval outperformed pre-parsed PDF delivery in both disposition accuracy and operational scalability across the full test suite. The relational structure preserved the internal logic of each guideline, giving the LLM a clean, consistently formatted context at every retrieval call.

Model selection: iterative testing across three GPT versions

The team evaluated three GPT model versions across proof-of-concept builds v0.3.x through v9.1: GPT-4.1, GPT-5, and GPT-5.1. All models were operated with HIPAA compliance configurations, a non-negotiable requirement given the sensitivity of patient scenario data.

GPT-5 and GPT-5.1 consistently outperformed GPT-4.1 across the guideline test suite. The performance gap was most pronounced in clinically complex scenarios requiring multi-step reasoning across branching TAQ sequences. GPT-5.1 was selected as the production model, offering the strongest balance of disposition accuracy and reasoning reliability within the compliance constraints.

Testing and validation: 329 scenarios, 16 guidelines

Clinical scenarios used to evaluate the system were developed through a rigorous 14-step validation process managed jointly by STCC and Vstorm. STCC Nurse Editors drafted scenarios for each guideline; scenarios were revised or discarded where editorial agreement could not be reached. Remaining scenarios were submitted to a panel of five expert telehealth triage nurses for validation. Only scenarios where the panel’s aggregate Correct Recommended Disposition (CRD) rate reached 95% or higher were retained and reviewed by the Senior Medical Editor.

The final test set comprised 329 validated clinical scenarios across 16 adult after-hours telehealth guidelines. Chatbot disposition accuracy was measured against this expert-validated benchmark; not against theoretical correctness, but against the standard established by practising clinicians.

“Research, testing and validation of the content is at the core of what we do. Nurses need to trust the guidelines and the trust is built on reliability.” — Laurie O’Bryan, RN, Nurse Editor, STCC / TAG