Reaching expert-level medical triage with guideline-executing LLMs.

The challenge

Logic-based guidelines in a language-model world

STCC recognised that content built over decades and validated against expert clinical benchmarks had to be made machine-accessible. Competitors building AI-native tools would fill the gap, and public LLMs were already at risk of absorbing proprietary guideline content through uncontrolled data exposure. STCC came to Vstorm for a controlled, validated, HIPAA-compliant system that could execute those guidelines at clinical accuracy — without inventing findings the intake note does not support.

The first challenge was not architecture. It was data. STCC guidelines are not written in natural language. They are structured as decision trees: sequential yes/no questions arranged in descending order of urgency, each linked to a recommended disposition such as “Go to ED Now” or “Home Care.”

That format is efficient for human practitioners and a specific failure risk for large language models. Queried as free-form clinical reasoners, LLMs are poorly calibrated for telehealth triage: they over-escalate routine cases and cannot justify recommendations against an auditable clinical standard. The engineering brief was therefore precise: treat the guidelines as the exclusive source of truth, make the model answer narrow triage assessment questions, and derive the disposition from the guideline traversal — not from the model’s parametric knowledge.

How it works

A four-stage pipeline, then a database instead of a PDF

Vstorm designed a four-stage agentic pipeline to handle the full scenario-to-disposition workflow. The pipeline runs on the PydanticAI framework, with Logfire providing observability across every stage. Every decision is traceable. Every step is auditable.

Stage 1 extracts patient age and gender so the applicable guideline set is filtered before any clinical reasoning begins. Stage 2 identifies the primary and secondary reasons for the visit, so retrieval receives clean inputs rather than raw prose. Stage 3 matches that reason against a pre-indexed guideline vector store using retrieval-augmented generation, then classifies the most appropriate guideline — the stage where selection errors concentrate on clinically ambiguous presentations. Stage 4 evaluates Triage Assessment Questions concurrently across disposition levels under a closed-world assumption: anything the scenario does not mention is treated as false. The disposition itself is not generated by the model; it follows from the first positive TAQ in acuity order.

In earlier Proof of Value builds, guideline content reached retrieval as a pre-parsed PDF. That was fast to prototype and introduced structural noise: fragmented chunks broke the logical continuity of individual guidelines, particularly around multi-step TAQ sequences. The team moved to a relational database. Guidelines were indexed as structured records with explicit relationships between questions, urgency tiers, and disposition outcomes. Database retrieval outperformed PDF delivery in both disposition accuracy and operational scalability across the full test suite.

Model selection across three GPT versions

The team evaluated three GPT model versions across Proof of Value builds v0.3.x through v9.1: GPT-4.1, GPT-5, and GPT-5.1. All models ran with HIPAA compliance configurations, a non-negotiable requirement given the sensitivity of patient scenario data.

GPT-5 and GPT-5.1 consistently outperformed GPT-4.1 across the guideline test suite. The gap was most pronounced in clinically complex scenarios that require multi-step reasoning across branching TAQ sequences. GPT-5.1 was selected as the production model, offering the strongest balance of disposition accuracy and reasoning reliability within the compliance constraints.

Testing and validation: 591 scenarios, plus an open 50-scenario benchmark

Clinical scenarios used to evaluate the system were authored by the guideline publisher’s clinical content team and validated by an expert triage-nurse panel, covering 14 disposition labels across the full acuity spectrum. A curated 50-scenario subset is released as an open benchmark with reference dispositions for direct comparison against that expert-validated telehealth standard.

Chatbot disposition accuracy is measured against those expert-validated references — not against theoretical correctness, and not as a bare “zero hallucination” claim. Guideline-grounded execution is what closes the gap from a 44% raw-LLM baseline to expert-level performance on the open subset.

Results

From a 44% raw LLM to expert-level triage

On the curated 50-scenario open benchmark, ALLMTS reaches 98% accuracy — on par with the 94.4% correct-disposition rate recorded by the five-nurse expert panel on that same subset. On the full 591 expert-validated scenarios spanning 14 dispositions, it reaches 93% disposition accuracy and 94% weighted F1, against 44% for the strongest raw-LLM baseline.

Safety profile matters as much as headline accuracy. Baselines over-triage up to about 46% of cases; ALLMTS keeps error low in both directions (at most about 6% under-triage and about 2% over-triage on the reported runs). Run-to-run stability also improves: the system reproduces the reference acuity band on all ten runs for 47 of 50 open-benchmark scenarios, versus 28 for the baseline.

Remaining misses concentrate on deterministic failure modes: no positive TAQ identified after correct guideline selection, and incorrect guideline selection on overlapping or ambiguous chief complaints. Those are retrieval and matching gaps — engineering backlog — not a license to market a bare zero-hallucination rate. When the system cannot identify an answer, it reports the absence of a match and provides traceable source references, or the explicit absence of them.

What is still in flight

The system remains in active development. TAQ matching logic is being extended to handle scenarios where no positive TAQ is present: a condition the current pipeline flags but does not yet resolve. Guideline selection is being improved for ambiguous chief-complaint presentations through enhanced retrieval ranking and disambiguation logic.

In parallel, Vstorm is developing a locally deployable model variant designed to run on client-owned infrastructure. That addresses healthcare environments with stricter data residency constraints, where patient history and treatment records cannot leave the organisation's perimeter. The local model keeps the same guideline-executing contract and explicit data boundary controls, so a higher degree of patient context can be passed into the system safely.

Related research

Academic context for guideline-executing clinical AI

Adjacent work on clinical guideline adherence, evidence integration, and LLM-executable decision support — linked for readers who want the research framing around this build.

• MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models — evaluates whether LLMs can follow structured clinical guidelines.
• MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline — deep evidence integration against expert-level guidelines.
• CPGPrompt: Translating Clinical Guidelines into LLM-Executable Decision Support — clinical practice guidelines as LLM-executable decision trees (also in JAMIA).