What happened when an LLM followed the triage guidelines nurses use
Alone, an LLM picked the reference triage disposition 47% of the time. Given the guideline, 90%. Executing its rules in ALLMTS, 95%.
Our paper, ALLMTS: Automated LLM-based Medical Triage System, has been accepted at the Generative AI for Health workshop at NeurIPS 2026. It describes a triage system we built for LaGrange Medical Software, the company behind Schmitt-Thompson Clinical Content.
The headline result: given a patient scenario and a list of possible recommendations, an LLM on its own selected the reference recommendation 47% of the time. Giving the same model the selected clinical guideline raised accuracy to 90%. A pipeline that explicitly evaluates the guideline's questions and applies its decision rules reached 95%. All three numbers come from 566 written patient scenarios validated by triage nurses.
Triage already runs on expert-written guidelines #
When a patient calls a telephone triage line with symptoms, a nurse uses those symptoms to recommend an appropriate next step: emergency care, an appointment within a particular time window, or care at home. That recommendation is called a disposition.
Much of the expertise behind this process already exists in maintained clinical guidelines. They specify which questions matter, how findings affect urgency, and when another guideline applies. We built ALLMTS around a corpus of adult telehealth triage guidelines from Schmitt-Thompson Clinical Content.
How ALLMTS reaches a recommendation #
The engineering challenge was connecting free-form intake notes to that structured decision process.
ALLMTS first extracts patient demographics and reasons for seeking care. Retrieval narrows the available guidelines, and the model selects an applicable one from a constrained set. It then evaluates the selected guideline's Triage Assessment Questions (TAQs) against the note.
The final disposition follows a fixed rule: take the first positive TAQ in the guideline's order, from highest to lowest acuity. If that TAQ directs the system to another guideline, the pipeline follows that route and evaluates again.
This creates an inspectable record: the selected guideline, the answers to its TAQs, and the TAQ that determined the recommendation.
The LLM still performs consequential work: misreading a symptom or selecting the wrong guideline can produce the wrong outcome. Deterministic rules do not remove those errors, but they make the path from interpretation to recommendation explicit. That gives us something concrete to debug, rather than relying on a plausible-sounding paragraph explaining a final answer.
Why this matters beyond healthcare #
This approach belongs to a broader line of research on guideline adherence. CPGPrompt translates clinical guidelines into decision trees that an LLM can navigate. MedGUIDE evaluates whether models make decisions consistent with oncology guidelines. These projects investigate how well models can apply an external, explicit standard to a particular case.
That question is relevant to organizations with carefully maintained procedures, troubleshooting guides, or decision criteria. Those materials encode choices experts have already worked through. They provide a concrete starting point for an AI project: identify the procedure, define what correct execution means, and test whether a model can connect real inputs to it.
The guideline did most of the work #
Our results show why both the content and the execution deserve attention. We compared four approaches using the same underlying model, GPT-5.2, at the same reasoning setting:
| Approach | Disposition accuracy on 566 scenarios |
|---|---|
| LLM with disposition definitions, without the guideline corpus | 47% |
| LLM with web search | 50% |
| LLM given the selected guideline in its prompt | 90% |
| ALLMTS: explicit TAQ evaluation and guideline routing | 95% |
Access to the selected guideline accounts for most of the improvement. Explicit TAQ evaluation and routing provide an additional five percentage points in this comparison.
The test set outlasts the model #
Building the evaluation set was a substantial part of the project. The scenarios began as LLM-generated drafts. Nurse editors reviewed and refined them before they went through independent second review, expert nurse validation, and senior medical approval. Each scenario was designed to test a particular TAQ and included a reference disposition.
A curated 50-scenario subset forms the paper's open benchmark. ALLMTS matched the reference disposition on 49 of those 50 cases. During validation, a panel of five expert telehealth triage nurses achieved a 94.4% correct disposition rate on the same subset.
That provides a useful human reference point within this task. It does not establish superiority to nurses in clinical practice: the sample is small, the cases are synthetic, and interpreting a prepared note covers only part of a nurse's work.
The evaluation also gives us a durable engineering asset. If we change retrieval, prompting, or the underlying model, we can test against the same cases. We can examine whether the change improves guideline selection, changes particular answers, or shifts errors toward excessive or insufficient urgency.
From written notes to conversation #
The same test set matters for the next phase. We began with a bounded task:
Written scenario → recommended disposition.
The intended interactive system has an additional responsibility: gather the information needed to make that recommendation. It must recognize what remains unknown, ask an appropriate follow-up question, interpret the answer, and decide when it has enough information or needs to defer to a nurse.
Evaluating those capabilities needs an explicit representation of missing information and tests of the conversation itself.
Starting with written scenarios lets us investigate guideline application before combining it with information gathering. When a recommendation is wrong, we can inspect whether the problem lies in guideline selection, interpretation of the note, or application of the decision rules. As we introduce dialogue, we gain another question to test: did the system obtain the information it needed?
Start with a procedure and a set of cases #
ALLMTS rests on two things. The first is a procedure experts already trust: the Schmitt-Thompson guidelines that telehealth triage nurses use. The second is a set of cases those experts can grade: 566 scenarios, each with a reference disposition reviewed by triage nurses and approved by a senior medical reviewer.
The guidelines gave the model something to execute. The cases showed us whether it executed them correctly, and where it did not.
Many teams have the first ingredient in some form: a troubleshooting guide, an underwriting checklist, a returns policy. The second usually has to be built, as it was here. If your team is considering a similar project, that is where we at Vstorm would start.


