Is higher accuracy always the goal?
A “minimum 95% accuracy” promise can be worth nothing. An interactive walkthrough of precision, recall and what to ask before any number goes in writing.
A legal team reviews contracts by hand, hunting for liability clauses. It is slow and mistakes are expensive, so they are shopping for an AI search tool. Every proposal they receive opens with the same line: “minimum 95% accuracy.”
That number tells them nothing about whether the tool solves their problem. This piece shows why, on one realistic example, and what belongs on that slide instead.
The panel beside this article is the example, running live. A lawyer read 200 contract passages by hand and marked the 10 that contain a liability clause — that is the answer key. The tool never sees it; all it does is sort the 200 passages from most to least likely. One decision is left: how many passages from the top of that ranking land on the lawyer's desk. Move the slider and everything below follows from where it sits.
- Recall
- 0%
- Precision
- —
- Accuracy
- 95.0%
0 of 10 clauses caught
nothing passed along
baseline (do nothing): 95.0%
A weather forecaster in the desert #
Picture a weather station where it rains 5 days a year. A forecaster who announces “no rain” every morning, without ever looking at the sky, is right on 360 days out of 365. That is 98.6% accuracy for never forecasting anything. A smoke detector with dead batteries works the same way: right every day there is no fire.
Every score has a version of this — the result you would get for free. Call it the baseline. With the slider at zero the tool has passed nothing along and found nothing, and it scores 95% accuracy, because for each of the 190 irrelevant passages, “hold it back” happened to be the right call.
So the proposal is not lying. It is quoting a score whose baseline is 95%, which means it has promised nothing at all. And note what the word “minimum” did: that is not a boast, it is a commitment. The guaranteed floor is precisely the score for doing nothing.
The promise does not travel either. Here clauses turn up in 1 passage in 20, so 95% comes free. In an archive where they appear in 1 passage in 3, doing nothing scores 67% and 95% becomes genuinely hard. The identical sentence is empty in one deal and demanding in the next, and nothing on the slide tells you which one you are reading.
All 10 clauses found. Accuracy: 45% #
The legal team's requirement is simple: not one clause may slip through. Move the slider until recall reads 100% and watch what accuracy does.
It falls to 45% — half the do-nothing baseline. The setting that does exactly what the client asked reads as failure on that scoreboard.
Because accuracy does not grade the job. It grades every single call, walking through all 200 passages and asking whether each was sorted correctly. To reach that tenth clause the tool had to pass along 120 passages, which means calling 110 irrelevant ones “relevant”. Under the hood it is an average of two separate tests, weighted by how big each one is:
| Test | Score | Weight |
|---|---|---|
| Spot the 10 relevant passages | 100% | 5% |
| Spot the 190 irrelevant ones | 42% | 95% |
| Accuracy | 45.0% | — |
The test the tool aced — the client's highest-priority test — carries 5% of the grade. Not by anyone's choice, simply because clauses are rare. Needle in a haystack: accuracy gives the hay 95% of the weight.
Two complaints, one slider, opposite directions #
Run the pilot for a month and two complaints come back. “It buries us in junk — one useful passage out of every three” is a precision problem: of what the tool passes along, how much is actually relevant. “It missed a clause in the contract, it is right there” is a recall problem: of what exists, how much the tool catches.
Try to push recall above 90% and precision above 50% at the same time. With this engine's ranking it cannot be done. Both complaints answer to the same slider, moved in opposite directions: pass along more and you miss less but drown in junk; pass along less and the junk goes, taking clauses with it.
The way out of that trade is not a setting. It is a better ranking — and that is engine work.
Same headline numbers, very different engines #
Two search engines can quote near-identical precision and recall and be nothing alike. Switch the panel to Engine B and give it the same job: find all 10 clauses.
Engine A needs 120 passages to catch everything. Engine B needs 174 — 87% of the entire archive. The reason is visible in the panel: where the marked passages sit. With A they cluster near the front of the list. With B they are scattered all the way to the back.
- Engine A — eight of the ten clauses sit in the first 21 results; only the last one hides deep, at position 120.
- Engine B — half the clauses sit beyond position 70, and the last one at 174.
The one-number summary of this is ranking strength, usually quoted as AUC: how consistently relevant passages rank ahead of irrelevant ones. It is only a summary — the positions are the evidence. They are why, with A, nearly every miss is a small widening away, while with B there is no low-workload setting anywhere near full coverage.
That is the difference between adjusting a setting and starting a separate project — and it is why you look at where the clauses land before you promise a date.
The engineer's report and the lawyer's report #
Put the panel back on Engine A at a middle setting: 21 passages go to the lawyer, and 8 of the 10 clauses are among them. Now listen to two people describe that same state.
The engineer says: “We catch 8 clauses out of 10, and we falsely flag only 13 of the 190 irrelevant passages — a 7% false-alarm rate.” The lawyer says: “There are 21 passages on my desk and 13 of them are junk.” Same tool, same setting, very different mood.
Both reports are true. They divide by different things, and the panel shows it directly: the red block is the same 13 mistakes in both bars. In the engineer's bar it is a sliver of 190. In the lawyer's bar it is most of the 21 that actually reached the desk.
Every slider position gives one row of results:
| Setting | Clauses caught | On the desk | Share useful |
|---|---|---|---|
| Strict | 3 of 10 | 5 | 60% |
| Middle | 8 of 10 | 21 | 38% |
| Thorough | 10 of 10 | 120 | 8% |
The desk view can be compressed into one number, properly called Average Precision: each time another real clause turns up in the list, check how much of the list so far is useful, then average those checks. Engine A scores 0.44, Engine B scores 0.10. Plain reading: across most practical settings, B leaves the lawyer with a dirtier desk.
The engineer's view has its own summary — the ROC curve, and its area. It reads 0.90 for A and 0.62 for B, a much narrower gap. When the thing you are hunting is rare, an engine can hold a high ROC score while most of what reaches the desk is junk. If a supplier shows you only ROC, ask for the other one.
Why bother with one number when the table says more? Because the table is not what goes in a deck. Suppliers quote the row that flatters them, and three documents arrive with three different settings that do not line up. A score that covers every setting is harder to game than one flattering row. The score compares engines; the table picks the setting.
What the client should actually be told #
You do not hand a client a threshold, and you do not hand them a menu of imperfect settings either. You hand them one recommendation, with the reasoning, priced in their own working hours.
The strict setting fits in 10 hours a month but delivers 3 clauses out of 10 — unacceptable against this team's requirement. The thorough setting found all 10 known clauses at 250 hours of reading a month. Today the same questions cost them 417 hours, because a lawyer walks the entire contract every time. That is roughly 40% of the review time back, with the lawyer still making every final call.
Notice what that framing does: it ends on the saving, not on the limitation. And not one metric appears in it, though every number stands on one.
Four questions to ask before anyone quotes you a number #
If you are the one buying, these four settle more than any accuracy figure will:
| What to ask | What it settles |
|---|---|
| What costs more: missing a clause, or reviewing irrelevant passages? | Which mistake hurts — decides whether to aim for recall or precision. |
| In a representative sample of your documents, how often do these clauses actually appear? | The baseline. Without it, no quoted score has a scale. |
| Who reviews the results, and how long does checking one take? | Review capacity — where the setting can realistically sit. |
| Is there anything the system absolutely must not miss? | The acceptance criterion, and the reference point for every later complaint. |
Three claims that should prompt a question #
These turn up in most proposals. None is a lie; each is incomplete in a way that matters.
- “We'll catch more and cut the junk, both.” Fine as a goal, wrong as a quick fix. Within one setup those two move in opposite directions. Getting both means improving the ranking itself — different search technology, different document splitting. That is a project with a price, not a configuration change.
- “Minimum 95% accuracy.” A guarantee with no baseline attached. Ask two things: what it was measured on, and what a do-nothing model would score on that same data.
- “It found everything in the demo.” The demo ran on a handful of clean documents someone picked. Your archive is scans, appendices and templates from 2009. The question to build a habit around is: measured on whose files?
And one sentence worth agreeing before any of it goes in writing: on an agreed sample of your own documents, the system must find at least X% of the validated clauses while showing no more than Y passages per question. Two numbers you sign, not adjectives — and success reported as hours returned to the team, never as an accuracy percentage alone.
