Confident and Wrong in a High-Risk Workflow

The question and what it scores

"Harbourline's coverage assistant tells an adjuster that water damage from a burst pipe is covered. The policy has an exclusion for pipes in unheated spaces, and the model never mentioned it. The tone was confident and the citation looked real. What do you change, and in what order?"

Three things are being scored. Whether you recognise that fluency is not evidence, so the fix is structural and not a better instruction. Whether you can order changes by cost and effect, because "add retrieval, verification, a human and an eval" as an unordered pile sounds like a list from a blog. And whether you measure: a claim that hallucinations went down needs a labelled set and a number. The cost of error sets the design. A wrong deductible quoted to a customer is a complaint. A wrong coverage decision is a regulatory issue, so the same model in the same product needs different controls on the two surfaces.

Why order matters

Each step below removes a different kind of error, and later steps assume earlier ones. Verification against a source needs a source, so grounding comes first. A confidence threshold is meaningless on an answer that is free text, so constraint comes before calibration. A human reviewer is expensive, so the earlier steps exist to shrink what reaches them. Run the steps in this order, and measure after each, so you know which one paid.

Ground the answer, then check the citation is real

A model answering from memory about a specific policy is guessing in a fluent voice. Retrieval puts the policy's actual clauses in the prompt, and the instruction is to answer only from them. How retrieval fails is its own topic, covered in chapter 01 and, for the pattern list, the compliance guide. Here the point is narrower: retrieval reduces unsupported claims, and it cannot reduce omitted ones. In the burst-pipe case, the exclusion clause might not have been retrieved at all. The answer is grounded in the clause that was there, and wrong because of the one that was not.

Ask for the supporting quote. Anthropic's guidance for long documents is to have the model extract word-for-word quotes first and then answer from them, and to have it retract any claim it cannot back with a quote. The API also has a citations feature, where the model's references are parsed into document locations and the cited text is returned by the API, not written by the model. The docs say this guarantees citations contain valid pointers to the documents you provided. Read that carefully: a valid pointer shows the passage exists. It does not show the passage supports the claim. You still check support, in the verify step.

Make abstention a first-class output

A model that must answer will answer. Give it a legitimate way out: a not_in_documents value, or the Anthropic docs' wording of permission to say it does not have enough information. The docs report that allowing "I don't know" can drastically reduce false information, and they also state that none of these techniques eliminates it.

Then measure the abstention rate, because it moves in both directions. Too low and the model is still guessing. Too high and adjusters stop using the tool. Track it on two sets: questions whose answer is in the documents, where abstention is a miss, and questions whose answer is not, where answering is a miss. Without the second set you cannot tell good abstention from refusal to work.

Constrain the output to options

Free text lets the model hedge, embellish and drift. A coverage decision is a choice among a few options plus evidence, so ask for exactly that: a decision from a closed set, the clause id, the supporting quote, and a confidence. This is the schema technique from chapter 13. It helps in three ways. It makes the answer checkable by code. It stops prose from slipping in claims that have no source. And it makes the evaluation a simple comparison with a label. The cost is that nuance moves into a narrower channel, so keep a short free-text explanation field for the human, and never let downstream code act on it.

Verify the claim against the source

Cheap checks first, in code. Was the cited clause id in the retrieved set? Does the quoted text appear in that clause as a substring? A model that invents a clause or paraphrases a quote fails here, at no model cost. This is the same runtime check the compliance guide recommends for citations.

Then a semantic check for what code cannot see: does the quote support the decision? Use a second model call with a narrow task, "given this clause and this claim, does the clause support the decision, yes or no, and which exclusions apply?", ideally a different model or at least a different prompt, so errors are less correlated. The Anthropic docs also describe best-of-N: run the same prompt several times and treat disagreement as a sign of a hallucination. That multiplies cost by N, so reserve it for the high-risk slice, such as decisions that would auto-approve.

Checkers have their own error rate. A verifier that says "supported" 95% of the time on wrong answers is a decoration. Measure it on the labelled set like any other component.

Confidence, thresholds and the human

Use confidence to decide who sees the answer, not whether it is true. Researchers at Anthropic reported in 2022 that larger models are fairly well calibrated when asked whether their own answer is correct, in the right format, and also that the calibration of a trained "do I know this" signal transfers poorly to new tasks. The practical reading: do not trust a number the model emits as a probability until you have checked it on your own data. Plot confidence against accuracy on the labelled set, pick the threshold where auto-handled answers meet your error target, and re-plot when the model or prompt changes.

The routing function below is the whole policy in a few lines, and it runs as written. The order of checks matters: the cheap, certain ones first.

def route(answer, retrieved, threshold=0.85):
    """answer: {"decision", "clause_id", "quote", "confidence"}; retrieved: {clause_id: text}"""
    if answer["decision"] == "not_in_documents":
        return "human:no_source"
    text = retrieved.get(answer["clause_id"])
    if text is None:
        return "human:citation_not_retrieved"      # cited a clause the model never saw
    if answer["quote"] not in text:
        return "human:quote_not_in_clause"          # pointer is real, words are not
    if answer["confidence"] < threshold:
        return "human:low_confidence"
    return "auto"


clauses = {"4.2": "Water damage from a burst pipe is covered up to the policy limit."}
ok = {"decision": "covered", "clause_id": "4.2", "quote": "burst pipe is covered", "confidence": 0.93}
print(route(ok, clauses))
print(route({**ok, "clause_id": "9.9"}, clauses))
print(route({**ok, "quote": "flood is covered"}, clauses))
print(route({**ok, "confidence": 0.6}, clauses))

Where the human sits matters more than whether there is one. A reviewer who sees only the model's verdict will rubber-stamp it. Put the clause text, the highlighted quote and the retrieved-but-unused clauses on screen, so the reviewer is checking a claim against a source and not judging the model. Show the reviewer why the case was routed. And for the highest-risk class, such as claim denials, require a human regardless of confidence, because the cost of a wrong denial is not recovered by a good average.

Measure, and slice

Build a labelled set of real questions, with the correct decision, the correct clause and, for the abstention cases, a marker that no answer exists. Start with a few hundred and add every production miss. Report, per release: unsupported-claim rate, citation validity, decision accuracy, abstention rate on answerable and unanswerable questions, and the share routed to humans.

Slice by question type, because the average hides the dangerous cell. Deductible lookups may be 99% right while exclusion questions are at 80%. Slice by surface as well: the threshold for the customer chat can be looser than the one for coverage decisions. The setup for the set and the ship decision is in chapter 07.

What not to do

Two moves sound like fixes and are not. Raising or lowering temperature: temperature changes how the next token is sampled, not whether the model has the fact, so a confident error at temperature 0 is still a confident error, and repeating it identically makes it look consistent. And adding "be accurate" or "do not hallucinate" to the prompt: the model has no accuracy dial for that sentence to turn. Instructions help when they change the task, such as "answer only from the clauses below, and quote them". They do not help when they ask for a quality the model cannot self-assess. A third move, fine-tuning to reduce hallucination, is a late option: it needs labelled data you do not yet have, and it does not fix missing context.

The worked answer

State the trade-offs. Each check adds latency and cost, and verification doubles model calls on the checked slice. A tight threshold moves work to humans, so the real cost of a safer system is adjuster time, and you should quote it. The recommendation flips by surface: for the help-centre assistant answering deductible questions, grounding, abstention and a visible source may be enough, with spot checks. For coverage decisions the full chain applies.

Interview probes

✓ Check yourself

Answer from memory before opening each one.

1. An answer cites clause 4.2, which was in the retrieved set, and quotes a sentence that appears in clause 4.2. An adjuster still finds the decision wrong. Which checks passed, and which did not run?

   Show answer

   The retrieved-set and substring checks passed; they only prove the pointer and the words are real. Nothing checked that the clause supports the decision, or that an exclusion elsewhere applies. See verify.

2. You add "if unsure, say you do not know" and the abstention rate goes from 1% to 30%. Accuracy on answered questions barely moves. What do you conclude, and what do you measure next?

   Show answer

   The model is abstaining on questions it used to answer correctly, so usefulness fell without a gain in precision. Measure abstention separately on answerable and unanswerable questions, and tune toward abstaining on the second set only. See abstention.

3. Customer chat quotes deductibles and the claims tool recommends coverage decisions. Should they share one confidence threshold?

   Show answer

   No. The cost of error differs: a complaint against a regulatory issue. Each surface gets its own threshold from its own labelled set, and denials always go to a human. See routing.

4. A teammate says the model's stated confidence of 0.9 means it is right 90% of the time. What do you ask them?

   Show answer

   Whether they measured it. Calibration depends on model, prompt and task, and a self-reported number can transfer poorly to a new task. Plot accuracy by confidence bucket on your own labelled set. See routing.

5. Overall accuracy is 96%, up from 94%. Why might you still block the release?

   Show answer

   The average can hide a regression in the high-risk slice, such as exclusion questions. Block if that slice fell, or if it sits below its target. See measure.

Takeaway

• Fluency is not evidence, so the fixes are structural: ground, abstain, constrain, verify, route and measure, in that order of cost.
• Retrieval removes invented facts but not omitted ones, so a missing exclusion clause still produces a confident wrong answer.
• A valid citation proves the passage exists, not that it supports the claim; check support with code first and a second call after.
• Abstention must be a legal output, and its rate must be measured on answerable and unanswerable questions separately.
• Confidence only decides who sees an answer; set the threshold from your own labelled data, per surface.
• A human loop works only if the reviewer sees the source, and you test it with seeded errors.
• Raising temperature or asking the model to be accurate does not change what it knows.

The last chapter turns these into the story of a system you shipped, with your own failure and your own numbers. → The System You Shipped

Further reading

• Reduce hallucinations — Anthropic docs. Permission to abstain, quote-first grounding, citation verification, best-of-N.
• Citations — Anthropic docs. What the API guarantees about citation pointers, and what it leaves to you.
• Language Models (Mostly) Know What They Know — Kadavath et al., 2022. The evidence on self-assessed correctness and where it breaks.
• Structured outputs — Anthropic docs. For constraining the answer to a closed set of options.