An LLM Behind a Reliable Contract

The question and what it scores

"Harbourline's claims workflow calls an LLM to decide whether a claim falls under a policy clause. The workflow engine expects a function, not a chatbot. How do you put the model behind a contract the engine can depend on?"

The interviewer is scoring four things. First, whether you treat the model as an unreliable dependency, like a flaky third-party API, and not as a magic function. Second, whether you know each failure has its own remedy: malformed output, slow output, no output, wrong output. Third, whether you can say what the caller sees when everything fails. Fourth, whether you version the prompt and the model with the discipline you apply to code. Candidates who talk only about prompts score low. This question is about the layer around the prompt.

Four ways the call fails, and why one fix cannot cover them

Start from the failure, because the contract is just the list of failures you chose to handle. The coverage-check service takes a claim and a policy clause set and returns a verdict. It can fail in four distinct ways.

• Transport failure. The request errors or times out. The Claude API returns 429 for rate limits, 500 for internal errors, 504 for timeouts and 529 when it is overloaded (docs, as of November 2026). These are retryable.
• Shape failure. The call succeeds but the body is not what you asked for: prose around the JSON, a missing field, a decision outside your enum, or a response cut off at the token limit.
• Semantic failure. The body is valid and wrong, for example a clause id that does not exist in this policy. No schema catches this; chapter 15 covers it.
• Drift. Nothing errors, but a model or prompt change last Tuesday moved the answers. This is a versioning problem, not a retry problem.

Retrying fixes the first. Constrained decoding and validation fix the second. Cross-checks fix the third. Pinning and canaries fix the fourth. A retry loop wrapped around all four turns drift and semantic failure into more cost and the same wrong answer.

The schema is the contract, and the model gets only part of it

Define the request and response as typed objects, version them, and publish them to the caller. For the response, three fields carry the design: a decision from a closed enum (covered, not_covered, needs_review), a clause_id, and a source field that says whether the verdict came from the model or a fallback. The last one is the part candidates forget. A caller that cannot tell a model verdict from a default cannot measure how often the default fires.

To get the model to emit the shape, use the API's structured outputs. On the Claude API you pass a JSON schema in output_config.format with type: "json_schema", and constrained decoding makes the output match it. The feature is stable as of November 2026, but read the limits. Objects need additionalProperties: false. Recursive schemas, minLength/maxLength and numeric minimum/maximum are rejected with a 400. The first request with a new schema pays a grammar-compile cost, and the compiled grammar is cached for 24 hours after last use, so a schema edit means a cold start in production. The output can still miss the schema when stop_reason is refusal or max_tokens, both of which return HTTP 200.

Two consequences follow. Your own validator still runs on every response, because it enforces what the schema cannot express: length bounds, "clause_id is required unless the decision is needs_review", and the cross-field rules. And the validator reads stop_reason before it reads the body, because a truncated body that parses is the most dangerous kind of success.

Validate, then repair once, then stop

When validation fails, the cheapest recovery is a repair call: send the original input, the bad output and the validator's message, and ask for a corrected object. This works because the failure is usually narrow, a missing field or an enum miss, and the error text names it. Allow one repair, maybe two. A model that failed twice with the error in front of it will mostly fail a third time, and each attempt costs a full round trip against a deadline.

Keep the repair loop separate from the transport-retry loop, with its own counter. Mixing them is how a service ends up making nine calls per request. The sketch below shows the whole contract in one function, with stdlib only so it runs as written.

import json
import random
import time

DECISIONS = {"covered", "not_covered", "needs_review"}


class Deadline:
    """One budget for the whole request, passed down every layer."""

    def __init__(self, seconds):
        self.end = time.monotonic() + seconds

    def left(self):
        return self.end - time.monotonic()


def validate(raw):
    """Return (obj, None) if raw is a valid verdict, else (None, reason)."""
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError as e:
        return None, f"not JSON: {e.msg}"
    if obj.get("decision") not in DECISIONS:
        return None, "decision must be one of " + ", ".join(sorted(DECISIONS))
    if obj["decision"] != "needs_review" and not obj.get("clause_id"):
        return None, "clause_id is required unless decision is needs_review"
    return obj, None


def coverage_check(call_model, claim, deadline, max_attempts=3):
    """call_model(claim, timeout, feedback) -> raw string. Raises TimeoutError, ConnectionError."""
    feedback = None
    for attempt in range(max_attempts):
        timeout = min(8.0, deadline.left())
        if timeout <= 0.5:                       # not enough time for another try
            break
        try:
            raw = call_model(claim, timeout, feedback)
        except (TimeoutError, ConnectionError):
            pause = random.uniform(0, min(2.0, 0.25 * 2 ** attempt))   # full jitter
            time.sleep(min(pause, max(deadline.left() - 0.5, 0)))
            continue
        obj, problem = validate(raw)
        if obj:
            return {**obj, "source": "model"}
        feedback = problem                       # repair: tell the model what was wrong
    # fallback: a deterministic default the caller can act on
    return {"decision": "needs_review", "clause_id": None, "source": "fallback"}


if __name__ == "__main__":
    random.seed(1)
    replies = iter([ConnectionError(), "Sure! Covered.", '{"decision": "covered", "clause_id": "4.2"}'])

    def flaky(claim, timeout, feedback):
        r = next(replies)
        if isinstance(r, Exception):
            raise r
        print("model saw feedback:", feedback)
        return r

    print(coverage_check(flaky, {"claim_id": "C-1"}, Deadline(20)))
    def dead(claim, timeout, feedback):
        raise TimeoutError()
    print(coverage_check(dead, {"claim_id": "C-2"}, Deadline(3)))

Retries, jitter and deadlines

Retry only what is transient: 429 with a retry-after header, 500, 504, 529 and dropped connections. Do not retry a 400, and do not retry a 429 that is a monthly spend cap; the API docs say that one carries no retry-after header and keeps failing until access resumes. The official SDK already retries transient errors twice by default with exponential backoff and honours retry-after. So decide where retries live. If the SDK retries twice, your service retries three times and the workflow engine retries three times, one slow dependency becomes 27 calls. Amazon's builders' library makes the same point with a five-deep stack and three retries per layer: 243 times the load on the bottom layer. Retry at one layer, and cap it with a budget, such as a token bucket that allows retries only while recent calls are mostly succeeding.

Add jitter. Exponential backoff alone makes every client that failed together retry together. Random delay in [0, backoff] spreads them out.

Timeouts need a number. Amazon's guidance is to pick an acceptable false-timeout rate, say 0.1%, and set the timeout at the matching latency percentile, p99.9, measured on the real dependency. For an LLM call that percentile is far above the median, because output length varies, so a fixed 30 seconds is a guess. Use streaming for long generations, which the Claude docs recommend for requests that may run past ten minutes, and enforce a timeout per attempt inside an overall deadline. The deadline travels with the request: if the workflow engine gave this call ten seconds and the first attempt took eight, the service must not start a second ten-second attempt. In the sketch, Deadline does that job.

Harbourline's engine routes about 6,000 new claims a day, and the coverage check sits on the intake path, so the deadline is ten seconds and the engine does not retry. That keeps one retry layer. The case where this flips: the nightly batch classification jobs have no human waiting. They can use the Message Batches API and retry whole batches hours later, because latency does not matter there.

What the caller sees when the model cannot answer

Decide the failure behaviour before the first incident, in this order of preference.

1. A cached verdict, when the same claim and policy version were already decided. Safe only if the cache key includes the prompt version and the policy version.
2. A smaller or different model, behind the same schema. It is cheaper and faster, and a cross-vendor fallback also survives a provider outage. It will be less accurate, so record source and the model id, and expect a higher human-review rate downstream.
3. A deterministic default. For coverage that is needs_review. A wrong auto-decision is a regulatory issue, so the default must be the conservative one, never covered or not_covered.
4. An explicit error with a retryable flag, when even the default makes no sense. A claim missing its policy number gets a 422, not a guess.

The caller's contract then has two outcomes it can program against: a valid verdict with a source, or a typed error. It never receives free text and never has to parse a hope.

Idempotency, because retries meet side effects

The coverage check is a pure function of its inputs, but the service around it is not: it writes an audit record and may create an adjuster task. A retry from the workflow engine after a timeout can double-write both. The fix is an idempotency key sent by the caller, one per logical request, stored with the saved result. A retry with the same key returns the saved result and runs nothing.

Stripe's API is the reference design. The key travels in an Idempotency-Key header, up to 255 characters, and the server saves the status code and body of the first request. Keys can be pruned after 24 hours, and the server compares parameters on a reused key and errors if they differ, which catches a caller that reuses a key for a different claim. Copy that last behaviour. A reused key with different input must fail loudly, not return the old answer.

Idempotency does not make the model deterministic. Two fresh calls with the same input can differ, even at temperature 0. That is the argument for saving the first result and replaying it, instead of regenerating on retry.

Version prompts and models like code

Model and prompt are two dependencies of the service, and both change behaviour without a code diff. Treat them as such.

• Pin. Use a dated model id where one exists, not an alias that moves. Store the prompt in the repo, with a version string that appears in every log line and every saved verdict.
• Changelog. Each prompt or model change is a pull request that links an eval run, using the labelled set from chapter 07 and the practices in the evals chapter of the compliance guide.
• Canary. Send 5% of traffic to the new version, and compare fallback rate, validation-failure rate, latency and, once labels arrive, verdict agreement with the old version. Promote or roll back on those numbers.
• Deprecations. Providers retire models on a published schedule. Put the retirement date in the calendar when you pin, not when the 404 arrives.

Remember that a schema change invalidates the compiled grammar and, if you use prompt caching, a change to output_config.format invalidates the cache for that thread. Schema edits are deploys, with warm-up.

An SLO that matches the contract

Define the service level on what the caller experiences: the share of requests that return a schema-valid verdict or a typed error within the deadline. Say it is 99.9% over four weeks. Google's SRE workbook gives the arithmetic: a 99.9% objective leaves 0.1% of requests as budget, so a million requests allow a thousand failures. At 6,000 claims a day, four weeks is about 168,000 requests and the budget is about 168 failures. Its example policy says that when the budget is spent, changes stop except for urgent fixes. For this service that rule has teeth: a prompt change that raises validation failures freezes the next prompt change.

Add a second, separate measure: fallback rate. A service can meet its availability target while falling back on 15% of claims, which is a quality outage hidden inside an availability number. Alert on both.

The worked answer

The trade-offs to name: more repair attempts raise success and cost, and eat the deadline. A cache saves money and risks serving a verdict from an old prompt. A smaller fallback model keeps the pipeline moving and shifts load to human reviewers. The recommendation flips for the nightly batch jobs, as noted above, and for any caller that wants a streamed answer: then the contract is a stream of events, validation happens at the end, and a mid-stream error arrives after a 200 status.

Interview probes

✓ Check yourself

Answer from memory before opening each one.

1. A response arrives with HTTP 200, parses as JSON, and passes your schema, yet the decision field is wrong in an obvious way. Name two reasons the schema could not have prevented it.

   Show answer

   Constrained decoding enforces shape, not truth, so a valid enum value can still be the wrong one, and the clause it cites may not exist in this policy. A truncated or refused response can also come back with 200. See the schema section.

2. The SDK retries twice, your service wraps it in three attempts, and the workflow engine retries three times. The provider is slow, not down. How many calls does one claim produce, and what should you change?

   Show answer

   Up to 3 × 3 × 3 = 27, each waiting out a timeout, which deepens the slowness. Keep retries in one layer and cap them with a budget. See retries.

3. A caller times out at 10 seconds and retries with the same idempotency key, but the first request was still running and finishes. What should the server do, and what if the retry arrives with a different claim under that key?

   Show answer

   Return the saved result of the first request for the same key, without a second model call or a second audit write. If the parameters differ, reject loudly, because reusing the old answer for a different claim is worse than failing. See idempotency.

4. Over four weeks the service shows 99.95% availability, yet adjusters complain that more claims land in their queue. Which number did you fail to watch, and how can the first number look healthy while this one is bad?

   Show answer

   Fallback rate. A needs-review default is a valid, on-time response, so it counts as success in availability while shifting work to humans. Track the two separately. See the SLO section.

5. Your colleague proposes pointing the production alias at the provider's newest model on release day, since "it is strictly better". What do you ask for before agreeing?

   Show answer

   An eval run on the labelled set against the pinned version, a canary slice comparing fallback rate, validation failures and verdict agreement, and a rollback path. Better on benchmarks does not mean better on Harbourline's clauses. See versioning.

6. When would you choose a long retry horizon over a short deadline for the same model call, and what changes in the design?

   Show answer

   When no human waits, as in the nightly claim classification. Use the batch API, retry whole batches later, and drop the fallback-to-default behaviour in favour of requeueing, since a late correct answer beats a fast conservative one. Idempotency keys still apply to downstream writes.

Takeaway

• A reliable LLM service is a list of failures chosen in advance: transport, shape, meaning and drift each get their own remedy.
• Structured outputs remove most parse errors, but refusals and token-limit stops still return 200, so your own validator stays.
• Repair once with the validator's message, then fall back; a third attempt rarely differs from the second.
• Retry at one layer, with jitter, a budget and a deadline that travels with the request.
• On failure the caller gets a verdict tagged with its source, a conservative default, or a typed error, never an unmarked guess.
• Idempotency keys make retries safe around side effects, and a reused key with different input must fail.
• Prompt and model are versioned dependencies: pin, eval, canary, and measure fallback rate beside availability.

The next failure is one no retry fixes, an agent that keeps calling tools without ever finishing. → The Agent That Loops Forever

Further reading

• Structured outputs — Anthropic docs. Parameter names, unsupported schema features, and the refusal and token-limit exceptions.
• Claude API errors — Anthropic docs. Which status codes are transient, SDK default retries, and long-request guidance.
• Timeouts, retries and backoff with jitter — Amazon Builders' Library. Retry amplification, percentile-based timeouts, retry budgets.
• Idempotent requests — Stripe API reference. A concrete idempotency-key design to copy.
• Example error budget policy — Google SRE Workbook. How an SLO becomes a release rule.