Section E · Reference

Domain Context — Vocabulary

The terms practitioners use, with one-line definitions you can deploy in the interview. Skim this; come back to it when you hit a word you can't define quickly.

Rust runtime & type vocab

Send: A type that can be transferred across thread boundaries. Arc<T> is Send if T is.
Sync: A type safe to share by reference across threads. &T is Send iff T is Sync.
'static: Lifetime "lives for the duration of the program." Spawned tasks require captured values to be 'static.
Future: A value that, when polled, eventually produces a result. Rust futures are lazy — nothing happens until polled.
Pinning (Pin<P>): Guarantees that the pointed-to value won't be moved. Required for self-referential futures (the state machine generated by async).
Typestate pattern: Encoding state in types so invalid transitions are compile errors. Builder<Unsealed> vs Builder<Sealed>.
RAII: Resource Acquisition Is Initialization — resources released in Drop. Lock guards, file handles, span enters.
Zero-cost abstraction: An abstraction that compiles to the same code you'd write by hand. Iterator chains, Result<T,E>.
Monomorphization: Generic code specialized per concrete type at compile time. Why generics are fast and binaries are big.

Async & concurrency vocab

Tokio runtime: The dominant async executor for Rust. Multi-threaded work-stealing scheduler by default.
Work stealing: Idle worker threads steal tasks from busy threads' local queues to keep CPUs saturated.
Cooperative scheduling: Tasks yield at .await points. A task that doesn't yield hogs its worker thread.
Backpressure: Mechanism by which a slow consumer signals "slow down" to producers. Bounded channels are the basic tool.
Structured concurrency: Child tasks bound to parent scope; cancelling the parent cancels children. Achievable with CancellationToken.
Cancellation safety: Property of a future that says "you can cancel me mid-way without corrupting state." Documented per-API.
Actor: A task that owns state and processes messages from a channel. Common pattern to avoid shared-mutable state.
spawn_blocking: Move synchronous, CPU-bound work to a dedicated thread pool so it doesn't starve the async executor.

Networking & RPC vocab

gRPC: HTTP/2-based binary RPC framework using Protocol Buffers. Streaming, deadlines, status codes.
tonic: The standard Rust gRPC implementation. Built on hyper + tower.
Protocol Buffers (protobuf): Binary serialization format with schema (.proto). Code-generated bindings.
Streaming RPC: gRPC modes: server-stream (one req, many resp), client-stream (many req, one resp), bidi (many both).
SSE (Server-Sent Events): HTTP streaming pattern: server pushes text/event-stream frames over an open connection. How LLM token streams reach browsers.
HTTP/2 multiplexing: Multiple concurrent requests over one TCP connection. Standard for gRPC and modern HTTP.
tower: Rust middleware framework. Services are Service<Req, Resp>; layers compose.
hyper: Low-level HTTP/1+2 implementation. Underpins reqwest, axum, tonic.
axum: Ergonomic tower-based web framework on hyper.
Deadline propagation: gRPC concept: the upstream's remaining time budget flows through to downstream calls.

ML serving vocab

vLLM: High-throughput open-source LLM serving engine. PagedAttention, continuous batching. OpenAI-compatible HTTP API.
Triton: NVIDIA's inference server. Multi-framework (TensorRT, PyTorch, ONNX). HTTP/gRPC.
TGI (Text Generation Inference): Hugging Face's LLM serving server. SSE streaming, batching.
KV cache: Per-request cache of attention keys/values from prior tokens. Avoids recomputation during decode. Memory-intensive.
Prefix caching: Reusing KV cache across requests that share a prefix (system prompt). Big throughput win.
Continuous batching: Inference server batches in-flight requests on each decode step, adding new ones each step. Maximizes GPU utilization.
Prefill vs decode: Prefill = process input prompt (parallel over tokens). Decode = generate output tokens one at a time. Different perf profiles.
TTFT (Time to First Token): Latency from request to first emitted token. Dominated by queueing + prefill.
ITL (Inter-Token Latency): Pause between consecutive tokens. Dominated by decode step.
Speculative decoding: Use a small draft model to predict multiple tokens, validated by the big model in one step. Latency win, no quality loss.
Quantization: Reducing model weight/activation precision (FP16 → INT8 → INT4). Less memory, faster, small quality cost.

Agent & orchestration vocab

Tool use / function calling: The model emits a structured "call this function with these args" instead of free text. The orchestrator runs the function and feeds the result back.
ReAct (reason-act-observe): Agent loop: model produces thought + action; environment returns observation; loop.
Plan-then-execute: Agent emits a multi-step plan once; executor walks it. More predictable than ReAct.
Agent state machine: Explicit states (pending, running, awaiting_approval, complete, failed, cancelled) with allowed transitions.
Tool router / dispatcher: The service that maps a tool name to a handler, validates args, enforces auth, dispatches.
Idempotency key: Caller-generated unique ID per side-effecting call so retries don't duplicate side effects.
Sub-agent: An agent launched by another agent. Isolated state, bounded scope, trace-linked to the parent.
Approval gate: Pause point in agent execution waiting for human sign-off on a high-tier action.
Hedging: Fire a duplicate request to a second backend after a delay; take whichever returns first.
Breakglass: Audited emergency override of normal approval/policy gates.

Observability vocab

OpenTelemetry (OTel): Vendor-neutral observability standard. Traces, metrics, logs. OTLP is the wire protocol.
Span: A single unit of work in a trace — start time, end time, attributes, parent.
Trace ID: Unique ID shared by all spans in one request's path. Propagated across services via headers.
Trace context (W3C): Standard headers (traceparent, tracestate) for cross-service trace propagation.
Tail-based sampling: Decide which traces to keep after seeing their outcome (e.g., always keep errors).
RED metrics: Rate, Errors, Duration. The three you need per endpoint.
USE metrics: Utilization, Saturation, Errors. Per-resource (CPU, memory, queue).
SLI / SLO / SLA: Service Level Indicator (the metric) / Objective (the target) / Agreement (the contract with consequences).
Error budget: Quantified amount of SLO breach you can afford before freezing risky changes.
Burn rate: Rate at which the error budget is being consumed. Multi-window burn-rate alerts catch fast and slow degradations.

Distributed systems vocab

CAP theorem: In a partition, you choose Consistency or Availability. Most systems are AP with tunable consistency.
Linearizability: The strongest single-key consistency: every read sees a real-time-ordered prior write.
Eventual consistency: Replicas converge given no new writes. Reads may be stale.
Quorum (R+W>N): Read and write quorum sizes sum > replication factor → guarantees overlap → consistent reads.
Sharding: Partition data across nodes by key. Consistent hashing is the standard scheme.
Consistent hashing: Hash key + node IDs onto a ring; key maps to the next clockwise node. Adding/removing nodes only remaps a fraction of keys.
Optimistic concurrency control (OCC): Read-modify-write with a version check; retry on conflict. No locks held.
Two-phase commit (2PC): Coordinator-based atomic commit across multiple resources. Blocks if coordinator fails — rarely used in modern systems.
Saga: Long-running transaction as a sequence of local transactions, each with a compensating action for rollback. Preferred over 2PC.
Bulkhead: Resource isolation pattern — separate pools/queues per workload so one failure doesn't drain everything.
Cascading failure: One service's slowness propagates upstream via queue buildup → upstream timeouts → upstream's upstream backs up.
Thundering herd: Many clients retry simultaneously after a transient failure, re-overwhelming the recovering service. Mitigation: jitter, retry budgets.

✓ Check yourself

Answer from memory before opening each one.

A spawned task captures an Arc around a RefCell and the compiler rejects it. Explain in terms of Send and Sync.

Show answer
tokio::spawn needs the captured value to be Send. An Arc<T> is Send only when T is both Send and Sync, and RefCell is not Sync because its borrow tracking is not thread-safe. Use a Mutex or RwLock instead. See Rust runtime & type vocab.
One workload sends 20,000-token prompts and expects short answers. Another sends short prompts and expects long answers. Which latency measure suffers in each, and why?

Show answer
The long-prompt workload suffers in time to first token, since prefill must process every input token before output starts. The long-answer workload accumulates delay as inter-token latency across many sequential decode steps, so its total time grows with output length. See ML serving vocab.
Decode latency is too high and the product team will not accept any change in output quality. Would you propose quantization or speculative decoding, and what does each cost?

Show answer
Speculative decoding. A small draft model proposes tokens and the large model verifies them, so output is unchanged and latency improves, at the cost of running a second model. Quantization saves memory and time but carries a small quality cost. See ML serving vocab.
A prefix-aware router uses consistent hashing over 10 backends and an eleventh is added. Roughly what fraction of keys move, and how does that compare with hashing by key modulo the node count?

Show answer
About one eleventh of keys move with consistent hashing, because only keys that now map to the new node shift. Modulo hashing remaps most keys when the node count changes, which would cold-start most prefix caches at once. See Distributed systems vocab.
A store keeps five replicas. Which of R=3, W=3 and R=2, W=2 gives reads that always see the latest write, and how would you tune for read-heavy traffic?

Show answer
R=3, W=3, since R plus W exceeds N=5, so every read quorum overlaps the latest write quorum. R=2, W=2 can miss it. For read-heavy traffic, shrink R and enlarge W, for example R=1 and W=5, accepting slower, less available writes. See Distributed systems vocab.
An agent workflow reserves funds, places an order and sends a notification, each in a different service. Choose between two-phase commit and a saga, and say what you must build for the one you choose.

Show answer
A saga. Two-phase commit blocks if the coordinator fails and ties the services together, while a saga commits each step locally and runs a compensating action on failure. You must write and test a compensation for each step, such as releasing the reservation. See Distributed systems vocab.

Takeaway

Send and Sync decide what can cross threads, so spawned tasks must capture Send values and shared state needs Arc plus a lock.
Async Rust is cooperative: tasks yield only at awaits, so anything that does not yield hogs a worker, and blocking work goes to spawn_blocking.
gRPC gives streaming, deadlines and status codes; SSE carries token streams to browsers; hyper, tower, tonic and axum fit together in layers.
Prefill sets time to first token, decode sets inter-token latency, and prefix caching and continuous batching are the main throughput levers.
Speculative decoding keeps output unchanged while quantization trades a little quality for memory and speed.
Use RED metrics for endpoints and USE for resources, with SLOs, error budgets and burn-rate alerts for decisions.
Quorum overlap, consistent hashing, optimistic concurrency and sagas are the distributed-systems tools that come up most in this role.

Vocabulary is not enough on the day; drill answers out loud, starting with the practice question bank. → Practice interview questions