Section E · Reference

Domain Context — Vocabulary

The terms practitioners use, with one-line definitions you can deploy in the interview. Skim this; come back to it when you hit a word you can't define quickly.

Rust runtime & type vocab

Send: A type that can be transferred across thread boundaries. Arc<T> is Send if T is.
Sync: A type safe to share by reference across threads. &T is Send iff T is Sync.
'static: Lifetime "lives for the duration of the program." Spawned tasks require captured values to be 'static.
Future: A value that, when polled, eventually produces a result. Rust futures are lazy — nothing happens until polled.
Pinning (Pin<P>): Guarantees that the pointed-to value won't be moved. Required for self-referential futures (the state machine generated by async).
Typestate pattern: Encoding state in types so invalid transitions are compile errors. Builder<Unsealed> vs Builder<Sealed>.
RAII: Resource Acquisition Is Initialization — resources released in Drop. Lock guards, file handles, span enters.
Zero-cost abstraction: An abstraction that compiles to the same code you'd write by hand. Iterator chains, Result<T,E>.
Monomorphization: Generic code specialized per concrete type at compile time. Why generics are fast and binaries are big.

Async & concurrency vocab

Tokio runtime: The dominant async executor for Rust. Multi-threaded work-stealing scheduler by default.
Work stealing: Idle worker threads steal tasks from busy threads' local queues to keep CPUs saturated.
Cooperative scheduling: Tasks yield at .await points. A task that doesn't yield hogs its worker thread.
Backpressure: Mechanism by which a slow consumer signals "slow down" to producers. Bounded channels are the basic tool.
Structured concurrency: Child tasks bound to parent scope; cancelling the parent cancels children. Achievable with CancellationToken.
Cancellation safety: Property of a future that says "you can cancel me mid-way without corrupting state." Documented per-API.
Actor: A task that owns state and processes messages from a channel. Common pattern to avoid shared-mutable state.
spawn_blocking: Move synchronous, CPU-bound work to a dedicated thread pool so it doesn't starve the async executor.

Networking & RPC vocab

gRPC: HTTP/2-based binary RPC framework using Protocol Buffers. Streaming, deadlines, status codes.
tonic: The standard Rust gRPC implementation. Built on hyper + tower.
Protocol Buffers (protobuf): Binary serialization format with schema (.proto). Code-generated bindings.
Streaming RPC: gRPC modes: server-stream (one req, many resp), client-stream (many req, one resp), bidi (many both).
SSE (Server-Sent Events): HTTP streaming pattern: server pushes text/event-stream frames over an open connection. How LLM token streams reach browsers.
HTTP/2 multiplexing: Multiple concurrent requests over one TCP connection. Standard for gRPC and modern HTTP.
tower: Rust middleware framework. Services are Service<Req, Resp>; layers compose.
hyper: Low-level HTTP/1+2 implementation. Underpins reqwest, axum, tonic.
axum: Ergonomic tower-based web framework on hyper.
Deadline propagation: gRPC concept: the upstream's remaining time budget flows through to downstream calls.

ML serving vocab

vLLM: High-throughput open-source LLM serving engine. PagedAttention, continuous batching. OpenAI-compatible HTTP API.
Triton: NVIDIA's inference server. Multi-framework (TensorRT, PyTorch, ONNX). HTTP/gRPC.
TGI (Text Generation Inference): Hugging Face's LLM serving server. SSE streaming, batching.
KV cache: Per-request cache of attention keys/values from prior tokens. Avoids recomputation during decode. Memory-intensive.
Prefix caching: Reusing KV cache across requests that share a prefix (system prompt). Big throughput win.
Continuous batching: Inference server batches in-flight requests on each decode step, adding new ones each step. Maximizes GPU utilization.
Prefill vs decode: Prefill = process input prompt (parallel over tokens). Decode = generate output tokens one at a time. Different perf profiles.
TTFT (Time to First Token): Latency from request to first emitted token. Dominated by queueing + prefill.
ITL (Inter-Token Latency): Pause between consecutive tokens. Dominated by decode step.
Speculative decoding: Use a small draft model to predict multiple tokens, validated by the big model in one step. Latency win, no quality loss.
Quantization: Reducing model weight/activation precision (FP16 → INT8 → INT4). Less memory, faster, small quality cost.

Agent & orchestration vocab

Tool use / function calling: The model emits a structured "call this function with these args" instead of free text. The orchestrator runs the function and feeds the result back.
ReAct (reason-act-observe): Agent loop: model produces thought + action; environment returns observation; loop.
Plan-then-execute: Agent emits a multi-step plan once; executor walks it. More predictable than ReAct.
Agent state machine: Explicit states (pending, running, awaiting_approval, complete, failed, cancelled) with allowed transitions.
Tool router / dispatcher: The service that maps a tool name to a handler, validates args, enforces auth, dispatches.
Idempotency key: Caller-generated unique ID per side-effecting call so retries don't duplicate side effects.
Sub-agent: An agent launched by another agent. Isolated state, bounded scope, trace-linked to the parent.
Approval gate: Pause point in agent execution waiting for human sign-off on a high-tier action.
Hedging: Fire a duplicate request to a second backend after a delay; take whichever returns first.
Breakglass: Audited emergency override of normal approval/policy gates.

Observability vocab

OpenTelemetry (OTel): Vendor-neutral observability standard. Traces, metrics, logs. OTLP is the wire protocol.
Span: A single unit of work in a trace — start time, end time, attributes, parent.
Trace ID: Unique ID shared by all spans in one request's path. Propagated across services via headers.
Trace context (W3C): Standard headers (traceparent, tracestate) for cross-service trace propagation.
Tail-based sampling: Decide which traces to keep after seeing their outcome (e.g., always keep errors).
RED metrics: Rate, Errors, Duration. The three you need per endpoint.
USE metrics: Utilization, Saturation, Errors. Per-resource (CPU, memory, queue).
SLI / SLO / SLA: Service Level Indicator (the metric) / Objective (the target) / Agreement (the contract with consequences).
Error budget: Quantified amount of SLO breach you can afford before freezing risky changes.
Burn rate: Rate at which the error budget is being consumed. Multi-window burn-rate alerts catch fast and slow degradations.

Distributed systems vocab

CAP theorem: In a partition, you choose Consistency or Availability. Most systems are AP with tunable consistency.
Linearizability: The strongest single-key consistency: every read sees a real-time-ordered prior write.
Eventual consistency: Replicas converge given no new writes. Reads may be stale.
Quorum (R+W>N): Read and write quorum sizes sum > replication factor → guarantees overlap → consistent reads.
Sharding: Partition data across nodes by key. Consistent hashing is the standard scheme.
Consistent hashing: Hash key + node IDs onto a ring; key maps to the next clockwise node. Adding/removing nodes only remaps a fraction of keys.
Optimistic concurrency control (OCC): Read-modify-write with a version check; retry on conflict. No locks held.
Two-phase commit (2PC): Coordinator-based atomic commit across multiple resources. Blocks if coordinator fails — rarely used in modern systems.
Saga: Long-running transaction as a sequence of local transactions, each with a compensating action for rollback. Preferred over 2PC.
Bulkhead: Resource isolation pattern — separate pools/queues per workload so one failure doesn't drain everything.
Cascading failure: One service's slowness propagates upstream via queue buildup → upstream timeouts → upstream's upstream backs up.
Thundering herd: Many clients retry simultaneously after a transient failure, re-overwhelming the recovering service. Mitigation: jitter, retry budgets.