Model Deployment — AWS & GCP

What "deployment" means for this role

In agentic-LLM work, you typically don't train and host large models — you consume them via API (Claude API directly, or via Bedrock/Vertex). What you do deploy:

1. The orchestrator (n8n, custom Python, Airflow) — the workflow layer.
2. MCP servers (custom Python or TypeScript) exposing tools to agents.
3. Inference glue (Lambda / Cloud Run / containers) that wraps API calls with retries, caching, validation.
4. Vector DBs / retrieval indexes.
5. Audit / observability stacks.
6. Evaluation harnesses (often as scheduled jobs).
7. Maybe: fine-tuned smaller models (open-source, 7-70B) for narrow tasks where API isn't right.

That last bullet is where SageMaker / Vertex AI / Bedrock get interesting.

AWS — Foundation models / hosted LLMs

• Amazon Bedrock: managed access to Anthropic, Meta, Mistral, Cohere, Amazon's own models via a single API. The most relevant for this role. Bedrock Claude is Anthropic's Claude API under AWS auth/billing/governance. Compliance teams in AWS shops often prefer Bedrock for SOC/PCI/HIPAA inheritance.
  • Bedrock Agents: AWS's first-party agent framework. Tool use, action groups, knowledge bases. Lighter-weight than rolling your own.
  • Bedrock Knowledge Bases: managed RAG — point at S3, get a vector store, get retrieval.
  • Bedrock Guardrails: pre/post filtering for PII, profanity, off-topic.
• Amazon SageMaker: end-to-end ML platform. Train, fine-tune, host. Less central for API-consumed Claude; central if you're hosting your own models.
  • SageMaker JumpStart: deploy a pre-trained open-source model in a few clicks.
  • SageMaker Endpoints: hosted inference for your models.

AWS — Compute for orchestration / inference glue

• Lambda: serverless functions. Right for: short-lived MCP servers, webhook handlers, post-processing. Cold starts matter for low-latency UX.
• AWS Fargate / ECS: containers without EC2 management. Right for: longer-running services, MCP HTTP servers, n8n self-hosted.
• EKS: Kubernetes when scale or complexity justifies. Often overkill for this role.
• App Runner: simple managed-container service. Easiest path for "just run this image as an HTTP service."
• Step Functions: workflow orchestration with built-in state, retries, fan-out. Compelling for compliance flows where state and audit-friendliness matter.

AWS — Storage / data & vector search

• S3: object storage. Foundational. Includes Object Lock for WORM compliance.
• DynamoDB: serverless NoSQL. Good for session state, idempotency keys, light metadata.
• Aurora / RDS PostgreSQL: relational. Aurora supports pgvector for vector search.
• OpenSearch: full-text + vector search.
• Kinesis Data Streams / MSK (managed Kafka): streaming.

Vector / search

• OpenSearch with vector engine: hybrid (BM25 + vector) at managed scale.
• Aurora pgvector: pragmatic, single store for app + vector.
• Bedrock Knowledge Bases: managed end-to-end RAG.
• Third-party (Pinecone, Weaviate) integrate easily.

AWS — Observability / governance

• CloudWatch: logs, metrics, alarms.
• CloudTrail: API audit log. Critical for compliance — every AWS API call is logged.
• AWS Config: configuration history and drift detection.
• Secrets Manager / Parameter Store: credential management.
• IAM: identity, scoped roles. Get this right or nothing else matters.

Compliance / certification

AWS publishes attestations: SOC 1/2/3, PCI DSS, HIPAA, FedRAMP, ISO 27001, etc. Inheriting these via AWS infra is part of why regulated firms use it. For a multi-jurisdictional firm, region selection (e.g. eu-west for EU customer data) is part of the compliance design.

GCP — the parallel services

Foundation models

• Vertex AI Model Garden: hosts Anthropic Claude, Google Gemini, Llama, Mistral, etc. via single API. Parallel to Bedrock.
• Vertex AI Agent Builder: agent framework with tool use and knowledge integration.
• Vertex AI Search / Vertex AI Vector Search: managed retrieval / RAG.

Compute

• Cloud Functions: serverless functions. Lambda parallel.
• Cloud Run: managed containers, scales to zero. Often the sweet spot for HTTP services in a GCP-heavy org.
• GKE: Kubernetes.
• Workflows: managed orchestration (Step Functions parallel).

Storage / data

• GCS: object storage with retention policies.
• Firestore: managed NoSQL.
• AlloyDB, Cloud SQL: PostgreSQL with pgvector support.
• BigQuery: warehouse + ML extensions; central piece in GCP.
• Pub/Sub: streaming.

Observability / governance

• Cloud Logging, Cloud Monitoring: logs and metrics.
• Cloud Audit Logs: equivalent of CloudTrail.
• Secret Manager: credentials.
• IAM: identity.

Compliance attestations are similar to AWS. Both are credible bases for regulated workloads.

Reference 1: Compliance agent on AWS

The shape for an alert-pre-screening agent or case-narrative-drafting agent:

Key choices to articulate

• Bedrock over direct Anthropic API: easier compliance inheritance, single billing, AWS-native auth.
• ECS over Lambda for MCP servers: longer-lived, state-friendlier, no cold-start hit.
• DynamoDB + S3 for audit: events small/queryable, content immutable/cheap.
• Object Lock on S3: regulatory retention enforcement at infra layer.

Reference 2: Compliance agent on GCP

Self-hosted vs managed — the tradeoff

For this kind of role, default is managed (Bedrock / Vertex / direct Anthropic). Self-hosting has high TCO; only justify if there's a compliance constraint that forces it.

CI / CD for AI systems

Deployment isn't only "running services" — it's also the lifecycle around them.

• Model versioning: pin specific versions (claude-opus-5-5, not aliases). Promote between dev/staging/prod via config.
• Prompt versioning: prompts in a registry, semver-tagged, deployed alongside code.
• Eval gating: PR can't merge unless offline eval suite passes. CI runs evals against canonical dataset.
• Shadow / canary: new prompt or model runs alongside existing in a small % of traffic, compared automatically.
• Rollback: any change must have an immediate rollback path. Config flag, prompt-pinning, model alias.
• Infra-as-code: Terraform / Pulumi / CDK / Cloud Deployment Manager for the deployment itself.
• Secrets rotation: API keys, JWTs, service-account credentials all rotate on schedule. No human reads them.

Containerization

You should be able to write a basic Dockerfile and Compose / Kubernetes manifest.

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]

Have notes ready on:

• Multi-stage builds: smaller production images.
• Distroless / minimal base: smaller attack surface.
• Health checks: liveness and readiness probes.
• Resource limits: CPU and memory caps prevent runaway containers.
• Non-root user: don't run as root inside the container.

Network / security at the deployment layer

• VPC / Private endpoints: keep traffic off the public internet. AWS PrivateLink, GCP Private Service Connect.
• WAF: web application firewall in front of public endpoints.
• mTLS between internal services.
• KMS / Cloud KMS: encryption keys managed centrally.
• Customer-managed keys (CMK): required by some compliance regimes for the most sensitive data.

Observability stack — the practical shape

For an LLM-heavy system, you need traces that include LLM call details, not just HTTP latencies.

The LLM-specific layer is the new piece many candidates miss. Mentioning OpenTelemetry GenAI semantic conventions or a tool like Langfuse signals current awareness.

Common deployment design questions

• "How would you deploy a multi-tenant MCP server on AWS such that tenants are isolated?"
• "Walk me through the deployment of a RAG-backed agent on GCP."
• "How would you scale your inference glue layer when traffic 10×'s?"
• "What's your rollout strategy for a new prompt that's been A/B-tested in eval?"
• "How would you keep a regulatory document index fresh without re-embedding everything daily?"
• "What does the failure path look like when Bedrock returns a 5xx?"

For each, the answer structure: components → data flow → scale assumptions → failure modes → cost/security tradeoffs. Even if you don't know the exact service name, the structure makes you sound architect-shaped.

Talking-point: "How would you deploy this on AWS?"

Memorize the structure; substitute GCP services if asked.

✓ Check yourself

Answer from memory before opening each one.

1. A firm already runs on AWS and wants Claude for case drafting. Why might it choose Bedrock over the direct API, and what does it inherit?

   Show answer

   The same Claude models are reached through the firm's existing cloud identity, billing and governance, and the firm inherits the cloud's attestations such as SOC and PCI. Regions can also be chosen to meet residency rules. It still owns its own application controls. See AWS foundation models.

2. Choose between ECS or Fargate and Lambda for an MCP tool server, and for a webhook handler that post-processes agent output.

   Show answer

   Containers for the MCP server: it is longer-lived, state-friendly and free of cold starts. Lambda for the webhook handler: short, stateless and event-driven. See AWS compute.

3. Design the audit store for the same agent on GCP, mirroring the object-lock design on AWS.

   Show answer

   Trace events go to BigQuery for queries, and large payloads go to a Cloud Storage content store with retention policies. Cloud Audit Logs cover the infrastructure layer. The agent runs on Cloud Run with its tool servers beside it. See reference architecture on GCP.

4. A prompt change passes review but changes behavior in production. Which parts of the delivery process should have stopped it?

   Show answer

   An eval gate in CI against the canonical dataset, a shadow or canary rollout comparing it with the old version, and a rollback path through a config flag or version pin. Prompts are versioned and deployed alongside code. See CI/CD for AI systems.

5. When would you self-host an open model for alert triage rather than call a managed API?

   Show answer

   When a strict data-residency rule cannot be met by providers, when latency needs are extreme, when a narrow high-volume task is better served by a fine-tuned model, and when the team has real ML platform skills. Otherwise default to managed, because self-hosting carries high total cost. See self-hosted versus managed.

6. HTTP latency in your traces looks normal, yet answers got worse. Which layer of the observability stack is missing?

   Show answer

   LLM-specific tracing that records prompts, model versions, tool calls, tokens and cost, with online eval scores on top. Standard traces show requests but not what the model saw or said. See the observability stack.

Takeaway

• For agentic LLM work you mostly consume models over an API and deploy the glue around them: the orchestrator, MCP servers, retrieval, audit and eval jobs.
• Bedrock and Vertex give the same Claude models under your cloud's identity, billing, region and attestations, which is the usual compliance reason to choose them.
• Pick the cloud the team already runs on, and add a second only when a requirement forces it.
• Long-lived tool servers belong on containers, while short stateless handlers fit functions.
• The audit store is events in a queryable store plus content in object storage under lock or retention.
• Ship prompts and models like code: pinned versions, eval gates, shadow or canary rollout and an instant rollback.
• Keep traffic private, encrypt with managed keys, and add LLM-specific tracing to the standard telemetry.

Deployment is the last technical piece; the next chapter gives you the compliance vocabulary and risk framework that explain why it looks this way. → Compliance Context

Further reading

• Anthropic Claude models — AWS documentation, accessed 2026. How Claude is called through Bedrock.
• Anthropic's Claude on Google Cloud — Google Cloud documentation, accessed 2026. Using Claude through Vertex AI.
• Locking objects with Object Lock — AWS documentation, accessed 2026. Write-once retention for audit archives.
• Financial Services Industry Lens — AWS Well-Architected Framework, accessed 2026. Resilience, security and compliance guidance for regulated workloads.
• What is Cloud Run — Google Cloud documentation, accessed 2026. The managed-container option the GCP reference uses.
• Building best practices — Docker documentation, accessed 2026. Multi-stage builds, small bases and non-root users.