Cloud Fundamentals

Why the cloud: rent instead of own

For decades, running software meant buying physical servers, racking them in a room, and operating them yourself. The cloud's pitch is simple: stop owning hardware and start renting it by the hour from a provider — Amazon Web Services (AWS), Google Cloud (GCP), Microsoft Azure — who runs gigantic data centers and rents you slices of them over the internet. Three things make this transformative for data work:

• Elasticity. You can ask for 100 machines for one hour to crunch a huge job, then give them back. You pay for the hour, not the machines. Your storage can grow from a gigabyte to a petabyte without you provisioning a single disk. The capacity scales up and down on demand — something owned hardware can never do.
• Managed services. The provider doesn't just rent you a bare server; it rents you a running database, a running message queue, a running data warehouse — patched, backed up, and kept alive by their engineers. You operate less and build more. Someone else is on call at 3am when the database disk fills up.
• Speed. Provisioning a server used to take weeks of purchasing. Now it's an API call that returns in seconds. You can spin up an entire platform, try an idea, and tear it down before lunch.

The tradeoff is real, so name it honestly. Renting can be more expensive than owning if you run resources 24/7 at steady load (the meter never stops). And building on a provider's proprietary managed services creates lock-in — moving your platform from BigQuery to Redshift later is real work, not a config flip. Most teams accept these tradeoffs because elasticity and not operating databases are worth it — but you accept them with eyes open.

Service models & the shared-responsibility model

Cloud offerings sit on a spectrum of "how much does the provider operate for you." The classic three-letter labels describe where the line falls:

The further down this list you go, the less you operate and the more you pay the provider to operate for you. A data engineer's life is mostly PaaS — managed warehouses, managed orchestration, managed storage — with occasional IaaS when you need a plain VM.

Underlying all of this is the single most important security idea in the cloud: the shared-responsibility model. Security is a partnership. The provider secures the cloud; you secure what you put in it. The line moves depending on the service model:

The core primitives a data engineer actually uses

Each provider has hundreds of services, which is intimidating until you realize a data platform leans on just a few categories. Learn these four and you can read 90% of architecture diagrams. Think of them as the building blocks your mini-platform's components map onto.

1. Object storage — the backbone of the data lake

Object storage holds files ("objects") in flat containers called buckets, addressed by a key (a path-like name). It is cheap, effectively infinite, and durable (the provider replicates copies for you). It is not a filesystem and not a database — you put and get whole objects over HTTP. Almost every modern data lake is "Parquet files sitting in object storage," and nearly every other service reads from and writes to it. The names: S3 on AWS, Google Cloud Storage (GCS) on GCP. You've already met the local stand-in: MinIO in your mini-platform speaks the S3 API, so the bucket reasoning you learned there transfers directly.

2. Compute — running your code

Three flavors, from most to least that you manage:

• Virtual machines (VMs) — a whole rented computer you SSH into and run anything on. Maximum control, you patch the OS. (EC2 / Compute Engine.)
• Containers — your Docker image from Chapter 02, run on managed infrastructure without you owning a VM. (ECS/EKS / Cloud Run/GKE.)
• Serverless functions — you upload just a function; the provider runs it on demand and bills per invocation. No servers to think about. (Lambda / Cloud Functions.)

3. Managed warehouses — analytics SQL at scale

The columnar analytical databases from Course 3, run for you and scaled to terabytes: Redshift (AWS), BigQuery (GCP), and the cross-cloud Snowflake. You load data and write SQL; the provider handles the distributed engine underneath. This is usually the heart of a production data platform.

4. Managed data services — the glue

The moving parts of a pipeline, operated for you: managed streaming (MSK = managed Kafka on AWS, Pub/Sub on GCP), and managed orchestration (MWAA = managed Airflow on AWS, Cloud Composer on GCP). Instead of running Kafka or Airflow yourself, you rent a running, patched instance.

Identity & access management (IAM)

If object storage is the backbone, IAM is the nervous system — and the part beginners struggle with most. IAM answers one question for every action in the cloud: "Is this identity allowed to do this thing to this resource?" Three words you'll use constantly:

The governing principle is least privilege: grant the minimum permission needed to do the job, and nothing more. Your nightly pipeline that writes one bucket should be able to write that one bucket — not read every bucket, not delete databases, not create users. Least privilege limits the blast radius when (not if) a credential leaks.

Regions & availability zones

The cloud isn't one place. Providers split the planet into regions (e.g. us-east-1, europe-west1) — geographic areas where they operate data centers. Each region contains several availability zones (AZs): physically separate data centers, close enough for fast links but isolated so a fire or power loss in one doesn't take down the others. Spreading a system across AZs is how you survive a single data center failing. Three things every data engineer weighs when choosing a region:

• Latency. Data and compute should live near each other and near your users. A query reaching across an ocean to its storage is needlessly slow.
• Data residency. Laws often require certain data to physically stay in a country or bloc (EU personal data under GDPR, for instance). The region you pick decides where bytes actually live — recall the export-control and residency constraints from the systems-design chapter; region choice is how you honor them.
• Cost & availability. Prices and the menu of available services differ between regions; the newest service may not be everywhere yet.

The cost model — and why platforms get expensive fast

The cloud bills pay-per-use: storage by the gigabyte-month, compute by the second or hour, queries by the data scanned or by uptime. Used well, you pay only for what you consume. Used carelessly, the bill is a horror story — and data platforms are especially prone to surprise costs. The classic gotchas:

The discipline of treating cost as a first-class engineering concern — visible, attributed, and optimized — is called FinOps. The mindset is simple: every resource has a meter running, so design with the meter in mind. Tag resources so you know what each team spends, set budget alerts, and make "turn it off" the default for anything not in use.

Hands-on: reason about bucket access

You don't need a paid account to think like a cloud engineer. Your local MinIO from the mini-platform speaks the S3 API, so the exact same access reasoning applies. The question to internalize: who is allowed to do what to this bucket?

If you had real credentials, listing a bucket with the AWS CLI looks like this — and the shape is identical whether it hits AWS or your local MinIO endpoint:

# List your buckets (uses the identity configured in your credentials)
aws s3 ls

# List objects inside one bucket
aws s3 ls s3://griddp-raw/

# Point the SAME command at local MinIO instead of real AWS:
aws --endpoint-url http://localhost:9000 s3 ls s3://griddp-raw/

# Copy a local file up into the bucket
aws s3 cp ./events.parquet s3://griddp-raw/events/events.parquet

Now reason about it without running anything. Suppose aws s3 ls s3://griddp-raw/ returns Access Denied. Walk the IAM question: which identity is the command using? Does its policy grant the s3:ListBucket action on that specific bucket? Under least privilege, a write-only pipeline identity legitimately can't list — and that's the system working, not breaking. The fix is a policy that grants exactly the action needed, not admin everywhere.

✓ Check yourself

• Can you explain elasticity and managed services to a friend — and name the tradeoff (cost, lock-in)?
• On the shared-responsibility model, which side is "your data and who can access it" on — always?
• Can you name the four core primitives and give the AWS and GCP name for each?
• Why does choosing a region involve law, not just latency?
• Name three ways a data platform's cloud bill blows up.

Next

You can reason about the cloud's building blocks; now let's automate getting your code onto them safely and repeatably. → CI/CD for Data