Quick disclaimer: I have found it a lil tough to translate my mental model about this stuff into words for juniors. I can explain it to them fine, but I am making up half the analogies on the fly when I do it. So I am writing this down properly once.
When you start debugging servers in prod, people toss around terms like iowait, %steal, CPU throttling, swap, IOPS vs throughput, and memory overcommit.
Easiest way to picture how CPU, RAM, and Disk interact (and how each one breaks) is a restaurant kitchen:
- CPU = the chef doing the actual chopping and cooking (clock speed in GHz = how fast a chef chops; cores = how many chefs you have in the kitchen).
- RAM = the kitchen counter right in front of the chef. Super fast to grab ingredients from, but space is limited.
- CPU Cache (L1/L2/L3) = the tiny spice rack literally in the chef’s hand. Tiny (megabytes), instant access.
- Disk (EBS / SSD / HDD) = the big storeroom down the hall. Holds terabytes of stuff, survives power loss, but walking down the hall to fetch from it is orders of magnitude slower than grabbing off the counter (RAM).
Here is what all the common Linux/cloud performance problems look like in that kitchen.
1. Swapping (when RAM runs out)
Your app needs to work on some data. If the data is already sitting on the counter (RAM), the chef grabs it in nanoseconds.
What happens when the counter (RAM) gets 100% full? Rather than crashing immediately, the OS starts moving stuff off the counter back into the storeroom (disk) and pretending a corner of the disk is extra RAM. That’s swap.
Because disk is thousands of times slower than RAM, the moment a server starts swapping heavily, latency goes through the roof and the box feels completely frozen.
2. Page Cache (why Linux “free RAM” looks low)
Juniors often run free -m, see almost all RAM used up by buff/cache, and panic. Don’t!
If you have empty space on your kitchen counter (free RAM), Linux smartly keeps recently read files sitting on the counter in the page cache just in case you ask for the same file again. The moment an application actually needs that RAM, the kernel drops the cached files immediately. Unused RAM is wasted RAM.
3. Disk: IOPS vs Bandwidth (Throughput)
When an EBS volume or disk is slow, people just say “disk is bottlenecked”. There are actually two completely different limits on a disk:
- IOPS (Input/Output Operations Per Second): How many separate trips to the storeroom you can make per second. An OLTP database doing 10,000 tiny 4 KB random reads/writes per second will max out your IOPS limit long before it hits your MB/s bandwidth limit.
- Bandwidth / Throughput (MB/s): How much total weight you can carry per second in big sequential chunks (like reading a 10 GB video file or running
mysqldump). Here IOPS is low, but you hit the MB/s pipe cap.
4. CPU iowait
Suppose you look at top and see iowait is at 60%.
That does not mean your CPU is overloaded doing math. It means the chef (CPU) is standing there doing nothing because he’s waiting for a read/write from the disk storeroom to come back! Adding more CPU cores won’t fix high iowait - you have a slow disk (or an unindexed DB query doing full-table disk scans).
One important gotcha: waiting on the network does NOT show up as iowait. iowait is specifically waiting on block storage (disk). If your app is blocked waiting 5 seconds for an external HTTP API call, CPU just shows as idle.
5. CPU %steal vs CPU Throttling (noisy neighbors)
Why is your app slow when your own CPU usage looks normal? Two different cloud/k8s culprits:
%steal(VM level): Remember your AWS EC2 instance is a VM sharing physical hardware with other customers’ VMs. If the physical host is oversubscribed and the hypervisor makes your VM wait for a physical CPU core, Linux reports that as%stealintop. If%stealis high on an EC2 box, stop debugging your code - stop/start the instance so it lands on a new physical host (or move off burstablet2/t3instances after you’ve burned your CPU credits).- CPU Throttling (Kubernetes / container level): In k8s, if you set a container’s CPU
limits: 500m, the Linux CFS scheduler gives your container 50ms of CPU time per 100ms window (cpu_quota / cpu_period). If your multi-threaded app burns through its 50ms allowance in the first 10ms of the window, the kernel freezes your container for the remaining 90ms! Your node has plenty of idle CPU, your average CPU graph looks fine (50%), and yet your p99 latency is terrible because of CFS throttling.
6. Memory Overcommit & the OOM Killer
By default, Linux lies to processes when they ask to allocate memory (malloc). A process asks for 8 GB, the kernel says “sure, here you go”, gambling that the process won’t actually write to all 8 GB at once.
When all processes suddenly cash in those promises at the same time and physical RAM + swap actually hits 100%, the Linux kernel has only one move left to save the OS from hanging: the OOM Killer (Out Of Memory Killer) wakes up, picks the process with the highest OOM score, and shoots it in the head with SIGKILL (Exit Code 137 / OOMKilled in Kubernetes).
Summary table
| Metric / Symptom | What’s actually happening | Where to look |
|---|---|---|
High us / sy CPU | CPU is actually busy running app or kernel code | Flamegraphs, thread dumps, add cores |
High wa (iowait) | CPU is blocked waiting on disk reads/writes | Check disk IOPS/throughput (iostat -xz 1), DB slow queries |
High st (%steal) | Hypervisor / noisy neighbor stealing physical CPU | Check t3 CPU credit balance, or replace the EC2 instance |
High k8s container_cpu_cfs_throttled_seconds_total | Container hitting its k8s CPU limit | Raise or remove CPU limits on the pod |
| Swap usage climbing | Out of RAM, spilling onto disk | Memory leak, or raise RAM |
OOMKilled (exit 137) | Exceeded container memory limit or host RAM | Raise pod memory limit or fix memory spike |