#linux #vmstat #performance #troubleshooting #devops #sre #monitoring #loadaverage #iostat #systemadministration ## The Problem You Have Seen Before It is 2am. The on-call alert fires: "High load average on prod-web-07." You SSH in, run `uptime`, and see load average sitting at 8.5. The server has 4 cores, so that is more than double capacity. You expect to find CPUs pegged at 100%. Instead, `top` shows CPU usage at 25%. The numbers do not make sense. What is actually wrong with this server? This scenario plays out in production environments constantly, and it exposes a fundamental misunderstanding: **load average does not measure CPU usage**. It measures something different — and if you do not understand what that something is, you will waste time chasing the wrong problem while your actual bottleneck continues degrading service. The tool that cuts through this confusion is `vmstat`. It is not new. It is not exciting. But it is one of the most powerful diagnostic tools in Linux, and most engineers do not use it effectively. This article explains what server load actually means, why load average misleads you, and how to use `vmstat` to identify the real bottleneck in under a minute. --- ## What Server Load Actually Means Load average is the **average number of processes** in a runnable or uninterruptible wait state over 1, 5, and 15 minutes. That definition matters. A process in a runnable state is ready to execute but waiting for CPU time. A process in an uninterruptible wait state is blocked waiting for IO — typically disk or network. Both count toward load average equally, but they represent completely different problems. ### The CPU Core Relationship Load needs context. A load average of 2.0 means: - **On a 2-core system:** You have exactly enough work to keep both cores busy. This is normal. - **On a 16-core system:** You are using 12.5% of available capacity. This is light load. The rule of thumb: if load average is below your core count, you have spare capacity. If it exceeds core count significantly, you have more work than the system can handle. But this rule assumes the load is CPU-bound, which is often wrong. ### The IO Wait Problem Here is where it breaks down. Imagine a server with 4 cores and a load average of 8. You assume 8 processes are fighting for 4 CPUs. But what if the truth is: - 1 process using CPU - 7 processes blocked waiting for a slow disk Load average counts all 8. `top` shows 25% CPU usage. The bottleneck is not CPU — it is IO. Load average alone cannot tell you which scenario you are in. You need to see **where** those processes are waiting. --- ## Why Traditional Tools Are Not Enough Most engineers reach for `top` or `htop` when diagnosing load. These tools are good for a quick overview, but they hide critical information: **`top` shows you:** - Current CPU usage percentage - Memory usage - Top processes by CPU or memory **`top` does NOT show you:** - How many processes are blocked on IO - Whether the system is swapping - Scheduling queue depth - Sustained vs momentary resource usage `uptime` gives you load average, but as we have established, that number is almost useless without context. You see "load 8.5" and have no idea if that is 8.5 processes waiting for CPU or 8.5 processes waiting for disk IO to complete. What you need is a tool that shows the **breakdown** of where processes are actually spending time. That tool is `vmstat`. --- ## Introduction to vmstat `vmstat` reports virtual memory statistics, but it actually shows much more: process states, memory usage, swap activity, IO activity, system calls, and CPU utilization. The critical advantage over `top` is that `vmstat` shows you the **scheduling queue** — the number of processes waiting to run vs blocked on IO. ### Basic Usage ```bash vmstat 1 ``` This samples system statistics every 1 second. The output looks like: ``` procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 245328 89564 823412 0 0 3 12 45 89 5 2 93 0 0 1 0 0 245124 89564 823568 0 0 0 28 312 581 8 3 89 0 0 3 0 0 244916 89564 823712 0 0 0 0 289 567 6 2 92 0 0 ``` The first line is average since boot — ignore it. Watch the subsequent lines. ### Column Breakdown **procs (Process States):** - `r` — Number of processes waiting for CPU time (runnable). This is your CPU queue depth. - `b` — Number of processes in uninterruptible sleep, waiting for IO to complete. This is your IO queue depth. **memory:** - `swpd` — Virtual memory used (swap). Non-zero means you have run out of RAM. - `free` — Idle memory. - `buff` — Memory used for buffers. - `cache` — Memory used for cache. **swap:** - `si` — Memory swapped in from disk (KB/s). - `so` — Memory swapped out to disk (KB/s). **io:** - `bi` — Blocks received from block devices (blocks/s). - `bo` — Blocks sent to block devices (blocks/s). **cpu (as percentage of total CPU time):** - `us` — Time spent running user space processes. - `sy` — Time spent running kernel code. - `id` — Idle time. - `wa` — Time spent waiting for IO. - `st` — Stolen time (time taken by hypervisor for other VMs). Only relevant in virtualized environments. The two columns that matter most for diagnosing load: **`r` and `wa`**. These tell you whether your bottleneck is CPU or IO. --- ## Real Production Diagnosis Examples ### Example 1: CPU Bottleneck You SSH into a server with high load. You run: ```bash vmstat 1 ``` Output: ``` procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 12 0 0 156432 42180 612844 0 0 1 5 145 234 85 12 3 0 0 11 0 0 156128 42180 613024 0 0 0 0 423 891 88 10 2 0 0 13 0 0 155840 42180 613156 0 0 0 8 398 856 86 11 3 0 0 ``` **Interpretation:** - `r = 12` — There are 12 processes waiting for CPU time. If this server has 4 cores, you have 3x more work than capacity. - `b = 0` — No processes blocked on IO. The disk is not the problem. - `us = 85-88%` — User space processes are consuming most CPU. - `wa = 0%` — No time spent waiting for IO. **Diagnosis:** This is a CPU-bound workload. You are either undersized for the load, or an application is burning cycles inefficiently. Next step: identify the top CPU consumers with `top` or `htop`, then investigate why they are consuming so much CPU. --- ### Example 2: Disk IO Bottleneck You run `vmstat 1` and see: ``` procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 8 0 312456 18920 445632 0 0 145 1823 201 345 12 6 42 40 0 2 9 0 311840 18920 446012 0 0 198 2156 189 298 15 8 35 42 0 1 7 0 311324 18920 446380 0 0 167 1934 223 401 10 5 38 47 0 ``` **Interpretation:** - `r = 1-2` — Only 1-2 processes waiting for CPU. CPU queue is fine. - `b = 7-9` — Up to 9 processes blocked waiting for IO. This is your bottleneck. - `wa = 40-47%` — CPU is spending almost half its time idle, waiting for disk. - `bo = 1800-2100` — Heavy write activity to disk. **Diagnosis:** The disk subsystem cannot keep up. Processes are piling up waiting for disk writes to complete. This is common on: - Servers with slow disks (spinning rust, not SSD). - Cloud instances with burstable IOPS that have exhausted their burst credits. - Databases writing large volumes without sufficient write cache. Next step: Use `iotop` to identify which process is hammering the disk, then either optimize the workload or upgrade the storage tier. --- ### Example 3: Memory Pressure with Swapping You run `vmstat 1` and see: ``` procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 2 145680 8432 5124 23456 412 856 523 934 389 678 25 18 32 25 0 4 3 148920 6780 5124 22180 567 923 612 1045 412 723 28 20 28 24 0 2 4 151340 5896 5124 21456 489 734 498 876 356 689 22 16 35 27 0 ``` **Interpretation:** - `swpd = 145-151 MB` — The system is using swap. You have run out of physical RAM. - `si = 400-600 KB/s` — Actively swapping in from disk. - `so = 700-900 KB/s` — Actively swapping out to disk. - `wa = 24-27%` — Significant time waiting for IO. - `b = 2-4` — Processes blocked on IO (related to swap). **Diagnosis:** The server is thrashing. It ran out of memory and is now using swap as overflow. Swap is on disk, so every memory access that hits swap becomes a disk IO operation. This creates a vicious cycle: processes need memory → system swaps to disk → processes block on IO → load climbs. Next step: Identify the memory hog with `top`, then either kill it, optimize it, or add more RAM. Swapping on a production server is never acceptable for performance-critical workloads. --- ### Example 4: Container Overload on Kubernetes Node You are debugging a Kubernetes node reporting high load. You run `vmstat 1`: ``` procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 18 0 0 412560 89340 1923440 0 0 2 18 4521 8934 45 38 17 0 0 16 0 0 408920 89340 1925680 0 0 0 24 4893 9245 48 40 12 0 0 19 0 0 405780 89340 1927120 0 0 0 0 5124 9678 50 42 8 0 0 ``` **Interpretation:** - `r = 16-19` — Large CPU queue. Many processes waiting. - `b = 0` — No IO blocking. - `us = 45-50%` — User space CPU usage. - `sy = 38-42%` — Kernel CPU usage is extremely high. This is unusual. - `cs = 8900-9700` — Context switches per second are very high. **Diagnosis:** High system CPU combined with high context switches indicates the kernel is spending a lot of time scheduling. This is characteristic of: - Too many containers/pods on the node, each with small CPU limits. - Thousands of threads competing for CPU time. - Kernel overhead from excessive context switching. On a Kubernetes node, this often means the node is over-committed. The scheduler is thrashing trying to give every pod a time slice. Next step: Check pod density on the node and consider redistributing workloads or increasing node size. --- ## Essential Diagnostic Tools Reference Before diving into the workflow, here is the toolkit you need and what each tool does: ### 1. uptime — Load Average at a Glance ```bash uptime ``` **What it shows:** Current time, uptime, number of users, and load average for 1, 5, and 15 minutes. **Example output:** ``` 14:23:45 up 12 days, 3:42, 2 users, load average: 8.45, 6.12, 4.89 ``` **Use it for:** Quick initial assessment. Is load high? Is it trending up or stabilizing? **Limitation:** Tells you load is high, but not _why_. --- ### 2. nproc — CPU Core Count ```bash nproc ``` **What it shows:** Number of processing units available to the current process (typically equals CPU cores). **Example output:** ``` 8 ``` **Use it for:** Context for load average. Load 8 on 8 cores is different from load 8 on 2 cores. **Quick math:** Load average / core count = load per core. Values above 1.0 indicate saturation. --- ### 3. top — Real-Time Process View ```bash top ``` **What it shows:** CPU and memory usage per process, sorted by resource consumption. Header shows overall CPU breakdown. **Key metrics in header:** ``` %Cpu(s): 24.5 us, 8.2 sy, 0.0 ni, 52.8 id, 14.5 wa, 0.0 hi, 0.0 si, 0.0 st ``` - `us` (user) — Application CPU usage - `sy` (system) — Kernel CPU usage - `wa` (wait) — Time waiting for IO - `id` (idle) — Unused CPU capacity **Use it for:** Identifying top CPU or memory consumers. Quick overview of CPU breakdown. **Limitation:** Does not show process queue depth or blocked processes. Cannot distinguish between CPU-bound and IO-bound load. --- ### 9. htop — Interactive Process Manager ```bash htop ``` **What it shows:** Color-coded, interactive version of `top` with better UI and easier sorting. **Key features:** - Press `F6` to sort by CPU%, MEM%, or other metrics - Per-core CPU usage bars at the top - Tree view to see process hierarchies (`F5`) - Kill processes with `F9` **Use it for:** Finding which specific process is consuming resources after `vmstat` identifies the bottleneck type. **Pro tip:** `htop` is not always installed by default. Install with `apt install htop` or `yum install htop`. --- ### 5. iotop — IO Activity by Process ```bash sudo iotop -o ``` **What it shows:** Disk read/write activity per process, similar to `top` but for IO instead of CPU. **Example output:** ``` Total DISK READ: 45.67 M/s | Total DISK WRITE: 123.45 M/s PID USER DISK READ DISK WRITE SWAPIN IO COMMAND 1234 postgres 12.34 M/s 89.12 M/s 0.00 % 45.2% postgres: writer 5678 mysql 8.45 M/s 34.23 M/s 0.00 % 28.6% mysqld ``` **Use it for:** Identifying which process is causing high IO wait (`wa` in `vmstat`). **Pro tip:** The `-o` flag shows only processes actually doing IO, filtering out idle processes. --- ### 6. iostat — IO Statistics by Device ```bash iostat -xz 1 ``` **What it shows:** Detailed statistics for each storage device: throughput, utilization, average wait times, and queue depth. **Example output:** ``` Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util sda 12.4 89.3 145.2 1234.8 0.12 4.56 0.96 4.86 2.34 45.67 3.21 11.71 13.82 8.45 92.34 nvme0n1 8.2 34.1 98.5 456.3 0.05 1.23 0.61 3.48 0.89 4.23 0.45 12.01 13.38 1.23 15.67 ``` **Key columns:** - `r/s`, `w/s` — Reads and writes per second - `rkB/s`, `wkB/s` — Read and write throughput in KB/s - `r_await`, `w_await` — Average time (ms) for read/write requests to be served - `%util` — Percentage of time the device was busy handling requests (disk saturation indicator) **Use it for:** - Determining which disk is saturated (`%util` near 100%) - Measuring disk latency (`r_await`, `w_await`) - Identifying whether the problem is read or write heavy **When to use it:** After `vmstat` shows high `wa` (IO wait), use `iostat` to see which disk is the bottleneck, then use `iotop` to find which process. **Pro tip:** The `-x` flag shows extended statistics, `-z` omits devices with no activity. --- ### 7. ps aux — Process Snapshot ```bash ps aux ``` **What it shows:** Snapshot of all running processes with CPU, memory, and state information. **Example output:** ``` USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.1 16864 8432 ? Ss Feb18 0:12 /sbin/init postgres 1234 12.4 8.3 245680 89432 ? Ss 08:23 4:23 postgres: writer process mysql 5678 45.2 15.6 892340 167234 ? Ssl Feb18 89:45 /usr/sbin/mysqld www-data 9012 2.3 1.2 56784 12456 ? S 14:02 0:34 nginx: worker process ``` **Key columns:** - `%CPU` — CPU usage percentage (sampled at snapshot time) - `%MEM` — Memory usage percentage - `VSZ` — Virtual memory size - `RSS` — Resident set size (actual physical memory used) - `STAT` — Process state (R=running, S=sleeping, D=uninterruptible sleep/IO wait, Z=zombie) **Use it for:** - Finding processes in uninterruptible sleep (`STAT` = `D`) — these contribute to load average and are waiting for IO - Getting a full process list for scripting or further analysis - Checking process states when `top`/`htop` are not sufficient **Common use cases:** ```bash # Find processes in uninterruptible sleep (IO wait) ps aux | awk '$8 ~ /D/ {print}' # Sort by memory usage ps aux --sort=-%mem | head -20 # Sort by CPU usage ps aux --sort=-%cpu | head -20 # Find all processes for a specific user ps aux | grep postgres ``` **Pro tip:** `ps` shows a snapshot at one moment in time. For sustained high resource usage, prefer `top` or `htop`. For understanding process states and IO-blocked processes, `ps aux` is invaluable. --- ### 8. vmstat — Virtual Memory Statistics ```bash vmstat 1 ``` **What it shows:** Process queue depth, memory usage, swap activity, IO activity, and CPU breakdown — all in one view. **Use it for:** Understanding _where_ processes are waiting: CPU queue (`r`), IO queue (`b`), or swap activity. **Why it is critical:** This is the only tool that shows you the **breakdown** of load between CPU-bound and IO-bound processes. (See detailed column explanation in Section 4 above.) --- ### 10. docker stats — Container Resource Usage ```bash docker stats --no-stream ``` **What it shows:** CPU, memory, network, and disk IO usage for each running container. **Example output:** ``` CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O a1b2c3d4e5f6 web-app 45.2% 1.2GiB / 4GiB 30.0% 12MB / 8MB 145MB / 89MB f6e5d4c3b2a1 postgres 12.8% 890MiB / 2GiB 43.5% 2MB / 3MB 2.3GB / 1.8GB ``` **Use it for:** Identifying which container is consuming resources on a Docker host. **Limitation:** Shows container-level metrics, but not process-level detail inside the container. --- ### 11. kubectl top — Kubernetes Pod and Node Metrics ```bash # Node-level resource usage kubectl top nodes # Pod-level resource usage kubectl top pods -A ``` **What it shows:** CPU and memory usage for Kubernetes nodes and pods. **Example output (nodes):** ``` NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% node-1 3420m 42% 12Gi 75% node-2 1890m 23% 8Gi 50% ``` **Example output (pods):** ``` NAMESPACE NAME CPU(cores) MEMORY(bytes) production web-app-7d4f8c9b-xk2pl 245m 512Mi production api-server-5f6a8d-mn9qr 890m 1.2Gi ``` **Use it for:** Identifying which pod or node is under load in a Kubernetes cluster. **Pro tip:** Requires metrics-server to be installed in the cluster. --- ## How to Quickly Diagnose Load in 30 Seconds When you SSH into a server with high load, follow this sequence: ### 1. Check load average and core count ```bash uptime nproc ``` If load is below core count, the issue is likely transient. If load is 2x+ core count, continue. ### 2. Run vmstat for 10 seconds ```bash vmstat 1 10 ``` Watch for patterns: - **High `r`, low `wa`:** CPU bottleneck. Check which processes with `htop`. - **High `b`, high `wa`:** IO bottleneck. Check which processes with `iotop`. - **Non-zero `si`/`so`:** Swapping. Memory pressure. Check with `free -h` and `htop`. - **High `sy`, high `cs`:** Kernel overhead, possibly from too many threads. ### 3. Identify the culprit Based on `vmstat` output: - **CPU bottleneck:** `htop` sorted by CPU (`F6` → CPU%). - **IO bottleneck:** - First, use `iostat -xz 1 5` to see which disk is saturated (`%util` near 100%) - Then use `sudo iotop -o` to identify which process is hammering that disk - **Memory bottleneck:** `htop` sorted by memory (`F6` → MEM%). - **Many processes in uninterruptible sleep:** `ps aux | awk '$8 ~ /D/'` to list them ### 4. Check containerized workloads (if applicable) If running Docker: ```bash docker stats --no-stream ``` If running Kubernetes: ```bash kubectl top nodes kubectl top pods -A --sort-by=cpu ``` This identifies which container or pod is responsible for the load. --- ### Complete 30-Second Workflow ```bash # 1. Initial assessment (5 seconds) uptime && nproc # 2. Detailed breakdown (10 seconds) vmstat 1 10 # 3. Based on vmstat results: # If high 'r' (CPU bottleneck): htop # Press F6, sort by CPU% # If high 'b' and 'wa' (IO bottleneck): iostat -xz 1 5 # Identify which disk sudo iotop -o # Identify which process # If processes stuck in uninterruptible sleep: ps aux | awk '$8 ~ /D/ {print $2, $11}' # List PIDs and commands # If containers are running: docker stats --no-stream # OR kubectl top pods -A --sort-by=cpu ``` This takes 30 seconds and gives you a clear direction. No guessing, no chasing the wrong metric. --- ## Best Practices for Production Systems ### Normal vs Dangerous Values These are rough guidelines, not hard rules. Context matters. **CPU queue depth (`r`):** - **Normal:** `r` is less than or equal to number of cores. - **Warning:** `r` is 1.5x core count. - **Critical:** `r` is 2x+ core count sustained for more than a few minutes. **IO wait (`wa`):** - **Normal:** `wa` below 10%. - **Warning:** `wa` between 10-30%. - **Critical:** `wa` above 30% sustained. The disk is a bottleneck. **Swap activity (`si`/`so`):** - **Normal:** Both zero. - **Critical:** Any non-zero value sustained. Swapping kills performance. **Context switches (`cs`):** - **Normal:** Varies widely by workload. Baseline your system. - **Warning:** Sudden spike (10x normal). - **Critical:** Sustained high values (50k+/s on modern hardware). ### Acceptable Run Queue Size A common question: how many processes in the run queue is too many? The answer depends on what those processes are doing. If you have 8 processes in the queue and they are all short-lived request handlers that execute for 10ms each, the queue drains quickly and users see low latency. If you have 8 processes that each run for 10 seconds, new work sits in the queue for over a minute before starting. The safer rule: **watch latency, not just queue depth**. A queue depth of 20 that drains in 100ms is fine. A queue depth of 5 that takes 10 seconds to drain is a problem. Use application-level latency monitoring (p95, p99 response times) as your real signal. Use `vmstat` to identify _why_ latency is bad. ### What to Baseline Do not wait for an incident to learn what normal looks like. On a new server or after a deployment, run: ```bash vmstat 1 60 > vmstat_baseline.txt ``` Capture 60 seconds of normal operation. When you are troubleshooting later, you can compare against this baseline to spot anomalies. --- ## Conclusion Load average is a misleading metric if you do not understand what it measures. A high load average can mean CPU saturation, disk IO blocking, memory thrashing, or kernel scheduling overhead — and you cannot tell which from the number alone. Chasing load average without understanding the underlying cause wastes time. `vmstat` is one of the most powerful and underused tools in the Linux diagnostic toolkit. It shows you the breakdown: how many processes are waiting for CPU, how many are blocked on IO, whether the system is swapping, and where CPU time is actually going. With `vmstat`, you identify the bottleneck in under a minute and know exactly where to focus your effort. The next time you see a high load alert, do not guess. Do not assume. Run `vmstat 1` for 10 seconds and let the data tell you the real story. CPU bottleneck, IO bottleneck, memory pressure — the numbers do not lie. Learn to read them, and you will diagnose production issues faster than engineers who rely on load average alone. --- _Vladimiras Levinas is a Lead DevOps Engineer with 18+ years in fintech infrastructure. He runs a production K3s homelab and writes about AI infrastructure at doc.thedevops.dev_