Diagnosing High iowait on a Linux Server With iostat (September 2026)Expert Reviews

You SSH into a Linux server and everything feels sluggish. Commands take seconds to respond. The load average is climbing past 20. You run top and the CPU usage looks fine, maybe even idle, yet something is clearly wrong. The %wa column tells the story: your system is drowning in disk I/O.

High iowait is one of the most confusing performance issues Linux system administrators face. The symptom, a high load average with seemingly idle CPUs, looks contradictory at first glance. And the fix is never obvious because iowait is a symptom, not a cause. You need to dig deeper to find what is actually saturating your storage.

This guide walks through the complete diagnostic process for diagnosing high iowait on a Linux server with iostat. We cover every tool in the chain: top for initial detection, vmstat for system-wide confirmation, iostat for device-level analysis, iotop and pidstat for process identification, and dmesg for hardware failures. We also cover Pressure Stall Information (PSI), cloud volume throttling, and post-incident analysis with sar that most tutorials skip.

Whether you manage bare-metal servers, AWS EC2 instances, or a fleet of containers, the workflow below will help you go from “something is slow” to “this specific process is hammering this specific device, and here is the fix.”

What Is iowait and Why Does It Spike?

iowait (also written as %iowait or wa in top) is the percentage of time the CPU is idle while at least one outstanding disk I/O request is waiting to complete. The key insight: iowait is a subset of idle time, not a subset of CPU usage. When you see 90% iowait, your CPU is not working hard. It is sitting idle because the tasks it wants to run are blocked waiting for disk.

The Linux kernel tracks this in /proc/stat. Every CPU’s time is divided into categories: user, nice, system, idle, iowait, irq, softirq, and steal. The kernel measures time in units called USER_HZ ticks, which is typically 100 ticks per second (10 milliseconds each). When a CPU core has no runnable tasks in its run queue and there is at least one task blocked on I/O, the kernel accrues those ticks as iowait rather than plain idle.

This creates a counterintuitive effect. If you add more CPU-intensive work to a system with high iowait, the iowait percentage often drops. The CPU was never busier in the iowait state; it just has something to do now, so the idle-iowait ticks get replaced by user-system ticks. This is why iowait alone is a poor metric and why you need iostat to understand what is actually happening at the device level.

How Much iowait Is Too Much?

There is no universal threshold for acceptable iowait. On a fast NVMe SSD with a well-tuned database, sustained iowait above 5% is worth investigating. On a 5400 RPM laptop drive or a heavily loaded NFS mount, iowait near 80% during large file copies can be completely normal for that hardware.

What matters is the context. A sudden spike from 2% to 60% on a server that normally sits at 1% is a problem regardless of the absolute number. Sustained iowait above 25% on production database servers usually translates to noticeable latency for end users.

Focus on changes from your baseline rather than chasing a specific number. If your monitoring shows a new pattern, investigate. If iowait has always been 15% on a particular server and performance is acceptable, that is your normal.

Quick Diagnostic Checklist: 5 Steps to Find the Bottleneck

When you need a fast answer, follow this sequence. Each tool narrows the problem down further:

1. Run top to confirm high iowait and check the load average. Press 1 to see per-core breakdown.

2. Run vmstat 1 to see system-wide I/O activity. Look at the wa, bi, and bo columns.

3. Run iostat -x 1 to identify which specific block device is saturated. Check %util, r_await, w_await, and aqu-sz.

4. Run iotop -oPa or pidstat -d 1 to find the exact process generating the I/O.

5. Check dmesg for hardware errors, ATA exceptions, or filesystem corruption before blaming the workload.

This five-step flow takes under two minutes and tells you whether the problem is a failing disk, a runaway process, or cloud throttling. The sections below explain each step in detail.

Step 1: Confirm High iowait With the top Command

Start with top because it is available on every Linux system with zero installation. The third line of the header shows CPU state percentages: us (user), sy (system), ni (nice), id (idle), wa (iowait), hi (hardware interrupt), si (software interrupt), and st (steal).

Look at %wa. If it is consistently above 20-30%, your system is spending significant time waiting for I/O. The load average (shown on the first line) will likely be elevated too, but load average counts processes in uninterruptible sleep (D state) as “running,” so it overstates CPU load during I/O bottlenecks.

Press 1 inside top to expand the CPU section and show each core individually. This matters because iowait can concentrate on specific cores handling I/O interrupts. If you see one core at 90% iowait while others sit near 0%, that narrows the problem to interrupt handling or NUMA effects.

top confirms the problem exists, but it cannot tell you which disk or which process is responsible. Move to vmstat next.

Step 2: Confirm the Problem With vmstat

vmstat 1 gives you a one-line-per-second system-wide view. Run it for 10-20 seconds to observe the pattern. The key columns are:

• wa under CPU: iowait percentage. Consistent values above 20% confirm a storage bottleneck.

• bi (block in): blocks read from disk per second.

• bo (block out): blocks written to disk per second.

• r: number of runnable processes (waiting for CPU).

• b: number of processes in uninterruptible sleep (blocked on I/O, D state).

If wa is high but bi and bo are near zero, something strange is happening. This can occur on systems where iowait decays slowly, or when I/O completions are delayed by driver issues. In most cases, high wa comes with elevated bi or bo.

The b column is particularly useful. A high value means many processes are stuck in D state waiting for I/O. This is the direct cause of system sluggishness. If b is 15 on a 4-core machine, 15 processes are blocked on I/O at that instant, and the system will feel unresponsive.

vmstat confirms the system-wide picture but cannot identify specific devices. That is where iostat takes over.

Step 3: Pinpoint the Device With iostat

This is the core step in diagnosing high iowait on a Linux server with iostat. The iostat command provides per-device I/O statistics that tell you exactly which disk is saturated and in what way.

First, make sure the sysstat package is installed:

sudo apt install sysstat (Debian/Ubuntu)
sudo dnf install sysstat (RHEL/CentOS/Fedora)

Then run extended statistics with one-second intervals:

iostat -x 1

The extended output shows every block device with detailed columns. Here is what each column means and when to care about it:

• r/s and w/s: Read and write requests completed per second. High values (tens of thousands) indicate an IOPS-heavy workload.

• rkB/s and wkB/s: Kilobytes read and written per second. This tells you the actual throughput.

• r_await and w_await: Average time in milliseconds for read and write requests to complete, including queue wait time. This is the most important latency indicator. Values consistently above 10-20 ms on SSD, or 50+ ms on HDD, indicate the device cannot keep up.

• rareq-sz and wareq-sz: Average request size in kilobytes. Small sizes (4-8 KB) with high r/s suggest an IOPS-bound workload. Large sizes (128 KB+) suggest sequential throughput.

• aqu-sz: Average queue size for the device. A growing queue means requests are piling up faster than the device can process them. Anything consistently above 2-4 on a single HDD, or above the device queue depth on SSDs, signals saturation.

• %util: Percentage of time the device had at least one request in flight. At 100%, the device was busy the entire interval.

A critical warning about %util: on modern NVMe drives with multiple hardware queues, %util can hit 100% while the device still has plenty of capacity. The metric measures busyness, not saturation. A fast NVMe SSD can sustain massive IOPS at 100% %util with r_await under 1 ms. Always cross-reference %util with latency columns (r_await, w_await) and queue depth (aqu-sz) before drawing conclusions.

For a practical example, if you see %util at 100%, w_await at 85 ms, aqu-sz at 12, and wareq-sz at 4 KB on device sda, that device is IOPS-bound. The small request size combined with high latency and growing queue tells you the disk is drowning in small random writes. That is a very different problem from %util at 100% with w_await at 3 ms and wareq-sz at 256 KB, which just means sequential throughput is maxing out.

Step 4: Find the Process Causing High iowait

Once iostat identifies the saturated device, you need to find which process is responsible. Two tools handle this: iotop for real-time monitoring and pidstat for per-process statistics.

iotop requires root and shows a live, top-style view sorted by I/O activity. Run it with flags that filter out idle processes:

sudo iotop -oPa

The -o flag shows only processes doing I/O. The -P flag shows processes instead of threads. The -a flag accumulates I/O counts. The columns DISK READ and DISK WRITE show per-process bandwidth. SWAPIN and IO percentages indicate how much the process is waiting on I/O.

If iotop is not available (it requires the kernel Python module and can be unavailable in minimal containers), use pidstat instead:

pidstat -d 1

This shows per-process I/O statistics updated every second. Look for processes with high kB_rd/s or kB_wr/s. The pidstat approach works in containers and minimal environments where iotop fails.

When a process is stuck in I/O wait, it enters the D state (uninterruptible sleep). You can find these processes with: ps aux | awk '$8 ~ /^D/'. D-state processes cannot be killed with SIGTERM or SIGKILL until the I/O completes. If the underlying device is truly hung (not just slow), these processes stay in D state indefinitely, and you may need to address the hardware or driver issue.

Step 5: Check dmesg for Hardware and Driver Errors

Before you blame your workload, check dmesg for hardware failures. Disk errors, ATA bus resets, and filesystem corruption all produce high iowait, and no amount of process tuning fixes a failing drive.

Run dmesg | tail -100 and look for messages like:

• blk_update_request: I/O error

• ata1: hard resetting link

• BTRFS critical or EXT4-fs error

• nvme controller is down

• device disconnected

Any of these indicate hardware or driver problems. If you see repeated ATA resets or I/O errors on a specific device, run a SMART check: sudo smartctl -a /dev/sdX. Look for reallocated sectors, pending sectors, or uncorrectable errors. A drive with growing reallocated sector counts is failing and should be replaced.

NVMe drives log controller errors differently. Check /sys/class/nvme/nvme0/device/firmware_log or use nvme smart-log /dev/nvme0 to read controller health. Media errors, CRC errors, or thermal throttling all produce high latency that shows up as iowait.

If dmesg is clean, your hardware is likely fine and the problem is workload-related. Proceed to the next sections.

Throughput-Bound or IOPS-Bound? Choosing the Right Fix

The fix for high iowait depends on whether your workload is IOPS-bound or throughput-bound. You can tell the difference from iostat columns.

IOPS-bound workloads show high request counts (r/s or w/s in the thousands) with small request sizes (rareq-sz or wareq-sz under 16 KB). Latency (r_await, w_await) is high because the device is processing too many individual operations. This pattern is common with databases doing random reads and writes.

Fixes for IOPS-bound problems: upgrade to faster SSD or NVMe storage, increase database cache sizes to reduce disk access, tune application read-ahead, batch small writes into larger operations, or add more IOPS capacity on cloud volumes (switching from gp2 to gp3, provisioning more IOPS).

Throughput-bound workloads show large request sizes (128 KB and above) with high rkB/s or wkB/s. The device is saturated by sequential data transfer. This pattern occurs during backups, large file copies, RAID rebuilds, and log writes.

Fixes for throughput-bound problems: move large operations to off-peak hours, use faster interconnects (SAS to NVMe), compress data before writes, spread writes across multiple devices, or upgrade to higher-throughput cloud volumes.

The distinction matters because the wrong fix wastes money and time. Adding IOPS to a throughput-bound system does nothing. Upgrading bandwidth on an IOPS-bound system does nothing. Read the iostat columns first.

Common Causes of High iowait and How to Fix Each

Here are the workloads and conditions we see most often causing high iowait in production:

Swap thrashing. When physical RAM is exhausted and the system starts swapping to disk, every memory access becomes a disk operation. vmstat shows high si and so (swap in/out) columns. Fix: add RAM, tune vm.swappiness down to 10 or lower, or reduce memory-hungry processes. free -h confirms available memory.

Database checkpoints and log writes. PostgreSQL, MySQL, and other databases periodically flush dirty pages to disk during checkpoints. This produces bursts of sequential writes. Fix: tune checkpoint_completion_target (PostgreSQL), increase innodb_buffer_pool_size (MySQL), or provision higher-throughput storage for WAL/log volumes.

RAID rebuild. When a drive fails in a RAID array, the rebuild process generates massive I/O that can last hours. cat /proc/mdstat shows rebuild progress. Fix: schedule rebuilds during maintenance windows, use hot spares to start rebuilds immediately, and ensure the array has dedicated I/O bandwidth.

Large file copies and backups. Copying 500 GB to a single disk will saturate it. Fix: use ionice to lower the priority of backup jobs: ionice -c2 -n7 rsync -a /source/ /dest/. Schedule backups during low-traffic periods.

NFS mount slowdowns. Network filesystems can produce high iowait when the network is slow or the NFS server is overloaded. nfsstat -c shows retransmission rates. Fix: tune NFS mount options (rsize, wsize, actimeo), switch to NFS over TCP, or investigate server-side load.

Container I/O contention. Multiple containers writing to the same underlying device compete for I/O bandwidth. Docker and Kubernetes do not enforce I/O limits by default. Fix: use --device-write-iops and --device-write-bps in Docker, or configure I/O limits in Kubernetes storage classes.

Cloud Volume Throttling: When the Disk Is Fine But the Cloud Slows Us Down

If you run on AWS, Azure, or GCP, your “disk” is a network-attached volume. Cloud providers throttle IOPS and throughput based on volume type and size, and throttling looks identical to a slow disk in iostat.

The most common trap is the AWS EBS gp2 burst credit system. gp2 volumes get a baseline performance of 3 IOPS per GB, plus a burst bucket that allows short spikes. Small volumes (under 1 TB) have low baselines: a 100 GB gp2 volume gets only 300 IOPS sustained. Once the burst credits run out, IOPS drops to the baseline and r_await spikes to 50-100 ms or higher.

You can detect this in iostat by watching for sudden latency spikes on the device with no corresponding change in request patterns. The workload is the same, but the device suddenly takes 10 times longer to respond. That is throttling, not a hardware problem.

On AWS, switch to gp3 volumes. They decouple IOPS and throughput from volume size, and you can provision exactly the IOPS you need (up to 16,000 per volume) at a lower cost than gp2. Check the BurstBalance and VolumeReadOps CloudWatch metrics to confirm throttling before making changes.

Azure Premium SSDs have similar tier-based throttling. GCP persistent disks throttle based on the provisioned IOPS limit. In all cases, cross-reference iostat output with your cloud provider’s performance metrics to distinguish throttling from real workload issues.

Pressure Stall Information (PSI): A Modern Way to Measure I/O Pressure

Pressure Stall Information, available on kernels 4.20 and later, provides a more accurate picture of I/O pressure than iowait alone. PSI measures how much time tasks spend waiting for resources, broken down by resource type: CPU, memory, and I/O.

Read the I/O pressure file:

cat /proc/pressure/io

Output looks like: some avg10=12.50 avg60=5.00 avg300=1.20 total=84567234

The some line shows the percentage of time at least one task was stalled on I/O. The full line shows the percentage of time all tasks were stalled simultaneously. The avg10, avg60, and avg300 fields are 10-second, 60-second, and 5-minute rolling averages.

PSI is better than iowait for modern systems because it measures actual task stall time rather than CPU idle categorization. On multi-core systems, one core can show high iowait while the system as a whole is fine. PSI accounts for this by tracking real task delays. If avg10 for the full line is above 5-10%, tasks are meaningfully stalled and users will notice.

You can also set PSI thresholds that trigger actions via psimon or custom daemon scripts, making PSI useful for automated alerting in ways iowait cannot support.

Post-Incident Analysis: Finding What Caused iowait After the Fact

One of the most common questions on r/linuxadmin is how to figure out what caused a high iowait spike after it already happened. By the time you SSH in, the system is back to normal and all the real-time tools show nothing. The answer is sar, part of the sysstat package.

If sysstat is installed and data collection is enabled, sar stores historical system metrics at regular intervals (usually every 10 minutes). Enable it by editing /etc/default/sysstat (set ENABLED="true") and restarting the service.

Check historical CPU iowait: sar -u. This shows CPU usage including %iowait over time. To look at a specific date: sar -u -f /var/log/sysstat/saDD where DD is the day of the month.

Check historical disk activity: sar -d. This shows per-device I/O statistics over time. Combine with -p for readable device names. You can pinpoint exactly when a device started saturating and correlate it with other system events.

If sar data is not available because sysstat was never enabled, enable it now for future incidents. Also consider setting up atop with daily logging, which captures per-process snapshots for post-incident forensic analysis. The combination of sar for system-level trends and atop for process-level snapshots gives you complete visibility.

Frequently Asked Questions

Wrapping Up

Diagnosing high iowait on a Linux server with iostat comes down to a disciplined workflow: confirm the problem with top and vmstat, pinpoint the device with iostat -x, identify the process with iotop or pidstat, and rule out hardware failures with dmesg. Then apply the right fix based on whether you are IOPS-bound or throughput-bound.

The tools covered here, from the basic five-step checklist to advanced techniques like PSI monitoring, cloud throttling detection, and post-incident sar analysis, give you a complete diagnostic toolkit. Keep sysstat collection enabled on all production servers so you are never caught without historical data during the next incident.

If you take away one thing from this guide: iowait is idle time, not work. The CPU is waiting. Your job is to find out what it is waiting for and why, and iostat -x 1 is the single best command to start that investigation.