How to Benchmark VM Storage Performance With fio (September 2026)?

When a workload runs slow and the CPU looks bored, the bottleneck is almost always storage. Inside a virtual machine, that storage is another layer of software on top of someone else’s storage, so guessing gets you nowhere. This guide shows you, step by step, how to benchmark storage performance inside a VM with fio using the same commands production engineers run every day.

I have rebuilt the workflow below while testing NVMe-backed cloud disks, Proxmox virtio drives, and cheap SATA SSDs on a home lab. The commands are copy-paste safe on any modern Linux VM. By the end you will know what each parameter does, how to read the output, and which pitfalls trip up most people the first time.

What Is fio and Why Use It for VM Storage Benchmarking?

fio (Flexible I/O Tester) is a command-line workload generator that produces controlled read and write traffic against a block device or a filesystem. Instead of copying a single big file like dd does, fio simulates the I/O patterns real applications create: sequential streams, random 4K reads, mixed read/write ratios, deep queues. After the run, it reports three numbers that matter in production: IOPS, throughput (MB/s), and latency.

fio is the de facto storage benchmark because it is flexible, deterministic, and honest. You tell it what block size, queue depth, and pattern to use, and it produces repeatable numbers you can compare against later. Google Cloud, Nutanix, Arm, and most cloud providers publish their official disk benchmarks on fio for exactly this reason.

For VMs specifically, fio gives you the right unit of measurement. A VM does not see the raw NVMe drive; it sees a virtual disk presented by virtio, SCSI, or a hypervisor-specific controller. Running fio inside the guest tells you what the guest actually gets, after every virtualization layer has had its say.

Why not just use dd or bonnie++

dd only measures sequential throughput with a single thread. It cannot tell you about random IOPS, queue depth, or latency percentiles. bonnie++ is friendlier but its results are harder to reproduce and harder to script. fio covers all of these cases with one binary.

Installing fio on Linux

fio ships in the package repository of every major Linux distribution. Pick the line that matches your guest OS and run it as root (or with sudo).

Debian / Ubuntu:

sudo apt update && sudo apt install fio -y

RHEL / Rocky / AlmaLinux / CentOS Stream:

sudo dnf install epel-release -y && sudo dnf install fio -y

openSUSE:

sudo zypper install fio

Arch Linux:

sudo pacman -S fio

Verify the install worked and check the version:

fio --version

If you would rather build from source, fio is a small C project with no exotic dependencies. On a host with gcc and make, run git clone https://github.com/axboe/fio.git, then ./configure && make && sudo make install. Most readers should stick with the packaged version.

Basic fio Command Syntax and Key Parameters

Every fio job is described by a job file or by inline options on the command line. The command-line form looks like this:

fio --name=mytest --filename=/tmp/fiotest.bin --rw=randread --bs=4k --size=2G --runtime=30 --time_based --ioengine=libaio --iodepth=32 --direct=1 --numjobs=1 --group_reporting

That single line runs a 30-second random read test against a 2 GB test file, bypassing the page cache, using the asynchronous ioengine libaio. Most of the time you will only change a handful of flags:

–rw= the workload pattern. Common values are read, write, randread, randwrite, randrw (mixed), and rw (sequential read then write).

–bs= or –blocksize= the size of each I/O. 4k is realistic for databases, 64k and larger are realistic for streaming workloads.

–ioengine= how fio issues I/O. libaio is the right choice on Linux for async I/O; psync and sync are simpler but slower.

–iodepth= how many outstanding I/O requests fio keeps in flight. Higher iodepth means more queue pressure.

–numjobs= how many worker processes to fork. numjobs=4 with iodepth=8 produces the same load as numjobs=1 with iodepth=32, but behaves differently under schedulers.

–direct=1 bypasses the OS page cache so you measure the device, not RAM.

–runtime= and –time_based force fio to run for a fixed duration instead of stopping when the file is exhausted. time_based is essential for repeatable results on systems with very fast disks.

–group_reporting aggregates statistics across all numjobs into one neat summary instead of one report per worker.

A note about job files

When you outgrow one-liners, fio’s job files (INI format) become easier to read. Save a file such as random-read.fio and run it with fio random-read.fio. Most serious benchmarking suites you find online are shipped as job files.

Hardware Verification Before Testing

Garbage in, garbage out. Before you trust an fio number, confirm the VM is actually pointed at the storage you think it is.

From inside the guest, run:

lsblk (to see block devices and their mount points)

cat /sys/block/vda/queue/rotational (0 means SSD/NVMe, 1 means spinning disk)

cat /sys/block/vda/queue/scheduler (none or mq-deadline are fine; cfq has been removed from modern kernels)

df -hT (to confirm filesystem type and available space)

Never benchmark the root filesystem while it is in active use. A dedicated test file or, better, a dedicated virtual disk keeps the test isolated and protects the OS from running out of space.

Allocate at least twice as much free space as your --size value, plus room for the page cache if you skip --direct=1. fio does not truncate between runs unless you tell it to.

Direct I/O vs Buffered I/O

This is the single most common source of “fio numbers that look too good”. When you forget --direct=1, fio writes into the kernel page cache instead of the device. The first run may show 50 GB/s because you are benchmarking RAM, not disk.

Use --direct=1 when you want to measure what an application with O_DIRECT semantics sees, which is most databases (PostgreSQL, MySQL, MongoDB) and most modern filesystems. Use buffered I/O (omit direct) when you want to model warm-cache performance of a desktop workload, where reads come from RAM after the first pass.

Filesystems also matter. ext4 and xfs both support O_DIRECT cleanly. ZFS and btrfs require extra care: ZFS has its own ARC cache, and btrfs may write to CoW metadata that confuses simple --size=2G jobs. Run with a separate dataset or virtual disk for cleanest numbers.

Sequential Read and Write Benchmark Examples

Sequential tests model large file copies, video streaming, and backup workloads. They mostly stress throughput, not IOPS.

Sequential read, 1 GB test file, 4k block size, 30 seconds:

fio --name=seq_read --filename=/mnt/data/seqtest.bin --rw=read --bs=1M --size=1G --numjobs=1 --ioengine=libaio --iodepth=16 --direct=1 --runtime=30 --time_based --group_reporting

Sequential write, same file:

fio --name=seq_write --filename=/mnt/data/seqtest.bin --rw=write --bs=1M --size=1G --numjobs=1 --ioengine=libaio --iodepth=16 --direct=1 --runtime=30 --time_based --group_reporting

A 1 MB block size is overkill for storage IOPs but realistic for streaming. If you are sizing how fast you can ingest a database dump, bump --bs to 1M. If you are sizing log writes, 64k or 128k is a fairer reflection of reality.

Random Read and Write Benchmark Examples

Random tests stress IOPS and latency, which is what database and virtualization workloads usually care about most.

Random read, 4k block size, deep queue:

fio --name=rand_read --filename=/mnt/data/randtest.bin --rw=randread --bs=4k --size=4G --numjobs=1 --ioengine=libaio --iodepth=32 --direct=1 --runtime=60 --time_based --group_reporting

Random write with fsync (mimics a database commit):

fio --name=sync_randwrite --filename=/mnt/data/syncrand.bin --rw=randwrite --bs=4k --size=4G --numjobs=1 --ioengine=libaio --iodepth=1 --direct=1 --sync=1 --runtime=60 --time_based --group_reporting

Adding --sync=1 forces every write to call fsync() before being counted as complete, which matches PostgreSQL’s WAL behavior and many key-value store defaults. Single-digit millisecond latency on this test is a healthy sign.

Mixed 70/30 read/write (typical OLTP read-heavy):

fio --name=mixed --filename=/mnt/data/mixed.bin --rw=randrw --rwmixread=70 --bs=4k --size=4G --numjobs=1 --ioengine=libaio --iodepth=16 --direct=1 --runtime=60 --time_based --group_reporting

How to Interpret fio Output (IOPS, Throughput, Latency)

A short fio run produces a wall of text, but you really only need three sections. The key metrics live near the bottom of the report under “READ” and “WRITE”.

IOPS (I/O Operations Per Second): the count of completed read or write operations per second. For random 4K workloads this is the headline number. Modern cloud NVMe often reports 50k to 200k read IOPS inside a properly tuned VM; SATA SSDs sit between 20k and 80k; spinning disks at 4k random read are lucky to break 200 IOPS.

Throughput (BW): reported in KiB/s, MiB/s, or KB/s. This is bandwidth, useful for sequential workloads. A single NVMe can easily exceed 1 GiB/s for sequential reads; a SATA SSD tops out around 550 MB/s.

Latency (clat, lat): the time from when fio issues an I/O to when the kernel confirms completion. fio prints mean, p50, p99, p99.9, p99.99, and max. Pay attention to p99 and p99.9. A mean of 1 ms with p99 of 50 ms means most operations are fast but a long tail will wreck a database.

slat (submission latency): how long it takes the kernel to submit the I/O. Usually microseconds.

clat (completion latency): how long between submission and completion. This is what users feel.

For a healthy VM disk in 2026, look for clat p99 under 10 ms for read workloads and under 20 ms for write workloads on cloud SSDs. NVMe on local storage should sit under 1 ms for p99.

VM-Specific Considerations and Pitfalls

Benchmarking inside a VM is not the same as benchmarking bare metal. The virtualization layer adds overhead, throttles I/O, and sometimes lies about what it is doing.

Virtio vs emulated controllers: Virtio-blk or virtio-scsi (with single-queue virtio-scsi and the iothread option) is the fastest path. Avoid the default IDE or SATA emulation unless you are testing a legacy migration; they add extra translation layers and slow everything down.

Queue configuration: On KVM/QEMU, modern virtio-scsi and virtio-blk support multiple queues. Set num-queues=N on the host to match the number of vCPUs the guest is allowed to use, otherwise you bottleneck on a single submission queue no matter how many CPUs fio spawns.

Caching modes: In Proxmox, libvirt, or OpenStack, the disk can be configured as cache=none, cache=writeback, cache=writethrough, or cache=directsync. For benchmarking you want cache=none so the host does not silently buffer I/O. cache=writeback can double your write numbers but they are not real.

Hyperconverged stacks: Platforms like Nutanix require load-balancing to be enabled on VM attachments before fio results become reproducible. The Nutanix KB calls this out explicitly.

Cloud disk quotas: On AWS gp3, Azure Premium SSD, and Google Compute persistent disks, the cloud provider throttles you to whatever IOPS and throughput tier you bought. Your fio run will clamp to that ceiling, which is actually a useful test in itself; you can validate that you are getting what you paid for.

VM vs bare-metal gap: Expect a 5 to 20 percent overhead inside a well-tuned VM compared to bare metal, due to scheduler hops and queue contention. If your VM numbers are far worse than that, suspect host contention, balloon drivers (VMware), or throttling from other tenants.

Common fio Mistakes Inside VMs and How to Avoid Them

These are the issues I see in forum threads and on our own internal mailing lists every quarter.

Forgetting –time_based. Without it, fio stops as soon as it writes 1 GB. On a fast NVMe that means the test runs for three seconds, then declares victory. Always add --runtime=30 --time_based for a stable result.

Running too small. A 100 MB test file fits in the host’s page cache on many systems and skews results toward RAM. Use at least 2x RAM, and ideally much more, on the disk you are testing.

Mixing buffered and direct runs. If you compare a buffered fio run from yesterday with a direct=1 run from today, the numbers do not mean the same thing. Pick a mode and stick to it.

Ignoring OOM in memory-constrained VMs. fio with huge --size and many --numjobs allocates memory aggressively. In a 512 MB VM with --numjobs=8 and --size=10G you can hang the guest. Scale the workload to the VM size.

Not warming the disk. The first fio run on a fresh SSD writes to fresh blocks and can be artificially fast. Two or three short warm-up runs before the measurement run give you numbers you can defend.

Comparing different block sizes. A 4k random read result and a 1M sequential read result measure different things. When you write up a comparison, match bs, rw, iodepth, and numjobs across every row.

Automating fio With a Reusable Script

Once you have run a few tests by hand, you will want a script. Save the following as vm-storage-bench.sh, make it executable with chmod +x, and call it any time you want a quick look at a new VM disk.

#!/usr/bin/env bash
# vm-storage-bench.sh - quick fio sweep against $TARGET
set -euo pipefail
TARGET="${1:-/mnt/data/fiotest.bin}"
SIZE="${2:-4G}"
RUNTIME="${3:-30}"
echo "Target: $TARGET Size: $SIZE Runtime per job: ${RUNTIME}s"

for rw in read write randread randwrite; do
fio --name="vm_$rw"
--filename="$TARGET"
--rw="$rw"
--bs=4k
--size="$SIZE"
--numjobs=1
--ioengine=libaio
--iodepth=32
--direct=1
--runtime="$RUNTIME"
--time_based
--group_reporting
done

Run ./vm-storage-bench.sh /mnt/data/fiotest.bin 4G 30 and you get sequential read, sequential write, random read, and random write in one shot. Pipe the output through tee results-$(date +%F).log and you have a baseline you can compare against later when storage changes.

Frequently Asked Questions

Wrapping Up: Benchmark Storage Performance Inside a VM With fio

You now have a reliable workflow to benchmark storage performance inside a VM with fio on any Linux guest. Install fio, verify the disk, run a sequential and a random test with direct=1, scan the IOPS, throughput, and p99 latency lines, and keep a log of baseline numbers to compare against later.

Pick two more scripts to write this week: a database-shaped job file (8K random 70/30 with sync=1) and a streaming-shaped job file (1 MB sequential reads). Once you have those, every new VM, cloud disk, or storage tier you test will produce a clean number you can defend in a capacity review, instead of a guess based on a single dd run.