Replace a Failed Drive in a Degraded ZFS Pool (September 2026)

When ZFS flags a pool as DEGRADED, you are looking at a warning, not a catastrophe — yet. Your data is still accessible, but you have lost a layer of redundancy and every minute counts. I have replaced dozens of failed disks across Proxmox, Ubuntu, and TrueNAS servers over the years, and the procedure is remarkably consistent once you understand the pattern.

This guide walks through how to diagnose and replace a failed drive in a degraded ZFS pool from start to finish. You will learn how to read zpool status output, identify the physical disk behind the failure label, swap it out, and confirm the pool is healthy again. I will also cover Proxmox boot-pool specifics, common errors that trip people up, and the best practices I follow every time.

Whether you are running a homelab NAS or a production server, the steps below apply. The commands are the same on Linux, FreeBSD, and Proxmox VE.

Understanding ZFS Pool Health States: FAULTED, REMOVED, and DEGRADED

ZFS reports pool health at two levels: the pool itself and each individual vdev. The pool inherits the worst state among its vdevs, so a single degraded mirror drags the entire pool status down.

Here is how to interpret the three states you will encounter during a drive failure:

DEGRADED means one or more disks in a redundant vdev have failed, but the pool is still operating. Data is accessible through the surviving disks. This is the most common state and your window to act before a second failure causes data loss.

FAULTED is more serious. The vdev has lost enough disks that it can no longer guarantee data integrity. For a mirror, this means all mirrors are down. For RAIDZ1, it means two disks have failed. A faulted pool is typically not mountable and requires immediate recovery action.

UNAVAIL (sometimes shown as REMOVED) means the device is not present at all. The system cannot communicate with it. This can happen if a drive is physically removed, a cable disconnects, or the controller fails.

The distinction matters because your recovery path changes. A DEGRADED pool lets you run zpool replace while it serves data. A FAULTED pool may require booting from alternate media or importing in read-only mode.

To check your current pool state, run:

zpool status

The output shows each pool, its vdevs, and individual disk states. Look for anything that says DEGRADED, FAULTED, or UNAVAIL.

Identifying the Failed Disk with zpool status

Once you know the pool is degraded, the next step is pinpointing exactly which disk failed. ZFS does not always make this obvious because it reports devices by their system path, not by their physical bay location.

Run the verbose status command:

zpool status -v yourpool

In the output, look for a line like this under the vdev listing:

ata-WDC_WD40EFRX-68N32N0_WD-WCC7K4HXYZ-part1 DEGRADED 0 0 0

The device path shown here uses the /dev/disk/by-id/ naming convention, which includes the model, serial number, and partition suffix. This is far more reliable than /dev/sdb because device letters can change after a reboot.

If your pool uses /dev/sdX naming instead, I strongly recommend converting to by-id paths. But for now, note whatever path ZFS reports and map it to a physical drive.

To find the serial number for a given device, run:

lsblk -o NAME,SERIAL,MODEL,SIZE,TYPE

Match the serial number from lsblk output to the serial embedded in the by-id path. If the pool uses sdX naming, look up that device name in lsblk to find its serial.

For physical identification, many server chassis have drive bay LEDs you can blink:

ledctl locate=/dev/sdb

On systems without LED support, use the serial number from smartctl -a /dev/sdb and match it to the printed label on the physical drive.

Also check the error counters. The zpool status -v output shows READ, WRITE, and CKSUM error counts per disk. A disk with thousands of checksum errors is failing even if ZFS has not marked it DEGRADED yet.

Hot-Swap vs Shutdown: Choosing the Right Replacement Path

The most common question I hear is whether you can swap a disk while the server is running. The answer depends on your hardware.

If your server chassis has hot-swap bays with a supported SAS or SATA controller, you can pull the failed disk and insert the replacement without shutting down. This is standard on enterprise servers from Dell, HPE, and Supermicro. The controller handles the electrical disconnect cleanly.

For consumer-grade SATA connections on a motherboard, hot-swap may work if the port supports it in your BIOS or UEFI settings. Enable “Aggressive Link Power Management” or the hot-plug option for the SATA port. I have done this successfully on many consumer boards, but it is not guaranteed.

If your drives are in a standard desktop case with SATA cables, shutting down is safer. Powering off prevents accidental damage to connectors and avoids the risk of a momentary electrical spike corrupting data on adjacent ports.

For Proxmox boot-pool disks, a shutdown is almost always required because boot disks are typically connected directly to the motherboard rather than through a hot-swap backplane.

Taking the Disk Offline Before Removal

This is the step most people skip, and skipping it is a mistake. Before physically removing a disk, you should tell ZFS to take it offline. This prevents the filesystem from attempting I/O operations on a device that is about to disappear, which can generate error storms and slow the pool.

Run the offline command using the same device path ZFS reports:

zpool offline yourpool ata-WDC_WD40EFRX-68N32N0_WD-WCC7K4HXYZ

Note that you drop the partition suffix (the -part1 part) for this command. ZFS knows which partitions belong to the device.

Verify the offline state:

zpool status yourpool

The disk should now show as OFFLINE rather than DEGRADED or UNAVAIL. Once you see that confirmation, it is safe to proceed with physical removal.

If the disk is already UNAVAIL because it died completely, you can skip this step. ZFS already knows it is gone.

Physical Drive Replacement Procedure

Now physically swap the failed disk for the replacement. Before you do, confirm the new disk meets the requirements.

The replacement disk must be at least as large as the failed disk. ZFS is strict about this. Even a 1MB difference can cause the replacement to fail if the new disk is smaller. If you are replacing with a larger disk, that works fine — ZFS will only use the amount needed to match the vdev, and you can grow the pool later.

Check the new disk size:

lsblk -o NAME,SIZE /dev/sdX

If this is a boot-pool disk (Proxmox rpool), you also need to replicate the partition table from a surviving disk. For standard data pools, ZFS handles partitioning automatically during the replace operation.

For boot pools, replicate partitions using sgdisk:

sgdisk -R /dev/sdX /dev/sdY

Where sdX is the new disk and sdY is a healthy boot disk. Then randomize the new disk’s GUID:

sgdisk -G /dev/sdX

Do not format the disk or create filesystems on it. ZFS manages its own partitions and does not need you to pre-format anything.

Using zpool replace to Attach the New Disk

This is the core command of the entire procedure. The zpool replace command tells ZFS to copy data from the surviving disks onto the new disk and then bring the new disk into the vdev.

Here is the basic syntax:

zpool replace yourpool ata-OLD_DISK_SERIAL ata-NEW_DISK_SERIAL

Replace the old device path with the one ZFS reported as failed, and the new device path with the replacement disk’s by-id identifier.

To find the new disk’s by-id path, run:

ls -la /dev/disk/by-id/ | grep sdX

You will often need the -f flag to force the replacement, especially if ZFS sees leftover partition data on the new disk:

zpool replace -f yourpool ata-OLD_DISK_SERIAL ata-NEW_DISK_SERIAL

The -f flag tells ZFS to ignore EFI labels or stale partition data that might confuse it. I use it routinely because it eliminates a common failure point.

For a mirror vdev, you can alternatively use zpool attach if you are adding a disk to a single-disk pool. But for replacing a failed disk in an existing mirror or RAIDZ, always use zpool replace. Using zpool attach adds a new disk to the vdev rather than replacing the failed one, which can change your topology.

For RAIDZ vdevs, the command is the same. RAIDZ1, RAIDZ2, and RAIDZ3 all use zpool replace identically. The difference is in resilver time and redundancy level during the process.

If autoreplace is enabled on the pool, ZFS may automatically detect the new disk and begin the replacement without a command. You can check this setting:

zpool get autoreplace yourpool

Enable it with zpool set autoreplace=on yourpool if you want future replacements to be automatic.

Monitoring the Resilver Process

Once the replace command succeeds, ZFS begins resilvering. Resilvering is the process of rebuilding data on the new disk by copying it from the surviving members of the vdev. It is more than a simple copy — ZFS validates checksums during the process, effectively performing an integrity scan.

Monitor progress with:

zpool status yourpool

The output will show a progress line like:

scan: resilver (draid) in progress since Wed Aug 12 09:14:32 2026
1.2T scanned at 500M/s, 800G issued at 350M/s
1.2T total (100% done), 0 days 00:45:12 to go

Resilver time varies widely based on pool size, disk speed, and system load. For a small SSD pool, expect minutes to an hour. For a 20TB spinning-disk RAIDZ2 array, plan for 12 to 24 hours or more.

I check progress every 30 minutes during the first hour to make sure it is advancing smoothly. If the time-to-completion keeps growing instead of shrinking, something is wrong — usually another disk is failing or the system is under heavy I/O load.

During resilvering, try to minimize unnecessary I/O on the pool. Heavy reads and writes compete with the resilver and extend completion time. If possible, schedule the replacement during a low-traffic window.

Do not reboot the system during resilvering unless absolutely necessary. ZFS can resume resilvering after a reboot, but it adds time and risk.

Once resilvering completes, the status output will show:

scan: resilver (draid) completed after 0 days 02:15:00 with 0 errors

Zero errors is what you want to see. Any errors here warrant further investigation.

Verifying Pool Health After Replacement

After resilvering completes, verify that the pool has returned to a fully healthy state. Run:

zpool status yourpool

The pool state should now show ONLINE with no DEGRADED indicators. Every disk in every vdev should show ONLINE.

Check the error counters one more time. Ideally, all READ, WRITE, and CKSUM values should be zero or at least not climbing. If the new disk shows checksum errors immediately, it may be defective or the SATA cable may be faulty.

Clear stale error counters from the old failed disk:

zpool clear yourpool

This resets the error counters without affecting data. It is good practice to clear after a successful replacement so future errors stand out clearly.

Now run a scrub to verify data integrity across the entire pool:

zpool scrub yourpool

A scrub reads every block and verifies checksums. It is the definitive confirmation that your data is intact after the replacement. For a large pool, this can take hours or days. Schedule it during off-peak time.

Check scrub progress:

zpool status yourpool

Look for a line like scan: scrub in progress or scan: scrub repaired 0 errors after completion.

Special Case: Replacing a Disk in a Proxmox Boot Pool (rpool)

Proxmox VE installs ZFS on the boot disks by default, creating a pool called rpool. Replacing a disk in this pool is more complex than a standard data pool because the disk also contains boot partitions, EFI system partitions, and a bootloader.

The basic zpool replace procedure still applies, but you must also replicate the boot structure onto the new disk.

After physically installing the new disk and replicating partitions with sgdisk (as described earlier), run the replace command:

zpool replace -f rpool /dev/disk/by-id/ata-NEW_DISK-part3

Notice the -part3 suffix. Proxmox rpool uses partition 3 for the ZFS member, while partitions 1 and 2 hold the BIOS boot and EFI system partitions.

After the ZFS replace completes, set up the bootloader on the new disk. First, check which bootloader your system uses:

proxmox-boot-tool status

This command lists each boot disk and whether it uses systemd-boot or GRUB. The output looks something like this:

System currently booted with: bios
The configuration is valid and is being used.
hd0 (gpt)
/dev/disk/by-id/ata-DISK1-part2 is configured as an ESP
/dev/disk/by-id/ata-DISK2-part2 is configured as an ESP

If the new disk is missing from this list, format and initialize it:

proxmox-boot-tool format /dev/sdX

proxmox-boot-tool init /dev/sdX

This sets up the EFI system partition and installs the bootloader. Without this step, the system will not boot from the new disk if the surviving boot disk also fails.

Verify the new disk appears in proxmox-boot-tool status after initialization. Both disks should show as configured ESP entries.

Finally, confirm rpool is healthy and resilvered using the standard zpool status rpool command. Boot pool disks are usually small and fast, so resilvering typically completes in minutes.

Troubleshooting Common ZFS Replace Errors

Here are the errors I encounter most frequently when replacing ZFS disks, and how to fix each one.

“cannot replace device with device” or “device already exists” — This happens when the new disk already has ZFS labels or partition data. Use the -f flag to force the replacement. If that fails, wipe the new disk entirely with wipefs -a /dev/sdX and sgdisk --zap-all /dev/sdX, then retry.

“no such device in pool” — You are referencing a device path that ZFS does not recognize as part of the pool. Double-check the exact path from zpool status. Remember that ZFS reports the path it originally used, which may differ from the current /dev/sdX assignment. Use by-id paths for consistency.

“cannot open pool: one or more devices is already unavailable” — The pool itself may be in a state where it cannot be modified. Try importing it read-only first, or check if a second disk has also failed. In multi-disk failures, address the additional failures before proceeding with replacement.

Replacement disk appears smaller than original — Even though the new disk has the same nominal capacity, it may have a slightly smaller actual sector count due to manufacturer differences. If the size difference is tiny, you can sometimes work around it by shrinking the partition slightly, but the cleanest fix is to use a disk with identical or larger capacity.

Resilvering fails or restarts repeatedly — This often indicates the new disk is defective or the SATA cable is bad. Run smartctl -a /dev/sdX on the new disk to check for reallocated sectors or other SMART warnings. Try a different SATA cable or port before assuming the disk is bad.

Pool still shows DEGRADED after successful replace — Clear the error counters with zpool clear yourpool. If it persists, run zpool status -v and check whether a different disk in the vdev now shows errors.

Device names changed after reboot — This is exactly why by-id paths matter. If your pool uses sdX naming, device letters can shuffle after a reboot, leaving ZFS confused. You can update the pool to use by-id paths, but this requires exporting and re-importing the pool with the correct -d flag.

Best Practices for ZFS Disk Replacement

Always use /dev/disk/by-id/ paths when creating and managing pools. This prevents the device-name-shuffle problem after reboots and makes identifying failed disks straightforward.

Set up monitoring so you learn about failures before they become critical. Tools like zed (ZFS Event Daemon) can send email or webhook alerts when a pool state changes. Proxmox includes this by default.

Keep a hot spare connected and configured. With autoreplace=on and a designated spare, ZFS can begin the replacement automatically when a failure is detected.

Schedule regular scrubs. Most admins recommend monthly for SSDs and weekly to monthly for spinning disks. Scrubs catch silent data corruption before it becomes a problem during a recovery scenario.

Document your pool topology and disk serial numbers before a failure happens. When a disk dies at 3 AM, you do not want to be figuring out which bay it lives in.

Frequently Asked Questions

Knowing how to diagnose and replace a failed drive in a degraded ZFS pool is a skill every system administrator should master before the first disk actually fails. The procedure breaks down into five phases: diagnose with zpool status, take the disk offline, physically swap it, run zpool replace, and verify with a scrub. Each step has its own pitfalls, but the pattern is the same whether you are running a 2-disk mirror or a 12-disk RAIDZ3 array.

The two most important habits I can recommend are using by-id device paths and running regular scrubs. Those two practices alone will save you from most of the headaches I see in forum posts and support channels. If you found this guide helpful, bookmark it — because the next drive failure is always a question of when, not if.