How to Fix a Proxmox Degraded ZFS Root Pool (September 2026)?

Nothing ruins a Monday morning faster than a Proxmox node that hangs at the boot splash screen, then drops you into a recovery shell because the rpool went degraded overnight. We have hit this exact scenario in our own homelab and on customer clusters, and the good news is that you can rebuild full redundancy on a running system without reinstalling Proxmox.

This guide walks through fixing a Proxmox degraded ZFS root pool from diagnosis to verification, drawing on real commands and forum threads from the Proxmox community. By the end, you will know exactly when to run zpool clear, when to run zpool replace, and how to keep both halves of your boot mirror healthy going forward.

What Causes a Proxmox Degraded ZFS Root Pool?

A ZFS pool becomes degraded when one or more virtual devices in a redundant configuration (mirror, RAID-Z, or RAID-Z2) stop responding. ZFS does not panic. Instead, it pulls the bad vdev out of service and keeps the pool online using whatever devices still work, while marking the pool state as DEGRADED.

Several real-world patterns drive this state on Proxmox nodes:

• Disk hardware failure. SSDs and SATA HDDs die in the field. SMART shows reallocated sectors, command timeouts, or just stops responding entirely. This is the most common reason an rpool goes degraded.

• Cable or backplane issues. A loose SATA cable or failing HBA port can make a perfectly healthy drive look faulted to ZFS. We have seen this on Supermicro chassis where one backplane connector oxidized.

• Transient I/O errors. Bad sectors, kernel I/O errors, or momentary host bus adapter freezes trigger a FAULTED state even though SMART reports the drive as healthy.

• Kernel or driver bugs. Some older kernels intermittently reset NVMe drives under load, which ZFS treats as a persistent error.

The boot impact is real. The boot pool is a separate, small ZFS mirror (often called rpool, sometimes bp, depending on installer version) that holds the root filesystem. When boot pool redundancy drops to one device, BIOS or UEFI firmware sometimes pauses for an unusually long time while probing the missing disk. Users on the Proxmox forum regularly report boot times stretching from 30 seconds to 15 minutes, exactly matching a failing boot drive.

Diagnosing the Failed Disk with zpool status

Before you swap anything, you need to confirm which disk actually failed. The single most useful command is zpool status -v.

Run it from a root shell. If your node booted into a recovery shell, type zpool import -f rpool first so you can read status data. On a normally running system, you can simply SSH in.

The output groups information into three blocks: pool-level state, scan activity (resilver or scrub), and per-vdev error counters. In a healthy boot mirror you see state: ONLINE and both disks listed as ONLINE. A degraded pool shows state: DEGRADED, one disk as ONLINE, and the failing disk as FAULTED or UNAVAIL.

Here is a representative output from the Proxmox forum that matches the symptom we are fixing:

  pool: rpool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications are unaffected.
action: Restore the file in question if possible.  Restore will
        be performed by resilver.
  scan: resilver in progress since ...
config:
        NAME                     STATE     READ WRITE CKSUM
        rpool                    DEGRADED       0     0     0
          mirror-0               DEGRADED       0     0     0
            sda3                 ONLINE         0     0     0
            sdb3                 FAULTED      187     0     0

The exact device name (sda3, sdb3, nvme0n1p3, or a by-id path) matters. Note it down. You will also see a column of read, write, and checksum error counts. Those numbers increasing on one device are a strong signal that the device is failing, even when another tool says otherwise.

Cross-check with SMART to differentiate hardware failure from a transient error:

• smartctl -a /dev/sdb for the suspect disk. Look for reallocated sectors, pending sectors, and UDMA CRC errors. Non-zero reallocated sectors usually mean the drive needs replacement.

• smartctl -t long /dev/sdb followed by smartctl -l selftest /dev/sdb after the test finishes confirms mechanical issues.

• dmesg | grep -i error on the host surfaces kernel-level I/O complaints.

Prerequisites Before You Start

Trying to repair a degraded ZFS pool without preparation tends to produce a worse problem. Before touching anything, gather these items.

A replacement disk of equal or larger size. For a mirror, both vdevs must be the same size or larger. A smaller replacement will refuse to attach. We recommend enterprise-grade SSDs for boot pools because consumer drives trigger more SMART errors under sustained write load.

A verified backup of VMs and config. Test that vzdump backups to your NFS, CIFS, or PBS storage are current and restorable. If the system loses the remaining mirror vdev mid-repair, you want a clean restore path.

Live boot media. Have a Proxmox ISO on a USB stick. Even though the typical repair runs on the live system, you may need to boot from the installer environment to import rpool manually if the node refuses to mount the root filesystem.

Out-of-band management. IPMI, iLO, or iDRAC access saves you from racing to a data center when the boot stalls. It also lets you watch the boot screen if the kernel console is unreliable over serial.

Time on your side. Resilvering a boot pool on SSDs typically completes in 15 to 45 minutes per drive. Plan the maintenance window accordingly and avoid scrubbing or heavy writes during rebuild.

Step-by-Step: Fixing a Proxmox Degraded ZFS Root Pool

Now to the main recovery workflow. These steps assume a two-disk ZFS mirror as the boot pool, which is the standard Proxmox install pattern for UEFI systems.

Step 1: Note the IDs and device names of the failed drive

From zpool status -v, capture the by-id path. By-id paths survive reboots and are the preferred identifier for ZFS. Example: /dev/disk/by-id/ata-Samsung_SSD_870_EVO_500GB_S6PXNG0R400512A-part3.

Note the partition number too. Proxmox creates partition 3 as the ZFS member. Do not try to replace the whole disk. ZFS only manages the partition.

Step 2: Swap the physical drive

Shut down cleanly with shutdown -h now. Pull power. Replace the failed disk. Hot-swap bays work on most server chassis, but if your server does not advertise hot-swap support, a cold swap is the safer choice.

Power back on and confirm the new disk appears with lsblk or ls -la /dev/disk/by-id/. If the new disk does not show up, check cabling and backplane seating before moving on.

Step 3: Replicate the partition table

ZFS expects the new disk to have an identical partition layout. Use sgdisk to copy from the healthy partner:

sgdisk --replicate=/dev/sdb /dev/sda
sgdisk --randomize-guids /dev/sdb

--replicate copies the MBR or GPT structure. --randomize-guids generates fresh unique GUIDs so the new disk does not collide with the surviving one. Without that randomization, ZFS gets confused about which partition is which after reboot.

Step 4: Add the new partition to ZFS

Find the new partition’s by-id path. Then run the replace command:

zpool replace rpool /dev/disk/by-id/<old-faulted-id>-part3 /dev/disk/by-id/<new-disk-id>-part3

You may need -f if the new disk has filesystem signatures from prior use. The replace operation copies all blocks from the surviving vdev onto the new disk in the background. This is the resilver.

Step 5: Wait for the resilver to complete

Watch progress with zpool status -v rpool. You will see a scan: resilver in progress line with percentage and ETA. For a 500 GB boot pool on SSDs, expect 15 to 45 minutes.

Resilvering is I/O intensive. Avoid heavy workloads on the host. ZFS pauses scrub during resilver, which is fine for a boot pool. Do not unplug or reboot the new disk during this phase; corruption can occur.

Step 6: Refresh boot partitions with proxmox-boot-tool

Proxmox VE uses ESP partitions (EFI System Partition) on each mirror vdev to keep the boot loader chain in sync. After a replace, you must refresh those:

proxmox-boot-tool status
proxmox-boot-tool init /dev/disk/by-id/<new-disk-id>
proxmox-boot-tool refresh

The init subcommand formats and sets up a fresh ESP on the new drive if missing. refresh copies the current kernel and initrd from the running system onto every ESP so each disk can boot independently.

Step 7: Verify and reboot

Before rebooting, run:

• zpool status -v rpool — confirm both disks list ONLINE and the resilver completed successfully.

• proxmox-boot-tool status — confirm both ESPs are present and synced.

• zpool scrub rpool is optional but recommended after a hardware swap to validate checksums.

Then reboot. Watch the boot screen for normal boot time. If the system still stalls at the splash screen, the ESP on the new disk was not refreshed correctly, and you need to revisit Step 6.

Step 8: Clean up dangling UUID references

After the replace, ZFS occasionally remembers the old failed disk by its old GUID. Wipe those lingering entries with:

zpool clear rpool

Only run this if the new disk is ONLINE and the resilver succeeded. Otherwise you can mask a real fault.

When to Use zpool clear Instead of zpool replace

Not every degraded state demands a physical replacement. Knowing the difference saves hours of work, and you can avoid swapping a perfectly good drive.

Use zpool clear when:

• The disk shows up as healthy in SMART, with zero reallocated or pending sectors.

• The error counters on the FAULTED device are still low (single digits).

• The fault appeared immediately after a transient event: power blip, kernel panic, controller reset.

• You can read the partition contents successfully with dd or zdb -l /dev/sdX.

Use zpool replace when:

• SMART shows reallocated or pending sectors above zero.

• The kernel logs report I/O errors like ataX: soft reset failed or blk_update_request: I/O error repeatedly on the same device.

• The drive fails an extended self-test (smartctl -t long).

• Errors recur after zpool clear. That pattern is the most reliable indicator that the hardware is dying.

From the Proxmox forum, one sysadmin saw a pool flip between DEGRADED on reboot and ONLINE after a power cycle. ZFS clears the fault automatically during the import scan, but the underlying weakness remains. Run a scrub and SMART long test to confirm before you trust zpool clear alone.

Boot Repair: GRUB vs systemd-Boot Considerations

Modern Proxmox installations (8.x and later) ship with systemd-boot when installed on UEFI. Older installs and BIOS-mode setups use GRUB. The tool you need depends on which boot loader is active.

To detect, run:

• efibootmgr -v — if you see an entry pointing to EFIsystemdsystemd-bootx64.efi, you have systemd-boot.

• ls /boot/grub/ — if it returns grub.cfg, you are on GRUB.

For systemd-boot hosts, proxmox-boot-tool is mandatory for keeping both halves of the mirror bootable. For GRUB hosts, grub-install /dev/sdX on each mirror disk achieves the same goal.

Mixing the two is a common failure mode. We have seen nodes where one half of the mirror is GRUB-bootable and the other is systemd-boot, so the system only boots from a specific disk. Standardizing boot loaders and running proxmox-boot-tool refresh after each disk swap prevents this trap.

Verifying the Fix and Preventing Future Degradation

A repaired mirror is not the end of the story. Pool health is a continuous property and needs monitoring.

Schedule a monthly ZFS scrub. A scrub walks every block and verifies checksums against redundancy. It catches silent corruption that no SMART scan will detect. Add it through the Proxmox web interface under Datacenter, Options, Scrub, or via systemd timer.

Set up SMART monitoring. Install smartmontools and run smartctl -H /dev/sdX nightly via cron or a ZED (ZFS Event Daemon) rule. Pipe failing results into your alerting system.

Watch the kernel log. ZED posts alerts to /var/log/syslog for checksum errors, device removals, and resilver completion. Forward syslog to a central aggregator so you get notified before the pool becomes critical.

Replace disks proactively. When SMART shows reallocated sectors approaching threshold, swap the drive. Waiting for failure costs you the redundancy window during replacement.

Keep BIOS and firmware current. Many NVMe stability fixes ship in firmware updates. Vendor tools like Samsung Magician or Intel MAS make this easy.

Frequently Asked Questions

Final Thoughts

Fixing a Proxmox degraded ZFS root pool is one of those tasks you only need to learn once, because the steps transfer between servers, SSDs, and even different disk capacities. The combination of zpool status -v for diagnosis, sgdisk --replicate for partition cloning, and proxmox-boot-tool refresh for boot mirror upkeep covers roughly 90 percent of real-world cases.

Make a checklist out of this guide, save it to your runbook, and run an annual drill on a test node. When the next drive fails, you will diagnose, replace, and resilver in under an hour with zero data loss.