Multi-Disk ZFS Pools — mirrors, RAIDZ, and special vdevs

ZFS pools are built from one or more vdevs (virtual devices). Each vdev is a redundancy group — it can be a single disk, a mirror, a RAIDZ stripe, or a special device. The pool stripes data across all top-level vdevs for performance. Choosing the right topology is the most important storage decision you will make, and it cannot be changed after creation.

Pool topology in the kldload installer

The kldload web UI shows a pool topology selector when you have more than one disk available. Select your disks, then choose the vdev type. The installer validates that you have enough disks for the topology you chose before proceeding.

Topology options in the UI:

• Single disk — no redundancy, maximum capacity
• Mirror (2+ disks) — all data mirrored, survives all-but-one disk failing
• RAIDZ1 (3+ disks) — 1 parity disk, survives 1 failure per vdev
• RAIDZ2 (4+ disks) — 2 parity disks, survives 2 failures per vdev
• RAIDZ3 (5+ disks) — 3 parity disks, survives 3 failures per vdev

You can also enter a custom zpool create command in the advanced section if you need multi-vdev layouts (e.g., two 3-way mirrors striped together).

Vdev types — when to use each

Creating pools — real commands

# 2-disk mirror (recommended for most setups)
zpool create -o ashift=12 \
  -O compression=zstd -O atime=off \
  rpool mirror /dev/disk/by-id/ata-WD_Blue_1TB_AA \
               /dev/disk/by-id/ata-WD_Blue_1TB_BB

# 4-disk RAIDZ2 (balanced capacity + safety for bulk storage)
zpool create -o ashift=12 \
  -O compression=zstd -O atime=off \
  tank raidz2 \
    /dev/disk/by-id/ata-WD_Red_4TB_AA \
    /dev/disk/by-id/ata-WD_Red_4TB_BB \
    /dev/disk/by-id/ata-WD_Red_4TB_CC \
    /dev/disk/by-id/ata-WD_Red_4TB_DD

# Two 3-disk RAIDZ1 vdevs — more capacity than mirrors, stripes for throughput
zpool create -o ashift=12 \
  -O compression=zstd -O atime=off \
  tank \
    raidz /dev/sdb /dev/sdc /dev/sdd \
    raidz /dev/sde /dev/sdf /dev/sdg

# Check pool status
zpool status -v tank

Always use /dev/disk/by-id/ paths rather than /dev/sdb style names. Device names are not stable across reboots and bus changes. By-id paths are tied to the physical drive's serial number and survive hot-plug, cable moves, and kernel updates.

Adding disks after install

You can grow a pool in two ways: add a new vdev (instant), or expand an existing RAIDZ vdev (slow, online, OpenZFS 2.2+). The recommended path is always adding a new vdev of the same type.

# Add a new mirror vdev to an existing pool (pool grows immediately)
zpool add rpool mirror /dev/disk/by-id/new-disk-AA \
                       /dev/disk/by-id/new-disk-BB

# Add a single disk (no redundancy — adds a stripe vdev, rarely a good idea)
zpool add tank /dev/disk/by-id/new-disk-CC

# Add a RAIDZ1 vdev to an existing pool
zpool add tank raidz /dev/sde /dev/sdf /dev/sdg

# Verify the new vdev appeared
zpool status tank

Expected output after adding a mirror vdev:

  pool: rpool
 state: ONLINE
config:

        NAME                      STATE     READ WRITE CKSUM
        rpool                     ONLINE       0     0     0
          mirror-0                ONLINE       0     0     0
            ata-WD_Blue_1TB_AA    ONLINE       0     0     0
            ata-WD_Blue_1TB_BB    ONLINE       0     0     0
          mirror-1                ONLINE       0     0     0
            new-disk-AA           ONLINE       0     0     0
            new-disk-BB           ONLINE       0     0     0

Warning: RAIDZ vdevs cannot be expanded by adding a disk to them in older ZFS versions. Adding to a RAIDZ vdev is only supported in OpenZFS 2.2+ and is a slow online operation (every block must be rewritten). If you are on an older version, the only way to grow a RAIDZ pool is to add a completely new RAIDZ vdev, or to back up, destroy, and recreate with more disks.

Adding a SLOG (ZFS Intent Log)

A SLOG (Separate Intent Log) accelerates synchronous writes by giving ZFS a fast place to commit write intent before it hits spinning rust. It does not cache reads. It is most effective when your workload has many small synchronous writes — databases, NFS with sync semantics, or any application that calls fsync() frequently.

# Add a SLOG device (single NVMe for a lab or home server)
zpool add rpool log /dev/nvme0n1p1

# Add a mirrored SLOG (required for production — a failed SLOG can lose data)
zpool add rpool log mirror /dev/nvme0n1p1 /dev/nvme1n1p1

# Verify
zpool status rpool | grep -A5 logs

# Remove a SLOG if it is failing (pool continues without it, writes fall back to main vdevs)
zpool remove rpool nvme0n1p1

Production note: A single SLOG that fails while power is lost can cause data loss for in-flight synchronous writes. Mirror your SLOG in production. On NVMe-based pools where the NVMe itself is the bottleneck, a SLOG provides little benefit — measure before adding complexity.

Adding an L2ARC (read cache)

An L2ARC is a second-level read cache behind the ARC (which lives in RAM). It is useful when your working set is larger than your RAM but smaller than your storage tier — common in media servers, large VM hosts, and database replicas.

# Add an L2ARC device
zpool add rpool cache /dev/nvme0n1p2

# Multiple L2ARC devices stripe automatically for higher throughput
zpool add rpool cache /dev/nvme0n1p2 /dev/nvme1n1p2

# Check L2ARC hit rate
arc_summary | grep -i l2

# Remove L2ARC (safe at any time — it's only a cache)
zpool remove rpool nvme0n1p2

Adding a special vdev (metadata acceleration)

A special vdev stores metadata — and optionally small files — on a fast device while bulk data stays on slower storage. This dramatically improves random IOPS on pools with many small files (Docker images, Git repos, email spools, large directory trees).

# Add a mirrored special vdev (always mirror special vdevs — metadata loss = pool loss)
zpool add tank special mirror /dev/nvme0n1p3 /dev/nvme1n1p3

# Store small files on the special vdev too (files <= 32K go to the fast device)
zfs set special_small_blocks=32K tank

# Verify the special vdev is active
zpool status tank

Warning: If a special vdev fails and is not mirrored, the pool is unrecoverable — metadata loss is fatal to ZFS. Always mirror special vdevs. Never use a single drive as a special vdev in production.

Replacing a failed disk

# Check which disk failed
zpool status rpool
# Look for FAULTED or REMOVED in the output

# Replace with a new disk (resilver begins immediately)
zpool replace rpool /dev/disk/by-id/old-failed-disk \
                    /dev/disk/by-id/new-replacement-disk

# Watch resilver progress
watch -n 2 zpool status rpool
# scan: resilver in progress since ...
#   3.72G resilvered, 12.4% done, 00:08:20 to go

Capacity reference by topology

Usable capacity for N identical disks of size D:

# Mirror (2 disks):     D × 1   =  50% efficiency
# Mirror (3 disks):     D × 1   =  33% efficiency
# RAIDZ1 (3 disks):     D × 2   =  67% efficiency
# RAIDZ1 (4 disks):     D × 3   =  75% efficiency
# RAIDZ2 (4 disks):     D × 2   =  50% efficiency
# RAIDZ2 (6 disks):     D × 4   =  67% efficiency
# RAIDZ3 (6 disks):     D × 3   =  50% efficiency
# RAIDZ3 (9 disks):     D × 6   =  67% efficiency

For most general-purpose servers with 4–8 disks, a pair of 3-way mirrors (or a 4-disk RAIDZ2) gives the best balance of capacity, performance, and safety. RAIDZ2 with 6 disks is a common NAS sweet spot: 67% efficiency with 2-fault tolerance.