Proxmox ZFS Tuning — stop blaming ZFS, start tuning it.

Proxmox VE ships with first-class ZFS support. You can install the hypervisor on ZFS, store VM disks on ZFS, replicate between cluster nodes with ZFS send/receive, and snapshot from the GUI. But the defaults are not optimized for virtualization workloads. People install Proxmox, create VMs on ZFS, performance is terrible, and they blame ZFS. The problem is not ZFS. The problem is that nobody tuned it. This page is the complete guide to making Proxmox and ZFS work together at full speed.

Proxmox VE + ZFS architecture

Proxmox VE is a Debian-based hypervisor that manages KVM virtual machines and LXC containers. When you choose ZFS as the storage backend, PVE uses two distinct storage modes:

The critical distinction: VM disks are zvols (block devices). Container rootfs are datasets (filesystems). They have completely different tuning requirements. Most Proxmox ZFS problems come from not understanding this split.

Creating ZFS pools in PVE — GUI vs CLI

Proxmox offers a GUI for creating ZFS pools under Datacenter → Storage → ZFS (or per-node under Disks → ZFS). The GUI creates pools with sane defaults, but it does not expose all options. For production pools, always use the CLI.

GUI pool creation (what you get)

The GUI lets you pick disks, choose RAID level (mirror, raidz1/2/3, single), set compression, and pick ashift. That is all. It does not let you set dnodesize, xattr, create special vdevs, add SLOG, or control volblocksize for future zvols. The pool name defaults to rpool for root or your chosen name for data pools.

CLI pool creation (what you should do)

# Create a production VM storage pool — 4 disks, 2 mirror pairs
zpool create -o ashift=12 \
  -O compression=lz4 \
  -O atime=off \
  -O xattr=sa \
  -O dnodesize=auto \
  vmpool \
  mirror /dev/sda /dev/sdb \
  mirror /dev/sdc /dev/sdd

# Create the data dataset that PVE expects
zfs create vmpool/data

# Set default volblocksize for new zvols created under this dataset
zfs set volblocksize=16K vmpool

Storage configuration — /etc/pve/storage.cfg

After creating a ZFS pool, you must register it with Proxmox so the GUI and API know about it. This is done in /etc/pve/storage.cfg, which is automatically synced across cluster nodes via pmxcfs (the Proxmox cluster filesystem).

# /etc/pve/storage.cfg — ZFS pool for VM disks
zfspool: vmpool
    pool vmpool/data
    content images,rootdir
    sparse 1
    blocksize 16k

# Explanation of each line:
# zfspool: vmpool        — storage ID (what PVE calls it)
# pool vmpool/data       — the ZFS dataset path
# content images,rootdir — allowed content types (VM disks + CT rootfs)
# sparse 1               — create thin-provisioned zvols (CRITICAL — see below)
# blocksize 16k          — volblocksize for new zvols (PVE 7.x+ only)

The content field controls what PVE allows on this storage:

Zvol-based VM disks vs dataset-based CT storage

This is the most important concept for ZFS on Proxmox. Understanding the difference between zvols and datasets determines whether your VMs fly or crawl.

VM disk formats on ZFS — raw zvols only, never qcow2

The correct answer is always raw zvols. ZFS already provides everything qcow2 was designed to add: copy-on-write, snapshots, thin provisioning, compression. Using qcow2 on ZFS duplicates all of that functionality at a massive performance cost.

Thin provisioning with zvols

By default, Proxmox can create zvols in two modes: thick (pre-allocated) or thin/sparse (allocate on write). Thin provisioning is controlled by the sparse flag in storage.cfg and the -s flag on zfs create.

# Thick zvol — 40GB allocated immediately, even if the VM uses 2GB
zfs create -V 40G rpool/data/vm-100-disk-0

# Thin/sparse zvol — 0 bytes allocated, grows as VM writes data
zfs create -V 40G -s rpool/data/vm-100-disk-0

# Check actual vs referenced space
zfs list -o name,volsize,used,refer rpool/data/vm-100-disk-0
# NAME                          VOLSIZE  USED   REFER
# rpool/data/vm-100-disk-0         40G   2.1G   2.0G

Always use thin provisioning (sparse 1 in storage.cfg). There is no performance penalty — ZFS allocates blocks on write either way. Thick provisioning just wastes pool space by reserving it upfront. The only edge case where thick makes sense is when you need guaranteed space reservation and cannot overcommit the pool.

The volmode property controls how the zvol appears to the system. Proxmox uses volmode=dev (the default), which exposes /dev/zvol/pool/name. You can also set volmode=full to expose partition tables, or volmode=none to hide the device entirely (useful for zvols managed only via iSCSI).

The write amplification problem

# For general VM workloads — 16K is the sweet spot
zfs create -V 40G -s \
    -o volblocksize=16K \
    -o compression=lz4 \
    rpool/data/vm-100-disk-0

# For database VMs (PostgreSQL, MySQL) — match the DB page size
zfs create -V 40G -s \
    -o volblocksize=8K \
    rpool/data/vm-100-disk-0

# Set the default for all future zvols created by PVE
# In /etc/pve/storage.cfg, add: blocksize 16k
# Or set on the parent dataset:
zfs set volblocksize=16K rpool/data

PVE snapshot integration

When you click "Snapshot" in the Proxmox GUI for a VM on ZFS storage, PVE does two things:

1. Creates a ZFS snapshot of each zvol attached to the VM: zfs snapshot rpool/data/vm-100-disk-0@snap_name
2. Saves the VM configuration (CPU, RAM, network) and optionally the RAM state to the snapshot metadata

ZFS snapshots are instantaneous and free until data diverges. A PVE snapshot of a 500GB VM takes less than a second and consumes zero additional space at creation time. As the VM writes new data, the snapshot holds references to the old blocks — only the delta consumes space.

# PVE creates snapshots like this internally:
zfs snapshot rpool/data/vm-100-disk-0@__replicate_100-0_1712345678
zfs snapshot rpool/data/vm-100-disk-1@__replicate_100-0_1712345678

# List all snapshots for a VM's disk
zfs list -t snapshot -r rpool/data/vm-100-disk-0

# Check snapshot space consumption
zfs list -t snapshot -o name,used,refer -r rpool/data/vm-100-disk-0

Key limitation: PVE snapshots include RAM state only if you check "Include RAM" in the GUI. Without RAM state, rolling back stops the VM first. With RAM state, the snapshot file is large (equal to VM RAM) and stored on the same ZFS dataset. For production, skip RAM snapshots and use ZFS-only snapshots for speed.

PVE replication — built-in ZFS send/receive

Proxmox has built-in replication that uses zfs send and zfs receive to copy VM disks between cluster nodes. This is one of PVE's best features for ZFS users. It gives you near-instant failover without shared storage.

# Create a replication job via CLI (replicate VM 100 to node pve2, every 15 min)
pvesr create-local-job 100-0 pve2 --schedule '*/15'

# List replication jobs
pvesr list

# Check replication status
pvesr status

# Manually trigger replication
pvesr run 100-0

# Under the hood, PVE runs something like:
# zfs send -i rpool/data/vm-100-disk-0@prev rpool/data/vm-100-disk-0@new | \
#   ssh pve2 zfs receive rpool/data/vm-100-disk-0

Backup interaction — vzdump + ZFS snapshots

PVE's backup tool (vzdump) has a special ZFS mode. When backing up a VM on ZFS storage, vzdump:

1. Creates a ZFS snapshot of each VM disk (instantaneous, no I/O freeze)
2. Reads from the snapshot to create the backup file (the VM keeps running normally)
3. Removes the temporary snapshot when the backup completes

This means ZFS backups have near-zero impact on VM performance. Unlike LVM-thin backups (which create an LVM snapshot that degrades write performance), the ZFS snapshot is copy-on-write and does not slow down the running VM. The backup reads from a frozen-in-time view while the VM continues writing to new blocks.

# Backup modes for ZFS VMs:
# snapshot — uses ZFS snapshot, no VM pause (RECOMMENDED)
# suspend  — pauses VM briefly, then snapshots
# stop     — stops VM, snapshots, restarts

# Configure backup job (Datacenter → Backup)
# Or via CLI:
vzdump 100 --mode snapshot --storage backup-pool --compress zstd

# The backup file lands as a .vma.zst file in the backup storage
# Restore:
qmrestore /path/to/vzdump-qemu-100-2026_04_04-12_00_00.vma.zst 100

ARC tuning for PVE — leave room for KVM/QEMU

The Adaptive Replacement Cache (ARC) is ZFS's in-memory read cache. By default, ZFS claims up to 50% of system RAM for ARC. On a dedicated file server, that is fine. On a Proxmox host where KVM/QEMU VMs also need RAM, ARC and VMs compete for the same memory.

The rule: Total VM RAM + ARC + OS overhead must not exceed physical RAM. If you overcommit, the kernel's OOM killer starts killing QEMU processes — your VMs crash randomly with no warning.

# Set ARC limits — example for 128GB host
# /etc/modprobe.d/zfs.conf (persists across reboots)
options zfs zfs_arc_max=17179869184
options zfs zfs_arc_min=4294967296

# Apply immediately without reboot
echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max
echo 4294967296 > /sys/module/zfs/parameters/zfs_arc_min

# Verify current ARC usage
arc_summary | head -30

# Or the quick way:
cat /proc/spl/kstat/zfs/arcstats | grep -E '^size|^c_max'

Swap on ZFS — the memory pressure problem

# The fix: put swap on a dedicated small partition, NOT on ZFS
# During Proxmox install, create a 4-8GB swap partition on the boot disk

# If your root is already on ZFS, create a swap file on tmpfs or
# a dedicated non-ZFS partition:
mkswap /dev/sda2
swapon /dev/sda2

# Add to /etc/fstab:
# /dev/sda2  none  swap  sw  0 0

# Or disable swap entirely and rely on proper memory sizing:
swapoff -a
sed -i '/swap/d' /etc/fstab

# On PVE with ZFS root, the installer creates a small swap partition
# automatically. Verify it is NOT on ZFS:
swapon --show
# NAME      TYPE      SIZE  USED  PRIO
# /dev/sda2 partition   8G    0B    -2     ← Good, this is a partition

I/O tuning for VMs — the properties that matter

# 16K — general purpose VMs (Windows, Linux desktop/server)
# Matches the common 4K guest block size with 4:1 amplification
# Good balance of performance and compression ratio
zfs create -V 100G -s -o volblocksize=16K rpool/data/vm-100-disk-0

# 8K — database VMs (PostgreSQL 8K pages, MySQL/InnoDB 16K pages)
# Minimal amplification for DB page-aligned I/O
zfs create -V 200G -s -o volblocksize=8K rpool/data/vm-200-disk-0

# 64K — bulk storage VMs (file servers, media servers)
# Better compression ratio, acceptable for sequential workloads
zfs create -V 500G -s -o volblocksize=64K rpool/data/vm-300-disk-0

# standard (default) — honor guest fsync() calls, safest option
zfs set sync=standard rpool/data/vm-100-disk-0

# disabled — ignore fsync(), all writes are async
# DANGEROUS: data loss on power failure. Only for throwaway test VMs.
zfs set sync=disabled rpool/data/vm-999-disk-0

# always — force every write to be synchronous
# Overkill for VMs. Only useful for NFS/iSCSI targets.

# latency (default) — optimize ZIL for low-latency commits
# Best for general VM workloads, databases, anything that calls fsync()
zfs set logbias=latency rpool/data/vm-100-disk-0

# throughput — optimize ZIL for large sequential sync writes
# Use for VMs that do large sequential writes (e.g., video processing)
# Writes go directly to the pool, bypassing the ZIL/SLOG
zfs set logbias=throughput rpool/data/vm-300-disk-0

# all (default) — cache both data and metadata in ARC
zfs set primarycache=all rpool/data/vm-100-disk-0

# metadata — only cache metadata, not data blocks
# Use for VMs with huge working sets that thrash ARC
# (e.g., a 2TB database VM on a host with 16GB ARC)
zfs set primarycache=metadata rpool/data/vm-200-disk-0

SLOG for VM workloads

Virtual machines are one of the most SLOG-friendly workloads. Every guest OS issues fsync() calls for journaling, package installs, database commits, and log writes. Without a SLOG, these sync writes wait for the transaction group to flush to the data disks (5 seconds by default, or whenever the pool is busy). With a SLOG, sync writes commit to the fast log device in microseconds and the VM continues immediately.

# Add a SLOG — mirrored enterprise NVMe with power loss protection
zpool add rpool log mirror /dev/nvme2n1p1 /dev/nvme3n1p1

# Verify SLOG is active
zpool status rpool
#   ...
#   logs
#     mirror-1    ONLINE
#       nvme2n1p1  ONLINE
#       nvme3n1p1  ONLINE

# Check if your workload benefits from SLOG — look at sync write queue
zpool iostat -q rpool 5
# If the syncq_write column is consistently > 0, SLOG helps.

# SLOG sizing: 16-32GB is plenty. The ZIL only holds ~10 seconds
# of sync write data. Even at 1GB/s sync write rate, 16GB is overkill.

Special vdev for VM metadata

When your data pool is on HDDs, every metadata operation (block pointer lookups, free space tracking, dedup tables) hits spinning rust. A mirrored SSD special vdev accelerates all metadata operations for the entire pool. For VM workloads, this means faster snapshot creation, faster zfs list, faster pool scrubs, and faster replication delta calculation.

# Add a mirrored special vdev (MUST be mirrored — losing it kills the pool)
zpool add rpool special mirror /dev/sda /dev/sdb

# Store small blocks (under 64K) on the special vdev too
# This catches VM config files, container metadata, small writes
zfs set special_small_blocks=65536 rpool

# The special vdev is most effective when:
# - Data pool is on HDDs (the speed gap between HDD metadata and SSD is huge)
# - You run many VMs/CTs (more metadata operations)
# - You use heavy snapshot/replication workflows (snapshot metadata is on special vdev)

On an all-NVMe pool, a special vdev provides minimal benefit — the data vdevs are already fast enough for metadata. Special vdevs shine when there is a large speed gap between the data disks and the special vdev disks.

Use mirrors, not RAIDZ, for VMs

# BAD for VMs — RAIDZ2 across 6 disks
# Every write touches all 6 disks. IOPS = ~1 disk worth.
# zpool create rpool raidz2 /dev/sd{a,b,c,d,e,f}

# GOOD for VMs — 3 mirror pairs
# Each pair handles I/O independently. IOPS = ~3 disks worth.
zpool create -o ashift=12 rpool \
    mirror /dev/sda /dev/sdb \
    mirror /dev/sdc /dev/sdd \
    mirror /dev/sde /dev/sdf

Exception: if your Proxmox host primarily stores backups, ISOs, or media files (sequential, large-block workloads), RAIDZ2 is fine. Separate your VM storage (mirrors) from your bulk storage (RAIDZ) into different pools.

PVE cluster with ZFS — Ceph vs ZFS replication

Proxmox supports two approaches to multi-node storage: Ceph (distributed object storage) and ZFS replication (zfs send/receive between nodes). They solve different problems.

Monitoring ZFS in PVE

The Proxmox GUI shows pool status (online/degraded/faulted) and basic space usage. That is about 10% of what you need to monitor. The rest requires the CLI.

# Pool health — the first command to run when anything seems wrong
zpool status rpool

# Pool I/O statistics — 5-second interval
zpool iostat rpool 5

# Per-vdev I/O breakdown (which disks are slow?)
zpool iostat -v rpool 5

# I/O queue depth (are writes queuing?)
zpool iostat -q rpool 5

# ARC hit rate — should be > 80% for good performance
arc_summary | grep -A5 "ARC size"
# Or:
cat /proc/spl/kstat/zfs/arcstats | grep -E '^hits|^misses'

# ARC efficiency breakdown
arcstat 5
# Output columns: read  hits  miss  hit%  l2hits  l2miss

# Pool fragmentation — high fragmentation = degraded write performance
zpool list -o name,frag,cap rpool
# Keep capacity below 80%. Above 80%, fragmentation accelerates.

# Scrub status — scrubs should run weekly
zpool status rpool | grep scan
# If the last scrub found errors, investigate immediately.

# ZFS I/O latency histogram (where is time being spent?)
zpool iostat -w rpool 5

Common mistakes

Migration between PVE nodes with ZFS

Proxmox supports live migration of VMs between cluster nodes. With ZFS, migration uses zfs send/receive to transfer the disk, which is efficient but has caveats.

# Migrate VM 100 from current node to pve2 (online)
qm migrate 100 pve2 --online

# Migrate offline (stops VM, transfers, starts on target)
qm migrate 100 pve2

# If migration is slow, check:
# 1. Network bandwidth between nodes (iperf3 -s / iperf3 -c pve2)
# 2. Whether replication is configured (pre-synced data = fast migration)
# 3. zvol size — a 2TB zvol without replication takes a while

# Manual zvol transfer (useful for moving to non-PVE ZFS hosts)
zfs send rpool/data/vm-100-disk-0 | ssh target zfs receive tank/data/vm-100-disk-0
# Incremental (much faster after initial sync):
zfs send -i @snap1 rpool/data/vm-100-disk-0@snap2 | \
  ssh target zfs receive tank/data/vm-100-disk-0

PVE + ZFS vs PVE + Ceph vs PVE + LVM-thin

Converting existing PVE from LVM-thin to ZFS

There is no in-place conversion. LVM-thin and ZFS are fundamentally different storage architectures. You must migrate VMs disk-by-disk. Here is the procedure:

1. Add new disks and create a ZFS pool alongside the existing LVM-thin storage
2. Register the ZFS pool in /etc/pve/storage.cfg
3. For each VM: shut down, use qm move-disk to move each disk from LVM-thin to ZFS
4. Boot the VM on ZFS storage, verify everything works
5. Once all VMs are migrated, remove the LVM-thin storage

# Step 1: Create the ZFS pool (new disks)
zpool create -o ashift=12 \
  -O compression=lz4 -O atime=off -O xattr=sa -O dnodesize=auto \
  vmpool mirror /dev/sdc /dev/sdd
zfs create vmpool/data

# Step 2: Register in storage.cfg
cat >> /etc/pve/storage.cfg << 'EOF'

zfspool: vmpool
    pool vmpool/data
    content images,rootdir
    sparse 1
    blocksize 16k
EOF

# Step 3: Move VM disks (VM must be stopped)
qm stop 100
qm move-disk 100 scsi0 vmpool --delete 1
# --delete 1 removes the source disk after successful move

# Step 4: Verify and start
qm start 100
qm agent 100 ping  # check guest agent responds

# Repeat for each VM. For many VMs, script it:
for vmid in 100 101 102 103 104; do
  qm stop $vmid
  qm move-disk $vmid scsi0 vmpool --delete 1
  qm start $vmid
  echo "VM $vmid migrated to ZFS"
done

PVE ZFS boot — root on ZFS

The Proxmox installer supports installing the OS directly on ZFS (root-on-ZFS). This gives you ZFS benefits for the entire system: the OS, logs, container rootfs, and VM disks all live on ZFS. Boot environments, rollback, and system-level snapshots work.

# Snapshot before PVE upgrade
zfs snapshot rpool/ROOT/pve-1@before-upgrade-$(date +%Y%m%d)

# Upgrade PVE
apt update && apt dist-upgrade

# If something breaks, rollback:
zfs rollback rpool/ROOT/pve-1@before-upgrade-20260404

# Separate data pool for VMs (recommended — keep OS and data pools separate)
# During install, choose ZFS mirror for the boot disk
# After install, create a separate pool on dedicated disks:
zpool create -o ashift=12 \
  -O compression=lz4 -O atime=off -O xattr=sa -O dnodesize=auto \
  vmpool mirror /dev/sdc /dev/sdd mirror /dev/sde /dev/sdf

Performance expectations

Real-world numbers from tuned Proxmox ZFS hosts. These assume correct volblocksize, mirrored vdevs, and properly sized ARC. Your results depend on disk hardware.

Note the massive difference between mirrors and RAIDZ for random I/O. Four HDD mirror pairs deliver 300-600 random read IOPS. A 6-disk RAIDZ2 of the same drives delivers 80-150. For VMs, that difference is the difference between "responsive" and "why is everything so slow."

Quick reference: Proxmox ZFS tuning