Why Proxmox VMs feel slow on default ZFS

Proxmox defaults to 128K recordsize for datasets. But VM disk I/O operates in 4K-8K blocks. When a VM writes 8K, ZFS has to:

1. Read the full 128K record that contains the 8K block
2. Decompress it (if compression is on)
3. Modify the 8K portion
4. Recompress the full 128K
5. Write the new 128K record to a new location (CoW)

That's 16x write amplification for every VM I/O operation. Your VMs aren't slow because ZFS is slow. They're slow because ZFS is reading and writing 128K to change 8K.

Fix 1: Use zvols with correct volblocksize

# DON'T do this (Proxmox default — dataset-backed, 128K records)
# zfs create rpool/data/vm-100-disk-0

# DO this — zvol with 16K block size
zfs create -V 40G -s \
    -o volblocksize=16K \
    -o compression=lz4 \
    rpool/data/vm-100-disk-0

# For database VMs, use 8K
zfs create -V 40G -s \
    -o volblocksize=8K \
    rpool/data/vm-100-disk-0

16K volblocksize = 2x amplification instead of 16x. That's an 8x improvement from changing one number.

Fix 2: Tune ARC for VM workloads

# Proxmox defaults restrict ARC aggressively
# Set ARC to use up to 50% of RAM (e.g., 16GB on a 32GB host)
echo "options zfs zfs_arc_max=17179869184" > /etc/modprobe.d/zfs.conf

# Set minimum ARC (don't let the kernel starve ZFS)
echo "options zfs zfs_arc_min=4294967296" >> /etc/modprobe.d/zfs.conf

# Apply without reboot
echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max
echo 4294967296 > /sys/module/zfs/parameters/zfs_arc_min

Fix 3: Add a SLOG for sync writes

# VMs use sync writes for data integrity
# Without SLOG, every sync write waits for spinning rust

# Add an enterprise NVMe as SLOG (MUST have power loss protection)
zpool add rpool log /dev/nvme1n1

# Verify
zpool status rpool | grep log

Fix 4: Add a special vdev for metadata

# Metadata operations (directory listings, file lookups) are slow on HDDs
# A mirrored SSD special vdev stores metadata on fast storage

zpool add rpool special mirror /dev/sda /dev/sdb

# Set small_blocks threshold (files smaller than this go to special vdev)
zfs set special_small_blocks=64K rpool

Fix 5: Use mirrors, not RAIDZ, for VMs

This is the most common mistake on Proxmox. RAIDZ has terrible random write performance. VMs generate random I/O. Mirrors handle random I/O linearly — each mirror pair serves requests independently.

# BAD for VMs:
# zpool create rpool raidz2 /dev/sd{a,b,c,d,e,f}

# GOOD for VMs:
zpool create rpool \
    mirror /dev/sda /dev/sdb \
    mirror /dev/sdc /dev/sdd \
    mirror /dev/sde /dev/sdf