ZFS on root. Every distro. Automatic. Identical.

Every kldload install creates the same ZFS dataset hierarchy. The pool layout, compression settings, snapshot policies, and mount points are identical across every supported distro — CentOS, Debian, Ubuntu, Fedora, Rocky, RHEL, Arch, and Alpine. Switch distros and your ZFS muscle memory transfers completely. The pool does not care which kernel is above it.

ZFS on root — the foundation

Every kldload install — regardless of distro, profile, or topology — puts the operating system directly on ZFS. There is no ext4 boot partition. There is no LVM layer. There is no mdraid array. The disk is partitioned into exactly two parts: a 512MB EFI System Partition (FAT32) and the rest of the disk as a ZFS pool named rpool.

The partitioning is handled by sgdisk in the installer. On a single-disk install:

# Single-disk layout: EFI + rpool
sgdisk -n1:1M:+512M -t1:EF00 -c1:"EFI System Partition" /dev/sda
sgdisk -n2:0:0      -t2:BF01 -c2:"KLDload rpool"        /dev/sda

Partition 1 is the ESP — formatted FAT32, mounted at /boot/efi. This holds the EFI bootloader (ZFSBootMenu) and nothing else. Partition 2 is the entire remaining disk, used as the sole vdev in the rpool pool. On multi-disk topologies, only the boot disk gets an ESP; the data disks are used raw as pool vdevs.

Boot pool vs root pool

Some ZFS-on-root guides create two pools: a small bpool for /boot and a larger rpool for everything else. This exists to work around GRUB's limited ZFS feature support. kldload does not use GRUB. kldload uses ZFSBootMenu, which supports all modern ZFS features. Therefore kldload creates a single pool: rpool. One pool. No feature restrictions. No compatibility hacks.

The rpool pool is created with -R /target (altroot) during installation, which temporarily mounts everything under /target. After installation completes, the pool is exported and the target reboots. ZFSBootMenu imports the pool at boot with the correct mountpoints.

ESP handling

The EFI System Partition is 512MB FAT32. It holds ZFSBootMenu's EFI binary, the ZFSBootMenu initramfs (which contains the ZFS kernel module), and a fallback kernel. On multi-disk topologies, only the boot disk (KLDLOAD_DISK, typically /dev/sda) has an ESP. The rpool lives on separate data disks. This means the boot disk can be a small SSD while the data disks are large spinners or NVMe.

# What lives on the ESP
/boot/efi/
  EFI/
    BOOT/
      BOOTX64.EFI          # ZFSBootMenu EFI binary
    kldload/
      vmlinuz-zbm           # ZFSBootMenu kernel
      initramfs-zbm.img     # ZFSBootMenu initramfs (contains zfs.ko)

Default dataset layout

The desktop and server profiles create a specific dataset hierarchy. Every dataset exists for a specific operational reason. This is not a guess. This is the result of running ZFS in production across hundreds of machines and learning what breaks when the layout is wrong.

This is the exact code from storage-zfs.sh that creates the hierarchy:

# Root dataset hierarchy
zfs create -o canmount=off -o mountpoint=none rpool/ROOT
zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/${KLDLOAD_HOSTNAME}
zfs mount rpool/ROOT/${KLDLOAD_HOSTNAME}

# Data datasets
zfs create -o mountpoint=/root      rpool/root
zfs create -o mountpoint=/home      rpool/home
zfs create -o mountpoint=/home/todd rpool/home/todd   # per-user dataset
zfs create -o mountpoint=/srv       rpool/srv
zfs create -o mountpoint=/opt       rpool/opt

# /usr namespace
zfs create -o canmount=off -o mountpoint=/usr rpool/usr
zfs create -o mountpoint=/usr/local rpool/usr/local

# /var namespace
zfs create -o canmount=off -o mountpoint=/var rpool/var
zfs create -o mountpoint=/var/cache rpool/var/cache
zfs create -o mountpoint=/var/lib   rpool/var/lib
zfs create -o mountpoint=/var/log   rpool/var/log
zfs create -o mountpoint=/var/spool rpool/var/spool
zfs create -o mountpoint=/var/tmp   rpool/var/tmp

# /tmp — hardened: no suid, no exec, no devices, async writes
zfs create -o mountpoint=/tmp -o sync=disabled \
  -o setuid=off -o exec=off -o devices=off rpool/tmp

kldload's ZFS defaults — and why each was chosen

Every property set on the pool at creation time is inherited by every child dataset unless explicitly overridden. These are the properties kldload sets, and the engineering reason behind each one.

The exact zpool create command from the installer:

zpool create -f \
  -o ashift=12 \
  -o autotrim=on \
  -O acltype=posixacl \
  -O canmount=off \
  -O compression=lz4 \
  -O dnodesize=auto \
  -O normalization=formD \
  -O relatime=on \
  -O xattr=sa \
  -O mountpoint=none \
  -R /target \
  rpool /dev/sda2

Pool design — topologies and disk selection

kldload supports four pool topologies. The web UI presents them as choices during install. The topology determines how many disks are needed, how redundancy works, and how the rpool vdev is constructed.

sgdisk -n1:1M:+512M -t1:EF00 /dev/sda
sgdisk -n2:0:0      -t2:BF01 /dev/sda
# rpool on partition 2
rpool_vdevs=( /dev/sda2 )

# EFI on /dev/sda, rpool mirrors sdb+sdc
rpool_vdevs=( mirror /dev/sdb /dev/sdc )

# EFI on /dev/sda, rpool raidz1 across sdb-sde
rpool_vdevs=( raidz1 /dev/sdb /dev/sdc /dev/sdd /dev/sde )

# EFI on /dev/sda, rpool RAID10 across sdb-sde
rpool_vdevs=(
  mirror /dev/sdb /dev/sdc
  mirror /dev/sdd /dev/sde
)

ashift auto-detection

kldload always forces ashift=12 (4K sectors). ZFS can auto-detect the physical sector size, but many drives lie about it. Samsung SSDs report 512-byte sectors when they use 4K internally. WD Reds report 512e (512-byte emulated). If ZFS trusts these drives and uses ashift=9, every 4K physical write requires a read-modify-write cycle on the drive controller. Performance drops 30-50%. Since ashift cannot be changed after pool creation, kldload always sets 12. Using ashift=12 on a true 512-byte drive wastes at most a few percent of space. Using ashift=9 on a 4K drive wastes half your IOPS permanently.

Special vdev acceleration

kldload supports an optional special vdev for metadata and small-block acceleration. If you provide one or two fast NVMe drives as KLDLOAD_ZFS_SPECIAL_DISKS, the installer adds a special vdev to the pool after creation. ZFS stores metadata (directory entries, file sizes, block pointers) and optionally small files on the special vdev instead of the main data vdevs.

# Two NVMe drives as mirrored special vdev
zpool add rpool special mirror /dev/nvme0n1 /dev/nvme1n1

# Direct small blocks (under 64K) to the special vdev
zfs set special_small_blocks=65536 rpool

A special vdev on NVMe in front of a raidz1 of spinning disks transforms performance. Metadata lookups that normally require seeking across 4+ HDDs complete in microseconds on NVMe. Directory listings, find commands, ls -la — all become instant. The main vdevs handle large sequential I/O, which is what HDDs are good at.

Snapshots and replication

kldload pre-configures Sanoid for automatic snapshot management on desktop and server profiles. Sanoid handles creation and pruning. Syncoid handles replication. Both are installed and configured at install time. You never have to think about snapshot rotation unless you want to change the defaults.

Snapshot tiers

Default sanoid.conf

# /etc/sanoid/sanoid.conf — kldload default

[rpool/ROOT]
  use_template = production
  recursive = yes

[rpool/home]
  use_template = production
  recursive = yes

[rpool/srv]
  use_template = frequent
  recursive = yes

[rpool/var/log]
  use_template = production

[rpool/var/lib]
  use_template = production

#---- Templates ----
[template_production]
  frequently = 0
  hourly = 48
  daily = 30
  monthly = 6
  yearly = 0
  autosnap = yes
  autoprune = yes

[template_frequent]
  frequently = 4
  hourly = 48
  daily = 30
  monthly = 6
  yearly = 0
  autosnap = yes
  autoprune = yes

Replication with Syncoid

Syncoid replicates datasets to any ZFS target: another kldload node, TrueNAS, any Linux with OpenZFS. Initial full send, then incremental (only changed blocks). Over SSH, over WireGuard, over anything that can carry a byte stream.

# Replicate /srv to a backup host every 15 minutes
syncoid rpool/srv backup-host:backup/srv

# Replicate recursively (all child datasets)
syncoid -r rpool/home backup-host:backup/home

# Cron job: replicate every 15 minutes over WireGuard
*/15 * * * * /usr/sbin/syncoid --no-sync-snap rpool/srv wg-backup:backup/srv

# Manual zfs send/recv for full control
zfs send -R rpool/srv@snap | ssh backup "zfs recv -F backup/srv"

# Incremental (only changes since last sync)
zfs send -R -I @old @new rpool/srv | ssh backup "zfs recv backup/srv"

# Encrypted replication (ciphertext only — receiver cannot read)
zfs send -w rpool/srv@snap | ssh backup "zfs recv backup/srv"

Boot environments

A boot environment is a snapshot or clone of rpool/ROOT/<hostname>. Every kernel update, every major package upgrade, every risky change — create a new boot environment first. If the change breaks the system, select the previous BE in ZFSBootMenu and boot it. Total downtime: one reboot.

kldload uses ZFSBootMenu as its bootloader. ZFSBootMenu scans all datasets under rpool/ROOT and presents them as bootable entries. The bootfs property on the pool controls which one boots by default.

# Create a boot environment before a risky upgrade
zfs snapshot rpool/ROOT/myhost@before-upgrade
zfs clone rpool/ROOT/myhost@before-upgrade rpool/ROOT/myhost-backup

# List all boot environments
zfs list -r -o name,used,creation rpool/ROOT

# Roll back to the previous boot environment
zpool set bootfs=rpool/ROOT/myhost-backup rpool
# Reboot — ZFSBootMenu boots the backup BE automatically

# Or use ZFSBootMenu interactively at boot (press ESC during boot countdown)
# Select any BE from the menu, boot it, done

# After confirming the old BE works, promote it
zpool set bootfs=rpool/ROOT/myhost-backup rpool

# Clean up the broken BE
zfs destroy rpool/ROOT/myhost

ZFSBootMenu kernel commandline

kldload sets the kernel commandline via a ZFS property so it is stored in the pool itself, not in a separate bootloader config file:

# Set during install — inherited by all boot environments
zfs set org.zfsbootmenu:commandline="rw console=tty1 console=ttyS0,115200" rpool/ROOT

# console=tty1       — VGA output for physical machines
# console=ttyS0      — serial console for Proxmox/KVM/headless
# Both are set so it works everywhere without reconfiguration

Storage for virtual machines

kldload's desktop and server profiles install KVM with ZFS zvols as VM storage. A zvol is a ZFS dataset that presents itself as a raw block device at /dev/zvol/rpool/vms/name. QEMU sees a block device, not a file. There is no qcow2 layer. No double copy-on-write. ZFS handles snapshots, clones, compression, and replication transparently.

# Create a 100GB zvol for a VM
zfs create -V 100G -o volblocksize=64K rpool/vms/webserver

# The zvol appears as a block device
ls -la /dev/zvol/rpool/vms/webserver
# lrwxrwxrwx 1 root root 9 Apr  4 15:30 /dev/zvol/rpool/vms/webserver -> ../../zd0

# Snapshot before upgrade — instant, O(1)
zfs snapshot rpool/vms/webserver@before-upgrade

# Upgrade goes wrong — rollback in milliseconds
zfs rollback rpool/vms/webserver@before-upgrade

# Clone a VM — instant, zero disk space used initially
zfs snapshot rpool/vms/webserver@template
zfs clone rpool/vms/webserver@template rpool/vms/webserver-staging
# New VM is ready immediately — shares all blocks with parent via COW

# Replicate a VM to another host
zfs send -R rpool/vms/webserver@snap | ssh dr-host "zfs recv backup/vms/webserver"

volblocksize tuning

The volblocksize is the ZFS block size for the zvol. It must match or be a multiple of the guest filesystem's block size. The default is 8K, but for VM workloads:

Storage for containers

Docker and Podman on kldload use the ZFS storage driver. Each container layer is a ZFS dataset. Each container gets its own dataset. Copy-on-write is native — no overlayfs, no devicemapper, no loopback files. The ZFS driver is the most efficient storage backend for containers on Linux when you already have ZFS.

# Docker ZFS storage driver configuration
# /etc/docker/daemon.json (set automatically by kldload)
{
  "storage-driver": "zfs",
  "storage-opts": [
    "zfs.fsname=rpool/var/lib/docker"
  ]
}

# Verify Docker is using ZFS
docker info | grep "Storage Driver"
# Storage Driver: zfs

# Each image layer is a ZFS dataset
zfs list -r rpool/var/lib/docker | head
# NAME                                              USED  AVAIL  REFER  MOUNTPOINT
# rpool/var/lib/docker                              4.2G   180G   24K   legacy
# rpool/var/lib/docker/abc123...                    1.1G   180G  1.1G   legacy
# rpool/var/lib/docker/def456...                    256M   180G  256M   legacy

# Snapshot all container storage
zfs snapshot -r rpool/var/lib/docker@backup-$(date +%Y%m%d)

# Per-container datasets mean per-container quotas
# No single container can fill the disk

The ZFS storage driver provides real copy-on-write at the filesystem level. When Docker creates a new container from an image, ZFS clones the image dataset. The clone is instant and uses zero additional space until the container writes data. Overlayfs achieves similar COW semantics but at the VFS layer with significant overhead for deep layer stacks. The ZFS driver eliminates that overhead entirely.

# Podman per-user ZFS storage
zfs create rpool/home/todd/.local/share/containers
podman info --format '{{.Store.GraphDriverName}}'
# zfs

NFS and iSCSI — sharing ZFS storage

ZFS datasets can be shared via NFS with a single property. ZFS zvols can be shared via iSCSI with targetcli. No separate volume manager needed. No LVM. No mdraid. The ZFS pool is the storage layer, NFS/iSCSI is the transport.

NFS exports

# Share a dataset via NFS — one command
zfs set sharenfs="rw=@10.0.0.0/24,no_root_squash" rpool/srv/shared

# ZFS writes /etc/exports.d/zfs.exports automatically
# exportfs picks it up — no manual /etc/exports editing

# Verify the export
showmount -e localhost
# /srv/shared  10.0.0.0/24

# Client mount
mount -t nfs server:/srv/shared /mnt/shared

# Performance tuning for NFS on ZFS
zfs set recordsize=128K rpool/srv/shared    # match NFS rsize/wsize
zfs set sync=standard rpool/srv/shared      # NFS requires sync writes for NFSv3
# For NFSv4 with async mount: zfs set sync=disabled (unsafe but fast)

# Multiple shares with different ACLs
zfs set sharenfs="rw=@10.0.0.0/24,ro=@10.0.1.0/24" rpool/srv/shared
zfs set sharenfs="rw=@10.0.0.5,no_root_squash" rpool/srv/admin-share

iSCSI targets

# Create a zvol for iSCSI
zfs create -V 500G -o volblocksize=4K rpool/iscsi/database-lun

# Configure targetcli
targetcli /backstores/block create db-lun /dev/zvol/rpool/iscsi/database-lun
targetcli /iscsi create iqn.2026-04.com.kldload:storage
targetcli /iscsi/iqn.2026-04.com.kldload:storage/tpg1/luns create /backstores/block/db-lun
targetcli /iscsi/iqn.2026-04.com.kldload:storage/tpg1/acls create iqn.2026-04.com.kldload:client1

# Save the targetcli config
targetcli saveconfig

# Client discovery and login
iscsiadm -m discovery -t sendtargets -p 10.0.0.1:3260
iscsiadm -m node --login

# Performance tuning for iSCSI on ZFS
zfs set volblocksize=4K rpool/iscsi/database-lun     # match DB page size
zfs set compression=off rpool/iscsi/database-lun     # database does its own compression
zfs set primarycache=metadata rpool/iscsi/database-lun  # DB has its own buffer cache
zfs set logbias=throughput rpool/iscsi/database-lun  # skip ZIL for async workloads

Encryption — native ZFS, not LUKS

kldload offers optional ZFS-native encryption at install time. This is not LUKS. Not dm-crypt. Not a block device encryption layer under ZFS. It is AES-256-GCM encryption built into the ZFS storage layer itself. Per-dataset. Hardware-accelerated on any CPU with AES-NI (every x86-64 CPU since 2010).

When encryption is enabled, the installer adds these properties to the zpool create command. Every dataset inherits encryption from the pool root. The passphrase is entered once at boot in ZFSBootMenu; all datasets unlock together.

# Pool creation with encryption (from storage-zfs.sh)
zpool create -f \
  -o ashift=12 -o autotrim=on \
  -O acltype=posixacl -O canmount=off -O compression=lz4 \
  -O dnodesize=auto -O normalization=formD -O relatime=on -O xattr=sa \
  -O encryption=aes-256-gcm \
  -O keyformat=passphrase \
  -O keylocation=prompt \
  -O mountpoint=none \
  -R /target \
  rpool /dev/sda2

# Verify encryption status
zfs get encryption,keystatus rpool
# NAME   PROPERTY      VALUE            SOURCE
# rpool  encryption    aes-256-gcm      -
# rpool  keystatus     available        -

# Every child dataset inherits encryption
zfs get encryption rpool/home
# NAME        PROPERTY    VALUE          SOURCE
# rpool/home  encryption  aes-256-gcm    inherited from rpool

Per-dataset encryption

Even on a non-encrypted pool, individual datasets can be encrypted with different keys. This enables multi-tenant scenarios where each tenant's data has a separate passphrase:

# Encrypt a specific dataset on an unencrypted pool
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase \
  -o keylocation=prompt rpool/srv/tenant-a

# Different key for a different tenant
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase \
  -o keylocation=prompt rpool/srv/tenant-b

# Lock a dataset (unmount and unload key)
zfs unmount rpool/srv/tenant-a
zfs unload-key rpool/srv/tenant-a

# Unlock and mount
zfs load-key rpool/srv/tenant-a
zfs mount rpool/srv/tenant-a

Encrypted replication

The -w (raw) flag on zfs send sends encrypted blocks as ciphertext. The receiving machine stores data it cannot decrypt. This is data sovereignty at the storage layer — replicate to an untrusted host and they physically cannot read your data.

# Send encrypted — receiver stores ciphertext, cannot decrypt
zfs send -w rpool/srv@snap | ssh untrusted-host "zfs recv backup/srv"

# The key never leaves the source machine
# The receiving host cannot: load-key, mount, or read the data
# Perfect for offsite DR to cloud or partner infrastructure

Monitoring — health checks, scrubs, ARC, and alerts

kldload configures ZFS monitoring out of the box. Pool health checks, automatic scrubs, ARC utilization tracking, and capacity alerts are all set up at install time. The monitoring stack integrates with the kldload observability tools if installed.

Scrub scheduling

# Weekly scrub via systemd timer (installed by kldload)
# /etc/systemd/system/zfs-scrub@.timer
[Timer]
OnCalendar=Sun *-*-* 02:00:00
Persistent=true

# Manual scrub
zpool scrub rpool

# Check scrub status
zpool status rpool | grep scan
#   scan: scrub repaired 0B in 00:12:34 with 0 errors on Sun Apr  4 02:12:34 2026

# Pause a scrub (if it's impacting production I/O)
zpool scrub -p rpool

# Resume
zpool scrub rpool

Pool health and status

# Full pool status — health, scrub, errors, vdev layout
zpool status rpool
#   pool: rpool
#  state: ONLINE
# status: One or more devices has experienced an unrecoverable error.
# action: Determine if the device needs to be replaced.
#   scan: scrub repaired 4K in 00:12:34 with 0 errors
# config:
#   NAME          STATE     READ WRITE CKSUM
#   rpool         ONLINE       0     0     0
#     mirror-0    ONLINE       0     0     0
#       sdb       ONLINE       0     0     0
#       sdc       ONLINE       0     0     1

# Quick health check (scriptable)
zpool status -x
# all pools are healthy

# Pool I/O statistics (like iostat for ZFS)
zpool iostat rpool 5
#               capacity     operations     bandwidth
# pool        alloc   free   read  write   read  write
# rpool       45.2G   180G    125    340  4.12M  12.5M

# Per-vdev I/O stats
zpool iostat -v rpool 5

ARC monitoring

# ARC (Adaptive Replacement Cache) statistics
arc_summary

# Quick ARC hit ratio
arcstat 5
#     time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c
# 15:30:01  1.2K    45   3.7%    30  2.5%    15  1.2%     0  0.0%  3.8G  4.0G

# ARC size and target
cat /proc/spl/kstat/zfs/arcstats | grep -E "^(size|c_max|c_min)"
# size     4   4089446400  (current ARC size ~3.8GB)
# c_min    4   1073741824  (minimum target ~1GB)
# c_max    4   8589934592  (maximum target ~8GB)

# Set maximum ARC size (e.g., limit to 4GB on a 16GB machine)
echo 4294967296 > /sys/module/zfs/parameters/zfs_arc_max
# Persist across reboots:
echo "options zfs zfs_arc_max=4294967296" > /etc/modprobe.d/zfs.conf

Capacity alerts

# Check pool capacity — ZFS degrades above 80%, critical above 90%
zpool list -o name,size,alloc,free,cap,health
# NAME    SIZE  ALLOC   FREE   CAP  HEALTH
# rpool   224G  45.2G   179G   20%  ONLINE

# Simple alert script (runs via cron)
#!/bin/bash
CAP=$(zpool list -Hp -o capacity rpool)
if [ "$CAP" -gt 80 ]; then
  echo "WARNING: rpool at ${CAP}% capacity" | \
    mail -s "ZFS capacity alert" admin@example.com
fi

# Per-dataset usage breakdown
zfs list -o name,used,avail,refer,compressratio -r rpool | head -20

# Find datasets consuming the most space
zfs list -o name,used -s used -r rpool | tail -10

# Check compression savings
zfs get compressratio rpool
# NAME   PROPERTY       VALUE  SOURCE
# rpool  compressratio  1.82x  -
# That means 45GB on disk is storing 82GB of logical data

Comparison — kldload ZFS vs the alternatives

Every storage decision is a tradeoff. ZFS is not the answer to every problem. But for the use cases kldload targets — OS root, VM storage, container storage, NFS/iSCSI serving, single-node and replicated setups — it is the correct choice. Here is why.

kldload ZFS vs ext4 + LVM

kldload ZFS vs Ceph

kldload ZFS vs GlusterFS

When NOT to use ZFS

Summary — what kldload gives you out of the box