KVM Virtual Machines on kldload

The kvm profile is a kldload OS template — select it during install and you get a fully configured KVM hypervisor with ZFS zvol storage, bridged networking, ARC tuning, automated hourly snapshots, and CLI tools for creating, cloning, and replicating VMs. No post-install configuration required.

Selecting the KVM profile

During kldload install, select the kvm profile in the web UI (or set KLDLOAD_PROFILE=kvm for unattended installs). The profile gives you:

• qemu-kvm, libvirt, virt-install, bridge-utils, OVMF, dnsmasq
• ZFS datasets: rpool/vms (compression=off, recordsize=64K, primarycache=metadata) + rpool/vms/isos (zstd)
• ARC capped at 50% of RAM — the rest is for VM guests
• Kernel tuning: swappiness=1, dirty page thresholds, bridge-nf disabled
• Hourly kvm-snapshot timer (48h retention per VM)
• Tools: kvm-create, kvm-clone, kvm-snap, kvm-replicate, kvm-delete, kvm-list
• libvirtd enabled at boot

Adding KVM to an existing install

If you installed with a different profile (desktop, server) and want to add KVM later:

# CentOS/RHEL/Rocky/Fedora
dnf install -y qemu-kvm libvirt virt-install bridge-utils ovmf dnsmasq libguestfs-tools
systemctl enable --now libvirtd

# Debian/Ubuntu
apt install -y qemu-kvm libvirt-daemon-system virtinst bridge-utils ovmf dnsmasq-base libguestfs-tools
systemctl enable --now libvirtd

You'll also need to create the ZFS datasets and tuning manually — see the sections below.

Verify

virsh list --all
virt-host-validate

ZFS storage architecture for KVM

What the kldload installer creates

When you select the KVM profile (or set KLDLOAD_ENABLE_KVM=1), the installer automatically creates this ZFS layout:

# VM disk images — compression OFF (qcow2/zvols handle their own compression)
# recordsize=64K matches zvol volblocksize for optimal alignment
# primarycache=metadata avoids ARC bloat from VM I/O
zfs create -o mountpoint=/var/lib/libvirt/images \
           -o compression=off \
           -o recordsize=64K \
           -o primarycache=metadata \
           -o atime=off \
           rpool/vms

# ISO storage — compressed with zstd (read-mostly workload)
zfs create -o mountpoint=/var/lib/libvirt/isos \
           -o compression=zstd \
           -o atime=off \
           rpool/vms/isos

Per-VM zvols (the kldload way)

kldload uses ZFS zvols — virtual block devices — for VM disks, not qcow2 files sitting on datasets. Each VM gets its own zvol at /dev/zvol/rpool/vms/<name>. This gives VMs direct block device access with zero qcow2 overhead.

# kvm-create does this automatically, but here's what's happening:
# Thin-provisioned zvol — only uses space as the VM writes data
zfs create -V 40G -s -o compression=off -o volblocksize=64K rpool/vms/web-1

# The zvol appears as a block device
ls -la /dev/zvol/rpool/vms/web-1
# /dev/zvol/rpool/vms/web-1 -> ../../zd0

# virt-install uses it directly — no qcow2 layer
virt-install --name web-1 ... \
  --disk "/dev/zvol/rpool/vms/web-1,bus=virtio,cache=none"

The kvm-create tool

kldload ships kvm-create which handles all of this automatically:

# Create a VM with sensible defaults (2GB RAM, 2 vCPUs, 20GB disk)
kvm-create myvm

# Create a VM with custom resources
kvm-create webserver --ram 4096 --cpus 4 --disk 40

# Create a VM from an ISO
kvm-create testbox --iso /var/lib/libvirt/isos/debian-13.iso --ram 2048 --os debian12

# What kvm-create does:
# 1. Creates thin-provisioned zvol: rpool/vms/ (volblocksize=64K)
# 2. Runs virt-install with: q35 machine, host-model CPU, virtio disk, UEFI boot
# 3. Disk is at /dev/zvol/rpool/vms/ — direct block device, no qcow2

ARC and kernel tuning (installed automatically)

The KVM profile installs tuning for ZFS + KVM coexistence:

# /etc/modprobe.d/zfs.conf — installed by the KVM profile
options zfs zfs_arc_max=<50% of RAM>   # leave memory for VM guests
options zfs zfs_txg_timeout=10          # flush writes more frequently
options zfs zfs_vdev_scheduler=none     # let the disk scheduler handle it
options zfs l2arc_noprefetch=0          # allow L2ARC prefetch

# /etc/sysctl.d/99-zfs-kvm.conf — installed by the KVM profile
vm.swappiness = 1                       # don't swap — let ARC shrink instead
vm.vfs_cache_pressure = 50              # balance inode/dentry cache
vm.dirty_ratio = 10                     # flush dirty pages sooner
vm.dirty_background_ratio = 5           # start background flush earlier
net.ipv4.ip_forward = 1                 # required for bridged VMs
net.bridge.bridge-nf-call-iptables = 0  # don't firewall bridge traffic

What a properly configured KVM+ZFS server looks like

$ zfs list -o name,used,avail,refer,volsize,compression -r rpool/vms
NAME                USED  AVAIL  REFER  VOLSIZE  COMPRESS
rpool/vms          34.8G   400G   196K        -  off
rpool/vms/isos      2.1G   400G  2.1G        -  zstd
rpool/vms/web-1     8.2G   400G    64K     40G  lz4
rpool/vms/web-2     7.9G   400G    64K     40G  lz4
rpool/vms/db-1     18.7G   400G    64K    200G  lz4

$ zfs list -t snapshot -r rpool/vms
NAME                                     USED  AVAIL  REFER
rpool/vms/web-1@clone-20260401_120000    1.2G      -    64K
rpool/vms/web-1@auto-20260402T100000Z     42M      -    64K
rpool/vms/db-1@auto-20260401T100000Z      89M      -    64K
rpool/vms/db-1@auto-20260402T100000Z     112M      -    64K

$ ls -la /dev/zvol/rpool/vms/
web-1  web-2  db-1

Create a VM

Using kvm-create (recommended)

# Simple — sensible defaults
kvm-create my-server

# With an ISO
kvm-create my-server --ram 4096 --cpus 4 --disk 40 \
  --iso ~/kldload-free/live-build/output/kldload-free-latest.iso \
  --os centos-stream9

Manual virt-install (for custom setups)

# Create the zvol first
zfs create -V 40G -s -o compression=off -o volblocksize=64K rpool/vms/my-server

virt-install \
  --name my-server \
  --ram 4096 \
  --vcpus 4 \
  --cpu host-model \
  --machine q35 \
  --disk "/dev/zvol/rpool/vms/my-server,bus=virtio,cache=none" \
  --cdrom ~/kldload-free/live-build/output/kldload-free-latest.iso \
  --os-variant centos-stream9 \
  --network bridge=br0 \
  --graphics vnc,listen=0.0.0.0 \
  --boot uefi \
  --noautoconsole

Connect via VNC to complete the install:

virsh vncdisplay my-server
# :0  → connect to host:5900

Create a VM from a cloud image

CentOS Stream 9

curl -LO https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2
cp CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2 /var/lib/libvirt/images/centos-cloud.qcow2

# Resize the disk
qemu-img resize /var/lib/libvirt/images/centos-cloud.qcow2 40G

# Set a root password with virt-customize
virt-customize -a /var/lib/libvirt/images/centos-cloud.qcow2 \
  --root-password password:changeme \
  --hostname my-centos-vm

virt-install \
  --name centos-cloud \
  --ram 2048 --vcpus 2 \
  --disk /var/lib/libvirt/images/centos-cloud.qcow2 \
  --os-variant centos-stream9 \
  --network bridge=br0 \
  --boot uefi \
  --import --noautoconsole

Debian

curl -LO https://cloud.debian.org/images/cloud/trixie/daily/latest/debian-13-generic-amd64-daily.qcow2
cp debian-13-generic-amd64-daily.qcow2 /var/lib/libvirt/images/debian-cloud.qcow2
qemu-img resize /var/lib/libvirt/images/debian-cloud.qcow2 40G

virt-customize -a /var/lib/libvirt/images/debian-cloud.qcow2 \
  --root-password password:changeme \
  --hostname my-debian-vm

virt-install \
  --name debian-cloud \
  --ram 2048 --vcpus 2 \
  --disk /var/lib/libvirt/images/debian-cloud.qcow2 \
  --os-variant debian12 \
  --network bridge=br0 \
  --boot uefi \
  --import --noautoconsole

Golden images and CoW cloning

Create one base image, then clone it instantly for each new VM:

Build the golden image

# Install a VM, configure it how you want, then shut it down
virsh shutdown my-server

# Generalize it (remove machine-specific state)
virt-sysprep -d my-server \
  --operations defaults,-ssh-userdir \
  --hostname localhost

# Mark it as a template (prevent accidental boot)
virsh dominfo my-server

Clone with kvm-clone (the kldload way)

# Create a template VM, configure it, shut it down
kvm-create template --ram 2048 --cpus 2 --disk 20 --iso /var/lib/libvirt/isos/centos9.iso
# ... install OS, configure, shut down ...

# Instant clone — zero bytes copied, shares all blocks via ZFS CoW
kvm-clone template web-1

# Another clone — also instant
kvm-clone template web-2

# Clone from a specific named snapshot
kvm-snap template          # creates a timestamped snapshot
kvm-clone template web-3 --snap @stable

# What kvm-clone does under the hood:
# 1. Pauses the source VM briefly (if running) for a consistent snapshot
# 2. zfs snapshot rpool/vms/template@clone-
# 3. zfs clone rpool/vms/template@clone- rpool/vms/web-1
# 4. virt-clone --original template --name web-1 --file /dev/zvol/rpool/vms/web-1
# 5. Resumes the source VM
# Total time: under 2 seconds for any disk size

# Spin up 10 VMs from a template in a loop
for i in $(seq 1 10); do
  kvm-clone template web-${i}
  virsh start web-${i}
done
# 10 VMs from one template in under 30 seconds. Each is its own zvol.

Clone with qemu-img (alternative — for non-zvol setups)

# CoW clone — near-instant, near-zero space
qemu-img create -f qcow2 \
  -b /var/lib/libvirt/images/my-server.qcow2 \
  -F qcow2 \
  /var/lib/libvirt/images/web-1.qcow2

# Register the clone as a new VM
virt-install \
  --name web-1 \
  --ram 2048 --vcpus 2 \
  --disk /var/lib/libvirt/images/web-1.qcow2 \
  --os-variant centos-stream9 \
  --network bridge=br0 \
  --boot uefi \
  --import --noautoconsole

Clone with virt-clone (copies everything)

# Full copy (slower but independent — no backing file dependency)
virt-clone \
  --original my-server \
  --name web-2 \
  --auto-clone

Customize after cloning

# Set unique hostname on the clone
virt-customize -d web-1 --hostname web-1.infra.local

# Or SSH in after boot:
virsh start web-1
ssh root@<ip> hostnamectl set-hostname web-1.infra.local

Snapshots

kvm-snap (recommended)

kldload ships kvm-snap for VM snapshots — it pauses the VM briefly for a consistent snapshot, then resumes it:

# Snapshot a VM (pauses briefly for consistency, then resumes)
kvm-snap web-1

# List all snapshots for a VM
kvm-snap web-1 list
# NAME                                     CREATION              USED
# rpool/vms/web-1@2026-04-01_153022        Wed Apr  1 15:30      42M
# rpool/vms/web-1@auto-20260402T100000Z    Thu Apr  2 10:00      89M

# Roll back to the most recent snapshot
kvm-snap web-1 rollback

# Roll back to a specific snapshot
kvm-snap web-1 rollback @2026-04-01_153022

# Delete a snapshot
kvm-snap web-1 delete @2026-04-01_153022

Or use raw ZFS commands for the same thing:

# Snapshot the zvol directly
zfs snapshot rpool/vms/web-1@before-update

# Roll back
virsh destroy web-1    # force stop
zfs rollback rpool/vms/web-1@before-update
virsh start web-1

Libvirt snapshots

# Create
virsh snapshot-create-as web-1 --name before-update --description "pre-update snapshot"

# List
virsh snapshot-list web-1

# Revert
virsh snapshot-revert web-1 before-update

# Delete
virsh snapshot-delete web-1 before-update

VM lifecycle

# Start / stop / restart
virsh start web-1
virsh shutdown web-1       # graceful
virsh destroy web-1        # force stop (like pulling the power cord)
virsh reboot web-1

# Pause / resume
virsh suspend web-1
virsh resume web-1

# Auto-start on host boot
virsh autostart web-1
virsh autostart --disable web-1

# Delete VM and its zvol (clean removal)
kvm-delete web-1            # must be stopped first
kvm-delete web-1 --force    # force-stop then delete
# Automatically cleans up orphaned clone snapshots

# List all VMs with ZFS disk usage
kvm-list

# Console access
virsh console web-1        # serial console (Ctrl+] to exit)

Resource management

# Change RAM (requires shutdown)
virsh setmaxmem web-1 8G --config
virsh setmem web-1 8G --config

# Change vCPUs (requires shutdown)
virsh setvcpus web-1 4 --config --maximum
virsh setvcpus web-1 4 --config

# Hot-add a disk
qemu-img create -f qcow2 /var/lib/libvirt/images/web-1-data.qcow2 100G
virsh attach-disk web-1 /var/lib/libvirt/images/web-1-data.qcow2 vdb \
  --driver qemu --subdriver qcow2 --persistent

# Hot-add a network interface
virsh attach-interface web-1 bridge br0 --model virtio --persistent

Monitoring

# List all VMs with state
virsh list --all

# CPU/memory stats
virt-top

# Disk I/O stats
virsh domblkstat web-1 vda

# Network stats
virsh domifstat web-1 vnet0

# Get IP address of a VM (requires qemu-guest-agent)
virsh domifaddr web-1

Automated VM snapshots

The KVM profile installs an hourly snapshot timer that automatically snapshots every VM zvol and keeps 48 hours of history:

# The kvm-snapshot.timer is installed and enabled automatically
systemctl status kvm-snapshot.timer

# It runs hourly and does:
# 1. Snapshots every zvol under rpool/vms with a timestamped @auto- name
# 2. Prunes auto-snapshots older than 48 hours per zvol

# Verify snapshots are being created
zfs list -t snapshot -r rpool/vms -o name,creation -s creation | tail -10

For longer retention (daily, weekly, monthly), add sanoid on top:

# /etc/sanoid/sanoid.conf — extended retention
[rpool/vms]
  use_template = vm-parent
  recursive = yes

[template_vm-parent]
  hourly = 48
  daily = 7
  weekly = 4
  monthly = 1
  autosnap = yes
  autoprune = yes

VM replication to a DR site

Replicate a single VM with kvm-replicate

kldload ships kvm-replicate which handles incremental ZFS send/receive automatically:

# First run: full send of db-1 to the DR host
kvm-replicate rpool/vms/db-1 dr-host

# Subsequent runs: incremental — only blocks changed since last replication
kvm-replicate rpool/vms/db-1 dr-host

# Replicate to a different dataset name on the remote
kvm-replicate rpool/vms/db-1 dr-host rpool/replicas/db-1

# What kvm-replicate does:
# 1. Creates a @repl- snapshot
# 2. Finds the previous @repl- snapshot
# 3. Sends the incremental delta: zfs send -i @prev @new | ssh remote zfs recv
# 4. Cleans up old replication snapshots (keeps last 2)

Replicate all VMs with syncoid

# syncoid handles recursive replication with automatic incrementals
syncoid --recursive rpool/vms root@dr-host:rpool/vms

Automated replication with systemd timer

cat > /etc/systemd/system/vm-replicate.service <<'EOF'
[Unit]
Description=Replicate VM zvols to DR site
After=wg-quick@wg0.service
Requires=wg-quick@wg0.service

[Service]
Type=oneshot
# Replicate each VM zvol individually
ExecStart=/bin/bash -c 'for ds in $(zfs list -H -o name -t volume -r rpool/vms); do /usr/local/sbin/kvm-replicate "$ds" dr-host; done'
EOF

cat > /etc/systemd/system/vm-replicate.timer <<'EOF'
[Unit]
Description=Replicate VMs every hour

[Timer]
OnCalendar=hourly
Persistent=true
RandomizedDelaySec=300

[Install]
WantedBy=timers.target
EOF

systemctl daemon-reload
systemctl enable --now vm-replicate.timer

Failover: bring up a replicated VM on the DR host

# On the DR host — the datasets are already there from replication
zfs list -r rpool/vms
# rpool/vms/db-1   18.7G   ...

# Register the VM with libvirt on the DR host
virt-install --name db-1 --ram 4096 --vcpus 4 \
  --disk /var/lib/libvirt/images/db-1/db-1.qcow2 \
  --os-variant centos-stream9 --network bridge=br0 \
  --boot uefi --import --noautoconsole

# The VM boots with data from the last replication snapshot.
# RPO (Recovery Point Objective) = your replication interval (1 hour with the timer above)
# RTO (Recovery Time Objective) = time to run virt-install + boot (under 2 minutes)

Migrate VMs between hosts

# === Near-zero-downtime migration with ZFS ===

# Pre-stage: send bulk data while VM is running
zfs snapshot rpool/vms/web-1@pre-migrate
zfs send rpool/vms/web-1@pre-migrate | ssh other-host zfs receive rpool/vms/web-1

# Cutover: shut down, send the tiny delta, boot on new host
virsh shutdown web-1
zfs snapshot rpool/vms/web-1@migrate-final
zfs send -i @pre-migrate rpool/vms/web-1@migrate-final | ssh other-host zfs receive rpool/vms/web-1
# The incremental send is only the blocks that changed since pre-stage

# Define and start on the new host
ssh other-host "virt-install --name web-1 --ram 2048 --vcpus 2 \
  --disk /var/lib/libvirt/images/web-1/web-1.qcow2 \
  --os-variant centos-stream9 --network bridge=br0 \
  --boot uefi --import --noautoconsole"

# Clean up on the old host
virsh undefine web-1 --nvram

# === True live migration (requires shared storage or pre-copy) ===
virsh migrate --live web-1 qemu+ssh://other-host/system

What KVM gets from native ZFS on root

Troubleshooting

# VM won't start — check logs
virsh start web-1 2>&1
journalctl -u libvirtd --since "5 minutes ago"
cat /var/log/libvirt/qemu/web-1.log

# Permission denied on disk
ls -la /var/lib/libvirt/images/web-1.qcow2
# Should be owned by qemu:qemu (CentOS) or libvirt-qemu:kvm (Debian)
chown qemu:qemu /var/lib/libvirt/images/web-1.qcow2  # CentOS
chown libvirt-qemu:kvm /var/lib/libvirt/images/web-1.qcow2  # Debian

# SELinux blocking (CentOS only)
ausearch -m avc --ts recent
restorecon -Rv /var/lib/libvirt/images/