| pick your distro, get ZFS on root
kldload — your platform, your way, free
Source
Tutorials › Storage

kupgrade — safe system upgrades with automatic rollback

kupgrade wraps the native package manager upgrade with a ZFS safety net. It works on both dnf (CentOS/RHEL) and apt (Debian) distros. One command. If the upgrade produces a broken system, roll back to exactly where you were in under a minute.

The problem with raw upgrades: dnf upgrade -y or apt dist-upgrade work fine until they do not. A new kernel, a dependency conflict, or a ZFS DKMS build failure can leave your system unbootable. On a plain ext4 system there is no quick way back. On ZFS with kldload, there is.

What kupgrade adds: a boot environment snapshot before the upgrade starts, a DKMS verification pass after packages are installed, and a clear rollback path if anything fails.

What kupgrade does, step by step

  1. Check pool health. If rpool is DEGRADED or FAULTED, kupgrade refuses to proceed. Upgrading a degraded pool risks data loss.
  2. Create boot environment snapshot. Clones rpool/ROOT/default to rpool/ROOT/pre-upgrade-YYYYMMDD-HHMMSS. This is instantaneous.
  3. Run the upgrade. Calls dnf upgrade -y or apt-get dist-upgrade -y depending on the distro. Full output is shown in real time and logged to /var/log/kupgrade.log.
  4. Remove orphaned packages. dnf autoremove or apt autoremove cleans up dependencies that are no longer needed.
  5. Verify ZFS DKMS for every installed kernel. Enumerates all kernel versions under /lib/modules/ and checks that zfs.ko exists and loads cleanly for each one.
  6. Re-sign DKMS modules if Secure Boot is active. Checks the MOK database and re-runs mokutil signing if needed.
  7. Print a summary report. Lists what changed, any warnings, and the rollback command if you need it.

Running kupgrade

kupgrade

Full expected output on a CentOS system:

[kupgrade] checking pool health... rpool ONLINE
[kupgrade] creating boot environment: pre-upgrade-20260325-091530
[kupgrade] snapshot: rpool/ROOT/pre-upgrade-20260325-091530 (instant)

[kupgrade] running: dnf upgrade -y
Last metadata expiration check: 0:12:34 ago.
Dependencies resolved.
================================================================================
 Package                Arch    Version                  Repository      Size
================================================================================
Upgrading:
 kernel                 x86_64  5.14.0-570.el9           baseos          7.0 M
 kernel-core            x86_64  5.14.0-570.el9           baseos          18 M
 zfs                    x86_64  2.2.8-1.el9              zfs             812 k
 zfs-dkms               noarch  2.2.8-1.el9              zfs             2.2 M
 ... (14 more packages)

Transaction Summary
================================================================================
Upgrade  17 Packages

Complete!

[kupgrade] running: dnf autoremove -y
No packages marked for removal.

[kupgrade] verifying ZFS DKMS modules...
  kernel 5.14.0-570.el9.x86_64 ... building zfs DKMS module
  DKMS: build completed.
  kernel 5.14.0-570.el9.x86_64 ... zfs.ko OK
  kernel 5.14.0-503.el9.x86_64 ... zfs.ko OK (pre-existing)

[kupgrade] Secure Boot: not active, skipping module signing

[kupgrade] DONE
  upgraded:  17 packages
  new kernel: 5.14.0-570.el9 (reboot required)
  ZFS DKMS:  OK for all kernels
  boot env:  pre-upgrade-20260325-091530 (rollback available)

To roll back:
  kbe activate pre-upgrade-20260325-091530 && reboot

What happens if DKMS fails

ZFS on Linux uses DKMS (Dynamic Kernel Module Support) to build zfs.ko for each installed kernel. If a new kernel ships and the ZFS DKMS build fails — due to a missing header, a compile error, or an incompatible kernel API — your system will not be able to mount ZFS pools after the next reboot.

kupgrade catches this before you reboot:

[kupgrade] verifying ZFS DKMS modules...
  kernel 5.14.0-580.el9.x86_64 ... building zfs DKMS module
  ERROR: DKMS build failed for kernel 5.14.0-580.el9.x86_64
  build log: /var/lib/dkms/zfs/2.2.8/build/make.log

[kupgrade] DKMS FAILURE — upgrade complete but ZFS may not load on new kernel

  DO NOT REBOOT until this is resolved, or use the rollback:
  kbe activate pre-upgrade-20260325-091530 && reboot

  Manual fix options:
  1. Wait for a new ZFS DKMS package update and re-run kupgrade
  2. Pin the old kernel: dnf install kernel-5.14.0-570.el9.x86_64
  3. Roll back: kbe activate pre-upgrade-20260325-091530 && reboot

The old kernel is still installed and still has working zfs.ko. You can boot into it explicitly from ZFSBootMenu by pressing E at the boot menu and selecting the previous kernel version. This gives you a working system while you wait for a DKMS fix.

Manual rollback

If you have already rebooted into a broken system:

# From a working shell on the broken system
kbe activate pre-upgrade-20260325-091530
reboot

# From ZFSBootMenu (if the system does not boot at all):
# Press any key at the ZFSBootMenu timeout
# Arrow to the pre-upgrade BE
# Press Enter to boot it

If the pool does not import at all, boot from the kldload ISO and use krecovery:

# Boot from USB, then:
krecovery import rpool
krecovery list-be
krecovery activate pre-upgrade-20260325-091530
reboot

kupgrade vs raw dnf/apt upgrade

Raw dnf upgrade -y

Upgrades packages. If a new kernel breaks ZFS DKMS, you discover this after rebooting into a system that cannot mount its root filesystem. Recovery requires booting from external media and doing manual ZFS imports.

no snapshot · no DKMS check · no rollback path

kupgrade

Creates a boot environment snapshot first. Runs the same upgrade. Verifies DKMS for every kernel before you reboot. If anything fails, prints the exact rollback command. Recovery is one command and a reboot.

snapshot → upgrade → DKMS verify → report

Both run the same underlying package manager commands. kupgrade is not magic — it is the same upgrade with a safety net on each side. The cost is a few extra seconds for the snapshot and DKMS check. The benefit is a system you can always get back to.

Debian / apt mode

kupgrade auto-detects the package manager. On Debian-based systems:

[kupgrade] detected: apt (Debian trixie)
[kupgrade] creating boot environment: pre-upgrade-20260325-092100

[kupgrade] running: apt-get update
Hit:1 http://deb.debian.org/debian trixie InRelease
...

[kupgrade] running: apt-get dist-upgrade -y
Reading package lists... Done
Building dependency tree... Done
The following packages will be upgraded:
  linux-image-6.14.0-1-amd64 linux-headers-6.14.0-1-amd64
  zfs-dkms zfsutils-linux libzfs4linux (5)

[kupgrade] running: apt-get autoremove -y

[kupgrade] verifying ZFS DKMS modules...
  kernel 6.14.0-1-amd64 ... zfs.ko OK
  kernel 6.13.0-1-amd64 ... zfs.ko OK

[kupgrade] DONE — reboot when ready

Scheduling and automation

kupgrade is safe to run unattended. It exits non-zero if DKMS fails, so you can gate automation on the result:

# Cron: run kupgrade weekly, alert on failure
0 3 * * 0 /usr/local/bin/kupgrade >> /var/log/kupgrade.log 2>&1 || \
  echo "kupgrade failed — check /var/log/kupgrade.log" | mail -s "kupgrade FAILED" root

Boot environments accumulate over time. Clean them up monthly:

# List BEs older than 30 days
kbe list

# Destroy old ones
kbe destroy pre-upgrade-20260101-120000