Documentation

Package Management

kldload does not replace or modify your distro’s package manager. apt, dnf, pacman — they all work exactly as they do on a stock install. Use them directly if you prefer.

kpkg is an optional convenience wrapper that calls the native package manager underneath and adds automatic ZFS snapshots before every operation. It works identically across CentOS, Debian, Ubuntu, Fedora, Rocky, RHEL, and Arch.

The real feature isn’t the wrapper. It’s the snapshot.

On a stock distro, apt install nginx installs nginx. If nginx’s postinst script breaks your Apache config, removes a library, or conflicts with something you didn’t expect — your options are: apt remove nginx and hope, manually fix what broke, or reinstall the OS. There is no undo.

On kldload, kpkg install nginx snapshots the entire root filesystem before the install runs. If anything goes wrong — anything at all — ksnap rollback / reverts the filesystem to the exact state before the install. Not just nginx. Every file that changed, every library that was updated, every config that was modified. Atomic undo of the entire operation. This is what package management looks like on a copy-on-write filesystem.

You can also do this manually: zfs snapshot rpool/ROOT/default@before-nginx && apt install nginx. The k-tool just does it automatically so you don’t have to remember.

Basic operations

# Install a package
kpkg install nginx

# Remove a package
kpkg remove nginx

# Search for a package
kpkg search redis

# Show package info
kpkg info nginx

# Update package lists
kpkg update

# Upgrade all packages
kpkg upgrade

# List installed packages
kpkg list

Every install, remove, and upgrade creates a ZFS snapshot first (kpkg-YYYYMMDD-HHMMSS). If the operation breaks something, roll back:

# List kpkg snapshots
ksnap list | grep kpkg

# Roll back
ksnap rollback /

Darksites are the offline superpower. Most people think of kldload as "ZFS on root installer." The darksites are the other half. Every ISO ships with complete package mirrors — RPM for CentOS/Rocky/Fedora, APT for Debian/Ubuntu, pacman for Arch, apk for Alpine. Thousands of packages pre-resolved with full dependency trees. Install on a submarine, a factory floor, a classified facility — no internet, no DNS, no upstream mirror. The ISO is the mirror.

After install, the darksites stay on the target system. You can install additional packages from the darksite without internet. When you're ready to connect to the internet, the normal vendor repos are pre-configured — dnf update or apt update works immediately.

Offline package install from the darksite

kldload bakes complete package mirrors into the ISO. After install, these mirrors are available at /root/darksite/.

CentOS/RHEL — RPM darksite

The RPM mirror is pre-configured as a local repo:

# Already configured — just install
dnf install tmux   # pulls from /root/darksite/rpm/

# Check available repos
dnf repolist

If the local repo isn’t configured:

cat > /etc/yum.repos.d/kldload-darksite.repo << 'EOF'
[kldload-darksite]
name=kldload offline mirror
baseurl=file:///root/darksite/rpm/
enabled=1
gpgcheck=0
EOF

Debian — APT darksite

The APT mirror runs as a local HTTP server on port 3142:

# Check if the mirror service is running
systemctl status kldload-apt-mirror

# If not running, start it
systemctl start kldload-apt-mirror

# Verify
curl -s http://localhost:3142/dists/trixie/Release | head -5

The sources list is pre-configured:

cat /etc/apt/sources.list.d/kldload-darksite.list
# deb [trusted=yes] http://localhost:3142 trixie main

System upgrades with kupgrade

kupgrade is a safe upgrade tool that creates a boot environment snapshot before upgrading:

kupgrade

What it does: 1. Creates pre-upgrade-YYYYMMDD-HHMMSS boot environment 2. Runs apt-get update + apt-get dist-upgrade (Debian) or dnf upgrade (CentOS) 3. Runs apt-get autoremove to clean up 4. Verifies ZFS DKMS modules built for every installed kernel 5. Re-signs DKMS modules with MOK key if Secure Boot is enabled

If the upgrade breaks something:

# Reboot, select the pre-upgrade boot environment from ZFSBootMenu
# Or from the command line:
kbe activate pre-upgrade-20260321-143000
reboot

kupgrade solves the most common ZFS-on-root failure mode. A kernel upgrade without a matching ZFS module rebuild = a system that won't boot. On stock Linux, apt upgrade updates the kernel, DKMS tries to rebuild ZFS, and if it fails silently — you find out at 3 AM after an unattended reboot. kupgrade does the upgrade, explicitly rebuilds DKMS for every installed kernel, verifies the ZFS module exists, re-signs it for Secure Boot if needed, and only then considers the upgrade complete. If any step fails, the pre-upgrade boot environment is one reboot away. This isn't paranoia. This is the 3 AM scenario that breaks real systems, automated away.

Adding packages to the darksite (build time)

To include additional packages in future ISO builds, add them to the package list files:

For CentOS/RHEL installs

# Base packages (all installs)
echo "htop" >> build/darksite/config/package-sets/target-base.txt

# Desktop-only packages
echo "gimp" >> build/darksite/config/package-sets/target-desktop.txt

# Server-only packages
echo "postgresql-server" >> build/darksite/config/package-sets/target-server.txt

For Debian/Ubuntu installs

# Debian packages
echo "htop" >> build/darksite-debian/config/package-sets/target-base.txt

# Ubuntu packages (separate darksite, ubuntu-specific package names)
echo "htop" >> build/darksite-ubuntu/config/package-sets/target-base.txt

For Arch installs

# Arch packages
echo "htop" >> build/darksite-arch/config/package-sets/target-base.txt

Dependencies resolve automatically when you rebuild:

./deploy.sh build-debian-darksite   # rebuild Debian APT mirror
./deploy.sh build-ubuntu-darksite   # rebuild Ubuntu APT mirror
./deploy.sh build-arch-darksite     # rebuild Arch pacman mirror
PROFILE=desktop ./deploy.sh build   # rebuild ISO (RPM darksite rebuilds automatically)

Package differences between distros

Task	CentOS/Rocky/Fedora/RHEL	Debian/Ubuntu	Arch
Install	`dnf install pkg`	`apt install pkg`	`pacman -S pkg`
Search	`dnf search pkg`	`apt search pkg`	`pacman -Ss pkg`
List installed	`dnf list installed`	`dpkg -l`	`pacman -Q`
Show deps	`dnf deplist pkg`	`apt depends pkg`	`pactree pkg`
Clean cache	`dnf clean all`	`apt clean`	`pacman -Sc`
Upgrade all	`dnf upgrade`	`apt dist-upgrade`	`pacman -Syu`
eBPF tools	`bcc-tools`	`bpfcc-tools`	`bcc-tools`

kpkg picks the right command automatically across all three families. But the native commands always work. kpkg is a shortcut, not a gate.

This table is why kpkg exists. Three package managers, three syntaxes, three ways to do the same thing. If you manage CentOS servers and Debian servers — or worse, if you’re writing automation that runs on both — you’re maintaining two code paths for every package operation. kpkg install htop works on all of them. More importantly, it snapshots before every operation on all of them. One command, one behavior, every distro. The wrapper is trivial. The consistency saves you from remembering whether it’s dpkg -l or rpm -qa at 3 AM.

← kst — System status The answers file →