30+ tools that make ZFS and Linux feel like they were always meant to work this way.
These are optional quality-of-life shortcuts — not requirements. You never have to use them.
Under the hood they just abstract the differences between apt, dnf, pacman, and zypper
so you don't have to care which distro you're on. For example, kpkg automatically snapshots before upgrading,
runs the right package manager for your distro, and creates a boot environment — one command instead of five.
That's it: automation for the boring stuff. Use them or don't. Your distro's native tools work exactly the same as always.
Note: These tools are included in the desktop and server profiles only. The core profile ships nothing but ZFS on root, ZFSBootMenu, and a stock distro — no k-commands, no extras, no opinions.
The philosophy: single-purpose scripts, the Unix way
Every kldload tool is a single bash script that does one thing. ksnap manages snapshots.
kvm-create creates VMs. kdf shows disk usage. No tool tries to do everything.
They are composable — pipe the output of one into the input of another. They are transparent —
cat $(which ksnap) shows you exactly what it does. They are replaceable — if you outgrow
ksnap, you use zfs snapshot directly. The native commands always work.
The design contract: Every kldload tool is a bash script you can read in under two minutes.
No compiled binaries. No vendor SDKs. No obfuscated code. No telemetry. No phone-home. If you don't trust a tool,
read it. If you don't like a tool, change it. If you don't need a tool, delete it. These scripts live in
/usr/local/bin/ and they belong to you. The source code is the documentation.
The consistency guarantee: Every tool works the same on CentOS, Debian, Ubuntu, Fedora,
Rocky, RHEL, Arch, and Alpine. kpkg install nginx calls apt on Debian,
dnf on CentOS, pacman on Arch. ksnap works identically on all of them
because ZFS is ZFS regardless of the distro above it. The abstraction layer is the kernel, not the tool.
These tools exist because ZFS is powerful but verbose. zfs list -t snapshot -r -o name,used,refer,creation -S creation rpool is correct. It's also 70 characters nobody wants to type at 3 AM. ksnap list does the same thing. The native commands still work — these are shortcuts, not replacements. If you outgrow them, you're using the native commands directly. That's the goal.
Every tool auto-elevates to root when it needs to — they call exec sudo "$0" "$@" at the top if $EUID != 0. You don't need to remember which commands need sudo. You just type the command. If it needs root, it asks for your password once. If it doesn't, it runs as your user. No surprises.
Single purpose
Each tool does one thing well. ksnap manages snapshots. kvm-create creates VMs.
kdf shows disk usage. No Swiss Army knives. Small tools that compose.
Distro-agnostic
Every tool auto-detects the underlying package manager and distro. Write scripts once, run them on any kldload system. The tool handles the translation.
Transparent
Every tool is a readable bash script at /usr/local/bin/. No binaries. No black boxes.
cat $(which ksnap) shows every line of logic.
Safe by default
Destructive tools require confirmation. kvm-delete won't remove a running VM without --force.
kpkg snapshots before every operation. Rollback is always one command away.
ZFS tools
ZFS is the foundation. Every other tool on this page builds on it. These tools make the most
common ZFS operations — snapshots, clones, disk usage, dataset creation — fast enough
that you'll do them reflexively. The goal is to make ZFS feel as natural as ls and cd.
ksnap — snapshot management
ksnap is the tool you'll use most. It creates, lists, rolls back, and destroys ZFS snapshots
with short commands and sensible defaults. No flags to memorize. Run ksnap with no arguments
and it snapshots every key dataset on the system. Run ksnap list and it shows all snapshots
sorted by creation time with human-readable ages and space usage.
ksnap resolves the path to its ZFS dataset, creates a timestamped snapshot, and confirms.Example output — ksnap list:
$ ksnap list SNAPSHOT CREATED USED -------- ------- ---- rpool/ROOT/default@kpkg-20260404-091522 Sat Apr 4 9:15 2026 148K rpool/ROOT/default@auto-20260404-090000 Sat Apr 4 9:00 2026 88K rpool/home/alice@ksnap-20260404-083012 Sat Apr 4 8:30 2026 12.4M rpool/srv@ksnap-20260404-080000 Sat Apr 4 8:00 2026 2.1M rpool/ROOT/default@kpkg-20260403-221845 Fri Apr 3 22:18 2026 64K rpool/home/alice@ksnap-20260403-170000 Fri Apr 3 17:00 2026 8.7M
The naming convention matters. kpkg- prefixed snapshots were created by the package manager before an install or upgrade. ksnap- prefixed ones were created manually. auto- prefixed ones were created by the systemd timer. When you're looking at a list of 50 snapshots at 3 AM trying to figure out which one to roll back to, those prefixes tell you exactly what happened before each snapshot was taken. That's not an accident.
kclone — instant copy-on-write clones
kclone creates ZFS clones — instant copies that share all blocks with the source until
either one writes new data. A clone of a 500GB dataset takes milliseconds and uses zero extra disk space
initially. This is the single most powerful operation in ZFS and kclone makes it a one-liner.
Example session:
$ kclone /srv/postgres /srv/postgres-staging [kclone] Snapshotting rpool/srv/postgres@clone-20260404-101530 [kclone] Cloning to rpool/srv/postgres-staging [kclone] Mounted at /srv/postgres-staging [kclone] Done. Clone uses 0 bytes until data diverges. $ ls /srv/postgres-staging/ base global pg_commit_ts pg_dynshmem pg_logical pg_multixact ... $ zfs list -o name,used,refer rpool/srv/postgres rpool/srv/postgres-staging NAME USED REFER rpool/srv/postgres 42.3G 42.3G rpool/srv/postgres-staging 12K 42.3G
Look at that output. The staging copy references 42.3 GB but uses 12 KB. That's copy-on-write. Every block is shared. The moment you modify a row in the staging database, ZFS copies only that one block. Everything else remains shared. This is why ZFS clones are the best testing mechanism in existence — you can clone a terabyte database in under a second, run destructive tests, and throw away the clone without ever touching the original. Try doing that with cp -r.
kdf — ZFS-aware disk usage
The standard df command doesn't understand ZFS properly. It shows the pool's total size for every
dataset, which is misleading. kdf shows what you actually need: per-dataset usage, available space,
compression ratio, quotas, and mountpoints. Color-coded so problems jump out.
$ kdf DATASET USED AVAIL RATIO QUOTA MOUNTPOINT ─────── ──── ───── ───── ───── ────────── rpool/ROOT/default 8.2G 412.5G 1.82x - / rpool/home 1.1G 412.5G 1.24x - /home rpool/home/alice 892.0M 412.5G 1.31x 50G /home/alice rpool/home/bob 204.0M 412.5G 1.18x 50G /home/bob rpool/srv 42.6G 412.5G 2.14x - /srv rpool/srv/postgres 42.3G 412.5G 2.17x - /srv/postgres rpool/var/log 318.0M 412.5G 4.82x 10G /var/log rpool/vms 82.1G 412.5G 1.45x - /vms rpool/vms/webserver 40.2G 412.5G 1.52x - - rpool/vms/database 41.9G 412.5G 1.38x - -
The RATIO column is the compression ratio. 4.82x on /var/log means
ZFS is storing log files at nearly 5:1 compression — 318 MB of real disk space holds what would be
1.5 GB uncompressed. Green means excellent compression (>1.5x). Yellow means moderate. The numbers tell
you whether compression is doing its job without running a separate query.
kdir — create datasets instead of directories
kdir is mkdir for ZFS. Instead of creating a plain directory, it creates a
ZFS dataset mounted at that path. The dataset gets its own snapshot timeline, its own
compression ratio, its own quota if you set one, its own encryption key if you want one. It transforms
a path from a dumb folder into an independent storage domain.
rpool/home/alice. Alice's home is now an independent dataset with its own snapshots, quota, and compression.mkdir -p but every level is a ZFS dataset.du cron jobs, no quota daemons — ZFS enforces it at the block layer.adduser alice on a kldload system automatically calls kdir under the hood via the adduser.local hook. Every new user gets a ZFS dataset, not a directory. That means Alice's home has its own snapshot timeline from day one. You can roll back her home to yesterday without touching anyone else's. You can replicate her home to another machine with zfs send. You can set a quota so she can't fill the pool. Try doing any of that with mkdir /home/alice.
KVM virtualization tools
The kvm-* tools wrap virsh, virt-install, and zfs
into a coherent VM lifecycle. Create a VM with a ZFS zvol as its disk. Clone it instantly.
Snapshot it atomically. Delete it cleanly. List them all with ZFS stats. Every operation that
would normally require 5-10 commands across two different tools becomes a single command with
sensible defaults.
kvm-create — create a VM on a ZFS zvol
kvm-create handles the full VM creation pipeline: creates the ZFS zvol, sets up the
libvirt domain with virt-install, configures virtio disk, networking, and console access.
Sensible defaults for everything, flags for everything you'd want to customize.
rpool/vms/myvm, bridged networking on br0.Example session:
$ sudo kvm-create webserver --ram 4096 --cpus 4 --disk 40 --os centos-stream9 [kvm-create] Creating ZFS zvol: rpool/vms/webserver (40G) [kvm-create] zvol created at /dev/zvol/rpool/vms/webserver [kvm-create] Creating VM with virt-install... Name: webserver RAM: 4096 MB vCPUs: 4 Disk: /dev/zvol/rpool/vms/webserver (40G, virtio) Network: bridge=br0, model=virtio Console: serial + VNC [kvm-create] VM created and started. [kvm-create] Connect: virsh console webserver
The disk is a ZFS zvol, not a qcow2 file. No double copy-on-write. ZFS handles compression, checksumming, and snapshots at the storage layer. QEMU just reads and writes raw blocks. This is the correct architecture for KVM on ZFS.
kvm-clone — instant copy-on-write VM clones
kvm-clone creates a full copy of a VM in under one second. The clone shares all blocks
with the source via ZFS copy-on-write — a 200GB VM clones in the time it takes to type the command.
The clone gets a new MAC address, a new UUID, and its own libvirt domain. It's a fully independent VM.
template VM to webserver-01. Creates a snapshot, clones the zvol, generates a new libvirt domain. Under one second.$ sudo kvm-clone golden web-1 [kvm-clone] Snapshotting rpool/vms/golden@clone-20260404-110000 [kvm-clone] Cloning zvol: rpool/vms/golden -> rpool/vms/web-1 [kvm-clone] Generating new MAC address: 52:54:00:a3:7b:22 [kvm-clone] Creating libvirt domain: web-1 [kvm-clone] Done. web-1 ready to start. $ sudo kvm-clone golden web-2 [kvm-clone] Cloning zvol: rpool/vms/golden -> rpool/vms/web-2 [kvm-clone] Done. web-2 ready to start. $ zfs list -o name,used,refer rpool/vms/golden rpool/vms/web-1 rpool/vms/web-2 NAME USED REFER rpool/vms/golden 8.2G 8.2G rpool/vms/web-1 0B 8.2G rpool/vms/web-2 0B 8.2G
Zero bytes used. Both clones reference 8.2 GB but store nothing new. Every block is shared with the golden template. The first time web-1 writes a new block — a log entry, a config change, an updated package — only that one block is copied. Everything else stays shared. This is why ZFS clones make VM sprawl a non-issue: 50 VMs from the same template use barely more disk space than one VM. Compare to VMware linked clones (30-120 seconds each) or full copies (minutes, double the disk). ZFS does this in kernel space, atomically, in under a second. Every time.
kvm-snap — snapshot a running VM
kvm-snap takes an atomic ZFS snapshot of a VM's zvol. The VM keeps running — the snapshot
is crash-consistent at the block level. Create, list, rollback, and delete VM snapshots with one tool.
myvm's zvol. The VM stays running. Completes in milliseconds.$ sudo kvm-snap webserver [kvm-snap] Snapshotting rpool/vms/webserver@20260404-111530 [kvm-snap] Done. $ sudo kvm-snap webserver list SNAPSHOT CREATED USED rpool/vms/webserver@20260404-111530 Sat Apr 4 11:15 2026 0B rpool/vms/webserver@20260404-080000 Sat Apr 4 8:00 2026 42.1M rpool/vms/webserver@pre-upgrade-20260403 Fri Apr 3 22:00 2026 1.8G $ sudo kvm-snap webserver rollback @pre-upgrade-20260403 [kvm-snap] VM webserver must be stopped for rollback. [kvm-snap] Shutting down webserver... [kvm-snap] Rolling back rpool/vms/webserver to @pre-upgrade-20260403 [kvm-snap] Done. Start with: virsh start webserver
kvm-delete — clean VM removal
kvm-delete removes a VM and its ZFS zvol in one step. It undefines the libvirt domain,
destroys all snapshots and clones of the zvol, and removes the zvol itself. Clean, complete, no orphaned
resources. Requires the VM to be stopped unless --force is specified.
$ sudo kvm-delete testbox [kvm-delete] Undefining libvirt domain: testbox [kvm-delete] Destroying ZFS zvol: rpool/vms/testbox (3 snapshots) [kvm-delete] Done. VM and all storage removed.
kvm-list — VM inventory with ZFS stats
kvm-list shows every VM on the system with its state, RAM, vCPUs, and ZFS zvol usage.
That last column is the killer feature — it tells you how much real disk space each VM is consuming,
including compression. No other VM listing tool shows you this because no other hypervisor stores VMs on ZFS.
$ sudo kvm-list NAME STATE RAM VCPUS ZVOL USED ---- ----- --- ----- --------- golden shut off 4096MB 4 8.2G web-1 running 4096MB 4 342M web-2 running 4096MB 4 289M web-3 running 4096MB 4 156M database running 8192MB 8 41.9G monitoring running 2048MB 2 3.1G ZFS pool space: NAME USED AVAIL rpool/vms 54.0G 412.5G
web-1 through web-3 are all clones of the golden template. The template uses 8.2G. The three running clones together use about 787M of additional space — that's just the data each one has written since being cloned. Three VMs running, total disk usage for all of them combined: about 9G. Without ZFS clones, three copies of the same template would use 24.6G. Disk savings: 63%. And that's with only three clones. At 50 clones from the same template, the savings are over 95%. This is not a gimmick. This is how production hypervisors should work.
System tools
kst — system health at a glance
kst is the first thing you run when you SSH into a box. One command shows pool health,
disk usage, snapshot counts, boot environment status, service status, memory, and uptime. Everything
a sysadmin needs to know in one screen. Color-coded — green means healthy, yellow means watch,
red means act now.
$ kst
kldloadOS 1.0.3 webserver.lab.internal
────────────────────────────────────────────────────────
Pool Health
● rpool ONLINE no errors scrub: 2 days ago
Disk Usage
Root (/) 8.2G used of 420.7G (2%)
Home 1.1G used
Srv 42.6G used
Snapshots 47 total (12 auto, 28 kpkg, 7 manual)
Boot Environment rpool/ROOT/default (active)
Compression 1.82x average (saving 152G across pool)
Services
● sshd active ● zfs-zed active
● wireguard@wg0 active ● sanoid.timer active
● libvirtd active ● kldload-rag active
Memory 3.1G / 16G (19%) ARC: 4.2G (target: 8G)
Uptime 47 days, 3:22
The ARC line is important. ZFS uses unused RAM as a read cache (the ARC). kst shows you both
system memory usage and ARC size so you can tell the difference between "the system is using 12GB" and
"the system is using 3GB and ZFS is caching 9GB." That distinction matters. ARC memory is released
immediately when applications need it.
kst-dashboard — live tmux monitoring
kst-dashboard launches a tmux session with four panes: system health (kst on refresh),
ZFS pool/snapshot status, btop for CPU/memory/disk/network, and service logs. One command
gives you a full monitoring dashboard that stays open. Detach with Ctrl-a d, reattach anytime.
tmux attach -t dashboard.Layout: ┌───���─────────────────┬──────────────────┐ │ │ ZFS pools │ │ kst (health) │──────────────────│ │ │ snapshots │ ├─────────────────────┼──────────────────│ │ btop │ services │ └─────────────────────┴──────────────────┘
kpkg — universal package manager
kpkg detects whether you're on a Debian-based system (apt), an RPM-based
system (dnf), or Arch (pacman), and runs the right command. But the real value
is the automatic ZFS snapshot before every operation. Install a package that breaks something?
ksnap rollback / fixes it. The snapshot is free — it's literally a pointer, takes zero time,
uses zero space until data diverges.
apt search, dnf search, or pacman -Ss depending on distro.$ kpkg install nginx [kpkg] Detected package manager: dnf [kpkg] Creating snapshot: rpool/ROOT/default@kpkg-20260404-143022 [kpkg] Running: dnf install -y nginx CentOS Stream 9 - BaseOS 4.2 MB/s | 12 MB 00:02 ... Installed: nginx-1:1.24.0-4.el9.x86_64 Complete! [kpkg] Done. Rollback: ksnap rollback /
The implementation is 60 lines of bash. It detects the package manager with three command -v checks, snapshots root if ZFS is available, then dispatches to the native command. That's it. No abstraction layer. No configuration files. No daemon. The beauty is in the simplicity — kpkg is a convenience wrapper, not a replacement. If dnf exits non-zero, kpkg exits non-zero. If you need dnf-specific flags, pass them through. The tool gets out of your way.
kupgrade — safe system upgrades with boot environments
kupgrade combines a boot environment snapshot with a full system upgrade. It creates a
named boot environment before the upgrade so you can boot back into the old system if the upgrade
breaks anything. This is the Linux equivalent of Solaris/FreeBSD boot environments — the feature
that makes OS upgrades a non-event.
$ sudo kupgrade [kupgrade] Creating boot environment: pre-upgrade-20260404 [kupgrade] Snapshot: rpool/ROOT/default@pre-upgrade-20260404 [kupgrade] Running full system upgrade... [kpkg] Detected package manager: dnf [kpkg] Running: dnf upgrade -y ... [kupgrade] Upgrade complete. 47 packages updated. [kupgrade] Reboot when ready. Rollback: kbe activate pre-upgrade-20260404
kexport — golden image export
kexport converts this system's disk to a portable image file: qcow2 (KVM/Proxmox/OpenStack),
vmdk (VMware), vhd (Azure/Hyper-V), ova (VirtualBox/VMware portable),
or raw (dd-ready). Uses qemu-img convert under the hood. The output is a
compressed, portable image you can import into any hypervisor.
/root/exports/ by default.
$ sudo kexport /dev/vda qcow2
[kexport] Source disk: /dev/vda (40G)
[kexport] Format: qcow2 (compressed)
[kexport] Output: /root/exports/kldload-export-20260404-1430.qcow2
[kexport] Converting... (this may take a few minutes)
(28.00/40.00 GB, 70.0%)
[kexport] Done. Size: 6.8G (83% compression)
[kexport] Import into Proxmox:
qm importdisk 100 kldload-export-20260404-1430.qcow2 local-zfs
kimage — golden image lifecycle
kimage is the full golden image pipeline in one tool. It seals the current system for cloning
(clears machine-id, removes SSH host keys, enables cloud-init), exports the disk image, and optionally
stamps out N VMs from it. The entire Packer workflow in a single command.
$ sudo kimage build [kimage] Sealing system for cloning... [kimage] Clearing machine-id: /etc/machine-id [kimage] Removing SSH host keys: /etc/ssh/ssh_host_* [kimage] Enabling cloud-init datasources: NoCloud, ConfigDrive, GCE, EC2 [kimage] Exporting ZFS pools... [kimage] System sealed. Ready for export. $ sudo kimage export qcow2 /srv/images [kimage] Auto-detected root disk: /dev/vda [kimage] Exporting to: /srv/images/kldload-golden-20260404.qcow2 [kimage] Done. 6.2G compressed image ready.
Network tools
kldload installs WireGuard on every system and configures up to four WireGuard planes automatically: wg0 for enrollment, wg1 for management (SSH, Salt), wg2 for Kubernetes backend traffic, and wg3 for storage replication. The firstboot service generates keypairs, writes configs, and brings up the tunnels. No manual WireGuard configuration required.
WireGuard planes
wg0 — enrollment
Bootstrap plane. The first tunnel established during firstboot. Used only for initial node enrollment and key exchange. Never bind services here — it's unreliable on some hardware during early boot.
wg1 — management
SSH, Salt master, monitoring agents. All management traffic flows over this encrypted plane. Even if the LAN is compromised, management commands travel through WireGuard.
wg2 — Kubernetes backend
Pod-to-pod traffic, etcd, API server. Kubernetes control plane and data plane traffic isolated on its own encrypted WireGuard tunnel. Separate from management, separate from storage.
wg3 — storage replication
ZFS send/receive traffic. syncoid replication runs over this plane. Storage traffic
never competes with management or application traffic. Dedicated bandwidth, dedicated encryption.
The planes are configured by kldload-firstboot on the installed system. The hub node's
WireGuard public keys are baked into /etc/kldload/hub.env during installation. Each node
generates its own keypairs in /etc/kldload/secrets/ and brings up all four tunnels on first boot.
$ wg show
interface: wg0
public key: aB3xY...base64...
private key: (hidden)
listening port: 51820
peer: Qm7zR...hub-pubkey...
endpoint: 192.168.1.1:51820
allowed ips: 10.77.0.0/16
latest handshake: 12 seconds ago
transfer: 1.42 MiB received, 892.31 KiB sent
persistent keepalive: every 25 seconds
interface: wg1
public key: cD4wZ...base64...
listening port: 51821
...
$ ip addr show wg1
5: wg1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN
inet 10.78.0.3/16 scope global wg1
AI tools
kldload ships a local AI assistant that understands your infrastructure. It runs entirely on your machine — no cloud API keys, no data leaving your network. The RAG (Retrieval-Augmented Generation) service indexes kldload documentation and your system's state, then answers questions with context from both. Ask it how to configure ZFS encryption and it considers your actual pool layout.
kai — AI infrastructure assistant
kai is the query interface. Ask it anything about your infrastructure and it combines
documentation knowledge with live system context — pool status, memory, services — to give
you relevant answers. If the RAG service is running, it uses retrieval-augmented generation for
documentation-grounded responses. If not, it falls back to the base model with system context.
zpool status output, identifies the failed device, and explains recovery steps specific to your pool topology.$ kai 'how do I replace a failed disk in my mirror?' Based on your pool status (rpool: DEGRADED, /dev/sdb FAULTED): 1. Identify the replacement disk: $ lsblk # find your new disk, e.g., /dev/sdc 2. Replace the faulted device: $ sudo zpool replace rpool /dev/sdb /dev/sdc 3. Monitor the resilver: $ watch zpool status rpool The resilver will run in the background. Your pool remains online and accessible during the entire process. Current pool size: 465G, estimated resilver time: ~45 minutes at current I/O rates. Sources: ZFS Admin Guide ch.4, kldload docs/tutorials/multi-disk.html
kai-rag — RAG service
kai-rag is the RAG query tool that talks to the local RAG service on port 8400.
It collects system context (pool status, memory, services) and sends it alongside your question
to the RAG endpoint. The RAG service retrieves relevant documentation chunks and feeds them to the
model along with your system state. This is what makes answers specific to your machine.
kai-index — document indexing
kai-index indexes documentation into the RAG service. By default it indexes the kldload
documentation at /usr/local/share/kldload-ai/html/. You can point it at any directory
of HTML or text files to make the AI knowledgeable about your own documentation, runbooks, or playbooks.
kai can answer questions about your internal procedures.
$ kai-index /srv/docs/runbooks
Indexing documentation from: /srv/docs/runbooks
{
"indexed": 47,
"skipped": 3,
"total_chunks": 892,
"source_dir": "/srv/docs/runbooks"
}
Done. Test with: kai 'what is OpenZFS?'
The entire AI stack runs locally. The RAG service is a Python process. The model runs via Ollama. The document index is a local vector store. No API keys, no cloud endpoints, no data leaving your network. Ask it about your ZFS pool and it reads zpool status output in real time. Ask it about PostgreSQL tuning and it considers your actual RAM. This is what AI assistants should be — tools that understand your specific environment, not generic chatbots that give you Stack Overflow answers.
Installer tools
The kldload installer is two components: a web UI (Python + HTML) that collects user choices, and a backend script (bash) that executes the installation. They communicate via WebSocket. The backend sources nine library files that handle ZFS, bootloader, bootstrapping, profiles, security, networking, logging, and distro-specific package management.
kldload-webui — browser-based installer
kldload-webui is a single Python file that serves the installer web interface and exposes
a WebSocket API. The browser drives the entire install — select a distro, choose a profile,
configure networking, pick an export format. Every action the browser can do, you can also do from
a terminal. The web UI is a convenience layer, not a requirement.
The web UI is a single HTML file per edition (/usr/local/share/kldload-webui/free/index.html
for the free edition). No JavaScript frameworks. No npm dependencies. No build step. One HTML file with
inline CSS and vanilla JS. It loads instantly, works on any browser, and communicates with the backend
over a local WebSocket connection.
kldload-install-target — the installation backend
kldload-install-target is the script that actually installs the operating system. It reads
an answers file (environment variables), partitions the disk, creates ZFS pools, bootstraps the target
distro, installs packages from darksites, configures the bootloader, sets up WireGuard, and seals the
system for first boot. It sources nine libraries from /usr/lib/kldload-installer/lib/:
dnf --installroot (RPM distros), debootstrap (Debian/Ubuntu), or pacstrap (Arch)./var/log/kldload/install.log on the target.The installer is modular for a reason. Adding a new distro means writing one function in bootstrap.sh that calls the distro's package manager, one function in bootloader.sh that configures the bootloader, and one package list file. The rest — ZFS pool creation, WireGuard, profile tools — is shared. That's how we went from 4 distros to 8 in two releases. The architecture makes new distros a weekend project, not a rewrite.
kldload-firstboot — post-install initialization
kldload-firstboot runs once on the first boot of an installed system. It reads the install
manifest, generates WireGuard keypairs for all four planes, writes WireGuard configs, brings up tunnels,
registers with the hub node, and performs role-specific setup (Salt minion enrollment, Kubernetes node join,
etc.). After firstboot completes, the node is fully operational and connected to the fleet.
Supporting tools
kldload-snapshot.timer.
$ sudo kldload-recovery import
pool: rpool
id: 13284920387412309
state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier.
config:
rpool ONLINE
nvme0n1p3 ONLINE
$ sudo kldload-recovery list-datasets
NAME USED AVAIL REFER MOUNTPOINT
rpool 52.1G 412G 192K /
rpool/ROOT 8.2G 412G 192K none
rpool/ROOT/default 8.2G 412G 8.14G /
rpool/home 1.1G 412G 192K /home
...
Package management and darksites
kldload is designed for offline installation. The ISO carries complete package mirrors ("darksites") for every supported RPM and APT distro. When you install CentOS, every package comes from the local mirror baked into the ISO — no internet required. Arch Linux is the exception: as a rolling-release distro, it requires network access to pull current packages.
How darksites work
RPM darksites
CentOS, Fedora, RHEL, Rocky packages are pre-downloaded with dnf download --resolve --alldeps
during the ISO build. Stored at /root/darksite/ on the live ISO. The installer points
dnf --installroot at this local repo. File-based — no HTTP server needed.
Debian darksite
Debian packages are resolved and downloaded inside a debian:trixie-slim container
during ISO build. Served on port 3142 on the live ISO. The installer configures
debootstrap to use http://localhost:3142/debian/ as its mirror.
Ubuntu darksite
Ubuntu packages are built the same way as Debian, including the universe component
(required for ZFS packages). Served on port 3143 on the live ISO.
Package lists are plain text files in the build tree. Adding a package to the darksite means adding one line to a text file and rebuilding the ISO. Dependencies resolve automatically.
# Adding a package to the darksite: $ echo "htop" >> build/darksite/config/package-sets/base.txt $ echo "htop" >> build/darksite-debian/config/package-sets/base.txt $ echo "htop" >> build/darksite-ubuntu/config/package-sets/base.txt # Rebuild the ISO (darksites rebuild incrementally if cached) $ PROFILE=server ./deploy.sh build # On the installed system, the package is available offline: $ kpkg install htop # works without internet
The darksite architecture is why kldload works in air-gapped environments. The ISO is completely self-contained for RPM and Debian/Ubuntu distros. Boot from USB, install CentOS with ZFS on root, reboot into a working system — never touch the internet. The darksite cache persists between builds, so rebuilding the ISO doesn't re-download 10GB of packages. Only new or updated packages are fetched. The first build is slow (downloading everything). Every subsequent build is fast (incremental). That's the correct trade-off.
Boot environment management
Boot environments are the killer feature. Snapshot your entire OS, upgrade, and if it breaks, reboot
into the old version. Fifteen seconds. No reinstall. This is the Linux equivalent of what FreeBSD
and illumos have had for years with beadm. On Linux, ZFSBootMenu makes it possible.
kldload's boot environment tools make it effortless.
BSD has had boot environments for years — beadm on FreeBSD, beadm on illumos. It's the reason sysadmins on those platforms laugh at the concept of "I upgraded and now it won't boot." On Linux, this was impossible until ZFSBootMenu. kldload's kbe commands are the Linux equivalent of what BSD admins have had for a decade. The difference: on FreeBSD it's built in. On Linux, someone had to build it for you.
$ kbe list NAME CREATED SIZE ACTIVE rpool/ROOT/default 2026-03-15 09:00 8.2G active (N/R) rpool/ROOT/pre-upgrade 2026-04-03 22:00 8.1G - rpool/ROOT/stable-march 2026-03-20 14:30 7.9G - # Before a risky upgrade: $ kbe create pre-kernel-upgrade [kbe] Created: rpool/ROOT/default@pre-kernel-upgrade # If the upgrade breaks boot: $ kbe activate pre-kernel-upgrade [kbe] Next boot: rpool/ROOT/pre-kernel-upgrade $ reboot
Snapshot automation
Snapshots happen automatically. You don't have to remember. The most important snapshot is the one you forgot to take — that's why kldload takes them for you.
kpkg-./srv) snapshotted every 15 minutes via systemd timer. Retention: last 4. Prefix: auto-.kldload-snapshot.timer. Always have a known-good state. Prefix: auto-.The retention policy is the other half. Unlimited snapshots sounds great until the pool fills up with thousands of them holding old blocks. kldload's defaults keep the last 10 package snapshots, last 4 service snapshots, and last 48 hourly snapshots. Old ones are pruned automatically by sanoid. You get a rolling window of recovery points without thinking about it.
System administration
adduser alice creating a ZFS dataset instead of a directory is the kind of thing that changes how you think about users. Alice's home isn't a folder on the root filesystem. It's an independent storage domain with its own compression, its own quota, its own snapshot timeline, its own encryption key if you want one. You can replicate Alice's home to another machine with zfs send. You can roll back her home to yesterday without touching anyone else's. You can set a 50GB quota so she can't fill the pool. Try doing any of that with mkdir /home/alice.
adduser.local hook. Proper ownership, proper permissions, proper isolation.kldload-apply-platform-holds.kldload-help.
$ kldload-help
╔═══════════════════════════════════════════════════════════════════════════╗
║ kldloadOS — Command Reference ║
╚═══════════════════════════════════════════════════════════════════════════╝
── kldload Tools ───────────────────────────────────────────────────────────
kst — System health at a glance
Pool health, disk usage, snapshots, boot environments, services.
$ kst
kst-dashboard — Live tmux monitoring dashboard
Four-pane view: health, pools, btop, services. Auto-refreshes.
$ kst-dashboard # launch and attach
$ kst-dashboard --kill # stop the dashboard
ksnap — ZFS snapshot manager
Create, list, rollback, and destroy snapshots in one command.
$ ksnap # snapshot all key datasets
$ ksnap list # show all snapshots with ages
$ ksnap /home/admin # snapshot a specific path
$ ksnap rollback /home/admin # roll back to last snapshot
...
Shell aliases and functions
The kldload .bashrc loads 50+ aliases and functions automatically. They are context-aware
— Kubernetes aliases only appear if kubectl is installed. Docker aliases only appear
if docker is installed. Helm, Salt, virsh, ZFS — each group is gated by a
command -v check. You never get "command not found" from an alias pointing at a missing tool.
ZFS
zls — list datasets
zsnap — list snapshots
zbe — list boot environments
Kubernetes
k, kgp, kgn, kgs — get pods/nodes/svc
kns — switch namespace
ksh <ns> <pod> — shell into pod
kshow <ns> — full namespace overview
Infrastructure
vls, vstart, vstop — virsh shortcuts
dps, dcu, dcd — Docker shortcuts
ports — listening ports (ss -tuln)
Helm
h — helm
hls — list all releases
hs <rel> — release status
hv <rel> — release values
hown <rel> — all resources owned by release
Salt (fleet management)
slist — table of all minions with OS, IPs, CPUs, RAM, roles
sping — test.ping all nodes
skeys — list/accept Salt keys
sknodes / skpods — K8s via Salt
Modern replacements
ls → eza (git status, icons, tree)
cat → bat (syntax highlighting)
find → fd (fast, ergonomic)
grep → rg (ripgrep, fast)
cd → z (zoxide, learns dirs)
Kubernetes functions in detail
The Kubernetes functions go beyond simple aliases. kshow gives you a complete namespace
overview — workloads, services, ingress, config, RBAC, storage, and recent events in one command.
ksh handles both pod names and label selectors. cmd launches a
netshoot debug pod on the same node as the target pod. These are the functions that save
you 10 minutes of typing during an incident.
$ kshow production
== ns: production ==
── workloads ──
NAME READY UP-TO-DATE AVAILABLE AGE NODE
deploy/web-frontend 3/3 3 3 47d worker-02
deploy/api-backend 2/2 2 2 47d worker-01
sts/postgres 1/1 1 1 47d worker-03
── services/ingress ──
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
svc/web-frontend ClusterIP 10.96.45.22 <none> 80/TCP
svc/api-backend ClusterIP 10.96.45.23 <none> 8080/TCP
svc/postgres ClusterIP 10.96.45.24 <none> 5432/TCP
── events (last 25) ──
(no warnings or errors)
$ ksh production app=api-backend
root@api-backend-7c8f5d9b4-x2k9m:/# curl localhost:8080/health
{"status":"ok","uptime":4082400}
$ cmd production app=api-backend
# Launches netshoot debug pod on same node
root@dbg-api-backend-12345:/# tcpdump -i any port 8080
...
Salt fleet management
If Salt is installed (hub/master nodes), the slist function gives you a table of every
minion in the fleet with hostname, OS, IP addresses, CPU count, RAM, and roles. One command shows
your entire infrastructure. The Salt aliases make fleet-wide operations one-liners.
$ slist MINION HOST OS IPs CPUs RAM ROLES web-01.lab.internal web-01 CentOS Stream 9 10.78.0.3 192.168.1.103 4 8192 worker web-02.lab.internal web-02 CentOS Stream 9 10.78.0.4 192.168.1.104 4 8192 worker db-01.lab.internal db-01 Debian 13 10.78.0.5 192.168.1.105 8 32768 worker,storage hub.lab.internal hub CentOS Stream 9 10.78.0.1 192.168.1.100 8 16384 master,etcd
Terminal tools
kldload bundles a curated set of modern terminal tools. These aren't kldload-specific — they're best-in-class open-source tools that every sysadmin should have. kldload pre-installs them, configures them, and aliases the traditional commands to their modern replacements. The originals are still available if you need them.
Ctrl-a. Split panes with | and -. Move with Alt+arrows. Ctrl-a S shows kst health. Ctrl-a D launches dashboard.Ctrl-R for history search, Ctrl-T for file finder, Alt-C for directory jump. Pipe anything into fzf to pick interactively.cat. Syntax highlighting, line numbers, git diff markers. catp for paged output. Also used as the man pager.ls. Git status integration, file icons, tree view. ll for long list with git status. tree for recursive tree view.find replacement. Faster, ergonomic syntax. fd '\.conf$' finds all .conf files. Respects .gitignore.grep. Extremely fast content search. Respects .gitignore, handles binary files, recursive by default.cd that learns your directories. z logs jumps to /var/log. zi opens interactive picker with fzf.
# tmux keybindings (prefix: Ctrl-a)
C-a | split horizontal C-a - split vertical
Alt+arrow move between panes Shift+arrow resize panes
Alt+1-9 switch windows Alt+h/l prev/next window
C-a z zoom pane (toggle) C-a d detach session
C-a S kst health C-a D launch dashboard
C-a H kldload-help C-a r reload config
# fzf — the most useful tool you've never used
$ vim $(fzf) # pick a file to edit
$ kpkg install $(apt list 2>/dev/null | fzf) # pick a package
$ ssh $(grep Host ~/.ssh/config | awk '{print $2}' | fzf) # pick a host
kldload tools vs manual commands
Every kldload tool saves typing. Some save a few keystrokes. Some save entire multi-step workflows. Here's what each tool replaces and how much work it does for you.
| kldload tool | Manual equivalent | Chars saved | Steps saved |
|---|---|---|---|
ksnap list |
zfs list -t snapshot -r -o name,used,refer,creation -S creation rpool |
60+ | Flag memorization |
ksnap /home |
ds=$(df --output=source /home | tail -1); zfs snapshot "${ds}@manual-$(date +%Y%m%d-%H%M%S)" |
80+ | 2 commands → 1 |
kdf |
zfs list -o name,used,avail,compressratio,quota,mountpoint -r rpool |
60+ | Color-coded output |
kpkg install nginx |
zfs snapshot rpool/ROOT/default@pre-install-$(date +%s); dnf install -y nginx |
70+ | Snapshot + install |
kvm-create myvm |
zfs create -V 20G rpool/vms/myvm; virt-install --name myvm --ram 2048 --vcpus 2 --disk /dev/zvol/rpool/vms/myvm --network bridge=br0 --os-variant generic --graphics none --console pty,target_type=serial |
150+ | 2 tools → 1 |
kvm-clone tmpl web-1 |
zfs snapshot rpool/vms/tmpl@clone; zfs clone rpool/vms/tmpl@clone rpool/vms/web-1; virt-clone --original tmpl --name web-1 --file /dev/zvol/rpool/vms/web-1 --mac RANDOM |
140+ | 3 commands → 1 |
kvm-snap myvm |
zfs snapshot rpool/vms/myvm@$(date +%Y%m%d-%H%M%S) |
40+ | Consistent naming |
kvm-list |
virsh list --all; for vm in $(virsh list --all --name); do zfs list rpool/vms/$vm 2>/dev/null; done |
80+ | Merged view |
kvm-delete myvm |
virsh shutdown myvm; virsh undefine myvm --nvram; zfs destroy -r rpool/vms/myvm |
60+ | 3 commands → 1 |
kst |
zpool status; zfs list; zfs list -t snapshot | wc -l; systemctl --no-pager list-units --state=active; free -h; uptime |
100+ | 6 commands → 1 |
kdir /srv/app |
zfs create rpool/srv/app; zfs set mountpoint=/srv/app rpool/srv/app |
50+ | Auto mountpoint |
kclone /srv/db /srv/db-test |
ds=$(df --output=source /srv/db | tail -1); zfs snapshot ${ds}@clone-$(date +%s); zfs clone ${ds}@clone-$(date +%s) rpool/srv/db-test |
100+ | 3 commands → 1 |
kexport /dev/vda qcow2 |
qemu-img convert -f raw -O qcow2 -c /dev/vda /root/exports/image.qcow2 |
40+ | Auto naming + format hints |
The character count isn't the point. The point is cognitive load. kvm-create myvm --ram 4096 --cpus 4 --disk 40 is readable English. The manual equivalent requires knowing ZFS zvol syntax, virt-install flag names, the exact path to the zvol device node, and the correct virtio configuration. That's four different domains of knowledge for one operation. The kldload tool compresses those four domains into flags you can guess: --ram, --cpus, --disk. You don't have to know that the zvol appears at /dev/zvol/rpool/vms/myvm. You don't have to remember --console pty,target_type=serial. The tool knows.
Tool architecture
Understanding where tools live and how they're installed helps when you want to customize, extend, or audit them. The layout is intentionally simple — standard FHS paths, no custom framework, no hidden config files.
Directory layout
kst, ksnap, kdf, kdir, kpkg, kclone, kexport, kimage, kvm-create, kvm-clone, kvm-snap, kvm-delete, kvm-list, kai, kai-rag, kai-index, kldload-help, kldload-webui.kldload-install-target (installer backend), kldload-firstboot (first-boot init), kldload-recovery (emergency recovery), kldload-snapshot (timer-driven snapshots), kldload-apply-platform-holds (package holds).common.sh, answers.sh, storage-zfs.sh, bootstrap.sh, bootloader.sh, profiles.sh, security.sh, infra.sh, logging.sh.free/index.html, active/index.html. Single HTML file with inline CSS and vanilla JS.install-manifest.env (install-time choices), hub.env (hub WireGuard keys), secrets/ (WireGuard keypairs, held mode 0700).How tools are installed
During the ISO build, everything under live-build/config/includes.chroot/ in the source tree
is mirrored directly into the live ISO's root filesystem. The tools at includes.chroot/usr/local/bin/
land at /usr/local/bin/ on the ISO. During installation, profiles.sh copies the
profile-appropriate tools from the live environment into the target root filesystem. The core profile
skips all kldload tools. The server and desktop profiles copy everything.
Extending and customizing tools
Every tool is a bash script with no external dependencies beyond standard coreutils and ZFS. To customize:
# Read the source: $ cat $(which ksnap) # 80 lines of bash. No mysteries. # Edit in place (your machine, your rules): $ sudo vim $(which ksnap) # Create your own tool: $ sudo vim /usr/local/bin/mycheck #!/usr/bin/env bash set -euo pipefail echo "Pool: $(zpool get -H -o value health rpool)" echo "Snaps: $(zfs list -t snapshot -H | wc -l)" echo "Disk: $(zfs get -H -o value used rpool)" $ sudo chmod +x /usr/local/bin/mycheck $ mycheck Pool: ONLINE Snaps: 47 Disk: 52.1G
The tools are deliberately simple so you can modify them. ksnap is 80 lines. kvm-create is about 120 lines. kpkg is 60 lines. These aren't complex programs. They're the kind of scripts a competent sysadmin writes in an afternoon. The difference is that we wrote them once, tested them across 8 distros, and included them in every install. You get the benefit without the afternoon of work. And when you need something these tools don't do, you have the native commands and the source code as a starting point.
Quick reference — every tool at a glance
Print this section. Pin it to your monitor. These are the commands you'll reach for daily.
| Tool | Purpose | Common usage |
|---|---|---|
kst |
System health overview | kst |
kst-dashboard |
Live tmux monitoring | kst-dashboard |
ksnap |
ZFS snapshot management | ksnap / ksnap list / ksnap rollback /path |
kclone |
Instant CoW clones | kclone /src /dst |
kdf |
ZFS-aware disk usage | kdf |
kdir |
Create ZFS datasets | kdir /path / kdir -p /deep/path / kdir -o quota=50G /path |
kpkg |
Universal package manager | kpkg install pkg / kpkg upgrade / kpkg search term |
kupgrade |
Safe system upgrade | kupgrade |
kexport |
Export disk image | kexport /dev/vda qcow2 / kexport /dev/sda all |
kimage |
Golden image lifecycle | kimage build / kimage export / kimage full 10 |
kvm-create |
Create VM on ZFS zvol | kvm-create name --ram 4096 --cpus 4 --disk 40 |
kvm-clone |
Instant VM clone | kvm-clone template new-vm |
kvm-snap |
Snapshot VM zvol | kvm-snap name / kvm-snap name list / kvm-snap name rollback |
kvm-delete |
Remove VM + zvol | kvm-delete name / kvm-delete name --force |
kvm-list |
VM inventory + ZFS stats | kvm-list |
kai |
AI infrastructure assistant | kai 'your question' |
kai-rag |
RAG-powered queries | kai-rag 'your question' |
kai-index |
Index documentation | kai-index / kai-index /path/to/docs |
kbe |
Boot environments | kbe list / kbe create name / kbe activate name |
khelp |
Built-in reference | khelp |
khold |
Hold critical packages | khold |
kpoof |
Scrub sensitive data | kpoof |
krecovery |
Emergency recovery | krecovery import / krecovery list-pools |
kldload-webui |
Browser installer UI | kldload-webui / kldload-webui --port 8080 |
kldload-test |
Smoke tests | kldload-test auto / kldload-test server |
ksnap works, cat $(which ksnap).
That's the point.
No compiled binaries. That's not laziness — it's a security decision. Every tool on this system is a bash script you can read, audit, and modify. cat $(which ksnap) shows you exactly what it does. No obfuscated binary. No vendor SDK phoning home. No "trust us." If you don't like how kpkg handles snapshots, open it in vim and change it. It's your machine. These are your tools. The source code is the documentation.