50+ tools that make ZFS, KVM, Kubernetes, and Linux feel like they were always meant to work this way.
These are optional quality-of-life shortcuts — not requirements. You never have to use them.
Under the hood they just abstract the differences between apt, dnf, pacman, and zypper
so you don't have to care which distro you're on. For example, kpkg automatically snapshots before upgrading,
runs the right package manager for your distro, and creates a boot environment — one command instead of five.
That's it: automation for the boring stuff. Use them or don't. Your distro's native tools work exactly the same as always.
Note on profile inclusion: The full toolchain ships with Desktop, Server, KVM Host, Kubernetes, klab, and OpenZFS Suite profiles. Kubernetes, klab, and OpenZFS Suite layer additional kube-*, klab, and kzfs-* tools on top of the Server set. The Core profile ships nothing but ZFS on root, ZFSBootMenu, and a stock distro — no k* commands, no extras, no opinions.
The philosophy: single-purpose scripts, the Unix way
Every kldload tool is a single bash script that does one thing. ksnap manages snapshots.
kvm-create creates VMs. kdf shows disk usage. No tool tries to do everything.
They are composable — pipe the output of one into the input of another. They are transparent —
cat $(which ksnap) shows you exactly what it does. They are replaceable — if you outgrow
ksnap, you use zfs snapshot directly. The native commands always work.
The design contract: Every kldload tool is a bash script you can read in under two minutes.
No compiled binaries. No vendor SDKs. No obfuscated code. No telemetry. No phone-home.
Read it. Change it. Delete it. These scripts live in
/usr/local/bin/ and they belong to you. The source code is the documentation.
The consistency guarantee: Every tool works the same on CentOS, Debian, Ubuntu, Fedora,
Rocky, RHEL, Arch, and Alpine. kpkg install nginx calls apt on Debian,
dnf on CentOS, pacman on Arch. ksnap works identically on all of them
because ZFS is ZFS regardless of the distro above it. The abstraction layer is the kernel, not the tool.
These tools exist because ZFS is powerful but verbose. zfs list -t snapshot -r -o name,used,refer,creation -S creation rpool is correct. It's also 70 characters nobody wants to type at 3 AM. ksnap list does the same thing. The native commands still work — these are shortcuts, not replacements. If you outgrow them, you're using the native commands directly. That's the goal.
Every tool auto-elevates to root when it needs to — they call exec sudo "$0" "$@" at the top if $EUID != 0. You don't need to remember which commands need sudo. You just type the command. If it needs root, it asks for your password once. If it doesn't, it runs as your user. No surprises.
Single purpose
Each tool does one thing well. ksnap manages snapshots. kvm-create creates VMs.
kdf shows disk usage. No Swiss Army knives. Small tools that compose.
Distro-agnostic
Every tool auto-detects the underlying package manager and distro. Write scripts once, run them on any kldload system. The tool handles the translation.
Transparent
Every tool is a readable bash script at /usr/local/bin/. No binaries. No black boxes.
cat $(which ksnap) shows every line of logic.
Safe by default
Destructive tools require confirmation. kvm-delete won't remove a running VM without --force.
kpkg snapshots before every operation. Rollback is always one command away.
ZFS tools
ZFS is the foundation. Every other tool on this page builds on it. These tools make the most
common ZFS operations — snapshots, clones, disk usage, dataset creation — fast enough
that you'll do them reflexively. The goal is to make ZFS feel as natural as ls and cd.
ksnap — snapshot management
ksnap is the tool you'll use most. It creates, lists, rolls back, and destroys ZFS snapshots
with short commands and sensible defaults. No flags to memorize. Run ksnap with no arguments
and it snapshots every key dataset on the system. Run ksnap list and it shows all snapshots
sorted by creation time with human-readable ages and space usage.
ksnap resolves the path to its ZFS dataset, creates a timestamped snapshot, and confirms.Example output — ksnap list:
$ ksnap list SNAPSHOT CREATED USED -------- ------- ---- rpool/ROOT/default@kpkg-20260404-091522 Sat Apr 4 9:15 2026 148K rpool/ROOT/default@auto-20260404-090000 Sat Apr 4 9:00 2026 88K rpool/home/alice@ksnap-20260404-083012 Sat Apr 4 8:30 2026 12.4M rpool/srv@ksnap-20260404-080000 Sat Apr 4 8:00 2026 2.1M rpool/ROOT/default@kpkg-20260403-221845 Fri Apr 3 22:18 2026 64K rpool/home/alice@ksnap-20260403-170000 Fri Apr 3 17:00 2026 8.7M
The naming convention matters. kpkg- prefixed snapshots were created by the package manager before an install or upgrade. ksnap- prefixed ones were created manually. auto- prefixed ones were created by the systemd timer. When you're looking at a list of 50 snapshots at 3 AM trying to figure out which one to roll back to, those prefixes tell you exactly what happened before each snapshot was taken. That's not an accident.
kclone — instant copy-on-write clones
kclone creates ZFS clones — instant copies that share all blocks with the source until
either one writes new data. A clone of a 500GB dataset takes milliseconds and uses zero extra disk space
initially. This is the single most powerful operation in ZFS and kclone makes it a one-liner.
Example session:
$ kclone /srv/postgres /srv/postgres-staging [kclone] Snapshotting rpool/srv/postgres@clone-20260404-101530 [kclone] Cloning to rpool/srv/postgres-staging [kclone] Mounted at /srv/postgres-staging [kclone] Done. Clone uses 0 bytes until data diverges. $ ls /srv/postgres-staging/ base global pg_commit_ts pg_dynshmem pg_logical pg_multixact ... $ zfs list -o name,used,refer rpool/srv/postgres rpool/srv/postgres-staging NAME USED REFER rpool/srv/postgres 42.3G 42.3G rpool/srv/postgres-staging 12K 42.3G
Look at that output. The staging copy references 42.3 GB but uses 12 KB. That's copy-on-write. Every block is shared. The moment you modify a row in the staging database, ZFS copies only that one block. Everything else remains shared. This is why ZFS clones are the best testing mechanism in existence — you can clone a terabyte database in under a second, run destructive tests, and throw away the clone without ever touching the original. Try doing that with cp -r.
kdf — ZFS-aware disk usage
The standard df command doesn't understand ZFS properly. It shows the pool's total size for every
dataset, which is misleading. kdf shows what you actually need: per-dataset usage, available space,
compression ratio, quotas, and mountpoints. Color-coded so problems jump out.
$ kdf DATASET USED AVAIL RATIO QUOTA MOUNTPOINT ─────── ──── ───── ───── ───── ────────── rpool/ROOT/default 8.2G 412.5G 1.82x - / rpool/home 1.1G 412.5G 1.24x - /home rpool/home/alice 892.0M 412.5G 1.31x 50G /home/alice rpool/home/bob 204.0M 412.5G 1.18x 50G /home/bob rpool/srv 42.6G 412.5G 2.14x - /srv rpool/srv/postgres 42.3G 412.5G 2.17x - /srv/postgres rpool/var/log 318.0M 412.5G 4.82x 10G /var/log rpool/vms 82.1G 412.5G 1.45x - /vms rpool/vms/webserver 40.2G 412.5G 1.52x - - rpool/vms/database 41.9G 412.5G 1.38x - -
The RATIO column is the compression ratio. 4.82x on /var/log means
ZFS is storing log files at nearly 5:1 compression — 318 MB of real disk space holds what would be
1.5 GB uncompressed. Green means excellent compression (>1.5x). Yellow means moderate. The numbers tell
you whether compression is doing its job without running a separate query.
kdir — create datasets instead of directories
kdir is mkdir for ZFS. Instead of creating a plain directory, it creates a
ZFS dataset mounted at that path. The dataset gets its own snapshot timeline, its own
compression ratio, its own quota if you set one, its own encryption key if you want one. It transforms
a path from a dumb folder into an independent storage domain.
rpool/home/alice. Alice's home is now an independent dataset with its own snapshots, quota, and compression.mkdir -p but every level is a ZFS dataset.du cron jobs, no quota daemons — ZFS enforces it at the block layer.adduser alice on a kldload system automatically calls kdir under the hood via the adduser.local hook. Every new user gets a ZFS dataset, not a directory. That means Alice's home has its own snapshot timeline from day one. You can roll back her home to yesterday without touching anyone else's. You can replicate her home to another machine with zfs send. You can set a quota so she can't fill the pool. Try doing any of that with mkdir /home/alice.
KVM virtualization tools
The kvm-* tools wrap virsh, virt-install, and zfs
into a coherent VM lifecycle. Create a VM with a ZFS zvol as its disk. Clone it instantly.
Snapshot it atomically. Delete it cleanly. List them all with ZFS stats. Every operation that
would normally require 5-10 commands across two different tools becomes a single command with
sensible defaults.
kvm-create — create a VM on a ZFS zvol
kvm-create handles the full VM creation pipeline: creates the ZFS zvol, sets up the
libvirt domain with virt-install, configures virtio disk, networking, and console access.
Sensible defaults for everything, flags for everything you'd want to customize.
rpool/vms/myvm, bridged networking on br0.Example session:
$ sudo kvm-create webserver --ram 4096 --cpus 4 --disk 40 --os centos-stream9 [kvm-create] Creating ZFS zvol: rpool/vms/webserver (40G) [kvm-create] zvol created at /dev/zvol/rpool/vms/webserver [kvm-create] Creating VM with virt-install... Name: webserver RAM: 4096 MB vCPUs: 4 Disk: /dev/zvol/rpool/vms/webserver (40G, virtio) Network: bridge=br0, model=virtio Console: serial + VNC [kvm-create] VM created and started. [kvm-create] Connect: virsh console webserver
The disk is a ZFS zvol, not a qcow2 file. No double copy-on-write. ZFS handles compression, checksumming, and snapshots at the storage layer. QEMU just reads and writes raw blocks. This is the correct architecture for KVM on ZFS.
kvm-clone — instant copy-on-write VM clones
kvm-clone creates a full copy of a VM in under one second. The clone shares all blocks
with the source via ZFS copy-on-write — a 200GB VM clones in the time it takes to type the command.
The clone gets a new MAC address, a new UUID, and its own libvirt domain. It's a fully independent VM.
template VM to webserver-01. Creates a snapshot, clones the zvol, generates a new libvirt domain. Under one second.$ sudo kvm-clone golden web-1 [kvm-clone] Snapshotting rpool/vms/golden@clone-20260404-110000 [kvm-clone] Cloning zvol: rpool/vms/golden -> rpool/vms/web-1 [kvm-clone] Generating new MAC address: 52:54:00:a3:7b:22 [kvm-clone] Creating libvirt domain: web-1 [kvm-clone] Done. web-1 ready to start. $ sudo kvm-clone golden web-2 [kvm-clone] Cloning zvol: rpool/vms/golden -> rpool/vms/web-2 [kvm-clone] Done. web-2 ready to start. $ zfs list -o name,used,refer rpool/vms/golden rpool/vms/web-1 rpool/vms/web-2 NAME USED REFER rpool/vms/golden 8.2G 8.2G rpool/vms/web-1 0B 8.2G rpool/vms/web-2 0B 8.2G
Zero bytes used. Both clones reference 8.2 GB but store nothing new. Every block is shared with the golden template. The first time web-1 writes a new block — a log entry, a config change, an updated package — only that one block is copied. Everything else stays shared. This is why ZFS clones make VM sprawl a non-issue: 50 VMs from the same template use barely more disk space than one VM. Compare to VMware linked clones (30-120 seconds each) or full copies (minutes, double the disk). ZFS does this in kernel space, atomically, in under a second. Every time.
kvm-snap — snapshot a running VM
kvm-snap takes an atomic ZFS snapshot of a VM's zvol. The VM keeps running — the snapshot
is crash-consistent at the block level. Create, list, rollback, and delete VM snapshots with one tool.
myvm's zvol. The VM stays running. Completes in milliseconds.$ sudo kvm-snap webserver [kvm-snap] Snapshotting rpool/vms/webserver@20260404-111530 [kvm-snap] Done. $ sudo kvm-snap webserver list SNAPSHOT CREATED USED rpool/vms/webserver@20260404-111530 Sat Apr 4 11:15 2026 0B rpool/vms/webserver@20260404-080000 Sat Apr 4 8:00 2026 42.1M rpool/vms/webserver@pre-upgrade-20260403 Fri Apr 3 22:00 2026 1.8G $ sudo kvm-snap webserver rollback @pre-upgrade-20260403 [kvm-snap] VM webserver must be stopped for rollback. [kvm-snap] Shutting down webserver... [kvm-snap] Rolling back rpool/vms/webserver to @pre-upgrade-20260403 [kvm-snap] Done. Start with: virsh start webserver
kvm-delete — clean VM removal
kvm-delete removes a VM and its ZFS zvol in one step. It undefines the libvirt domain,
destroys all snapshots and clones of the zvol, and removes the zvol itself. Clean, complete, no orphaned
resources. Requires the VM to be stopped unless --force is specified.
$ sudo kvm-delete testbox [kvm-delete] Undefining libvirt domain: testbox [kvm-delete] Destroying ZFS zvol: rpool/vms/testbox (3 snapshots) [kvm-delete] Done. VM and all storage removed.
kvm-list — VM inventory with ZFS stats
kvm-list shows every VM on the system with its state, RAM, vCPUs, and ZFS zvol usage.
That last column is the killer feature — it tells you how much real disk space each VM is consuming,
including compression. No other VM listing tool shows you this because no other hypervisor stores VMs on ZFS.
$ sudo kvm-list NAME STATE RAM VCPUS ZVOL USED ---- ----- --- ----- --------- golden shut off 4096MB 4 8.2G web-1 running 4096MB 4 342M web-2 running 4096MB 4 289M web-3 running 4096MB 4 156M database running 8192MB 8 41.9G monitoring running 2048MB 2 3.1G ZFS pool space: NAME USED AVAIL rpool/vms 54.0G 412.5G
web-1 through web-3 are all clones of the golden template. The template uses 8.2G. The three running clones together use about 787M of additional space — that's just the data each one has written since being cloned. Three VMs running, total disk usage for all of them combined: about 9G. Without ZFS clones, three copies of the same template would use 24.6G. Disk savings: 63%. And that's with only three clones. At 50 clones from the same template, the savings are over 95%. This is not a gimmick. This is how production hypervisors should work.
Kubernetes tools
The kube-* family spins production Kubernetes on ZFS-backed zvol VMs — CRI-O,
Cilium eBPF datapath, MetalLB, local-path PV with WaitForFirstConsumer auto-binder, ArgoCD demo.
Bundled with the Kubernetes profile. Every tool is a readable bash script that orchestrates
virsh, cloud-init, kubeadm, and kubectl against
the local ZFS host — no opinionated YAML, no Helm-of-Helm-charts, just steps you can read.
kube-cluster — the one-command cluster
kube-cluster up stands up a complete K8s cluster on this host. It builds VMs from the
Kubernetes golden image, joins them, installs Cilium and MetalLB, and writes a working kubeconfig
to your home directory. Cluster sizes from 1 to N nodes; defaults to a 3-node cluster (control plane + 2 workers).
kubectl get nodes.kube-demo — PetClinic + ArgoCD smoke test
End-to-end deployment of the PetClinic microservices reference app (customers, vets, visits, API gateway, config server, admin server, Eureka) plus ArgoCD watching the manifests. Use it as a known-good baseline to prove a fresh cluster works, then tear it down and rebuild to learn the pieces.
kube-status — cluster health summary
Quick health summary that combines kubectl get nodes, Cilium connectivity probes, the
Hubble flow rate, MetalLB advertised IPs, persistent volume binding state, and Helm release status
into a single screen. The web UI surfaces the same data.
Supporting helpers
kube-init bootstraps the control plane (called by kube-cluster).
kube-join joins a new worker.
kube-reset nukes and rebuilds the cluster from scratch.
kube-network tweaks Cilium / MetalLB / PodCIDR after the fact.
kube-load-images preloads images into the CRI-O cache from the offline darksite (essential for air-gapped clusters).
kube-smoke-test runs a battery of assertions you can fire from CI.
kube-setup configures the local kubeconfig + tab-completion.
The Kubernetes profile is a complete cluster you can read — not a black box. cat $(which kube-cluster) shows the exact virt-install commands, the cloud-init data, the kubeadm flags, and the Cilium helm values used. Want a different CNI? Replace one section. Want metallb on different VLAN? Change one variable. Same script, your decisions.
klab tools — multi-distro test platform
klab gives you golden ZFS-backed VMs of every supported distro, blue/green deploys via instant clone, fault injection, and a Distro Matrix Runner that fires the same change at every distro in parallel. Bundled with the klab profile.
klab — the umbrella command
Supporting helpers
klab-vm-debug-bundle collects everything from a VM (logs, ZFS state, package list, syslog)
into a tarball for off-host inspection.
klab-exporter is the Prometheus exporter that surfaces golden state and clone health to Grafana.
klab-prom-targets regenerates the Prometheus scrape targets file when the matrix changes.
OpenZFS Suite tools
Dedicated tooling for OpenZFS contributors and storage teams chasing regressions. Bundled with the OpenZFS Suite profile, separate from klab so you can run the upstream test suite without interfering with regular klab matrix work.
zfs build for that distro).ztest — the in-userland multi-vdev fuzz tester used to find on-disk format regressions.ztest alone won't surface.System tools
kst — system health at a glance
kst is the first thing you run when you SSH into a box. One command shows pool health,
disk usage, snapshot counts, boot environment status, service status, memory, and uptime. Everything
a sysadmin needs to know in one screen. Color-coded — green means healthy, yellow means watch,
red means act now.
$ kst
kldloadOS 1.0.3 webserver.lab.internal
────────────────────────────────────────────────────────
Pool Health
● rpool ONLINE no errors scrub: 2 days ago
Disk Usage
Root (/) 8.2G used of 420.7G (2%)
Home 1.1G used
Srv 42.6G used
Snapshots 47 total (12 auto, 28 kpkg, 7 manual)
Boot Environment rpool/ROOT/default (active)
Compression 1.82x average (saving 152G across pool)
Services
● sshd active ● zfs-zed active
● wireguard@wg0 active ● sanoid.timer active
● libvirtd active ● kldload-rag active
Memory 3.1G / 16G (19%) ARC: 4.2G (target: 8G)
Uptime 47 days, 3:22
The ARC line is important. ZFS uses unused RAM as a read cache (the ARC). kst shows you both
system memory usage and ARC size so you can tell the difference between "the system is using 12GB" and
"the system is using 3GB and ZFS is caching 9GB." That distinction matters. ARC memory is released
immediately when applications need it.
kst-dashboard — live tmux monitoring
kst-dashboard launches a tmux session with four panes: system health (kst on refresh),
ZFS pool/snapshot status, btop for CPU/memory/disk/network, and service logs. One command
gives you a full monitoring dashboard that stays open. Detach with Ctrl-a d, reattach anytime.
tmux attach -t dashboard.Layout: ┌───���─────────────────┬──────────────────┐ │ │ ZFS pools │ │ kst (health) │──────────────────│ │ │ snapshots │ ├─────────────────────┼──────────────────│ │ btop │ services │ └─────────────────────┴──────────────────┘
kpkg — universal package manager
kpkg detects whether you're on a Debian-based system (apt), an RPM-based
system (dnf), or Arch (pacman), and runs the right command. But the real value
is the automatic ZFS snapshot before every operation. Install a package that breaks something?
ksnap rollback / fixes it. The snapshot is free — it's literally a pointer, takes zero time,
uses zero space until data diverges.
apt search, dnf search, or pacman -Ss depending on distro.$ kpkg install nginx [kpkg] Detected package manager: dnf [kpkg] Creating snapshot: rpool/ROOT/default@kpkg-20260404-143022 [kpkg] Running: dnf install -y nginx CentOS Stream 9 - BaseOS 4.2 MB/s | 12 MB 00:02 ... Installed: nginx-1:1.24.0-4.el9.x86_64 Complete! [kpkg] Done. Rollback: ksnap rollback /
The implementation is 60 lines of bash. It detects the package manager with three command -v checks, snapshots root if ZFS is available, then dispatches to the native command. That's it. No abstraction layer. No configuration files. No daemon. The beauty is in the simplicity — kpkg is a convenience wrapper, not a replacement. If dnf exits non-zero, kpkg exits non-zero. If you need dnf-specific flags, pass them through. The tool gets out of your way.
kupgrade — safe system upgrades with boot environments
kupgrade combines a boot environment snapshot with a full system upgrade. It creates a
named boot environment before the upgrade so you can boot back into the old system if the upgrade
breaks anything. This is the Linux equivalent of Solaris/FreeBSD boot environments — the feature
that makes OS upgrades a non-event.
$ sudo kupgrade [kupgrade] Creating boot environment: pre-upgrade-20260404 [kupgrade] Snapshot: rpool/ROOT/default@pre-upgrade-20260404 [kupgrade] Running full system upgrade... [kpkg] Detected package manager: dnf [kpkg] Running: dnf upgrade -y ... [kupgrade] Upgrade complete. 47 packages updated. [kupgrade] Reboot when ready. Rollback: kbe activate pre-upgrade-20260404
kexport — golden image export
kexport converts this system's disk to a portable image file: qcow2 (KVM/Proxmox/OpenStack),
vmdk (VMware), vhd (Azure/Hyper-V), ova (VirtualBox/VMware portable),
or raw (dd-ready). Uses qemu-img convert under the hood. The output is a
compressed, portable image you can import into any hypervisor.
/root/exports/ by default.
$ sudo kexport /dev/vda qcow2
[kexport] Source disk: /dev/vda (40G)
[kexport] Format: qcow2 (compressed)
[kexport] Output: /root/exports/kldload-export-20260404-1430.qcow2
[kexport] Converting... (this may take a few minutes)
(28.00/40.00 GB, 70.0%)
[kexport] Done. Size: 6.8G (83% compression)
[kexport] Import into Proxmox:
qm importdisk 100 kldload-export-20260404-1430.qcow2 local-zfs
kimage — golden image lifecycle
kimage is the full golden image pipeline in one tool. It seals the current system for cloning
(clears machine-id, removes SSH host keys, enables cloud-init), exports the disk image, and optionally
stamps out N VMs from it. The entire Packer workflow in a single command.
$ sudo kimage build [kimage] Sealing system for cloning... [kimage] Clearing machine-id: /etc/machine-id [kimage] Removing SSH host keys: /etc/ssh/ssh_host_* [kimage] Enabling cloud-init datasources: NoCloud, ConfigDrive, GCE, EC2 [kimage] Exporting ZFS pools... [kimage] System sealed. Ready for export. $ sudo kimage export qcow2 /srv/images [kimage] Auto-detected root disk: /dev/vda [kimage] Exporting to: /srv/images/kldload-golden-20260404.qcow2 [kimage] Done. 6.2G compressed image ready.
Network tools
kldload installs WireGuard on every system and configures up to four WireGuard planes automatically: wg0 for enrollment, wg1 for management (SSH, Salt), wg2 for Kubernetes backend traffic, and wg3 for storage replication. The firstboot service generates keypairs, writes configs, and brings up the tunnels. No manual WireGuard configuration required.
WireGuard planes
wg0 — enrollment
Bootstrap plane. The first tunnel established during firstboot. Used only for initial node enrollment and key exchange. Never bind services here — it's unreliable on some hardware during early boot.
wg1 — management
SSH, Salt master, monitoring agents. All management traffic flows over this encrypted plane. Even if the LAN is compromised, management commands travel through WireGuard.
wg2 — Kubernetes backend
Pod-to-pod traffic, etcd, API server. Kubernetes control plane and data plane traffic isolated on its own encrypted WireGuard tunnel. Separate from management, separate from storage.
wg3 — storage replication
ZFS send/receive traffic. syncoid replication runs over this plane. Storage traffic
never competes with management or application traffic. Dedicated bandwidth, dedicated encryption.
The planes are configured by kldload-firstboot on the installed system. The hub node's
WireGuard public keys are baked into /etc/kldload/hub.env during installation. Each node
generates its own keypairs in /etc/kldload/secrets/ and brings up all four tunnels on first boot.
$ wg show
interface: wg0
public key: aB3xY...base64...
private key: (hidden)
listening port: 51820
peer: Qm7zR...hub-pubkey...
endpoint: 192.168.1.1:51820
allowed ips: 10.77.0.0/16
latest handshake: 12 seconds ago
transfer: 1.42 MiB received, 892.31 KiB sent
persistent keepalive: every 25 seconds
interface: wg1
public key: cD4wZ...base64...
listening port: 51821
...
$ ip addr show wg1
5: wg1: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN
inet 10.78.0.3/16 scope global wg1
Bob — local AI assistant
Bob is the bundled local AI assistant — Ollama-served LLM with a RAG index over the kldload codebase and docs (ChromaDB + nomic-embed-text), hands-free voice control, multi-terminal awareness, packet-flow vision, and tool-use against the rest of the toolchain. Runs on the local NVIDIA GPU via CUDA time-slicing, sharing the card with whatever else is using it. No cloud, no telemetry, no data leaves your network.
kai — the Bob query CLI
kai is the typed-input front-end to Bob. Ask anything about your infrastructure and Bob
combines its RAG index over kldload docs + your codebase with live system context — pool status,
memory, running services, K8s pods, recent Hubble flows — to give you grounded answers. If you
approve, Bob can also call kldload tools directly (ksnap, kbe, kube-status,
klab) and produce real changes, not just suggestions.
zpool status output, identifies the failed device, and explains recovery steps specific to your pool topology.$ kai 'how do I replace a failed disk in my mirror?' Based on your pool status (rpool: DEGRADED, /dev/sdb FAULTED): 1. Identify the replacement disk: $ lsblk # find your new disk, e.g., /dev/sdc 2. Replace the faulted device: $ sudo zpool replace rpool /dev/sdb /dev/sdc 3. Monitor the resilver: $ watch zpool status rpool The resilver will run in the background. Your pool remains online and accessible during the entire process. Current pool size: 465G, estimated resilver time: ~45 minutes at current I/O rates. Sources: ZFS Admin Guide ch.4, kldload docs/tutorials/multi-disk.html
Voice, vision, multi-terminal
Beyond kai, Bob has three modes most assistants don't:
- Voice in/out via whisper.cpp + piper running on the same GPU. Talk to Bob while you have your hands on a console.
- Multi-terminal awareness — Bob can read tmux panes (with permission), reason across what's running in each, and propose the next step in the right pane. Great for parallel install/debug across the klab matrix.
- Packet-flow vision — Bob ingests Hubble flow data, eBPF events, and Cilium policy drops, so it can reason about what's happening on the wire, not just on the host.
kai-rag — RAG service
kai-rag is the RAG query tool that talks to the local RAG service on port 8400.
It collects system context (pool status, memory, services, K8s state) and sends it alongside your
question to the RAG endpoint. The service retrieves relevant documentation + codebase chunks
from the ChromaDB vector store and feeds them to the model along with your system state. This is
what makes Bob's answers specific to your machine instead of generic Stack Overflow paraphrasing.
kai-index — document indexing
kai-index (and kldload-rag-index for the full codebase index) feeds
documents into the RAG service. By default it indexes the kldload documentation at
/usr/local/share/kldload-ai/html/ plus the bundled kldload source tree. You can point
it at any directory of HTML, markdown, source code, or runbooks to make Bob knowledgeable about
your own infrastructure.
kai can answer questions about your internal procedures.
$ kai-index /srv/docs/runbooks
Indexing documentation from: /srv/docs/runbooks
{
"indexed": 47,
"skipped": 3,
"total_chunks": 892,
"source_dir": "/srv/docs/runbooks"
}
Done. Test with: kai 'what is OpenZFS?'
The entire AI stack runs locally. The RAG service is a Python process. The model runs via Ollama. The document index is a ChromaDB vector store with nomic-embed-text embeddings. Voice in is whisper.cpp; voice out is piper. No API keys, no cloud endpoints, no data leaving your network. Ask Bob about your ZFS pool and it reads zpool status output in real time. Ask Bob about PostgreSQL tuning and it considers your actual RAM. Ask Bob to "snapshot the rpool, then rebuild the cluster" and — with your approval — it actually does it via ksnap and kube-cluster. This is what AI assistants should be: tools that understand your specific environment and can act on it.
Console & monitoring tools
Live, keyboard-first views of what's happening on the box — complementary to the web UI for users who never leave the terminal.
kldload-console — tmux F-key cockpit
A tmux session with live eBPF panels and a help popup bound to F1. F-keys swap entire
cockpits without leaving the terminal: F2 = TCP flows (tcplife + tcpconnect),
F3 = kernel TCP state, F4 = process execs (execsnoop), F5 = disk latency
(biolatency), F6 = ZFS events, F7 = K8s events, F8 = Hubble flows.
kldload-doctor — subsystem health check
Runs through every kldload subsystem (ZFS pool health, hostid sync, DKMS state, WireGuard tunnels, K8s connectivity, Cilium agent, sanoid timer, exporters) and reports pass/fail/warning with a one-line fix suggestion per failed check. The web UI's "System Doctor" panel calls the same script.
kldload-loadtest — synthetic workload generator
Builds traffic so dashboards show signal instead of empty graphs. Configurable mix of CPU
(stress-ng), disk I/O (fio), network (iperf3, HTTP),
K8s pod churn, and ZFS snapshot churn. Use it before demos or screenshots.
kldload-collect — debug bundle
Collects logs, ZFS state, KVM state, K8s state, journals, and recent eBPF events into a single
tarball for off-host inspection or a support thread. Mirrors what klab-vm-debug-bundle
does for klab VMs, but for the host itself.
Other helpers
kldload-dash opens the web UI in your default browser.
kldload-overview prints a one-screen summary of host + pools + VMs + cluster.
kldload-follow tails the unified system log (journald + zed + kldload).
kldload-help is the CLI help index.
kldload-lh ("local host") prints the kldload box's LAN IP via ip -o addr show scope global — not getaddrinfo, so it works even when /etc/hosts only has localhost.
kldload-obs-check verifies the Prometheus + Grafana + Loki stack is healthy.
kldload-mgmt is the management-plane control script.
kldload-inventory exports the host's hardware + ZFS topology to JSON.
kldload-db manages the local SQLite state DB.
kinspect is a one-shot "what is this box?" prober.
kimage + kexport handle the golden-image lifecycle (covered above).
Installer tools
The kldload installer is two components: a web UI (Python + HTML) that collects user choices, and a backend script (bash) that executes the installation. They communicate via WebSocket. The backend sources nine library files that handle ZFS, bootloader, bootstrapping, profiles, security, networking, logging, and distro-specific package management.
kldload-webui — browser-based installer
kldload-webui is a single Python file that serves the installer web interface and exposes
a WebSocket API. The browser drives the entire install — select a distro, choose a profile,
configure networking, pick an export format. Every action the browser can do, you can also do from
a terminal. The web UI is a convenience layer, not a requirement.
The web UI is a single HTML file per edition (/usr/local/share/kldload-webui/free/index.html
for the free edition). No JavaScript frameworks. No npm dependencies. No build step. One HTML file with
inline CSS and vanilla JS. It loads instantly, works on any browser, and communicates with the backend
over a local WebSocket connection.
kldload-install-target — the installation backend
kldload-install-target is the script that actually installs the operating system. It reads
an answers file (environment variables), partitions the disk, creates ZFS pools, bootstraps the target
distro, installs packages from darksites, configures the bootloader, sets up WireGuard, and seals the
system for first boot. It sources nine libraries from /usr/lib/kldload-installer/lib/:
dnf --installroot (RPM distros), debootstrap (Debian/Ubuntu), or pacstrap (Arch)./var/log/kldload/install.log on the target.The installer is modular for a reason. Adding a new distro means writing one function in bootstrap.sh that calls the distro's package manager, one function in bootloader.sh that configures the bootloader, and one package list file. The rest — ZFS pool creation, WireGuard, profile tools — is shared. That's how we went from 4 distros to 8 in two releases. The architecture makes new distros a weekend project, not a rewrite.
kldload-firstboot — post-install initialization
kldload-firstboot runs once on the first boot of an installed system. It reads the install
manifest, generates WireGuard keypairs for all four planes, writes WireGuard configs, brings up tunnels,
registers with the hub node, and performs role-specific setup (Salt minion enrollment, Kubernetes node join,
etc.). After firstboot completes, the node is fully operational and connected to the fleet.
Supporting tools
kldload-snapshot.timer.
$ sudo kldload-recovery import
pool: rpool
id: 13284920387412309
state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier.
config:
rpool ONLINE
nvme0n1p3 ONLINE
$ sudo kldload-recovery list-datasets
NAME USED AVAIL REFER MOUNTPOINT
rpool 52.1G 412G 192K /
rpool/ROOT 8.2G 412G 192K none
rpool/ROOT/default 8.2G 412G 8.14G /
rpool/home 1.1G 412G 192K /home
...
Package management and darksites
kldload is designed for offline installation. The ISO carries complete package mirrors ("darksites") for every supported RPM and APT distro. When you install CentOS, every package comes from the local mirror baked into the ISO — no internet required. Arch Linux is the exception: as a rolling-release distro, it requires network access to pull current packages.
How darksites work
RPM darksites
CentOS, Fedora, RHEL, Rocky packages are pre-downloaded with dnf download --resolve --alldeps
during the ISO build. Stored at /root/darksite/ on the live ISO. The installer points
dnf --installroot at this local repo. File-based — no HTTP server needed.
Debian darksite
Debian packages are resolved and downloaded inside a debian:trixie-slim container
during ISO build. Served on port 3142 on the live ISO. The installer configures
debootstrap to use http://localhost:3142/debian/ as its mirror.
Ubuntu darksite
Ubuntu packages are built the same way as Debian, including the universe component
(required for ZFS packages). Served on port 3143 on the live ISO.
Package lists are plain text files in the build tree. Adding a package to the darksite means adding one line to a text file and rebuilding the ISO. Dependencies resolve automatically.
# Adding a package to the darksite: $ echo "htop" >> build/darksite/config/package-sets/base.txt $ echo "htop" >> build/darksite-debian/config/package-sets/base.txt $ echo "htop" >> build/darksite-ubuntu/config/package-sets/base.txt # Rebuild the ISO (darksites rebuild incrementally if cached) $ PROFILE=server ./deploy.sh build # On the installed system, the package is available offline: $ kpkg install htop # works without internet
The darksite architecture is why kldload works in air-gapped environments. The ISO is completely self-contained for RPM and Debian/Ubuntu distros. Boot from USB, install CentOS with ZFS on root, reboot into a working system — never touch the internet. The darksite cache persists between builds, so rebuilding the ISO doesn't re-download 10GB of packages. Only new or updated packages are fetched. The first build is slow (downloading everything). Every subsequent build is fast (incremental). That's the correct trade-off.
Boot environment management
Boot environments are the killer feature. Snapshot your entire OS, upgrade, and if it breaks, reboot
into the old version. Fifteen seconds. No reinstall. This is the Linux equivalent of what FreeBSD
and illumos have had for years with beadm. On Linux, ZFSBootMenu makes it possible.
kldload's boot environment tools make it effortless.
BSD has had boot environments for years — beadm on FreeBSD, beadm on illumos. It's the reason sysadmins on those platforms laugh at the concept of "I upgraded and now it won't boot." On Linux, this was impossible until ZFSBootMenu. kldload's kbe commands are the Linux equivalent of what BSD admins have had for a decade. The difference: on FreeBSD it's built in. On Linux, someone had to build it for you.
$ kbe list NAME CREATED SIZE ACTIVE rpool/ROOT/default 2026-03-15 09:00 8.2G active (N/R) rpool/ROOT/pre-upgrade 2026-04-03 22:00 8.1G - rpool/ROOT/stable-march 2026-03-20 14:30 7.9G - # Before a risky upgrade: $ kbe create pre-kernel-upgrade [kbe] Created: rpool/ROOT/default@pre-kernel-upgrade # If the upgrade breaks boot: $ kbe activate pre-kernel-upgrade [kbe] Next boot: rpool/ROOT/pre-kernel-upgrade $ reboot
Snapshot automation
Snapshots happen automatically. You don't have to remember. The most important snapshot is the one you forgot to take — that's why kldload takes them for you.
kpkg-./srv) snapshotted every 15 minutes via systemd timer. Retention: last 4. Prefix: auto-.kldload-snapshot.timer. Always have a known-good state. Prefix: auto-.The retention policy is the other half. Unlimited snapshots sounds great until the pool fills up with thousands of them holding old blocks. kldload's defaults keep the last 10 package snapshots, last 4 service snapshots, and last 48 hourly snapshots. Old ones are pruned automatically by sanoid. You get a rolling window of recovery points without thinking about it.
System administration
adduser alice creating a ZFS dataset instead of a directory is the kind of thing that changes how you think about users. Alice's home isn't a folder on the root filesystem. It's an independent storage domain with its own compression, its own quota, its own snapshot timeline, its own encryption key if you want one. You can replicate Alice's home to another machine with zfs send. You can roll back her home to yesterday without touching anyone else's. You can set a 50GB quota so she can't fill the pool. Try doing any of that with mkdir /home/alice.
adduser.local hook. Proper ownership, proper permissions, proper isolation.kldload-apply-platform-holds.kldload-help.
$ kldload-help
╔═══════════════════════════════════════════════════════════════════════════╗
║ kldloadOS — Command Reference ║
╚═══════════════════════════════════════════════════════════════════════════╝
── kldload Tools ───────────────────────────────────────────────────────────
kst — System health at a glance
Pool health, disk usage, snapshots, boot environments, services.
$ kst
kst-dashboard — Live tmux monitoring dashboard
Four-pane view: health, pools, btop, services. Auto-refreshes.
$ kst-dashboard # launch and attach
$ kst-dashboard --kill # stop the dashboard
ksnap — ZFS snapshot manager
Create, list, rollback, and destroy snapshots in one command.
$ ksnap # snapshot all key datasets
$ ksnap list # show all snapshots with ages
$ ksnap /home/admin # snapshot a specific path
$ ksnap rollback /home/admin # roll back to last snapshot
...
Shell aliases and functions
The kldload .bashrc loads 50+ aliases and functions automatically. They are context-aware
— Kubernetes aliases only appear if kubectl is installed. Docker aliases only appear
if docker is installed. Helm, Salt, virsh, ZFS — each group is gated by a
command -v check. You never get "command not found" from an alias pointing at a missing tool.
ZFS
zls — list datasets
zsnap — list snapshots
zbe — list boot environments
Kubernetes
k, kgp, kgn, kgs — get pods/nodes/svc
kns — switch namespace
ksh <ns> <pod> — shell into pod
kshow <ns> — full namespace overview
Infrastructure
vls, vstart, vstop — virsh shortcuts
dps, dcu, dcd — Docker shortcuts
ports — listening ports (ss -tuln)
Helm
h — helm
hls — list all releases
hs <rel> — release status
hv <rel> — release values
hown <rel> — all resources owned by release
Salt (fleet management)
slist — table of all minions with OS, IPs, CPUs, RAM, roles
sping — test.ping all nodes
skeys — list/accept Salt keys
sknodes / skpods — K8s via Salt
Modern replacements
ls → eza (git status, icons, tree)
cat → bat (syntax highlighting)
find → fd (fast, ergonomic)
grep → rg (ripgrep, fast)
cd → z (zoxide, learns dirs)
Kubernetes functions in detail
The Kubernetes functions go beyond simple aliases. kshow gives you a complete namespace
overview — workloads, services, ingress, config, RBAC, storage, and recent events in one command.
ksh handles both pod names and label selectors. cmd launches a
netshoot debug pod on the same node as the target pod. These are the functions that save
you 10 minutes of typing during an incident.
$ kshow production
== ns: production ==
── workloads ──
NAME READY UP-TO-DATE AVAILABLE AGE NODE
deploy/web-frontend 3/3 3 3 47d worker-02
deploy/api-backend 2/2 2 2 47d worker-01
sts/postgres 1/1 1 1 47d worker-03
── services/ingress ──
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
svc/web-frontend ClusterIP 10.96.45.22 <none> 80/TCP
svc/api-backend ClusterIP 10.96.45.23 <none> 8080/TCP
svc/postgres ClusterIP 10.96.45.24 <none> 5432/TCP
── events (last 25) ──
(no warnings or errors)
$ ksh production app=api-backend
root@api-backend-7c8f5d9b4-x2k9m:/# curl localhost:8080/health
{"status":"ok","uptime":4082400}
$ cmd production app=api-backend
# Launches netshoot debug pod on same node
root@dbg-api-backend-12345:/# tcpdump -i any port 8080
...
Salt fleet management
If Salt is installed (hub/master nodes), the slist function gives you a table of every
minion in the fleet with hostname, OS, IP addresses, CPU count, RAM, and roles. One command shows
your entire infrastructure. The Salt aliases make fleet-wide operations one-liners.
$ slist MINION HOST OS IPs CPUs RAM ROLES web-01.lab.internal web-01 CentOS Stream 9 10.78.0.3 192.168.1.103 4 8192 worker web-02.lab.internal web-02 CentOS Stream 9 10.78.0.4 192.168.1.104 4 8192 worker db-01.lab.internal db-01 Debian 13 10.78.0.5 192.168.1.105 8 32768 worker,storage hub.lab.internal hub CentOS Stream 9 10.78.0.1 192.168.1.100 8 16384 master,etcd
Terminal tools
kldload bundles a curated set of modern terminal tools. These aren't kldload-specific — they're widely-used open-source tools. kldload pre-installs them, configures them, and aliases the traditional commands to their modern replacements. The originals are still available if you need them.
Ctrl-a. Split panes with | and -. Move with Alt+arrows. Ctrl-a S shows kst health. Ctrl-a D launches dashboard.Ctrl-R for history search, Ctrl-T for file finder, Alt-C for directory jump. Pipe anything into fzf to pick interactively.cat. Syntax highlighting, line numbers, git diff markers. catp for paged output. Also used as the man pager.ls. Git status integration, file icons, tree view. ll for long list with git status. tree for recursive tree view.find replacement. Faster, ergonomic syntax. fd '\.conf$' finds all .conf files. Respects .gitignore.grep. Extremely fast content search. Respects .gitignore, handles binary files, recursive by default.cd that learns your directories. z logs jumps to /var/log. zi opens interactive picker with fzf.
# tmux keybindings (prefix: Ctrl-a)
C-a | split horizontal C-a - split vertical
Alt+arrow move between panes Shift+arrow resize panes
Alt+1-9 switch windows Alt+h/l prev/next window
C-a z zoom pane (toggle) C-a d detach session
C-a S kst health C-a D launch dashboard
C-a H kldload-help C-a r reload config
# fzf — the most useful tool you've never used
$ vim $(fzf) # pick a file to edit
$ kpkg install $(apt list 2>/dev/null | fzf) # pick a package
$ ssh $(grep Host ~/.ssh/config | awk '{print $2}' | fzf) # pick a host
kldload tools vs manual commands
Every kldload tool saves typing. Some save a few keystrokes. Some save entire multi-step workflows. Here's what each tool replaces and how much work it does for you.
| kldload tool | Manual equivalent | Chars saved | Steps saved |
|---|---|---|---|
ksnap list |
zfs list -t snapshot -r -o name,used,refer,creation -S creation rpool |
60+ | Flag memorization |
ksnap /home |
ds=$(df --output=source /home | tail -1); zfs snapshot "${ds}@manual-$(date +%Y%m%d-%H%M%S)" |
80+ | 2 commands → 1 |
kdf |
zfs list -o name,used,avail,compressratio,quota,mountpoint -r rpool |
60+ | Color-coded output |
kpkg install nginx |
zfs snapshot rpool/ROOT/default@pre-install-$(date +%s); dnf install -y nginx |
70+ | Snapshot + install |
kvm-create myvm |
zfs create -V 20G rpool/vms/myvm; virt-install --name myvm --ram 2048 --vcpus 2 --disk /dev/zvol/rpool/vms/myvm --network bridge=br0 --os-variant generic --graphics none --console pty,target_type=serial |
150+ | 2 tools → 1 |
kvm-clone tmpl web-1 |
zfs snapshot rpool/vms/tmpl@clone; zfs clone rpool/vms/tmpl@clone rpool/vms/web-1; virt-clone --original tmpl --name web-1 --file /dev/zvol/rpool/vms/web-1 --mac RANDOM |
140+ | 3 commands → 1 |
kvm-snap myvm |
zfs snapshot rpool/vms/myvm@$(date +%Y%m%d-%H%M%S) |
40+ | Consistent naming |
kvm-list |
virsh list --all; for vm in $(virsh list --all --name); do zfs list rpool/vms/$vm 2>/dev/null; done |
80+ | Merged view |
kvm-delete myvm |
virsh shutdown myvm; virsh undefine myvm --nvram; zfs destroy -r rpool/vms/myvm |
60+ | 3 commands → 1 |
kst |
zpool status; zfs list; zfs list -t snapshot | wc -l; systemctl --no-pager list-units --state=active; free -h; uptime |
100+ | 6 commands → 1 |
kdir /srv/app |
zfs create rpool/srv/app; zfs set mountpoint=/srv/app rpool/srv/app |
50+ | Auto mountpoint |
kclone /srv/db /srv/db-test |
ds=$(df --output=source /srv/db | tail -1); zfs snapshot ${ds}@clone-$(date +%s); zfs clone ${ds}@clone-$(date +%s) rpool/srv/db-test |
100+ | 3 commands → 1 |
kexport /dev/vda qcow2 |
qemu-img convert -f raw -O qcow2 -c /dev/vda /root/exports/image.qcow2 |
40+ | Auto naming + format hints |
The character count isn't the point. The point is cognitive load. kvm-create myvm --ram 4096 --cpus 4 --disk 40 is readable English. The manual equivalent requires knowing ZFS zvol syntax, virt-install flag names, the exact path to the zvol device node, and the correct virtio configuration. That's four different domains of knowledge for one operation. The kldload tool compresses those four domains into flags you can guess: --ram, --cpus, --disk. You don't have to know that the zvol appears at /dev/zvol/rpool/vms/myvm. You don't have to remember --console pty,target_type=serial. The tool knows.
Tool architecture
Understanding where tools live and how they're installed helps when you want to customize, extend, or audit them. The layout is intentionally simple — standard FHS paths, no custom framework, no hidden config files.
Directory layout
kst, ksnap, kdf, kdir, kpkg, kclone, kexport, kimage, kvm-create, kvm-clone, kvm-snap, kvm-delete, kvm-list, kai, kai-rag, kai-index, kldload-help, kldload-webui.kldload-install-target (installer backend), kldload-firstboot (first-boot init), kldload-recovery (emergency recovery), kldload-snapshot (timer-driven snapshots), kldload-apply-platform-holds (package holds).common.sh, answers.sh, storage-zfs.sh, bootstrap.sh, bootloader.sh, profiles.sh, security.sh, infra.sh, logging.sh.free/index.html, active/index.html. Single HTML file with inline CSS and vanilla JS.install-manifest.env (install-time choices), hub.env (hub WireGuard keys), secrets/ (WireGuard keypairs, held mode 0700).How tools are installed
During the ISO build, everything under live-build/config/includes.chroot/ in the source tree
is mirrored directly into the live ISO's root filesystem. The tools at includes.chroot/usr/local/bin/
land at /usr/local/bin/ on the ISO. During installation, profiles.sh copies the
profile-appropriate tools from the live environment into the target root filesystem. The core profile
skips all kldload tools. The server and desktop profiles copy everything.
Extending and customizing tools
Every tool is a bash script with no external dependencies beyond standard coreutils and ZFS. To customize:
# Read the source: $ cat $(which ksnap) # 80 lines of bash. No mysteries. # Edit in place (your machine, your rules): $ sudo vim $(which ksnap) # Create your own tool: $ sudo vim /usr/local/bin/mycheck #!/usr/bin/env bash set -euo pipefail echo "Pool: $(zpool get -H -o value health rpool)" echo "Snaps: $(zfs list -t snapshot -H | wc -l)" echo "Disk: $(zfs get -H -o value used rpool)" $ sudo chmod +x /usr/local/bin/mycheck $ mycheck Pool: ONLINE Snaps: 47 Disk: 52.1G
The tools are deliberately simple so you can modify them. ksnap is 80 lines. kvm-create is about 120 lines. kpkg is 60 lines. These aren't complex programs. They're the kind of scripts a competent sysadmin writes in an afternoon. The difference is that we wrote them once, tested them across 8 distros, and included them in every install. You get the benefit without the afternoon of work. And when you need something these tools don't do, you have the native commands and the source code as a starting point.
Quick reference — every tool at a glance
Print this section. Pin it to your monitor. These are the commands you'll reach for daily.
| Tool | Purpose | Common usage |
|---|---|---|
kst |
System health overview | kst |
kst-dashboard |
Live tmux monitoring | kst-dashboard |
ksnap |
ZFS snapshot management | ksnap / ksnap list / ksnap rollback /path |
kclone |
Instant CoW clones | kclone /src /dst |
kdf |
ZFS-aware disk usage | kdf |
kdir |
Create ZFS datasets | kdir /path / kdir -p /deep/path / kdir -o quota=50G /path |
kpkg |
Universal package manager | kpkg install pkg / kpkg upgrade / kpkg search term |
kupgrade |
Safe system upgrade | kupgrade |
kexport |
Export disk image | kexport /dev/vda qcow2 / kexport /dev/sda all |
kimage |
Golden image lifecycle | kimage build / kimage export / kimage full 10 |
kvm-create |
Create VM on ZFS zvol | kvm-create name --ram 4096 --cpus 4 --disk 40 |
kvm-clone |
Instant VM clone | kvm-clone template new-vm |
kvm-snap |
Snapshot VM zvol | kvm-snap name / kvm-snap name list / kvm-snap name rollback |
kvm-delete |
Remove VM + zvol | kvm-delete name / kvm-delete name --force |
kvm-list |
VM inventory + ZFS stats | kvm-list |
kai |
AI infrastructure assistant | kai 'your question' |
kai-rag |
RAG-powered queries | kai-rag 'your question' |
kai-index |
Index documentation | kai-index / kai-index /path/to/docs |
kbe |
Boot environments | kbe list / kbe create name / kbe activate name |
khelp |
Built-in reference | khelp |
khold |
Hold critical packages | khold |
kpoof |
Scrub sensitive data | kpoof |
krecovery |
Emergency recovery | krecovery import / krecovery list-pools |
kldload-webui |
Browser installer + control plane | kldload-webui / kldload-webui --port 8080 |
kldload-test |
Smoke tests | kldload-test auto / kldload-test server |
| Kubernetes | ||
kube-cluster |
Bring up production K8s on ZFS | kube-cluster up / kube-cluster up --nodes 5 / kube-cluster down |
kube-demo |
PetClinic + ArgoCD smoke deploy | kube-demo deploy / kube-demo rollback / kube-demo destroy |
kube-status |
Cluster health summary | kube-status |
kube-reset |
Nuke and rebuild cluster | kube-reset |
kube-load-images |
Preload images from offline darksite | kube-load-images |
kube-init / kube-join / kube-network / kube-setup / kube-smoke-test |
Cluster bootstrap + maintenance helpers | called by kube-cluster, runnable standalone |
| klab — multi-distro test platform | ||
klab |
klab umbrella (build / deploy / promote / fault / matrix / diff / destroy) | klab build all / klab deploy blue rocky / klab matrix run play.yml |
klab-vm-debug-bundle |
Collect everything from a klab VM into a tarball | klab-vm-debug-bundle rocky-blue |
klab-exporter / klab-prom-targets |
Prometheus exporter + dynamic target file | systemd-managed |
| OpenZFS Suite | ||
kzfs-lab |
Manage OpenZFS test goldens | kzfs-lab build all / kzfs-lab list |
kzfs-test |
Run zfs-test-suite / ztest / zloop on a golden | kzfs-test run rocky / kzfs-test ztest / kzfs-test zloop --runs 1000 |
kztest-tail |
Live-tail running test results | kztest-tail |
| Bob — local AI | ||
kai |
Bob query CLI (typed input) | kai 'your question' |
kai-rag |
RAG query (with system context bundle) | kai-rag 'your question' |
kai-index / kldload-rag-index |
Index docs / codebase into the RAG vector store | kai-index / kai-index /path/to/runbooks |
| Console & monitoring | ||
kldload-console |
tmux F-key cockpit (eBPF panels) | kldload-console — F1 help, F2… cockpits |
kldload-doctor |
Subsystem health check (web UI's System Doctor) | kldload-doctor |
kldload-loadtest |
Synthetic workload generator (for dashboards/demos) | kldload-loadtest cpu+io+net |
kldload-collect |
Host debug bundle (logs + ZFS + KVM + K8s + eBPF) | kldload-collect |
kldload-overview |
One-screen summary of host + pools + VMs + cluster | kldload-overview |
kldload-follow |
Tail the unified system log (journald + zed + kldload) | kldload-follow |
kldload-lh |
Print this box's LAN IP (via ip -o addr, not getaddrinfo) |
kldload-lh |
kldload-obs-check |
Verify the Prom + Grafana + Loki stack | kldload-obs-check |
kldload-inventory |
Export hardware + ZFS topology to JSON | kldload-inventory |
ksnap works, cat $(which ksnap).
That's the point.
No compiled binaries — a security decision, not laziness. Every tool is a bash script you can read, audit, and modify. cat $(which ksnap) shows you exactly what it does. No obfuscated binary, no vendor SDK phoning home. If you don't like how kpkg handles snapshots, open it in vim and change it. It's your machine. These are your tools. The source code is the documentation.