kldload 1.1.0 — Hardware Reality

1. Live environment: CentOS Stream 9 → Fedora 44

The 1.0.x line shipped on CentOS Stream 9 with kernel 5.14 and OpenZFS 2.2. Fine for VMs. On real hardware with newer NVMe controllers, recent NVIDIA cards, USB 3.2 sticks, and Secure Boot firmwares from 2024 onwards, the boot path was getting frayed at the edges. 1.1.0 cuts the live env over to Fedora 44.

2. kldload-proxy — single-port TLS reverse proxy

The biggest architectural change since 1.0.0. Pre-1.1.0 the platform was a constellation of services on different ports, each with its own self-signed cert. 1.1.0 introduces a Go-native single-port TLS reverse proxy that fronts the whole stack:

https://<host>/             →  kldload-webui
https://<host>/grafana/     →  Grafana (3000)
https://<host>/prometheus/  →  Prometheus (9090)
https://<host>/headlamp/    →  k8s Headlamp (4466)
https://<host>/console/     →  libvirt VNC console
https://<host>/k9s/         →  ttyd-k9s embedded terminal
wss://<host>/                →  Bob chat over WebSocket

• One cert issued by kldload-ca covers every backend
• TLS terminated once at the proxy; backends run plain HTTP on the loopback
• WebSocket-aware — preserves Connection: Upgrade, Transfer-Encoding: chunked
• Concurrent cert issuance serialized + atomic install
• Stage-2 nginx + Headlamp scaffold — second proxy layer for sub-path routing, optional

3. Unified web UI — sub-tabbed workspaces

Single-page app with sub-tabbed workspaces for every resource type, behind the proxy, with Grafana / Headlamp / k9s embedded as iframes that share the same TLS cert. HTTPS out of the box on port 8443, self-signed cert auto-renewed, explicit "Grant microphone access" button next to the mic, install/klab subprocesses run under systemd-run transient units so a crashing install can’t blow up the UI tracking it.

Plus: encrypted credentials store — AES-256-GCM-encrypted ZFS dataset (rpool/kldload/secrets) on installed systems; live ISO refuses to adopt a target-disk rpool; secret values live as 0600 files inside the encrypted dataset and are never logged.

4. klab — the multi-distro test sandbox

By 1.1.0 klab is the substrate the rest of the platform runs on: hypervisor + multi-distro test runner + WireGuard mesh + observability stack + web UI in one. Run the real OpenZFS test suite across 7 distros from one host: CentOS Stream 9, Debian 13, Ubuntu 24.04, Fedora 44, Rocky 9, RHEL 9, Arch.

5. OpenZFS observability stack

The demo you’d bring to OpenZFS maintainers. Click a test failure → see kernel stack, last zio, D-state blocked task, SMART status, ARC state — all correlated on one timeline. Bundle is paste-ready for upstream.

Eight Grafana dashboards bundled: zfs-pool-health, scrub-history, compression-trend, disk-health-smart, block-io-latency, kernel-messages, klab-test-matrix, klab-test-debug.

zed → Loki: /etc/zfs/zed.d/all-loki.sh pushes every zpool event (checksum error, resilver, scrub, trim, vdev state change) into Loki with {class, pool} labels.

6. Tetragon — eBPF runtime security

Tetragon now ships in 1.1.0 with full Grafana plumbing.

• Process flow tracing — every exec, fork, exit captured
• Syscall flow — read/write/connect/socket events
• Packet flow attribution — packets tagged with the originating process, visible in the Traffic Map
• kprobe-based observation — kernel-level visibility, no application instrumentation
• zfs-execs TracingPolicy audits zfs/zpool/zdb/ztest/zfs-tests.sh execve events
• eBPF deep-dive demos in kube-demo (demos 22-24)

This is the "advanced messaging" surface — kernel-to-application event correlation, not just metric aggregation.

7. Bob AI — eyes, ears, and a voice

Bob graduated from "kldload-aware chat" to "agent that can actually do things on the system." Bob can answer "why is hubble-relay not ready?" by calling k8s_events → k8s_describe → k8s_logs → kernel_dmesg in sequence.

8. Operator console — 24-key tmux drawer

Every watchable pane is one keystroke away. The drawer reshapes the main content area via a CSS variable so panels, VMs, logs never hide behind it. Every key is a toggle.

9. Hardware enablement — laptop-ready desktop

~350 MB of hardware enablement, codecs, fonts. The 1.0.4 desktop booted clean but then you had a laptop with no WiFi, no webcam, no printer, and Liberation Sans. Not anymore.

10. Install reliability — the hardware-truth pass

Each item below was found by installing onto real hardware and watching it fail. Each is a discrete commit with a comment in the source explaining the failure mode.

11. Secure Boot architecture

The shipped chain is the smallest possible Secure Boot trust footprint for a ZFS-on-root distro. The only thing we sign is zfs.ko and the staged vmlinuz (with the kldload MOK key). Everything else is already signed by trusted vendor certs.

firmware
  └ shim.efi          (Microsoft-signed, distro-shipped)
       └ grubx64.efi   (RH/CentOS/Rocky/Fedora distro-signed,
                              already trusted by shim’s vendor cert)
            └ vmlinuz   (distro-signed by the same vendor cert,
                              re-signed with kldload MOK leaf in 1.1.0)
                 └ zfs.ko (MOK-signed via DKMS sign_tool)

• Per-install MOK leaf cert generated fresh per install, single EKU, no CA reuse
• kldload-ca init creates the per-install CA (separate from the MOK)
• kldload-grub-refresh.path auto-refreshes /boot/efi/EFI/BOOT/grubx64.efi on distro upgrade
• kldload-secure-boot CLI — enable | disable | status | reenroll
• Unified trust root — same kldload CA roots TLS certs (browser trust) + MOK signing (kernel module trust)

12. Kubernetes / kube-cluster

• Pre-flight libvirt default network before virt-install — fixes bootstrap on fresh hosts where virbr0 has no carrier yet
• kubeconfig + kubectl published to host BEFORE step 7 — you can kubectl immediately
• Atomic download of cloud qcow2 — fixes a race where parallel invocations corrupted the source image
• AI / Ollama enabled by default for k8s/kvm/zfslab tiles
• metrics-server auto-installs — powers kubectl top + live CPU/Mem overlays
• Cilium-operator scrape only fires on the node it’s actually running on
• kube-demo — 21+ interactive demos, eBPF deep-dives at 22-24

13. kspawn — ZFS-native multi-runtime cluster spawner

New top-level CLI: kspawn spawn --name web --distro debian --count 5 clones 5 VMs from the klab debian golden in parallel, injects cloud-init (hostname + SSH key per node), boots them, writes a JSON manifest at /var/lib/kspawn/clusters/<name>/manifest.json. Subcommands: spawn / list / status / ssh / destroy. Because all state is derivable from the manifest + ZFS clones, there is nothing to "upgrade" — destroy and re-spawn.

14. Known issues

15. Upgrade notes

Major version. Live env, kernel, ZFS, shim, dnf are all stepping forward. No in-place upgrade from 1.0.x — do a fresh install onto a different ZFS pool, or wipe and reinstall on the same disk. Restore state from ZFS snapshots if you had them.

# Get it
curl -L -o /tmp/kldload.iso https://dl.kldload.com/kldload-free-latest.iso

# Burn it (USB at /dev/sda — verify with lsblk first)
sudo bash -c 'wipefs -af /dev/sda && \
  dd if=/tmp/kldload.iso of=/dev/sda bs=4M oflag=direct \
     status=progress conv=fsync && sync && eject /dev/sda'

Full commit-by-commit changelog: RELEASE_NOTES-1.1.0.md · git log --oneline v1.0.4..v1.1.0