kldload 1.2.0 — Full Stack Automation
1.1.0 made the install reliable on real hardware and got the live env onto Fedora 44. 1.2.0 takes the next step: end-to-end automation. A click in the webui spawns a full Java + Postgres microservices workload (Spring PetClinic), every Grafana dashboard lights up, drop a table, click "Recover," and you're back in under thirty seconds. On every supported distro.
Underneath: a new state & reconciliation layer that owns apps, ingress, monitoring, and (eventually) AWS managed services from one declarative YAML in /var/lib/kldload/state/. The same workload declaration runs against in-cluster Postgres or RDS depending on the state-layer context.
1. The PetClinic demo — full microservices
A click-to-deploy Spring PetClinic Microservices stack (seven services, not the monolith — the microservices version lights up L7 Hubble and the Cilium service map, which is the whole point) with an embedded Postgres statefulset, sanoid snapshots on a five-minute frequency, syncoid replication on tap, and a smoke test that proves rollback works.
Webui Demo Mode
New sub-page (not just a button): deploy progress, snapshot inventory, replication lag, ArgoCD sync state, embedded dashboard links, disaster and recover buttons. Gated by KLDLOAD_DEMO_MODE=1.
Click-to-recover rollback
One button. No shell required. recover.sh handles both the zfs rollback path and the kubectl delete pvc re-stamp path.
Trigger-disaster buttons
Several pre-built failure modes you can fire on demand: drop a table, kill a pod, partition the network, fill the disk. Each one drives the recovery story.
Data-integrity verifier
kldload-demo verify-data runs sanity queries against PetClinic so recover.sh can assert "data is good" before declaring success.
Full ArgoCD — not Argo Core
ArgoCD server + repo-server + controller + dex + redis + notifications. Available on its own MetalLB VIP at 10.100.10.31.
Pre-warmed PetClinic
Ten warm-up requests fire, p99 settles, then the system declares ready. No "demo ran cold and the latency tile looked broken" moments.
2. State & reconciliation layer
A new declarative layer at /var/lib/kldload/state/. Continuous reconciliation. Owns apps, ingress, monitoring, and (in 1.3) AWS managed services. Same workload declaration runs against in-cluster Postgres or RDS depending on context.
This is the architectural pivot. Today kldload is a great installer with great tools; 1.2 makes it a substrate the workloads sit declaratively on top of.
3. Install ordering — deterministic sequence
The 1.1.0 install raced parts of the stack: kldload-autodeploy and klab-firstboot.service fought for the same resources, and reporting components started before what they were reporting on existed. 1.2.0 enforces a strict order:
- Control plane up
- Workers Ready
- Cilium + Hubble
- kube-prometheus-stack + Loki + Tempo
- Tetragon + Pyroscope + ebpf_exporter
- Dashboard smoke-curl
- THEN klab-firstboot (parallel-safe at this point)
Six minutes faster on a fresh install, and the failure modes are now obvious instead of "something fell over somewhere in the first ten seconds."
4. Backup is on tap — not a vendor
sanoid + syncoid wired into the default install with sensible policies: 5-minute snapshots on PetClinic, hourly on /srv, daily on the OS. Boot environments before every package transaction. syncoid replicates incrementally to a peer (or to the same box, on a different pool) at a frequency you set with one flag.
There's no third-party backup vendor to install, no separate agent to configure, no quota to hit. Snapshots, replication, and restore are how the filesystem already works.
5. Webui — substrate-first messaging
The webui refocuses on the two things kldload actually exists to do: ZFS on Linux, properly, on eight distributions; and a bad-ass KVM hypervisor with ZFS underneath. Every profile — Desktop, Server, KVM Host, Kubernetes, klab, OpenZFS Suite, Core — is an example of what the substrate can become, not a separate product.
6. PoC framing — reproducible runbooks, not marketing
Per the design direction: PoCs are now apples-to-apples reproducible runbooks with artifact directories per incident, judged by the customer. No "demo mode buttons" doing things the runbook can't.
7. Coming next (1.3+)
- AWS-managed-services portability via the state layer — same workload, different runtime context
- HA Postgres — demo path currently uses single-replica
- Remote backup target across WireGuard — today's syncoid policy is local pool only
- Microservices PetClinic with Tetragon mesh enforcement
- AWS-as-metal —
PROFILE=awsprovisions EC2 i4i nodes with ZFS on NVMe ephemeral and S3 snapshot offload