kldload 1.2.0 — Full Stack Automation

Status: shipped in 1.3.1  ·  License: BSD-3-Clause
Source: github.com/kldload/kldload
Shipped as part of 1.3.1 — The Kernel-Loaded Desktop. 1.2.0 was never cut as a standalone release; this full-stack automation work landed in the first public 1.3.x build. This page records the scope that shipped.

1.1.0 made the install reliable on real hardware and got the live env onto Fedora 44. 1.2.0 takes the next step: end-to-end automation. A click in the webui spawns a full Java + Postgres microservices workload (Spring PetClinic), every Grafana dashboard lights up, drop a table, click "Recover," and you're back in under thirty seconds. On every supported distro.

Underneath: a new state & reconciliation layer that owns apps, ingress, monitoring, and (eventually) AWS managed services from one declarative YAML in /var/lib/kldload/state/. The same workload declaration runs against in-cluster Postgres or RDS depending on the state-layer context.

1. The PetClinic demo — full microservices

A click-to-deploy Spring PetClinic Microservices stack (seven services, not the monolith — the microservices version lights up L7 Hubble and the Cilium service map, which is the whole point) with an embedded Postgres statefulset, sanoid snapshots on a five-minute frequency, syncoid replication on tap, and a smoke test that proves rollback works.

Webui Demo Mode

New sub-page (not just a button): deploy progress, snapshot inventory, replication lag, ArgoCD sync state, embedded dashboard links, disaster and recover buttons. Gated by KLDLOAD_DEMO_MODE=1.

Click-to-recover rollback

One button. No shell required. recover.sh handles both the zfs rollback path and the kubectl delete pvc re-stamp path.

Trigger-disaster buttons

Several pre-built failure modes you can fire on demand: drop a table, kill a pod, partition the network, fill the disk. Each one drives the recovery story.

Data-integrity verifier

kldload-demo verify-data runs sanity queries against PetClinic so recover.sh can assert "data is good" before declaring success.

Full ArgoCD — not Argo Core

ArgoCD server + repo-server + controller + dex + redis + notifications. Available on its own MetalLB VIP at 10.100.10.31.

Pre-warmed PetClinic

Ten warm-up requests fire, p99 settles, then the system declares ready. No "demo ran cold and the latency tile looked broken" moments.

2. State & reconciliation layer

A new declarative layer at /var/lib/kldload/state/. Continuous reconciliation. Owns apps, ingress, monitoring, and (in 1.3) AWS managed services. Same workload declaration runs against in-cluster Postgres or RDS depending on context.

This is the architectural pivot. Today kldload is a great installer with great tools; 1.2 makes it a substrate the workloads sit declaratively on top of.

3. Install ordering — deterministic sequence

The 1.1.0 install raced parts of the stack: kldload-autodeploy and klab-firstboot.service fought for the same resources, and reporting components started before what they were reporting on existed. 1.2.0 enforces a strict order:

  1. Control plane up
  2. Workers Ready
  3. Cilium + Hubble
  4. kube-prometheus-stack + Loki + Tempo
  5. Tetragon + Pyroscope + ebpf_exporter
  6. Dashboard smoke-curl
  7. THEN klab-firstboot (parallel-safe at this point)

Six minutes faster on a fresh install, and the failure modes are now obvious instead of "something fell over somewhere in the first ten seconds."

4. Backup is on tap — not a vendor

sanoid + syncoid wired into the default install with sensible policies: 5-minute snapshots on PetClinic, hourly on /srv, daily on the OS. Boot environments before every package transaction. syncoid replicates incrementally to a peer (or to the same box, on a different pool) at a frequency you set with one flag.

There's no third-party backup vendor to install, no separate agent to configure, no quota to hit. Snapshots, replication, and restore are how the filesystem already works.

5. Webui — substrate-first messaging

The webui refocuses on the two things kldload actually exists to do: ZFS on Linux, properly, on eight distributions; and a bad-ass KVM hypervisor with ZFS underneath. Every profile — Desktop, Server, KVM Host, Kubernetes, klab, OpenZFS Suite, Core — is an example of what the substrate can become, not a separate product.

6. PoC framing — reproducible runbooks, not marketing

Per the design direction: PoCs are now apples-to-apples reproducible runbooks with artifact directories per incident, judged by the customer. No "demo mode buttons" doing things the runbook can't.

7. Coming next (1.3+)

  • AWS-managed-services portability via the state layer — same workload, different runtime context
  • HA Postgres — demo path currently uses single-replica
  • Remote backup target across WireGuard — today's syncoid policy is local pool only
  • Microservices PetClinic with Tetragon mesh enforcement
  • AWS-as-metalPROFILE=aws provisions EC2 i4i nodes with ZFS on NVMe ephemeral and S3 snapshot offload