Disposable Infrastructure

Most operators treat infrastructure as precious because rebuilding it is expensive — the rebuild takes hours, depends on procedures nobody’s tested, and risks losing state. So you protect what exists. You patch in place. You SSH-fix things at 2am. You don’t destroy and recreate, because you’re afraid the recreation won’t come back clean.

kldload changes that arithmetic. When destroying a Kubernetes cluster takes 20 seconds and rebuilding from scratch takes 5 minutes, the cost of recreation drops below the cost of protection. You stop preserving infrastructure that should be replaced. Test environments become per-PR instead of per-team. DR exercises become weekly instead of annual. Production stops meaning fragile.

This masterclass is the operator-side consequence of the substrate thesis (see First-Class Infrastructure for the substrate fundamentals). Same architecture, different lens: what changes about how you run a system when rebuild is no longer scary.

Pick any infrastructure operation you treat as “risky” today. Cluster rebuild. Database failover. Site cutover. Cross-region migration. The reason it’s risky isn’t the operation itself — it’s that you’ve never actually done the operation in production conditions, because each rehearsal would cost hours and might not come back clean.

Disposable infrastructure inverts that. The operations are the rehearsals. If your install procedure produces the same outcome as your DR procedure, and your DR procedure runs in 10 minutes on commodity hardware, then every install is a DR drill. There’s no separate runbook to drift from production. There’s no separate vendor to write a contract with. There’s no annual procedure-binder review with check boxes for things nobody’s tested in three years.

The mental shift is from “protect what exists” to “trust what regenerates.”

The numbers below are wall-clock measurements on a kldload box running build #50, demoed in real time. Every number is observed, not extrapolated.

If destroying a Kubernetes cluster takes 20 seconds, your DR architecture is fundamentally different. Not because kldload added a DR feature — because the operations other platforms treat as scary, rare, runbook-driven are operations kldload treats as “I’ll just do it again in front of you live.”

“Fast K8s nodes on one box” usually means kind or k3d — both of which use Docker containers as nodes. kldload doesn’t. Every node in the table above is a real qemu-kvm virtual machine with its own kernel, filesystem, boot, distribution. This matters more than it sounds.

The trade-off is honest: VMs cost more RAM and start ~20 seconds slower than containers. What you get back is every capability that depends on having your own kernel — and once you need any of them, containers can’t substitute.

The kldload demos require real VMs:

• The OpenZFS test suite runs zfs-tests.sh against each VM’s own kernel — doesn’t work in containers.
• klab’s multi-distro testing only means something because each VM is actually a different distribution running a different kernel — containers all share the host’s kernel, so “test CentOS + Fedora + Ubuntu” is fiction when it’s the same Linux underneath.
• ZFS DKMS regression testing rebuilds the module against per-VM kernels.
• WireGuard kernel module + Cilium eBPF behave differently per kernel.
• Per-VM Secure Boot, MOK keys, custom kernel command lines — kernel features, kernel territory.

If your workload is “schedule pods, watch them run” — kind is fine and faster. If your workload touches the kernel in any way — you need real VMs, and kldload is the fastest way to get a fleet of them on one box. That’s the lane.

The same USB that installs kldload also exports it. kexport produces qcow2, raw, vmdk, vhd, OVA — pick the destination, get the format. Disaster recovery is just “install procedure, with a different output target.”

The recovery sequence on commodity hardware:

# 1. Plug kldload USB into ANY commodity computer.
# 2. Boot live env (entirely in RAM).
# 3. In the installer, pick the template matching what was lost.
template = kldload-kvm                # or k8s, klab, zfslab, etc.
inventory = ./my-50-vm-fleet.yaml

# 4. Pick the output format for the destination.
format = qcow2                        # or raw / vmdk / vhd / OVA

# 5. Hit Export. kldload assembles the image from the live env.
#    No source system needed -- there isn't one anymore.
# 6. Once built, dd the result back onto the same USB.
#    The live env runs from RAM, doesn't care about overwriting
#    its own boot media.
$ dd if=output.qcow2 of=/dev/sdX bs=4M oflag=direct conv=fsync

# 7. Upload to the provider, boot one instance.
#    Autodeploy reconstitutes the environment in parallel.
$ aws ec2 import-image ...            # or Azure / GCP / OVH equivalent

About an hour later, the infrastructure shape is back. Be honest about scope: this rebuilds the shape (VMs, K8s, mesh, observability). The data in that infrastructure has to come from somewhere that survived — usually your syncoid → survivor site ZFS replication stream. The USB doesn’t magic up data that doesn’t exist. But infrastructure-shape recovery is the part that traditionally takes weeks; once it’s back, restoring state from a ZFS receive is bounded by network throughput.

Pair this masterclass with Backup & Disaster Recovery for the data-side coverage.

Take any one of the three away and the disposability story breaks. All three are properties of the substrate, not features kldload adds on top:

Most platforms have one of the three. A few have two. Almost none have all three together — which is why almost none of them can offer the “install procedure IS the DR procedure” pitch with a straight face.

Once destroy-and-rebuild is cheap, your team starts doing things differently. Not because anyone told them to — because the new economics make the new behaviours obviously right.

These aren’t culture changes you have to drive. They’re the rational responses to a substrate where the cost ratios are different. Cheap rebuild → casual rebuild → behavioural change. The platform shapes the operator, not the other way around.

Worth being honest about scope — especially when the numbers sound too good. Disposable infrastructure is not:

Three follow-on reads if this changed how you think about infrastructure lifecycle: