Three roles, three ISOs

The timeline

# Minute 0: Build ISOs (one per role)
./deploy.sh build   # builds master.iso, etcd-1.iso, worker-1.iso, worker-2.iso

# Minute 1: Deploy to Proxmox (or KVM, or bare metal)
# Each ISO: preseed install → poweroff → boot from disk → postinstall → poweroff

# Minute 15: All VMs powered off after postinstall
# Master has: Salt master, WireGuard hub, kubectl
# etcd nodes have: containerd, kubelet, Salt minion, WireGuard client
# Workers have: containerd, kubelet, Salt minion, WireGuard client

# Minute 16: All VMs started for production
# Master's wg-sync-salt.timer runs every 30s:
#   - Discovers Salt minions
#   - Reads WireGuard pubkeys from grains
#   - Pushes WireGuard configs to all nodes
#   - Generates Ansible inventory from Salt grains

# Minute 18: cluster-autobootstrap.timer runs every 60s:
#   - Finds first etcd node (control-plane)
#   - Runs kubeadm init on it
#   - Installs Calico CNI (Tigera operator)
#   - Joins remaining etcd nodes as control-planes
#   - Joins workers
#   - Pushes admin.conf to all control-planes
#   - Marks done when all nodes are Ready

# Minute 25: kubectl get nodes
# NAME      STATUS   ROLES           VERSION
# etcd-1    Ready    control-plane   v1.35.x
# worker-1  Ready    <none>          v1.35.x
# worker-2  Ready    <none>          v1.35.x

Define your cluster in environment variables

# Node spec: vmid name role lan_ip wg_cidr
MASTER_ID=999     MASTER_NAME=master    MASTER_LAN=10.100.10.20
ETCD_COUNT=1      ETCD_LAN_BASE=10.100.10.30
WORKER_COUNT=2    WORKER_LAN_BASE=10.100.10.40

# Results in:
# 999  master    master  10.100.10.20  10.78.0.1/16
# 1000 etcd-1    etcd    10.100.10.30  10.78.0.5/32
# 1010 worker-1  worker  10.100.10.40  10.78.0.11/32
# 1011 worker-2  worker  10.100.10.41  10.78.0.12/32

# Scale by changing counts:
ETCD_COUNT=3 WORKER_COUNT=5 ./deploy.sh build
# Builds 9 ISOs. Deploy. Walk away. Full HA cluster.

wg-sync-salt: automatic peer management

Every 30 seconds, the master's wg-sync-salt timer:

1. Queries Salt grains for every minion's WireGuard pubkey, CIDR, role, and LAN IP
2. Regenerates the master's wg0.conf with all peers
3. Pushes WireGuard client configs to each minion via Salt
4. Runs wg syncconf to apply changes without restarting the interface
5. Generates Ansible inventory from Salt grains (master/etcd/workers groups)

New node boots → Salt minion connects → grains include WireGuard pubkey → wg-sync-salt picks it up → pushes config → node joins the mesh. Automatic.

ZFS makes Kubernetes disposable

# Snapshot the entire cluster state
for node in master etcd-1 worker-1 worker-2; do
  ssh admin@${node} "sudo ksnap"
done

# Worker dies? Don't fix it. Clone from golden and rejoin.
zfs clone rpool/vms/worker-golden@base rpool/vms/worker-3
# Boot → postinstall → Salt minion → WireGuard → kubeadm join → Ready

# Upgrade the cluster? Snapshot first.
ssh admin@etcd-1 "sudo kbe create pre-k8s-upgrade"
# Run kubeadm upgrade. If it breaks:
ssh admin@etcd-1 "sudo kbe rollback pre-k8s-upgrade"
# 15 seconds. Cluster is back to pre-upgrade state.

# Replicate the entire cluster to DR site
for node in master etcd-1 worker-1 worker-2; do
  ssh admin@${node} "sudo zfs send -R rpool@backup | ssh dr-site zfs recv backup/${node}"
done