Build your own. Understand what you built. That's the whole point.
kldload is not a product you deploy. It is a re-packer you use to build infrastructure that belongs to you. You pick the distro. You pick the profile. You pick the packages, the ZFS layout, the network topology, the encryption policy. kldload assembles it into a bootable image with ZFS on root, WireGuard, boot environments, and offline package mirrors. The output is your infrastructure, not kldload's. kldload is the factory. What rolls off the line is yours.
This page is the landing page for everything in the Build Your Own section. It explains the concept, the build process, the three installation paths, the profiles, the distro selection logic, the customization layers, and the real-world architectures people build with this system. The sidebar links go deeper into each recipe. This page tells you how to think about building.
The thesis: Infrastructure you understand is infrastructure you can fix. Infrastructure you downloaded as a black-box appliance is infrastructure you can only restart and hope. kldload does not hide what it does. Every script is bash you can read. Every config file is a file on disk you can edit. Every package is a package you chose. The goal is not to make infrastructure easy — it is to make infrastructure yours.
You will learn more building one kldload system from scratch than you will from a year of clicking through managed dashboards. Not because kldload is educational software — because building things teaches you how things work. That is the oldest lesson in engineering.
The re-packer concept
Most infrastructure tools are opinionated products. They make choices for you: which distro, which filesystem, which init system, which network stack. You get their opinions, pre-baked. kldload makes zero choices for you. It presents every option and builds exactly what you specify. Nine distros. Four profiles. Any package set. Any ZFS layout. Any network configuration. Offline or online. Interactive or unattended. The ISO is a factory that builds what you tell it to build.
Not a distro
kldload is not a Linux distribution. It installs real vendor distros —
CentOS, Debian, Ubuntu, Fedora, RHEL, Rocky, Arch, Alpine — directly from vendor
packages. The installed system IS that distro. cat /etc/os-release shows
Debian, not kldload. Package management uses the vendor's repos. Updates come from the
vendor. kldload adds ZFS on root, boot environments, and tooling on top. The distro is
the distro. kldload is the installer.
Not an appliance
Appliances hide their internals. kldload exposes them. Every installer script is bash
in /usr/lib/kldload-installer/lib/. Every systemd service has a unit file
you can read. Every ZFS dataset has properties you tuned. Every WireGuard tunnel has a
config in /etc/wireguard/. There is no compiled binary, no obfuscated
config format, no proprietary management layer. It is bash, systemd, and standard Linux
tools all the way down.
Not a cloud service
kldload runs on your hardware. Your NVMe drives. Your RAM. Your network. No API keys. No subscription. No telemetry. No "free tier" that becomes paid when you need it. The ISO is 4 GB. Download it once. Burn it to USB. Install as many machines as you want, as many times as you want, forever. The darksites (offline package mirrors) mean you do not even need internet after the initial download.
The re-packer metaphor is precise. You get a box of parts: a kernel, a userland, ZFS modules, WireGuard, boot environment tooling, snapshot automation, eBPF observability, a web UI installer. You assemble them into whatever you need. An edge server. A CI runner. A NAS. A firewall. A Kubernetes cluster. A dev workstation. The parts are the same. The assembly is yours. And because you assembled it, you know where every part is, what it does, and how to replace it when it wears out.
This is fundamentally different from downloading a pre-built thing. When you download TrueNAS or Proxmox or OpenMediaVault, you get someone else's opinions about how storage or virtualization should work. When those opinions collide with your requirements — and they always do, eventually — you are stuck. You cannot change the internals because you did not build them. With kldload, there are no internals. There is only your build, assembled from parts you understand.
The re-packer gives you four fundamental primitives. Every build on kldload — from a single Raspberry Pi to a 200-node cluster — combines these same four things. Once you understand them, you can build anything in the sidebar. And things that are not in the sidebar.
1. Datasets
Every service gets its own ZFS dataset with tuned recordsize, compression, and quota.
Your database gets recordsize=8K. Your media library gets recordsize=1M.
Your CI workspace gets sync=disabled. Each one is independently snapshotted,
encrypted, and replicable. Datasets are the unit of storage in kldload. Not directories.
Not partitions. Datasets.
2. Tunnels
WireGuard connects your nodes. Encrypted at the kernel level. No certificate authority. No daemon to crash. Your services talk over the mesh without knowing it is there. Replication, monitoring, orchestration — all ride the tunnel. A two-node homelab and a 200-node enterprise use the same WireGuard mesh. It scales because it is stateless.
3. Boot environments
Every risky operation gets a snapshot first. Upgrade the kernel? Snapshot. Install a new service? Snapshot. Change a config? Snapshot. If it breaks, boot the old snapshot from ZFSBootMenu. 15 seconds to a known-good state. No reinstall. No rescue disk. No hoping your backup script actually ran last night.
4. Replication
zfs send moves datasets between machines at the block level. Incremental.
Checksummed. Encrypted in flight over WireGuard. Build on one machine, replicate to 100.
Your deployment artifact is a ZFS snapshot, not a tarball. Your DR strategy is a cron job,
not a product with a license fee.
Those four primitives combine into everything in the sidebar. A NAS server is datasets + replication. A KVM hypervisor is datasets (zvols) + boot environments. A cluster is all four. A key management platform is datasets (per-customer encryption) + tunnels (replication over WireGuard). A CI runner is datasets (ephemeral build workspaces with sync=disabled) + boot environments (rollback after each build). A monitoring stack is datasets (Prometheus TSDB with tuned recordsize) + tunnels (scraping remote nodes over WireGuard) + replication (TSDB backup to cold storage).
The recipes in the sidebar are examples of how to combine the primitives. This overview is here to explain the building blocks so you can combine them yourself. If you understand datasets, tunnels, boot environments, and replication, you do not need recipes. You need ideas.
What you can build
The sidebar has specific recipes. This section is the catalog — an overview of every category of thing people build with kldload, what makes each one interesting on ZFS, and where to go deeper. Every card links to a dedicated recipe or tutorial.
Edge servers
Remote sites, branch offices, retail locations. Boot from USB, install offline from the darksite, replicate back to HQ over WireGuard. ZFS encryption at rest for physical security. Boot environments so a failed update does not strand a technician. The edge server recipe builds a self-healing node that phones home but works independently when the WAN is down.
CI runners
Ephemeral build environments on ZFS. Each build gets a clone of a golden dataset —
O(1) creation, zero disk cost until the build writes data. When the build finishes, destroy
the clone. No container overhead. No shared filesystem corruption between builds. Full
kernel access for builds that need DKMS, eBPF, or KVM. sync=disabled on
build datasets for raw throughput.
Dev workstations
GNOME desktop with ZFS on root. Per-project datasets with instant snapshots before
every git rebase. Docker and Podman on ZFS storage drivers. KVM for local
VMs. Boot environments so you can test a kernel upgrade and roll back in 15 seconds.
NVIDIA drivers pre-installed for CUDA/ML development.
Security appliances
nftables firewall, WireGuard VPN concentrator, Suricata IDS, CrowdSec threat intelligence. ZFS encryption for evidence datasets. eBPF for kernel-level network visibility without packet capture overhead. Boot environments so security patches never brick the perimeter.
Classroom labs
One golden image, 30 ZFS clones. Each student gets an independent system that shares storage with the template until they diverge. Reset a student's machine by destroying the clone and re-cloning. The entire lab resets in under a minute. No PXE. No network boot. No imaging server. ZFS clone is the imaging server.
Disaster recovery sites
Continuous ZFS replication over WireGuard to a remote site. Incremental, checksummed,
encrypted. The DR site has every dataset from production, seconds behind. Failover is
zfs promote + DNS change. No third-party DR product. No replication license.
No cloud egress fees. ZFS send is the replication engine. WireGuard is the transport.
NAS / file servers
ZFS is the best NAS filesystem in existence. Checksummed reads. Self-healing with mirrors or RAID-Z. NFS, SMB, iSCSI serving. Per-share datasets with independent compression, encryption, and quotas. Snapshot-based Time Machine for macOS clients. Sanoid for automated snapshot management. No proprietary NAS OS — just Linux + ZFS + the file sharing protocols you need.
Firewalls and gateways
nftables for stateful packet filtering. WireGuard for site-to-site VPN. BIRD for BGP when you need dynamic routing. Alpine or Debian core profile for minimal attack surface. ZFS boot environments so firewall rule changes are atomic and reversible. eBPF XDP for wire-speed packet processing when nftables is not fast enough.
Media servers
Plex, Jellyfin, or Emby on ZFS with recordsize=1M for large media files.
Per-library datasets — movies, TV, music, photos — each with independent
compression and snapshot policies. Hardware transcoding with Intel QSV or NVIDIA NVENC.
ZFS send for offsite backup of irreplaceable media libraries.
AI/ML workstations
NVIDIA drivers pre-installed. CUDA toolkit on ZFS. Per-model datasets with snapshots
at each training checkpoint. recordsize=1M for large model files.
sync=disabled for scratch datasets that hold intermediate training
state. KVM with GPU passthrough for isolated training environments. Bob AI for
infrastructure-aware local LLM.
Databases on ZFS
PostgreSQL with recordsize=8K matching its page size. MySQL/MariaDB
with recordsize=16K. WAL on a dedicated dataset with
logbias=throughput. Snapshots are consistent point-in-time backups
without pg_dump overhead. Replication via zfs send for
DR. ZFS compression gives 2-3x space savings on typical database workloads.
Container hosts
Docker or Podman with ZFS storage driver. Each container layer is a ZFS dataset. Snapshots, clones, and replication work at the container layer. Pull an image once, clone it for every container instance. ZFS compression reduces image storage. Boot environments protect the host OS from container runtime upgrades.
Kubernetes clusters
KVM hypervisor on ZFS with zvol-backed VMs for control plane and workers. Or bare metal with kubeadm. OpenZFS CSI driver for persistent volumes. WireGuard mesh for inter-node encryption. Cilium with eBPF for CNI. ZFS replication for etcd backup. Boot environments on every node for safe kernel upgrades across the cluster.
Monitoring stacks
Prometheus with ZFS-tuned TSDB storage. Grafana dashboards. Alertmanager. Node exporter on every machine. eBPF-based metrics collection with zero application instrumentation. WireGuard mesh for secure scraping across sites. ZFS replication for TSDB backup. Per-retention datasets — hot (7 days, SSD), warm (90 days, HDD), cold (1 year, compressed + replicated).
Every card above shares the same foundation: ZFS on root, boot environments, WireGuard, snapshots. The differentiation is in the datasets (tuned per workload), the packages (chosen per role), and the postinstall script (custom per deployment). That is the re-packer pattern. Same foundation, infinite variation. You are not learning 14 different products. You are learning four primitives and applying them 14 different ways.
If something you want to build is not listed, it does not mean kldload cannot build it. It means nobody has written the recipe yet. The primitives are general purpose. Anything that runs on Linux, runs on kldload. The question is just how to tune the ZFS datasets and which packages to include. Everything else is standard Linux administration.
How kldload builds images
The build process has two phases: ISO assembly (happens once, on your build machine) and target installation (happens every time you boot the ISO on hardware). Understanding both phases is key to understanding the re-packer.
Phase 1: ISO assembly
This is the ./deploy.sh build pipeline. It runs inside containers on your build
machine. It produces the bootable ISO. Five stages, all automated:
The five build stages
Stage 1: Builder image CentOS Stream 9 container with lorax, xorriso,
squashfs-tools, dracut, mtools
Stage 2: Debian darksite debian:trixie-slim container resolves + downloads
all APT packages for offline install
Cached at live-build/darksite-debian-cache/
Stage 3: Ubuntu darksite ubuntu:noble container resolves + downloads all APT
packages (including universe for ZFS)
Cached at live-build/darksite-ubuntu-cache/
Stage 4: RPM darksite dnf download --resolve --alldeps inside builder
Downloads CentOS/Fedora/RHEL/Rocky packages
Stage 5: ISO assembly Bootstraps rootfs via dnf --installroot, builds ZFS
DKMS, embeds all darksites, creates squashfs + EFI +
ISO with xorriso. Output: one bootable ISO.The darksites are cached after the first build. Subsequent builds skip stages 2-4 unless you explicitly rebuild them. A full first build takes 30-60 minutes depending on your internet speed. Incremental rebuilds take 5-10 minutes.
The darksite concept is what makes offline installation possible. A "darksite" is a complete, self-contained package repository embedded in the ISO. When the installer runs, it does not reach out to the internet for packages. It points the package manager at the local darksite — a directory on the live ISO's RAM filesystem. APT distros (Debian, Ubuntu) get served via a local HTTP mirror on port 3142/3143. RPM distros (CentOS, Fedora, RHEL, Rocky) use file:///root/darksite/ directly. The installer does not know or care that the packages are local. It thinks it is talking to a normal repository.
This is why the ISO is 4 GB. It contains complete package sets for every supported distro. That is the trade-off: a large ISO in exchange for zero internet dependency at install time. For air-gapped environments, classified networks, remote sites with satellite internet, or anywhere the network is unreliable, this is not a trade-off. It is a requirement.
Phase 2: Target installation
Boot the ISO on target hardware (or a VM). The live CentOS Stream 9 environment starts. From here, two things can happen: the web UI launches for interactive install, or a seed disk triggers unattended install. Either way, the installation sequence is identical:
The installation sequence
1. Detect disks Scan block devices, present options to user
2. Partition GPT table: EFI System Partition + ZFS partition
3. Create pool zpool create with selected topology (single/mirror/raidz)
4. Create datasets Root dataset, /home, /var, /srv, boot environment layout
5. Bootstrap dnf --installroot (RPM) or debootstrap (Debian/Ubuntu)
or pacstrap (Arch) — packages come from the darksite
6. Configure Timezone, locale, hostname, users, SSH, networking
7. Build ZFS modules DKMS compiles ZFS against the installed kernel
8. Install bootloader ZFSBootMenu or systemd-boot with ZFS support
9. Enable services Snapshot timers, package holds, WireGuard, firstboot
10. Seal (if exporting) Clear machine-id, SSH keys, enable cloud-init
11. Export (if requested) qemu-img convert to qcow2/vmdk/vhd/ova/rawSteps 1-9 happen on every install. Steps 10-11 only happen when you are producing a golden image for cloning. The entire sequence takes 8-20 minutes depending on hardware speed and the selected package set.
The installer is one bash script (kldload-install-target)
that sources nine library files. It is not compiled. It is not obfuscated. Every step is
a bash function you can read, understand, and modify. The web UI is a single Python file
that sends WebSocket messages to the installer. The HTML is a single file per edition.
The entire installer codebase is under 5,000 lines of bash.
The deliberate simplicity of the installer is a feature. Nine bash libraries. One main script. One Python web server. One HTML file. No framework. No build system for the installer itself. No transpilation. No minification. No dependency tree. You can read the entire installer in an afternoon. You can modify it with a text editor. You can add a new distro by writing one new bootstrap function and adding it to the dispatch table.
Compare this to Anaconda (Fedora/RHEL installer): 200,000+ lines of Python across hundreds of modules. Or debian-installer: a maze of udebs, preseeds, and d-i components that take weeks to understand. Or Calamares: a C++/Qt framework with a plugin system and build dependencies that fill a page. kldload's installer is bash that calls standard tools. The complexity is in ZFS and the boot chain, not in the installer framework, because there is no installer framework.
Three installation paths
Same ISO, same installer, same result. The difference is how you provide answers. Pick the path that matches your workflow.
Path 1: Interactive (Web UI)
Boot the ISO. Open a browser to http://<machine-ip>:8080. Select your
distro, profile, disk, hostname, users, networking, ZFS topology, features. Click Install.
Watch the progress in real time via WebSocket. Best for: first installs, learning, one-off
machines, hardware testing, development.
Path 2: Unattended (Answers file)
Write an answers.env file with your choices. Put it on a FAT32 USB labeled
KLDLOAD-SEED. Insert it alongside the ISO. Boot. The installer finds the seed
disk, reads the answers, installs to the target disk, and powers off. Zero interaction.
Best for: repeatable deployments, air-gapped sites, field installs, anyone who hates
clicking through UIs.
Path 3: Pipeline (Packer / CI)
Packer boots the ISO in a VM, sends install commands via the WebSocket API, waits for completion, exports the result as a golden image (qcow2, vmdk, vhd, ova). The golden image feeds into Terraform, cloud-init, or manual deployment. Best for: automated image pipelines, cloud templates, multi-environment builds, CI/CD systems that produce infrastructure artifacts alongside application artifacts.
Choosing your path
Decision tree
Are you building one machine?
├── Yes → Interactive (Web UI)
│ Boot, click, done. Learn the options.
│
└── No → Are you building many identical machines?
├── Yes → Do you have a CI/CD pipeline?
│ ├── Yes → Pipeline (Packer)
│ │ Build once, clone forever.
│ │
│ └── No → Unattended (Answers file)
│ One USB stick per machine.
│ Or one USB stick + PXE for many.
│
└── No → Are you building many different machines?
└── Unattended with per-host answers files.
One answers.env per hostname.
Serve via HTTP or individual seed USBs.Most people start with Path 1 (Interactive). They boot the ISO, poke around the web UI, install a test system, break it, reinstall. That is the learning path. Once they know what they want, they write an answers file (Path 2) and never touch the web UI again. The answers file is the crystallized version of their interactive choices — a text file they can version-control, diff, and review.
Path 3 (Pipeline) is for teams. One engineer writes the Packer template. CI builds the golden image on every merge to main. Terraform deploys the golden image to staging, runs tests, promotes to production. The kldload ISO is stage zero of the pipeline — it produces the base image that everything else layers on top of. Nobody touches the ISO manually after the Packer template is written. The pipeline does it.
The important thing: all three paths produce the same machine. The ZFS layout is the same. The packages are the same. The services are the same. The boot chain is the same. The difference is only the delivery mechanism for the configuration. Web form, text file, or API call. Same installer. Same result.
Combining paths
The paths are not mutually exclusive. A common pattern: use Interactive to prototype a new
system type. Once you are happy, copy the install manifest (/etc/kldload/install-manifest.env)
from the installed system — it contains every choice you made. Turn it into an answers
file. Use that answers file for all future installs of this system type. If you have a pipeline,
feed the answers file to Packer. Three paths, one progression: interactive to learn, unattended
to repeat, pipeline to scale.
# After an interactive install, grab the manifest:
scp root@new-machine:/etc/kldload/install-manifest.env ./answers.env
# Review and clean it up (remove machine-specific values):
vim answers.env # Change hostname, adjust IP, review options
# Now use it for every future install of this machine type:
# Option A: Seed USB
mkfs.vfat -n KLDLOAD-SEED /dev/sdb1 && mount /dev/sdb1 /mnt
cp answers.env /mnt/ && umount /mnt
# Option B: Packer template
# Point Packer's http_content at the answers file
# Packer boots the ISO, serves answers via HTTP, triggers installProfiles explained
A profile controls what gets installed on top of the base distro. The base is always the same: kernel, ZFS modules, boot environments, core utilities. The profile adds layers on top. Four profiles, from most opinionated to least:
Desktop
GNOME + full kldload tools + everything. The complete experience.
Desktop environment, Firefox, terminal emulator, file manager, all k-tools
(kupgrade, ksnapshot, krestore, kstatus),
sanoid for automated snapshots, the kldload web UI for management, WireGuard tools,
eBPF observability stack. For workstations, development machines, demo systems, or
any machine where a human sits at the keyboard and wants a graphical environment.
Server
Headless SSH + full kldload tools. Everything Desktop has minus the GUI. All k-tools, sanoid, web UI on :8080, WireGuard, eBPF. Accessed via SSH or the web interface. This is the workhorse profile for production servers, VMs, appliances, and anything that runs headless. The web UI provides management without a desktop environment. Most common profile for production deployments.
Core
ZFS on root. Boot environments. Stock distro. Nothing else. No k-tools.
No sanoid. No web UI. No WireGuard configuration. No eBPF stack. No opinions. Just the
vendor distro with ZFS on root and boot environments. You get zfs and
zpool commands and a bootloader that understands ZFS datasets. Everything
else is your choice. This is the blank canvas for people who know exactly what they
want to build.
KVM (hypervisor)
Server profile + libvirt + QEMU/KVM. Everything Server has plus a complete virtualization stack. ZFS zvols as VM disks. Cockpit for web-based VM management. OVMF for UEFI VMs. PCI passthrough support. For building your own hypervisor with ZFS-backed storage that supports snapshots, clones, and replication of entire VMs at the storage level.
What each profile includes
Profile comparison matrix
Feature Desktop Server Core KVM
───────────────────────── ─────── ────── ──── ───
ZFS on root yes yes yes yes
Boot environments yes yes yes yes
ZFSBootMenu yes yes yes yes
k-tools (kupgrade, etc.) yes yes - yes
Sanoid snapshots yes yes - yes
Web UI (:8080) yes yes - yes
WireGuard configured yes yes - yes
eBPF tools yes yes - yes
Darksite (offline pkgs) yes yes - yes
GNOME desktop yes - - -
Firefox yes - - -
libvirt / QEMU / KVM - - - yes
Cockpit - - - yes
OVMF (UEFI VMs) - - - yesCore is the re-packer's re-packer. It gives you a stock vendor distro with ZFS on root and nothing else. No opinions. No tools. No scripts. Just the foundation. Use it as a Packer base image. Feed it to Terraform. Build your own postinstall that turns it into whatever you need. Core is for the person who read everything on this site, understood the primitives, and wants to build their own thing from scratch.
The reason Core exists: not everyone wants kldload's opinions. Some people want Debian with ZFS on root and literally nothing else. They will install their own packages, write their own systemd services, configure their own snapshot policy. Core gives them the one hard thing — ZFS on root with boot environments — and gets out of the way. If the Server profile is a furnished apartment, Core is a concrete slab with plumbing and electrical. Habitable, but you are bringing your own walls.
The KVM profile is for the person who wants to build their own Proxmox. A ZFS-backed hypervisor with web management, live migration support, and storage that understands snapshots and clones at the block level. Every VM disk is a ZFS zvol. Snapshotting a VM is zfs snapshot. Cloning a VM is zfs clone. Replicating a VM to another host is zfs send. The hypervisor storage layer IS ZFS. No abstraction. No translation layer. No storage API that maps to a different backend. Just ZFS.
Distro selection guide
Nine distros. Same ISO. Same installer. Same ZFS on root. The difference is the userland: package manager, release cadence, support lifecycle, available packages, and community. Picking the right distro is the first decision and the most consequential one. Here is when to pick each.
CentOS Stream 9
When to pick: You want RHEL compatibility without the subscription. You want a stable, well-tested base for production servers. You want packages that are vetted by Red Hat QE before they reach you. You are building infrastructure that needs to run for 5+ years without major changes.
Trade-offs: Packages are older than Fedora. Some bleeding-edge software requires third-party repos (EPEL). The "stream" model means it is slightly ahead of RHEL point releases, which some people dislike.
Debian 13 (Trixie)
When to pick: You want the most minimal, cleanest base image. You
value stability and predictability above all else. You are building servers, appliances,
or containers that need a small footprint. You prefer APT and .deb packages.
You trust the Debian project's legendary release process.
Trade-offs: Packages can be older than Ubuntu. The Debian ecosystem is smaller than RHEL's enterprise ecosystem. Some commercial software only ships RPMs.
Ubuntu 24.04 (Noble)
When to pick: You want the largest package ecosystem in Linux. You want first-class support from every software vendor (most ship Ubuntu packages first). You want LTS with 5 years of support. You are building something that needs a specific package that only exists in the Ubuntu universe repository. You want PPAs for bleeding-edge software without building from source.
Trade-offs: Snap packages can be controversial. Canonical makes choices that not everyone agrees with (cloud-first bias, snap enforcement for some packages). Heavier than Debian.
Fedora 41
When to pick: You want the latest kernel, the latest GNOME, the latest everything. You are willing to upgrade every 13 months. You are developing software that targets the newest Linux features (io_uring, eBPF programs that need 6.x kernels, Wayland-only apps). You want what RHEL will be in 2-3 years, today.
Trade-offs: Short lifecycle (13 months per release). Things break more often than CentOS or Debian. Not ideal for servers that must run untouched for years.
RHEL 9
When to pick: Your organization requires commercial support. You need FIPS 140-3 validated cryptography. You need a vendor to call when something breaks. You are in a regulated industry (finance, healthcare, government) that mandates supported platforms. You already have Red Hat subscriptions.
Trade-offs: Requires a subscription for package access. Packages are deliberately conservative. The RHEL ecosystem is enterprise-oriented, which means more documentation but more bureaucracy.
Rocky Linux 9
When to pick: You want RHEL binary compatibility without the subscription. You are migrating from CentOS Linux (pre-Stream). You want the same packages, the same paths, the same behavior as RHEL, but free. You need enterprise stability but not enterprise support contracts.
Trade-offs: Slightly behind RHEL on point releases. No vendor support (community only). Some RHEL-specific features (Insights, Satellite integration) are not available.
Arch Linux
When to pick: You want a rolling release with the absolute latest packages. You understand Linux deeply and want full control over every installed package. You are building a development workstation or a system where you need the newest kernel and the newest userland simultaneously. You enjoy the AUR.
Trade-offs: Requires internet (no darksite — rolling releases cannot be snapshotted into an offline mirror that stays current). Updates can break things. Not for production servers unless you enjoy living dangerously. The wiki is excellent but the learning curve is real.
Alpine Linux
When to pick: You want the smallest possible base image. You are building containers, firewalls, gateways, or embedded systems where every megabyte counts. You want musl instead of glibc. You want OpenRC instead of systemd. You want a system that boots in 2 seconds and uses 50 MB of RAM.
Trade-offs: musl libc has compatibility issues with some software that assumes glibc. No systemd means some software requires adaptation. Smaller package repository than Debian or Ubuntu. Not ideal for desktop use.
The distro question generates religious wars, and most of them miss the point. On kldload, the distro choice matters less than you think, because the hard parts are already solved. ZFS on root works on all nine distros. Boot environments work on all eight. WireGuard works on all eight. The k-tools work on all eight. The darksite works on all eight (except Arch). The postinstall mechanism works on all eight.
The distro choice matters for: package availability (does the specific software you need have a package for this distro?), support lifecycle (how long before you have to upgrade?), and organizational requirements (does your company mandate RHEL?). That is it. For everything else, pick the one you know best. Familiarity with the package manager and the ecosystem is worth more than any technical difference between them.
If you genuinely have no preference: Debian for servers, Fedora for workstations, Rocky for enterprise, Alpine for appliances. Those are safe defaults that nobody will argue with.
Customization layers
kldload gives you a layered customization model. Each layer builds on the one below it. You can stop at any layer. Most people use two or three. The layers are independent — you can use a postinstaller without cloud-init, or cloud-init without config management.
The four layers
Layer 0: Base image kldload ISO → install → ZFS on root + boot environments
The foundation. Same for everyone.
You control: distro, profile, ZFS topology, encryption.
Layer 1: Postinstaller postinstall.sh runs after kldload finishes the base.
Installs your packages, creates your datasets,
writes your config files, enables your services.
Baked into the ISO. Runs at install time. Offline.
Layer 2: Cloud-init Runs on first boot of the installed (or cloned) system.
Sets hostname, users, SSH keys, networking, scripts.
Personalizes a golden image for a specific machine.
Data comes from: NoCloud seed ISO, HTTP metadata, cloud API.
Layer 3: Config mgmt Ansible / Salt / Puppet / Chef — runs after the machine
is up and reachable. Handles application-layer changes:
deploy apps, write runtime configs, manage secrets, update.
Connects to a finished machine, not a blank one.Layer 0: Base image
This is what kldload builds. A bootable system with ZFS on root, boot environments, and your selected profile's tooling. You control the distro, the profile, the ZFS topology (single, mirror, raidz), encryption, networking, and features (KVM, eBPF, AI stack). The base image is defined entirely by the answers file or web UI choices. It is the same on every machine that uses the same answers.
Layer 1: Postinstaller scripts
The postinstaller is the most powerful customization layer. It runs after kldload finishes the base install, while the target disk is still mounted. It has full access to the installed system's filesystem. It can install packages, create ZFS datasets, write config files, enable systemd services, set up cron jobs — anything. It is a bash script embedded in the ISO that runs at install time.
# postinstall.sh — web server with ZFS-tuned datasets
#!/bin/bash
set -euo pipefail
# Install application packages
chroot /mnt apt install -y nginx postgresql-16 redis-server certbot
# Create ZFS datasets tuned per workload
zfs create -o recordsize=8K -o logbias=throughput rpool/srv/postgres
zfs create -o recordsize=8K -o logbias=latency rpool/srv/postgres/wal
zfs create -o recordsize=128K -o compression=zstd rpool/srv/redis
zfs create -o recordsize=128K -o compression=zstd rpool/var/www
zfs create -o recordsize=1M -o compression=off rpool/srv/media
# Set mountpoints
zfs set mountpoint=/var/lib/postgresql/16/main rpool/srv/postgres
zfs set mountpoint=/var/lib/redis rpool/srv/redis
zfs set mountpoint=/var/www rpool/var/www
zfs set mountpoint=/srv/media rpool/srv/media
# Copy application configs from the ISO
cp /root/app-configs/nginx.conf /mnt/etc/nginx/nginx.conf
cp /root/app-configs/postgresql.conf /mnt/etc/postgresql/16/main/postgresql.conf
cp /root/app-configs/redis.conf /mnt/etc/redis/redis.conf
# Enable services
chroot /mnt systemctl enable nginx postgresql redis-server
# Snapshot policy for application data
cat > /mnt/etc/cron.d/app-snapshots << 'SNAP'
*/15 * * * * root zfs snapshot -r rpool/srv@auto-$(date +\%Y\%m\%dT\%H\%M\%SZ)
0 */6 * * * root zfs snapshot -r rpool/var/www@auto-$(date +\%Y\%m\%dT\%H\%M\%SZ)
SNAP
echo "Postinstall complete: nginx + postgres + redis on ZFS"The postinstaller is the mechanism that turns kldload from an installer into a re-packer. Without postinstallers, kldload gives you a base OS with ZFS. With postinstallers, kldload gives you a complete application stack on ZFS. The postinstaller is your recipe. It encodes your infrastructure decisions: which packages, which datasets, which recordsizes, which configs, which services. Version-control it. Review it. Diff it between environments. It is your infrastructure as code, but simpler than Ansible and more powerful than a Dockerfile.
The postinstaller runs at install time, inside the installer environment, with the target filesystem mounted at /mnt. It has access to the darksite packages (offline) and the full installer environment. This means your postinstaller works in air-gapped environments. No internet required. The packages were already downloaded into the darksite during ./deploy.sh build. The postinstaller just installs them from local storage.
Layer 2: Cloud-init
Cloud-init runs on first boot of the installed system (or a clone of it). It personalizes the machine: hostname, users, SSH keys, networking, and first-boot scripts. This is the standard mechanism for customizing golden images across cloud and on-premises platforms. kldload configures cloud-init with multi-datasource support when sealing images for export.
The key insight: postinstallers define the role (what this machine does). Cloud-init defines the identity (who this machine is). A postinstaller says "this is a web server with nginx and postgres." Cloud-init says "this specific web server is called web-prod-01, has this IP, these SSH keys, and these DNS entries." The role is baked into the image. The identity is applied at boot time.
Layer 3: Configuration management
Ansible, Salt, Puppet, Chef — they all work on kldload machines because kldload machines are standard Linux. But they connect to a finished machine, not a blank one. The base is installed. The ZFS layout is done. The packages are installed. The services are running. Configuration management handles the application layer on top: deploying your app code, writing runtime configs, rotating secrets, pushing updates. 40 tasks instead of 400. 2 minutes instead of 20.
Where each layer runs
Layer When Where Requires network?
────── ───────────────────── ───────────────────── ─────────────────
Base During ISO install On target disk via ISO No (darksite)
Postinst During ISO install On target disk via ISO No (darksite)
Cloud-in On first boot On running system Depends on datasource
Config After machine is up Via SSH from mgmt host YesThe layering model is what makes kldload composable. You can stop at Layer 0 and have a perfectly good ZFS-on-root system. You can add a postinstaller (Layer 1) and have a complete application stack. You can add cloud-init (Layer 2) and have a cloneable golden image. You can add Ansible (Layer 3) and have a fully managed infrastructure. Each layer is optional. Each layer adds capability without requiring the layers above it.
Most small deployments (homelab, small business) use Layers 0-1 only. Build the ISO with a postinstaller, burn to USB, install. Done. No cloud-init because you are not cloning. No Ansible because you have three machines, not three hundred. Most enterprise deployments use all four layers. The ISO is built in CI, the postinstaller defines the role, cloud-init personalizes the clone, and Ansible manages the application lifecycle. Same kldload. Different scale. Different layers.
Example architectures
Theory is useful. Real-world examples are more useful. Here are four deployment architectures that people actually build with kldload, at different scales, for different purposes. Each one shows which distro, which profile, which customization layers, and how the four primitives (datasets, tunnels, boot environments, replication) combine.
Architecture 1: The homelab
One person, 2-5 machines, learning and hosting
┌─────────────────────────────────────────────────────────┐
│ HOMELAB │
│ │
│ ┌──────────────┐ WireGuard ┌──────────────┐ │
│ │ hypervisor │◄────────────►│ NAS/backup │ │
│ │ Fedora/KVM │ 10.100.0.x │ Debian/Core │ │
│ │ 4x NVMe │ │ 6x HDD RAIDZ2│ │
│ │ 64 GB RAM │ │ 2x SSD mirror │ │
│ └──────┬───────┘ └──────┬────────┘ │
│ │ zvol VMs │ zfs recv │
│ ┌────┴────┐ │ │
│ │ Desktop │ Fedora/Desktop │ │
│ │ k8s-dev │ Ubuntu/Server │ │
│ │ pihole │ Alpine/Core │ │
│ └─────────┘ │ │
│ │ │ │
│ └──── zfs send ────────────────┘ │
│ (nightly, incremental, over WireGuard) │
└─────────────────────────────────────────────────────────┘
Distros: Fedora (hypervisor), Debian (NAS), mixed (VMs)
Profiles: KVM (hypervisor), Core (NAS), mixed (VMs)
Layers: Base + Postinstaller (no cloud-init, no Ansible)
Primitives: Datasets (per-VM zvols, per-share NAS datasets)
Tunnels (WireGuard between hypervisor and NAS)
Boot environments (on hypervisor and NAS)
Replication (nightly zfs send from hypervisor to NAS)The homelab is where most people start. One machine with KVM, one machine for storage. WireGuard between them. ZFS replication every night. The hypervisor runs VMs on zvols — each VM disk is a ZFS dataset you can snapshot, clone, and replicate independently. The NAS receives replicated datasets and serves files via NFS/SMB. If the hypervisor dies, the NAS has every VM's data from last night. Restore is zfs send from NAS to a replacement machine.
The beauty of this architecture: it scales. Add a third machine and it becomes a mini-cluster. Add a fourth at a friend's house and it becomes an offsite DR site. The primitives are the same. The topology changes. The tooling does not.
Architecture 2: Small business
5-20 employees, 3-8 machines, production workloads
┌───────────────────────────────────────────────────────────────┐
│ SMALL BUSINESS (Office + Colo) │
│ │
│ ┌── Office ─────────────────┐ WG ┌── Colo ────────────┐ │
│ │ │ mesh │ │ │
│ │ ┌────────┐ ┌────────┐ │◄─────►│ ┌────────────┐ │ │
│ │ │ firewall│ │ NAS │ │ │ │ web server │ │ │
│ │ │ Alpine │ │ Rocky │ │ │ │ Debian │ │ │
│ │ │ Core │ │ Server │ │ │ │ Server │ │ │
│ │ └────────┘ └────┬───┘ │ │ └────────────┘ │ │
│ │ │ │ │ ┌────────────┐ │ │
│ │ ┌────────────┐ │ │ │ │ DB primary │ │ │
│ │ │ 5x Desktop │ │ │ │ │ Rocky │ │ │
│ │ │ Ubuntu │ │ │ │ │ Server │ │ │
│ │ │ Desktop │ │ │ │ └─────┬──────┘ │ │
│ │ └────────────┘ │ │ │ │ │ │
│ └────────────────────┘ │ └────────┘ │ │
│ │ │ │
│ zfs send: DB → NAS (hourly) │ zfs send: NAS → offsite │ │
│ zfs send: web → NAS (daily) │ (weekly, encrypted) │ │
└───────────────────────────────────────────────────────────────┘
Distros: Alpine (firewall), Rocky (NAS, DB), Debian (web), Ubuntu (desktops)
Profiles: Core (firewall), Server (NAS, DB, web), Desktop (workstations)
Layers: Base + Postinstaller + Unattended answers files
Primitives: Datasets (DB with recordsize=8K, web with recordsize=128K)
Tunnels (WireGuard mesh: office ↔ colo, encrypted)
Boot environments (on every server)
Replication (DB → NAS hourly, everything → offsite weekly)The small business architecture uses different distros for different roles. Alpine for the firewall because it is tiny and has minimal attack surface. Rocky for the database because it is enterprise-grade and long-lived. Debian for the web server because it has the best nginx and certbot packaging. Ubuntu for desktops because it has the largest application ecosystem. All connected via WireGuard. All backed by ZFS replication.
Architecture 3: Enterprise edge
50+ sites, standardized builds, central management
┌────────────────────────────────────────────────────────────────┐
│ ENTERPRISE EDGE │
│ │
│ ┌── HQ / Data Center ────────────────────────────────────┐ │
│ │ Packer builds golden images from kldload ISO (nightly) │ │
│ │ Terraform deploys to edge sites via IPMI + seed USB │ │
│ │ Ansible manages application layer post-deploy │ │
│ │ Prometheus + Grafana monitor all sites via WireGuard │ │
│ │ Central NAS receives ZFS replication from all sites │ │
│ └─────────────────────────────┬──────────────────────────┘ │
│ │ WireGuard hub-spoke │
│ ┌────────────────────┼────────────────────┐ │
│ │ │ │ │
│ ┌── Site A ──────┐ ┌── Site B ──────┐ ┌── Site C ──────┐ │
│ │ Rocky / Server │ │ Rocky / Server │ │ Rocky / Server │ │
│ │ edge-a-01 │ │ edge-b-01 │ │ edge-c-01 │ │
│ │ edge-a-02 │ │ edge-b-02 │ │ edge-c-02 │ │
│ │ ZFS encrypted │ │ ZFS encrypted │ │ ZFS encrypted │ │
│ └────────────────┘ └────────────────┘ └────────────────┘ │
│ │
│ All edge nodes: same golden image, same answers file │
│ Differentiated by: hostname, IP, site-specific cloud-init │
│ Replication: continuous to HQ, hourly between site peers │
└────────────────────────────────────────────────────────────────┘
Distros: Rocky (standardized across all sites for support lifecycle)
Profiles: Server (all edge nodes)
Layers: All four: Base + Postinstaller + Cloud-init + Ansible
Primitives: Datasets (ZFS encrypted for physical security at edge)
Tunnels (hub-spoke WireGuard: every site → HQ)
Boot environments (safe remote upgrades, rollback via IPMI)
Replication (continuous to HQ, hourly site-to-site)The enterprise edge architecture is where all four customization layers come together. Packer builds the golden image from the kldload ISO with a postinstaller (Layers 0-1). Cloud-init personalizes each clone with site-specific identity (Layer 2). Ansible manages the application lifecycle after deployment (Layer 3). The CI pipeline produces a new golden image on every merge to main. Terraform deploys it to staging edge sites. Tests run. Green light → promote to production edge sites. Blue/green deployment at the edge, powered by ZFS boot environments.
ZFS encryption is critical at the edge. These are physical machines in branch offices, retail stores, warehouses. Someone could steal one. With ZFS native encryption, the data at rest is encrypted with a key that is not stored on the machine — it comes from a remote key server via WireGuard on boot. If someone steals the hardware, they get encrypted blocks. If they unplug the WAN, the machine continues to operate with cached data but cannot decrypt new datasets without the key server.
Boot environments make remote upgrades safe. Push a new image via ZFS replication. Set it as the next boot environment. Reboot remotely via IPMI. If the new image fails to boot (watchdog timeout), ZFSBootMenu rolls back to the previous boot environment automatically. The site is down for one reboot cycle, then back on the known-good image. No technician dispatched. No panic. The rollback is structural.
Architecture 4: Development team
5-15 developers, shared infrastructure, fast iteration
┌────────────────────────────────────────────────────────────────┐
│ DEVELOPMENT TEAM │
│ │
│ ┌── Dev workstations ──────────────────────────────────────┐ │
│ │ Each developer: Ubuntu or Fedora / Desktop profile │ │
│ │ ZFS on root. Docker on ZFS storage driver. │ │
│ │ Per-project datasets. Snapshots before every rebase. │ │
│ │ KVM for local testing VMs (cloned from shared golden). │ │
│ └──────────────────────────┬───────────────────────────────┘ │
│ │ WireGuard mesh │
│ ┌── Shared infra ──────────┴───────────────────────────────┐ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
│ │ │ CI runner │ │ staging │ │ registry │ │ │
│ │ │ Debian │ │ Rocky │ │ Debian │ │ │
│ │ │ Server │ │ Server │ │ Core │ │ │
│ │ │ ephemeral │ │ clone of │ │ Harbor │ │ │
│ │ │ ZFS clones│ │ prod img │ │ on ZFS │ │ │
│ │ └──────────┘ └──────────┘ └──────────┘ │ │
│ │ │ │
│ │ CI: clone golden dataset → build → test → destroy clone │ │
│ │ Staging: zfs clone of production golden image │ │
│ │ Registry: Harbor container images on ZFS datasets │ │
│ └───────────────────────────────────────────────────────────┘ │
│ │
│ Golden images built by CI from kldload ISO + postinstallers │
│ Developers clone golden images for local testing │
│ Staging is a ZFS clone of the production golden image │
│ Everything connected via WireGuard. Everything snapshotted. │
└────────────────────────────────────────────────────────────────┘
Distros: Mixed (Ubuntu/Fedora desktops, Debian CI/registry, Rocky staging)
Profiles: Desktop (workstations), Server (CI, staging), Core (registry)
Layers: Base + Postinstaller + Cloud-init (for cloned VMs)
Primitives: Datasets (per-project, per-build, ephemeral clones)
Tunnels (WireGuard mesh for all dev infra)
Boot environments (safe upgrades on every machine)
Replication (golden images distributed via zfs send)The development team architecture leverages ZFS clones aggressively. The CI runner clones a golden build dataset for each build — O(1) creation, zero cost, destroyed after the build. Staging is a ZFS clone of the production golden image — identical to production at the block level, but writable and independent. Developers clone the staging image to their workstations for local testing. Every clone is instant, free, and disposable. This is the ZFS workflow that has no equivalent on ext4 or XFS.
The pattern across all four architectures: kldload builds the base, postinstallers customize per role, ZFS primitives handle the hard problems (storage, backup, cloning, replication), and WireGuard handles the networking. The scale changes. The tooling does not. A homelab with 2 machines and an enterprise with 200 use the same four primitives, the same ISO, the same profiles, the same postinstaller mechanism. The difference is the number of machines and the number of customization layers.
That is the re-packer promise. Learn the primitives once. Apply them at any scale. The recipes in the sidebar show you how. The architectures on this page show you why. The zero-to-hero guide shows you the commands. Pick a starting point and build.
The re-packer philosophy
kldload re-packs a vendor distro with kernel modules baked in. You can re-pack kldload's
output with your application baked in. The mechanism is postinstall.sh —
a hook script that runs after the base install completes. Put your packages, your configs,
your services, your datasets in the postinstall. Build the ISO. Every machine deployed from
that ISO has your application pre-installed on ZFS with snapshots, encryption, and
replication — before anyone touches a keyboard.
# postinstall.sh — runs after kldload installs the base system
#!/bin/bash
# Your application
apt install -y nginx postgresql redis
# Your datasets — tuned per workload
zfs create -o recordsize=8K -o logbias=throughput rpool/srv/postgres
zfs create -o recordsize=128K -o compression=zstd rpool/srv/redis
zfs create -o recordsize=1M -o compression=off rpool/srv/media
# Your config
cp /path/to/your/nginx.conf /etc/nginx/nginx.conf
systemctl enable nginx postgresql redis
# Your snapshot policy
echo '0 * * * * root zfs snapshot -r rpool/srv@auto-$(date +\%Y\%m\%dT\%H\%M\%SZ)' \
> /etc/cron.d/app-snapshots
That is the entire deployment pipeline. One postinstall script. Every machine gets it.
No Ansible playbook. No Puppet manifest. No Chef cookbook. The image is the artifact.
The postinstall is your recipe. ./deploy.sh build produces a USB that installs
your complete stack — OS, storage, networking, application — offline, in minutes.
The re-packer philosophy applies at every level. kldload re-packs Debian with ZFS. You re-pack kldload with nginx. Your customer re-packs your output with their application data. Each layer adds value without understanding or modifying the layers below. Debian does not know about ZFS. kldload does not know about nginx. Your postinstaller does not know about the customer's data. Clean separation. Each layer does one thing.
This is the Unix philosophy applied to infrastructure. Small, composable tools. Each one does one thing well. The re-packer is not one monolithic system — it is a pipeline of independent stages that compose into whatever you need.
What changes when you build on ZFS
Every recipe in the sidebar, every architecture above, every build you do with kldload inherits ZFS. This is not a cosmetic difference. It changes how you think about storage, backup, deployment, and failure recovery at a structural level.
The structural difference
On ext4:
Install app → hope nothing breaks → write a backup script
→ schedule the backup → test the restore → maintain it forever
→ hope the backup script still works after you updated rsync
→ hope the target disk has enough space → hope the cron job ran
On ZFS:
Install app → it's already on a snapshotted, checksummed, replicable dataset
→ rollback is one command → replication is one command → no backup script
→ no backup schedule → no restore testing → it's structural, not bolted on
→ snapshots are free (O(1), milliseconds, zero space until divergence)
→ replication is incremental (only changed blocks, checksummed, resumable)
Every recipe below inherits this. The Plex server has per-library datasets. The hypervisor
has zvol-backed VMs. The database has tuned recordsizes. The cluster replicates over WireGuard.
The CI runner uses ephemeral clones. The classroom lab resets by destroying and re-cloning.
None of them mention "backups" because backups are snapshots that already happened. None of
them mention "restore testing" because restoring a ZFS snapshot is zfs rollback
— it has never failed. The filesystem is the safety net. You do not bolt one on.
Per-workload tuning
Every application gets a dataset with properties tuned for its access pattern.
recordsize=8K for databases. recordsize=1M for media.
sync=disabled for ephemeral CI workspaces. compression=zstd
for everything compressible. encryption=aes-256-gcm for sensitive data.
On ext4, all of this is one partition with one set of mount options. On ZFS, every
workload is independently optimized.
Cloning as deployment
ZFS clone is O(1) — instant, regardless of dataset size. This means deployment IS cloning. Spin up a new VM? Clone the golden zvol. Spin up a CI build? Clone the golden workspace. Spin up a staging environment? Clone production. The clone shares all blocks with the parent until it writes new data. 100 clones of a 50 GB image use 50 GB total, not 5 TB. Cloning replaces copying, imaging, and provisioning.
Replication as backup
zfs send | zfs recv moves datasets between machines at the block level.
Incremental sends transmit only changed blocks since the last snapshot. Checksummed
end to end. Resumable if interrupted. Encrypted in flight over WireGuard. This is not
rsync. This is not tar. This is block-level replication that the filesystem understands
natively. Your "backup strategy" is a cron job: zfs send -i @prev @now | ssh
remote zfs recv.
Boot environments as safety
Every time you upgrade the kernel, update packages, or change a system config, you can snapshot the root dataset first. If the change breaks the system, boot the previous snapshot from ZFSBootMenu. 15 seconds. The machine is back to exactly where it was before the change. This is not "restore from backup." This is "boot a different dataset." It is instant, local, and guaranteed to work because the old data was never modified.
The reason every recipe and every architecture on this site starts with ZFS is not because ZFS is trendy or because it has cool features. It is because ZFS solves the hard problems structurally. Backup is solved by snapshots. Cloning is solved by copy-on-write. Replication is solved by zfs send. Data integrity is solved by checksumming. Boot recovery is solved by boot environments. These are not features you turn on. They are properties of the filesystem that exist whether you use them or not.
When you build on ext4 or XFS, you bolt on solutions to these problems: rsync for backup, cp or dd for cloning, rsync again for replication, nothing for data integrity, nothing for boot recovery. Every bolted-on solution is another script to maintain, another cron job to monitor, another thing that can fail silently at 3 AM. ZFS eliminates the bolted-on solutions because the filesystem IS the solution. That is the structural difference. That is why kldload builds everything on ZFS.
Where to start
You have read the concept, the build process, the paths, the profiles, the distro guide, the customization layers, and the architectures. Now pick a starting point.
Never used kldload before?
Start with Zero to Hero.
It takes you from git clone to a running system in every command. You will
build the ISO, boot it, install a system, and understand every step. Takes about an hour.
Know what you want to build?
Go straight to the recipe. Edge Server, CI Runner, NAS Server, Firewall, Database, Containers, Kubernetes. Each recipe includes the answers file, the postinstaller, and the ZFS dataset layout.
Want to automate everything?
Read IaC Quickstart and Automation. Covers unattended install, golden image workflow, Packer integration, cloud-init, and the full CI/CD pipeline for infrastructure images.
Want to understand ZFS first?
Start with ZFS Zero to Hero or the ZFS Wiki. ZFS is the foundation of everything kldload builds. Understanding datasets, snapshots, clones, and replication will make every recipe and architecture on this site click.
The re-packer has a learning curve. ZFS has concepts you have never encountered if you have only used ext4. Boot environments have no equivalent in traditional Linux. WireGuard mesh networking requires understanding key exchange. Darksites require understanding how package managers resolve dependencies. None of this is hard. All of it is new. Give yourself an afternoon to build your first system. A weekend to break it and rebuild it. A week to build something from the sidebar. A month to build something that is not in the sidebar.
The payoff is infrastructure you understand completely, that you built yourself, that runs on your hardware, that you can fix at 3 AM because you know where every file is and what every service does. Not because you memorized it, but because you put it there. That is the whole point of building your own.