Kernel vs Userland — why boot order matters.
There are two ways to add capabilities to a Linux system: load them into the kernel, or run them as a process in userspace. Both work. But they have fundamentally different security properties, and the difference comes down to when they start.
The question isn't "does this feature exist on the system?"
The question is: "At what point during boot is this feature protecting me?"
What boots when
Two models
Push Mode / Userland Daemon
A process starts, reads configuration, and pushes eBPF programs or firewall rules into the kernel. Examples: Cilium, Calico, Falco, Datadog agent.
Native / Kernel Module (DKMS)
A module loads with the kernel itself. It's in memory before any process exists. Examples: WireGuard, ZFS, NVIDIA drivers, eBPF subsystem.
The trade-offs
| Userland (push) | Kernel (native) | |
|---|---|---|
| Available at | 3–15 seconds after boot | Instant — loads with the kernel |
| Vulnerability window | Yes — unprotected until daemon starts | None — active from second zero |
| Can be killed | Yes — crash, OOM, misconfiguration | No — cannot be killed from userspace |
| Update method | Reload config, restart daemon. No reboot. | DKMS rebuild on kernel upgrade. May need reboot. |
| Hot reload | Yes — change policy in seconds | Limited — module parameters, not full reconfiguration |
| Attack surface | Daemon process, config files, API endpoints | Kernel memory only — no API, no config parsing at runtime |
| Trusted boot chain | Not part of it — loads after boot completes | Signed via MOK — tamper-evident from firmware |
| Maintenance | Simple — package update, restart | DKMS rebuild on kernel upgrades across distros |
| Flexibility | High — dynamic policy, runtime changes | Low — behaviour defined at load time |
There are actually three models, not two. The table above groups "kernel module" as one thing, but how the module gets onto the machine changes everything.
Model 1: Userland daemon. Starts 3-15 seconds after boot. Can crash. Can be killed. Has a config file, an API, a PID file, a log, a restart policy. The attack surface is the daemon itself.
Model 2: DKMS (compile after install). You install the OS, then apt install zfs-dkms. DKMS downloads the source and compiles it against your kernel headers. This happens at install time and again after every kernel update. It requires gcc, make, autoconf, and kernel-devel on the target machine. If any of those are missing, wrong version, or the kernel update changes an ABI — the module doesn't build. The machine boots, systemd starts, and ZFS isn't loaded. Monitoring says green. Your data is unreachable. This is how most people run ZFS on Linux today. The most common failure mode: an unattended kernel update at 3am, a DKMS build that fails silently, and a machine that hangs at boot because ZFS can't mount the root filesystem. You find out at 9am when the dashboard goes red.
Model 3: Build-time (what kldload does). The module is compiled and signed during image creation — before it ever touches a target machine. The kernel and the module are a matched pair. They ship together in the same image. No compiler on the target. No kernel headers. No DKMS in the boot path. The module is embedded in the initramfs and loaded by the bootloader before userland exists. If the machine boots, the module is loaded. Period. There is no state where the OS is up and the module isn't.
The signing chain: During image build, DKMS generates a Machine Owner Key (MOK) keypair and signs every out-of-tree module (zfs.ko, spl.ko) using the kernel's sign-file tool. The signed modules and the MOK public certificate ship in the image. On Secure Boot systems, the MOK is enrolled in firmware on first boot — after that, only modules signed by that key will load. A tampered or replaced module fails signature verification and the kernel refuses it. DKMS on a production machine? The module is unsigned by default. Anyone who can write to /lib/modules can replace it with anything and it loads without question.
This is the actual difference between kldload and "just install ZFS." The module is identical. The operational model is completely different. DKMS is a build system that runs on your production machine. kldload moves that build to image creation time and ships the result as a signed, verified artifact.
How kldload layers them
This isn't a choice between one model or the other. They're complementary. The kernel layer closes the vulnerability window. The userland layer provides flexibility on top. kldload ships both — pre-assembled, in the right order.
Layer 1 — Kernel (second zero)
WireGuard — network encryption active before any process starts. ZFS — checksummed, encrypted filesystem from first mount. eBPF subsystem — the tracing infrastructure is live in kernel memory. NVIDIA — GPU available without waiting for a runtime.
Layer 2 — Userland (seconds later)
eBPF tools — execsnoop, tcplife, biolatency attach to the kernel subsystem. Cilium — loads network policies via eBPF (on K8s workloads). Ollama — AI inference daemon. Sanoid — automated ZFS snapshot scheduling.
Why this matters
Most platforms ship userland tools and call it security. The daemon starts, loads its rules, and then you're protected. But what about the seconds before that? What about when the daemon crashes at 3am?
kldload closes the window at the kernel level. WireGuard encrypts the network before systemd starts. ZFS checksums every block before any process reads it. The eBPF subsystem is tracing before any observability daemon initializes.
By the time Cilium loads its first policy, kldload has been protecting the system for 10 seconds.
That's not a minor architectural detail. That's the difference between a system that is secure by default and one that becomes secure eventually.