Defense in depth. No trust by default. No agents. No subscriptions.

kldload's security model starts at the firmware and works upward through every layer of the stack: verified boot chain, signed kernel modules, encrypted storage, encrypted networking, kernel-level intrusion detection, and air-gapped supply chain. Each layer operates independently. None of them require userland agents, cloud services, or vendor subscriptions. They are kernel primitives, available from second zero.

Security philosophy — defense in depth

Defense in depth is not a marketing term. It means that every layer assumes the layer above and below it has been compromised. The boot chain verifies the kernel. The kernel verifies its modules. ZFS verifies every block. WireGuard encrypts every packet. eBPF watches every syscall. Each layer is a complete security boundary on its own. Together, they create overlapping fields of verification that an attacker must defeat simultaneously.

Out-of-tree kernel modules — compiled and signed at build time

kldload depends on four out-of-tree kernel modules: ZFS (storage), WireGuard (networking), eBPF CO-RE programs (monitoring), and optionally NVIDIA (GPU compute). These modules are not included in the upstream kernel source tree. They must be compiled against the running kernel headers and, for Secure Boot, signed with a trusted key.

Most Linux distributions handle this with DKMS (Dynamic Kernel Module Support) — the module source is installed on the target, and DKMS compiles it when the kernel updates. The problem: DKMS-compiled modules are unsigned. If Secure Boot is enabled, unsigned modules will not load. The administrator's choices are: disable Secure Boot (bad), or sign modules manually on every kernel update (tedious, error-prone).

kldload takes a different approach. Modules are compiled and signed at ISO build time, inside the builder container. The build generates a unique MOK (Machine Owner Key) keypair, compiles every out-of-tree module against the target kernel, signs each .ko file with the MOK private key, and embeds both the signed modules and the MOK public certificate into the ISO.

# What happens during ISO build (inside builder/build-iso.sh):

# 1. Generate unique MOK keypair for this build
openssl req -new -x509 -newkey rsa:2048 -keyout MOK.priv -outform DER \
  -out MOK.der -nodes -days 36500 -subj "/CN=kldload-$(date +%Y%m%d)/"

# 2. Compile ZFS against the target kernel headers
./configure --with-linux=/lib/modules/$(uname -r)/build
make -j$(nproc)

# 3. Sign every .ko file with the MOK key
for mod in zfs.ko spl.ko znvpair.ko zcommon.ko zunicode.ko zavl.ko \
           zlua.ko icp.ko zzstd.ko; do
  /usr/src/kernels/$(uname -r)/scripts/sign-file sha256 \
    MOK.priv MOK.der "$mod"
done

# 4. Verify signatures
modinfo zfs.ko | grep "^sig"
# sig_id:         PKCS#7
# signer:         kldload-20260404
# sig_key:        A3:2F:... (MOK key fingerprint)
# sig_hashalgo:   sha256

MOK enrollment on first boot

The signed modules will only load if the UEFI firmware trusts the MOK certificate. On first boot with Secure Boot enabled, the user must enroll the MOK certificate through the shim/MokManager interface. This is a one-time process:

# Stage the MOK certificate for enrollment
mokutil --import /root/MOK.der
# Enter a one-time password (used only during the next reboot)

# Reboot → MokManager appears before the bootloader
# Select "Enroll MOK" → confirm → enter the password → continue boot

# Verify enrollment
mokutil --list-enrolled | grep kldload
# CN=kldload-20260404

# Verify module loads with signature
modprobe zfs
dmesg | grep -i "module verification"
# [    2.14] zfs: module verification succeeded

Why this matters for supply chain security

Module signing is not just about Secure Boot compliance. It is a supply chain verification mechanism. When the kernel enforces module signatures, an attacker who gains root cannot load a malicious kernel module without possessing the signing key. Even if they replace /lib/modules/.../zfs.ko on disk, the kernel will reject the tampered module on the next load because the signature will not match.

# Attempt to load an unsigned or tampered module with Secure Boot active
modprobe evil_rootkit
# modprobe: ERROR: could not insert 'evil_rootkit': Required key not available

# Check kernel lockdown status
cat /sys/kernel/security/lockdown
# integrity
# (or "confidentiality" if LOCKDOWN_CONFIDENTIALITY is enabled)

# Kernel lockdown prevents:
#   - Loading unsigned modules
#   - Writing to /dev/mem or /dev/kmem
#   - Accessing MSRs from userland
#   - Using kexec to load unsigned kernels
#   - Writing to ACPI tables

WireGuard 4-plane backplane

kldload does not run a single WireGuard tunnel for all traffic. It creates four separate WireGuard interfaces, each with its own keypair, its own subnet, and its own firewall rules. This is the backplane architecture. The four planes are:

# Four separate WireGuard interfaces, four separate key pairs
ip link show type wireguard
# wg-enroll:  10.200.0.0/24   (enrollment — temporary)
# wg-mgmt:   10.201.0.0/24   (management — SSH, Ansible)
# wg-mon:    10.202.0.0/24   (monitoring — Prometheus, syslog)
# wg-stor:   10.203.0.0/24   (storage — ZFS replication, NFS)

# Each interface has its own private key
wg show wg-mgmt | head -3
# interface: wg-mgmt
#   public key: aB3d...Kx8=
#   listening port: 51821

wg show wg-stor | head -3
# interface: wg-stor
#   public key: Zy9f...Qm4=    ← different key
#   listening port: 51823

# Verify plane isolation — management cannot reach storage subnet
ip route get 10.203.0.5 from 10.201.0.1
# RTNETLINK answers: Network is unreachable (by design)

# nftables enforces strict source/destination per plane
nft list chain inet filter wg_plane_isolation

Backplane configuration

# /etc/wireguard/wg-mgmt.conf — management plane
[Interface]
PrivateKey = <management-private-key>
Address = 10.201.0.1/24
ListenPort = 51821

[Peer]
PublicKey = <peer-management-public-key>
AllowedIPs = 10.201.0.2/32
Endpoint = 203.0.113.50:51821
PersistentKeepalive = 25

# /etc/wireguard/wg-stor.conf — storage plane (different key!)
[Interface]
PrivateKey = <storage-private-key>
Address = 10.203.0.1/24
ListenPort = 51823

[Peer]
PublicKey = <peer-storage-public-key>
AllowedIPs = 10.203.0.2/32
Endpoint = 203.0.113.50:51823
PersistentKeepalive = 25

# Bring up all planes at boot
systemctl enable wg-quick@wg-mgmt wg-quick@wg-mon wg-quick@wg-stor

# Verify all planes are up
wg show all | grep "latest handshake"

ZFS encryption — per-dataset AES-256-GCM

ZFS native encryption operates at the dataset level, not the disk level. Each dataset (or zvol) has its own encryption key. The key is wrapped with a user-supplied passphrase or keyfile. This means you can have dozens of datasets, each with a different owner and a different key, on the same pool. Unlock Alice's home directory without unlocking Bob's. Unlock the application dataset without unlocking the audit logs.

# Create an encrypted dataset with passphrase
zfs create -o encryption=aes-256-gcm -o keyformat=passphrase \
  -o keylocation=prompt rpool/secrets
# Enter passphrase: ********

# Create an encrypted dataset with a keyfile (better for automation)
dd if=/dev/urandom of=/root/.keys/app-data.key bs=32 count=1
chmod 600 /root/.keys/app-data.key
zfs create -o encryption=aes-256-gcm -o keyformat=raw \
  -o keylocation=file:///root/.keys/app-data.key rpool/app-data

# Child datasets inherit encryption by default
zfs create rpool/secrets/certs     # encrypted, same key as parent
zfs create rpool/secrets/tokens    # encrypted, same key as parent

# Child dataset with its OWN key (different from parent)
zfs create -o keyformat=passphrase -o keylocation=prompt \
  rpool/secrets/root-only
# This dataset requires a separate unlock — compromise of the parent
# key does not expose this child

# Check encryption status
zfs get encryption,keystatus,keyformat rpool/secrets
# NAME            PROPERTY     VALUE
# rpool/secrets   encryption   aes-256-gcm
# rpool/secrets   keystatus    available
# rpool/secrets   keyformat    passphrase

Key management and boot-time unlock

The boot dataset (rpool/ROOT) is not encrypted by default. The kernel and initramfs must be readable by the bootloader without a passphrase. Encrypted datasets are unlocked after boot, either interactively or via a systemd unit that reads a keyfile.

# Lock a dataset (unload the key — data becomes inaccessible)
zfs unload-key rpool/secrets
zfs get keystatus rpool/secrets
# keystatus  unavailable

# Unlock at boot via systemd
# /etc/systemd/system/zfs-unlock-secrets.service
[Unit]
Description=Unlock encrypted ZFS datasets
After=zfs-mount.service

[Service]
Type=oneshot
ExecStart=/sbin/zfs load-key -a
# -a loads keys for ALL encrypted datasets using their keylocation

[Install]
WantedBy=multi-user.target

# For passphrase-based datasets, use a keyfile instead for automated boot:
zfs change-key -o keyformat=raw \
  -o keylocation=file:///root/.keys/secrets.key rpool/secrets

# Verify key rotation worked
zfs get keylocation rpool/secrets
# keylocation  file:///root/.keys/secrets.key

Raw send — encrypted replication to untrusted targets

zfs send -w (raw send) transmits encrypted blocks without decrypting them. The destination receives and stores ciphertext. It can serve the dataset, scrub it, snapshot it — but it cannot read the data. The decryption key never leaves the source.

# Send encrypted dataset to an untrusted remote host
zfs snapshot rpool/secrets@backup-2026-04-04
zfs send -w rpool/secrets@backup-2026-04-04 | \
  ssh backup-host zfs recv tank/offsite/secrets

# On the remote host — the data is there but unreadable
ssh backup-host zfs get encryption,keystatus tank/offsite/secrets
# encryption   aes-256-gcm
# keystatus    unavailable    ← the remote cannot decrypt

# The remote can do incremental receives (still encrypted)
zfs snapshot rpool/secrets@backup-2026-04-05
zfs send -w -i @backup-2026-04-04 rpool/secrets@backup-2026-04-05 | \
  ssh backup-host zfs recv tank/offsite/secrets

# Data sovereignty: cloud provider stores your backups,
# physically cannot read them. No envelope encryption.
# No key escrow. The key stays on your hardware.

eBPF security monitoring — the kernel watches itself

eBPF (extended Berkeley Packet Filter) allows small, verified programs to run inside the Linux kernel at specific hook points. kldload uses eBPF for security monitoring: process execution auditing, file integrity monitoring, network intrusion detection, and container escape detection. All without userland agents. All at kernel speed. All invisible to the processes being monitored.

Process execution auditing

# Trace every process execution on the system — kernel-level, no agent
bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
  printf("%s [pid=%d uid=%d] %s\n",
    comm, pid, uid, str(args->filename));
}'
# sshd [pid=1423 uid=0] /usr/sbin/sshd
# bash [pid=1424 uid=1000] /bin/bash
# curl [pid=1425 uid=1000] /usr/bin/curl

# Detect suspicious executions — shells spawned by web servers
bpftrace -e 'tracepoint:syscalls:sys_enter_execve
  /comm == "nginx" || comm == "httpd" || comm == "apache2"/ {
  printf("ALERT: web server spawned process: %s [pid=%d] %s\n",
    comm, pid, str(args->filename));
}'

# Log all setuid/setgid executions (privilege escalation attempts)
bpftrace -e 'tracepoint:syscalls:sys_enter_execve /uid != 0/ {
  @[comm, str(args->filename)] = count();
}'

File integrity monitoring

# Monitor writes to critical system files — kernel-level FIM
bpftrace -e 'tracepoint:syscalls:sys_enter_openat
  /str(args->filename) == "/etc/passwd" ||
   str(args->filename) == "/etc/shadow" ||
   str(args->filename) == "/etc/sudoers"/ {
  printf("ALERT: %s [pid=%d uid=%d] opened %s flags=%d\n",
    comm, pid, uid, str(args->filename), args->flags);
}'

# Detect writes to /usr/bin or /usr/sbin (binary tampering)
bpftrace -e 'kprobe:vfs_write {
  $path = str(((struct file *)arg0)->f_path.dentry->d_name.name);
  if (strncmp($path, "/usr/bin/", 9) == 0 ||
      strncmp($path, "/usr/sbin/", 10) == 0) {
    printf("ALERT: write to system binary: %s by %s [pid=%d]\n",
      $path, comm, pid);
  }
}'

# Compare: traditional FIM (AIDE)
#   - Runs periodically (cron)
#   - Detects changes after the fact
#   - Attacker modifies file, AIDE detects on next scan (hours later)
#
# eBPF FIM:
#   - Runs continuously in the kernel
#   - Detects the write syscall in real time
#   - Attacker's write triggers an alert before the syscall returns

Network intrusion detection

# Detect outbound connections to unexpected destinations
bpftrace -e 'tracepoint:syscalls:sys_enter_connect /uid > 0/ {
  $sa = (struct sockaddr_in *)args->uservaddr;
  if ($sa->sin_family == 2) {  // AF_INET
    printf("%s [pid=%d uid=%d] → %s:%d\n",
      comm, pid, uid,
      ntop($sa->sin_addr.s_addr), ntohs($sa->sin_port));
  }
}'

# Detect port scanning — rapid connection attempts to sequential ports
bpftrace -e 'tracepoint:syscalls:sys_enter_connect {
  @scan[pid, comm] = count();
}
interval:s:5 {
  print(@scan);
  clear(@scan);
}
# Any process with >50 connections in 5 seconds is probably scanning

# XDP-based packet filtering (runs before the kernel network stack)
# Drop packets from known-bad sources at wire speed
# See: /usr/lib/kldload-installer/ebpf/xdp-blocklist.c

Container escape detection

# Detect namespace changes — container escape indicator
bpftrace -e 'kprobe:switch_task_namespaces {
  printf("ALERT: namespace switch by %s [pid=%d uid=%d]\n",
    comm, pid, uid);
}'

# Detect mount namespace breakouts
bpftrace -e 'tracepoint:syscalls:sys_enter_mount {
  printf("ALERT: mount syscall by %s [pid=%d] src=%s type=%s\n",
    comm, pid, str(args->dev_name), str(args->type));
}'

# Detect access to host PID namespace from container
bpftrace -e 'tracepoint:syscalls:sys_enter_ptrace {
  printf("ALERT: ptrace by %s [pid=%d uid=%d] target_pid=%d\n",
    comm, pid, uid, args->pid);
}'

Firewall — nftables default deny

kldload configures nftables with a default deny policy on all profiles. Every port is closed unless explicitly opened. Management access (SSH) is restricted to the WireGuard management plane — it is not reachable from the physical network interface. The firewall is active from first boot, before any service starts.

# /etc/nftables.conf — kldload default ruleset
table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;

    # Allow established/related connections
    ct state established,related accept
    ct state invalid drop

    # Allow loopback
    iif lo accept

    # Allow ICMP (ping) — rate limited
    ip protocol icmp limit rate 10/second accept
    ip6 nexthdr icmpv6 limit rate 10/second accept

    # Allow SSH only on WireGuard management interface
    iifname "wg-mgmt" tcp dport 22 accept

    # Allow Prometheus scrape only on monitoring plane
    iifname "wg-mon" tcp dport 9090 accept
    iifname "wg-mon" tcp dport 9100 accept

    # Allow ZFS replication only on storage plane
    iifname "wg-stor" tcp dport 22 accept

    # Allow WireGuard UDP on physical interface
    udp dport { 51820, 51821, 51822, 51823 } accept

    # Everything else: drop (default policy)
    # No log — dropped packets are the norm, not the exception
  }

  chain forward {
    type filter hook forward priority 0; policy drop;
    # No forwarding by default — this is not a router
  }

  chain output {
    type filter hook output priority 0; policy accept;
    # Outbound is unrestricted by default
    # Restrict for high-security environments:
    # policy drop;
    # oifname "wg-*" accept
    # ct state established,related accept
  }
}

# Plane isolation — prevent cross-plane routing
table inet wg_isolation {
  chain forward {
    type filter hook forward priority 10; policy drop;
    # Traffic on wg-mgmt cannot route to wg-stor subnet
    iifname "wg-mgmt" oifname "wg-stor" drop
    iifname "wg-stor" oifname "wg-mgmt" drop
    iifname "wg-mon"  oifname "wg-mgmt" drop
    iifname "wg-mon"  oifname "wg-stor" drop
  }
}

# Verify the firewall is active
nft list ruleset | head -20

# Test from outside the WireGuard tunnel — SSH is unreachable
ssh root@physical-ip-address
# Connection refused (port 22 is not open on the physical interface)

# Test from inside the management tunnel — SSH works
ssh -o ProxyJump=none root@10.201.0.1
# Welcome to kldload...

# List active rules with counters
nft list chain inet filter input -a
# tcp dport 22 accept  # packets 1423  bytes 189204  (management plane only)

# Temporarily open a port for debugging (does not persist across reboot)
nft add rule inet filter input tcp dport 8080 accept

# Persist changes
nft list ruleset > /etc/nftables.conf
systemctl restart nftables

SSH hardening

kldload hardens SSH from first boot. Password authentication is disabled for root. Key-only authentication is the default. The SSH daemon listens only on the WireGuard management interface, not on the physical network. An attacker on the LAN cannot even see an SSH port.

# /etc/ssh/sshd_config — kldload defaults
PermitRootLogin prohibit-password     # Key-only for root
PasswordAuthentication no             # No passwords for anyone
PubkeyAuthentication yes              # Keys only
AuthorizedKeysFile .ssh/authorized_keys

# Listen only on WireGuard management plane
ListenAddress 10.201.0.1              # wg-mgmt address only
# NOT 0.0.0.0 — SSH is invisible on the physical interface

# Hardened algorithms
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
HostKeyAlgorithms ssh-ed25519

# Session limits
MaxAuthTries 3
MaxSessions 5
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2

# Disable unused features
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no                # Enable per-user if needed
PermitTunnel no

eBPF-based brute force detection

Traditional fail2ban parses log files with regex and adds iptables rules. It works. It is also slow, fragile, and reactive — the auth failure must be written to a log, the log must be read by fail2ban, the regex must match, and then a rule is added. With eBPF, you can detect authentication failures at the kernel level and react in microseconds:

# eBPF-based SSH brute force detection
# Attach to the audit subsystem — no log parsing required
bpftrace -e 'tracepoint:syscalls:sys_exit_accept4 /comm == "sshd"/ {
  @connections[ntop(((struct sockaddr_in *)args->upeer_sockaddr)->sin_addr.s_addr)] = count();
}
interval:s:10 {
  print(@connections);
  clear(@connections);
}'

# More than 5 connections from the same IP in 10 seconds = brute force
# Feed this into an nftables set for automatic blocking:

# Create a dynamic blocklist in nftables
nft add set inet filter ssh_blocklist '{ type ipv4_addr; timeout 1h; }'
nft add rule inet filter input ip saddr @ssh_blocklist drop

# Add offending IPs (from eBPF detection script)
nft add element inet filter ssh_blocklist '{ 192.168.1.100 }'

# The IP is automatically removed after 1 hour (timeout)

Jump host patterns

# ~/.ssh/config — access kldload hosts through WireGuard bastion
Host bastion
  HostName 203.0.113.10
  User admin
  IdentityFile ~/.ssh/kldload-mgmt
  Port 22

Host kldload-*
  ProxyJump bastion
  User admin
  IdentityFile ~/.ssh/kldload-mgmt
  StrictHostKeyChecking yes
  UserKnownHostsFile ~/.ssh/kldload_known_hosts

Host kldload-web
  HostName 10.201.0.10     # wg-mgmt address

Host kldload-db
  HostName 10.201.0.20     # wg-mgmt address

Host kldload-mon
  HostName 10.201.0.30     # wg-mgmt address

# Usage: ssh kldload-web
# Automatically jumps through bastion → WireGuard → target
# The target's SSH port is never exposed to the internet

SELinux and AppArmor — MAC across 8 distros

Mandatory Access Control (MAC) is not optional in production. The challenge is that Linux distributions do not agree on which MAC framework to use. kldload handles this per-distro:

# SELinux distros (enforcing by default)
CentOS Stream 9:  SELinux enforcing   ← RHEL-derived, targeted policy
RHEL 9:           SELinux enforcing   ← same targeted policy
Rocky Linux 9:    SELinux enforcing   ← same targeted policy
Fedora 41:        SELinux enforcing   ← same targeted policy, newer labels
Alpine Linux:     No MAC by default   ← minimal, add manually if needed

# AppArmor distros
Debian 13:        AppArmor enabled    ← default since Debian 10
Ubuntu 24.04:     AppArmor enabled    ← default since Ubuntu 7.10

# Neither by default
Arch Linux:       Neither             ← user's choice, AppArmor available

# Check status on any distro
getenforce                            # SELinux: Enforcing/Permissive/Disabled
aa-status                             # AppArmor: profiles loaded/enforced

ZFS and SELinux

ZFS is an out-of-tree filesystem. SELinux policies are written for ext4, XFS, and the standard filesystem hierarchy. ZFS datasets mounted at non-standard paths need SELinux context labels. kldload handles this during installation:

# Relabel ZFS mountpoints after installation (RHEL/CentOS/Rocky/Fedora)
restorecon -Rv /home /var /srv /opt

# Set SELinux context on ZFS datasets via xattr
zfs set xattr=sa rpool/data           # Store xattrs in system attributes (faster)

# Verify SELinux labels on ZFS-mounted directories
ls -lZ /home/
# drwx------. alice alice unconfined_u:object_r:user_home_dir_t:s0 alice
# drwx------. bob   bob   unconfined_u:object_r:user_home_dir_t:s0 bob

# If labels are wrong after ZFS mount:
restorecon -Rv /path/to/mountpoint

# For custom services on ZFS paths, write a policy module:
cat > kldload-zfs.te << 'EOF'
module kldload-zfs 1.0;
require { type httpd_t; type default_t; class file { read open getattr }; }
allow httpd_t default_t:file { read open getattr };
EOF
checkmodule -M -m -o kldload-zfs.mod kldload-zfs.te
semodule_package -o kldload-zfs.pp -m kldload-zfs.mod
semodule -i kldload-zfs.pp

AppArmor and ZFS

# AppArmor profiles reference paths. ZFS mounts at standard paths
# (/home, /var, /srv) work out of the box. Custom mount points
# need profile adjustments.

# Check if a service is confined
aa-status | grep nginx
# /usr/sbin/nginx (enforce)

# Allow nginx to read from a ZFS dataset mounted at /srv/www
cat >> /etc/apparmor.d/local/usr.sbin.nginx << 'EOF'
/srv/www/ r,
/srv/www/** r,
EOF
apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx

# Verify profile allows the path
aa-logprof    # Review and accept any remaining denials

Secure Boot chain — firmware to ZFS

The complete Secure Boot chain on a kldload system verifies every component from UEFI firmware to the ZFS kernel module. Each stage cryptographically verifies the next. A tampered component at any point breaks the chain and halts the boot process.

# The complete verification chain:

UEFI firmware (immutable, vendor-signed)
  │ verifies →
shim (signed by Microsoft UEFI CA)
  │ verifies →
ZFSBootMenu EFI binary (signed by Distro or MOK)
  │ verifies →
Linux kernel (signed by Distro — Red Hat, Canonical, etc.)
  │ verifies →
zfs.ko, spl.ko, etc. (signed by kldload build-time MOK)
  │ loads →
Root filesystem on ZFS (checksummed by ZFS on every read)

# Verify the chain manually:

# 1. Check Secure Boot status
mokutil --sb-state
# SecureBoot enabled

# 2. List trusted keys in UEFI
mokutil --list-enrolled
# [key 1] CN=kldload-20260404    ← MOK from build
# [key 2] CN=CentOS Stream 9     ← distro key

# 3. Verify shim is Microsoft-signed
pesign -S -i /boot/efi/EFI/centos/shimx64.efi
# signer: Microsoft Corporation UEFI CA 2011

# 4. Verify kernel is distro-signed
modinfo /boot/vmlinuz-$(uname -r) 2>/dev/null || \
  pesign -S -i /boot/vmlinuz-$(uname -r)
# signer: CentOS Stream 9 Secure Boot

# 5. Verify ZFS module is MOK-signed
modinfo zfs | grep -E "^sig|^signer"
# sig_id:    PKCS#7
# signer:    kldload-20260404
# sig_hashalgo: sha256

What happens when Secure Boot is enabled

With Secure Boot enabled and the MOK enrolled, the system operates in kernel lockdown mode. This restricts what even root can do:

# Kernel lockdown restrictions (when Secure Boot is active):

# Cannot load unsigned modules
modprobe unsigned_module
# ERROR: could not insert: Required key not available

# Cannot write to /dev/mem (no direct memory access)
dd if=/dev/urandom of=/dev/mem bs=1 count=1
# dd: failed to open '/dev/mem': Operation not permitted

# Cannot access CPU MSRs
rdmsr 0x1a0
# rdmsr: open: Operation not permitted

# Cannot use kexec with unsigned kernels
kexec -l /boot/vmlinuz-unsigned
# kexec: Cannot load unsigned kernel

# Cannot modify ACPI tables
# Cannot access ioperm/iopl
# Cannot write to kernel parameters via /sys

# Check lockdown status
cat /sys/kernel/security/lockdown
# integrity

Supply chain security — build once, deploy offline

Supply chain security is a build-time problem, not a deploy-time problem. If your installer downloads packages at deploy time, you are trusting the network, the mirror, the DNS resolver, and every hop in between. If any of those are compromised, every machine you deploy is compromised. kldload eliminates this by embedding complete package mirrors — called darksites — directly into the ISO.

# Build pipeline — packages are downloaded and verified ONCE

# 1. RPM darksite (CentOS, RHEL, Rocky, Fedora)
dnf download --resolve --alldeps --destdir=/root/darksite/rpms \
  @package-set-desktop.txt
# GPG verification happens here — dnf checks RPM signatures against
# the distro's GPG key. A tampered package fails verification.

# 2. Debian darksite (Debian 13 Trixie)
apt-get download $(cat package-set-debian.txt)
# APT verifies InRelease signatures against the Debian archive key.
# A tampered Packages file or .deb fails verification.

# 3. Ubuntu darksite (Ubuntu 24.04 Noble)
apt-get download $(cat package-set-ubuntu.txt)
# Same GPG verification against Ubuntu's archive key.

# 4. All packages are embedded into the ISO image
# The ISO is checksummed (SHA-256) at build time

# At deploy time — ZERO network access required
# The installer reads from the embedded darksite:
#   RPM:    file:///root/darksite/
#   Debian: http://localhost:3142/  (local APT mirror on the ISO)
#   Ubuntu: http://localhost:3143/  (local APT mirror on the ISO)

# Verify the chain:
sha256sum kldload-desktop-1.0.2.iso
# e3b0c442... kldload-desktop-1.0.2.iso
# This checksum must match what your build host produced.
# Any difference = tampered image. Do not deploy.

Air-gapped deployment

# The ISO is the deployment artifact. Write it to USB:
dd if=kldload-desktop-1.0.2.iso of=/dev/sda bs=4M oflag=direct status=progress

# Boot the target machine from USB.
# The installer runs entirely from RAM.
# No DHCP. No DNS. No internet. No phone-home.
# Every package comes from the embedded darksite.
# Every module is pre-compiled and pre-signed.

# After installation, the target machine has:
#   ✓ Signed kernel modules (ZFS, WireGuard)
#   ✓ Complete package set (no missing dependencies)
#   ✓ nftables firewall (default deny)
#   ✓ SSH hardened (key-only, WireGuard-only)
#   ✓ ZFS on root with checksumming active
#   ✓ Package holds on kernel/ZFS/bootloader

# The first time the machine needs network access is when YOU decide.
# Not when the installer decides. Not when a post-install script decides.

FIPS 140-3 — what is compliant and what is not

FIPS 140-3 is the US federal standard for cryptographic modules. If you sell to the US government (or work in healthcare, finance, or defense contracting), certain components must use FIPS-validated cryptographic implementations. Here is an honest assessment of where kldload stands.

# FIPS-COMPLIANT components:

ZFS encryption:        AES-256-GCM
                       → AES-256-GCM is a FIPS-approved algorithm
                       → BUT: OpenZFS's implementation is NOT CMVP-validated
                       → The algorithm is correct; the module lacks certification

SSH (OpenSSH):         AES-256-GCM, HMAC-SHA-256
                       → OpenSSH uses OpenSSL (FIPS-validated on RHEL 9)
                       → RHEL 9 ships a FIPS-validated OpenSSL module
                       → Run: update-crypto-policies --set FIPS

Disk encryption:       AES-256-XTS (LUKS, if used alongside ZFS)
                       → Kernel crypto module is FIPS-validated on RHEL 9

TLS:                   AES-256-GCM, ECDHE-P256/P384
                       → OpenSSL FIPS module on RHEL 9 is validated

# NOT FIPS-COMPLIANT components:

WireGuard:             ChaCha20-Poly1305
                       → ChaCha20 is NOT a FIPS-approved algorithm
                       → WireGuard has NO FIPS mode
                       → For FIPS compliance, use IPsec with AES-GCM instead

ZFS crypto module:     Not CMVP-validated
                       → OpenZFS uses AES-256-GCM (approved algorithm)
                       → But the implementation has not undergone CMVP testing
                       → For strict FIPS: use LUKS under ZFS (LUKS is validated)

eBPF:                  Not applicable
                       → eBPF is not a cryptographic module

Practical FIPS guidance

# Enable FIPS mode on RHEL 9 / CentOS Stream 9 / Rocky 9
fips-mode-setup --enable
reboot

# Verify FIPS mode is active
fips-mode-setup --check
# FIPS mode is enabled.

cat /proc/sys/crypto/fips_enabled
# 1

# FIPS mode enforces:
#   ✓ Only FIPS-approved algorithms in OpenSSL
#   ✓ Only FIPS-approved ciphers in SSH
#   ✓ Only FIPS-approved algorithms in the kernel crypto API
#   ✗ Does NOT affect WireGuard (ChaCha20 still used)
#   ✗ Does NOT affect ZFS native encryption (not kernel crypto API)

# For WireGuard replacement in FIPS environments:
# Use IPsec with AES-256-GCM via Libreswan
dnf install libreswan
# Configure IKEv2 with AES-256-GCM + ECDHE-P256
# This replaces WireGuard for the management/monitoring/storage planes

# For ZFS encryption in strict FIPS environments:
# Option 1: Use unencrypted ZFS on top of LUKS (FIPS-validated)
cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 /dev/nvme0n1p3
cryptsetup open /dev/nvme0n1p3 crypt-pool
zpool create rpool /dev/mapper/crypt-pool

# Option 2: Use ZFS native encryption and document the deviation
# AES-256-GCM is the correct algorithm; the module lacks CMVP paperwork
# Many auditors accept this with a risk acceptance memo

Compliance mapping — SOC 2, PCI-DSS, HIPAA

Compliance frameworks ask for specific controls. kldload provides the technical implementations. This is not a complete compliance program — that requires policies, procedures, and organizational controls beyond any single tool. But these are the technical controls that auditors ask for, and how kldload addresses each.

SOC 2 (Trust Service Criteria)

# CC6.1 — Logical access security
  SSH key-only auth, no root password login          → sshd_config
  WireGuard-only management access                   → nftables + wg-mgmt
  Per-dataset encryption with independent keys       → ZFS encryption

# CC6.6 — Encryption of data in transit
  WireGuard kernel-level encryption on all planes    → wg-mgmt, wg-mon, wg-stor
  TLS for web services                               → OpenSSL / Let's Encrypt

# CC6.7 — Encryption of data at rest
  ZFS AES-256-GCM per-dataset encryption             → zfs create -o encryption=on
  Raw send for encrypted replication                  → zfs send -w

# CC7.1 — Detection of unauthorized changes
  ZFS block-level checksumming                       → zpool scrub (weekly)
  eBPF file integrity monitoring                     → bpftrace on vfs_write
  Signed kernel modules (MOK)                        → Secure Boot + lockdown

# CC7.2 — Monitoring of system components
  eBPF process execution auditing                    → tracepoint:sys_enter_execve
  eBPF network connection monitoring                 → tracepoint:sys_enter_connect
  Prometheus + node_exporter on monitoring plane      → wg-mon:9100

# CC8.1 — Change management
  ZFS snapshots before every change                  → kupgrade, zfs snapshot
  Boot environments for atomic rollback              → ZFSBootMenu
  Package holds on critical components               → dnf versionlock / apt-mark hold

PCI-DSS v4.0

# Requirement 1 — Network security controls
  nftables default deny on all interfaces            → /etc/nftables.conf
  WireGuard plane isolation                          → wg_isolation table
  No services on physical interface except WireGuard → firewall + ListenAddress

# Requirement 2 — Secure configurations
  Hardened SSH (no passwords, no root, key-only)     → /etc/ssh/sshd_config
  SELinux enforcing (RHEL/CentOS/Rocky/Fedora)       → getenforce = Enforcing
  Unnecessary services disabled                      → systemctl list-unit-files

# Requirement 3 — Protect stored account data
  ZFS encryption on cardholder data datasets         → encryption=aes-256-gcm
  Per-dataset keys (limit decryption scope)          → zfs load-key per dataset
  Raw send to backup site (ciphertext only)          → zfs send -w

# Requirement 5 — Protect from malicious software
  Signed kernel modules prevent rootkits             → Secure Boot + MOK
  eBPF detects unauthorized process execution        → execve tracing
  Kernel lockdown prevents /dev/mem, kexec abuse     → lockdown=integrity

# Requirement 8 — Identify users and authenticate access
  SSH key-only authentication                        → PubkeyAuthentication yes
  No shared accounts                                 → individual keys per user
  Session recording via eBPF                         → execve + tty tracing

# Requirement 10 — Log and monitor all access
  eBPF audit trail (kernel-level, tamper-resistant)   → bpftrace programs
  ZFS snapshots preserve log state                   → immutable point-in-time
  Prometheus metrics with retention                  → wg-mon plane

# Requirement 11 — Test security regularly
  zpool scrub (weekly integrity verification)        → systemd timer
  eBPF continuous monitoring (real-time)             → always-on kernel probes
  Snapshot-based evidence preservation               → forensic holds

HIPAA (Technical Safeguards)

# § 164.312(a)(1) — Access control
  Per-dataset encryption (ePHI datasets locked independently)
  SSH key-only auth on WireGuard management plane
  nftables default deny

# § 164.312(a)(2)(iv) — Encryption and decryption
  ZFS AES-256-GCM on all ePHI datasets
  Raw send for encrypted replication to DR site
  WireGuard encryption for all data in transit

# § 164.312(b) — Audit controls
  eBPF process execution logging (who ran what, when)
  eBPF file access monitoring (who opened ePHI files)
  ZFS snapshots for point-in-time audit trail

# § 164.312(c)(1) — Integrity
  ZFS checksumming on every read (detects tampering)
  Signed kernel modules (prevents rootkit injection)
  Kernel lockdown (prevents memory/kernel modification)

# § 164.312(d) — Person or entity authentication
  SSH Ed25519 keys (one per person, no shared keys)
  WireGuard keys per machine (32-byte Curve25519)
  No password authentication anywhere

# § 164.312(e)(1) — Transmission security
  WireGuard on all inter-host communication
  No cleartext replication (zfs send -w or zfs send | ssh)
  TLS 1.3 for web interfaces

Forensic capabilities — snapshots, eBPF, and immutable audit

When a security incident occurs, the first priority is evidence preservation. The second priority is understanding what happened. The third is proving what happened to a third party (legal, regulatory, insurance). kldload provides kernel-level primitives for all three.

ZFS snapshots for evidence preservation

# Incident detected — immediately snapshot everything
zfs snapshot -r rpool@incident-2026-04-04-1423

# The -r flag snapshots recursively: every dataset, every child
# This is atomic — all datasets are snapshotted at the same txg
# Time to complete: milliseconds, regardless of data size

# The snapshot is immutable. It cannot be modified, even by root.
# An attacker who has root CANNOT delete a snapshot hold:

zfs hold forensic rpool@incident-2026-04-04-1423
zfs hold forensic rpool/home@incident-2026-04-04-1423
zfs hold forensic rpool/var/log@incident-2026-04-04-1423
# Held snapshots cannot be destroyed until the hold is released

# Attempt to destroy a held snapshot:
zfs destroy rpool@incident-2026-04-04-1423
# cannot destroy 'rpool@incident-2026-04-04-1423': dataset is busy

# Send the forensic snapshot to an evidence preservation server
zfs send -R rpool@incident-2026-04-04-1423 | \
  ssh evidence-server zfs recv -F tank/evidence/host1

# The evidence server now has a byte-identical copy of every dataset
# at the exact moment of detection. Compare pre-incident and post:

zfs diff rpool@daily-2026-04-03 rpool@incident-2026-04-04-1423
# M  /usr/bin/sshd            ← modified binary (backdoor?)
# +  /tmp/.hidden/payload      ← new file (malware?)
# M  /etc/pam.d/sshd          ← modified PAM config (persistence?)

eBPF for real-time forensic capture

# Capture ALL process executions during incident response
bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
  time("%H:%M:%S ");
  printf("pid=%d ppid=%d uid=%d comm=%s exe=%s\n",
    pid, (curtask->real_parent->pid), uid, comm,
    str(args->filename));
}' > /evidence/execve-log-$(date +%s).txt &

# Capture all network connections
bpftrace -e 'tracepoint:syscalls:sys_enter_connect {
  $sa = (struct sockaddr_in *)args->uservaddr;
  if ($sa->sin_family == 2) {
    time("%H:%M:%S ");
    printf("pid=%d comm=%s → %s:%d\n",
      pid, comm, ntop($sa->sin_addr.s_addr),
      ntohs($sa->sin_port));
  }
}' > /evidence/connect-log-$(date +%s).txt &

# Capture all file opens (who accessed what)
bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
  time("%H:%M:%S ");
  printf("pid=%d uid=%d comm=%s file=%s flags=%d\n",
    pid, uid, comm, str(args->filename), args->flags);
}' > /evidence/open-log-$(date +%s).txt &

# These logs are kernel-level. A userland rootkit cannot evade them.
# A kernel rootkit would need to be loaded as a signed module (blocked
# by Secure Boot) or modify the running kernel (blocked by lockdown).

Immutable audit logs with ZFS

# Create a dedicated dataset for audit logs
zfs create -o compression=zstd -o atime=off rpool/audit

# Configure rsyslog to write to the audit dataset
cat >> /etc/rsyslog.d/50-audit.conf << 'EOF'
auth,authpriv.*    /rpool/audit/auth.log
*.info;mail.none   /rpool/audit/syslog
kern.*             /rpool/audit/kern.log
EOF
systemctl restart rsyslog

# Snapshot audit logs every hour (immutable point-in-time)
cat > /etc/systemd/system/audit-snapshot.timer << 'EOF'
[Unit]
Description=Hourly audit log snapshot

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target
EOF

cat > /etc/systemd/system/audit-snapshot.service << 'EOF'
[Unit]
Description=Snapshot audit logs

[Service]
Type=oneshot
ExecStart=/sbin/zfs snapshot rpool/audit@auto-%Y%m%d-%H%M
ExecStartPost=/sbin/zfs hold audit rpool/audit@auto-%Y%m%d-%H%M
EOF

systemctl enable audit-snapshot.timer

# Result: hourly immutable snapshots of all audit logs
# An attacker who gains root can modify the CURRENT log file
# They CANNOT modify the last hourly snapshot (held + immutable)
# They CANNOT delete the snapshot (hold prevents destruction)
# The forensic trail is preserved at hourly granularity

# List all audit snapshots
zfs list -t snapshot -r rpool/audit
# rpool/audit@auto-20260404-1300    42K
# rpool/audit@auto-20260404-1400    38K
# rpool/audit@auto-20260404-1500    41K

# Compare what changed between snapshots
zfs diff rpool/audit@auto-20260404-1400 rpool/audit@auto-20260404-1500
# M  /rpool/audit/auth.log     ← normal: new log entries
# M  /rpool/audit/syslog       ← normal: new log entries

Replicate audit logs off-host in real time

# Send audit snapshots to a hardened log aggregator via storage plane
# Run every hour after the snapshot is created

cat > /usr/local/bin/replicate-audit.sh << 'SCRIPT'
#!/bin/bash
LATEST=$(zfs list -t snapshot -o name -s creation -r rpool/audit | tail -1)
PREV=$(zfs list -t snapshot -o name -s creation -r rpool/audit | tail -2 | head -1)

# Incremental send — only the delta since last snapshot
zfs send -w -i "$PREV" "$LATEST" | \
  ssh -o ConnectTimeout=10 -i /root/.ssh/audit-replication \
  log-server zfs recv tank/audit/$(hostname)
SCRIPT
chmod +x /usr/local/bin/replicate-audit.sh

# The log server receives encrypted audit data via the storage plane
# Even if the source host is completely compromised and wiped,
# the audit trail exists on the log server, with hourly granularity,
# held and immutable.

Ephemeral live session

# What kpoof does before shutdown:
shred -u /root/.ssh/*               # Destroy SSH keys
shred -u /etc/wireguard/*.conf      # Destroy WireGuard configs
shred -u /tmp/*                     # Destroy temp files
echo 3 > /proc/sys/vm/drop_caches  # Drop page cache, dentries, inodes
sync                                 # Flush any remaining buffers

# The live ISO never touches the target disk until you explicitly
# run the installer. Booting the ISO is non-destructive.

Package holds — the three most dangerous packages are locked

# RPM-based distros (CentOS, RHEL, Rocky, Fedora)
dnf versionlock list
# kernel-*
# zfs-*
# zfs-kmod-*
# grub2-*

# Attempting to update held packages:
dnf update kernel
# Last metadata expiration check: ...
# Dependencies resolved. Nothing to do.

# Deliberate upgrade with safety net:
kupgrade
# 1. zfs snapshot -r rpool@pre-upgrade-20260404
# 2. dnf versionlock delete kernel-* zfs-* grub2-*
# 3. dnf update kernel zfs zfs-kmod grub2-efi-x64
# 4. Recompile ZFS against new kernel headers
# 5. Re-sign modules with MOK key
# 6. dnf versionlock add kernel-* zfs-* grub2-*
# 7. Reboot into new kernel
# 8. If boot fails → ZFSBootMenu → select pre-upgrade snapshot

# APT-based distros (Debian, Ubuntu)
apt-mark showhold
# linux-image-amd64
# zfs-dkms
# zfsutils-linux
# grub-efi-amd64-signed

Auditable — every line of code is readable

# Read the entire installer — it is bash
wc -l /usr/sbin/kldload-install-target
# ~2000 lines of bash

# Read any kldload tool
cat $(which kupgrade)
cat $(which kstatus)
cat $(which kpoof)

# Read the web UI server — it is one Python file
wc -l /usr/local/bin/kldload-webui
# ~400 lines of Python

# Read the installer libraries
ls /usr/lib/kldload-installer/lib/
# bootloader.sh  bootstrap.sh  darksite.sh  disk.sh
# encryption.sh  networking.sh  packages.sh  profiles.sh  zfs.sh

# Every function, every variable, every decision is readable.
# No compiled binaries. No Go executors. No Rust daemons.
# If you do not trust it, read it. That is the point.

Current status — what is done, what is not

Done:
  Secure Boot with signed modules (per-build MOK)
  ZFS block-level checksumming and self-healing
  Per-dataset AES-256-GCM encryption
  WireGuard kernel-level encryption (4-plane backplane)
  eBPF security monitoring (process, file, network, container)
  nftables default deny firewall
  SSH hardening (key-only, WireGuard-only)
  SELinux enforcing on RHEL-derived distros
  AppArmor on Debian/Ubuntu
  Air-gap / offline deployment with GPG-verified darksites
  Ephemeral live session with RAM scrubbing (kpoof)
  Package holds on critical boot components
  100% auditable bash/Python source
  ZFS forensic snapshots with holds
  Immutable audit log snapshots

Not yet:
  TPM-2 measured boot (planned)
  Remote attestation before cluster admission (planned)
  Kubernetes admission webhook verifying boot integrity (planned)
  Automated MOK enrollment (currently manual on first boot)
  FIPS-validated ZFS encryption module (requires CMVP submission)
  Hardware security key integration for ZFS key management (planned)