System logs via journald

systemd-journald captures every log message from every service, the kernel ring buffer, and stdout/stderr of every systemd unit. Logs are structured (key=value pairs), indexed, and queryable. On kldload, journal storage lives on ZFS — compressed, checksummed, and snapshotted automatically by sanoid.

# View all logs since last boot
journalctl -b

# Follow logs in real time
journalctl -f

# Logs from a specific service
journalctl -u sshd --since "1 hour ago"

# Logs from a specific PID
journalctl _PID=1234

# Kernel messages only
journalctl -k

# Export structured JSON for external processing
journalctl -o json --since "2024-01-01" | head -100

# Disk usage of journal storage
journalctl --disk-usage

ZFS events via ZED

The ZFS Event Daemon watches every pool event: scrub completions, checksum errors, device failures, resilver progress, import/export operations. On kldload, ZED is configured to log to journald and optionally send email alerts.

# View all ZFS events
journalctl -t zed

# Check ZED configuration
cat /etc/zfs/zed.d/zed.rc

# List enabled ZED scripts
ls -la /etc/zfs/zed.d/

# Recent pool events
zpool events -v | head -50

# Checksum error history (critical for audit)
zpool events | grep -i checksum

# Scrub history
zpool history rpool | grep scrub

WireGuard connection logs

Every WireGuard peer connection is logged via the kernel's dynamic debug facility. The handshake timestamp, peer public key, and endpoint IP are recorded. On kldload systems with WireGuard enabled, this provides a complete record of every peer that connected, when, and from where.

# Enable WireGuard debug logging
echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control

# View WireGuard handshake events
journalctl -k | grep wireguard

# Show current peer status (latest handshake times)
wg show all

# Show specific interface peer details
wg show wg0 latest-handshakes

# Parse handshake timestamps for audit report
wg show wg0 latest-handshakes | while read key ts; do
  echo "$key  $(date -d @$ts '+%Y-%m-%d %H:%M:%S')"
done

SSH authentication logs

Every SSH login attempt — successful or failed — is logged to journald with the username, source IP, authentication method, and key fingerprint. kldload disables root SSH login by default, so every legitimate session maps to a named user.

# All SSH authentication events
journalctl -u sshd | grep -E "(Accepted|Failed|Invalid)"

# Failed login attempts (brute force detection)
journalctl -u sshd | grep "Failed password" | awk '{print $11}' | sort | uniq -c | sort -rn

# Successful logins with key fingerprints
journalctl -u sshd | grep "Accepted publickey"

# Active SSH sessions right now
ss -tnp | grep :22
who -u

Process execution logging

Track every process that runs on the system. Every exec() call is captured with the full command line, parent PID, user ID, and timestamp. This is more comprehensive than auditd because eBPF sees the exec before the process starts.

# Real-time process execution trace (BCC)
execsnoop-bpfcc

# Output includes: PID, PPID, COMM, RET, ARGS
# Example output:
# PID    PPID   COMM       RET ARGS
# 18234  18220  curl         0 curl https://example.com
# 18235  1      crond        0 /usr/sbin/crond -n

# Log to file for audit retention
execsnoop-bpfcc -T > /var/log/audit/exec-$(date +%Y%m%d).log &

# With bpftrace: custom exec tracer with UID
bpftrace -e 'tracepoint:syscalls:sys_enter_execve {
  printf("%s uid=%d pid=%d ppid=%d %s\n",
    strftime("%H:%M:%S", nsecs),
    uid, pid, curtask->parent->pid,
    str(args->filename));
}'

# Filter for specific users (e.g., uid 1000)
bpftrace -e 'tracepoint:syscalls:sys_enter_execve /uid == 1000/ {
  printf("%s %s %s\n", strftime("%H:%M:%S", nsecs), comm, str(args->filename));
}'

File integrity monitoring

Watch specific files or directories for any open, read, write, or unlink operation. This is real-time file integrity monitoring at the kernel level — no polling, no checksumming delay. Every file access is captured the instant it happens.

# Watch all file opens in /etc (configuration tampering detection)
opensnoop-bpfcc -p 0 | grep /etc/

# Watch writes to critical files
bpftrace -e 'tracepoint:syscalls:sys_enter_openat
  /str(args->filename) == "/etc/passwd"/ {
  printf("%s uid=%d pid=%d comm=%s OPEN /etc/passwd\n",
    strftime("%H:%M:%S", nsecs), uid, pid, comm);
}'

# Watch file deletions system-wide
bpftrace -e 'tracepoint:syscalls:sys_enter_unlinkat {
  printf("%s uid=%d pid=%d comm=%s DELETE %s\n",
    strftime("%H:%M:%S", nsecs), uid, pid, comm, str(args->pathname));
}'

# Monitor /etc/shadow access (password file)
bpftrace -e 'tracepoint:syscalls:sys_enter_openat
  /str(args->filename) == "/etc/shadow"/ {
  printf("ALERT: %s (pid=%d uid=%d) accessed /etc/shadow at %s\n",
    comm, pid, uid, strftime("%H:%M:%S", nsecs));
}'

# Watch all writes to /var/log (log tampering detection)
opensnoop-bpfcc | grep -E "W.*/var/log/"

Network connection tracking

Every TCP connection established on the system — inbound and outbound — is logged with source IP, destination IP, port, PID, and process name. This creates a complete network audit trail at the kernel level.

# All new TCP connections (outbound)
tcpconnect-bpfcc

# Output: PID, COMM, IP, SADDR, DADDR, DPORT
# Example:
# PID    COMM         IP SADDR         DADDR          DPORT
# 4523   curl         4  10.0.0.5      93.184.216.34  443
# 4531   sshd         4  10.0.0.5      10.0.0.10      22

# All new TCP connections (inbound)
tcpaccept-bpfcc

# Combined: all TCP state changes with duration
tcplife-bpfcc
# PID   COMM    LADDR        LPORT RADDR        RPORT TX_KB RX_KB MS
# 4523  curl    10.0.0.5     44820 93.184.216   443   1     12    234

# DNS queries (what hostnames is the system resolving?)
bpftrace -e 'tracepoint:net:net_dev_xmit /args->len > 0/ {
  printf("%s pid=%d comm=%s len=%d\n",
    strftime("%H:%M:%S", nsecs), pid, comm, args->len);
}'

# Log all outbound connections to file for audit
tcpconnect-bpfcc -T > /var/log/audit/netconn-$(date +%Y%m%d).log &

Privilege escalation detection

Detect any process that changes its effective UID (sudo, su, setuid binaries, exploits). Every privilege change is visible to eBPF because the kernel itself performs the credential swap.

# Watch setuid/setgid syscalls
bpftrace -e 'tracepoint:syscalls:sys_enter_setuid {
  printf("PRIVESC: %s pid=%d uid=%d -> target_uid=%d\n",
    comm, pid, uid, args->uid);
}'

# Watch all capability grants
bpftrace -e 'kprobe:cap_capable {
  printf("%s pid=%d uid=%d comm=%s cap=%d\n",
    strftime("%H:%M:%S", nsecs), pid, uid, comm, arg2);
}'

# Detect sudo usage
execsnoop-bpfcc | grep -E "^.*sudo"

# Watch for kernel module loading (rootkit detection)
bpftrace -e 'tracepoint:syscalls:sys_enter_finit_module {
  printf("ALERT: module load by pid=%d uid=%d comm=%s\n", pid, uid, comm);
}'

# Watch for ptrace (debugger attach, often used in exploits)
bpftrace -e 'tracepoint:syscalls:sys_enter_ptrace {
  printf("ALERT: ptrace by pid=%d uid=%d comm=%s request=%d\n",
    pid, uid, comm, args->request);
}'

Snapshots as point-in-time evidence

A ZFS snapshot captures the exact state of a dataset at a specific moment. It is read-only. It cannot be modified. If an attacker modifies a file, the original version still exists in the snapshot. If an attacker deletes a log, the snapshot still has it.

# Create an evidence snapshot (millisecond precision)
zfs snapshot rpool/ROOT/cs@evidence-$(date +%Y%m%d-%H%M%S)

# Create recursive snapshot of entire system
zfs snapshot -r rpool@incident-$(date +%Y%m%d-%H%M%S)

# List all snapshots with creation times
zfs list -t snapshot -o name,creation,used,refer -s creation

# Access snapshot contents (read-only, no mount needed)
ls /rpool/ROOT/cs/.zfs/snapshot/evidence-20240115-143022/etc/passwd
cat /rpool/ROOT/cs/.zfs/snapshot/evidence-20240115-143022/var/log/secure

# Compare current state to snapshot (what changed?)
diff /etc/passwd /rpool/ROOT/cs/.zfs/snapshot/evidence-20240115-143022/etc/passwd

# Automated hourly evidence snapshots via sanoid
# sanoid runs by default on kldload and creates:
#   - hourly snapshots (kept for 48 hours)
#   - daily snapshots (kept for 30 days)
#   - monthly snapshots (kept for 6 months)
sanoid --cron

zfs diff — exactly what changed

zfs diff compares two snapshots (or a snapshot and the live filesystem) and lists every file that was added, modified, deleted, or renamed. This is the single most powerful audit command on any filesystem.

# What changed since this morning's snapshot?
zfs diff rpool/ROOT/cs@autosnap_2024-01-15_06:00:00_hourly

# Output format:
# M  /etc/passwd                    (modified)
# +  /tmp/suspicious-binary         (added)
# -  /var/log/secure                (deleted)
# R  /etc/shadow -> /etc/shadow.bak (renamed)

# Compare two specific snapshots
zfs diff rpool/ROOT/cs@before-change rpool/ROOT/cs@after-change

# Find all changes in the last 24 hours (compare oldest hourly to now)
OLDEST=$(zfs list -t snapshot -o name -s creation rpool/ROOT/cs | grep hourly | head -1)
zfs diff "$OLDEST"

# Pipe to file for evidence
zfs diff rpool/ROOT/cs@evidence-20240115-143022 > /root/incident-changes.txt

# Show only deleted files (potential evidence destruction)
zfs diff rpool/ROOT/cs@autosnap_2024-01-15_06:00:00_hourly | grep "^-"

zfs send/receive for evidence preservation

Preserve the exact state of a system off-site. zfs send creates a byte-for-byte stream of a snapshot that can be received on any other ZFS system. The evidence is cryptographically identical to the source.

# Send evidence snapshot to a remote evidence server
zfs send rpool/ROOT/cs@evidence-20240115-143022 | \
  ssh evidence-server zfs receive evidence-pool/case-2024-0115

# Send encrypted (raw) for chain of custody
zfs send --raw rpool/ROOT/cs@evidence-20240115-143022 | \
  ssh evidence-server zfs receive evidence-pool/case-2024-0115

# Send incremental (only changes between two snapshots)
zfs send -i rpool/ROOT/cs@before rpool/ROOT/cs@after | \
  ssh evidence-server zfs receive evidence-pool/case-delta

# Verify the received snapshot matches the source
zfs get -o value written rpool/ROOT/cs@evidence-20240115-143022
ssh evidence-server zfs get -o value written evidence-pool/case-2024-0115

# Save to file for offline evidence storage
zfs send rpool/ROOT/cs@evidence-20240115-143022 | \
  gzip > /mnt/usb/evidence-case-2024-0115.zfs.gz

Immutable snapshots with holds

A ZFS hold prevents a snapshot from being destroyed, even by root. This is critical for evidence preservation — you can take a snapshot, place a hold on it, and nobody can delete it until the hold is explicitly released.

# Place a hold on an evidence snapshot (nobody can delete it now)
zfs hold legal-hold rpool/ROOT/cs@evidence-20240115-143022

# Attempt to destroy it (this FAILS)
zfs destroy rpool/ROOT/cs@evidence-20240115-143022
# cannot destroy 'rpool/ROOT/cs@evidence-20240115-143022': dataset is busy

# List all holds
zfs holds rpool/ROOT/cs@evidence-20240115-143022
# NAME                                          TAG          TIMESTAMP
# rpool/ROOT/cs@evidence-20240115-143022        legal-hold   Mon Jan 15 14:30 2024

# List all held snapshots across the pool
zfs list -t snapshot -o name,defer_destroy | grep -v "-$"

# Release a hold (requires explicit action)
zfs release legal-hold rpool/ROOT/cs@evidence-20240115-143022

# Place holds on all snapshots from an incident
for snap in $(zfs list -t snapshot -o name | grep "incident-20240115"); do
  zfs hold forensic-hold "$snap"
done

Install and enable auditd

# CentOS / Rocky / RHEL / Fedora (usually pre-installed)
dnf install -y audit
systemctl enable --now auditd

# Debian / Ubuntu
apt install -y auditd audispd-plugins
systemctl enable --now auditd

# Arch
pacman -S audit
systemctl enable --now auditd

# Verify it is running
auditctl -s
# enabled 1
# pid 1234
# ...

# Check audit log location
ls -la /var/log/audit/audit.log

Audit rules for critical files

Watch specific files and directories for read, write, execute, and attribute changes. These rules generate audit events every time the specified file is accessed.

# Add rules to /etc/audit/rules.d/kldload.rules

# Watch password and authentication files
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers

# Watch SSH configuration
-w /etc/ssh/sshd_config -p wa -k sshd_config
-w /etc/ssh/ssh_config -p wa -k ssh_config
-w /root/.ssh/ -p wa -k root_ssh

# Watch system startup scripts
-w /etc/systemd/ -p wa -k systemd_config
-w /etc/cron.d/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -p wa -k cron

# Watch kernel module loading
-w /sbin/insmod -p x -k kernel_modules
-w /sbin/modprobe -p x -k kernel_modules
-w /sbin/rmmod -p x -k kernel_modules

# Watch ZFS configuration
-w /etc/zfs/ -p wa -k zfs_config
-w /etc/modprobe.d/zfs.conf -p wa -k zfs_config

# Watch WireGuard configuration
-w /etc/wireguard/ -p wa -k wireguard_config

# Load the rules
augenrules --load
auditctl -l   # verify rules are active

Audit rules for syscalls

Monitor specific syscalls across the entire system. These rules catch actions that file watches alone cannot — like mounting filesystems, changing the system time, or modifying network configuration.

# Add to /etc/audit/rules.d/kldload-syscalls.rules

# Log all execve calls by root (every command root runs)
-a always,exit -F arch=b64 -S execve -F euid=0 -k root_commands

# Log mount/unmount operations
-a always,exit -F arch=b64 -S mount -S umount2 -k mounts

# Log time changes (critical for log integrity)
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -S clock_settime -k time_change

# Log network configuration changes
-a always,exit -F arch=b64 -S sethostname -S setdomainname -k network_config

# Log user/group modification
-a always,exit -F arch=b64 -S setuid -S setgid -S setreuid -S setregid -k priv_escalation

# Log failed file access attempts (permission denied)
-a always,exit -F arch=b64 -S open -S openat -F exit=-EACCES -k access_denied
-a always,exit -F arch=b64 -S open -S openat -F exit=-EPERM -k access_denied

# Log kernel module operations
-a always,exit -F arch=b64 -S init_module -S finit_module -S delete_module -k kernel_modules

# Make the audit configuration immutable until reboot
-e 2

Searching and reporting with ausearch and aureport

# Search for all events related to /etc/passwd
ausearch -k identity

# Search for events by specific user
ausearch -ua 1000

# Search for failed login attempts
ausearch -m USER_LOGIN --success no

# Search by time range
ausearch --start 01/15/2024 08:00:00 --end 01/15/2024 17:00:00

# Search for root command execution
ausearch -k root_commands -ts today

# Generate summary report
aureport --summary

# Authentication report (logins, failures)
aureport --auth

# File access report
aureport --file --summary

# Executable report (what binaries ran)
aureport --executable --summary

# Failed events report
aureport --failed

# User activity report
aureport --user

# Generate report for specific time period
aureport --start 01/01/2024 --end 01/31/2024 --auth --summary

FedRAMP audit requirements

FedRAMP inherits NIST 800-53 audit controls. kldload maps to the following control families:

AU-2 (Audit Events) — auditd rules cover authentication, privilege use, file access, system changes. eBPF adds kernel-level process and network visibility.

AU-3 (Content of Audit Records) — journald and auditd capture timestamp, source, user, action, outcome, and object for every event.

AU-4 (Audit Storage Capacity) — ZFS compression reduces log storage by 3-10x. Dynamic pool expansion adds capacity without downtime.

AU-5 (Response to Audit Failures) — auditd -f 2 halts the system if audit cannot write. ZED alerts on pool errors.

AU-6 (Audit Review) — ausearch, aureport, journalctl, Grafana dashboards for systematic review.

AU-9 (Protection of Audit Information) — ZFS snapshots with holds. Checksummed storage. Off-site replication via zfs send.

AU-11 (Audit Record Retention) — ZFS snapshots retained per sanoid policy. Compressed storage makes 12+ month retention practical.

Promtail — ship logs from every node

Promtail reads journald and ships logs to Loki over HTTP (or WireGuard-encrypted HTTP). Install on every kldload node.

# /etc/promtail/config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki.internal:3100/loki/api/v1/push

scrape_configs:
  # Scrape systemd journal
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
        host: ${HOSTNAME}
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'
      - source_labels: ['__journal_priority_keyword']
        target_label: 'level'

  # Scrape audit logs
  - job_name: audit
    static_configs:
      - targets: [localhost]
        labels:
          job: auditd
          host: ${HOSTNAME}
          __path__: /var/log/audit/audit.log

  # Scrape eBPF audit logs
  - job_name: ebpf-audit
    static_configs:
      - targets: [localhost]
        labels:
          job: ebpf
          host: ${HOSTNAME}
          __path__: /var/log/audit/exec-*.log
          __path__: /var/log/audit/netconn-*.log

Loki — log storage on ZFS

Loki stores log chunks and indexes. On kldload, put Loki's data directory on a dedicated ZFS dataset with compression enabled — logs compress 5-10x with LZ4.

# Create a dedicated ZFS dataset for Loki
zfs create -o compression=lz4 -o recordsize=128k rpool/loki
zfs create rpool/loki/chunks
zfs create rpool/loki/index

# /etc/loki/config.yml (relevant storage section)
storage_config:
  boltdb_shipper:
    active_index_directory: /rpool/loki/index/active
    cache_location: /rpool/loki/index/cache
    shared_store: filesystem
  filesystem:
    directory: /rpool/loki/chunks

# Retention: keep 90 days of logs
limits_config:
  retention_period: 2160h

# Start Loki
systemctl enable --now loki

# Verify Loki is receiving logs
curl -s http://localhost:3100/ready
curl -s http://localhost:3100/metrics | grep loki_ingester_chunks_stored_total

Grafana — query and alert

Grafana connects to Loki and provides log search, dashboards, and alerting. Query all audit events across every node in the fleet from one interface.

# Grafana Loki queries (LogQL)

# All SSH failures across the fleet
{job="systemd-journal"} |= "Failed password"

# All sudo commands by user "deploy"
{job="systemd-journal", unit="sudo.service"} |= "deploy"

# All auditd events for /etc/passwd changes
{job="auditd"} |= "identity"

# eBPF exec events for curl (exfiltration detection)
{job="ebpf"} |= "curl"

# All events from a specific host
{host="web-prod-01"} | json | level="err"

# Rate of failed SSH logins (alerting)
rate({job="systemd-journal"} |= "Failed password" [5m])

Log retention on ZFS

ZFS makes long-term log retention practical. Logs compress 5-10x with LZ4. Snapshots are incremental (only new data costs space). A 1TB ZFS pool can store years of audit logs for a mid-size fleet.

# Check compression ratio on log storage
zfs get compressratio rpool/loki
# NAME         PROPERTY       VALUE  SOURCE
# rpool/loki   compressratio  6.82x  -

# Check actual space used vs. logical size
zfs list -o name,used,logicalused,compressratio rpool/loki

# Snapshot log storage for compliance retention
zfs snapshot rpool/loki@eom-$(date +%Y%m)

# Replicate log snapshots off-site
zfs send -i rpool/loki@eom-202312 rpool/loki@eom-202401 | \
  ssh log-archive zfs receive archive/loki

# Set quota to prevent log storage from consuming the pool
zfs set quota=500G rpool/loki

Git for configuration tracking

Store /etc in a local git repository. Every configuration change gets a commit with a timestamp, author, and message. git log and git diff show exactly who changed what and when.

# Initialize git tracking for /etc
cd /etc
git init
git add -A
git commit -m "baseline configuration after kldload install"

# After making any configuration change:
cd /etc
git add -A
git diff --cached   # review what changed
git commit -m "enable auditd, add audit rules for PCI-DSS"

# Who changed sshd_config and when?
git log --oneline -10 -- ssh/sshd_config

# What exactly changed in the last commit?
git show HEAD -- ssh/sshd_config

# Compare current config to the original baseline
git diff HEAD~10 -- ssh/sshd_config

# Blame: who wrote each line of the current config?
git blame ssh/sshd_config

ZFS snapshots for filesystem state

ZFS snapshots capture the entire filesystem state, not just tracked files. Sanoid creates automatic snapshots on schedule. Use zfs diff to see what changed between any two snapshots — including files git does not track.

# List recent automatic snapshots
zfs list -t snapshot -o name,creation -s creation rpool/ROOT/cs | tail -20

# Before a maintenance window: manual snapshot
zfs snapshot rpool/ROOT/cs@pre-maintenance-$(date +%Y%m%d)

# After maintenance: what changed?
zfs diff rpool/ROOT/cs@pre-maintenance-20240115

# Save the diff as a change record
zfs diff rpool/ROOT/cs@pre-maintenance-20240115 > \
  /root/change-records/maintenance-20240115.diff

# If maintenance went wrong: rollback
zfs rollback rpool/ROOT/cs@pre-maintenance-20240115

eBPF for real-time change tracking

eBPF sees changes as they happen. During a maintenance window, run eBPF monitors to capture a complete record of every command executed, every file written, every network connection made.

# Start a complete change audit session
mkdir -p /root/change-audit/$(date +%Y%m%d)
AUDIT_DIR="/root/change-audit/$(date +%Y%m%d)"

# Log every command executed
execsnoop-bpfcc -T > "$AUDIT_DIR/exec.log" &
EXEC_PID=$!

# Log every file opened for writing
opensnoop-bpfcc -T | grep -E "O_WRONLY|O_RDWR" > "$AUDIT_DIR/writes.log" &
WRITE_PID=$!

# Log every network connection
tcpconnect-bpfcc -T > "$AUDIT_DIR/netconn.log" &
NET_PID=$!

echo "Change audit running. PIDs: exec=$EXEC_PID writes=$WRITE_PID net=$NET_PID"

# ... perform maintenance ...

# Stop the audit
kill $EXEC_PID $WRITE_PID $NET_PID

# Review what happened
wc -l "$AUDIT_DIR"/*.log
head -50 "$AUDIT_DIR/exec.log"

RPM verification (CentOS, Rocky, Fedora, RHEL)

rpm -Va verifies every installed file against the RPM database. It checks size, checksum (SHA-256), permissions, ownership, modification time, and symlink targets. Any discrepancy is flagged.

# Verify ALL installed packages
rpm -Va 2>/dev/null

# Output format:
# S.5....T.  c /etc/ssh/sshd_config     (size, checksum, time changed — config file)
# ..5....T.    /usr/bin/suspicious        (checksum and time changed — ALERT)
#
# Flags: S=size, M=mode, 5=checksum, D=device, L=link, U=user, G=group, T=time
# "c" means config file (expected to change)

# Verify a specific package
rpm -V openssh-server

# Verify only non-config files (find actual tampering)
rpm -Va 2>/dev/null | grep -v "^.*c /"

# Verify that RPM database itself has not been tampered with
rpm -qa --qf '%{SIGPGP:pgpsig}\n' | grep -c "Key ID"

# List packages with failed GPG signatures
rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE} %{SIGPGP:pgpsig}\n' | grep -i "not"

# Reinstall a package that fails verification
dnf reinstall openssh-server

dpkg/debsums verification (Debian, Ubuntu)

debsums verifies installed file checksums against the package database. dpkg -V provides a similar check. Both catch modified binaries and libraries.

# Install debsums if not present
apt install -y debsums

# Verify ALL installed packages
debsums -c   # show only changed files

# Verify a specific package
debsums openssh-server

# Show all files that have been modified
debsums -ac 2>/dev/null

# Alternative: dpkg --verify (similar to rpm -Va)
dpkg -V

# Verify package signatures
apt-key list
apt-cache policy openssh-server

# Check that apt sources have valid signatures
apt update 2>&1 | grep -i "NO_PUBKEY\|EXPKEYSIG"

# Reinstall a package that fails verification
apt install --reinstall openssh-server

pacman verification (Arch)

# Verify all installed packages
pacman -Qkk 2>/dev/null | grep -v "0 altered"

# Verify a specific package
pacman -Qkk openssh

# Check package signatures
pacman -Qkq 2>/dev/null | sort

# List packages that were not installed from a repository (manual/AUR)
pacman -Qm

# Verify trust of keyring
pacman-key --verify

# Reinstall a package that fails verification
pacman -S openssh

File checksum baselines

For files that package verification does not cover (custom scripts, configuration managed outside packages), create a checksum baseline after install and verify against it periodically.

# Create a baseline of critical files after install
find /usr/sbin /usr/bin /usr/lib -type f -exec sha256sum {} \; \
  > /root/baseline-checksums-$(date +%Y%m%d).txt

# Store the baseline on a held ZFS snapshot
zfs snapshot rpool/ROOT/cs@baseline-$(date +%Y%m%d)
zfs hold baseline rpool/ROOT/cs@baseline-$(date +%Y%m%d)

# Verify against baseline (run periodically via cron)
sha256sum -c /root/baseline-checksums-20240115.txt 2>/dev/null | grep FAILED

# Compare current binaries to the baseline snapshot
diff <(sha256sum /usr/sbin/*) \
     <(cd /rpool/ROOT/cs/.zfs/snapshot/baseline-20240115 && sha256sum usr/sbin/*)

# Verify kernel modules are signed
for mod in $(lsmod | awk 'NR>1{print $1}'); do
  modinfo "$mod" 2>/dev/null | grep -q "sig_id" && echo "$mod: SIGNED" || echo "$mod: UNSIGNED"
done

Frozen snapshot, not a live mirror

Each darksite is a point-in-time snapshot of official upstream packages. During installation, the installer uses only the darksite — no internet mirror is contacted. The package versions installed are exactly the versions that were cached at build time.

After installation, the system's package manager is configured with standard internet repositories. The user runs apt update && apt upgrade, dnf update, or pacman -Syu to get current at their own pace.

Daily audit summary

Generate a daily audit summary that covers authentication events, system changes, and security alerts. Run via cron or systemd timer.

#!/bin/bash
# /usr/local/bin/daily-audit-report.sh
REPORT="/var/log/audit/daily-report-$(date +%Y%m%d).txt"

echo "=== DAILY AUDIT REPORT $(date) ===" > "$REPORT"
echo "" >> "$REPORT"

echo "--- Authentication Summary ---" >> "$REPORT"
aureport --auth --summary >> "$REPORT" 2>/dev/null
echo "" >> "$REPORT"

echo "--- Failed Logins ---" >> "$REPORT"
aureport --auth --failed >> "$REPORT" 2>/dev/null
echo "" >> "$REPORT"

echo "--- Privilege Escalation Events ---" >> "$REPORT"
ausearch -k priv_escalation -ts today --format text >> "$REPORT" 2>/dev/null
echo "" >> "$REPORT"

echo "--- File Integrity Changes ---" >> "$REPORT"
ausearch -k identity -ts today --format text >> "$REPORT" 2>/dev/null
echo "" >> "$REPORT"

echo "--- ZFS Pool Status ---" >> "$REPORT"
zpool status >> "$REPORT"
echo "" >> "$REPORT"

echo "--- ZFS Snapshot Count ---" >> "$REPORT"
zfs list -t snapshot -o name | wc -l >> "$REPORT"
echo "" >> "$REPORT"

echo "--- Filesystem Changes (last 24h) ---" >> "$REPORT"
LATEST_SNAP=$(zfs list -t snapshot -o name -s creation rpool/ROOT/cs | \
  grep hourly | tail -24 | head -1)
if [ -n "$LATEST_SNAP" ]; then
  zfs diff "$LATEST_SNAP" 2>/dev/null | wc -l >> "$REPORT"
  echo "files changed since $LATEST_SNAP" >> "$REPORT"
fi

echo "" >> "$REPORT"
echo "--- Package Verification ---" >> "$REPORT"
rpm -Va 2>/dev/null | grep -v "^.*c /" | wc -l >> "$REPORT"
echo "non-config files with verification failures" >> "$REPORT"

echo "Report saved to $REPORT"

Compliance posture check

Verify that audit controls are properly configured. Run before an audit engagement to identify gaps.

#!/bin/bash
# /usr/local/bin/compliance-check.sh
echo "=== COMPLIANCE POSTURE CHECK ==="

# Check auditd is running
echo -n "[1] auditd running: "
systemctl is-active auditd 2>/dev/null && echo "PASS" || echo "FAIL"

# Check audit rules are loaded
echo -n "[2] audit rules loaded: "
RULES=$(auditctl -l 2>/dev/null | wc -l)
[ "$RULES" -gt 5 ] && echo "PASS ($RULES rules)" || echo "FAIL ($RULES rules)"

# Check audit log exists and is being written
echo -n "[3] audit log active: "
[ -f /var/log/audit/audit.log ] && echo "PASS ($(wc -l < /var/log/audit/audit.log) lines)" || echo "FAIL"

# Check ZFS pool health
echo -n "[4] ZFS pool healthy: "
zpool status -x 2>/dev/null | grep -q "all pools are healthy" && echo "PASS" || echo "FAIL"

# Check ZFS snapshots exist
echo -n "[5] ZFS snapshots exist: "
SNAPS=$(zfs list -t snapshot -o name 2>/dev/null | wc -l)
[ "$SNAPS" -gt 1 ] && echo "PASS ($SNAPS snapshots)" || echo "FAIL"

# Check sanoid is running (automatic snapshots)
echo -n "[6] sanoid timer active: "
systemctl is-active sanoid.timer 2>/dev/null && echo "PASS" || echo "FAIL"

# Check SSH config
echo -n "[7] SSH root login disabled: "
grep -q "^PermitRootLogin no" /etc/ssh/sshd_config 2>/dev/null && echo "PASS" || echo "FAIL"

echo -n "[8] SSH password auth disabled: "
grep -q "^PasswordAuthentication no" /etc/ssh/sshd_config 2>/dev/null && echo "PASS" || echo "WARN (enabled)"

# Check time sync
echo -n "[9] time synchronization: "
timedatectl show -p NTPSynchronized --value 2>/dev/null | grep -q "yes" && echo "PASS" || echo "FAIL"

# Check firewall
echo -n "[10] nftables active: "
systemctl is-active nftables 2>/dev/null && echo "PASS" || echo "FAIL"

# Check journal persistence
echo -n "[11] persistent journal: "
[ -d /var/log/journal ] && echo "PASS" || echo "FAIL"

# Check WireGuard (if configured)
echo -n "[12] WireGuard interfaces: "
WG=$(wg show interfaces 2>/dev/null | wc -w)
echo "$WG interfaces"

echo ""
echo "=== END POSTURE CHECK ==="

Security posture assessment

A deeper assessment that checks kernel security parameters, file permissions, and system hardening. Useful for security reviews and penetration test preparation.

# Kernel security parameters
sysctl kernel.randomize_va_space      # should be 2 (full ASLR)
sysctl kernel.dmesg_restrict           # should be 1 (non-root cannot read dmesg)
sysctl kernel.kptr_restrict            # should be 1 or 2
sysctl net.ipv4.conf.all.rp_filter     # should be 1 (reverse path filtering)
sysctl net.ipv4.tcp_syncookies         # should be 1 (SYN flood protection)

# SUID binaries (potential privilege escalation vectors)
find / -perm -4000 -type f 2>/dev/null | sort

# World-writable files (excluding /tmp and /proc)
find / -xdev -perm -o+w -type f 2>/dev/null | grep -v -E "^/(tmp|proc|sys|dev)"

# Unowned files
find / -xdev -nouser -o -nogroup 2>/dev/null

# Listening services
ss -tlnp

# Loaded kernel modules
lsmod | wc -l
echo "kernel modules loaded"

# Check for unauthorized cron jobs
for user in $(cut -d: -f1 /etc/passwd); do
  crontab -l -u "$user" 2>/dev/null | grep -v "^#" | grep -v "^$" && \
    echo "  ^ cron for: $user"
done

# Check systemd timers
systemctl list-timers --all

# Check for containers running as root
podman ps --format "{{.Names}} {{.User}}" 2>/dev/null
docker ps --format "{{.Names}} {{.User}}" 2>/dev/null

Network access audit

Does this system phone home? No. There is no telemetry, no analytics, no license check, no heartbeat. Verify it yourself.

# Capture all outbound traffic for 60 seconds
timeout 60 tcpdump -i any -nn -q 'not src host 127.0.0.1' \
  -w /root/network-audit-$(date +%Y%m%d).pcap

# Analyze the capture
tcpdump -r /root/network-audit-*.pcap -nn | \
  awk '{print $5}' | cut -d. -f1-4 | sort | uniq -c | sort -rn

# Real-time: what DNS queries is the system making?
tcpdump -i any -nn port 53

# What processes have outbound connections RIGHT NOW?
ss -tnp | grep ESTAB | awk '{print $5, $6}'

# eBPF: log every outbound connection for the next hour
timeout 3600 tcpconnect-bpfcc -T > /root/outbound-audit.log

Same source + same mirrors = same ISO

Two builds from the same commit, run against the same mirror state, will produce darksites with identical packages. The ISO layout is deterministic. Timestamps and build metadata will differ, but the installed packages are byte-identical.

For full bit-for-bit reproducibility, pin the mirror snapshot (Debian has snapshot.debian.org, Arch has archive.archlinux.org) and set SOURCE_DATE_EPOCH in the build environment.

Source code

Every file is readable. No obfuscation, no minification, no compiled code.

deploy.sh — build orchestration (darksite builds, ISO assembly, VM deploy)

builder/build-iso.sh — rootfs bootstrap, squashfs, EFI, xorriso

build/darksite-*/build-darksite-*.sh — per-distro package downloaders

kldload-install-target — installer entry point

lib/bootstrap.sh — per-distro bootstrap functions (dnf, apt, pacman, apk)

kldload-webui — single-file Python web UI backend

Build environment

The builder container needs internet to fetch packages from upstream mirrors. You can inspect exactly what it downloads by running the darksite build scripts individually and watching the output. Every URL points to an official distro mirror.

# Watch what gets downloaded
./deploy.sh build-debian-darksite  2>&1 | grep -i download
./deploy.sh build-arch-darksite    2>&1 | grep -i download
./deploy.sh build-alpine-darksite  2>&1 | grep -i download

Installed system

# 1. Verify all packages are unmodified
rpm -Va 2>/dev/null | grep -v "^.*c /"    # RPM distros
debsums -c 2>/dev/null                     # Debian/Ubuntu
pacman -Qkk 2>/dev/null | grep -v "0 altered"  # Arch

# 2. Verify ZFS pool integrity
zpool status
zpool scrub rpool && sleep 5 && zpool status rpool

# 3. Verify auditd is running and logging
systemctl status auditd
auditctl -l | wc -l

# 4. Verify snapshots exist
zfs list -t snapshot | wc -l

# 5. Verify no unexpected network connections
ss -tnp | grep ESTAB

# 6. Verify no unexpected listening services
ss -tlnp

# 7. Verify no unexpected SUID binaries
find / -perm -4000 -type f 2>/dev/null | wc -l

# 8. Run package verification + zfs diff for full integrity check
BASELINE=$(zfs list -t snapshot -o name -s creation rpool/ROOT/cs | head -2 | tail -1)
echo "Changes since baseline ($BASELINE):"
zfs diff "$BASELINE" 2>/dev/null | wc -l