Part 2: The back plane

The back plane is the part you usually have to build yourself.

Every working API developer already understands that production services have a back plane — a side-band channel for the things the request/response path shouldn't handle. Replication. Encryption in transit. Snapshots. Metrics. Audit. Disaster recovery. The question isn't whether you know the back plane exists. It's where it lives.

On most stacks, the back plane is a separate stack of products you provision next to your service: an APM agent, a backup daemon, a VPN client, a sidecar for mTLS, an object-storage SDK for off-host copies, a Prometheus exporter, a log forwarder. Each one is a dependency to install, a config file to keep in sync, a thing to forget on the staging box, a thing to break on a deploy. Your API code ends up coordinating across them — signalling the backup service, signalling the replication service, signalling the metrics endpoint — each over its own protocol.

kldload moves the back plane into the kernel. The same primitives are already there because they were compiled in at image build time. Your code signals ZFS for a snapshot, WireGuard for an encrypted destination, eBPF for a probe, Cilium for a network policy — not a daemon wrapping them, the kernel module itself. There's no agent to crash, no sidecar to scale, no config drift between staging and prod. Same kernel, same signals, every machine.

Think about what's happening underneath a typical web service:

Front plane (your API):
  POST /api/upload → accept file → store → return 200

Back plane (what the infrastructure handles):
  → ZFS checksums every block as it's written (data integrity)
  → ZFS compresses the data transparently (storage efficiency)
  → Snapshot fires automatically before the write (undo capability)
  → WireGuard encrypts the replication stream (secure distribution)
  → zfs send pushes the dataset to DR site (disaster recovery)
  → eBPF traces the write latency (observability)

Your API wrote one line: store(file). The back plane did six things
your API never asked for, never configured, and can't break.

This is what "baked into the image" actually means. When the back plane lives in the kernel — not in a sidecar, not in an agent, not in a config file that gets missed during deployment — every application on the machine gets it for free. Your API doesn't import a ZFS library or call a snapshot endpoint. When it does want to reach into the back plane explicitly, it signals the kernel module directly: zfs snapshot, zfs send over a WireGuard peer, an eBPF probe attached to a syscall, a Cilium policy applied to a label. The signal goes to the kernel, not to a userland coordinator that wraps it.

Now think about what you can add to your API when you know the back plane exists:

# Your API can now offer these features with zero additional infrastructure:

# Point-in-time recovery for any customer
GET /api/customer/acme/restore?snapshot=2026-04-01T14:00
→ zfs rollback rpool/customers/acme@2026-04-01T14:00

# Instant staging environment from production data
POST /api/staging/create?from=production
→ zfs clone rpool/production@latest rpool/staging/ticket-4521
→ returns in 0.2 seconds regardless of dataset size

# Encrypted replication to customer's own infrastructure
POST /api/customer/acme/replicate?target=acme-dr.example.com
→ zfs send | ssh wg-peer zfs recv
→ your customer gets a verified copy, never decrypted in transit

# Real-time I/O observability without APM agents
GET /api/debug/slow-queries
→ eBPF traces disk latency per dataset, returns top offenders

None of these features required a new dependency. No new database. No new queue. No new service to deploy and monitor. They're API wrappers around kernel primitives that already exist on the machine because they were baked into the image at build time.

This is the power of embedding the back plane at image creation: every machine you deploy has these capabilities from second zero. You don't configure them per-host. You don't install them post-deploy. You don't forget them on the staging server. They're structural. They're in the image. They're everywhere, always.

The image is the deployment. The deployment is the infrastructure.

Traditional infrastructure has stages: provision the machine, install the OS, configure networking, add storage, deploy the app, bolt on monitoring, set up backups. Each stage is a separate tool, a separate config, a separate failure mode. Miss one and you have a machine that's 90% right — which is the same as wrong.

When you bake the back plane into the image, the stages collapse:

Traditional:
  provision → install OS → configure networking → add storage
  → deploy app → add monitoring → configure backups → harden
  → pray nothing was missed
  (8 stages, 8 tools, 8 places to make a mistake)

kldload:
  boot the image
  (the image IS the OS + networking + storage + snapshot policy + security hardening)
  → deploy your app
  (2 stages. The first one is turning the machine on.)

This isn't about saving time, although it does. It's about eliminating the gaps between stages where things go wrong. The machine that doesn't have monitoring because someone forgot the Ansible role. The server that doesn't have backups because the cron job wasn't in the golden image. The production box running without encryption because the WireGuard config was "TODO."

When it's in the image, it's everywhere. When it's a post-deploy step, it's everywhere you remembered to put it.

Part 3: Datasets as service boundaries

Datasets are service boundaries. Design them that way.

Any service that manipulates files — an upload processor, a transcoder, a log aggregator, a CI runner — is reading and writing to a path. That path is a directory. On ext4, all directories share one filesystem, one set of I/O characteristics, one failure domain. A runaway log writer starves the upload processor. A transcoder's sequential 4GB writes thrash the CI runner's random 4K reads. Everything competes for the same disk, the same cache, the same throughput.

On ZFS, you separate them into datasets. The service doesn't know the difference — it still reads and writes to a path. But now each path has its own tuning:

# Upload processor — large incoming files, compress well, limit space
zfs create -o recordsize=1M -o compression=zstd -o quota=500G \
  rpool/srv/uploads

# Transcoder — huge sequential reads/writes, skip compression (already encoded)
zfs create -o recordsize=1M -o compression=off -o atime=off \
  rpool/srv/transcode

# Log aggregator — append-only, compress heavily, rotate via snapshots
zfs create -o recordsize=1M -o compression=zstd-19 -o atime=off \
  rpool/srv/logs

# CI runner — small random I/O, fast metadata, throw away between builds
zfs create -o recordsize=16K -o compression=lz4 -o sync=disabled \
  rpool/srv/ci-workspace

# Database — tuned to page size, protect every write
zfs create -o recordsize=8K -o logbias=throughput -o compression=lz4 \
  rpool/srv/postgres

Each service gets I/O characteristics matched to its workload. The transcoder's 1M sequential writes don't pollute the database's 8K random reads in the ARC. The CI runner's throwaway workspace has sync=disabled because losing it on power failure doesn't matter — but the database has full sync because every write matters. Never use sync=disabled on anything you can't afford to lose on power failure. The log aggregator compresses at zstd-19 because it's write-once-read-rarely and saves 80% of disk space.

Now the interesting part — you can optimize the routing between them:

# User uploads a video. Traditional approach:
#   write to /tmp → copy to /uploads → copy to /transcode → copy to /output
#   Three copies of a 4GB file. 12GB of I/O for one upload.

# ZFS approach:
#   write to rpool/srv/uploads (one write)
#   zfs clone rpool/srv/uploads@snap rpool/srv/transcode/job-123 (instant, zero I/O)
#   transcode in place on the clone
#   zfs send the result to rpool/srv/output (block-level, only changed blocks)
#   destroy the clone

# One write. One clone (free). One incremental send. Total I/O: ~4GB instead of 12GB.
# The upload processor, transcoder, and output service never copied a file.
# They worked on datasets. ZFS moved the blocks.

This is what happens when you stop thinking of services as programs that read files and start thinking of them as consumers of datasets. The file manipulation becomes a dataset operation. The copy becomes a clone. The transfer becomes a send. The backup is a snapshot that already happened. You cut the I/O in half or better, and every stage is atomic, checksummed, and reversible.

The services never know. They read a path. They write to a path. But underneath, each path is a tuned, isolated, snapshotted, replicable storage domain — and the data flows between them at the block level instead of being copied through userland.