Cilium Masterclass

This guide goes deep on Cilium — the eBPF-native CNI that replaces kube-proxy, iptables, and traditional overlay networking with kernel programs that make routing decisions at wire speed. If you have read the Networking tutorial and the Kubernetes on KVM guide, this is the natural next step: you have a cluster, now make the networking layer invisible, unbreakable, and observable.

1. What Cilium Is

Cilium is a Kubernetes CNI plugin — but that undersells it. Traditional CNI plugins (Flannel, Calico, Weave) run in userspace daemons that program iptables rules and VXLAN tunnels. Every packet decision goes through those iptables chains. Cilium is different at the foundation: it compiles network policy into eBPF programs and loads them directly into the Linux kernel. Packet decisions happen in the kernel, before any userspace code runs, with no context switches.

2. Why Cilium on kldload

kldload already ships the eBPF toolchain — bcc, bpftrace, and the full kernel headers needed to compile eBPF programs. The kernel shipped with kldload (5.14+ on CentOS Stream 9, 6.1+ on Debian 13) has BTF (BPF Type Format) enabled, which is what Cilium needs for its CO-RE (Compile Once — Run Everywhere) programs. You are not bolting Cilium onto a foreign kernel. You are running it on a kernel built for this.

3. Install Cilium

Install the Cilium CLI

# Download and install the Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz

tar xzvf cilium-linux-amd64.tar.gz
sudo mv cilium /usr/local/bin/
rm cilium-linux-amd64.tar.gz

# Verify
cilium version --client

Quick install (dev/test)

# Installs Cilium with sensible defaults
# kubeProxyReplacement=true removes kube-proxy entirely
cilium install \
  --set kubeProxyReplacement=true

# Watch it come up
cilium status --wait

Production install with Helm

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=10.80.0.1 \
  --set k8sServicePort=6443 \
  --set ipam.mode=kubernetes \
  --set routingMode=tunnel \
  --set tunnelProtocol=vxlan \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set encryption.enabled=true \
  --set encryption.type=wireguard

If you installed kube-proxy — remove it first

# Delete the kube-proxy DaemonSet before enabling kubeProxyReplacement
kubectl -n kube-system delete daemonset kube-proxy

# Also clean up the ConfigMap
kubectl -n kube-system delete configmap kube-proxy

# Flush any iptables rules kube-proxy left behind
# Run this on every node:
iptables-save | grep -v KUBE | iptables-restore
ip6tables-save | grep -v KUBE | ip6tables-restore

Verify

# Check overall health — should show all green
cilium status

# Run the full connectivity test suite
# Creates a test namespace, deploys pods, tests all traffic paths
cilium connectivity test

# Check which nodes have the Cilium agent running
kubectl -n kube-system get pods -l k8s-app=cilium -o wide

4. Cilium Replaces kube-proxy

kube-proxy's job is simple to describe but expensive to implement: when a pod calls a ClusterIP service, something has to translate that virtual IP to a real pod IP. kube-proxy does this by writing iptables DNAT rules — one per endpoint, chained with probability weights for load balancing.

How kube-proxy works

# What kube-proxy writes to iptables (simplified):
# For a service with 3 endpoints:
-A KUBE-SVC-XYZ -m statistic --mode random --probability 0.333 -j KUBE-SEP-A
-A KUBE-SVC-XYZ -m statistic --mode random --probability 0.500 -j KUBE-SEP-B
-A KUBE-SVC-XYZ -j KUBE-SEP-C

-A KUBE-SEP-A -p tcp -j DNAT --to-destination 10.0.1.5:8080
-A KUBE-SEP-B -p tcp -j DNAT --to-destination 10.0.1.6:8080
-A KUBE-SEP-C -p tcp -j DNAT --to-destination 10.0.1.7:8080

# And a conntrack entry is created for every connection to track the NAT.
# At 5000 services with 10 endpoints each = 50,000+ rules to walk per packet.

How Cilium does it

# Cilium stores service endpoints in a BPF map (a kernel hash table):
# cilium bpf lb list shows the current state

# Service VIP → backend list (one lookup, O(1))
cilium bpf lb list

# Output (abbreviated):
# SERVICE ADDRESS    BACKEND ADDRESS
# 10.96.0.10:53      10.0.0.50:53 (weight 1)
# 10.96.0.10:53      10.0.0.51:53 (weight 1)
# 10.96.120.45:80    10.0.1.22:8080 (weight 1)
# 10.96.120.45:80    10.0.1.23:8080 (weight 1)

# The eBPF program attached to the socket intercepts the connect() syscall
# *before* the packet is even built, rewrites the destination to a backend IP,
# and records it in a local socket cookie. No conntrack. No NAT. No iptables.
# The kernel handles this in the socket layer — the packet that hits the NIC
# already has the correct destination.

Concrete example: pod calls a ClusterIP service

# 1. Pod calls connect("10.96.120.45", 80)
# 2. Cilium's eBPF program intercepts at the socket layer (cgroup/connect4)
# 3. BPF map lookup: 10.96.120.45:80 → [10.0.1.22:8080, 10.0.1.23:8080]
# 4. Select backend: consistent hashing (or round-robin, or maglev)
# 5. Rewrite destination: 10.96.120.45:80 → 10.0.1.22:8080
# 6. Store mapping in socket cookie (no global conntrack table entry)
# 7. Packet leaves pod with real destination. No NAT on return path.

# Compare: kube-proxy flow
# 1. Pod calls connect("10.96.120.45", 80) — kernel builds packet normally
# 2. Packet hits iptables OUTPUT chain
# 3. Walk KUBE-SERVICES → KUBE-SVC-XYZ → probability match → KUBE-SEP-A
# 4. DNAT: rewrite destination
# 5. conntrack entry created to track the NAT for return traffic
# 6. Return packet hits conntrack, gets un-DNAT'd
# Every connection = one conntrack entry. conntrack table is finite.

5. Network Policy with Cilium

Standard Kubernetes NetworkPolicy is L3/L4 — you can allow or deny traffic based on IP addresses, pod selectors, namespaces, and ports. Cilium supports all of that natively, and extends it to L7 (application layer) and DNS-based rules.

L3/L4 policy — standard Kubernetes NetworkPolicy

# Allow frontend pods to reach backend on port 8080
# Deny everything else to backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

L7 HTTP policy — Cilium extension

# Allow GET /api/health from frontend
# Deny POST /api/admin from everywhere
# This is a CiliumNetworkPolicy — Cilium's CRD, not the standard K8s type
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: backend-http-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: GET
          path: /api/health
        - method: GET
          path: /api/v1/.*   # regex paths supported

DNS-based egress policy

# Allow pods to reach *.example.com but nothing else on the internet
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: egress-dns-allowlist
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend
  egress:
  - toFQDNs:
    - matchPattern: "*.example.com"
    - matchName: "api.github.com"
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY

Default-deny namespace policy

# Lock down an entire namespace — deny all by default
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  endpointSelector: {}   # matches all pods in namespace
  ingress:
  - {}                   # allow nothing (no fromEndpoints = implicit deny)
  egress:
  - {}                   # allow nothing
---
# Then layer explicit allow rules on top for each service that needs them

6. Hubble — Network Observability

Hubble is Cilium's observability layer. It is built into the Cilium agent and exports a structured stream of every network flow between pods — with pod names, namespaces, labels, verdict (forwarded/dropped/policy-denied), and the specific policy that caused a drop. This is not pcap. It is structured metadata at the Kubernetes identity layer.

Enable Hubble

# Enable Hubble relay and UI (if not done at install time)
cilium hubble enable --ui

# Or via Helm:
helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set hubble.metrics.enabled="{dns,drop,tcp,flow,icmp,http}"

Install the Hubble CLI

HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/hubble/releases/download/${HUBBLE_VERSION}/hubble-linux-amd64.tar.gz

tar xzvf hubble-linux-amd64.tar.gz
sudo mv hubble /usr/local/bin/
rm hubble-linux-amd64.tar.gz

# Set up port-forward to the Hubble relay
cilium hubble port-forward &

# Verify
hubble status

Watch live flows

# All flows in the cluster — live
hubble observe

# Filter to a specific namespace
hubble observe --namespace production

# Filter to flows involving a specific pod
hubble observe --pod production/backend-7f9c8b-xkp2q

# Watch only dropped packets (policy violations, unreachable endpoints)
hubble observe --verdict DROPPED

# Watch only policy-denied flows
hubble observe --verdict POLICY_DENIED

# Trace the path of a specific request
# (shows source, destination, verdict at each hop)
hubble observe \
  --from-pod production/frontend-abc123 \
  --to-pod production/backend-xyz789 \
  --protocol TCP

Hubble UI

# Open the Hubble UI in a browser (port-forward to the UI service)
cilium hubble ui

# This opens http://localhost:12000 — a graphical network map
# showing all pods as nodes and traffic flows as edges,
# color-coded by verdict (green=forwarded, red=dropped)

Hubble metrics for Prometheus

# Hubble exports Prometheus metrics on port 9965 of each Cilium agent pod
# Key metrics:
# hubble_flows_processed_total{verdict="FORWARDED"}
# hubble_flows_processed_total{verdict="DROPPED"}
# hubble_drop_total{reason="POLICY_DENIED"}
# hubble_http_requests_total{method="GET",protocol="HTTP/1.1"}
# hubble_dns_queries_total{rcode="NOERROR"}

kubectl -n kube-system port-forward daemonset/cilium 9965:9965 &
curl http://localhost:9965/metrics | grep hubble_flows

7. Transparent Encryption (WireGuard Mode)

Cilium can encrypt all pod-to-pod traffic crossing node boundaries using WireGuard. Every Cilium agent generates a WireGuard keypair. Peers exchange public keys automatically via the Kubernetes API. No certificates, no PKI, no manual configuration. Every cross-node pod packet is encrypted without any application changes.

Enable WireGuard encryption

# Enable at install time (recommended)
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set encryption.enabled=true \
  --set encryption.type=wireguard \
  --set encryption.wireguard.userspaceFallback=false

# Or enable on an existing installation
helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set encryption.enabled=true \
  --set encryption.type=wireguard

# Restart Cilium agents to pick up new config
kubectl -n kube-system rollout restart daemonset/cilium

Verify encryption is active

# Check Cilium status — should show "WireGuard (enabled)"
cilium status | grep Encryption

# Inspect the WireGuard interface that Cilium creates on each node
# (SSH to a node, or exec into the Cilium pod)
kubectl -n kube-system exec -ti daemonset/cilium -- cilium-dbg encrypt status

# Output shows:
# Encryption: WireGuard
# Interface: cilium_wg0
# Public key: 
# Peers:  connected

# Confirm with a tcpdump — you should see WireGuard UDP (port 51871)
# but NO readable HTTP/TCP inside those packets
tcpdump -i eth0 -n 'udp port 51871'

Node keys are managed automatically

# Cilium stores WireGuard public keys as Kubernetes node annotations
kubectl get node k8s-worker-1 -o jsonpath='{.metadata.annotations}' | jq .

# Output includes:
# "network.cilium.io/wg-pub-key": "base64pubkey=="

# When a new node joins, it reads the public keys of all existing nodes
# from these annotations and establishes WireGuard sessions automatically.
# No manual peer configuration. No key distribution problem.

8. Service Mesh Without Sidecars

Traditional service meshes (Istio, Linkerd) inject a proxy container (Envoy or similar) into every pod. This proxy intercepts all traffic, adds mTLS, collects telemetry, and enforces retries and circuit breakers. The sidecar model works, but it is expensive: each proxy consumes 50–100 MB of RAM, 0.1–0.5 CPU cores, and adds a userspace hop to every request.

Cilium's service mesh mode does mTLS, traffic shaping, and observability in eBPF — no sidecars, no Envoy per pod, no extra memory.

Enable Cilium Service Mesh

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set ingressController.enabled=true \
  --set ingressController.default=true \
  --set gatewayAPI.enabled=true

Traffic splitting (canary deployments)

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend-canary
  namespace: production
spec:
  parentRefs:
  - name: cilium-gateway
  rules:
  - backendRefs:
    - name: backend-stable
      port: 8080
      weight: 90
    - name: backend-canary
      port: 8080
      weight: 10

Retries and timeouts

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend-with-retries
  namespace: production
spec:
  parentRefs:
  - name: cilium-gateway
  rules:
  - backendRefs:
    - name: backend
      port: 8080
    filters:
    - type: ExtensionRef
      extensionRef:
        group: cilium.io
        kind: CiliumHTTPRetryPolicy
        name: standard-retries

9. BGP with Cilium

Cilium has a built-in BGP speaker that can announce Kubernetes LoadBalancer service IPs to your LAN router. This replaces MetalLB. Combined with the FRRouting BGP setup from the Networking tutorial, your router learns about both kldload host subnets and Kubernetes service IPs from the same BGP session.

Enable Cilium BGP

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set bgpControlPlane.enabled=true

Configure a BGP peering policy

# Tell Cilium which nodes should peer with which router
# and what ASN to use
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPPeeringPolicy
metadata:
  name: bgp-peering-policy
spec:
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux    # all nodes
  virtualRouters:
  - localASN: 65001              # your cluster ASN
    exportPodCIDR: false         # don't advertise pod CIDRs (service IPs only)
    neighbors:
    - peerAddress: 192.168.1.1/32   # your LAN router
      peerASN: 65000               # your router's ASN
    serviceSelector:
      matchExpressions:
      - key: somekey
        operator: NotIn
        values:
        - never-select             # match all services

Assign an IP pool for LoadBalancer services

# Define a pool of IPs that Cilium can assign to LoadBalancer services
# These IPs will be announced via BGP
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: lan-pool
spec:
  cidrs:
  - cidr: 192.168.10.0/24    # a dedicated range on your LAN
  serviceSelector:
    matchLabels:
      announce: bgp

Create a service and watch it get announced

apiVersion: v1
kind: Service
metadata:
  name: my-app
  labels:
    announce: bgp    # matches the pool selector above
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080

# Watch Cilium assign an IP and announce it
kubectl get service my-app -w

# Check BGP session status
kubectl get ciliumbgppeeringstatus -A

# Verify the route appeared on your router
# (on your FRRouting instance or LAN router)
vtysh -c 'show ip bgp summary'
vtysh -c 'show ip route bgp'

10. Bandwidth Manager & Rate Limiting

Cilium's Bandwidth Manager enforces per-pod egress bandwidth limits using eBPF and the kernel's EDT (Earliest Departure Time) mechanism. Unlike tc rate limiting or CNI chaining hacks, this is a single eBPF program attached to each pod's network interface — zero overhead when there is no limit, precise enforcement when there is.

Enable the Bandwidth Manager

helm upgrade cilium cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set bandwidthManager.enabled=true \
  --set bandwidthManager.bbr=true    # use BBR congestion control (recommended)

Set per-pod limits via annotations

apiVersion: v1
kind: Pod
metadata:
  name: rate-limited-app
  annotations:
    # Egress bandwidth limit for this pod
    kubernetes.io/egress-bandwidth: "100M"
    # Ingress bandwidth limit
    kubernetes.io/ingress-bandwidth: "100M"
spec:
  containers:
  - name: app
    image: my-app:latest

Verify bandwidth enforcement

# Check that the bandwidth manager is active
cilium status | grep Bandwidth

# Inspect the eBPF programs attached to a pod's interface
# (get the pod's veth interface name first)
POD_IFACE=$(kubectl get pod rate-limited-app -o jsonpath='{.status.podIP}' | \
  xargs -I{} ip r get {} | awk '{print $NF}')

# View TC programs attached to that interface
tc filter show dev $POD_IFACE egress

# Run an iperf3 test to verify the limit
kubectl run iperf-server --image=networkstatic/iperf3 -- -s
kubectl run iperf-client --image=networkstatic/iperf3 \
  --annotations='kubernetes.io/egress-bandwidth=50M' \
  -- -c $(kubectl get pod iperf-server -o jsonpath='{.status.podIP}') -t 10

11. Troubleshooting

Basic health check

# Overall Cilium health — first thing to check
cilium status

# Run the full connectivity test (creates a test namespace)
cilium connectivity test

# Check agent logs on a specific node
kubectl -n kube-system logs daemonset/cilium --previous 2>&1 | tail -100

# Get a shell in the Cilium agent pod for deeper inspection
kubectl -n kube-system exec -ti daemonset/cilium -- bash

Hubble for live debugging

# Watch all flows in real time — look for DROPPED or POLICY_DENIED
hubble observe --verdict DROPPED --follow

# Trace a specific pod's traffic
hubble observe --pod production/my-pod --follow

# Check DNS resolution for a pod
hubble observe --pod production/my-pod --protocol DNS

# See which policy is dropping traffic
hubble observe --verdict POLICY_DENIED -o json | jq '.flow.drop_reason_desc'

eBPF map inspection

# List all services in the BPF load balancer map
cilium bpf lb list

# Show the identity map (pod labels → numeric security identity)
cilium bpf identity list

# Show endpoint policy map for a specific endpoint
cilium bpf policy get

# Show nat table (should be empty or minimal with kubeProxyReplacement=true)
cilium bpf nat list

# Show connection tracking table
cilium bpf ct list global

Common issues

Kernel requirements (kldload is fine)

# Verify your kernel meets Cilium's requirements
# kldload ships kernels that pass all of these

# Minimum: 4.19 for basic eBPF
# Recommended: 5.10+ for full feature set
# kube-proxy replacement: 5.8+
# WireGuard encryption: 5.6+
# BBR bandwidth manager: 5.8+
uname -r

# Check BTF support (required for CO-RE, present on all kldload kernels)
ls /sys/kernel/btf/vmlinux

# Check BPF filesystem
mount | grep bpf

# Check that the required kernel config options are enabled
zcat /proc/config.gz | grep -E 'CONFIG_BPF|CONFIG_WIREGUARD'

12. The Full kldload Networking Stack

Putting it all together: every layer of the kldload networking stack has a specific job. None of them overlap. None of them replace each other. They compose.

Related pages

• Networking tutorial — VXLAN, BGP, eBPF dataplane foundations
• WireGuard Masterclass — deep dive on the host encryption layer
• WireGuard Mesh & Multi-Site — the four-plane mesh that Cilium runs over
• Kubernetes on KVM — building the cluster that Cilium runs on
• eBPF Reference — bcc and bpftrace alongside Cilium
• eBPF Security — kernel-level security programs
• Monitoring Stack Glossary (355 terms) Help & Links — Hubble metrics in Prometheus and Grafana