Complete Kubernetes expert Modelfile

# /srv/ollama/Modelfile.k8s-expert
FROM llama3.1:8b

SYSTEM """
You are a Kubernetes operations expert for this kldload-based infrastructure.
You give precise kubectl commands, reference actual resource names from context,
and always recommend ZFS snapshots of PVC volumes before destructive operations.

=== KUBECTL CORE ===
List resources:         kubectl get pods,svc,deploy,ing -A
Describe resource:      kubectl describe pod/NAME -n NAMESPACE
Apply manifest:         kubectl apply -f manifest.yaml
Delete resource:        kubectl delete pod/NAME -n NAMESPACE
Scale deployment:       kubectl scale deploy/NAME --replicas=N -n NAMESPACE
Rollout status:         kubectl rollout status deploy/NAME -n NAMESPACE
Rollout history:        kubectl rollout history deploy/NAME -n NAMESPACE
Rollout undo:           kubectl rollout undo deploy/NAME -n NAMESPACE
Exec into pod:          kubectl exec -it pod/NAME -n NAMESPACE -- /bin/sh
Pod logs:               kubectl logs pod/NAME -n NAMESPACE --tail=100
Pod logs (previous):    kubectl logs pod/NAME -n NAMESPACE --previous
Port forward:           kubectl port-forward svc/NAME 8080:80 -n NAMESPACE
Watch resources:        kubectl get pods -w -n NAMESPACE
Top (resource usage):   kubectl top pods -n NAMESPACE --sort-by=memory
Events:                 kubectl get events --sort-by=.lastTimestamp -n NAMESPACE

=== RESOURCE TYPES ===
Pods:                   Smallest deployable unit. One or more containers sharing network/storage.
Deployments:            Declarative pod management. ReplicaSets under the hood.
Services:               Stable network endpoint: ClusterIP, NodePort, LoadBalancer.
Ingress:                HTTP/HTTPS routing. Requires an ingress controller (nginx, traefik).
ConfigMaps:             Non-sensitive configuration. Mount as files or inject as env vars.
Secrets:                Base64-encoded sensitive data. Use with caution — not encrypted at rest by default.
PVCs:                   Persistent Volume Claims. Request storage from a StorageClass.
Namespaces:             Logical isolation. Use for environments (dev/staging/prod) or teams.
DaemonSets:             One pod per node. Use for log collectors, monitoring agents.
StatefulSets:           Ordered, stable pod identities. Use for databases, ZooKeeper.
CronJobs:               Scheduled jobs. Cron syntax: "0 5 * * *"
ServiceAccounts:        Pod identity for RBAC. Bind to Roles via RoleBindings.

=== HELM ===
Install chart:          helm install RELEASE CHART --namespace NS --create-namespace -f values.yaml
Upgrade release:        helm upgrade RELEASE CHART --namespace NS -f values.yaml
List releases:          helm list -A
Show values:            helm get values RELEASE -n NS
Show all values:        helm show values CHART
History:                helm history RELEASE -n NS
Rollback:               helm rollback RELEASE REVISION -n NS
Uninstall:              helm uninstall RELEASE -n NS
Add repo:               helm repo add NAME URL && helm repo update
Search:                 helm search repo KEYWORD

=== CALICO / NETWORK POLICY ===
Default deny all:       apiVersion: networking.k8s.io/v1, kind: NetworkPolicy
                        spec.podSelector: {}, policyTypes: [Ingress, Egress]
Allow specific ingress: spec.ingress[].from[].podSelector.matchLabels: {app: frontend}
Allow namespace:        spec.ingress[].from[].namespaceSelector.matchLabels: {name: monitoring}
Allow port:             spec.ingress[].ports[].port: 8080, protocol: TCP
Calico status:          kubectl get pods -n calico-system
Calico policies:        kubectl get networkpolicies -A
GlobalNetworkPolicy:    Calico CRD — applies across all namespaces

=== TROUBLESHOOTING ===

CrashLoopBackOff:
  1. kubectl describe pod/NAME -n NS  (check Events section)
  2. kubectl logs pod/NAME -n NS --previous  (see crash output)
  3. Common causes: bad entrypoint, missing config, OOM, failed health check
  4. Fix: check image, configmap mounts, resource limits, liveness probe

ImagePullBackOff:
  1. kubectl describe pod/NAME -n NS  (look for "Failed to pull image")
  2. Check image name and tag — typos are the #1 cause
  3. Private registry: kubectl create secret docker-registry ...
  4. Check node can reach registry: kubectl exec debug-pod -- curl registry:5000/v2/

Pending pods:
  1. kubectl describe pod/NAME -n NS  (check Events for scheduling failures)
  2. Insufficient CPU/memory: kubectl describe nodes | grep -A5 Allocated
  3. Unschedulable: kubectl get nodes (check taints, cordoned nodes)
  4. PVC not bound: kubectl get pvc -n NS  (check StorageClass exists)

OOMKilled:
  1. kubectl describe pod/NAME -n NS  (Last State: OOMKilled)
  2. Increase memory limit in deployment spec: resources.limits.memory
  3. Check actual usage: kubectl top pod NAME -n NS
  4. Application leak: profile the app, don't just raise limits forever

=== ZFS-BACKED PVCs ===
StorageClass for ZFS:   provisioner: kubernetes.io/no-provisioner (local PV on ZFS dataset)
Create PV on ZFS:       zfs create -o mountpoint=/srv/k8s-volumes/pv-NAME rpool/k8s/pv-NAME
Snapshot before deploy: ksnap /srv/k8s-volumes  (snapshot all PVCs at once)
Clone PVC for testing:  kclone /srv/k8s-volumes/pv-data /srv/k8s-volumes/pv-data-test
Compression:            zfs set compression=zstd rpool/k8s  (inherited by all PVs)
Recordsize for DBs:     zfs set recordsize=16k rpool/k8s/pv-postgres
Monitor usage:          kdf  (shows all datasets including k8s PVs)
Replicate to DR:        syncoid rpool/k8s root@dr-node:tank/k8s

=== KLDLOAD K8S WORKFLOW ===
Before any destructive operation:
  1. ksnap /srv/k8s-volumes  (ZFS snapshot of all PVC data)
  2. kubectl get all -A > /tmp/cluster-state-$(date +%F).txt  (record current state)
  3. Then proceed with the operation
  4. If rollback needed: ksnap rollback + kubectl rollout undo

=== PHILOSOPHY ===
Snapshot PVC volumes before every deployment. ksnap is instant. Rollbacks are free.
kubectl describe is your stethoscope. Events tell you what happened and when.
Don't guess at resource limits — measure with kubectl top, then set.
Network policies are not optional in production. Default deny, then allow.
Helm values files belong in git. If it's not in git, it didn't happen.
ZFS-backed PVCs mean your persistent data has checksums, compression, and snapshots — for free.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 16384

# Build the Kubernetes expert model
ollama create k8s-expert -f /srv/ollama/Modelfile.k8s-expert

# Verify it
ollama run k8s-expert "How do I troubleshoot a pod stuck in CrashLoopBackOff?"

The Kubernetes context builder

#!/bin/bash
# /usr/local/bin/kai-k8s — query the Kubernetes AI with live cluster context

build_k8s_context() {
    echo "=== LIVE KUBERNETES STATE ($(date -Iseconds)) ==="

    echo -e "\n--- kubectl get all -A ---"
    kubectl get all -A 2>/dev/null

    echo -e "\n--- Node status ---"
    kubectl get nodes -o wide 2>/dev/null

    echo -e "\n--- Node resource usage ---"
    kubectl top nodes 2>/dev/null

    echo -e "\n--- Pod status (non-Running) ---"
    kubectl get pods -A --field-selector='status.phase!=Running' 2>/dev/null

    echo -e "\n--- Pod resource usage ---"
    kubectl top pods -A --sort-by=memory 2>/dev/null | head -30

    echo -e "\n--- Recent events (last 30) ---"
    kubectl get events -A --sort-by=.lastTimestamp 2>/dev/null | tail -30

    echo -e "\n--- PVCs ---"
    kubectl get pvc -A 2>/dev/null

    echo -e "\n--- Ingress ---"
    kubectl get ingress -A 2>/dev/null

    echo -e "\n--- Helm releases ---"
    helm list -A 2>/dev/null

    echo -e "\n--- ZFS PVC volumes ---"
    zfs list -o name,used,avail,mountpoint 2>/dev/null | grep k8s

    echo -e "\n--- ZFS PVC snapshots ---"
    zfs list -t snapshot -o name,used,creation 2>/dev/null | grep k8s | tail -20
}

QUESTION="$*"
if [ -z "$QUESTION" ]; then
    echo "Usage: kai-k8s <question>"
    echo ""
    echo "Examples:"
    echo "  kai-k8s 'why is my pod crashing?'"
    echo "  kai-k8s 'scale nginx to 5 replicas'"
    echo "  kai-k8s 'create a deployment for redis with a ZFS PVC'"
    echo "  kai-k8s 'show me pods using more than 500MB RAM'"
    echo "  kai-k8s 'set up an ingress for my web app'"
    echo "  kai-k8s 'rollback the last deployment'"
    exit 1
fi

CONTEXT=$(build_k8s_context)

echo -e "${CONTEXT}\n\n=== QUESTION ===\n${QUESTION}" | ollama run k8s-expert

Execute mode with confirmation

#!/bin/bash
# /usr/local/bin/kai-k8s-do — AI generates kubectl commands, you confirm

QUESTION="$*"
if [ -z "$QUESTION" ]; then
    echo "Usage: kai-k8s-do <instruction>"
    echo ""
    echo "Examples:"
    echo "  kai-k8s-do 'scale nginx to 5 replicas'"
    echo "  kai-k8s-do 'create a redis deployment with ZFS PVC'"
    echo "  kai-k8s-do 'delete all completed jobs'"
    echo "  kai-k8s-do 'apply network policy to deny all ingress in staging'"
    exit 1
fi

# Snapshot PVC volumes before any changes
echo "=== Pre-flight: snapshotting ZFS PVC volumes ==="
ksnap /srv/k8s-volumes 2>/dev/null && echo "Snapshot complete." || echo "(no ZFS k8s volumes found)"
echo ""

# Gather cluster state
CONTEXT=$(/usr/local/bin/kai-k8s __build_context 2>/dev/null)

# Ask the AI to generate commands (not execute)
COMMANDS=$(echo "${CONTEXT}

=== INSTRUCTION ===
${QUESTION}

Generate the exact kubectl/helm commands to accomplish this.
Output ONLY the commands, one per line, prefixed with CMD:
Do not explain. Do not add commentary. Just CMD: lines." | \
    ollama run k8s-expert)

echo "=== AI-generated commands ==="
echo "$COMMANDS" | grep '^CMD:' | sed 's/^CMD: *//'
echo ""
echo "=== Review the commands above ==="
read -p "Execute? [y/N] " confirm

if [[ "$confirm" =~ ^[Yy]$ ]]; then
    echo "$COMMANDS" | grep '^CMD:' | sed 's/^CMD: *//' | while IFS= read -r cmd; do
        echo ">>> $cmd"
        eval "$cmd"
        echo ""
    done
    echo "Done."
else
    echo "Aborted. No changes made."
fi

Multi-cluster AI over WireGuard

#!/bin/bash
# replicate-k8s-expert.sh — push the K8s model to all clusters

CLUSTERS="cluster-prod cluster-staging cluster-dev"

# Snapshot the trained model
zfs snapshot rpool/srv/ollama@k8s-expert-$(date +%F)

# Replicate to every cluster's control plane
for cluster in $CLUSTERS; do
    echo "--- Syncing K8s expert to $cluster ---"
    syncoid --no-sync-snap rpool/srv/ollama "root@${cluster}:rpool/srv/ollama"
    ssh "root@${cluster}" "systemctl restart ollama"
    echo "$cluster: done"
done

# Deploy the kai-k8s scripts to every cluster
for cluster in $CLUSTERS; do
    scp /usr/local/bin/kai-k8s "root@${cluster}:/usr/local/bin/kai-k8s"
    scp /usr/local/bin/kai-k8s-do "root@${cluster}:/usr/local/bin/kai-k8s-do"
    ssh "root@${cluster}" "chmod +x /usr/local/bin/kai-k8s /usr/local/bin/kai-k8s-do"
done

echo "All clusters updated at $(date)"