| your Linux re-packer
kldload — your platform, your way, anywhere, free
Source

Keycloak & SELinux Masterclass

This guide covers two pillars of enterprise-grade authentication and access control: Keycloak for centralised identity management — SSO, OIDC, SAML, realm design, client registration, token lifecycle, and federation — and SELinux for mandatory access control at the kernel level: policies, contexts, booleans, custom modules, multi-category security, and the real-world troubleshooting workflow that keeps enforcing mode on in production. Together they answer the question every serious infrastructure asks: who are you, and what are you allowed to touch?

The premise: Authentication (who you are) and authorisation (what you can do) are different problems that most teams solve with the same tool — usually an LDAP directory or a flat user database. Keycloak gives you a proper identity layer: single sign-on, token-based auth, federated identity, fine-grained consent. SELinux gives you a proper enforcement layer: even if an attacker gets root, mandatory access control limits what root can touch. One protects the front door. The other protects every room inside.

What this page covers: Keycloak architecture and deployment, realm and client design, OIDC and SAML flows, token anatomy and validation, user federation (LDAP/AD), role-based and attribute-based access, reverse proxy integration, Kubernetes SSO. Then: SELinux fundamentals, policy types, file contexts, booleans, port labelling, custom policy modules with audit2allow, MCS/MLS, container SELinux, and the complete troubleshooting playbook. All grounded in the kldload stack.

Prerequisites: a running kldload system. The Kubernetes sections assume a cluster from the Kubernetes on KVM guide. TLS certificates from the TLS & PKI Masterclass.

Most teams disable SELinux on day one because it blocks something and nobody knows why. Then they run Keycloak in a Docker container with no TLS because "it's internal." The result is a system that authenticates users with a single password and authorises processes with no restrictions at all. This masterclass exists because both problems are solvable — Keycloak is straightforward once you understand realms and clients, and SELinux is straightforward once you understand contexts and booleans. The tooling has matured enormously. The reason people still skip them is not complexity — it is unfamiliarity. This guide fixes that.

1. Keycloak Fundamentals

Keycloak is an open-source identity and access management server. It handles user authentication, token issuance, single sign-on, identity brokering, and user federation. Your applications never see passwords — they receive cryptographically signed tokens from Keycloak and validate them locally.

Realm

A realm is an isolated authentication domain. Each realm has its own users, clients, roles, and identity providers. The master realm is for Keycloak administration only — never put application users in it. Create one realm per trust boundary: production, staging, corporate.

// Realm = a separate building with its own keys. Users in one realm cannot see another.

Client

A client is any application that delegates authentication to Keycloak. Web apps, CLI tools, API gateways, mobile apps — each registers as a client in a realm. Clients have a type: confidential (has a secret, e.g. backend API), public (no secret, e.g. SPA or mobile app), or bearer-only (validates tokens but never initiates login).

// Client = an application that says "Keycloak, please log this user in for me."

User & Role

Users live in a realm. They can have realm roles (global to the realm) or client roles (scoped to a single application). Roles are embedded in tokens as claims. Applications read claims to make authorisation decisions — Keycloak never tells an app what to do, it tells the app who the user is and what roles they hold.

// "Here is a signed token. The user is alice. She has roles: admin, db-reader. You decide what that means."

Identity Provider (IdP)

Keycloak can broker authentication to external providers: Google, GitHub, SAML-based corporate IdPs, LDAP directories, or another Keycloak instance. Users authenticate at the external IdP and Keycloak maps their identity into the local realm. This is how you get "Login with GitHub" or federate with Active Directory.

// "I don't know this user, but Google does. Google says they're legit. I'll issue my own token."

Token

Keycloak issues three tokens: an access token (short-lived, sent with API requests), a refresh token (longer-lived, used to get new access tokens without re-login), and an ID token (contains user profile claims for the client). All are JWTs signed with the realm's RSA or EC key.

// Access token = your badge. Refresh token = the ability to get a new badge without going back to reception.

Protocol: OIDC vs SAML

OpenID Connect (OIDC) is built on OAuth 2.0 and uses JSON/JWT. SAML 2.0 is XML-based. Use OIDC for everything new — it is simpler, faster, and better supported. SAML exists for legacy enterprise apps that predate OIDC. Keycloak supports both natively.

// OIDC = modern JSON badges. SAML = legacy XML badges. Same job, different era.
The single biggest conceptual shift is this: your application does not authenticate users. Keycloak authenticates users. Your application receives a signed token and trusts it because it trusts Keycloak's signing key. This means you can add SSO to ten applications by registering ten clients in one realm — users log in once and get tokens for all of them. It also means your application never handles passwords, which eliminates an entire class of vulnerabilities.

2. Deploying Keycloak on kldload

Keycloak runs as a Java application (Quarkus runtime since v17+). It needs a database for persistence (PostgreSQL recommended), TLS termination, and a reverse proxy in front.

Install on bare metal (systemd)

# Download Keycloak (check for latest version)
KC_VERSION=24.0.4
curl -LO https://github.com/keycloak/keycloak/releases/download/${KC_VERSION}/keycloak-${KC_VERSION}.tar.gz
tar xzf keycloak-${KC_VERSION}.tar.gz -C /opt/
ln -sfn /opt/keycloak-${KC_VERSION} /opt/keycloak

# Create a dedicated system user
useradd -r -s /sbin/nologin -d /opt/keycloak keycloak
chown -R keycloak:keycloak /opt/keycloak

# Create the PostgreSQL database
sudo -u postgres psql -c "CREATE USER keycloak WITH PASSWORD 'changeme';"
sudo -u postgres psql -c "CREATE DATABASE keycloak OWNER keycloak;"

Configure Keycloak

# /opt/keycloak/conf/keycloak.conf
# Database
db=postgres
db-url=jdbc:postgresql://localhost:5432/keycloak
db-username=keycloak
db-password=changeme

# Hostname — the public URL users see
hostname=auth.example.com
hostname-strict=true

# HTTP — Keycloak listens on 8080, TLS terminated by nginx
http-enabled=true
http-port=8080
proxy-headers=xforwarded

# Metrics and health (for monitoring)
health-enabled=true
metrics-enabled=true

# Logging
log=console,file
log-file=/var/log/keycloak/keycloak.log
log-level=INFO

Build and start

# Build the optimised runtime (Quarkus ahead-of-time compilation)
/opt/keycloak/bin/kc.sh build

# Bootstrap admin user (first run only)
KEYCLOAK_ADMIN=admin KEYCLOAK_ADMIN_PASSWORD=changeme \
  /opt/keycloak/bin/kc.sh start

systemd unit

# /etc/systemd/system/keycloak.service
[Unit]
Description=Keycloak Identity Server
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=exec
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/bin/kc.sh start --optimized
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

# Hardening
ProtectSystem=strict
ReadWritePaths=/opt/keycloak /var/log/keycloak
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now keycloak
systemctl status keycloak

Reverse proxy with nginx

# /etc/nginx/conf.d/keycloak.conf
server {
    listen 443 ssl http2;
    server_name auth.example.com;

    ssl_certificate     /etc/letsencrypt/live/auth.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/auth.example.com/privkey.pem;

    location / {
        proxy_pass         http://127.0.0.1:8080;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Real-IP         $remote_addr;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_buffer_size  128k;
        proxy_buffers      4 256k;
        proxy_busy_buffers_size 256k;
    }
}

server {
    listen 80;
    server_name auth.example.com;
    return 301 https://$host$request_uri;
}
Running Keycloak in a container is fine for dev, but for production on kldload the systemd approach gives you direct control over the JVM, clean log management, and the ability to run the database on ZFS with snapshots for backup. The Quarkus build step compiles the configuration into the runtime binary — it is not a traditional "configure at startup" application. Run kc.sh build after any config change, then kc.sh start --optimized.

3. Realm & Client Design

Good realm design prevents the mess that comes later. The rules are simple: one realm per trust boundary, never use the master realm for applications, and name things clearly because client IDs end up in every configuration file in your stack.

Create a realm

# Using the Keycloak admin CLI
/opt/keycloak/bin/kcadm.sh config credentials \
  --server http://localhost:8080 \
  --realm master --user admin --password changeme

# Create the production realm
/opt/keycloak/bin/kcadm.sh create realms \
  -s realm=production \
  -s enabled=true \
  -s displayName="Production Services" \
  -s loginTheme=keycloak \
  -s sslRequired=external

Register OIDC clients

# Confidential client — backend web application (e.g. Grafana)
/opt/keycloak/bin/kcadm.sh create clients -r production \
  -s clientId=grafana \
  -s name="Grafana Dashboard" \
  -s enabled=true \
  -s protocol=openid-connect \
  -s clientAuthenticatorType=client-secret \
  -s secret=GRAFANA_CLIENT_SECRET_HERE \
  -s 'redirectUris=["https://grafana.example.com/login/generic_oauth"]' \
  -s 'webOrigins=["https://grafana.example.com"]' \
  -s directAccessGrantsEnabled=false \
  -s standardFlowEnabled=true

# Public client — single-page application
/opt/keycloak/bin/kcadm.sh create clients -r production \
  -s clientId=dashboard-spa \
  -s name="Dashboard SPA" \
  -s enabled=true \
  -s protocol=openid-connect \
  -s publicClient=true \
  -s 'redirectUris=["https://dashboard.example.com/*"]' \
  -s 'webOrigins=["https://dashboard.example.com"]' \
  -s directAccessGrantsEnabled=false \
  -s standardFlowEnabled=true

# Bearer-only client — API that validates tokens but never initiates login
/opt/keycloak/bin/kcadm.sh create clients -r production \
  -s clientId=api-backend \
  -s name="API Backend" \
  -s enabled=true \
  -s protocol=openid-connect \
  -s bearerOnly=true

Define roles

# Realm-level roles (global)
/opt/keycloak/bin/kcadm.sh create roles -r production \
  -s name=platform-admin -s description="Full platform access"

/opt/keycloak/bin/kcadm.sh create roles -r production \
  -s name=platform-viewer -s description="Read-only platform access"

# Client-level roles (scoped to grafana)
GRAFANA_ID=$(/opt/keycloak/bin/kcadm.sh get clients -r production \
  -q clientId=grafana --fields id --format csv --noquotes)

/opt/keycloak/bin/kcadm.sh create clients/$GRAFANA_ID/roles -r production \
  -s name=admin -s description="Grafana admin"

/opt/keycloak/bin/kcadm.sh create clients/$GRAFANA_ID/roles -r production \
  -s name=editor -s description="Grafana editor"

/opt/keycloak/bin/kcadm.sh create clients/$GRAFANA_ID/roles -r production \
  -s name=viewer -s description="Grafana viewer"
Client roles are more useful than realm roles in practice. A realm role like admin is ambiguous — admin of what? Client roles like grafana:admin and vault:operator are unambiguous. The token carries both, so applications can check whichever scope they care about. Start with client roles. Use realm roles only for cross-cutting concerns like platform-admin that genuinely span every application.

4. The OIDC Authorisation Code Flow

This is the flow used by web applications. The user's browser redirects to Keycloak, the user authenticates, Keycloak redirects back with an authorisation code, and the backend exchanges the code for tokens. The tokens never pass through the browser URL bar.

Step 1 — Redirect to Keycloak

The application redirects the user's browser to Keycloak's authorisation endpoint with its client ID, redirect URI, requested scopes, and a random state parameter (CSRF protection). For public clients, a PKCE code_challenge is also included.

// "Go to the front desk, tell them who sent you, come back with a code."

Step 2 — User authenticates

Keycloak shows the login page (or the federated IdP's login page). The user enters credentials. If MFA is configured, the second factor is prompted. Keycloak validates everything.

// "Show your ID. Now show your fingerprint. You're verified."

Step 3 — Authorisation code returned

Keycloak redirects the browser back to the application's redirect URI with a short-lived authorisation code in the query string: ?code=abc123&state=xyz. The application checks the state matches what it sent.

// A one-time receipt. Useless on its own, but the backend can exchange it for tokens.

Step 4 — Token exchange

The backend makes a server-to-server POST to Keycloak's token endpoint, sending the authorisation code and its client secret (or PKCE verifier). Keycloak returns the access token, refresh token, and ID token. The browser never sees the tokens — they stay server-side.

// "Here's the receipt and my ID. Give me the actual badge." — happens entirely server-side.

Token anatomy

# Decode an access token (it's a JWT — three base64 parts separated by dots)
echo "$ACCESS_TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq .

# Typical payload:
{
  "exp": 1712345678,
  "iat": 1712345378,
  "jti": "a1b2c3d4-...",
  "iss": "https://auth.example.com/realms/production",
  "sub": "user-uuid-here",
  "typ": "Bearer",
  "azp": "grafana",
  "scope": "openid email profile",
  "realm_access": {
    "roles": ["platform-viewer"]
  },
  "resource_access": {
    "grafana": {
      "roles": ["editor"]
    }
  },
  "preferred_username": "alice",
  "email": "alice@example.com"
}

Validate tokens in your application

# Fetch the realm's public keys (JWKS endpoint)
curl -s https://auth.example.com/realms/production/protocol/openid-connect/certs | jq .

# In nginx — use lua-resty-openidc or oauth2-proxy
# In Go — use coreos/go-oidc
# In Python — use python-jose or PyJWT with the JWKS
# In Node — use jose or jsonwebtoken with JWKS

# The pattern: download the JWKS once, cache it, verify the JWT signature
# on every request. If the signature is valid and exp > now, the token is trusted.
The critical security property of OIDC is that your application validates tokens locally using the realm's public key. It never calls Keycloak on every request. This means Keycloak can go down after login and your applications keep working until tokens expire. It also means token validation is fast — it is a local RSA/EC signature check, not a network call. The flip side is that you cannot revoke an access token before it expires. Keep access token lifetimes short (5–15 minutes) and use refresh tokens for session continuity.

5. SAML 2.0 for Legacy Applications

SAML is XML-based and uses browser redirects with signed XML assertions. You will encounter it when integrating with enterprise applications that predate OIDC: older Java apps, on-premises Microsoft products, some government systems.

Register a SAML client

# Create a SAML client for a legacy application
/opt/keycloak/bin/kcadm.sh create clients -r production \
  -s clientId=https://legacy-app.example.com/saml \
  -s name="Legacy App" \
  -s enabled=true \
  -s protocol=saml \
  -s 'attributes={"saml.server.signature":"true","saml.assertion.signature":"true","saml.client.signature":"false","saml_force_name_id_format":"true","saml_name_id_format":"username"}' \
  -s 'redirectUris=["https://legacy-app.example.com/saml/acs"]' \
  -s frontchannelLogout=true \
  -s 'attributes.saml_single_logout_service_url_redirect=https://legacy-app.example.com/saml/slo'

SAML metadata exchange

# Download Keycloak's SAML metadata for the realm
curl -o keycloak-idp-metadata.xml \
  https://auth.example.com/realms/production/protocol/saml/descriptor

# Upload this to the legacy application's "Identity Provider" configuration

# Download the legacy app's SP metadata (if it provides one)
# and import it into Keycloak via Admin Console > Clients > Import
SAML Concept OIDC Equivalent Purpose
Assertion ID Token Signed statement about user identity
Service Provider (SP) Client (Relying Party) The application requesting authentication
Identity Provider (IdP) OpenID Provider (OP) Keycloak — the thing that authenticates users
NameID sub claim Unique user identifier
Attribute Statement Claims in JWT User attributes (email, roles, groups)
ACS URL Redirect URI Where the response goes after authentication
If you have the choice, always choose OIDC over SAML. SAML assertions are verbose XML documents that are easy to misconfigure and hard to debug. The only reason to use SAML is when the application you are integrating with only supports SAML. Keycloak handles both protocols for the same user base, so you can have some clients using OIDC and others using SAML in the same realm.

6. User Federation — LDAP & Active Directory

User federation lets Keycloak authenticate users against an external directory without importing their passwords. The directory remains the source of truth for credentials. Keycloak syncs user attributes, group memberships, and role mappings.

Configure LDAP federation

# Via kcadm (or use the Admin Console UI)
/opt/keycloak/bin/kcadm.sh create components -r production \
  -s name="Corporate LDAP" \
  -s providerId=ldap \
  -s providerType=org.keycloak.storage.UserStorageProvider \
  -s 'config.vendor=["ad"]' \
  -s 'config.connectionUrl=["ldaps://dc01.corp.example.com:636"]' \
  -s 'config.bindDn=["CN=keycloak-svc,OU=Service Accounts,DC=corp,DC=example,DC=com"]' \
  -s 'config.bindCredential=["SERVICE_ACCOUNT_PASSWORD"]' \
  -s 'config.usersDn=["OU=Users,DC=corp,DC=example,DC=com"]' \
  -s 'config.userObjectClasses=["person, organizationalPerson, user"]' \
  -s 'config.usernameAttribute=["sAMAccountName"]' \
  -s 'config.rdnAttribute=["cn"]' \
  -s 'config.uuidAttribute=["objectGUID"]' \
  -s 'config.editMode=["READ_ONLY"]' \
  -s 'config.syncRegistrations=["false"]' \
  -s 'config.searchScope=["2"]' \
  -s 'config.pagination=["true"]' \
  -s 'config.importEnabled=["true"]' \
  -s 'config.batchSizeForSync=["1000"]'

Group and role mapping

# Map LDAP groups to Keycloak roles
# In Admin Console: User Federation > Corporate LDAP > Mappers > Add
# Or via CLI:

# Create a group-ldap-mapper
LDAP_ID=$(/opt/keycloak/bin/kcadm.sh get components -r production \
  -q name="Corporate LDAP" --fields id --format csv --noquotes)

/opt/keycloak/bin/kcadm.sh create components -r production \
  -s name="group-mapper" \
  -s providerId=group-ldap-mapper \
  -s providerType=org.keycloak.storage.ldap.mappers.LDAPStorageMapper \
  -s parentId=$LDAP_ID \
  -s 'config.groups.dn=["OU=Groups,DC=corp,DC=example,DC=com"]' \
  -s 'config.group.name.ldap.attribute=["cn"]' \
  -s 'config.group.object.classes=["group"]' \
  -s 'config.membership.ldap.attribute=["member"]' \
  -s 'config.membership.attribute.type=["DN"]' \
  -s 'config.mode=["READ_ONLY"]' \
  -s 'config.drop.non.existing.groups.during.sync=["false"]'

# Trigger a full sync
/opt/keycloak/bin/kcadm.sh create user-storage/$LDAP_ID/sync \
  -r production -s action=triggerFullSync
The biggest gotcha with LDAP federation is edit mode. READ_ONLY means Keycloak reads from LDAP but cannot write back — password changes must happen in AD. WRITABLE means Keycloak can push password changes to LDAP. UNSYNCED means Keycloak stores a local copy that can diverge from LDAP. For Active Directory, use READ_ONLY and let AD remain the password authority. Keycloak will validate passwords by binding to LDAP with the user's credentials at login time.

7. Multi-Factor Authentication

Keycloak supports TOTP (Google Authenticator, FreeOTP), WebAuthn (hardware keys like YubiKey), and OTP via email. MFA is configured per-realm as an authentication flow.

Enable TOTP for all users

# Set the browser flow to require OTP
# Admin Console > Authentication > Flows > browser > OTP Form > Required

# Or via CLI — configure the realm to require OTP
/opt/keycloak/bin/kcadm.sh update realms/production \
  -s 'otpPolicyType=totp' \
  -s 'otpPolicyAlgorithm=HmacSHA256' \
  -s 'otpPolicyDigits=6' \
  -s 'otpPolicyPeriod=30' \
  -s 'otpPolicyInitialCounter=0'

WebAuthn (FIDO2 hardware keys)

# Admin Console > Authentication > Required Actions > Enable "Webauthn Register"
# Admin Console > Authentication > Flows > browser
#   Add "WebAuthn Authenticator" as an alternative to OTP

# Realm settings for WebAuthn
/opt/keycloak/bin/kcadm.sh update realms/production \
  -s 'webAuthnPolicyRpEntityName=kldload-production' \
  -s 'webAuthnPolicySignatureAlgorithms=["ES256","RS256"]' \
  -s 'webAuthnPolicyAttestationConveyancePreference=none' \
  -s 'webAuthnPolicyAuthenticatorAttachment=cross-platform' \
  -s 'webAuthnPolicyRequireResidentKey=No' \
  -s 'webAuthnPolicyUserVerificationRequirement=preferred'

Conditional MFA — require for admins only

# Create a custom authentication flow:
# 1. Duplicate the "browser" flow as "browser-conditional-mfa"
# 2. Add a "Conditional OTP" sub-flow
# 3. Add a "Condition - User Role" execution
#    - Set realm role = "platform-admin"
# 4. Add "OTP Form" execution after the condition
# 5. Bind the new flow as the realm's browser flow

# Result: regular users get username/password only.
# Users with platform-admin role must also provide OTP.
WebAuthn with hardware keys is the gold standard for MFA — it is phishing-resistant because the browser binds the credential to the origin. TOTP is vulnerable to real-time phishing (the attacker relays the OTP). For platform administrators, require WebAuthn. For regular users, TOTP is acceptable. The conditional MFA pattern is powerful: you can require stronger authentication for sensitive roles without annoying every user.

8. Practical SSO — Grafana, Vault, Kubernetes

Grafana OIDC

# /etc/grafana/grafana.ini
[auth.generic_oauth]
enabled = true
name = Keycloak
allow_sign_up = true
client_id = grafana
client_secret = GRAFANA_CLIENT_SECRET_HERE
scopes = openid email profile
auth_url = https://auth.example.com/realms/production/protocol/openid-connect/auth
token_url = https://auth.example.com/realms/production/protocol/openid-connect/token
api_url = https://auth.example.com/realms/production/protocol/openid-connect/userinfo
role_attribute_path = contains(resource_access.grafana.roles[*], 'admin') && 'Admin' || contains(resource_access.grafana.roles[*], 'editor') && 'Editor' || 'Viewer'
email_attribute_path = email
login_attribute_path = preferred_username
name_attribute_path = name

HashiCorp Vault OIDC

# Enable the OIDC auth method
vault auth enable oidc

# Configure Vault to use Keycloak
vault write auth/oidc/config \
  oidc_discovery_url="https://auth.example.com/realms/production" \
  oidc_client_id="vault" \
  oidc_client_secret="VAULT_CLIENT_SECRET_HERE" \
  default_role="default"

# Create a role mapping
vault write auth/oidc/role/default \
  bound_audiences="vault" \
  allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
  allowed_redirect_uris="http://localhost:8250/oidc/callback" \
  user_claim="preferred_username" \
  groups_claim="resource_access.vault.roles" \
  policies="default" \
  ttl=1h

Kubernetes OIDC

# kube-apiserver flags (add to /etc/kubernetes/manifests/kube-apiserver.yaml)
--oidc-issuer-url=https://auth.example.com/realms/production
--oidc-client-id=kubernetes
--oidc-username-claim=preferred_username
--oidc-username-prefix=oidc:
--oidc-groups-claim=groups
--oidc-groups-prefix=oidc:
--oidc-ca-file=/etc/kubernetes/pki/keycloak-ca.pem

# RBAC binding for a Keycloak group
kubectl create clusterrolebinding keycloak-admins \
  --clusterrole=cluster-admin \
  --group=oidc:platform-admin
# kubelogin — kubectl plugin for OIDC login
kubectl oidc-login setup \
  --oidc-issuer-url=https://auth.example.com/realms/production \
  --oidc-client-id=kubernetes \
  --oidc-client-secret=K8S_CLIENT_SECRET

# Add to kubeconfig
kubectl config set-credentials oidc-user \
  --exec-api-version=client.authentication.k8s.io/v1beta1 \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg=--oidc-issuer-url=https://auth.example.com/realms/production \
  --exec-arg=--oidc-client-id=kubernetes \
  --exec-arg=--oidc-client-secret=K8S_CLIENT_SECRET
Once Keycloak is running, adding SSO to a new application is a 15-minute task: register a client, configure the application with the client ID and Keycloak URLs, map roles. The three examples above — Grafana, Vault, Kubernetes — cover the three most common patterns: web app with role mapping, secrets manager with policy binding, and API server with RBAC groups. Every other integration follows the same pattern. The real work is in the realm and role design, not the integration.

9. SELinux Fundamentals

SELinux (Security-Enhanced Linux) is a mandatory access control (MAC) system built into the Linux kernel. Unlike traditional Unix permissions (discretionary access control — DAC), SELinux policies are enforced by the kernel and cannot be overridden by root. Even if an attacker compromises a service and escalates to root, SELinux confines the process to its labelled domain and prevents it from accessing anything outside that domain.

DAC vs MAC

DAC (traditional Unix): the file owner decides who can access it. Root bypasses everything. MAC (SELinux): the kernel enforces policy regardless of user. Root running as httpd_t can only access objects labelled for httpd_t. The policy is the law, not the user.

// DAC = the homeowner decides who enters. MAC = the building code decides what every room can be used for.

Labels (Security Contexts)

Every process, file, port, and socket has a label: user:role:type:level. For files: system_u:object_r:httpd_sys_content_t:s0. For processes: system_u:system_r:httpd_t:s0. The type field is what matters most — SELinux type enforcement (TE) is the primary access control mechanism.

// Every object has a name tag. The policy says which name tags can talk to which other name tags.

Type Enforcement (TE)

The core of SELinux. Processes run in a domain (type). Files have a type. The policy defines which domains can access which types, and how (read, write, execute, etc.). If the policy does not explicitly allow an access, it is denied.

// Default deny. If it's not in the allow rules, it doesn't happen.

Modes: Enforcing, Permissive, Disabled

Enforcing: policy is applied, violations are blocked and logged. Permissive: policy violations are logged but not blocked (for debugging). Disabled: SELinux is off entirely. Production systems must run enforcing. Permissive is for troubleshooting only.

// Enforcing = the guard stops you. Permissive = the guard writes your name down but lets you through.

Booleans

On/off switches for specific policy rules. httpd_can_network_connect controls whether Apache can make outbound network connections. Booleans let you tune policy without writing custom modules. There are hundreds of them.

// "Should the web server be allowed to connect to the database?" Flip the boolean, no custom policy needed.

Policy Type: Targeted

RHEL/CentOS/Rocky use the targeted policy: only specific daemons are confined. Everything else runs as unconfined_t (unrestricted). This is a pragmatic middle ground — critical services like httpd, sshd, named are confined; user sessions are not.

// Not every room has a lock. The targeted policy locks the rooms that matter most.
The mental model for SELinux is a whitelist, not a blacklist. Nothing is allowed unless the policy explicitly permits it. This is why it breaks things — a new application has no policy rules, so every access is denied. The fix is not to disable SELinux; it is to write or adjust the rules. The tooling (audit2allow, audit2why, semanage) makes this routine. The reason SELinux has a bad reputation is that people encounter it only when it blocks something, and they do not know the tools to fix it. This section teaches those tools.

10. SELinux Status & Configuration

# Check current mode
getenforce
# Enforcing

# Check detailed status
sestatus
# SELinux status:                 enabled
# SELinuxfs mount:                /sys/fs/selinux
# SELinux root directory:         /etc/selinux
# Loaded policy name:             targeted
# Current mode:                   enforcing
# Mode from config file:          enforcing
# Policy MLS status:              enabled
# Policy deny_unknown status:     allowed
# Memory protection checking:     actual (secure)
# Max kernel policy version:      33

# Temporarily switch to permissive (for debugging — reverts on reboot)
setenforce 0

# Back to enforcing
setenforce 1

# Permanent configuration
cat /etc/selinux/config
# SELINUX=enforcing
# SELINUXTYPE=targeted

View labels on everything

# File labels
ls -Z /var/www/html/
# system_u:object_r:httpd_sys_content_t:s0 index.html

# Process labels
ps auxZ | grep httpd
# system_u:system_r:httpd_t:s0  root  12345 ... /usr/sbin/httpd

# Port labels
semanage port -l | grep http
# http_port_t     tcp   80, 443, 488, 8008, 8009, 8443, 9000

# User mappings
semanage login -l
# Login Name       SELinux User     MLS/MCS Range
# root             unconfined_u     s0-s0:c0.c1023
# __default__      unconfined_u     s0-s0:c0.c1023

11. File Contexts & Relabelling

When SELinux blocks access, the most common cause is wrong file labels. Files get the wrong label when you copy them (cp preserves labels from the source), move them from a different directory, or create them in a non-standard location.

Check and fix file contexts

# See what the policy expects for a path
matchpathcon /var/www/html
# /var/www/html    system_u:object_r:httpd_sys_content_t:s0

# See the actual label
ls -Z /var/www/html
# If it says httpd_sys_content_t — it's correct
# If it says default_t or something else — relabel it

# Restore default context for a directory tree
restorecon -Rv /var/www/html/
# Relabeled /var/www/html from unconfined_u:object_r:default_t:s0
#   to system_u:object_r:httpd_sys_content_t:s0

# Restore the entire filesystem (after major changes — takes time)
touch /.autorelabel
reboot

Add custom file context rules

# Tell SELinux that /srv/myapp/static should have httpd content labels
semanage fcontext -a -t httpd_sys_content_t '/srv/myapp/static(/.*)?'
restorecon -Rv /srv/myapp/static/

# Tell SELinux that /opt/keycloak/data should be writable by Java
semanage fcontext -a -t var_lib_t '/opt/keycloak/data(/.*)?'
restorecon -Rv /opt/keycloak/data/

# List all custom file context rules
semanage fcontext -l -C
The number one SELinux troubleshooting pattern: something breaks, you check ls -Z, the label is wrong, you run restorecon, it works. This covers 60–70% of all SELinux issues. The second most common: a boolean needs to be flipped. The third: a port needs to be labelled. Custom policy modules are the last resort, not the first.

12. Booleans — Tuning Policy Without Writing Policy

# List all booleans (there are hundreds)
getsebool -a

# Search for relevant booleans
getsebool -a | grep httpd
# httpd_can_network_connect --> off
# httpd_can_network_connect_db --> off
# httpd_can_sendmail --> off
# httpd_enable_cgi --> on
# httpd_enable_homedirs --> off
# ...

# Enable a boolean (persistent across reboots)
setsebool -P httpd_can_network_connect on

# Enable temporarily (reverts on reboot — for testing)
setsebool httpd_can_network_connect on

# Common booleans you will need:
setsebool -P httpd_can_network_connect on       # nginx/Apache reverse proxy
setsebool -P httpd_can_network_connect_db on     # web app to database
setsebool -P httpd_can_network_relay on          # proxy/load balancer
setsebool -P nis_enabled on                       # NIS/LDAP client lookups
setsebool -P domain_can_mmap_files on            # JVM applications (Keycloak)
setsebool -P daemons_enable_cluster_mode on      # Pacemaker/Corosync

Booleans for Keycloak specifically

# Keycloak (Java) needs:
setsebool -P httpd_can_network_connect on        # if proxied via httpd
setsebool -P domain_can_mmap_files on            # JVM memory mapping

# If Keycloak connects to an external LDAP:
setsebool -P authlogin_nsswitch_use_ldap on

# If Keycloak sends email (password reset):
setsebool -P httpd_can_sendmail on
Booleans are the SELinux equivalent of configuration flags. They exist because the policy authors anticipated common needs — "web servers sometimes need to connect to databases" — and pre-wrote the allow rules behind a toggle. Before writing a custom policy module, always check if a boolean already covers your use case. getsebool -a | grep $keyword is the fastest diagnostic you have.

13. Port Labelling

SELinux controls which services can bind to which ports. If you run a service on a non-standard port, SELinux will block it unless you label that port for the service's domain.

# See current port labels
semanage port -l | grep -E 'http|ssh|postgres'
# http_port_t                    tcp      80, 443, 488, 8008, 8009, 8443, 9000
# ssh_port_t                     tcp      22
# postgresql_port_t              tcp      5432

# Allow nginx to bind to port 3000
semanage port -a -t http_port_t -p tcp 3000

# Allow SSH on a non-standard port
semanage port -a -t ssh_port_t -p tcp 2222

# Allow Keycloak on port 8080 (already http_port_t? check first)
semanage port -l | grep 8080
# If not listed, add it:
semanage port -a -t http_port_t -p tcp 8080

# Remove a custom port label
semanage port -d -t http_port_t -p tcp 3000

# If the port is already defined for another type, use -m (modify)
semanage port -m -t http_port_t -p tcp 8888

14. Troubleshooting SELinux Denials

When SELinux blocks something, it writes an AVC (Access Vector Cache) denial to the audit log. The troubleshooting workflow is always the same: find the denial, understand it, fix it.

Step 1 — Find the denial

# Recent AVC denials
ausearch -m avc -ts recent

# Denials for a specific service
ausearch -m avc -c httpd -ts today

# Tail the audit log in real time
tail -f /var/log/audit/audit.log | grep denied

# Using sealert (if setroubleshoot is installed)
sealert -a /var/log/audit/audit.log

Step 2 — Understand the denial

# Example AVC denial:
# type=AVC msg=audit(1712345678.123:456): avc: denied { name_connect }
#   for pid=1234 comm="nginx" dest=8080
#   scontext=system_u:system_r:httpd_t:s0
#   tcontext=system_u:object_r:http_port_t:s0
#   tclass=tcp_socket permissive=0

# Translation: nginx (httpd_t) tried to connect (name_connect)
# to port 8080 (http_port_t) and was denied.

# Use audit2why to get a human-readable explanation
ausearch -m avc -ts recent | audit2why
# Was caused by: The boolean httpd_can_network_connect was set to off.
# Fix: setsebool -P httpd_can_network_connect on

Step 3 — Fix it (in order of preference)

# 1. Fix the file label (most common)
restorecon -Rv /path/to/files

# 2. Flip a boolean
setsebool -P httpd_can_network_connect on

# 3. Label a port
semanage port -a -t http_port_t -p tcp 3000

# 4. Write a custom policy module (last resort)
ausearch -m avc -ts recent | audit2allow -M my_custom_policy
semodule -i my_custom_policy.pp
The workflow is mechanical: find the AVC, read it, fix it. audit2why tells you exactly what to do in most cases. The reason people get frustrated is they skip step 2 and go straight to setenforce 0. That is like turning off the fire alarm instead of finding the fire. The audit log tells you exactly what was denied, what process tried it, and what the fix is. Read it.

15. Writing Custom SELinux Policy Modules

When no boolean, file context, or port label fixes the issue, you need a custom policy module. The audit2allow tool generates one from AVC denials. The trick is to generate a minimal module that allows only what is needed, not a blanket permission.

Generate a module from denials

# Collect all recent denials for a service
ausearch -m avc -c keycloak -ts today > /tmp/keycloak-denials.txt

# Generate a human-readable policy
cat /tmp/keycloak-denials.txt | audit2allow
# #============= java_t ==============
# allow java_t self:netlink_route_socket { bind create getattr nlmsg_read read };
# allow java_t http_port_t:tcp_socket name_connect;

# Generate a compilable module
cat /tmp/keycloak-denials.txt | audit2allow -M keycloak_custom
# ******************** IMPORTANT ***********************
# To make this policy package active, execute:
# semodule -i keycloak_custom.pp

# Review the .te file before installing
cat keycloak_custom.te
# module keycloak_custom 1.0;
# require {
#     type java_t;
#     type http_port_t;
#     class tcp_socket name_connect;
#     class netlink_route_socket { bind create getattr nlmsg_read read };
# }
# #============= java_t ==============
# allow java_t self:netlink_route_socket { bind create getattr nlmsg_read read };
# allow java_t http_port_t:tcp_socket name_connect;

# Install the module
semodule -i keycloak_custom.pp

# Verify it is loaded
semodule -l | grep keycloak_custom

Write a module from scratch

# /root/selinux-modules/myapp.te
policy_module(myapp, 1.0)

# Declare types
type myapp_t;
type myapp_exec_t;
type myapp_data_t;

# Allow init to start the service
init_daemon_domain(myapp_t, myapp_exec_t)

# Networking
allow myapp_t self:tcp_socket { create bind listen accept read write getattr };
corenet_tcp_bind_generic_port(myapp_t)
corenet_tcp_connect_http_port(myapp_t)

# File access
allow myapp_t myapp_data_t:dir { read write add_name remove_name search };
allow myapp_t myapp_data_t:file { create read write unlink getattr open };

# Logging
logging_send_syslog_msg(myapp_t)

# DNS resolution
sysnet_dns_name_resolve(myapp_t)
# /root/selinux-modules/myapp.fc (file contexts)
/opt/myapp/bin/myapp    --  gen_context(system_u:object_r:myapp_exec_t,s0)
/opt/myapp/data(/.*)?       gen_context(system_u:object_r:myapp_data_t,s0)
# Compile and install
make -f /usr/share/selinux/devel/Makefile myapp.pp
semodule -i myapp.pp
restorecon -Rv /opt/myapp/

Manage modules

# List all installed modules
semodule -l

# Remove a module
semodule -r keycloak_custom

# Disable a module (keeps it installed)
semodule -d keycloak_custom

# Re-enable a disabled module
semodule -e keycloak_custom

# Check module priority (higher priority wins)
semodule -l --all | head
The audit2allow approach is quick but can be overly permissive — it allows everything that was denied during your test session, which might include things the service should not actually do. Always review the generated .te file. For production services, writing a module from scratch gives you tighter control. The selinux-policy-devel package provides macros (init_daemon_domain, corenet_tcp_bind_generic_port, etc.) that are the correct way to grant access — they handle the underlying allow rules and transitions that a raw allow statement might miss.

16. Multi-Category Security (MCS)

MCS extends type enforcement with categories. Two processes can have the same type but different categories — they cannot access each other's files. This is how container runtimes isolate containers under SELinux: each container gets a unique category pair.

# Category labels look like: s0:c1,c2
# A process with s0:c1,c2 can access files with s0:c1,c2 but NOT s0:c3,c4

# See a container's category
ps auxZ | grep container
# system_u:system_r:container_t:s0:c123,c456  root ... /usr/bin/myapp

# Files for that container:
ls -Z /var/lib/containers/storage/overlay/.../merged/
# system_u:object_r:container_file_t:s0:c123,c456

# Podman assigns unique categories automatically
podman run --security-opt label=level:s0:c200,c300 ...

# Disable SELinux for a specific container (NOT recommended)
podman run --security-opt label=disable ...

MCS for multi-tenant services

# Assign categories to users for multi-tenant isolation
semanage login -a -s user_u -r s0:c10,c20 tenant_alice
semanage login -a -s user_u -r s0:c30,c40 tenant_bob

# Alice's processes get s0:c10,c20 — they can only access files with s0:c10,c20
# Bob's processes get s0:c30,c40 — complete isolation at the kernel level

# This is useful for shared hosting, multi-tenant databases, or any scenario
# where multiple untrusted workloads share a kernel
MCS is how SELinux makes containers actually isolated at the kernel level, not just at the namespace level. Namespaces are a userspace abstraction — a kernel vulnerability can break out of them. SELinux MCS categories are enforced by the kernel's security module — a container with categories c123,c456 physically cannot read files labelled c789,c1000, even if it escapes the namespace. This is defense in depth that most people do not realise they are getting when they run Podman or CRI-O with SELinux enabled.

17. SELinux Policy for Keycloak

Running Keycloak under SELinux requires a few adjustments. Java applications run in the java_t domain by default, which is quite restricted.

Complete SELinux setup for Keycloak

# 1. Label the Keycloak directories
semanage fcontext -a -t java_exec_t '/opt/keycloak/bin(/.*)?'
semanage fcontext -a -t var_lib_t '/opt/keycloak/data(/.*)?'
semanage fcontext -a -t etc_t '/opt/keycloak/conf(/.*)?'
semanage fcontext -a -t var_log_t '/var/log/keycloak(/.*)?'
restorecon -Rv /opt/keycloak /var/log/keycloak

# 2. Set booleans
setsebool -P domain_can_mmap_files on
setsebool -P httpd_can_network_connect on

# 3. Label non-standard ports if needed
semanage port -a -t http_port_t -p tcp 8443   # HTTPS management
# 8080 is usually already http_port_t

# 4. Run Keycloak in permissive temporarily to collect denials
semanage permissive -a java_t

# 5. Exercise Keycloak (login, create realm, LDAP sync, etc.)
# 6. Generate custom module from collected denials
ausearch -m avc -c java -ts today | audit2allow -M keycloak_policy

# 7. Review the .te file, remove anything overly broad
cat keycloak_policy.te

# 8. Install and switch back to enforcing
semodule -i keycloak_policy.pp
semanage permissive -d java_t

# 9. Verify Keycloak works in enforcing mode
systemctl restart keycloak
curl -s https://auth.example.com/realms/production/.well-known/openid-configuration | jq .issuer
The semanage permissive -a java_t trick is the professional approach: it puts only the java_t domain into permissive mode while the rest of the system stays enforcing. This lets you collect all the denials Keycloak generates during normal operation without weakening security for anything else. Once you have the custom module, switch java_t back to enforcing. This is how RHEL consultants do it.

18. SELinux for Containers

# Podman and CRI-O are SELinux-aware by default
# Every container runs with a unique MCS label

# Verify container SELinux is working
podman run --rm fedora:latest cat /proc/1/attr/current
# system_u:system_r:container_t:s0:c123,c456

# Bind mounts need the :Z or :z flag
# :Z = private label (only this container can access)
# :z = shared label (multiple containers can access)
podman run -v /data/myapp:/app:Z myimage

# Without :Z, SELinux blocks the container from reading the mount
# because the host files have host labels, not container labels

# For ZFS datasets mounted into containers:
podman run -v /tank/app-data:/data:Z --security-opt label=type:container_file_t myimage

Kubernetes with SELinux

# Pod security context with SELinux options
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
spec:
  securityContext:
    seLinuxOptions:
      level: "s0:c123,c456"
      type: "container_t"
  containers:
  - name: app
    image: myapp:latest
    securityContext:
      seLinuxOptions:
        level: "s0:c123,c456"
# SELinux with Cilium — Cilium's eBPF programs need:
setsebool -P domain_kernel_load_modules on
# Or use a custom module for cilium_t if defined

# Check for SELinux denials from Cilium
ausearch -m avc -c cilium -ts today

19. Production Checklist

Item Keycloak SELinux
TLS All endpoints HTTPS. hostname-strict=true. Certificate files labelled cert_t.
Database PostgreSQL with TLS. Separate credentials per environment. postgresql_t domain. Data on ZFS labelled correctly.
Secrets Client secrets in Vault. Admin password rotated. Config files etc_t. No world-readable secrets.
Tokens Access token lifetime ≤15 min. Refresh token ≤8 hours. N/A
MFA WebAuthn for admins. TOTP for all users minimum. N/A
Backup PostgreSQL on ZFS → snapshot + replicate. Custom modules in git. semanage export in backups.
Monitoring Prometheus metrics at /metrics. Alert on login failures. Alert on AVC denials. ausearch -m avc in cron.
Mode Production build (kc.sh build + start --optimized). Enforcing. No permissive domains. No disabled.

Export and import SELinux customisations

# Export all local SELinux customisations (for backup or replication)
semanage export > /root/selinux-customisations.txt

# Import on another node
semanage import < /root/selinux-customisations.txt

# Example export output:
# boolean -m -1 httpd_can_network_connect
# port -a -t http_port_t -p tcp 3000
# fcontext -a -t httpd_sys_content_t '/srv/myapp/static(/.*)?'
The semanage export/import pattern is gold for infrastructure-as-code. Export your SELinux customisations on a working node, commit the file to git, and apply it to new nodes via your postinstaller. Combined with Keycloak's realm export (kc.sh export --dir /tmp/realms), you can reproduce your entire authentication and access control stack from a clean kldload install.

20. Quick Reference — Troubleshooting Table

Symptom Likely Cause Fix
403 Forbidden from nginx/httpd Wrong file context on web root restorecon -Rv /var/www/html
nginx cannot connect to upstream httpd_can_network_connect is off setsebool -P httpd_can_network_connect on
Service won't bind to port Port not labelled for service type semanage port -a -t TYPE -p tcp PORT
Container can't read bind mount Missing :Z on volume mount podman run -v /path:/path:Z ...
Keycloak OIDC redirect loop Incorrect redirectUris on client Check client config, must match app exactly
Token validation fails Clock skew between app and Keycloak chronyc tracking — fix NTP
LDAP sync fails Firewall or SELinux blocking LDAPS 636 semanage port -a -t ldap_port_t -p tcp 636
SAML assertion rejected Mismatched Entity ID or ACS URL Compare SP metadata with client config
Files created with wrong label Created in non-standard path semanage fcontext -a + restorecon

Related pages