Keycloak & SELinux Masterclass
This guide covers two pillars of enterprise-grade authentication and access control: Keycloak for centralised identity management — SSO, OIDC, SAML, realm design, client registration, token lifecycle, and federation — and SELinux for mandatory access control at the kernel level: policies, contexts, booleans, custom modules, multi-category security, and the real-world troubleshooting workflow that keeps enforcing mode on in production. Together they answer the question every serious infrastructure asks: who are you, and what are you allowed to touch?
The premise: Authentication (who you are) and authorisation (what you can do) are different problems that most teams solve with the same tool — usually an LDAP directory or a flat user database. Keycloak gives you a proper identity layer: single sign-on, token-based auth, federated identity, fine-grained consent. SELinux gives you a proper enforcement layer: even if an attacker gets root, mandatory access control limits what root can touch. One protects the front door. The other protects every room inside.
What this page covers: Keycloak architecture and deployment, realm and client design, OIDC and SAML flows, token anatomy and validation, user federation (LDAP/AD), role-based and attribute-based access, reverse proxy integration, Kubernetes SSO. Then: SELinux fundamentals, policy types, file contexts, booleans, port labelling, custom policy modules with audit2allow, MCS/MLS, container SELinux, and the complete troubleshooting playbook. All grounded in the kldload stack.
Prerequisites: a running kldload system. The Kubernetes sections assume a cluster from the Kubernetes on KVM guide. TLS certificates from the TLS & PKI Masterclass.
1. Keycloak Fundamentals
Keycloak is an open-source identity and access management server. It handles user authentication, token issuance, single sign-on, identity brokering, and user federation. Your applications never see passwords — they receive cryptographically signed tokens from Keycloak and validate them locally.
Realm
A realm is an isolated authentication domain. Each realm has its own users, clients, roles, and identity providers. The master realm is for Keycloak administration only — never put application users in it. Create one realm per trust boundary: production, staging, corporate.
Client
A client is any application that delegates authentication to Keycloak. Web apps, CLI tools, API gateways, mobile apps — each registers as a client in a realm. Clients have a type: confidential (has a secret, e.g. backend API), public (no secret, e.g. SPA or mobile app), or bearer-only (validates tokens but never initiates login).
User & Role
Users live in a realm. They can have realm roles (global to the realm) or client roles (scoped to a single application). Roles are embedded in tokens as claims. Applications read claims to make authorisation decisions — Keycloak never tells an app what to do, it tells the app who the user is and what roles they hold.
Identity Provider (IdP)
Keycloak can broker authentication to external providers: Google, GitHub, SAML-based corporate IdPs, LDAP directories, or another Keycloak instance. Users authenticate at the external IdP and Keycloak maps their identity into the local realm. This is how you get "Login with GitHub" or federate with Active Directory.
Token
Keycloak issues three tokens: an access token (short-lived, sent with API requests), a refresh token (longer-lived, used to get new access tokens without re-login), and an ID token (contains user profile claims for the client). All are JWTs signed with the realm's RSA or EC key.
Protocol: OIDC vs SAML
OpenID Connect (OIDC) is built on OAuth 2.0 and uses JSON/JWT. SAML 2.0 is XML-based. Use OIDC for everything new — it is simpler, faster, and better supported. SAML exists for legacy enterprise apps that predate OIDC. Keycloak supports both natively.
2. Deploying Keycloak on kldload
Keycloak runs as a Java application (Quarkus runtime since v17+). It needs a database for persistence (PostgreSQL recommended), TLS termination, and a reverse proxy in front.
Install on bare metal (systemd)
# Download Keycloak (check for latest version)
KC_VERSION=24.0.4
curl -LO https://github.com/keycloak/keycloak/releases/download/${KC_VERSION}/keycloak-${KC_VERSION}.tar.gz
tar xzf keycloak-${KC_VERSION}.tar.gz -C /opt/
ln -sfn /opt/keycloak-${KC_VERSION} /opt/keycloak
# Create a dedicated system user
useradd -r -s /sbin/nologin -d /opt/keycloak keycloak
chown -R keycloak:keycloak /opt/keycloak
# Create the PostgreSQL database
sudo -u postgres psql -c "CREATE USER keycloak WITH PASSWORD 'changeme';"
sudo -u postgres psql -c "CREATE DATABASE keycloak OWNER keycloak;"
Configure Keycloak
# /opt/keycloak/conf/keycloak.conf
# Database
db=postgres
db-url=jdbc:postgresql://localhost:5432/keycloak
db-username=keycloak
db-password=changeme
# Hostname — the public URL users see
hostname=auth.example.com
hostname-strict=true
# HTTP — Keycloak listens on 8080, TLS terminated by nginx
http-enabled=true
http-port=8080
proxy-headers=xforwarded
# Metrics and health (for monitoring)
health-enabled=true
metrics-enabled=true
# Logging
log=console,file
log-file=/var/log/keycloak/keycloak.log
log-level=INFO
Build and start
# Build the optimised runtime (Quarkus ahead-of-time compilation)
/opt/keycloak/bin/kc.sh build
# Bootstrap admin user (first run only)
KEYCLOAK_ADMIN=admin KEYCLOAK_ADMIN_PASSWORD=changeme \
/opt/keycloak/bin/kc.sh start
systemd unit
# /etc/systemd/system/keycloak.service
[Unit]
Description=Keycloak Identity Server
After=network-online.target postgresql.service
Wants=network-online.target
[Service]
Type=exec
User=keycloak
Group=keycloak
ExecStart=/opt/keycloak/bin/kc.sh start --optimized
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
# Hardening
ProtectSystem=strict
ReadWritePaths=/opt/keycloak /var/log/keycloak
PrivateTmp=true
NoNewPrivileges=true
[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now keycloak
systemctl status keycloak
Reverse proxy with nginx
# /etc/nginx/conf.d/keycloak.conf
server {
listen 443 ssl http2;
server_name auth.example.com;
ssl_certificate /etc/letsencrypt/live/auth.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/auth.example.com/privkey.pem;
location / {
proxy_pass http://127.0.0.1:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
}
}
server {
listen 80;
server_name auth.example.com;
return 301 https://$host$request_uri;
}
kc.sh build after any config change, then kc.sh start --optimized.3. Realm & Client Design
Good realm design prevents the mess that comes later. The rules are simple: one realm per trust boundary, never use the master realm for applications, and name things clearly because client IDs end up in every configuration file in your stack.
Create a realm
# Using the Keycloak admin CLI
/opt/keycloak/bin/kcadm.sh config credentials \
--server http://localhost:8080 \
--realm master --user admin --password changeme
# Create the production realm
/opt/keycloak/bin/kcadm.sh create realms \
-s realm=production \
-s enabled=true \
-s displayName="Production Services" \
-s loginTheme=keycloak \
-s sslRequired=external
Register OIDC clients
# Confidential client — backend web application (e.g. Grafana)
/opt/keycloak/bin/kcadm.sh create clients -r production \
-s clientId=grafana \
-s name="Grafana Dashboard" \
-s enabled=true \
-s protocol=openid-connect \
-s clientAuthenticatorType=client-secret \
-s secret=GRAFANA_CLIENT_SECRET_HERE \
-s 'redirectUris=["https://grafana.example.com/login/generic_oauth"]' \
-s 'webOrigins=["https://grafana.example.com"]' \
-s directAccessGrantsEnabled=false \
-s standardFlowEnabled=true
# Public client — single-page application
/opt/keycloak/bin/kcadm.sh create clients -r production \
-s clientId=dashboard-spa \
-s name="Dashboard SPA" \
-s enabled=true \
-s protocol=openid-connect \
-s publicClient=true \
-s 'redirectUris=["https://dashboard.example.com/*"]' \
-s 'webOrigins=["https://dashboard.example.com"]' \
-s directAccessGrantsEnabled=false \
-s standardFlowEnabled=true
# Bearer-only client — API that validates tokens but never initiates login
/opt/keycloak/bin/kcadm.sh create clients -r production \
-s clientId=api-backend \
-s name="API Backend" \
-s enabled=true \
-s protocol=openid-connect \
-s bearerOnly=true
Define roles
# Realm-level roles (global)
/opt/keycloak/bin/kcadm.sh create roles -r production \
-s name=platform-admin -s description="Full platform access"
/opt/keycloak/bin/kcadm.sh create roles -r production \
-s name=platform-viewer -s description="Read-only platform access"
# Client-level roles (scoped to grafana)
GRAFANA_ID=$(/opt/keycloak/bin/kcadm.sh get clients -r production \
-q clientId=grafana --fields id --format csv --noquotes)
/opt/keycloak/bin/kcadm.sh create clients/$GRAFANA_ID/roles -r production \
-s name=admin -s description="Grafana admin"
/opt/keycloak/bin/kcadm.sh create clients/$GRAFANA_ID/roles -r production \
-s name=editor -s description="Grafana editor"
/opt/keycloak/bin/kcadm.sh create clients/$GRAFANA_ID/roles -r production \
-s name=viewer -s description="Grafana viewer"
admin is ambiguous — admin of what? Client roles like grafana:admin and vault:operator are unambiguous. The token carries both, so applications can check whichever scope they care about. Start with client roles. Use realm roles only for cross-cutting concerns like platform-admin that genuinely span every application.4. The OIDC Authorisation Code Flow
This is the flow used by web applications. The user's browser redirects to Keycloak, the user authenticates, Keycloak redirects back with an authorisation code, and the backend exchanges the code for tokens. The tokens never pass through the browser URL bar.
Step 1 — Redirect to Keycloak
The application redirects the user's browser to Keycloak's authorisation endpoint with its client ID, redirect URI, requested scopes, and a random state parameter (CSRF protection). For public clients, a PKCE code_challenge is also included.
Step 2 — User authenticates
Keycloak shows the login page (or the federated IdP's login page). The user enters credentials. If MFA is configured, the second factor is prompted. Keycloak validates everything.
Step 3 — Authorisation code returned
Keycloak redirects the browser back to the application's redirect URI with a short-lived authorisation code in the query string: ?code=abc123&state=xyz. The application checks the state matches what it sent.
Step 4 — Token exchange
The backend makes a server-to-server POST to Keycloak's token endpoint, sending the authorisation code and its client secret (or PKCE verifier). Keycloak returns the access token, refresh token, and ID token. The browser never sees the tokens — they stay server-side.
Token anatomy
# Decode an access token (it's a JWT — three base64 parts separated by dots)
echo "$ACCESS_TOKEN" | cut -d. -f2 | base64 -d 2>/dev/null | jq .
# Typical payload:
{
"exp": 1712345678,
"iat": 1712345378,
"jti": "a1b2c3d4-...",
"iss": "https://auth.example.com/realms/production",
"sub": "user-uuid-here",
"typ": "Bearer",
"azp": "grafana",
"scope": "openid email profile",
"realm_access": {
"roles": ["platform-viewer"]
},
"resource_access": {
"grafana": {
"roles": ["editor"]
}
},
"preferred_username": "alice",
"email": "alice@example.com"
}
Validate tokens in your application
# Fetch the realm's public keys (JWKS endpoint)
curl -s https://auth.example.com/realms/production/protocol/openid-connect/certs | jq .
# In nginx — use lua-resty-openidc or oauth2-proxy
# In Go — use coreos/go-oidc
# In Python — use python-jose or PyJWT with the JWKS
# In Node — use jose or jsonwebtoken with JWKS
# The pattern: download the JWKS once, cache it, verify the JWT signature
# on every request. If the signature is valid and exp > now, the token is trusted.
5. SAML 2.0 for Legacy Applications
SAML is XML-based and uses browser redirects with signed XML assertions. You will encounter it when integrating with enterprise applications that predate OIDC: older Java apps, on-premises Microsoft products, some government systems.
Register a SAML client
# Create a SAML client for a legacy application
/opt/keycloak/bin/kcadm.sh create clients -r production \
-s clientId=https://legacy-app.example.com/saml \
-s name="Legacy App" \
-s enabled=true \
-s protocol=saml \
-s 'attributes={"saml.server.signature":"true","saml.assertion.signature":"true","saml.client.signature":"false","saml_force_name_id_format":"true","saml_name_id_format":"username"}' \
-s 'redirectUris=["https://legacy-app.example.com/saml/acs"]' \
-s frontchannelLogout=true \
-s 'attributes.saml_single_logout_service_url_redirect=https://legacy-app.example.com/saml/slo'
SAML metadata exchange
# Download Keycloak's SAML metadata for the realm
curl -o keycloak-idp-metadata.xml \
https://auth.example.com/realms/production/protocol/saml/descriptor
# Upload this to the legacy application's "Identity Provider" configuration
# Download the legacy app's SP metadata (if it provides one)
# and import it into Keycloak via Admin Console > Clients > Import
| SAML Concept | OIDC Equivalent | Purpose |
|---|---|---|
Assertion |
ID Token |
Signed statement about user identity |
Service Provider (SP) |
Client (Relying Party) |
The application requesting authentication |
Identity Provider (IdP) |
OpenID Provider (OP) |
Keycloak — the thing that authenticates users |
NameID |
sub claim |
Unique user identifier |
Attribute Statement |
Claims in JWT |
User attributes (email, roles, groups) |
ACS URL |
Redirect URI |
Where the response goes after authentication |
6. User Federation — LDAP & Active Directory
User federation lets Keycloak authenticate users against an external directory without importing their passwords. The directory remains the source of truth for credentials. Keycloak syncs user attributes, group memberships, and role mappings.
Configure LDAP federation
# Via kcadm (or use the Admin Console UI)
/opt/keycloak/bin/kcadm.sh create components -r production \
-s name="Corporate LDAP" \
-s providerId=ldap \
-s providerType=org.keycloak.storage.UserStorageProvider \
-s 'config.vendor=["ad"]' \
-s 'config.connectionUrl=["ldaps://dc01.corp.example.com:636"]' \
-s 'config.bindDn=["CN=keycloak-svc,OU=Service Accounts,DC=corp,DC=example,DC=com"]' \
-s 'config.bindCredential=["SERVICE_ACCOUNT_PASSWORD"]' \
-s 'config.usersDn=["OU=Users,DC=corp,DC=example,DC=com"]' \
-s 'config.userObjectClasses=["person, organizationalPerson, user"]' \
-s 'config.usernameAttribute=["sAMAccountName"]' \
-s 'config.rdnAttribute=["cn"]' \
-s 'config.uuidAttribute=["objectGUID"]' \
-s 'config.editMode=["READ_ONLY"]' \
-s 'config.syncRegistrations=["false"]' \
-s 'config.searchScope=["2"]' \
-s 'config.pagination=["true"]' \
-s 'config.importEnabled=["true"]' \
-s 'config.batchSizeForSync=["1000"]'
Group and role mapping
# Map LDAP groups to Keycloak roles
# In Admin Console: User Federation > Corporate LDAP > Mappers > Add
# Or via CLI:
# Create a group-ldap-mapper
LDAP_ID=$(/opt/keycloak/bin/kcadm.sh get components -r production \
-q name="Corporate LDAP" --fields id --format csv --noquotes)
/opt/keycloak/bin/kcadm.sh create components -r production \
-s name="group-mapper" \
-s providerId=group-ldap-mapper \
-s providerType=org.keycloak.storage.ldap.mappers.LDAPStorageMapper \
-s parentId=$LDAP_ID \
-s 'config.groups.dn=["OU=Groups,DC=corp,DC=example,DC=com"]' \
-s 'config.group.name.ldap.attribute=["cn"]' \
-s 'config.group.object.classes=["group"]' \
-s 'config.membership.ldap.attribute=["member"]' \
-s 'config.membership.attribute.type=["DN"]' \
-s 'config.mode=["READ_ONLY"]' \
-s 'config.drop.non.existing.groups.during.sync=["false"]'
# Trigger a full sync
/opt/keycloak/bin/kcadm.sh create user-storage/$LDAP_ID/sync \
-r production -s action=triggerFullSync
READ_ONLY means Keycloak reads from LDAP but cannot write back — password changes must happen in AD. WRITABLE means Keycloak can push password changes to LDAP. UNSYNCED means Keycloak stores a local copy that can diverge from LDAP. For Active Directory, use READ_ONLY and let AD remain the password authority. Keycloak will validate passwords by binding to LDAP with the user's credentials at login time.7. Multi-Factor Authentication
Keycloak supports TOTP (Google Authenticator, FreeOTP), WebAuthn (hardware keys like YubiKey), and OTP via email. MFA is configured per-realm as an authentication flow.
Enable TOTP for all users
# Set the browser flow to require OTP
# Admin Console > Authentication > Flows > browser > OTP Form > Required
# Or via CLI — configure the realm to require OTP
/opt/keycloak/bin/kcadm.sh update realms/production \
-s 'otpPolicyType=totp' \
-s 'otpPolicyAlgorithm=HmacSHA256' \
-s 'otpPolicyDigits=6' \
-s 'otpPolicyPeriod=30' \
-s 'otpPolicyInitialCounter=0'
WebAuthn (FIDO2 hardware keys)
# Admin Console > Authentication > Required Actions > Enable "Webauthn Register"
# Admin Console > Authentication > Flows > browser
# Add "WebAuthn Authenticator" as an alternative to OTP
# Realm settings for WebAuthn
/opt/keycloak/bin/kcadm.sh update realms/production \
-s 'webAuthnPolicyRpEntityName=kldload-production' \
-s 'webAuthnPolicySignatureAlgorithms=["ES256","RS256"]' \
-s 'webAuthnPolicyAttestationConveyancePreference=none' \
-s 'webAuthnPolicyAuthenticatorAttachment=cross-platform' \
-s 'webAuthnPolicyRequireResidentKey=No' \
-s 'webAuthnPolicyUserVerificationRequirement=preferred'
Conditional MFA — require for admins only
# Create a custom authentication flow:
# 1. Duplicate the "browser" flow as "browser-conditional-mfa"
# 2. Add a "Conditional OTP" sub-flow
# 3. Add a "Condition - User Role" execution
# - Set realm role = "platform-admin"
# 4. Add "OTP Form" execution after the condition
# 5. Bind the new flow as the realm's browser flow
# Result: regular users get username/password only.
# Users with platform-admin role must also provide OTP.
8. Practical SSO — Grafana, Vault, Kubernetes
Grafana OIDC
# /etc/grafana/grafana.ini
[auth.generic_oauth]
enabled = true
name = Keycloak
allow_sign_up = true
client_id = grafana
client_secret = GRAFANA_CLIENT_SECRET_HERE
scopes = openid email profile
auth_url = https://auth.example.com/realms/production/protocol/openid-connect/auth
token_url = https://auth.example.com/realms/production/protocol/openid-connect/token
api_url = https://auth.example.com/realms/production/protocol/openid-connect/userinfo
role_attribute_path = contains(resource_access.grafana.roles[*], 'admin') && 'Admin' || contains(resource_access.grafana.roles[*], 'editor') && 'Editor' || 'Viewer'
email_attribute_path = email
login_attribute_path = preferred_username
name_attribute_path = name
HashiCorp Vault OIDC
# Enable the OIDC auth method
vault auth enable oidc
# Configure Vault to use Keycloak
vault write auth/oidc/config \
oidc_discovery_url="https://auth.example.com/realms/production" \
oidc_client_id="vault" \
oidc_client_secret="VAULT_CLIENT_SECRET_HERE" \
default_role="default"
# Create a role mapping
vault write auth/oidc/role/default \
bound_audiences="vault" \
allowed_redirect_uris="https://vault.example.com/ui/vault/auth/oidc/oidc/callback" \
allowed_redirect_uris="http://localhost:8250/oidc/callback" \
user_claim="preferred_username" \
groups_claim="resource_access.vault.roles" \
policies="default" \
ttl=1h
Kubernetes OIDC
# kube-apiserver flags (add to /etc/kubernetes/manifests/kube-apiserver.yaml)
--oidc-issuer-url=https://auth.example.com/realms/production
--oidc-client-id=kubernetes
--oidc-username-claim=preferred_username
--oidc-username-prefix=oidc:
--oidc-groups-claim=groups
--oidc-groups-prefix=oidc:
--oidc-ca-file=/etc/kubernetes/pki/keycloak-ca.pem
# RBAC binding for a Keycloak group
kubectl create clusterrolebinding keycloak-admins \
--clusterrole=cluster-admin \
--group=oidc:platform-admin
# kubelogin — kubectl plugin for OIDC login
kubectl oidc-login setup \
--oidc-issuer-url=https://auth.example.com/realms/production \
--oidc-client-id=kubernetes \
--oidc-client-secret=K8S_CLIENT_SECRET
# Add to kubeconfig
kubectl config set-credentials oidc-user \
--exec-api-version=client.authentication.k8s.io/v1beta1 \
--exec-command=kubectl \
--exec-arg=oidc-login \
--exec-arg=get-token \
--exec-arg=--oidc-issuer-url=https://auth.example.com/realms/production \
--exec-arg=--oidc-client-id=kubernetes \
--exec-arg=--oidc-client-secret=K8S_CLIENT_SECRET
9. SELinux Fundamentals
SELinux (Security-Enhanced Linux) is a mandatory access control (MAC) system built into the Linux kernel. Unlike traditional Unix permissions (discretionary access control — DAC), SELinux policies are enforced by the kernel and cannot be overridden by root. Even if an attacker compromises a service and escalates to root, SELinux confines the process to its labelled domain and prevents it from accessing anything outside that domain.
DAC vs MAC
DAC (traditional Unix): the file owner decides who can access it. Root bypasses everything. MAC (SELinux): the kernel enforces policy regardless of user. Root running as httpd_t can only access objects labelled for httpd_t. The policy is the law, not the user.
Labels (Security Contexts)
Every process, file, port, and socket has a label: user:role:type:level. For files: system_u:object_r:httpd_sys_content_t:s0. For processes: system_u:system_r:httpd_t:s0. The type field is what matters most — SELinux type enforcement (TE) is the primary access control mechanism.
Type Enforcement (TE)
The core of SELinux. Processes run in a domain (type). Files have a type. The policy defines which domains can access which types, and how (read, write, execute, etc.). If the policy does not explicitly allow an access, it is denied.
Modes: Enforcing, Permissive, Disabled
Enforcing: policy is applied, violations are blocked and logged. Permissive: policy violations are logged but not blocked (for debugging). Disabled: SELinux is off entirely. Production systems must run enforcing. Permissive is for troubleshooting only.
Booleans
On/off switches for specific policy rules. httpd_can_network_connect controls whether Apache can make outbound network connections. Booleans let you tune policy without writing custom modules. There are hundreds of them.
Policy Type: Targeted
RHEL/CentOS/Rocky use the targeted policy: only specific daemons are confined. Everything else runs as unconfined_t (unrestricted). This is a pragmatic middle ground — critical services like httpd, sshd, named are confined; user sessions are not.
audit2allow, audit2why, semanage) makes this routine. The reason SELinux has a bad reputation is that people encounter it only when it blocks something, and they do not know the tools to fix it. This section teaches those tools.10. SELinux Status & Configuration
# Check current mode
getenforce
# Enforcing
# Check detailed status
sestatus
# SELinux status: enabled
# SELinuxfs mount: /sys/fs/selinux
# SELinux root directory: /etc/selinux
# Loaded policy name: targeted
# Current mode: enforcing
# Mode from config file: enforcing
# Policy MLS status: enabled
# Policy deny_unknown status: allowed
# Memory protection checking: actual (secure)
# Max kernel policy version: 33
# Temporarily switch to permissive (for debugging — reverts on reboot)
setenforce 0
# Back to enforcing
setenforce 1
# Permanent configuration
cat /etc/selinux/config
# SELINUX=enforcing
# SELINUXTYPE=targeted
View labels on everything
# File labels
ls -Z /var/www/html/
# system_u:object_r:httpd_sys_content_t:s0 index.html
# Process labels
ps auxZ | grep httpd
# system_u:system_r:httpd_t:s0 root 12345 ... /usr/sbin/httpd
# Port labels
semanage port -l | grep http
# http_port_t tcp 80, 443, 488, 8008, 8009, 8443, 9000
# User mappings
semanage login -l
# Login Name SELinux User MLS/MCS Range
# root unconfined_u s0-s0:c0.c1023
# __default__ unconfined_u s0-s0:c0.c1023
11. File Contexts & Relabelling
When SELinux blocks access, the most common cause is wrong file labels. Files get the wrong label when you copy them (cp preserves labels from the source), move them from a different directory, or create them in a non-standard location.
Check and fix file contexts
# See what the policy expects for a path
matchpathcon /var/www/html
# /var/www/html system_u:object_r:httpd_sys_content_t:s0
# See the actual label
ls -Z /var/www/html
# If it says httpd_sys_content_t — it's correct
# If it says default_t or something else — relabel it
# Restore default context for a directory tree
restorecon -Rv /var/www/html/
# Relabeled /var/www/html from unconfined_u:object_r:default_t:s0
# to system_u:object_r:httpd_sys_content_t:s0
# Restore the entire filesystem (after major changes — takes time)
touch /.autorelabel
reboot
Add custom file context rules
# Tell SELinux that /srv/myapp/static should have httpd content labels
semanage fcontext -a -t httpd_sys_content_t '/srv/myapp/static(/.*)?'
restorecon -Rv /srv/myapp/static/
# Tell SELinux that /opt/keycloak/data should be writable by Java
semanage fcontext -a -t var_lib_t '/opt/keycloak/data(/.*)?'
restorecon -Rv /opt/keycloak/data/
# List all custom file context rules
semanage fcontext -l -C
ls -Z, the label is wrong, you run restorecon, it works. This covers 60–70% of all SELinux issues. The second most common: a boolean needs to be flipped. The third: a port needs to be labelled. Custom policy modules are the last resort, not the first.12. Booleans — Tuning Policy Without Writing Policy
# List all booleans (there are hundreds)
getsebool -a
# Search for relevant booleans
getsebool -a | grep httpd
# httpd_can_network_connect --> off
# httpd_can_network_connect_db --> off
# httpd_can_sendmail --> off
# httpd_enable_cgi --> on
# httpd_enable_homedirs --> off
# ...
# Enable a boolean (persistent across reboots)
setsebool -P httpd_can_network_connect on
# Enable temporarily (reverts on reboot — for testing)
setsebool httpd_can_network_connect on
# Common booleans you will need:
setsebool -P httpd_can_network_connect on # nginx/Apache reverse proxy
setsebool -P httpd_can_network_connect_db on # web app to database
setsebool -P httpd_can_network_relay on # proxy/load balancer
setsebool -P nis_enabled on # NIS/LDAP client lookups
setsebool -P domain_can_mmap_files on # JVM applications (Keycloak)
setsebool -P daemons_enable_cluster_mode on # Pacemaker/Corosync
Booleans for Keycloak specifically
# Keycloak (Java) needs:
setsebool -P httpd_can_network_connect on # if proxied via httpd
setsebool -P domain_can_mmap_files on # JVM memory mapping
# If Keycloak connects to an external LDAP:
setsebool -P authlogin_nsswitch_use_ldap on
# If Keycloak sends email (password reset):
setsebool -P httpd_can_sendmail on
getsebool -a | grep $keyword is the fastest diagnostic you have.13. Port Labelling
SELinux controls which services can bind to which ports. If you run a service on a non-standard port, SELinux will block it unless you label that port for the service's domain.
# See current port labels
semanage port -l | grep -E 'http|ssh|postgres'
# http_port_t tcp 80, 443, 488, 8008, 8009, 8443, 9000
# ssh_port_t tcp 22
# postgresql_port_t tcp 5432
# Allow nginx to bind to port 3000
semanage port -a -t http_port_t -p tcp 3000
# Allow SSH on a non-standard port
semanage port -a -t ssh_port_t -p tcp 2222
# Allow Keycloak on port 8080 (already http_port_t? check first)
semanage port -l | grep 8080
# If not listed, add it:
semanage port -a -t http_port_t -p tcp 8080
# Remove a custom port label
semanage port -d -t http_port_t -p tcp 3000
# If the port is already defined for another type, use -m (modify)
semanage port -m -t http_port_t -p tcp 8888
14. Troubleshooting SELinux Denials
When SELinux blocks something, it writes an AVC (Access Vector Cache) denial to the audit log. The troubleshooting workflow is always the same: find the denial, understand it, fix it.
Step 1 — Find the denial
# Recent AVC denials
ausearch -m avc -ts recent
# Denials for a specific service
ausearch -m avc -c httpd -ts today
# Tail the audit log in real time
tail -f /var/log/audit/audit.log | grep denied
# Using sealert (if setroubleshoot is installed)
sealert -a /var/log/audit/audit.log
Step 2 — Understand the denial
# Example AVC denial:
# type=AVC msg=audit(1712345678.123:456): avc: denied { name_connect }
# for pid=1234 comm="nginx" dest=8080
# scontext=system_u:system_r:httpd_t:s0
# tcontext=system_u:object_r:http_port_t:s0
# tclass=tcp_socket permissive=0
# Translation: nginx (httpd_t) tried to connect (name_connect)
# to port 8080 (http_port_t) and was denied.
# Use audit2why to get a human-readable explanation
ausearch -m avc -ts recent | audit2why
# Was caused by: The boolean httpd_can_network_connect was set to off.
# Fix: setsebool -P httpd_can_network_connect on
Step 3 — Fix it (in order of preference)
# 1. Fix the file label (most common)
restorecon -Rv /path/to/files
# 2. Flip a boolean
setsebool -P httpd_can_network_connect on
# 3. Label a port
semanage port -a -t http_port_t -p tcp 3000
# 4. Write a custom policy module (last resort)
ausearch -m avc -ts recent | audit2allow -M my_custom_policy
semodule -i my_custom_policy.pp
audit2why tells you exactly what to do in most cases. The reason people get frustrated is they skip step 2 and go straight to setenforce 0. That is like turning off the fire alarm instead of finding the fire. The audit log tells you exactly what was denied, what process tried it, and what the fix is. Read it.15. Writing Custom SELinux Policy Modules
When no boolean, file context, or port label fixes the issue, you need a custom policy module.
The audit2allow tool generates one from AVC denials. The trick is to generate a
minimal module that allows only what is needed, not a blanket permission.
Generate a module from denials
# Collect all recent denials for a service
ausearch -m avc -c keycloak -ts today > /tmp/keycloak-denials.txt
# Generate a human-readable policy
cat /tmp/keycloak-denials.txt | audit2allow
# #============= java_t ==============
# allow java_t self:netlink_route_socket { bind create getattr nlmsg_read read };
# allow java_t http_port_t:tcp_socket name_connect;
# Generate a compilable module
cat /tmp/keycloak-denials.txt | audit2allow -M keycloak_custom
# ******************** IMPORTANT ***********************
# To make this policy package active, execute:
# semodule -i keycloak_custom.pp
# Review the .te file before installing
cat keycloak_custom.te
# module keycloak_custom 1.0;
# require {
# type java_t;
# type http_port_t;
# class tcp_socket name_connect;
# class netlink_route_socket { bind create getattr nlmsg_read read };
# }
# #============= java_t ==============
# allow java_t self:netlink_route_socket { bind create getattr nlmsg_read read };
# allow java_t http_port_t:tcp_socket name_connect;
# Install the module
semodule -i keycloak_custom.pp
# Verify it is loaded
semodule -l | grep keycloak_custom
Write a module from scratch
# /root/selinux-modules/myapp.te
policy_module(myapp, 1.0)
# Declare types
type myapp_t;
type myapp_exec_t;
type myapp_data_t;
# Allow init to start the service
init_daemon_domain(myapp_t, myapp_exec_t)
# Networking
allow myapp_t self:tcp_socket { create bind listen accept read write getattr };
corenet_tcp_bind_generic_port(myapp_t)
corenet_tcp_connect_http_port(myapp_t)
# File access
allow myapp_t myapp_data_t:dir { read write add_name remove_name search };
allow myapp_t myapp_data_t:file { create read write unlink getattr open };
# Logging
logging_send_syslog_msg(myapp_t)
# DNS resolution
sysnet_dns_name_resolve(myapp_t)
# /root/selinux-modules/myapp.fc (file contexts)
/opt/myapp/bin/myapp -- gen_context(system_u:object_r:myapp_exec_t,s0)
/opt/myapp/data(/.*)? gen_context(system_u:object_r:myapp_data_t,s0)
# Compile and install
make -f /usr/share/selinux/devel/Makefile myapp.pp
semodule -i myapp.pp
restorecon -Rv /opt/myapp/
Manage modules
# List all installed modules
semodule -l
# Remove a module
semodule -r keycloak_custom
# Disable a module (keeps it installed)
semodule -d keycloak_custom
# Re-enable a disabled module
semodule -e keycloak_custom
# Check module priority (higher priority wins)
semodule -l --all | head
audit2allow approach is quick but can be overly permissive — it allows everything that was denied during your test session, which might include things the service should not actually do. Always review the generated .te file. For production services, writing a module from scratch gives you tighter control. The selinux-policy-devel package provides macros (init_daemon_domain, corenet_tcp_bind_generic_port, etc.) that are the correct way to grant access — they handle the underlying allow rules and transitions that a raw allow statement might miss.16. Multi-Category Security (MCS)
MCS extends type enforcement with categories. Two processes can have the same type but different categories — they cannot access each other's files. This is how container runtimes isolate containers under SELinux: each container gets a unique category pair.
# Category labels look like: s0:c1,c2
# A process with s0:c1,c2 can access files with s0:c1,c2 but NOT s0:c3,c4
# See a container's category
ps auxZ | grep container
# system_u:system_r:container_t:s0:c123,c456 root ... /usr/bin/myapp
# Files for that container:
ls -Z /var/lib/containers/storage/overlay/.../merged/
# system_u:object_r:container_file_t:s0:c123,c456
# Podman assigns unique categories automatically
podman run --security-opt label=level:s0:c200,c300 ...
# Disable SELinux for a specific container (NOT recommended)
podman run --security-opt label=disable ...
MCS for multi-tenant services
# Assign categories to users for multi-tenant isolation
semanage login -a -s user_u -r s0:c10,c20 tenant_alice
semanage login -a -s user_u -r s0:c30,c40 tenant_bob
# Alice's processes get s0:c10,c20 — they can only access files with s0:c10,c20
# Bob's processes get s0:c30,c40 — complete isolation at the kernel level
# This is useful for shared hosting, multi-tenant databases, or any scenario
# where multiple untrusted workloads share a kernel
17. SELinux Policy for Keycloak
Running Keycloak under SELinux requires a few adjustments. Java applications run in the
java_t domain by default, which is quite restricted.
Complete SELinux setup for Keycloak
# 1. Label the Keycloak directories
semanage fcontext -a -t java_exec_t '/opt/keycloak/bin(/.*)?'
semanage fcontext -a -t var_lib_t '/opt/keycloak/data(/.*)?'
semanage fcontext -a -t etc_t '/opt/keycloak/conf(/.*)?'
semanage fcontext -a -t var_log_t '/var/log/keycloak(/.*)?'
restorecon -Rv /opt/keycloak /var/log/keycloak
# 2. Set booleans
setsebool -P domain_can_mmap_files on
setsebool -P httpd_can_network_connect on
# 3. Label non-standard ports if needed
semanage port -a -t http_port_t -p tcp 8443 # HTTPS management
# 8080 is usually already http_port_t
# 4. Run Keycloak in permissive temporarily to collect denials
semanage permissive -a java_t
# 5. Exercise Keycloak (login, create realm, LDAP sync, etc.)
# 6. Generate custom module from collected denials
ausearch -m avc -c java -ts today | audit2allow -M keycloak_policy
# 7. Review the .te file, remove anything overly broad
cat keycloak_policy.te
# 8. Install and switch back to enforcing
semodule -i keycloak_policy.pp
semanage permissive -d java_t
# 9. Verify Keycloak works in enforcing mode
systemctl restart keycloak
curl -s https://auth.example.com/realms/production/.well-known/openid-configuration | jq .issuer
semanage permissive -a java_t trick is the professional approach: it puts only the java_t domain into permissive mode while the rest of the system stays enforcing. This lets you collect all the denials Keycloak generates during normal operation without weakening security for anything else. Once you have the custom module, switch java_t back to enforcing. This is how RHEL consultants do it.18. SELinux for Containers
# Podman and CRI-O are SELinux-aware by default
# Every container runs with a unique MCS label
# Verify container SELinux is working
podman run --rm fedora:latest cat /proc/1/attr/current
# system_u:system_r:container_t:s0:c123,c456
# Bind mounts need the :Z or :z flag
# :Z = private label (only this container can access)
# :z = shared label (multiple containers can access)
podman run -v /data/myapp:/app:Z myimage
# Without :Z, SELinux blocks the container from reading the mount
# because the host files have host labels, not container labels
# For ZFS datasets mounted into containers:
podman run -v /tank/app-data:/data:Z --security-opt label=type:container_file_t myimage
Kubernetes with SELinux
# Pod security context with SELinux options
apiVersion: v1
kind: Pod
metadata:
name: secure-app
spec:
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
type: "container_t"
containers:
- name: app
image: myapp:latest
securityContext:
seLinuxOptions:
level: "s0:c123,c456"
# SELinux with Cilium — Cilium's eBPF programs need:
setsebool -P domain_kernel_load_modules on
# Or use a custom module for cilium_t if defined
# Check for SELinux denials from Cilium
ausearch -m avc -c cilium -ts today
19. Production Checklist
| Item | Keycloak | SELinux |
|---|---|---|
| TLS | All endpoints HTTPS. hostname-strict=true. |
Certificate files labelled cert_t. |
| Database | PostgreSQL with TLS. Separate credentials per environment. | postgresql_t domain. Data on ZFS labelled correctly. |
| Secrets | Client secrets in Vault. Admin password rotated. | Config files etc_t. No world-readable secrets. |
| Tokens | Access token lifetime ≤15 min. Refresh token ≤8 hours. | N/A |
| MFA | WebAuthn for admins. TOTP for all users minimum. | N/A |
| Backup | PostgreSQL on ZFS → snapshot + replicate. | Custom modules in git. semanage export in backups. |
| Monitoring | Prometheus metrics at /metrics. Alert on login failures. |
Alert on AVC denials. ausearch -m avc in cron. |
| Mode | Production build (kc.sh build + start --optimized). |
Enforcing. No permissive domains. No disabled. |
Export and import SELinux customisations
# Export all local SELinux customisations (for backup or replication)
semanage export > /root/selinux-customisations.txt
# Import on another node
semanage import < /root/selinux-customisations.txt
# Example export output:
# boolean -m -1 httpd_can_network_connect
# port -a -t http_port_t -p tcp 3000
# fcontext -a -t httpd_sys_content_t '/srv/myapp/static(/.*)?'
semanage export/import pattern is gold for infrastructure-as-code. Export your SELinux customisations on a working node, commit the file to git, and apply it to new nodes via your postinstaller. Combined with Keycloak's realm export (kc.sh export --dir /tmp/realms), you can reproduce your entire authentication and access control stack from a clean kldload install.20. Quick Reference — Troubleshooting Table
| Symptom | Likely Cause | Fix |
|---|---|---|
| 403 Forbidden from nginx/httpd | Wrong file context on web root | restorecon -Rv /var/www/html |
| nginx cannot connect to upstream | httpd_can_network_connect is off |
setsebool -P httpd_can_network_connect on |
| Service won't bind to port | Port not labelled for service type | semanage port -a -t TYPE -p tcp PORT |
| Container can't read bind mount | Missing :Z on volume mount |
podman run -v /path:/path:Z ... |
| Keycloak OIDC redirect loop | Incorrect redirectUris on client |
Check client config, must match app exactly |
| Token validation fails | Clock skew between app and Keycloak | chronyc tracking — fix NTP |
| LDAP sync fails | Firewall or SELinux blocking LDAPS 636 | semanage port -a -t ldap_port_t -p tcp 636 |
| SAML assertion rejected | Mismatched Entity ID or ACS URL | Compare SP metadata with client config |
| Files created with wrong label | Created in non-standard path | semanage fcontext -a + restorecon |
Related pages
- TLS & PKI Masterclass — certificates for Keycloak and internal services
- Vault & Secrets Masterclass — storing Keycloak client secrets securely
- Security Hardening Masterclass — CIS benchmarks, Falco, intrusion detection
- FIPS 140-3 Compliance — cryptographic compliance for Keycloak and SELinux
- nftables Masterclass — firewall rules complementing SELinux
- Kubernetes Masterclass — OIDC integration and pod security
- Security — kldload security posture overview