Greetings, I am trying to run a keycloak cluster configuration in k8s of 3 replicas. I am using haproxy with TLS termination as a reverse proxy.
The version of keycloak is 21.0.2.
I use next configuration
KC_CACHE: ispn
KC_CACHE_STACK: kubernetes
KC_DB_ADDR: mariadbendpoint:3306
KC_DB_DATABASE: keycloak
KC_DB_POOL_MAX_SIZE: 15
KC_DB_POOL_MIN_SIZE: 5
KC_DB_USER: keycloak
KC_DB_VENDOR: mariadb
KC_HEALTH_ENABLED: true
KC_HTTP_RELATIVE_PATH: /auth
KC_LOG_LEVEL: INFO,org.jgroups.protocols:TRACE
KC_METRICS_ENABLED: true
KC_PROXY: edge
KC_HOSTNAME_STRICT: false
KC_HOSTNAME_STRICT_BACKCHANNEL: false
KC_HOSTNAME_STRICT_HTTPS: false
JAVA_OPTS_APPEND: -Djboss.as.management.blocking.timeout=300 -Djgroups.dns.query=keycloak-headless
The first problem I encountered was the inability to log into the Administration console, and main page (https://keycloak-endpoint.com/auth) has broken styles
According to the documentation, it is recommended to enable sticky-session on my reverse-proxy - Using a reverse proxy - Keycloak
I configured the backend as follows:
backend keycloak
mode http
option httplog
option forwardfor header X-Real-IP
option http-server-close
capture request header x-req-id len 16
capture request header Host len 32
capture request header Referer len 32
capture request header X-MCS-Request-Id len 64
timeout client 2m
timeout server 2m
timeout http-request 10s
http-request redirect code 301 location https://keycloak-endpoint.com/auth if { path -i / }
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header X-Forwarded-Port %[dst_port]
cookie REVERSE_PROXY_SESSION_ID maxidle 30m maxlife 12h insert nocache preserve indirect secure
server-template keycloak1 1-3 keycloak-headless.keycloak-vkcloud.svc.kube:8080 check inter 10s fastinter 1s downinter 1s resolvers kubedns-kube init-addr none cookie keycloak
The administration console is now working. And styles looks good on main page.
Next, I set up a new realm - cluster-realm and added some test users.
And now I want to get a list of users using the REST API as follows:
token=`curl -sk -v -X POST https://keycloak-endpoint.com/auth/realms/master/protocol/openid-connect/token -H 'Accept: application/json' -H 'Content-Type: application/x-www-form-urlencoded' -H 'cache-control: no-cache' -d 'grant_type=password&username=admin&password=s0mestr0ngp@ss&client_id=admin-cli' | jq -r '.access_token'`
# get all users of cluster-realm realm, use the token from above and use Bearer as prefix
curl -sk -v -X GET https://keycloak-endpoint.com/auth/admin/realms/cluster-realm/users -H "Authorization: Bearer $token" | jq
When I try to do this - I get a 401 error, although seeing by curl logs - sticky session is the same (which I configured on reverse-proxy earlier)
* Connected to keycloak-endpoint.com
> POST /auth/realms/master/protocol/openid-connect/token HTTP/2
> Host: keycloak-endpoint.com
< HTTP/2 200
< referrer-policy: no-referrer
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=31536000; includeSubDomains
< cache-control: no-store
< x-content-type-options: nosniff
< set-cookie: KEYCLOAK_LOCALE=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/master/; HttpOnly
< set-cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/master/; HttpOnly
< pragma: no-cache
< x-xss-protection: 1; mode=block
< content-type: application/json
< content-length: 1969
< set-cookie: BALANCER_SESSION_ID=vk-keycloak|ZDaGh|ZDaGh; path=/; Secure
<
{ [1969 bytes data]
* Connection #0 to host keycloak-endpoint.com left intact
some-token-here
> GET /auth/admin/realms/cluster-realm/users HTTP/2
> Host: keycloak-endpoint.com
> user-agent: curl/7.81.0
> accept: */*
< HTTP/2 401
> authorization: Bearer some-token-here
< referrer-policy: no-referrer
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< content-type: application/json
< content-length: 33
< set-cookie: BALANCER_SESSION_ID=vk-keycloak|ZDaGh|ZDaGh; path=/; Secure
<
{ [33 bytes data]
* Connection #0 to host keycloak-endpoint.com left intact
{
"error": "HTTP 401 Unauthorized"
}
If I scale replica number from 3 to 1 - everything works fine.
* Connected to keycloak-endpoint.com
> POST /auth/realms/master/protocol/openid-connect/token HTTP/2
> Host: keycloak-endpoint.com
< HTTP/2 200
< referrer-policy: no-referrer
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=31536000; includeSubDomains
< cache-control: no-store
< x-content-type-options: nosniff
< set-cookie: KEYCLOAK_LOCALE=; Version=1; Comment=Expiring cookie; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/master/; HttpOnly
< set-cookie: KC_RESTART=; Version=1; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Max-Age=0; Path=/auth/realms/master/; HttpOnly
< pragma: no-cache
< x-xss-protection: 1; mode=block
< content-type: application/json
< content-length: 1969
< set-cookie: BALANCER_SESSION_ID=vk-keycloak|ZDaAo|ZDaAo; path=/; Secure
<
{ [1969 bytes data]
* Connection #0 to host keycloak-endpoint.com left intact
some-token-here
> GET /auth/admin/realms/cluster-realm/users HTTP/2
> Host: keycloak-endpoint.com
> user-agent: curl/7.81.0
> accept: */*
> authorization: Bearer some-token-here
< HTTP/2 200
< referrer-policy: no-referrer
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=31536000; includeSubDomains
< cache-control: no-cache
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< content-type: application/json
< content-length: 1116
< set-cookie: BALANCER_SESSION_ID=vk-keycloak|ZDaAo|ZDaAo; path=/; Secure
<
{ [1116 bytes data]
* Connection #0 to host keycloak-endpoint.com left intact
[
{
"id": "f6c509c9-3758-4dea-9614-6d876f22e0be",
"createdTimestamp": 1681293358639,
"username": "admin",
"enabled": true,
"totp": false,
"emailVerified": false,
"firstName": "",
"lastName": "",
"email": "admin@one.com",
"disableableCredentialTypes": [],
"requiredActions": [],
"notBefore": 0,
"access": {
"manageGroupMembership": true,
"view": true,
"mapRoles": true,
"impersonate": true,
"manage": true
}
},
{
"id": "4ad854ff-7572-466f-a5fa-c697488fd272",
"createdTimestamp": 1681293333136,
"username": "user@one",
"enabled": true,
"totp": false,
"emailVerified": false,
"firstName": "",
"lastName": "",
"email": "user@one.com",
"disableableCredentialTypes": [],
"requiredActions": [],
"notBefore": 0,
"access": {
"manageGroupMembership": true,
"view": true,
"mapRoles": true,
"impersonate": true,
"manage": true
}
},
{
"id": "06baee2b-14cf-48a9-a2bd-6586df831099",
"createdTimestamp": 1681293345299,
"username": "user@two.com",
"enabled": true,
"totp": false,
"emailVerified": false,
"firstName": "",
"lastName": "",
"email": "user@two.com",
"disableableCredentialTypes": [],
"requiredActions": [],
"notBefore": 0,
"access": {
"manageGroupMembership": true,
"view": true,
"mapRoles": true,
"impersonate": true,
"manage": true
}
}
]
I checked the logs for cluster configuration and ispn cache - looks valid, DNS PING is in place and there is a connection between the pods.
2023-04-12 13:30:22,883 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-138,keycloak-0-24014) keycloak-0-24014: sending stability msg keycloak-0-24014: [829], keycloak-1-22812: [199], keycloak-2-42571: [172]
2023-04-12 13:30:22,883 TRACE [org.jgroups.protocols.TCP] (jgroups-138,keycloak-0-24014) keycloak-0-24014: sending msg to null, src=keycloak-0-24014, size=124, headers are STABLE: [STABILITY] view-id= [keycloak-0-24014|6], TP: [cluster=ISPN]
2023-04-12 13:30:22,884 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received stable digest keycloak-0-24014: [829 (829)], keycloak-1-22812: [199 (199)], keycloak-2-42571: [172 (172)]
2023-04-12 13:30:22,884 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-138,keycloak-0-24014) keycloak-0-24014: deleting msgs <= 829 from keycloak-0-24014
2023-04-12 13:30:22,884 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-138,keycloak-0-24014) keycloak-0-24014: deleting msgs <= 199 from keycloak-1-22812
2023-04-12 13:30:22,884 TRACE [org.jgroups.protocols.pbcast.NAKACK2] (jgroups-138,keycloak-0-24014) keycloak-0-24014: deleting msgs <= 172 from keycloak-2-42571
2023-04-12 13:30:23,339 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-138,keycloak-0-24014) keycloak-0-24014: updating the local digest with a stable message (coordinator): keycloak-0-24014: [829 (829)], keycloak-1-22812: [199 (199)], keycloak-2-42571: [172 (172)]
2023-04-12 13:30:23,339 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-138,keycloak-0-24014) keycloak-0-24014: handling digest from keycloak-0-24014:
mine: keycloak-0-24014: [-1], keycloak-1-22812: [-1], keycloak-2-42571: [-1]
sender: keycloak-0-24014: [829], keycloak-1-22812: [199], keycloak-2-42571: [172]
result: keycloak-0-24014: [829], keycloak-1-22812: [199], keycloak-2-42571: [172]
2023-04-12 13:30:23,590 TRACE [org.jgroups.protocols.TCP] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received message batch of 1 messages from keycloak-1-22812
2023-04-12 13:30:23,590 TRACE [org.jgroups.protocols.FD_ALL3] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received heartbeat from keycloak-1-22812
2023-04-12 13:30:24,061 TRACE [org.jgroups.protocols.TCP] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received message batch of 1 messages from keycloak-2-42571
2023-04-12 13:30:24,061 TRACE [org.jgroups.protocols.FD_ALL3] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received heartbeat from keycloak-2-42571
2023-04-12 13:30:24,558 TRACE [org.jgroups.protocols.TCP] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received [keycloak-2-42571 to 10.220.1.228:7800, 51 bytes, flags=OOB|DONT_BUNDLE], headers are DNS_PING: [GET_MBRS_REQ cluster=ISPN initial_discovery=false], TP: [cluster=ISPN]
2023-04-12 13:30:24,558 TRACE [org.jgroups.protocols.TCP] (jgroups-138,keycloak-0-24014) keycloak-0-24014: sending msg to 10.220.1.101:7800, src=keycloak-0-24014, size=95, headers are MERGE3: INFO: view_id=[keycloak-0-24014|6], logical_name=keycloak-0-24014, physical_addr=10.220.1.228:7800, TP: [cluster=ISPN]
2023-04-12 13:30:24,558 TRACE [org.jgroups.protocols.dns.DNS_PING] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received GET_MBRS_REQ from keycloak-2-42571, sending response keycloak-0-24014, name=keycloak-0-24014, addr=10.220.1.228:7800, coord
2023-04-12 13:30:24,559 TRACE [org.jgroups.protocols.TCP] (jgroups-138,keycloak-0-24014) keycloak-0-24014: sending msg to keycloak-2-42571, src=keycloak-0-24014, size=111, headers are DNS_PING: [GET_MBRS_RSP cluster=null initial_discovery=false], TP: [cluster=ISPN]
2023-04-12 13:30:24,561 TRACE [org.jgroups.protocols.TCP] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received message batch of 1 messages from keycloak-2-42571
2023-04-12 13:30:27,557 TRACE [org.jgroups.protocols.TCP] (jgroups-138,keycloak-0-24014) keycloak-0-24014: received [keycloak-1-22812 to keycloak-0-24014, 67 bytes, flags=OOB|NO_RELIABILITY], obj: keycloak-0-24014: [829 (829)], keycloak-1-22812: [199 (199)], keycloak-2-42571: [172 (172)], headers are STABLE: [STABLE_GOSSIP] view-id= [keycloak-0-24014|6], TP: [cluster=ISPN]
2023-04-12 13:30:27,557 TRACE [org.jgroups.protocols.pbcast.STABLE] (jgroups-138,keycloak-0-24014) keycloak-0-24014: handling digest from keycloak-1-22812:
mine: keycloak-0-24014: [829], keycloak-1-22812: [199], keycloak-2-42571: [172]
sender: keycloak-0-24014: [829], keycloak-1-22812: [199], keycloak-2-42571: [172]
result: keycloak-0-24014: [829], keycloak-1-22812: [199], keycloak-2-42571: [172]
Please help what I might have missed and why the cluster configuration might not work.