# How keycloak works in a cluster with cache **URL:** \ **Category:** Configuring the server\ **Tags:** clustering\ **Created:** [February 7, 2025, 12:33pm UTC](https://forum.keycloak.org/t/how-keycloak-works-in-a-cluster-with-cache/29553 "2025-02-07T12:33:25Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![alansatis](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/alansatis/32/11960_2.png) [@alansatis](https://forum.keycloak.org/u/alansatis)\ **Post date:** [February 7, 2025, 12:33pm UTC](https://forum.keycloak.org/t/how-keycloak-works-in-a-cluster-with-cache/29553/1 "2025-02-07T12:33:25Z")
Hello everyone! I hope you are all well! I’m trying to use Keycloak version 26.1.0 in cluster mode. My scenario: I have an internal load balancer that points to an Azure Scale-Set. In this scale-set, I have two VMs. In each one, I have a docker-compose that configures Keycloak for me. When I run my containers, Keycloak starts cluster mode successfully, but it uses the internal IP of the Docker network. This causes me to have several timeouts when it tries to communicate with the other instance, since the instances are in different VMs, with different internal networks. I effectively have communication between the VMs, through the VM IP and with the appropriate ports (7800). My Docker - VM01 keycloak01: image: keycloak:26.1.0 container\_name: keycloak01 command: - start - --truststore-paths=/opt/keycloak/certs/DigiCertGlobalRootCA.crt,/opt/keycloak/certs/DigiCertGlobalRootG2.crt.pem,/opt/keycloak/certs/MicrosoftRsaRootCertificateAuthority2017.cert environment: KC\_BOOTSTRAP\_ADMIN\_USERNAME: sreadmin KC\_BOOTSTRAP\_ADMIN\_PASSWORD: “admin” KC\_DB: postgres KC\_DB\_URL: “jdbc:postgresql://domain.postgres.database.azure.com:5432/keycloak?sslmode=verify-full&sslrootcert=/opt/keycloak/certs/DigiCertGlobalRootCA.crt” KC\_DB\_SCHEMA: public KC\_LOG\_LEVEL: INFO DB\_DATABASE: keycloak KC\_DB\_USERNAME: manager KC\_DB\_PASSWORD: “pass” KC\_PROXY: edge KC\_HOSTNAME: keycloak01 KC\_HTTP\_ENABLED: “true” KC\_HOSTNAME\_STRICT: true KC\_CACHE: ispn KC\_CACHE\_STACK: jdbc-ping # KC\_JGROUPS\_DISCOVERY\_PROTOCOL: JDBC\_PING JGROUPS\_DISCOVERY\_EXTERNAL\_IP: 10.6.6.5 JGROUPS\_BIND\_ADDR: 10.6.6.5 ports: - “80:8080” # Porta HTTP - “7800:7800” volumes: - ./certs:/opt/keycloak/certs:ro restart: always networks: - app-network My Docker - VM02 keycloak02: image: keycloak:26.1.0 container\_name: keycloak02 command: - start - --truststore-paths=/opt/keycloak/certs/DigiCertGlobalRootCA.crt,/opt/keycloak/certs/DigiCertGlobalRootG2.crt.pem,/opt/keycloak/certs/MicrosoftRsaRootCertificateAuthority2017.cert environment: KC\_BOOTSTRAP\_ADMIN\_USERNAME: sreadmin KC\_BOOTSTRAP\_ADMIN\_PASSWORD: “admin” KC\_DB: postgres KC\_DB\_URL: “jdbc:postgresql://domain.database.azure.com:5432/keycloak?sslmode=verify-full&sslrootcert=/opt/keycloak/certs/DigiCertGlobalRootCA.crt” KC\_DB\_SCHEMA: public KC\_LOG\_LEVEL: INFO DB\_DATABASE: keycloak KC\_DB\_USERNAME: manager KC\_DB\_PASSWORD: “pass” KC\_PROXY: edge KC\_HOSTNAME: keycloak02 KC\_HTTP\_ENABLED: “true” KC\_HOSTNAME\_STRICT: true KC\_CACHE: ispn KC\_CACHE\_STACK: jdbc-ping # KC\_JGROUPS\_DISCOVERY\_PROTOCOL: JDBC\_PING JGROUPS\_DISCOVERY\_EXTERNAL\_IP: 10.6.6.6 JGROUPS\_BIND\_ADDR: 10.6.6.6 ports: - “80:8080” # Porta HTTP - “7800:7800” volumes: - ./certs:/opt/keycloak/certs:ro restart: always networks: - app-network Logs: keycloak01 | 2025-02-07 11:46:33,240 INFO [org.infinispan.CLUSTER] (main) ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping` keycloak01 | 2025-02-07 11:46:33,242 INFO [org.jgroups.JChannel] (main) local\_addr: a5d5d19a-1af4-46a3-8d4f-36f6f603b0c0, name: af13e6b70dd6-35000 keycloak01 | 2025-02-07 11:46:33,253 INFO [org.jgroups.protocols.FD\_SOCK2] (main) server listening on \*.57800 keycloak01 | 2025-02-07 11:46:33,269 INFO [org.jgroups.protocols.pbcast.GMS] (main) af13e6b70dd6-35000: no members discovered after 14 ms: creating cluster as coordinator keycloak01 | 2025-02-07 11:46:33,293 INFO [org.infinispan.CLUSTER] (main) ISPN000094: Received new cluster view for channel ISPN: [af13e6b70dd6-35000|0] (1) [af13e6b70dd6-35000] keycloak01 | 2025-02-07 11:46:33,393 INFO [org.infinispan.CLUSTER] (main) ISPN000079: Channel `ISPN` local address is `af13e6b70dd6-35000`, physical addresses are `[172.28.0.2:7800]` keycloak01 | 2025-02-07 11:46:33,904 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: af13e6b70dd6-35000, Site name: null keycloak01 | 2025-02-07 11:46:33,910 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener keycloak01 | 2025-02-07 11:46:35,637 WARN [io.agroal.pool] (main) Datasource ‘’: JDBC resources leaked: 3 ResultSet(s) and 0 Statement(s) keycloak01 | 2025-02-07 11:46:35,837 INFO [io.quarkus] (main) Keycloak 26.1.0 on JVM (powered by Quarkus 3.15.2) started in 10.822s. Listening on: [http://0.0.0.0:8080](http://0.0.0.0:8080) keycloak01 | 2025-02-07 11:46:35,838 INFO [io.quarkus] (main) Profile prod activated. keycloak01 | 2025-02-07 11:46:35,838 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, narayana-jta, opentelemetry, reactive-routes, rest, rest-jackson, smallrye-context-propagation, vertx] keycloak01 | 2025-02-07 11:46:36,227 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,af13e6b70dd6-35000) JGRP000006: 172.28.0.2:7800: failed accepting connection from peer Socket[addr=/168.63.129.16,port=58536,localport=7800]: java.net.SocketTimeoutException: Read timed out keycloak01 | 2025-02-07 11:46:51,232 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,af13e6b70dd6-35000) JGRP000006: 172.28.0.2:7800: failed accepting connection from peer Socket[addr=/168.63.129.16,port=58785,localport=7800]: java.net.SocketTimeoutException: Read timed out … With the variables: JGROUPS\_DISCOVERY\_EXTERNAL\_IP: 10.6.6.6 JGROUPS\_BIND\_ADDR: 10.6.6.6 Shouldn’t jdbc use these IPs instead of the internal IPs? Could someone help me? Thanks 😃 ---
**Author:** ![dasniko](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/dasniko/32/2969_2.png) [@dasniko](https://forum.keycloak.org/u/dasniko)\ **Post date:** [February 7, 2025, 12:39pm UTC](https://forum.keycloak.org/t/how-keycloak-works-in-a-cluster-with-cache/29553/2 "2025-02-07T12:39:41Z")
The `JGROUPS_...` variables are legacy and outdated, they won’t be used! Instead, add this env var: ```auto JAVA_OPTS_APPEND: -Djgroups.external_addr= ``` IMHO the `bind_addr` isn’t needed (I didn’t need to set them yet explicitly). But if, simply add a `-Djgroups.bind_addr=...` to the value of `JAVA_OPTS_APPEND`. ---
**Author:** ![dasniko](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/dasniko/32/2969_2.png) [@dasniko](https://forum.keycloak.org/u/dasniko)\ **Post date:** [February 7, 2025, 12:40pm UTC](https://forum.keycloak.org/t/how-keycloak-works-in-a-cluster-with-cache/29553/3 "2025-02-07T12:40:39Z")
Additionally, for proper error detection/handling, you should also open port 57800 between the nodes. ---
**Author:** ![alansatis](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/alansatis/32/11960_2.png) [@alansatis](https://forum.keycloak.org/u/alansatis)\ **Post date:** [February 7, 2025, 12:50pm UTC](https://forum.keycloak.org/t/how-keycloak-works-in-a-cluster-with-cache/29553/4 "2025-02-07T12:50:56Z")
Hi, Thank you in advance for your response. After these changes, the IP was finally recognized correctly. keycloak01 | 2025-02-07 12:43:59,446 INFO [org.infinispan.CLUSTER] (main) ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping` keycloak01 | 2025-02-07 12:43:59,448 INFO [org.jgroups.JChannel] (main) local\_addr: e7cadcf9-d33f-4315-bfe4-52d5c53debeb, name: cbcf7aab088f-20801 keycloak01 | 2025-02-07 12:43:59,461 INFO [org.jgroups.protocols.FD\_SOCK2] (main) server listening on \*.57800 keycloak01 | 2025-02-07 12:44:01,476 WARN [org.jgroups.protocols.pbcast.GMS] (main) cbcf7aab088f-20801: JOIN(cbcf7aab088f-20801) sent to 7b880fcfef00-7598 timed out (after 2000 ms), on try 0 keycloak01 | 2025-02-07 12:44:03,481 WARN [org.jgroups.protocols.pbcast.GMS] (main) cbcf7aab088f-20801: JOIN(cbcf7aab088f-20801) sent to 7b880fcfef00-7598 timed out (after 2000 ms), on try 1 keycloak01 | 2025-02-07 12:44:07,574 INFO [org.infinispan.CLUSTER] (main) ISPN000094: Received new cluster view for channel ISPN: [7b880fcfef00-7598|1] (2) [7b880fcfef00-7598, cbcf7aab088f-20801] keycloak01 | 2025-02-07 12:44:07,663 INFO [org.infinispan.CLUSTER] (main) ISPN000079: Channel `ISPN` local address is `cbcf7aab088f-20801`, physical addresses are `[10.6.6.5:7800]` keycloak01 | 2025-02-07 12:44:07,973 INFO [org.infinispan.LIFECYCLE] () [Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [7b880fcfef00-7598, cbcf7aab088f-20801], phase READ\_OLD\_WRITE\_ALL, topology id 2 keycloak01 | 2025-02-07 12:44:08,016 INFO [org.infinispan.LIFECYCLE] (non-blocking-thread–p2-t1) [Context=org.infinispan.CONFIG] ISPN100010: Finished rebalance with members [7b880fcfef00-7598, cbcf7aab088f-20801], topology id 2 keycloak01 | 2025-02-07 12:44:08,082 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,cbcf7aab088f-20801) JGRP000006: 10.6.6.5:7800: failed accepting connection from peer Socket[addr=/168.63.129.16,port=50894,localport=7800]: java.net.SocketTimeoutException: Read timed out keycloak01 | 2025-02-07 12:44:08,142 INFO [org.infinispan.LIFECYCLE] () [Context=actionTokens] ISPN100002: Starting rebalance with members [7b880fcfef00-7598, cbcf7aab088f-20801], phase READ\_OLD\_WRITE\_ALL, topology id 2 keycloak01 | 2025-02-07 12:44:08,163 INFO [org.infinispan.LIFECYCLE] () [Context=actionTokens] ISPN100010: Finished rebalance with members [7b880fcfef00-7598, cbcf7aab088f-20801], topology id 2 keycloak01 | 2025-02-07 12:44:08,297 INFO [org.infinispan.LIFECYCLE] () [Context=authenticationSessions] ISPN100002: Starting rebalance with members [7b880fcfef00-7598, cbcf7aab088f-20801], phase READ\_OLD\_WRITE\_ALL, topology id 2 VM-02: keycloak02 | 2025-02-07 12:42:06,987 INFO [org.infinispan.CLUSTER] (main) ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping` keycloak02 | 2025-02-07 12:42:06,990 INFO [org.jgroups.JChannel] (main) local\_addr: d2f108b9-da74-4f13-adbe-1db792f7b5ea, name: 7b880fcfef00-7598 keycloak02 | 2025-02-07 12:42:07,002 INFO [org.jgroups.protocols.FD\_SOCK2] (main) server listening on \*.57800 keycloak02 | 2025-02-07 12:42:08,075 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,7b880fcfef00-7598) JGRP000006: 10.6.6.6:7800: failed accepting connection from peer Socket[addr=/168.63.129.16,port=51360,localport=7800]: java.net.SocketTimeoutException: Read timed out keycloak02 | 2025-02-07 12:42:09,020 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 0 keycloak02 | 2025-02-07 12:42:11,025 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 1 keycloak02 | 2025-02-07 12:42:13,029 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 2 keycloak02 | 2025-02-07 12:42:15,034 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 3 keycloak02 | 2025-02-07 12:42:17,038 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 4 keycloak02 | 2025-02-07 12:42:19,046 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 5 keycloak02 | 2025-02-07 12:42:21,050 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 6 keycloak02 | 2025-02-07 12:42:23,054 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 7 keycloak02 | 2025-02-07 12:42:23,085 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,7b880fcfef00-7598) JGRP000006: 10.6.6.6:7800: failed accepting connection from peer Socket[addr=/168.63.129.16,port=51624,localport=7800]: java.net.SocketTimeoutException: Read timed out keycloak02 | 2025-02-07 12:42:25,059 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 8 keycloak02 | 2025-02-07 12:42:27,066 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: JOIN(7b880fcfef00-7598) sent to af13e6b70dd6-35000 timed out (after 2000 ms), on try 9 keycloak02 | 2025-02-07 12:42:27,066 WARN [org.jgroups.protocols.pbcast.GMS] (main) 7b880fcfef00-7598: too many JOIN attempts (10): becoming singleton keycloak02 | 2025-02-07 12:42:27,215 INFO [org.infinispan.CLUSTER] (main) ISPN000094: Received new cluster view for channel ISPN: [7b880fcfef00-7598|0] (1) [7b880fcfef00-7598] keycloak02 | 2025-02-07 12:42:27,310 INFO [org.infinispan.CLUSTER] (main) ISPN000079: Channel `ISPN` local address is `7b880fcfef00-7598`, physical addresses are `[10.6.6.6:7800]` keycloak02 | 2025-02-07 12:42:27,813 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: 7b880fcfef00-7598, Site name: null keycloak02 | 2025-02-07 12:42:27,818 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener keycloak02 | 2025-02-07 12:42:29,446 WARN [io.agroal.pool] (main) Datasource ‘’: JDBC resources leaked: 3 ResultSet(s) and 0 Statement(s) keycloak02 | 2025-02-07 12:42:29,649 INFO [io.quarkus] (main) Keycloak 26.1.0 on JVM (powered by Quarkus 3.15.2) started in 30.928s. Listening on: [http://0.0.0.0:8080](http://0.0.0.0:8080) keycloak02 | 2025-02-07 12:42:29,649 INFO [io.quarkus] (main) Profile prod activated. And in the database I already have both records with their respective IPs. keycloak=\> SELECT \* FROM public.JGROUPS\_PING; address | name | cluster\_name | ip | coord ---------------------------------------------±-------------------±-------------±--------------±------ uuid://ID | 7b880fcfef00-7598 | ISPN | 10.6.6.6:7800 | t uuid://ID | cbcf7aab088f-20801 | ISPN | 10.6.6.5:7800 | f But why do I still keep getting these time-outs? keycloak01 | 2025-02-07 12:49:53,247 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,cbcf7aab088f-20801) JGRP000006: 10.6.6.5:7800: failed accepting connection from peer Socket[addr=/168.63.129.16,port=56827,localport=7800]: java.net.SocketTimeoutException: Read timed out keycloak02: keycloak02 | 2025-02-07 12:44:08,633 INFO [org.infinispan.LIFECYCLE] (non-blocking-thread–p2-t1) [Context=work] ISPN100010: Finished rebalance with members [7b880fcfef00-7598, cbcf7aab088f-20801], topology id 2 keycloak02 | 2025-02-07 12:44:08,643 INFO [org.infinispan.CLUSTER] () [Context=work] ISPN100009: Advancing to rebalance phase READ\_ALL\_WRITE\_ALL, topology id 3 keycloak02 | 2025-02-07 12:44:08,647 INFO [org.infinispan.CLUSTER] () [Context=work] ISPN100009: Advancing to rebalance phase READ\_NEW\_WRITE\_ALL, topology id 4 keycloak02 | 2025-02-07 12:44:08,650 INFO [org.infinispan.CLUSTER] () [Context=work] ISPN100010: Finished rebalance with members [7b880fcfef00-7598, cbcf7aab088f-20801], topology id 5 keycloak02 | 2025-02-07 12:44:23,186 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7800]-1,7b880fcfef00-7598) JGRP000006: 10.6.6.6:7800: failed accepting connection from peer Socket[addr=/168.63.129.16,port=53554,localport=7800]: java.net.SocketTimeoutException: Read timed out tkx ---
**Author:** ![dasniko](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/dasniko/32/2969_2.png) [@dasniko](https://forum.keycloak.org/u/dasniko)\ **Post date:** [February 7, 2025, 1:11pm UTC](https://forum.keycloak.org/t/how-keycloak-works-in-a-cluster-with-cache/29553/5 "2025-02-07T13:11:12Z")
Don’t know, most probably there’s something on your network, which blocks proper communication between the nodes. ---
**Author:** ![alansatis](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/alansatis/32/11960_2.png) [@alansatis](https://forum.keycloak.org/u/alansatis)\ **Post date:** [February 7, 2025, 2:15pm UTC](https://forum.keycloak.org/t/how-keycloak-works-in-a-cluster-with-cache/29553/6 "2025-02-07T14:15:00Z")
I’ve tested it and now, creating realms, users… in the instances separately, and it seems that everything is synchronized. The IP is also correct. Now I’m going to open port 57800 and check the details related to the time-out. One last question, is there a place where I can check the documentation that talks about these variables and the changes that will occur? Thanks for your help. 😃