Hello, i have two keycloak servers, each on its own vm runninf in docker and one database server. I’m using icdb ping. it works for long time and after update to 24, the second instance refuse to connect .
Please couldanyone give me some advice?
[root@keycloak2 keycloak]# docker compose up
[+] Building 0.0s (0/0) [+] Running 1/0
✔ Container keycloak-keycloak-1 Running 0.0s Attaching to keycloak-keycloak-1
keycloak-keycloak-1 | 2024-03-20 10:12:21,462 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
keycloak-keycloak-1 | 2024-03-20 10:12:21,725 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: aai.it4i.cz, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: true
keycloak-keycloak-1 | 2024-03-20 10:12:21,829 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000088: Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
keycloak-keycloak-1 | 2024-03-20 10:12:22,125 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
keycloak-keycloak-1 | 2024-03-20 10:12:22,129 INFO [org.jgroups.JChannel] (keycloak-cache-init) local_addr: 7aae64be-e755-43d9-acc4-83e38bd8c4c5, name: f24fc2dbd4c5-62278
keycloak-keycloak-1 | 2024-03-20 10:12:22,161 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
keycloak-keycloak-1 | 2024-03-20 10:12:22,161 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 20MB, but the OS only allocated 212.99KB
keycloak-keycloak-1 | 2024-03-20 10:12:22,161 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB
keycloak-keycloak-1 | 2024-03-20 10:12:22,161 WARN [org.jgroups.protocols.UDP] (keycloak-cache-init) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB
keycloak-keycloak-1 | 2024-03-20 10:12:22,231 INFO [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.23728
keycloak-keycloak-1 | 2024-03-20 10:12:24,295 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) f24fc2dbd4c5-62278: no members discovered after 2019 ms: creating cluster as coordinator
keycloak-keycloak-1 | 2024-03-20 10:12:24,320 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [f24fc2dbd4c5-62278|0] (1) [f24fc2dbd4c5-62278]
keycloak-keycloak-1 | 2024-03-20 10:12:24,350 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `f24fc2dbd4c5-62278`, physical addresses are `[172.19.0.2:39264]`
keycloak-keycloak-1 | 2024-03-20 10:12:24,364 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
keycloak-keycloak-1 | 2024-03-20 10:12:24,469 WARN [io.quarkus.agroal.runtime.DataSources] (JPA Startup Thread) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
keycloak-keycloak-1 | 2024-03-20 10:12:29,652 WARN [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread) HHH000342: Could not obtain connection to query metadata: java.lang.NullPointerException: Cannot invoke "org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(java.sql.SQLException, String)" because the return value of "org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.sqlExceptionHelper()" is null
keycloak-keycloak-1 | at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doTheWork(JtaIsolationDelegate.java:186)
keycloak-keycloak-1 | at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.lambda$delegateWork$1(JtaIsolationDelegate.java:75)
keycloak-keycloak-1 | at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doInSuspendedTransaction(JtaIsolationDelegate.java:107)
keycloak-keycloak-1 | at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.delegateWork(JtaIsolationDelegate.java:72)
keycloak-keycloak-1 | at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentUsingJdbcMetadata(JdbcEnvironmentInitiator.java:276)
keycloak-keycloak-1 | at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:107)
keycloak-keycloak-1 | at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:68)
keycloak-keycloak-1 | at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:130)
keycloak-keycloak-1 | at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263)
keycloak-keycloak-1 | at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:238)
keycloak-keycloak-1 | at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:215)
keycloak-keycloak-1 | at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:52)
keycloak-keycloak-1 | at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:136)
keycloak-keycloak-1 | at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:247)
keycloak-keycloak-1 | at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:215)
keycloak-keycloak-1 | at org.hibernate.boot.internal.SessionFactoryOptionsBuilder.<init>(SessionFactoryOptionsBuilder.java:276)
keycloak-keycloak-1 | at io.quarkus.hibernate.orm.runtime.recording.PrevalidatedQuarkusMetadata.buildSessionFactoryOptionsBuilder(PrevalidatedQuarkusMetadata.java:70)
keycloak-keycloak-1 | at io.quarkus.hibernate.orm.runtime.boot.FastBootEntityManagerFactoryBuilder.build(FastBootEntityManagerFactoryBuilder.java:84)
keycloak-keycloak-1 | at io.quarkus.hibernate.orm.runtime.FastBootHibernatePersistenceProvider.createEntityManagerFactory(FastBootHibernatePersistenceProvider.java:72)
keycloak-keycloak-1 | at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:80)
keycloak-keycloak-1 | at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
keycloak-keycloak-1 | at io.quarkus.hibernate.orm.runtime.JPAConfig$LazyPersistenceUnit.get(JPAConfig.java:157)
keycloak-keycloak-1 | at io.quarkus.hibernate.orm.runtime.JPAConfig$1.run(JPAConfig.java:64)
keycloak-keycloak-1 | at java.base/java.lang.Thread.run(Thread.java:840)
keycloak-keycloak-1 | 2024-03-20 10:12:30,619 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: f24fc2dbd4c5-62278, Site name: null
keycloak-keycloak-1 | 2024-03-20 10:12:30,622 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
keycloak-keycloak-1 | 2024-03-20 10:12:35,670 INFO [org.infinispan.CLUSTER] (main) ISPN000080: Disconnecting JGroups channel `ISPN`
keycloak-keycloak-1 | 2024-03-20 10:12:35,703 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
keycloak-keycloak-1 | 2024-03-20 10:12:35,704 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) Error details:: java.lang.RuntimeException: Failed to obtain JDBC connection
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.storage.database.jpa.AbstractJpaConnectionProviderFactory.getConnection(AbstractJpaConnectionProviderFactory.java:56)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.storage.legacy.database.QuarkusJpaConnectionProviderFactory.postInit(QuarkusJpaConnectionProviderFactory.java:111)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.integration.QuarkusKeycloakSessionFactory.init(QuarkusKeycloakSessionFactory.java:105)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.createSessionFactory(QuarkusKeycloakApplication.java:56)
keycloak-keycloak-1 | at org.keycloak.services.resources.KeycloakApplication.startup(KeycloakApplication.java:131)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication.onStartupEvent(QuarkusKeycloakApplication.java:46)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.integration.jaxrs.QuarkusKeycloakApplication_Observer_onStartupEvent_GNZ8m5QenZ9h9VNelo7awjUZFDE.notify(Unknown Source)
keycloak-keycloak-1 | at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346)
keycloak-keycloak-1 | at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:328)
keycloak-keycloak-1 | at io.quarkus.arc.impl.EventImpl.fire(EventImpl.java:82)
keycloak-keycloak-1 | at io.quarkus.arc.runtime.ArcRecorder.fireLifecycleEvent(ArcRecorder.java:155)
keycloak-keycloak-1 | at io.quarkus.arc.runtime.ArcRecorder.handleLifecycleEvents(ArcRecorder.java:106)
keycloak-keycloak-1 | at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy_0(Unknown Source)
keycloak-keycloak-1 | at io.quarkus.deployment.steps.LifecycleEventsBuildStep$startupEvent1144526294.deploy(Unknown Source)
keycloak-keycloak-1 | at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
keycloak-keycloak-1 | at io.quarkus.runtime.Application.start(Application.java:101)
keycloak-keycloak-1 | at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
keycloak-keycloak-1 | at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.KeycloakMain.start(KeycloakMain.java:117)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.cli.command.AbstractStartCommand.run(AbstractStartCommand.java:33)
keycloak-keycloak-1 | at picocli.CommandLine.executeUserObject(CommandLine.java:2026)
keycloak-keycloak-1 | at picocli.CommandLine.access$1500(CommandLine.java:148)
keycloak-keycloak-1 | at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
keycloak-keycloak-1 | at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
keycloak-keycloak-1 | at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
keycloak-keycloak-1 | at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
keycloak-keycloak-1 | at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
keycloak-keycloak-1 | at picocli.CommandLine.execute(CommandLine.java:2170)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.cli.Picocli.parseAndRun(Picocli.java:125)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.KeycloakMain.main(KeycloakMain.java:107)
keycloak-keycloak-1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
keycloak-keycloak-1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
keycloak-keycloak-1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
keycloak-keycloak-1 | at java.base/java.lang.reflect.Method.invoke(Method.java:568)
keycloak-keycloak-1 | at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:62)
keycloak-keycloak-1 | at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:33)
keycloak-keycloak-1 | Caused by: java.sql.SQLException: Acquisition timeout while waiting for new connection
keycloak-keycloak-1 | at io.agroal.pool.ConnectionPool.handlerFromSharedCache(ConnectionPool.java:321)
keycloak-keycloak-1 | at io.agroal.pool.ConnectionPool.getConnection(ConnectionPool.java:249)
keycloak-keycloak-1 | at io.agroal.pool.DataSource.getConnection(DataSource.java:86)
keycloak-keycloak-1 | at io.quarkus.hibernate.orm.runtime.customized.QuarkusConnectionProvider.getConnection(QuarkusConnectionProvider.java:23)
keycloak-keycloak-1 | at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator$ConnectionProviderJdbcConnectionAccess.obtainConnection(JdbcEnvironmentInitiator.java:428)
keycloak-keycloak-1 | at org.keycloak.quarkus.runtime.storage.database.jpa.AbstractJpaConnectionProviderFactory.getConnection(AbstractJpaConnectionProviderFactory.java:54)
keycloak-keycloak-1 | ... 35 more
keycloak-keycloak-1 | Caused by: java.util.concurrent.TimeoutException
keycloak-keycloak-1 | at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:204)
keycloak-keycloak-1 | at io.agroal.pool.ConnectionPool.handlerFromSharedCache(ConnectionPool.java:298)
keycloak-keycloak-1 | ... 40 more
keycloak-keycloak-1 |
keycloak-keycloak-1 exited with code 0
this is my cache-ispn-jdbc-ping.xml - NOT ENTIRE
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
xmlns="urn:infinispan:config:11.0">
<jgroups>
<stack name="postgres-jdbc-ping-tcp" extends="tcp">
<TCP external_addr="${env.EXTERNAL_IP}"/>
<JDBC_PING connection_driver="org.postgresql.Driver"
connection_username="${env.KC_DB_USERNAME}" connection_password="${env.KC_DB_PASSWORD}"
connection_url="jdbc:postgresql://${env.KC_DB_URL_HOST}/${env.KC_DB_URL_DATABASE}"
initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, ping_data BYTEA, constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
info_writer_sleep_time="500"
remove_all_data_on_view_change="true"
stack.combine="REPLACE"
stack.position="MPING" />
</stack>
</jgroups>
<cache-container name="keycloak">
<transport lock-timeout="60000" stack="${env.KC_DB}-jdbc-ping-tcp"/>
<local-cache name="realms">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/> </local-cache> <local-cache name="users"> <encoding> <key media-type="application/x-java-object"/> <value media-type="application/x-java-object"/> </encoding> <memory max-count="10000"/> </local-cache> <distributed-cache name="sessions" owners="2"> <expiration lifespan="-1"/> </distributed-cache>
EXTERNAL_IP - is ip of server.
db is postgres, ports are open, firewall is disabled.