Hi,
We are running a Keycloak 3.4.3, connecting to an old MySQL v5.5.59, the VM where is runs was upgraded from Debian 8 to Debian 9 a couple of months ago. Yesterday, we started getting some errors, I guess the 2 cores and 4GB of RAM was not good enough.
CPU: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
RAM was 4GB, now it is setup to 6GB, and 1 CPU 2 cores, now it is setup with 4 cores.
uname -a
Linux tor-keycloak 4.9.0-15-amd64 #1 SMP Debian 4.9.258-1 (2021-03-08) x86_64 GNU/Linux
lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.13 (stretch)
Release: 9.13
Codename: stretch
The way we run Keycloak is:
Environment=JBOSS_HOME=/opt/www/keycloak JBOSS_LOG_DIR=/opt/www/keycloak/standalone/log “JAVA_OPTS=-Xms1024m -Xmx2048m -XX:MaxPermSize=768m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true”
java -version
java version “1.8.0_162”
Java™ SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot™ 64-Bit Server VM (build 25.162-b12, mixed mode)
The Exception we got form the logs files “a bunch of exception like this”:
2021-06-06 08:03:58,753 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000001:-7354d2b:60a1615a:48751c in state RUN
2021-06-06 08:03:58,756 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffff7f000001:-7354d2b:60a1615a:48751c invoked while multiple threads active within it.
2021-06-06 08:03:58,758 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012381: Action id 0:ffff7f000001:-7354d2b:60a1615a:48751c completed with multiple threads - thread default task-3 was in progress with sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:138)
org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
org.keycloak.connections.httpclient.DefaultHttpClientFactory$1.get(DefaultHttpClientFactory.java:86)
org.keycloak.protocol.oidc.utils.JWKSHttpUtils.sendJwksRequest(JWKSHttpUtils.java:39)
org.keycloak.keys.loader.ClientPublicKeyLoader.loadKeys(ClientPublicKeyLoader.java:65)
org.keycloak.keys.infinispan.InfinispanPublicKeyStorageProvider$WrapperCallable.call(InfinispanPublicKeyStorageProvider.java:221)
org.keycloak.keys.infinispan.InfinispanPublicKeyStorageProvider$WrapperCallable.call(InfinispanPublicKeyStorageProvider.java:201)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
org.keycloak.keys.infinispan.InfinispanPublicKeyStorageProvider.getPublicKey(InfinispanPublicKeyStorageProvider.java:151)
org.keycloak.keys.loader.PublicKeyStorageManager.getClientPublicKey(PublicKeyStorageManager.java:45)
com.alertdriving.jwts.SignedJWTAuth.getSignatureValidationKey(SignedJWTAuth.java:203)
com.alertdriving.jwts.SignedJWTAuth.authenticate(SignedJWTAuth.java:112)
org.keycloak.authentication.DefaultAuthenticationFlow.processFlow(DefaultAuthenticationFlow.java:200)
org.keycloak.authentication.AuthenticationProcessor.authenticateOnly(AuthenticationProcessor.java:853)
org.keycloak.authentication.AuthenticationProcessor.authenticate(AuthenticationProcessor.java:722)
org.keycloak.protocol.AuthorizationEndpointBase.handleBrowserAuthenticationRequest(AuthorizationEndpointBase.java:145)
org.keycloak.protocol.oidc.endpoints.AuthorizationEndpoint.buildAuthorizationCodeAuthorizationResponse(AuthorizationEndpoint.java:395)
org.keycloak.protocol.oidc.endpoints.AuthorizationEndpoint.build(AuthorizationEndpoint.java:139)
sun.reflect.GeneratedMethodAccessor441.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138)
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107)
org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133)
org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101)
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406)
org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)
org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
org.keycloak.services.filters.KeycloakSessionServletFilter.doFilter(KeycloakSessionServletFilter.java:90)
io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction$$Lambda$990/933554452.call(Unknown Source)
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$991/1602428963.call(Unknown Source)
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$991/1602428963.call(Unknown Source)
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$991/1602428963.call(Unknown Source)
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction$$Lambda$991/1602428963.call(Unknown Source)
io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
io.undertow.server.Connectors.executeRootHandler(Connectors.java:326)
io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2021-06-06 08:03:58,758 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff7f000001:-7354d2b:60a1615a:48751c aborting with 1 threads active!
Any advice? how to assign more RAM to Keycloak, update the Java version?