Hi,
I have just upgraded to Keycloak 26.0.0 which I am running on Kubernetes inside the codecentric/keycloakx helm chart ( found here https://codecentric.github.io/helm-charts ).
I am getting some weird errors when I try and add or remove a user from an organization. This used to work without a problem in Keycloak 25. It seems to have something to with caching as far as I can tell. The error also only occurs when there is more than one replica deployed. If I have a single replica, or run Keycloak locally in development mode there are no errors.
Any help or guidance would be appreciated.
2024-10-10 11:00:28,247 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (jgroups-11,keycloak26-keycloakx-0-47006) ISPN000136: Error executing command PutKeyValueCommand on Cache ‘work’, writing keys [43d94b4f-347f-44bd-9b34-d11e24f71df3]: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from keycloak26-keycloakx-1-38314, see cause for remote stack trace
at org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:26)
at org.infinispan.remoting.transport.ValidSingleResponseCollector.withException(ValidSingleResponseCollector.java:37)
at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:21)
at org.infinispan.remoting.transport.impl.SingleTargetRequest.addResponse(SingleTargetRequest.java:75)
at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:46)
at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1556)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1457)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1649)
at org.jgroups.JChannel.up(JChannel.java:748)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:941)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:139)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:245)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:245)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:853)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:235)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1144)
at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:880)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:862)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:669)
at org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:105)
at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180)
at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:190)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:274)
at org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups.stack.Protocol.up(Protocol.java:360)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1230)
at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:95)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.ClassCastException] while invoking method [public void org.keycloak.cluster.infinispan.InfinispanNotificationsManager$CacheEntryListener.cacheEntryCreated(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.keycloak.cluster.infinispan.InfinispanNotificationsManager$CacheEntryListener@721fd59c
at org.infinispan.marshall.exts.ThrowableExternalizer.readMessageAndCause(ThrowableExternalizer.java:257)
at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:184)
at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:44)
at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49)
at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41)
at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191)
at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1548)
… 25 more
Caused by: java.lang.ClassCastException: class org.keycloak.models.cache.infinispan.events.CacheKeyInvalidatedEvent cannot be cast to class org.keycloak.models.cache.infinispan.events.UserCacheInvalidationEvent (org.keycloak.models.cache.infinispan.events.CacheKeyInvalidatedEvent and org.keycloak.models.cache.infinispan.events.UserCacheInvalidationEvent are in unnamed module of loader io.quarkus.bootstrap.runner.RunnerClassLoader @7e6cbb7a)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.infinispan.marshall.exts.ThrowableExternalizer.newThrowableInstance(ThrowableExternalizer.java:286)
at org.infinispan.marshall.exts.ThrowableExternalizer.readGenericThrowable(ThrowableExternalizer.java:264)
at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:239)
at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:44)
at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
at org.infinispan.marshall.exts.ThrowableExternalizer.readMessageAndCause(ThrowableExternalizer.java:256)
… 39 more
2024-10-10 11:00:28,249 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (executor-thread-1) Uncaught server error: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from keycloak26-keycloakx-1-38314, see cause for remote stack trace
at org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:26)
at org.infinispan.remoting.transport.ValidSingleResponseCollector.withException(ValidSingleResponseCollector.java:37)
at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:21)
at org.infinispan.remoting.transport.impl.SingleTargetRequest.addResponse(SingleTargetRequest.java:75)
at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:46)
at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1556)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1457)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1649)
at org.jgroups.JChannel.up(JChannel.java:748)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:941)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:139)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:245)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:245)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:853)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:235)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1144)
at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:880)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:862)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:669)
at org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:105)
at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180)
at org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:190)
at org.jgroups.protocols.MERGE3.up(MERGE3.java:274)
at org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups.stack.Protocol.up(Protocol.java:360)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1230)
at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:95)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Suppressed: org.infinispan.commons.util.logging.TraceException
at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:39)
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:240)
at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:329)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:112)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:94)
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1303)
at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:687)
at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:212)
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:121)
at org.infinispan.cache.impl.AbstractDelegatingCache.put(AbstractDelegatingCache.java:121)
at org.infinispan.cache.impl.EncoderCache.put(EncoderCache.java:191)
at org.keycloak.cluster.infinispan.InfinispanNotificationsManager.notify(InfinispanNotificationsManager.java:158)
at org.keycloak.cluster.infinispan.InfinispanClusterProvider.notify(InfinispanClusterProvider.java:135)
at org.keycloak.models.cache.infinispan.CacheManager.sendInvalidationEvents(CacheManager.java:211)
at org.keycloak.models.cache.infinispan.UserCacheSession.runInvalidations(UserCacheSession.java:156)
at org.keycloak.models.cache.infinispan.UserCacheSession$1.commit(UserCacheSession.java:168)
at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:146)
at org.keycloak.services.DefaultKeycloakSession.closeTransactionManager(DefaultKeycloakSession.java:414)
at org.keycloak.services.DefaultKeycloakSession.close(DefaultKeycloakSession.java:379)
at org.keycloak.models.KeycloakBeanProducer_ProducerMethod_getKeycloakSession_XoSEUTXOsE3bpqXlGMAykCiECUM_ClientProxy.close(Unknown Source)
at org.keycloak.quarkus.runtime.transaction.TransactionalSessionHandler.close(TransactionalSessionHandler.java:60)
at org.keycloak.quarkus.runtime.integration.jaxrs.CloseSessionFilter.closeSession(CloseSessionFilter.java:67)
at org.keycloak.quarkus.runtime.integration.jaxrs.CloseSessionFilter.filter(CloseSessionFilter.java:63)
at org.jboss.resteasy.reactive.server.handlers.ResourceResponseFilterHandler.handle(ResourceResponseFilterHandler.java:25)
at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:150)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:635)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
… 1 more
Caused by: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.ClassCastException] while invoking method [public void org.keycloak.cluster.infinispan.InfinispanNotificationsManager$CacheEntryListener.cacheEntryCreated(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.keycloak.cluster.infinispan.InfinispanNotificationsManager$CacheEntryListener@721fd59c
at org.infinispan.marshall.exts.ThrowableExternalizer.readMessageAndCause(ThrowableExternalizer.java:257)
at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:184)
at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:44)
at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49)
at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41)
at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191)
at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1548)
… 25 more
Caused by: java.lang.ClassCastException: class org.keycloak.models.cache.infinispan.events.CacheKeyInvalidatedEvent cannot be cast to class org.keycloak.models.cache.infinispan.events.UserCacheInvalidationEvent (org.keycloak.models.cache.infinispan.events.CacheKeyInvalidatedEvent and org.keycloak.models.cache.infinispan.events.UserCacheInvalidationEvent are in unnamed module of loader io.quarkus.bootstrap.runner.RunnerClassLoader @7e6cbb7a)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at org.infinispan.marshall.exts.ThrowableExternalizer.newThrowableInstance(ThrowableExternalizer.java:286)
at org.infinispan.marshall.exts.ThrowableExternalizer.readGenericThrowable(ThrowableExternalizer.java:264)
at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:239)
at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:44)
at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
at org.infinispan.marshall.exts.ThrowableExternalizer.readMessageAndCause(ThrowableExternalizer.java:256)
… 39 more