We have recently updated our Keycloak instance to 22.0.5. Randomly we are seeing below errors in one of the Keycloak pods
keycloak-3 keycloak 2024-02-22 09:56:01,433 WARN [org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction] (executor-thread-26047) Failed to replace entity 'e7fcaab4-975c-4cfd-ac17-4f4bd56f6e19' in cache 'clientSessions'
keycloak-3 keycloak 2024-02-22 09:56:01,436 WARN [org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction] (executor-thread-26063) Failed to replace entity 'e7fcaab4-975c-4cfd-ac17-4f4bd56f6e19' in cache 'clientSessions'
keycloak-3 keycloak 2024-02-22 09:56:01,440 WARN [org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction] (executor-thread-26050) Failed to replace entity 'e7fcaab4-975c-4cfd-ac17-4f4bd56f6e19' in cache 'clientSessions'
keycloak-3 keycloak 2024-02-22 09:56:01,452 WARN [org.keycloak.models.sessions.infinispan.changes.InfinispanChangelogBasedTransaction] (executor-thread-26064) Failed to replace entity 'e7fcaab4-975c-4cfd-ac17-4f4bd56f6e19' in cache 'clientSessions'
This is followed by the pod becoming unavailable because of readiness probe failure and after sometime this is followed by readiness probe failures in all pods. After a minute or two readiness probes succeed again and Keycloak works normally. We observed this happening around once a week.
We are using codecentric helm chart with 5 nodes in the cluster with embedded cache. During this error I do not see any issue with memory consumption
Related GitHub discussion: Keycloak freezes after cache warnings · keycloak/keycloak · Discussion #16405 · GitHub
Related GitHub issue Failed to replace entity Warnings in KC 22 · Issue #23634 · keycloak/keycloak · GitHub closed as not reproducible.
I see one possible root cause here Setup for storing large volume of durable sessions · keycloak/keycloak · Discussion #25170 · GitHub which I did not understood completely.
What can be the reason for this? How can I avoid the cascading failure of readiness probe in all pods? Is it possible to tweak the probe to ignore this warning?