Does Keycloak restart listeners for remote `work` cache after recreate of cache?

Cross-posting from Does Keycloak restart listeners for remote `work` cache after recreate of cache? · keycloak/keycloak · Discussion #37472 · GitHub

The setup is a group of Keycloak pods connecting to a remote Infinispan cache cluster that’s been created by the Infinispan operator (all current versions).

  1. Destroy and recreate the work cache on the Infinispan cluster
  2. See the re-connection of the Keycloak pods to the work cache in the logs
  3. Direct connect to the admin UI on one of the pods. Make a Realm or User change.
  4. Direct connect to the admin UI on a different pod. The Realm or User change is not propagated.
  5. Restart the Keycloak pods
  6. Do 3 and 4 again, and the change is propagated

I noticed from the RemoteInfinispanNotificationManager that the listener is only registered once to the work cache. However, I don’t know if this is the culprit of the above behavior (i.e. that is the expected behavior if the cache goes away, and won’t receive events even if it comes back), or if there is some other configuration issue.

Has anyone see this before? Can you comment on the expected behavior?

Also, does anyone know of a way to debug this on the Infinispan side?

Thanks!

Found the answer from the Infinispan team Public view of Infinispan | Zulip team chat

If you destroy the cache, all listeners will be destroyed as well. The server needs to store which clients have listeners in a specific cache, and this info is destroyed when the cache is destroyed.

If there is a network issue or a broken connection, the client will retry to install the listener.