Does Keycloak still create embedded infinispan cluster when using remote infinispan server

Getting advice
, ,
1

Hello there -
I am running Keycloak 23.0.3, 2 pods in Kubernetes and it uses remote infinispan cluster for sessions and client sessions cache. Below is my infinispan config xml. I am using 2 owners for remote sessions/client sessions caches as well as other distributed caches. When I do rolling restart for keycloak, I am noticing it is doing a state transfer. I am wondering if it is because I set 2 owners for remote caches and should actually be 1 as sessions are maintained in remote cluster. Also, does keycloak always create an embedded infinispan cluster even if using remote infinispan cluster? May be it does as I have owners set to 2 for other distributed caches like authentication sessions which isn’t a remote cache. State transfer is timing out during deployments and although I can increase the timeout, I am wondering if I can disable it for the remote caches.

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:13.0 http://www.infinispan.org/schemas/infinispan-config-13.0.xsd"
        xmlns="urn:infinispan:config:13.0">

    <cache-container name="keycloak">
        <transport lock-timeout="60000"/>
        <local-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="2">
            <expiration lifespan="-1"/>
            <remote-store cache="sessions" xmlns="urn:infinispan:config:store:remote:13.0"
                          fetch-state="false"
                          purge="false"
                          preload="false"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${infinispan.bind.port:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.REMOTE_ISPN_USERNAME}"
                                password="${env.REMOTE_ISPN_PASSWORD}"
                                realm="default"/>
                    </authentication>
                </security>
            </remote-store>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="2">
            <expiration lifespan="-1"/>
            <remote-store cache="clientSessions" xmlns="urn:infinispan:config:store:remote:13.0"
                          fetch-state="false"
                          purge="false"
                          preload="false"
                          shared="true" segmented="false"
                          connect-timeout="${env.KEYCLOAK_REMOTE_ISPN_CONN_TIMEOUT:2000}"
                          raw-values="true"
                          marshaller="org.keycloak.cluster.infinispan.KeycloakHotRodMarshallerFactory">
                <remote-server host="${env.INFINISPAN_HOST:infinispan}" port="${infinispan.bind.port:11222}"/>
                <security>
                    <authentication>
                        <digest username="${env.REMOTE_ISPN_USERNAME}"
                                password="${env.REMOTE_ISPN_PASSWORD}"
                                realm="default"/>
                    </authentication>
                </security>
            </remote-store>
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <local-cache name="authorization">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
        </replicated-cache>
        <local-cache name="keys">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <distributed-cache name="actionTokens" owners="2">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <memory max-count="-1"/>
        </distributed-cache>
    </cache-container>
</infinispan>
2

Why do you use an external Infinispan cluster with your Keycloak cluster?
For session persistence?

If yes, then I strongly recommend to update to the most recent 26.x version, as it has built-in session persistence and improved cache handling. And not only because of persistent sessions you should always upgrade to the most recent version. v23 is approx. 2 years old, outdated and receives no more update. There are a lot security issues with old versions.

Also, when using current versions of Keycloak, it’s not recommended to use a custom XML file for cache configuration, as you can configure the important settings through Keycloak config directly.

Again: this has really improved a lot in the past and you should give it a try!!
An external Infinispan cluster is only required if you want to have a multi-region cluster or if you disable persistent sessions in Keycloak and you want to have save sessions across server restarts. But in most cases, there’s no real requirement to to disable persistent sessions.
And yes, Keycloak will aways create embedded infinispan servers, which replicate with the external ones. You can’t get completely rid off the local Infinispan, unless you would maintain your custom Keycloak fork. (which is not desired, most likely)

3

@dasniko - thanks for response, yes I started using external infinispan starting keycloak v19 I believe. I noticed default persistence support in latest versions but to use that we’ve to let go of current sessions in external infinispan and everyone needs to re-login, timing hasn’t been right for us to do so. Thanks for confirming there is an embedded cluster as well, that helps my understanding.