# Experiencing Infinispan Timeouts

**URL:** <https://forum.keycloak.org/t/experiencing-infinispan-timeouts/4456>\
**Category:** Getting advice\
**Created:** [August 24, 2020, 10:46pm UTC](https://forum.keycloak.org/t/experiencing-infinispan-timeouts/4456 "2020-08-24T22:46:16Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![eloyot](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@eloyot](https://forum.keycloak.org/u/eloyot)\
**Post date:** [August 24, 2020, 10:46pm UTC](https://forum.keycloak.org/t/experiencing-infinispan-timeouts/4456/1 "2020-08-24T22:46:17Z")

</div>

We have been running Keycloak on AWS as an ECS application for about 3 months now. We occasionally experience infinispan timeouts (shown below) when under load. These cause a spate of login errors until the system recovers. Our infinispan stack is also shown below. Does anyone have any thoughts as to how we might debug this. We have reviewed other posts in this forum but nothing has helped so far. Thanks!

Error:

```auto
2020-08-09 01:44:13,170 ERROR [org.keycloak.services.error.KeycloakErrorHandler] (default task-1957) env=dev node_ip=10.116.53.26 ecs_cluster_name=keycloak-service-cluster ecs_service_name=keycloak-service Uncaught server error: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 30113426 from ip-10-116-52-251
	at org.infinispan@9.4.16.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
	at org.infinispan@9.4.16.Final//org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1918)
	at org.infinispan@9.4.16.Final//org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1474)
	at org.infinispan@9.4.16.Final//org.infinispan.cache.impl.DecoratedCache.putIfAbsent(DecoratedCache.java:695)

```

Infinispan stack configuration:

```auto
                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="JDBC_PING">
                        <property name="datasource_jndi_name">java:jboss/datasources/KeycloakDS
                        </property>
                        <property name="remove_old_coords_on_view_change">true</property>
                        <property name="remove_all_data_on_view_change">true</property>
                        <property name="initialize_sql">
                            CREATE TABLE IF NOT EXISTS JGROUPSPING (
                            own_addr varchar(200) NOT NULL,
                            bind_addr varchar(200) NOT NULL,
                            created timestamp NOT NULL,
                            cluster_name varchar(200) NOT NULL,
                            ping_data BYTEA,
                            constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name)
                            )
                        </property>
                        <property name="insert_single_sql">INSERT INTO JGROUPSPING (own_addr, bind_addr, created, cluster_name, ping_data) values (?, '${jboss.bind.address.private:UNKNOWN}', NOW(), ?, ?)</property>
                        <property name="delete_single_sql">DELETE FROM JGROUPSPING WHERE own_addr=? AND cluster_name=?</property>
                        <property name="select_all_pingdata_sql">SELECT ping_data, own_addr, cluster_name FROM JGROUPSPING WHERE cluster_name=?</property>
                    </protocol>
                    <protocol type="MERGE3"/>
                    <socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>
            </stacks>

```

---

<div class="post-metadata">

**Author:** ![sre.ian](https://avatars.discourse-cdn.com/v4/letter/s/dc4da7/32.png) [@sre.ian](https://forum.keycloak.org/u/sre.ian)\
**Post date:** [July 28, 2021, 4:19pm UTC](https://forum.keycloak.org/t/experiencing-infinispan-timeouts/4456/2 "2021-07-28T16:19:12Z")

</div>

Are you setup to use a cross-datacenter remote infinispan setup? So you have keycloak nodes and separate infinispan nodes?

The configuration you posted is jgroups TCP [jDBC\_PING](http://www.jgroups.org/manual/html/protlist.html#d0e5196) confuguration for cluster discovery. It’s used as a way for new keycloak nodes to discover existing nodes in the cluster to sync cache from.

There’s configuration for state-transfer timeouts for caches

> <https://github.com/wildfly/wildfly/blob/master/clustering/infinispan/extension/src/main/resources/schema/jboss-as-infinispan_4_0.xsd#L739>

```auto
<replicated-cache name="offlineSessions">
      <!---- default settings ---->
     <state-transfer timeout="240000" chunk-size="512"/>
</replicated-cache>

```

---

<div class="post-metadata">

**Author:** ![williamye](https://avatars.discourse-cdn.com/v4/letter/w/91b2a8/32.png) [@williamye](https://forum.keycloak.org/u/williamye)\
**Post date:** [December 21, 2021, 7:24am UTC](https://forum.keycloak.org/t/experiencing-infinispan-timeouts/4456/3 "2021-12-21T07:24:04Z")

</div>

Hi Eloyot,

I got the same issue, so how did you fix it?