# Keycloak Cluster JGroups/Infinispan on EC2

**URL:** <https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208>\
**Category:** Configuring the server\
**Created:** [November 1, 2022, 6:24am UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208 "2022-11-01T06:24:56Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![paulkilla](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/paulkilla/32/1045_2.png) [@paulkilla](https://forum.keycloak.org/u/paulkilla)\
**Post date:** [November 1, 2022, 6:24am UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/1 "2022-11-01T06:24:56Z")

</div>

Hello, we are running keycloak on EC2 VMs and want to run in currently a cluster of 2.

We are installing Keycloak 19, and attempting to setup clustering using S3.

I have in my keycloak.conf file the following entries in relation to the cache:

```auto
cache=ispn
cache-config-file=cache-ec2.xml
cache-stack=ec2

```

But I’m struggling to configure the cache-ec2.xml file. This is the current contents for that:

```auto
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
    xmlns="urn:infinispan:config:11.0">

  <!-- custom stack goes into the jgroups element -->
  <jgroups>
	<stack name="s3"> 
		<transport type="TCP" socket-binding="jgroups-tcp"/> 
		<protocol type="S3_PING" location="pauls-test-keycloak-bucket" /> 
		<protocol type="MERGE3"/> 
		<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/> 
		<protocol type="FD_ALL"/> 
		<protocol type="VERIFY_SUSPECT"/> 
		<protocol type="pbcast.NAKACK2" use_mcas_xmit="false" /> 
		<protocol type="UNICAST3"/> 
		<protocol type="pbcast.STABLE"/> 
		<protocol type="pbcast.GMS"/> 
		<protocol type="MFC"/> 
		<protocol type="FRAG2"/> 
	</stack> 
  </jgroups>

  <cache-container name="keycloak">
    <!-- custom stack must be referenced by name in the stack attribute of the transport element -->
    <transport lock-timeout="60000" stack="s3"/>
    <local-cache name="realms">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <local-cache name="users">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <distributed-cache name="sessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="authenticationSessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineSessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="clientSessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="offlineClientSessions" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <distributed-cache name="loginFailures" owners="2">
      <expiration lifespan="-1"/>
    </distributed-cache>
    <local-cache name="authorization">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <memory max-count="10000"/>
    </local-cache>
    <replicated-cache name="work">
      <expiration lifespan="-1"/>
    </replicated-cache>
    <local-cache name="keys">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <expiration max-idle="3600000"/>
      <memory max-count="1000"/>
    </local-cache>
    <distributed-cache name="actionTokens" owners="2">
      <encoding>
        <key media-type="application/x-java-object"/>
        <value media-type="application/x-java-object"/>
      </encoding>
      <expiration max-idle="-1" lifespan="-1" interval="300000"/>
      <memory max-count="-1"/>
    </distributed-cache>
  </cache-container>
</infinispan>

```

I’m sure I have something configured incorrectly there, but I get the following error:

```auto
2022-11-01 17:16:28,764 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (production) mode
2022-11-01 17:16:28,765 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start caches
2022-11-01 17:16:28,765 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: ISPN000085: Error while trying to create a channel using the specified configuration file: default-configs/default-jgroups-ec2.xml
2022-11-01 17:16:28,765 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: org.infinispan.commons.CacheConfigurationException: ISPN000085: Error while trying to create a channel using the specified configuration file: default-configs/default-jgroups-ec2.xml
2022-11-01 17:16:28,765 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: ISPN000085: Error while trying to create a channel using the specified configuration file: default-configs/default-jgroups-ec2.xml
2022-11-01 17:16:28,765 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: JGRP000002: unable to load protocol org.jgroups.aws.s3.NATIVE_S3_PING (either with relative - org.jgroups.aws.s3.NATIVE_S3_PING - or absolute - org.jgroups.protocols.org.jgroups.aws.s3.NATIVE_S3_PING - class name)
2022-11-01 17:16:28,765 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.

```

I’m struggling to get this working and have spent all day today trying to get to the bottom of it, has anyone got working configuration they are willing to share or point me in the right direction?

Any help is appreciated thanks.  
Paul

---

<div class="post-metadata">

**Author:** ![dasniko](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/dasniko/32/2969_2.png) [@dasniko](https://forum.keycloak.org/u/dasniko)\
**Post date:** [November 1, 2022, 8:47am UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/2 "2022-11-01T08:47:33Z")

</div>

> [@paulkilla](#):
>
> I have in my keycloak.conf file the following entries in relation to the cache:
> 
> ```auto
> cache=ispn
> cache-config-file=cache-ec2.xml
> cache-stack=ec2
> 
> ```

I don’t know how to configure the EC2 stack, BUT…  
It has been said quite multiple times here in the forum in various threads about cluster configuration, that, if you configure `cache-stack` _AND_ `cache-config-file`, the former will take higher precedence over the latter and your custom file will be ignored. So, if you want to use one of the built-in stacks, use `cache-stack`, if you want to use a custom stack with a xml file, specify it with `cache-config-file`.

---

<div class="post-metadata">

**Author:** ![paulkilla](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/paulkilla/32/1045_2.png) [@paulkilla](https://forum.keycloak.org/u/paulkilla)\
**Post date:** [November 1, 2022, 9:32am UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/3 "2022-11-01T09:32:11Z")

</div>

How does that work with S3 where you need to specify the bucket?  
Or do you specify it with ENV variables?

Would be good if there was a keycloak example xml file for each of the infinispan implementations, as there is obviously some wrapping around the default infinispan xml - at least from comparing JDBC\_PING, which didn’t work for me in AWS as has been reported as well.

---

<div class="post-metadata">

**Author:** ![dasniko](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/dasniko/32/2969_2.png) [@dasniko](https://forum.keycloak.org/u/dasniko)\
**Post date:** [November 1, 2022, 10:01am UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/4 "2022-11-01T10:01:33Z")

</div>

Maybe this helps (and maybe not only the linked chapter, but other resources of the Infinispan docs):

[https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-extras-properties\_cluster-transport](https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-extras-properties_cluster-transport)

---

<div class="post-metadata">

**Author:** ![weltonrodrigo](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/weltonrodrigo/32/3104_2.png) [@weltonrodrigo](https://forum.keycloak.org/u/weltonrodrigo)\
**Post date:** [November 1, 2022, 8:00pm UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/5 "2022-11-01T20:00:27Z")

</div>

Based on [this](https://github.com/jgroups-extras/jgroups-aws#testing) and [this](https://connect2id.com/blog/how-to-configure-infinispan-for-aws-s3ping-discovery) you need to select the ec2 stack and provide AWS credentials for the underlying library used by it.

I suppose you can just put those lines in your keycloak.conf (besides removing the `cache-stack` as stated in [the previous answers](http://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/2), you can use on or the other):

```nohighlight
cache=ispn
cache-stack=ec2

```

Run keycloak with those environment variables set (replace with your own values):

```bash
JAVA_OPTS_APPEND=-Djgroups.s3.bucket=MY_BUCKET_NAME -Djgroups.s3.access_key=MY_KEY_ID -Djgroups.s3.secret_access_key=MY_KEY_SECRET

```

---

<div class="post-metadata">

**Author:** ![paulkilla](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/paulkilla/32/1045_2.png) [@paulkilla](https://forum.keycloak.org/u/paulkilla)\
**Post date:** [November 1, 2022, 9:37pm UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/6 "2022-11-01T21:37:56Z")

</div>

I am away from the office today, so unable to test it… but does anyone know if you can skip providing the AWS credentials and use an IAM role in it’s place, I will be unable to create credentials due to security requirements but can use an IAM role attached to the EC2 (and the equivelent bucket policy applied for said role).

---

<div class="post-metadata">

**Author:** ![weltonrodrigo](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/weltonrodrigo/32/3104_2.png) [@weltonrodrigo](https://forum.keycloak.org/u/weltonrodrigo)\
**Post date:** [November 2, 2022, 2:43pm UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/7 "2022-11-02T14:43:39Z")

</div>

Looking at the code, it seems aws.S3\_PING just uses default AWS library with a call to

```java
DefaultAWSCredentialsProviderChain creds=DefaultAWSCredentialsProviderChain.getInstance();

```

So, the default methods used to configure the aws client will apply here.

Take a look at on how to configure the client to use a role [amazon web services - How to use IAM role with AWS Java SDK - Stack Overflow](https://stackoverflow.com/a/52805686/25612)

If the IAM role is attached to the ec2 instance itself, I suppose you can just set bucketname and credentials will be handled for you by the java SDK under the hood.

---

<div class="post-metadata">

**Author:** ![serhiiKalchenko](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/serhiikalchenko/32/7719_2.png) [@serhiiKalchenko](https://forum.keycloak.org/u/serhiiKalchenko)\
**Post date:** [February 21, 2023, 1:09pm UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/8 "2023-02-21T13:09:24Z")

</div>

Keycloak: `20.0.3`  
To implement default “ec2” Infinispan stacks protocol you should:

1. Put these files in `./providers` dir

- `jgroups-aws-2.0.1.Final.jar` (stacks protocol)
- `aws-java-sdk-core-1.12.410.jar` (access to AWS creds, etc.)
- `aws-java-sdk-s3-1.12.410.jar` (access to S3, etc.)
- `joda-time-2.12.2.jar` (is used)

1. JAVA\_OPTS\_APPEND=‘-Djgroups.s3.region\_name=us-east-1 -Djgroups.s3.bucket\_name=\<backet\_name\>’

- IAM profile role should be applied to EC2 instance (for AWS creds) and S3 bucket created

1. Build Keycloak with option `--cache-stack=ec2` (no `--cache-config-file` option!)

- `bin/kc.[sh|bat] build --cache-stack=ec2`

By me it’s working) GL& HF!

Logs:

```auto
2023-02-21 09:47:28,062 INFO [org.infinispan.server.core.transport.EPollAvailable] (keycloak-cache-init) ISPN005028: Native Epoll transport not available, using NIO instead: java.lang.ExceptionInInitializerError
2023-02-21 09:47:28,453 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2023-02-21 09:47:28,472 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2023-02-21 09:47:28,506 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2023-02-21 09:47:28,987 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2023-02-21 09:47:29,086 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000128: Infinispan version: Infinispan 'Triskaidekaphobia' 13.0.10.Final
2023-02-21 09:47:29,270 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN`
2023-02-21 09:47:29,290 WARN [org.jgroups.stack.Configurator] (keycloak-cache-init) NATIVE_S3_PING has been deprecated; please upgrade to a newer version of the protocol
2023-02-21 09:47:30,181 INFO [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) using Amazon S3 ping in region us-east-1 with bucket 'my-jgroups-s3-bucket-test' and prefix ''
2023-02-21 09:47:30,937 INFO [org.jgroups.aws.s3.NATIVE_S3_PING] (keycloak-cache-init) found bucket my-jgroups-s3-bucket-test
2023-02-21 09:48:04,101 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [ip-10-68-49-170-40943|3] (2) [ip-10-68-49-170-40943, ip-10-68-49-190-31671]
2023-02-21 09:48:04,111 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `ip-10-68-49-190-31671`, physical addresses are `[10.68.49.190:7800]`

2023-02-21 09:48:07,087 INFO [io.quarkus] (main) Keycloak 20.0.3 on JVM (powered by Quarkus 2.13.6.Final) started in 78.156s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
2023-02-21 09:48:07,087 INFO [io.quarkus] (main) Profile prod activated.

```

---

<div class="post-metadata">

**Author:** ![msquared](https://avatars.discourse-cdn.com/v4/letter/m/aeb1de/32.png) [@msquared](https://forum.keycloak.org/u/msquared)\
**Post date:** [May 5, 2023, 10:24pm UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/9 "2023-05-05T22:24:23Z")

</div>

I ended up going with jdbc\_ping but it also relies on jgroups-aws-2.0.1.Final.jar.

---

<div class="post-metadata">

**Author:** ![stropia](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/stropia/32/3778_2.png) [@stropia](https://forum.keycloak.org/u/stropia)\
**Post date:** [May 11, 2023, 4:17pm UTC](https://forum.keycloak.org/t/keycloak-cluster-jgroups-infinispan-on-ec2/18208/10 "2023-05-11T16:17:12Z")

</div>

Or you could do something like this:

```auto
    <jgroups>
        <stack name="cluster">
            <TCP bind_port="7600"/>
            <TCPPING
                    initial_hosts="{{groups['tag_group_' + ec2_tag_environment + '_iam_infinispan'] | join('[7600], ')}}[7600]"
                     port_range="0"/>
            <MERGE3 min_interval="10000" max_interval="30000"/>
            <FD_SOCK client_bind_port="57600" start_port="57600"/>
            <!-- Suspect node `timeout` to `timeout + timeout_check_interval` millis after the last heartbeat -->
            <FD_ALL timeout="10000" interval="2000" timeout_check_interval="1000"/>
            <VERIFY_SUSPECT timeout="1000"/>

            <pbcast.NAKACK2 use_mcast_xmit="false" xmit_interval="100" xmit_table_num_rows="50" xmit_table_msgs_per_row="1024"
                            xmit_table_max_compaction_time="30000" resend_last_seqno="true"/>
            <UNICAST3 xmit_interval="100" xmit_table_num_rows="50" xmit_table_msgs_per_row="1024" xmit_table_max_compaction_time="30000"/>
            <pbcast.STABLE stability_delay="500" desired_avg_gossip="5000" max_bytes="1M"/>
            <pbcast.GMS print_local_addr="false" join_timeout="5000"/>
            <UFC_NB max_credits="3m" min_threshold="0.40"/>
            <MFC_NB max_credits="3m" min_threshold="0.40"/>
            <FRAG3/>
        </stack>
    </jgroups>

```

tag\_group, ec2\_tag\_environment are provided via ansible during deployment.