# Configurations for a large cluster **URL:** \ **Category:** Configuring the server\ **Created:** [July 16, 2021, 7:42pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099 "2021-07-16T19:42:18Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jichen-amplify](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jichen-amplify](https://forum.keycloak.org/u/jichen-amplify)\ **Post date:** [July 16, 2021, 7:42pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/1 "2021-07-16T19:42:18Z")
Hi, We have been running a 3 instances Keycloak cluster on AWS EC2 with some modifications on the standalone-ha.xml file that comes with official Keycloak distribution. The instance type we are using is fairly large (c5.4xlarge) and the whole system is fairly stable. Recently, our company has moved to using AWS Fargate, which means we will have much less powerful machines. During our load testing, we are now running 18 instances of Keycloak and we are seeing a small amount of errors. Some requests are taking a really long time to complete. CPU and network usage also become very unstable, even after the load testing is done. Does anyone have any suggestions on areas of interest where we can look into to eliminate the issues? Or it would be really helpful too if anyone can point us to some information on how to configure/run a large Keycloak cluster. Much appreciated in advanced! Jia ---
**Author:** ![xgp](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/xgp/32/2589_2.png) [@xgp](https://forum.keycloak.org/u/xgp)\ **Post date:** [July 17, 2021, 6:49am UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/2 "2021-07-17T06:49:34Z")
Depending on your target number of users and sessions, I’d consider running an external infinispan cluster. I’ve been running ~15-20 keycloak instances on Fargate with 5 infinispan instances (r5.2xlarge) on EC2, and it’s much more stable than running everything on Fargate. There have been some discussions here about config: > [@How to configure standalone external infinispan server to domain cluster keycloak](http://forum.keycloak.org/t/how-to-configure-standalone-external-infinispan-server-to-domain-cluster-keycloak/8838): > > any suggestions FYI - keycloak domain cluster is done using JDBC\_PING > **[GitHub - Mu-Wahba/keycloak-for-production: Keycloak HA mode Installation for...](https://github.com/Mu-Wahba/keycloak-for-production)** > > Keycloak HA mode Installation for production environment with external infinispan cache cluster. - GitHub - Mu-Wahba/keycloak-for-production: Keycloak HA mode Installation for production environmen... ---
**Author:** ![jichen-amplify](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jichen-amplify](https://forum.keycloak.org/u/jichen-amplify)\ **Post date:** [July 19, 2021, 11:55pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/3 "2021-07-19T23:55:07Z")
@xgp Thanks for the information! I’ll give it a try for sure. Thanks again. ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [July 20, 2021, 12:25pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/4 "2021-07-20T12:25:03Z")
@jichen-amplify@xgp Can you please let us know the kind of traffic that you are managing with your deployments? Just trying to understand what should be the scale of the infrastructure, when the peak load is expected to go like 1000 authentications per second. ---
**Author:** ![xgp](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/xgp/32/2589_2.png) [@xgp](https://forum.keycloak.org/u/xgp)\ **Post date:** [July 20, 2021, 4:42pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/5 "2021-07-20T16:42:34Z")
Is that 1000 auths per second sustained? You’re really going to have almost 100 million auths per day!? That would certainly be one of the biggest deployments of Keycloak I’ve heard of. The cluster I described above does ~200 auths per second peak load, but definitely not sustained. Most auths in one day is ~2 million. ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [July 20, 2021, 4:50pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/6 "2021-07-20T16:50:31Z")
No, its not sustained. It would be peak load that may be experienced intermittently. So, we are integrating with a mobile app which will be used by about 15 to 20 million users. The app may send periodic notifications for some offers that it may have. During that period, we are expecting that lot of users may try to access the application, which would create a load on Keycloak to provide tokens to users who may need a new token or a new authentication. In your scenario where you have about 200 auth / second, what kind of infrastructure did you need , interms of sizing. Regards, Ashish ---
**Author:** ![xgp](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/xgp/32/2589_2.png) [@xgp](https://forum.keycloak.org/u/xgp)\ **Post date:** [July 20, 2021, 5:12pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/7 "2021-07-20T17:12:04Z")
The cluster I described above: > [@xgp](#): > > ~15-20 keycloak instances on Fargate with 5 infinispan instances (r5.2xlarge) on EC2 ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [July 20, 2021, 5:14pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/8 "2021-07-20T17:14:18Z")
ok so 15-20 instances is it needed all the time or its just the peak scaling of containers during auto-scale on Fargate? ---
**Author:** ![xgp](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/xgp/32/2589_2.png) [@xgp](https://forum.keycloak.org/u/xgp)\ **Post date:** [July 20, 2021, 5:26pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/9 "2021-07-20T17:26:26Z")
In the use case that I have, there would be no time to auto-scale to meet peaks, so we currently keep 16 instances running at all time. ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [July 20, 2021, 5:32pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/10 "2021-07-20T17:32:05Z")
OK, thanks a lot , this information helps us to decide the way forward. Just one last question, can you please share what is the total user volume that you did cater to ? ---
**Author:** ![xgp](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/xgp/32/2589_2.png) [@xgp](https://forum.keycloak.org/u/xgp)\ **Post date:** [July 20, 2021, 5:43pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/11 "2021-07-20T17:43:28Z")
There are just over 18 million users stored, but the MAU number is only 4 million. ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [July 20, 2021, 5:44pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/12 "2021-07-20T17:44:57Z")
Thanks again, this is really helpful. Just to understand the sizing, can I know how many vCPU and RAM each of your container had in the Fargate ? ---
**Author:** ![xgp](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/xgp/32/2589_2.png) [@xgp](https://forum.keycloak.org/u/xgp)\ **Post date:** [July 20, 2021, 6:09pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/13 "2021-07-20T18:09:08Z")
```auto cpu = 4096 memory = 8192 ``` Happy to share information, but I’d really recommend doing load testing against your own specific use cases before you put something of this scale into production. There are a few starts out there for Keycloak load testing frameworks (e.g. [GitHub - lgraf/keycloak-gatling: basic gatling simulation to load test your keycloak installation](https://github.com/lgraf/keycloak-gatling)). ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [July 20, 2021, 6:24pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/14 "2021-07-20T18:24:29Z")
Yes, we are going to do the load test, I was just trying to understand the sizing for similar scale. Thanks again ---
**Author:** ![atulchauhan01](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@atulchauhan01](https://forum.keycloak.org/u/atulchauhan01)\ **Post date:** [September 29, 2021, 7:21am UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/15 "2021-09-29T07:21:05Z")
@ashishtchaudhari, are you running Keycloak and Infinispan on Fargate or Infinispan on EC2 instance as suggested by @xgp ? If you are running on Fargate, can you please share the container sizing details (vCPU and Memory) for Keycloak and Infinispan? and how many instances are you running all the time? ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [September 29, 2021, 7:46am UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/16 "2021-09-29T07:46:09Z")
Hi Atul, At this point we are running it over auto-scaling group of EC2 instances. We have decided to go with R5b instances with 2 CPU an 16 GB of RAM. Our performance testing so far have showed that we would need more memory to store session cache. Our Infinispan servers are embedded and not external as of now. We are still evaluating the best approach. Regards, Ashish ---
**Author:** ![atulchauhan01](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@atulchauhan01](https://forum.keycloak.org/u/atulchauhan01)\ **Post date:** [September 29, 2021, 9:11am UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/17 "2021-09-29T09:11:32Z")
Thanks Ashish, can you share, how many users do you have in your database? Which database are you using and what is the configuration? ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [October 16, 2021, 9:45pm UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/18 "2021-10-16T21:45:51Z")
We have about 4 million users. ---
**Author:** ![ashishtchaudhari](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/ashishtchaudhari/32/6518_2.png) [@ashishtchaudhari](https://forum.keycloak.org/u/ashishtchaudhari)\ **Post date:** [October 19, 2021, 4:27am UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/19 "2021-10-19T04:27:29Z")
@xgp We are facing issues with keycloak in certain scenarios, if the rolling restart of the server happens, there are certain issues related to refresh token is seen. We have offline tokens configured, but even then it seems that in some cases, even if the token sent is valid, keycloak returns an error stating that the token is invalid. By anychance, did you face an issue similar to this, if yes, any guidance around this will help. ---
**Author:** ![xgp](https://yyz2.discourse-cdn.com/free1/user_avatar/forum.keycloak.org/xgp/32/2589_2.png) [@xgp](https://forum.keycloak.org/u/xgp)\ **Post date:** [October 19, 2021, 7:30am UTC](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099/20 "2021-10-19T07:30:43Z")
I haven’t seen that specific problem. If you could post more about what you’re seeing, we could try to recreate it and help you debug. [Next page](https://forum.keycloak.org/t/configurations-for-a-large-cluster/10099.md?page=2)