Best way to import realm in keycloak cluster for prod environments (container based)

[constantine]

Hello all,
We are currently trying to implement a software solution which will incorporate keycloak and will work under kubernetes. More specifically we will create a cluster of kubernetes pods.

We have currently created a custom keycloak container images which extend the default keycloak image and as part of it’s entrypoint it always performs an import of the current realm by utilizing the command “kc.sh import …”.

Things seem to go well for dev purposes. As soon as a keycloak pod is created the respective import is executed and the realm is recreated again in database (although this is not always necessary). The thing is that this doesn’t seem to be the ideal way in a production environment. Theoretically this could lead to situations like race conditions, inconsistency etc. Isn’t that right? The worst thing is that by incorporating this logic in the entrypoint of keycloak pods, this will be triggered even in a simple rolling update.

What is the recommended way of handling the realm import in kubernetes, when we use a keycloak cluster? I assume that it’s not safe to follow our approach which replaces each time the realm upon pod creation.

Does keycloak offer something as part of it’s cluster based functionality or should this be handled externally?

Would a creation of a kubernetes Job be more appropriate in order to ensure that the import happens only once and at the exact time that we need it? Is there any other alternative solution?

[xgp]

I posted something similar on this topic a few days ago. There are a few tools for declarative configuration and synchronization of realm state.

Tools or approaches for Keycloak DML Changes

First of all, I wasn’t aware of “Keycloakmigration”. Thanks for posting that. These are the tools I’m familiar with that I’ve used with varying degress of success: Terraform provider keycloak-config-cli scripting the Admin CLI (kcadm.sh) using the API directly to create “migrations” as in your example tool. Totally different approach, if you can live with not modifying config at runtime: GitHub - opdt/keycloak-extension-filestore: Custom Filestore-implementation (based on Keycloak 22 map-…

Most of the above can/should be run as a separate Job in k8s, or in some cases as an initContainer.

kc.sh import ... is good for dev or first time realm creation, but not good to run each time you start a container.

[constantine]

Thanks for the suggestions, I will have a look at them. It looks like we will probably end up defining the import process as a Job.

[djordje]

I have been using keycloak import strategy and importing a realm on each Keycloak startup.
There is a few things you have to consider for this approach and there is some advantages and disadvantages:

OVERWRITE_EXISTING

Switching to the OVERWRITE_EXISTING strategy would allow you to store realm configurations in our repository as JSON files, known as Realm Representations. There are several advantages and some disadvantages that need to be considered:

Advantages of Using JSON Files for Realm Imports

1. Better Backup and Restore Strategy:

• Storing realm configurations as JSON files enables easy backups. These files can be versioned and stored in a version control system, allowing us to restore a specific configuration state easily.

2. Reduction of Human Errors:

• Managing realm configurations through JSON files reduces the risk of errors that can occur with manual UI modifications. This is particularly beneficial for complex configurations where consistency is crucial.

3. Controlled Changes and Decision-Making:

• Changes to the realm configurations can be managed through pull requests (PRs). This involves multiple stakeholders in the decision-making process, ensuring that changes are reviewed and approved by the team.

4. Release management:

• All environments will have same configuration at release time, and you can test it before PROD.

Disadvantages

1. Less User-Friendly Management:

• Configuring realms through JSON files can be less intuitive compared to using the Keycloak UI. It requires a deeper understanding of Keycloak’s configuration and JSON syntax.

2. Reduced Real-Time Changes:

• Real-time modifications through the UI will be temporary. Changes made directly in the UI will only persist until the next application restart. Permanent changes must be made in the JSON files, which then need to be redeployed.

Also this worked for me since I have maybe different situation than you, but this may help you in your decision!

[constantine]

Hello,
Thanks for your response. We are actually also using a JSON based approach with the OVERWRITE_EXISTING strategy. The problem in our case is that we are currently doing it at startup, but this doesn’t seem like a good approach for a production env (where multiple pods create a cluster) where each one of this pods could be restarted and retrigger an import while the others are serving live requests at the same time.

[djordje]

Idk, we have no issues in this case, ofc looking when traffic is lowest for release, also that is not something that you will do every day. Also what you can check with devops is to terminate all current pods after first is up, so there will be no 2 pods with different versions, and one pod will serve traffic until another are up.

[embesozzi]

As xpg mentioned before, it’s hard to see the kc.sh import approach being used in a production environment.
Luckily, we have the Terraform provider option. Now that Keycloak has adopted it, it’s hard to see a better approach. However, it all depends on the technology stack and experience.