Keycloak 19.0.3 quarkus - ha in azure - jgroup configuration azure_ping

1

Hi,

I have 3 virtual machines in azure and I want enable Keycloak latest version of Quarkus in HA.
I have readed the documentation of:

https://www.keycloak.org/server/caching
https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-cloud-discovery-protocols_cluster-transport

I have added in keycloak.conf

cache=ispn
cache-stack=azure
cache-config-file=cache-ispn-azure.xml

But I don’t understand how inject the configuration in cache-ispn-azure.xml.
Maybe something like this

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
        xmlns="urn:infinispan:config:11.0">


    <jgroups>
        <stack name="tcp_azure" extends="azure">
            <TCP     bind_port="7800" />
            <AZURE_PING initial_hosts="redhattest1[7800],redhattest2[7800]" port_range="0" max_dynamic_hosts="2"/>
        </stack>
    </jgroups>

    <cache-container name="keycloak">
        <transport cluster="mykeycloak" lock-timeout="60000"  stack="tcp_azure" node-name="redhattest1"/>

But I don’t know how inject the storage explained here:

https://github.com/jgroups-extras/jgroups-azure

Somebody know the correct configuration for this custom cache-ispn.xml for azure?

Somehow I have to be able to indicate that I am using the azure ping, the credentials of the azure resource for its implementation and the machines that I want to form the infinispan cluster…

I’m working with Quarkus distro and the documentation is more structured that wildfly version but very short in principal elements like distributed cache.

Thanks so much!!

Xavier.

2

I believe you shoud put them as parameters in this element, as in the docs:

<azure.AZURE_PING
	storage_account_name="storage_account_name"
	storage_access_key="storage_account_key"
	container="storage_account_container_name"
/>

Based on what I see in the AZURE_PING.java, I suppose initial_hosts, port_range and max_dynamic_hosts are ignored.

You shouldn’t set cache-stack. This config selects a preconfigured cache coming with keycloak out of the box, not a stack you defined in the cache-config-file.

3

We seem to be having the same issue with setting up Keycoak v19 with Azure Ping. Did you get this working, and can you share what options you ended up using in your kc build/run commands and what your cache config file entries ended up looking like. Thanks

4

Hi,
Here is configuration that works for me. We are running Keycloak 24.0.1 (Quarkus Engine) in Azure Container Apps, but the same config should work with Azure VMs as well.

  1. We build a custom docker container to add some parameters and plugins needed.
FROM quay.io/keycloak/keycloak:24.0.1 as builder

# Configure a database vendor
ENV KC_DB=mysql

WORKDIR /opt/keycloak
COPY jar/azure-storage-8.6.6.jar /opt/keycloak/providers/azure-storage-8.6.6.jar
COPY jar/jgroups-azure-2.0.2.Final.jar /opt/keycloak/providers/jgroups-azure-2.0.2.Final.jar
RUN /opt/keycloak/bin/kc.sh build --health-enabled=true --metrics-enabled=true --cache=ispn --cache-stack=azure

FROM quay.io/keycloak/keycloak:24.0.1
COPY --from=builder /opt/keycloak/ /opt/keycloak/

ENV KC_HTTP_RELATIVE_PATH=/auth
ENV KC_HEALTH_ENABLED=true

Here in the jar folder, I put 2 plugins:
Link to jars:
https://repo1.maven.org/maven2/org/jgroups/azure/jgroups-azure/2.0.2.Final/jgroups-azure-2.0.2.Final.jar
https://repo1.maven.org/maven2/com/microsoft/azure/azure-storage/8.6.6/azure-storage-8.6.6.jar
2. You need to have Azure Storage account created with a container in it to store ISPN cache and pass this info via JAVA_OPTS_APPEND

JAVA_OPTS_APPEND
-Djboss.jgroups.azure_ping.storage_account_name="${storage_account_name}"
-Djboss.jgroups.azure_ping.storage_access_key="${storage_access_key}" 
-Djboss.jgroups.azure_ping.container="${storage_container}"

Here I passed storage_account_name, storage_access_key and storage_container as parameters.

Hope it will help. I spent couple of days to figure it out :slight_smile:

1 Like
Deploying Keycloak docker image 25.0 in Azure Container App
5

Hi,
I also would like to run Keycloak in HA with Azure Container Apps. Could you provide additional configuration of the individual containers? Which load balancer are you using?

6

Hi, here is part of the Bicep template do deploy Keycloak in Azure Container Apps:

    template: {
      revisionSuffix: docker_image_tag
      containers: [
        {
          image: 'yourcontainerregistry.azurecr.io/keycloak-custom:${docker_image_tag}'
          name: appname
          command: []
          resources: {
            cpu: json('1')
            memory: '2Gi'
          }
          args: ['start','--optimized']
          probes: [
            {
              failureThreshold: 3
              httpGet: {
                path: '/health/live'
                port: 8080
                scheme: 'HTTP'
              }
              initialDelaySeconds: 60
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 5
              type: 'Liveness'
            }
            {
              failureThreshold: 3
              httpGet: {
                path: '/health/ready'
                port: 8080
                scheme: 'HTTP'
              }
              initialDelaySeconds: 60
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 5
              type: 'Readiness'
            }
            {
              failureThreshold: 10
              httpGet: {
                path: '/health'
                port: 8080
                scheme: 'HTTP'
              }
              initialDelaySeconds: 60
              periodSeconds: 5
              successThreshold: 1
              timeoutSeconds: 5
              type: 'Startup'
            }
          ]
          env: [
            {
              name: 'KC_DB'
              value: 'mysql'
            }
            {
              name: 'KC_DB_URL'
              value: 'jdbc:mysql://yourmysqldb.mysql.database.azure.com:3306/keycloak?useSSL=true&requireSSL=true&enabledTLSProtocols=TLSv1.2&TransactionTimeout=30000'
            }
            {
              name: 'KC_DB_PASSWORD'
              secretRef: 'db-password'
            }
            {
              name: 'KC_DB_USERNAME'
              secretRef: 'db-username'
            }
            {
              name: 'KEYCLOAK_ADMIN_PASSWORD'
              secretRef: 'kc-password'
            }
            {
              name: 'KEYCLOAK_ADMIN'
              secretRef: 'kc-username'
            }
            {
              name: 'DB_QUERY_TIMEOUT'
              value: '3000'
            }
            {
              name: 'DB_VALIDATE_ON_MATCH'
              value: 'false'
            }
            {
              name: 'JAVA_OPTS_APPEND'
              value: '-Xms512M -Xmx1G -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=1G -Djboss.jgroups.azure_ping.storage_account_name="${storage_account_name}" -Djboss.jgroups.azure_ping.storage_access_key="${storage_access_key}" -Djboss.jgroups.azure_ping.container="${storage_container}" -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.as.management.blocking.timeout=300'
            }
            {
              name: 'KEYCLOAK_LOGLEVEL'
              value: 'DEBUG'
            }
            {
              name: 'PROXY_ADDRESS_FORWARDING'
              value: 'true'
            }
            {
              name: 'KEYCLOAK_FRONTEND_URL'
              value: 'https://yourkeycloakpublicdomain.com/auth/'
            }
            {
              name: 'KC_HOSTNAME'
              value: 'yourkeycloakpublicdomain.com'
            }
            {
              name: 'KC_HOSTNAME_STRICT_BACKCHANNEL'
              value: 'true'
            }
            {
              name: 'KC_PROXY_HEADERS'
              value: 'xforwarded'
            }
            {
              name: 'KC_HTTP_ENABLED'
              value: 'true'
            }
          ]
          volumeMounts: [
          ]
        }
      ]
      scale: {
        minReplicas: 2
        maxReplicas: 2
      }
      volumes: [
      ]
    }

As a load balancer we use Azure Application Gateway. For it to work properly make sure to add Rewrite Rule like this:

    rewriteRuleSets: [
      {
        name: 'keycloak-rewrite-rules'
        id: '${agwId}//rewriteRuleSets/keycloak-rewrite-rules'
        properties: {
          rewriteRules: [
            {
              ruleSequence: 100
              conditions: []
              name: 'headers-rewrite'
              actionSet: {
                requestHeaderConfigurations: [
                  {
                    headerName: 'X-Forwarded-For'
                    headerValue: '{var_add_x_forwarded_for_proxy}'
                  }
                  {
                    headerName: 'X-Forwarded-Proto'
                    headerValue: '{var_request_scheme}'
                  }
                  {
                    headerName: 'X-Forwarded-Host'
                    headerValue: '{var_host}'
                  }
                ]
                responseHeaderConfigurations: []
              }
            }
          ]
        }
      }

Hope this helps.

1 Like
8

Hi @reketen ,

I configured the all the above setting in container apps also in azure application gateway .

But in openID endpoint i could get the Mixed http and Https endpoints.

I tried adding proxy as edge in keycloak but no luck.

Is any way to fix it. i need it as https endpoint.

thanks in advance.
Guru