I would use standard debugging = increase log level and check the logs. Behaviour indicates problem with clustering, so check if nodes are discovered correctly and they are able to talk each other without any issue.
Thanks @jangaraj, I should have mentioned that I’ve done that, DEBUG level, and it reveals nothing.
I should also state that I’m using and external ProstgresDB instance that is being connected to.
OK.
I’ve done more investigation and it seems that the JDBC_PING is not being invoked.
I have made sure that the right JDBC_PING.cli file is in /opt/jboss/tools/cli/jgroups/discovery/
Have updated standalone-ha.xml to use JDBC_PING:
Jan I’m doing exactly that.
Here is my Dockerfiles:
FROM jboss/keycloak:15.0.2
USER root
RUN curl -O http://mirror.centos.org/centos/8/AppStream/x86_64/os/Packages/oniguruma-6.8.2-2.el8.x86_64.rpm
RUN rpm -ivh oniguruma-6.8.2-2.el8.x86_64.rpm
RUN curl -O http://mirror.centos.org/centos/8/AppStream/x86_64/os/Packages/jq-1.5-12.el8.x86_64.rpm
RUN rpm -ivh jq-1.5-12.el8.x86_64.rpm
USER jboss
WORKDIR /opt/jboss
COPY cli/JDBC_PING.cli tools/cli/jgroups/discovery/
COPY standalone-ha.xml keycloak/standalone/configuration/
COPY postgresql/ keycloak/modules/system/layers/keycloak/org/
COPY themes/ keycloak/themes/
COPY docker-entrypoint.sh .
ENTRYPOINT [ "/opt/jboss/docker-entrypoint.sh", "-b", "0.0.0.0" ]
EXPOSE 7600
I have to create my own image for the following reasons:
We use customised themes (Keycloak login is a proxy for our app login)
We have to load the Postgres JDBC driver
We have to load up a Postgres jdbc_ping.cli
We have been forced to load a custom standalone-ha.xml because jdbc_ping.cli doesn’t run and load the Postgres driver
we add JQ so we can get the ECS Task’s IP address in the following script
Consequently, here is our startup script:
#!/bin/bash
# use when testing jq
#ECS_CONTAINER_METADATA_URI_V4=`cat aws_test.json`
#
echo "getting the Container's IP Address for Keycloak clustering"
#echo "ECS_CONTAINER_METADATA_URI_V4 is ${ECS_CONTAINER_METADATA_URI_V4}"
curl "${ECS_CONTAINER_METADATA_URI_V4}" > container_metadta.json
#cat container_metadta.json
export EXTERNAL_ADDR=$(cat container_metadta.json | jq -r ".Networks[0].IPv4Addresses[0]")
echo "EXTERNAL_ADDR is ${EXTERNAL_ADDR}"
export JGROUPS_DISCOVERY_EXTERNAL_IP="${EXTERNAL_ADDR}"
exec /opt/jboss/keycloak/bin/standalone.sh --server-config=standalone-ha.xml "$@" -Djboss.bind.address="${EXTERNAL_ADDR}" -Djboss.bind.address.private="${EXTERNAL_ADDR}"
exit $?
What I found was:
JDBC_PING.cli just doesn’t run (maybe because if needs to be owned by the Jboss user, and losing it in the Dockerfile means that Jgroups doesn’t see it?)
unless you change this line in standalone-ha.xml (change the ‘udp’ value to ‘tcp’), jbdc-ping JUST DOES NOT GET NOTICED AT ALL:
The JDBC_PING.cli just isn’t run (maybe because the Jboss user has to own it and Groups doesn’t see it because root owns it after the Dockerfile update??), hence the change not being made above and the other changes need to be made in standalone-ha.xml
Unless BOTH of these parameters are passed in, you’ll get our friend “401 on whoami”, even with the jgroupsping table created and used:
However, none of this information is highlighted, or even documented. The last point I found purely by chance when searching on another issue (and it was in an obscure use post)!
@dasniko unfortunately, as the start of this issue states, I used the envars but I did not automate anything.
Which is why I opened this issue.
All this stuff may work nicely in standalone docker containers, but not when orchestrated in a Fargate instance in ECS
see also issue Keycloak clustering · Issue #95 · aws-samples/keycloak-on-aws · GitHub there. There is usually no need for any customization. I guess you made a problem with your own customization.
Recommendation: start simple with vanila docker image and then when standard Keycloak is running in clustered mode properly add your own customizations.
Thanks Jan, but we we migrating from an existing set of implementations (Test, Staging, Production) that were just single standalone implementations. The people that set these up had left the organisation, so we found it hard to find somewhere that described a standard cluster in ECS to do a base implementation from.
That link you gave, although referencing Fargate, uses Typescript and the aws-cdk, so would take almost as much unpacking as doing what I did.
But the most important aspect is that it only using standalone Fargate Instances, like standalone EC2 instances, NOT managed by ECS.
I really don’t understand. You are complaining that something is not working, but you didn’t mention in first place that you have a lot of customizations.
For example custom entrypoint, which apparently doesn’t execute vendor jgroups.sh, which configure jgroups configuration in the container.
You should provide reproducible example (+ infrastructure + hire someone). This looks like a very long shoot for community forum (so you didn’t open real issue, only discussion with random people, who are willing to read your problem). I have autoscaled apps on AWS ECS Fargate in prod (but not a Keycloak), but this problem is 🤷.
I really don’t see a reason for any your customization (only custom theme, but that can be delivered from side car container, so it won’t need a custom Keycloak image build at all).
Please don’t get aggressive because you can’t answer the question or solve the problem, being that keycloak has NO working example or documentation of running in Fargate in ECS.
If they tried it they would find out the it doesn’t work out of the box because the JDBC_PING.cli file is not run for an existing implementation (Read here data in the database).
The only customisation we have is the themes, nothing else.
Nothing any one that a has responded has been able to refute what I’m saying.
It’s not my fault if it doesn’t automatically work, and the documentation assumes it will.
In case you have misunderstood, keycloak IS working in Fargate in ECS, BUT only with the manual configs I’ve pointed out.