Keycloak - Domain mode - Management console behing Nginx - Error WFLYHC0001

Kortex · December 30, 2021, 3:21pm

Hi everyone,

I’m trying to achieve a simple objective: to build a Keycloak cluster (domain mode for the moment) with 2 nodes.

Details of what I’m doing:

node 1:
- domain controller (port 9990)
- host controller
- keycloak server (port 8180)
- nginx for the SSL offloading (self-signed certificate for my test)
node 2:
- domain controller (port 9990)
- host controller
- keycloak server (port 8180)

On Nginx side I created a first virtual host dedicated to Keycloak (so to answer on the /auth requests):

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

upstream keycloak-sg-01 {
    server 127.0.0.1:8180      weight=10    max_fails=3    fail_timeout=15s;
    server 192.168.1.2:8180    weight=5     max_fails=3    fail_timeout=15s    backup;
}

## HTTPS ##
server {
    server_name keycloak-test.local.net;

    access_log /var/log/nginx/access-keycloak-server.log;
    error_log /var/log/nginx/error-keycloak-server.log;

    listen 443 ssl;
    include snippets/self-signed.conf;

    location / {
        rewrite ^ /auth;
    }

    location /auth {
        add_header Front-End-Https off;
        add_header Cache-Control no-cache;

        proxy_redirect off;

        proxy_set_header    Authorization      $http_authorization;
        proxy_set_header    Host               $host;
        proxy_set_header    Origin             http://$host;
        proxy_set_header    X-Real-IP          $remote_addr;
        proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Host   $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Forwarded-Port   $server_port;
        proxy_set_header    X-Forwarded-Proto  $scheme;

        proxy_pass http://keycloak-sg-01/auth;
    }
}
## /HTTPS ##

This part works:

I can reach Keycloak
the SSL offloading is managed by Nginx (not perfect because the communication Nginx <-> 192.168.1.2:8180 is not crypted)

Now I tried to hide the console behind Nginx…

I created a second virtual host dedicated to the console:

map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

## HTTPS ##
server {
    server_name keycloak-console-test.local.net;

    access_log /var/log/nginx/access-keycloak-domain.log;
    error_log /var/log/nginx/error-keycloak-domain.log debug;

    listen 443 ssl;
    include snippets/self-signed.conf;

    location / {
        add_header Cache-Control 'no-store, no-cache';

        proxy_http_version 1.1;
        proxy_redirect off;

        proxy_set_header    Authorization      $http_authorization;
        proxy_set_header    Host               $host;
        proxy_set_header    Origin             http://$host;
        proxy_set_header    X-Real-IP          $remote_addr;
        proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Host   $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Forwarded-Port   $server_port;
        proxy_set_header    X-Forwarded-Proto  $scheme;

        proxy_pass http://127.0.0.1:9990;
    }

    location /console {
        add_header Cache-Control no-cache;

        proxy_http_version 1.1;
        proxy_redirect off;

        proxy_set_header    Authorization      $http_authorization;
        proxy_set_header    Host               $host;
        proxy_set_header    Origin             http://$host;
        proxy_set_header    X-Real-IP          $remote_addr;
        proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Host   $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Forwarded-Port   $server_port;
        proxy_set_header    X-Forwarded-Proto  $scheme;

        proxy_pass http://127.0.0.1:9990/console;
    }

    location /management {
        add_header Cache-Control no-cache;

        proxy_http_version 1.1;
        proxy_redirect off;

        proxy_set_header    Authorization      $http_authorization;
        proxy_set_header    Host               $host;
        proxy_set_header    Origin             http://$host;
        proxy_set_header    X-Real-IP          $remote_addr;
        proxy_set_header    X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Host   $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Forwarded-Port   $server_port;
        proxy_set_header    X-Forwarded-Proto  $scheme;

        proxy_pass http://127.0.0.1:9990/management;
    }
}
## /HTTPS ##

I can reach the HAL Management console with my browser on https://keycloak-console-test.local.net/console/ (I have an entry in my /etc/hosts file).

If I configure my host.xml with:

    <domain-controller>
        <remote username="myusername" authentication-context="hcAuthContext">
            <discovery-options>
                <static-discovery name="primary" protocol="${jboss.domain.master.protocol:remote+http}" host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9990}" />
            </discovery-options>
        </remote>
    </domain-controller>

It works when I start with this command:

/opt/keycloak/bin/domain.sh -c host.xml -Djboss.bind.address=0.0.0.0 -Djboss.bind.address.management=0.0.0.0 -Djboss.bind.address.private=127.0.0.1 -Djboss.tx.node.id=tx-srv-local-02 -Djboss.domain.master.address=keycloak-test.local.net -Djboss.domain.master.port=9990 -Djboss.management.http.port=9990

→ it works when I reach directly the console bypassing the Nginx server (so on the port 9990)

Now same configuration but I try to reach through Nginx:

/opt/keycloak/bin/domain.sh -c host.xml -Djboss.bind.address=0.0.0.0 -Djboss.bind.address.management=0.0.0.0 -Djboss.bind.address.private=127.0.0.1 -Djboss.tx.node.id=tx-srv-local-02 -Djboss.domain.master.address=keycloak-test.local.net -Djboss.domain.master.port=443 -Djboss.management.http.port=9990

If failed with this error:

[Host Controller] 15:05:19,012 WARN  [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0001: Could not connect to remote domain controller remote+http://keycloak-test.local.net:443: java.net.ConnectException: WFLYPRT0053: Could not connect to remote+http://keycloak-test.local.net:443. The connection failed
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionUtils.checkFuture(ProtocolConnectionUtils.java:142)
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:122)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection.lambda$openConnection$0(RemoteDomainConnection.java:211)
[Host Controller] 	at org.wildfly.common.context.Contextual.runExceptionAction(Contextual.java:108)
[Host Controller] 	at org.wildfly.security.auth.client.AuthenticationContext.run(AuthenticationContext.java:273)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection.openConnection(RemoteDomainConnection.java:211)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection$InitialConnectTask.connect(RemoteDomainConnection.java:582)
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection.connect(RemoteDomainConnection.java:141)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnectionService.register(RemoteDomainConnectionService.java:282)
[Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService.connectToDomainMaster(DomainModelControllerService.java:1008)
[Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService.boot(DomainModelControllerService.java:720)
[Host Controller] 	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:424)
[Host Controller] 	at java.lang.Thread.run(Thread.java:748)
[Host Controller] Caused by: org.xnio.http.UpgradeFailedException: Invalid response code 400
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState$UpgradeResultListener.handleEvent(HttpUpgrade.java:471)
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState.flushUpgradeChannel(HttpUpgrade.java:369)
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState.access$900(HttpUpgrade.java:165)
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState$ConnectionOpenListener.handleEvent(HttpUpgrade.java:340)
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState$ConnectionOpenListener.handleEvent(HttpUpgrade.java:320)
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState.upgradeExistingConnection(HttpUpgrade.java:315)
[Host Controller] 	at org.xnio.http.HttpUpgrade.performUpgrade(HttpUpgrade.java:144)
[Host Controller] 	at org.jboss.remoting3.remote.HttpUpgradeConnectionProvider$UpgradeListener.handleEvent(HttpUpgradeConnectionProvider.java:174)
[Host Controller] 	at org.jboss.remoting3.remote.HttpUpgradeConnectionProvider$UpgradeListener.handleEvent(HttpUpgradeConnectionProvider.java:153)
[Host Controller] 	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
[Host Controller] 	at org.xnio.nio.WorkerThread$ConnectHandle.handleReady(WorkerThread.java:333)
[Host Controller] 	at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
[Host Controller] 	at ...asynchronous invocation...(Unknown Source)
[Host Controller] 	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:599)
[Host Controller] 	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:561)
[Host Controller] 	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:549)
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionUtils.connect(ProtocolConnectionUtils.java:224)
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:118)
[Host Controller] 	... 12 more

So I tried to change my host.xml like this:

<static-discovery name="primary" protocol="${jboss.domain.master.protocol:remote+https}" host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9990}" />

→ remote-https instead of remote-http

Now it failed with this message:

[Host Controller] 15:07:56,807 WARN  [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0001: Could not connect to remote domain controller remote+https://keycloak-test.local.net:443: java.net.ConnectException: WFLYPRT0053: Could not connect to remote+https://keycloak-test.local.net:443. The connection failed
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionUtils.checkFuture(ProtocolConnectionUtils.java:142)
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:122)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection.lambda$openConnection$0(RemoteDomainConnection.java:211)
[Host Controller] 	at org.wildfly.common.context.Contextual.runExceptionAction(Contextual.java:108)
[Host Controller] 	at org.wildfly.security.auth.client.AuthenticationContext.run(AuthenticationContext.java:273)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection.openConnection(RemoteDomainConnection.java:211)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection$InitialConnectTask.connect(RemoteDomainConnection.java:582)
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection.connect(RemoteDomainConnection.java:141)
[Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnectionService.register(RemoteDomainConnectionService.java:282)
[Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService.connectToDomainMaster(DomainModelControllerService.java:1008)
[Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService.boot(DomainModelControllerService.java:720)
[Host Controller] 	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:424)
[Host Controller] 	at java.lang.Thread.run(Thread.java:748)
[Host Controller] Caused by: org.xnio.http.RedirectException: XNIO000816: Redirect encountered establishing connection
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState.handleRedirect(HttpUpgrade.java:513)
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState.access$1300(HttpUpgrade.java:165)
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState$UpgradeResultListener.handleEvent(HttpUpgrade.java:468)
[Host Controller] 	at org.xnio.http.HttpUpgrade$HttpUpgradeState$UpgradeResultListener.handleEvent(HttpUpgrade.java:400)
[Host Controller] 	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
[Host Controller] 	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
[Host Controller] 	at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
[Host Controller] 	at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
[Host Controller] 	at ...asynchronous invocation...(Unknown Source)
[Host Controller] 	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:599)
[Host Controller] 	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:561)
[Host Controller] 	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:549)
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionUtils.connect(ProtocolConnectionUtils.java:224)
[Host Controller] 	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:118)
[Host Controller] 	... 12 more

I tried a lot of things but I can’t find the solution…
I tested several Origin header like:

 proxy_set_header    Origin             $host;

 proxy_set_header    Origin             http://$host;

 proxy_set_header    Origin             http://$host:9990;

 proxy_set_header    Origin             https://$host;

 proxy_set_header    Origin             https://$host:9990;

→ same error

I tried to add http-upgrade-enabled=“true” on node 1 side : same error

Is anyone has an idea please ?

Topic		Replies	Views
Keycloak in docker behind reverse proxy Configuring the server admin-console	53	85387	May 1, 2024
Keycloak 17.0.1 behind NginX reverse proxy Configuring the server admin-console , documentation , upgrading	17	14328	April 25, 2022
KeyCloak 19.0.1 Admin Console behind k8s nginx gives invalid redirect_uri Configuring the server	9	4856	September 13, 2022
Keycloak Admin Console + NGINX Getting advice admin-console	2	2638	December 4, 2019
Domain problem with Keycloak login for an app \| NGINX Configuring the server admin-console , authentication , oidc	1	994	March 28, 2023

Keycloak - Domain mode - Management console behing Nginx - Error WFLYHC0001

Related topics