Keycloak fails X509 Authentication when OCSP is enabled

Getting advice
1

I have X509 Authentication working properly using Keycloak and SAML. When I enable OCSP and point it to our OCSP Server, I get the following error and I can no longer login.

2019-12-20 09:39:29,033 ERROR [org.keycloak.services] (default task-16) No trusted CA in certificate found: CN=DOD ID CA-49, OU=PKI, OU=DoD, O=U.S. Government, C=US. Add it to truststore SPI if valid.: java.security.GeneralSecurityException: No trusted CA in certificate found: CN=DOD ID CA-49, OU=PKI, OU=DoD, O=U.S. Government, C=US. Add it to truststore SPI if valid.

In the standalone.xml file I have added the following.

        <spi name="truststore">
            <provider name="file" enabled="true">
            <properties>
                    <property name="file" value="/usr/java/latest/jre/lib/security/cacerts"/>
                    <property name="password" value="changeit"/>
                    <property name="hostname-verification-policy" value="WILDCARD"/>
                    <property name="disabled" value="false"/>
            </properties>
            </provider>
        </spi>

When I run the keytool command like this it shows that DOD ID CA-49 and DOD ROOT CA 3 exist.

keytool -list -keystore /usr/java/latest/jre/lib/security/cacerts

dod_id-ca-49, Dec 13, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): 6C:D6:E8:BD:7A:CD:2F:08:E2:16:93:98:8A:30:9E:CA:67:72:C1:34
dod_root_ca_3, Dec 11, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): D7:3C:A9:11:02:A2:20:4A:36:45:9E:D3:22:13:B4:67:D7:CE:97:FB

I cannot figure out how to get keycloak to find the CA certs that I added. There is also these settings in the standalone.xml

       <security-realm name="ApplicationRealm">
            <server-identities>
                <ssl>
                    <keystore path="server.keystore" relative-to="jboss.server.config.dir" keystore-password="changeit" alias="server" key-password="changeit"/>
                </ssl>
            </server-identities>
    <authentication>
            <truststore path="cacerts" relative-to="jboss.server.config.dir" keystore-password="changeit" />
            <local default-user="$local"/>
            <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
    </authentication>
            <authorization>
                <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
            </authorization>
        </security-realm>
1 Like
2

Currently having the same problem. Were you able to resolve the issue?

3

Just for others who are looking here; the error happens because Keycloak is looking for the SSL_CLIENT_CERT_CHAIN values from Apache.

On the apache end, you’ll need something like this

RequestHeader set SSL_CLIENT_CERT_CHAIN_0 "%{SSL_CLIENT_CERT_CHAIN_0}s"
RequestHeader set SSL_CLIENT_CERT_CHAIN_1 "%{SSL_CLIENT_CERT_CHAIN_1}s"

and on the Keycloak end, you’ll need to add something like

    <spi name="x509cert-lookup">
        <default-provider>apache</default-provider>
        <provider name="apache" enabled="true">
            <properties>
                <property name="sslClientCert" value="SSL_CLIENT_CERT"/>
                <property name="sslCertChainPrefix" value="SSL_CLIENT_CERT_CHAIN"/>
                <property name="certificateChainLength" value="2"/>
            </properties>
        </provider>
    </spi>

As directed in the documentation for " Apache certificate lookup provider" in the Server Administration documentation.

Additionally, you’ll probably need to set the following variables in Apache where N is a 1024 multiple of your choosing, such as 65536 (depending on your certificate chain size)

LimitRequestFieldSize N
ProxyIOBufferSize N

Because without it, you’ll likely have an ssl_error_log that says

AH00976: ajp_marshal_into_msgb: Error appending the SSL certificates

Because your certificates will exceed the 8kb defaults. Hope this helps someone.

4

I am still getting this in the logs and not certain why. I thought I added all of these to the default Java keystore under /usr/java/latest/lib/security/cacerts

2020-11-18 17:17:21,281 ERROR [org.keycloak.services] (default task-2) No trusted CA in certificate found: CN=DOD ID CA-50, OU=PKI, OU=DoD, O=U.S. Government, C=US. Add it to truststore SPI if valid.: java.security.GeneralSecurityException: No trusted CA in certificate found: CN=DOD ID CA-50, OU=PKI, OU=DoD, O=U.S. Government, C=US. Add it to truststore SPI if valid.

5

I was also mystified because I had added the appropriate CA to the truststore SPI, but the message is a bit misleading.

The source of the error is that Keycloak wants to see the SSL_CLIENT_CERT_CHAIN_0 header passed from Apache (and SSL_CLIENT_CERT_CHAIN_1, etc for the rest of the chain as configured in Apache).

The RequestHeader set directives I posted above will pass it from Apache (or other load balancer) to Keycloak.

And for the SSL_CLIENT_CERT_CHAIN fields to populate properly, you’ll probably also need to set the SSLCACertificateFile directive pointing to the DoD CA chain you have.

This is all assuming you’re using Apache though, are you behind a load balancer?

6

Yes, I am using Apache 2.4 to proxy the request to the keycloak server. This is my config file, but it still gives me the following error.

2020-11-19 13:39:38,166 ERROR [org.keycloak.services.x509.AbstractClientCertificateFromHttpHeadersLookup] (default task-1) org.keycloak.common.util.PemException: java.io.IOException: Bad Base64 input character decimal 40 in array position 0: org.keycloak.common.util.PemException: org.keycloak.common.util.PemException: java.io.IOException: Bad Base64 input character decimal 40 in array position 0
2020-11-19 13:39:38,171 ERROR [org.keycloak.services] (default task-1) No trusted CA in certificate found: CN=DOD ID CA-50, OU=PKI, OU=DoD, O=U.S. Government, C=US. Add it to truststore SPI if valid.: java.security.GeneralSecurityException: No trusted CA in certificate found: CN=DOD ID CA-50, OU=PKI, OU=DoD, O=U.S. Government, C=US. Add it to truststore SPI if valid.

<VirtualHost 10.110.110.68:443>
ServerName auth.example.com
ProxyPreserveHost On
SSLEngine On
SSLProtocol -ALL +TLSv1.2
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
SSLProxyEngine On
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
SSLCertificateFile /etc/certs/example.com/example.com.crt
SSLCertificateKeyFile /etc/certs/example.com/example.com.key
SSLCACertificateFile /etc/certs/DoD_CAs.pem
RemoteIPHeader X-Forwarded-For

Enable higher values to properly pass the certificate chain to Keycloak

LimitRequestFieldSize 65536
ProxyIOBufferSize 65536

STIG V-92795 - Added by Jeff West 2020-04-08

Header always edit Set-Cookie ^(.*)$ $1;HttpOnly;secure

STIG V-92801 - Added by Jeff West 2020-04-08

Timeout 10

<Proxy *>
SSLRequireSSL
SSLVerifyClient      require
SSLVerifyDepth       2
SSLRequire           %{SSL_CIPHER_USEKEYSIZE} >= 128
AddDefaultCharset Off
Order deny,allow
Allow from all
</Proxy>

# initialize the special headers to a blank value to avoid http header forgeries
RequestHeader set SSL_CLIENT_S_DN           ""
RequestHeader set SSL_CLIENT_I_DN           ""
RequestHeader set SSL_SERVER_S_DN_OU        ""
RequestHeader set SSL_CLIENT_VERIFY         ""
RequestHeader set SSL_CLIENT_CERT           ""
RequestHeader set SSL_CLIENT_CERT_CHAIN_0           ""
RequestHeader set SSL_CLIENT_CERT_CHAIN_1           ""

<Location />
  # add all the SSL_* you need in the internal web application
  RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
  RequestHeader set SSL_CLIENT_I_DN "%{SSL_CLIENT_I_DN}s"
  RequestHeader set SSL_SERVER_S_DN_OU "%{SSL_SERVER_S_DN_OU}s"
  RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"
  RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
  RequestHeader set SSL_CLIENT_CERT_CHAIN_0 "%{SSL_CLIENT_CERT_CHAIN_0}s"
  RequestHeader set SSL_CLIENT_CERT_CHAIN_1 "%{SSL_CLIENT_CERT_CHAIN_1}s"
  ProxyPass https://<ip_address>:8443/