OAuth2 Standard Auth Flow Returns 404 Then Returns 200 on Hard Reload

,
1

I am trying to configure a basic OAuth2 client to authenticate users for JupyterHub. I have it working with one major annoyance:


  1. Unauthorized User enters this address into browser:
    https://datalab.example.com

  1. Unauthorized User is redirected to the following URL. This location initially returns 404, but then returns 200 when page is hard refreshed in Firefox (“Hard Refresh” entails holding down the Shift key while refreshing the URL: Shift+Cmd+R on Mac, Shift+Ctrl+R elsewhere). In Safari and Chrome, there is no getting past the 404:
    https://auth.example.com/realms/research/protocol/openid-connect/auth?response_type=code&redirect_uri=https%3A%2F%2Fdatalab.example.com%2Fhub%2Foauth_callback&client_id=example-datalab&code_challenge=GeWmRK8vA1_O6NjPXrv4kNcCCYgxVTloQS6LVqloYkA&code_challenge_method=S256&state=eyJzdGF0ZV9pZCI6ICJhZGQ3ZDc2ODgyODA0MzEyYjgyZjZkYmFhYjEwYzc2NCJ9&scope=openid+email+profile+roles

  1. After User successfully authenticates with Keycloak, User is redirected to this callback URL. This location initially returns 404, but
    returns 200 when page is hard refreshed in Firefox:
    https://datalab.example.com/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJhZGQ3ZDc2ODgyODA0MzEyYjgyZjZkYmFhYjEwYzc2NCJ9&session_state=87f068e0-e2da-453e-85c7-452a856502cc&iss=https%3A%2F%2Fauth.example.com%2Frealms%2Fresearch&code=320c0ce2-743b-4633-a705-6bf751108a6e.87f068e0-e2da-453e-85c7-452a856502cc.2a977078-9579-45a8-8764-9438aec50ba9
    After the hard refresh here, the User arrives in their newly-spawned JupyterLab environment as exepected.

I have attempted thorough debugging regarding the browser cache to no avail.

Keycloak instance is responsible for SSL termination. Keycloak sits behind Traefik proxy with TLS Passthrough. Keycloak and Traefik are each in their own Docker container, deployed via Docker Compose.

Login to the Keycloak Admin Console at auth.example.com works just fine (without any 404 responses). I can’t for the life of me understand what’s going on here, including why I can ultimately get it to work in Firefox, but not in Safari and Chrome. Any suggestions would be greatly appreciated.

2

I cleared all the logs for JupyterHub and Keycloak, then restarted their containers. I also cleared out the browser cache and the cookies for this site, then reproduced the error again. Here is what I believe to be a pertinent slice of the DEBUG log from Keycloak. The new log was 18,000 lines just for the few minutes that I was working on this - I’ve narrowed it down. I apologize, but I don’t know Java - please let me know if I should include more at either end of this:

2025-07-22 15:57:33,817 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) Couldn't find template in cache for "error.ftl"("en_US", UTF-8, parsed); will try to load it.
2025-07-22 15:57:33,818 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("error_en_US.ftl"): Not found
2025-07-22 15:57:33,818 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("error_en.ftl"): Not found
2025-07-22 15:57:33,818 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("error.ftl"): Found
2025-07-22 15:57:33,818 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) Loading template for "error.ftl"("en_US", UTF-8, parsed) from "jar:file:/opt/keycloak/lib/lib/main/org.keycloak.keycloak-themes-26.3.1.jar!/theme/base/login/error.ftl"
2025-07-22 15:57:33,820 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) Couldn't find template in cache for "template.ftl"("en_US", UTF-8, parsed); will try to load it.
2025-07-22 15:57:33,820 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("template_en_US.ftl"): Not found
2025-07-22 15:57:33,820 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("template_en.ftl"): Not found
2025-07-22 15:57:33,820 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("template.ftl"): Found
2025-07-22 15:57:33,820 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) Loading template for "template.ftl"("en_US", UTF-8, parsed) from "jar:file:/opt/keycloak/lib/lib/main/org.keycloak.keycloak-themes-26.3.1.jar!/theme/keycloak.v2/login/template.ftl"
2025-07-22 15:57:33,828 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) Couldn't find template in cache for "field.ftl"("en_US", UTF-8, parsed); will try to load it.
2025-07-22 15:57:33,828 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("field_en_US.ftl"): Not found
2025-07-22 15:57:33,828 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("field_en.ftl"): Not found
2025-07-22 15:57:33,828 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("field.ftl"): Found
2025-07-22 15:57:33,828 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) Loading template for "field.ftl"("en_US", UTF-8, parsed) from "jar:file:/opt/keycloak/lib/lib/main/org.keycloak.keycloak-themes-26.3.1.jar!/theme/keycloak.v2/login/field.ftl"
2025-07-22 15:57:33,832 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) Couldn't find template in cache for "footer.ftl"("en_US", UTF-8, parsed); will try to load it.
2025-07-22 15:57:33,832 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("footer_en_US.ftl"): Not found
2025-07-22 15:57:33,833 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("footer_en.ftl"): Not found
2025-07-22 15:57:33,833 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) TemplateLoader.findTemplateSource("footer.ftl"): Found
2025-07-22 15:57:33,833 FINE  [freemarker.cache] (vert.x-eventloop-thread-5) Loading template for "footer.ftl"("en_US", UTF-8, parsed) from "jar:file:/opt/keycloak/lib/lib/main/org.keycloak.keycloak-themes-26.3.1.jar!/theme/keycloak.v2/login/footer.ftl"
2025-07-22 15:57:33,844 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (vert.x-eventloop-thread-5) JtaTransactionWrapper rollback. Request Context: Non-HTTP task
2025-07-22 15:57:33,844 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (vert.x-eventloop-thread-5) JtaTransactionWrapper end. Request Context: Non-HTTP task
2025-07-22 15:57:33,844 DEBUG [io.quarkus.micrometer.runtime.binder.vertx.VertxHttpServerMetrics] (vert.x-eventloop-thread-5) responseEnd io.vertx.core.http.impl.Http2ServerResponse@4bba0a6d, HttpRequestMetric [initialPath=/hub/oauth_callback, currentRoutePath=/, templatePath=null, request=io.vertx.core.http.impl.Http2ServerRequest@28025ff2]
2025-07-22 15:57:33,844 DEBUG [org.infinispan.metrics.impl.MetricsRegistryImpl] (vert.x-eventloop-thread-5) Registered metric MeterId{name='http.server.requests', tags=[tag(method=GET),tag(outcome=CLIENT_ERROR),tag(status=404),tag(uri=NOT_FOUND)]}
2025-07-22 15:57:34,011 DEBUG [io.quarkus.micrometer.runtime.binder.vertx.VertxHttpServerMetrics] (vert.x-eventloop-thread-5) requestRouted null HttpRequestMetric [initialPath=/resources/r0l2g/common/keycloak/vendor/patternfly-v5/patternfly.min.css, currentRoutePath=null, templatePath=null, request=io.vertx.core.http.impl.Http2ServerRequest@49191264]
2025-07-22 15:57:34,012 DEBUG [io.quarkus.micrometer.runtime.binder.vertx.VertxHttpServerMetrics] (vert.x-eventloop-thread-5) requestRouted null HttpRequestMetric [initialPath=/resources/r0l2g/common/keycloak/vendor/patternfly-v5/patternfly.min.css, currentRoutePath=null, templatePath=null, request=io.vertx.core.http.impl.Http2ServerRequest@49191264]
2025-07-22 15:57:34,012 DEBUG [io.quarkus.micrometer.runtime.binder.vertx.VertxHttpServerMetrics] (vert.x-eventloop-thread-5) requestRouted / HttpRequestMetric [initialPath=/resources/r0l2g/common/keycloak/vendor/patternfly-v5/patternfly.min.css, currentRoutePath=null, templatePath=null, request=io.vertx.core.http.impl.Http2ServerRequest@49191264]
2025-07-22 15:57:34,013 DEBUG [org.keycloak.transaction.JtaTransactionWrapper] (executor-thread-5) new JtaTransactionWrapper. Was existing transaction suspended: false Request Context: HTTP GET /resources/r0l2g/common/keycloak/vendor/patternfly-v5/patternfly.min.css
3

Hard to guess without showing the (redacted) config

I use

    GenericOAuthenticator:
      admin_groups:
      - jupyter-admin
      allowed_groups:
      - jupyter-admin
      - ...
      authorize_url: https://<kcwith realm>/protocol/openid-connect/auth
      claim_groups_key: groups
      client_id: jupyterhub
      client_secret: ...
      login_service: keycloak
      manage_groups: true
      oauth_callback_url: https://<jupyter hub url>/hub/oauth_callback
      scope: openid microprofile-jwt profile email
      token_url: https://<kcwith realm>/protocol/openid-connect/token
      userdata_from_id_token: true
      userdata_params:
        state: state
      username_claim: preferred_username