Keycloak fine tuned LLM

1

I have been really enjoying working with keycloak! Thanks to all who contribute!
The best way for me to contribute to our community is building an AI kit for keycloak. The AI kit has these components:

  1. Scraper - collects public signals on keycloak.
  2. Indexer - indexes/chunks scrape chunks (uses voyager)
  3. Training - synthesizes 800 questions and answers (trained by claude Sonnet Model) + all other required training data (basically everything from the index)
  4. Fine tuning (Moving to Qwen3-14B )
  5. Query Router ( RAG+commercial LLM, RAG+Keycloak LLM, or hybrid (RAG+commercial LLM ++ RAG+KeyCloak LLM)

As you probably know generalist AI (commercial LLMs) really suck at keycloak questions and they hallucinate with a very high degree of confidence. :slight_smile: I’m still testing the AI Kit but the results look super promising. For my next step I was wondering if I could answer a few of the forum questions using the AI Kit. Or maybe if admins wanted to post some test questions?

Thanks
-Jonathan

2 Likes
2

Would this include development related queries?

Ask a commercial LLM to create a protocol mapper that does x, y, and x and you get a pretty looking solution that doesn’t work. Always a dozen errors for undefined method, classes that don’t exist in the keycloak version you specified, etc, etc, etc. At best you get a starting point that takes additional time debugging and fixing before it works.

3

+1 for @davsum ‘s response. If you’re planning on “trying” this with this forum, I’d suggest you run the LLM locally against the forum post, try out the response to validate that it will in fact solve the problem, and then post your response with 1. the LLM’s response, 2. what you had to change to get it working, and 3. the steps you took to test the solution.