What do you mean by “custom identity provider”. Does it work with a generic openid provider?
Do you have to work around some custom oauth2 claims?
Are you trying to achieve something similar to this: Link with Patreon - #18 by andymel? Then you could take that as a starting point and create your own provider.
In general keycloak can be extended without putting your changes into the base source - see Extending Server | keycloak-documentation.