Doing some more research here. In the front-channel OIDC spec, it says:
RPs supporting HTTP-based logout register a logout URI with the OP as part of their client registration. The domain, port, and scheme of this URL MUST be the same as that of a registered Redirection URI value.
I’ve added the Realm B OIDC logout url to the Realm A client redirect URIs, but no effect.