Unable to start the service on an ro file system

Hi group,
in my case the service must run on an ro file system, so I have spilt out the static and dynamic files.
The static part:

tree /usr/share/java/keycloak/
/usr/share/java/keycloak/
├── lib
│   ├── app
│   │   └── keycloak.jar
│   ├── lib
│   │   ├── boot
│   │   │   ├── io.github.crac.org-crac-0.1.1.jar
│   │   │   ├──  ....
│   │   │   └── org.wildfly.common.wildfly-common-1.5.4.Final-format-001.jar
│   │   ├── deployment
│   │   │   ├── appmodel.dat
│   │   │   ├── com.fasterxml.jackson.dataformat.jackson-dataformat-yaml-2.13.4.jar
│   │   │   ├── ...
│   │   │   └── org.testcontainers.vault-1.17.3.jar
│   │   └── main
│   │       ├── antlr.antlr-2.7.7.jar
│   │       ├── ...
│   │       └── org.yaml.snakeyaml-1.33.jar
│   └── quarkus-run.jar
├── providers
└── themes

The config part:

tree /etc/keycloak/
/etc/keycloak/
├── cache-ispn.xml
├── keycloak.conf

The dynamic part:

tree /var/lib/keycloak/
/var/lib/keycloak/
├── conf -> /etc/keycloak
├── data
│   ├── h2
│   │   ├── keycloakdb.mv.db
│   │   └── keycloakdb.trace.db
│   └── tmp
│       └── kc-gzip-cache
│           └── 3jagx
│               ├── admin
│               │   └── keycloak.v2
│               │       ├── assets
│               │       │   ├── index.0cb2e516.js.gz
│               │       │   └── index.e2193254.css.gz
│               │       └── favicon.svg.gz
│               ├── common
│               │   └── keycloak
│               │       └── node_modules
│               │           └── patternfly
│               │               └── dist
│               │                   └── css
│               │                       ├── patternfly-additions.css.gz
│               │                       └── patternfly.css.gz
│               └── welcome
│                   └── keycloak
│                       ├── css
│                       │   └── welcome.css.gz
│                       └── img
│                           └── favicon.ico.gz
├── lib
│   ├── app -> /usr/share/java/keycloak/lib/app
│   ├── lib -> /usr/share/java/keycloak/lib/lib
│   ├── quarkus
│   │   ├── build-system.properties
│   │   ├── generated-bytecode.jar
│   │   ├── quarkus-application.dat
│   │   └── transformed-bytecode.jar
│   └── quarkus-run.jar -> /usr/share/java/keycloak/lib/quarkus-run.jar
├── providers -> /usr/share/java/keycloak/providers
└── themes -> /usr/share/java/keycloak/themes

And put all together via symlinks on an rw part of the file system.
But the service will not start, because it will not read the quarkus from the rw space. It will try to load it form the static one.

Nov 29 07:33:15 keycloak[27790]: Appending additional Java properties to JAVA_OPTS: -Djava.net.preferIPv4Stack=false
Nov 29 07:33:15 keycloak[27794]: Exception in thread "main" java.nio.file.NoSuchFileException: /usr/share/java/keycloak/lib/quarkus/quarkus-application.dat
Nov 29 07:33:15 keycloak[27794]:         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
Nov 29 07:33:15 keycloak[27794]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Nov 29 07:33:15 keycloak[27794]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Nov 29 07:33:15 keycloak[27794]:         at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
Nov 29 07:33:15 keycloak[27794]:         at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
Nov 29 07:33:15 keycloak[27794]:         at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
Nov 29 07:33:15 keycloak[27794]:         at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
Nov 29 07:33:15 keycloak[27794]:         at java.base/java.nio.file.Files.newInputStream(Files.java:160)
Nov 29 07:33:15 keycloak[27794]:         at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:52)
Nov 29 07:33:15 keycloak[27794]:         at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:32)
Nov 29 07:33:15 systemd[1]: keycloak.service: Main process exited, code=exited, status=1/FAILURE
Nov 29 07:33:15 systemd[1]: keycloak.service: Failed with result 'exit-code'.

My systemd file:

[Unit]
Description=The Keycloak server
After=network-online.target

[Service]
User=keycloak
Group=keycloak
Environment="JAVA_HOME=/usr/lib/jvm/jre-17" "JAVA_OPTS_APPEND=-Djava.net.preferIPv4Stack=false"
ExecStart=/usr/bin/keycloak start
WorkingDirectory=/var/lib/keycloak
SuccessExitStatus=0 143
PrivateTmp=True

[Install]
WantedBy=multi-user.targed

On the startup script I only have simple adjust the path:

SERVER_OPTS="-Dkc.home.dir='/var/lib/keycloak'"
SERVER_OPTS="$SERVER_OPTS -Djboss.server.config.dir='/etc/keycloak'"
SERVER_OPTS="$SERVER_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
SERVER_OPTS="$SERVER_OPTS -Dquarkus-log-max-startup-records=10000"
CLASSPATH_OPTS="/var/lib/keycloak/lib/quarkus-run.jar"

As far I undestood the app, the quarkus folder (with the quarkus-application.dat file and so on) must be writable. But how must be the ro/rw spilt done?

Thanks for any Ideas.

@mdc_tux have you found the solution for this yet?

No :frowning: As an workaround I try it with an podman container.