Hello,
I’m using keycloak 20.0.1 in k8s set up using the operator.
I’m trying to figure out what are core metrics. According to Enabling Keycloak Metrics - Keycloak, there should be HTTP and database metrics but I can’t see any.
The metrics that I see:
# HELP base_classloader_loadedClasses_count Displays the number of classes that are currently loaded in the Java virtual machine.
# TYPE base_classloader_loadedClasses_count gauge
base_classloader_loadedClasses_count 19779.0
# HELP base_classloader_loadedClasses_total Displays the total number of classes that have been loaded since the Java virtual machine has started execution.
# TYPE base_classloader_loadedClasses_total counter
base_classloader_loadedClasses_total 20034.0
# HELP base_classloader_unloadedClasses_total Displays the total number of classes unloaded since the Java virtual machine has started execution.
# TYPE base_classloader_unloadedClasses_total counter
base_classloader_unloadedClasses_total 255.0
# HELP base_cpu_availableProcessors Displays the number of processors available to the Java virtual machine. This value may change during a particular invocation of the virtual machine.
# TYPE base_cpu_availableProcessors gauge
base_cpu_availableProcessors 2.0
# HELP base_cpu_processCpuLoad_percent Displays the "recent cpu usage" for the Java Virtual Machine process. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that none of the CPUs were running threads from the JVM process during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running threads from the JVM 100% of the time during the recent period being observed. Threads from the JVM include the application threads as well as the JVM internal threads. All values between 0.0 and 1.0 are possible depending of the activities going on in the JVM process and the whole system. If the Java Virtual Machine recent CPU usage is not available, the method returns a negative value.
# TYPE base_cpu_processCpuLoad_percent gauge
base_cpu_processCpuLoad_percent 0.0
# HELP base_cpu_systemLoadAverage Displays the system load average for the last minute. The system load average is the sum of the number of runnable entities queued to the available processors and the number of runnable entities running on the available processors averaged over a period of time. The way in which the load average is calculated is operating system specific but is typically a damped time-dependent average. If the load average is not available, a negative value is displayed. This attribute is designed to provide a hint about the system load and may be queried frequently. The load average may be unavailable on some platforms where it is expensive to implement this method.
# TYPE base_cpu_systemLoadAverage gauge
base_cpu_systemLoadAverage 20.76
# HELP base_gc_time_total Displays the approximate accumulated collection elapsed time in milliseconds. This attribute displays -1 if the collection elapsed time is undefined for this collector. The Java virtual machine implementation may use a high resolution timer to measure the elapsed time. This attribute may display the same value even if the collection count has been incremented if the collection elapsed time is very short.
# TYPE base_gc_time_total counter
base_gc_time_total_seconds{name="G1 Old Generation"} 0.0
base_gc_time_total_seconds{name="G1 Young Generation"} 180.698
# HELP base_gc_total Displays the total number of collections that have occurred. This attribute lists -1 if the collection count is undefined for this collector.
# TYPE base_gc_total counter
base_gc_total{name="G1 Old Generation"} 0.0
base_gc_total{name="G1 Young Generation"} 390.0
# HELP base_jvm_uptime_seconds Displays the time from the start of the Java virtual machine in milliseconds.
# TYPE base_jvm_uptime_seconds gauge
base_jvm_uptime_seconds 53613.289
# HELP base_memory_committedHeap_bytes Displays the amount of memory in bytes that is committed for the Java virtual machine to use. This amount of memory is guaranteed for the Java virtual machine to use.
# TYPE base_memory_committedHeap_bytes gauge
base_memory_committedHeap_bytes 1.59383552E8
# HELP base_memory_maxHeap_bytes Displays the maximum amount of heap memory in bytes that can be used for memory management. This attribute displays -1 if the maximum heap memory size is undefined. This amount of memory is not guaranteed to be available for memory management if it is greater than the amount of committed memory. The Java virtual machine may fail to allocate memory even if the amount of used memory does not exceed this maximum size.
# TYPE base_memory_maxHeap_bytes gauge
base_memory_maxHeap_bytes 5.36870912E8
# HELP base_memory_usedHeap_bytes Displays the amount of used heap memory in bytes.
# TYPE base_memory_usedHeap_bytes gauge
base_memory_usedHeap_bytes 1.12759888E8
# HELP base_thread_count Displays the current number of live threads including both daemon and non-daemon threads
# TYPE base_thread_count gauge
base_thread_count 31.0
# HELP base_thread_daemon_count Displays the current number of live daemon threads.
# TYPE base_thread_daemon_count gauge
base_thread_daemon_count 19.0
# HELP base_thread_max_count Displays the peak live thread count since the Java virtual machine started or peak was reset. This includes daemon and non-daemon threads.
# TYPE base_thread_max_count gauge
base_thread_max_count 38.0
# HELP vendor_agroal_acquire_count_total Number of times an acquire operation succeeded.
# TYPE vendor_agroal_acquire_count_total counter
vendor_agroal_acquire_count_total{datasource="default"} 251.0
# HELP vendor_agroal_active_count Number of active connections. These connections are in use and not available to be acquired.
# TYPE vendor_agroal_active_count gauge
vendor_agroal_active_count{datasource="default"} 0.0
# HELP vendor_agroal_available_count Number of idle connections in the pool, available to be acquired.
# TYPE vendor_agroal_available_count gauge
vendor_agroal_available_count{datasource="default"} 1.0
# HELP vendor_agroal_awaiting_count Approximate number of threads blocked, waiting to acquire a connection.
# TYPE vendor_agroal_awaiting_count gauge
vendor_agroal_awaiting_count{datasource="default"} 0.0
# HELP vendor_agroal_blocking_time_average_seconds Average time an application waited to acquire a connection.
# TYPE vendor_agroal_blocking_time_average_seconds gauge
vendor_agroal_blocking_time_average_seconds{datasource="default"} 0.016
# HELP vendor_agroal_blocking_time_max_seconds Maximum time an application waited to acquire a connection.
# TYPE vendor_agroal_blocking_time_max_seconds gauge
vendor_agroal_blocking_time_max_seconds{datasource="default"} 1.992
# HELP vendor_agroal_blocking_time_total_seconds Total time applications waited to acquire a connection.
# TYPE vendor_agroal_blocking_time_total_seconds gauge
vendor_agroal_blocking_time_total_seconds{datasource="default"} 4.131
# HELP vendor_agroal_creation_count_total Number of created connections.
# TYPE vendor_agroal_creation_count_total counter
vendor_agroal_creation_count_total{datasource="default"} 117.0
# HELP vendor_agroal_creation_time_average_seconds Average time for a connection to be created.
# TYPE vendor_agroal_creation_time_average_seconds gauge
vendor_agroal_creation_time_average_seconds{datasource="default"} 4.502
# HELP vendor_agroal_creation_time_max_seconds Maximum time for a connection to be created.
# TYPE vendor_agroal_creation_time_max_seconds gauge
vendor_agroal_creation_time_max_seconds{datasource="default"} 56.369
# HELP vendor_agroal_creation_time_total_seconds Total time waiting for connections to be created.
# TYPE vendor_agroal_creation_time_total_seconds gauge
vendor_agroal_creation_time_total_seconds{datasource="default"} 526.8
# HELP vendor_agroal_destroy_count_total Number of destroyed connections.
# TYPE vendor_agroal_destroy_count_total counter
vendor_agroal_destroy_count_total{datasource="default"} 116.0
# HELP vendor_agroal_flush_count_total Number of connections removed from the pool, not counting invalid / idle.
# TYPE vendor_agroal_flush_count_total counter
vendor_agroal_flush_count_total{datasource="default"} 0.0
# HELP vendor_agroal_invalid_count_total Number of connections removed from the pool for being idle.
# TYPE vendor_agroal_invalid_count_total counter
vendor_agroal_invalid_count_total{datasource="default"} 0.0
# HELP vendor_agroal_leak_detection_count_total Number of times a leak was detected. A single connection can be detected multiple times.
# TYPE vendor_agroal_leak_detection_count_total counter
vendor_agroal_leak_detection_count_total{datasource="default"} 0.0
# HELP vendor_agroal_max_used_count Maximum number of connections active simultaneously.
# TYPE vendor_agroal_max_used_count gauge
vendor_agroal_max_used_count{datasource="default"} 3.0
# HELP vendor_agroal_reap_count_total Number of connections removed from the pool for being idle.
# TYPE vendor_agroal_reap_count_total counter
vendor_agroal_reap_count_total{datasource="default"} 116.0
# HELP vendor_cpu_processCpuTime_seconds Displays the CPU time used by the process on which the Java virtual machine is running in nanoseconds. The returned value is of nanoseconds precision but not necessarily nanoseconds accuracy. This method returns -1 if the the platform does not support this operation.
# TYPE vendor_cpu_processCpuTime_seconds gauge
vendor_cpu_processCpuTime_seconds 157.99
# HELP vendor_cpu_systemCpuLoad_percent Displays the "recent cpu usage" for the whole system. This value is a double in the [0.0,1.0] interval. A value of 0.0 means that all CPUs were idle during the recent period of time observed, while a value of 1.0 means that all CPUs were actively running 100% of the time during the recent period being observed. All values betweens 0.0 and 1.0 are possible depending of the activities going on in the system. If the system recent cpu usage is not available, the method returns a negative value.
# TYPE vendor_cpu_systemCpuLoad_percent gauge
vendor_cpu_systemCpuLoad_percent 0.006558748421052632
# HELP vendor_memory_committedNonHeap_bytes Displays the amount of non heap memory in bytes that is committed for the Java virtual machine to use.
# TYPE vendor_memory_committedNonHeap_bytes gauge
vendor_memory_committedNonHeap_bytes 1.74206976E8
# HELP vendor_memory_freePhysicalSize_bytes Displays the amount of free physical memory in bytes.
# TYPE vendor_memory_freePhysicalSize_bytes gauge
vendor_memory_freePhysicalSize_bytes 1.5466381312E10
# HELP vendor_memory_freeSwapSize_bytes Displays the amount of free swap space in bytes.
# TYPE vendor_memory_freeSwapSize_bytes gauge
vendor_memory_freeSwapSize_bytes 0.0
# HELP vendor_memory_maxNonHeap_bytes Displays the maximum amount of used non-heap memory in bytes.
# TYPE vendor_memory_maxNonHeap_bytes gauge
vendor_memory_maxNonHeap_bytes 7.80140544E8
# HELP vendor_memory_usedNonHeap_bytes Displays the amount of used non-heap memory in bytes.
# TYPE vendor_memory_usedNonHeap_bytes gauge
vendor_memory_usedNonHeap_bytes 1.691332E8
# HELP vendor_memoryPool_usage_bytes Current usage of the memory pool denoted by the 'name' tag
# TYPE vendor_memoryPool_usage_bytes gauge
vendor_memoryPool_usage_bytes{name="CodeHeap 'non-nmethods'"} 1576704.0
vendor_memoryPool_usage_bytes{name="CodeHeap 'non-profiled nmethods'"} 1.1100416E7
vendor_memoryPool_usage_bytes{name="CodeHeap 'profiled nmethods'"} 3.1798272E7
vendor_memoryPool_usage_bytes{name="Compressed Class Space"} 1.3309E7
vendor_memoryPool_usage_bytes{name="G1 Eden Space"} 0.0
vendor_memoryPool_usage_bytes{name="G1 Old Gen"} 4.2039632E7
vendor_memoryPool_usage_bytes{name="G1 Survivor Space"} 4194304.0
vendor_memoryPool_usage_bytes{name="Metaspace"} 1.11348808E8
# HELP vendor_memoryPool_usage_max_bytes Peak usage of the memory pool denoted by the 'name' tag
# TYPE vendor_memoryPool_usage_max_bytes gauge
vendor_memoryPool_usage_max_bytes{name="CodeHeap 'non-nmethods'"} 1621376.0
vendor_memoryPool_usage_max_bytes{name="CodeHeap 'non-profiled nmethods'"} 1.1100416E7
vendor_memoryPool_usage_max_bytes{name="CodeHeap 'profiled nmethods'"} 3.1798272E7
vendor_memoryPool_usage_max_bytes{name="Compressed Class Space"} 1.3309E7
vendor_memoryPool_usage_max_bytes{name="G1 Eden Space"} 8.8080384E7
vendor_memoryPool_usage_max_bytes{name="G1 Old Gen"} 5.2195728E7
vendor_memoryPool_usage_max_bytes{name="G1 Survivor Space"} 9437184.0
vendor_memoryPool_usage_max_bytes{name="Metaspace"} 1.11348808E8```
Have I misconfigured anything?
I see that some people use SPI https://github.com/aerogear/keycloak-metrics-spi , but following the discussion here https://github.com/keycloak/keycloak/discussions/8490#discussioncomment-4837281 I'm not sure if it makes sense to do that as I've seen that there've been lots of changes introduced lately and the issue seems to be still ongoing so I guess that even more will appear soon
What is the recommendation at the moment?
Best regards,
Tomek