Prometheus监控学习笔记之使用JMX Exporter监控微服务JVM
0x00 概述
本文实现微服务JVM监控的方法为，使用volume HostPath挂载的JMX Exporter的方式在容器内以in-process的方式实现对微服务的JVM监控。
主要坑点为HostPath挂载JMX Exporter javaagent导致微服务无法启动。
0x02 部署
2.1 下载JMX Exporter javaagent包
$ mkdir -p /tmp/jmx_exporter $ wget -O /tmp/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.17.0/jmx_prometheus_javaagent-0.17.0.jar
2.2 配置YAML文件
此处镜像采用Deployment+Tomcat的镜像作为样例，自测的时候注意及时更换镜像；
''' 1. 采用HostPath的方式，将下载的JMX Exporter的jar包挂载到服务器容内启动； 2. JMX Exporter的配置文件以configmap的形式挂载到服务容器内； '''
Tomcat服务的Deployment文件：
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-with-jvmExporter
namespace: default
labels:
app: tomcat-with-jvmExporter
spec:
replicas: 1
selector:
matchLabels:
app: tomcat-with-jvmExporter
template:
metadata:
labels:
app: tomcat-with-jvmExporter
spec:
nodeSelector:
kubernetes.io/hostname: docker49
restartPolicy: Always
containers:
- name: tomcat-with-jvmExporter
image: tomcat:9.0.54-jdk8-openjdk
imagePullPolicy: Always
env:
# JVM env
- name: JAVA_OPTS
value: "-Xms500m -Xmx500m -javaagent:/jmx_prometheus_javaagent-0.17.0.jar=5555:/jmx/prometheus-jmx-config.yaml"
#value: "-Xms500m -Xmx500m"
volumeMounts:
- mountPath: /jmx_prometheus_javaagent-0.17.0.jar
name: jmx-prometheus-javaagent
- mountPath: /jmx
name: prometheus-jmx-config
resources:
limits:
cpu: 2000m
memory: "4096Mi"
requests:
cpu: 2000m
memory: "4096Mi"
volumes:
- name: jmx-prometheus-javaagent
hostPath:
path: /tmp/jmx_prometheus_javaagent-0.17.0.jar
- configMap:
name: prometheus-jmx-config
name: prometheus-jmx-config
''' 1. 通过hostpath挂载的方式，对于pod来说，不一定会调度到这个节点服务器上； 2. 如果pod没调度到有这个jar包的服务器上，在启动pod的时候，会发现pod挂载了jar包和配置文件，但是无法执行这个jar包； 3. 所以需要在yaml中通过nodeSelector指定pod调度到固定节点上，这样hostPath才能取到文件，pod才会成功挂载这个文件并正常运行 4. 上面我们将jar包下载到了docker49这台机器上，所以指定将tomcat的pod调度到docker49，注意红字部分 '''
未配置nodeSelector的话，会出现如下报错：
''' Error opening zip file or JAR manifest missing : /jmx_prometheus_javaagent-0.17.0.jar Error occurred during initialization of VM agent library failed to init: instrument '''
直接搜索以上报错日志，StackOverFlow上大部分都会说是权限问题，但是无论你改成755还是777都一样，配置nodeSelector，将pod调度到JMX Exporter jar包的机器上即可。
Tomcat服务的Service文件：
apiVersion: v1 kind: Service metadata: name: tomcat-with-jvmExporter-svc namespace: default labels: app: tomcat-with-jvmExporter-svc annotations: prometheus.io/port: "5555" prometheus.io/jvm: "true" spec: type: NodePort ports: - port: 8080 name: tomcat-with-jvmExporter-svc targetPort: 30899 nodePort: 30899 - name: jmx-metrics port: 5555 protocol: TCP selector: app: tomcat-with-jvmExporter
Tomcat服务的configmap文件（此处为方便测试采用最小化配置文件，官方配置文件样例）：
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-jmx-config
namespace: default
data:
prometheus-jmx-config.yaml: |
# lowercaseOutputLabelNames: true
# lowercaseOutputName: true
rules:
- pattern: ".*"
# - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
# name: tomcat_$3_total
# labels:
# port: "$2"
# protocol: "$1"
# help: Tomcat global $3
# type: COUNTER
# - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
# name: tomcat_servlet_$3_total
# labels:
# module: "$1"
# servlet: "$2"
# help: Tomcat servlet $3 total
# type: COUNTER
# - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
# name: tomcat_threadpool_$3
# labels:
# port: "$2"
# protocol: "$1"
# help: Tomcat threadpool $3
# type: GAUGE
# - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
# name: tomcat_session_$3_total
# labels:
# context: "$2"
# host: "$1"
# help: Tomcat session $3 total
# type: COUNTER
2.3 All in One yaml文件
tomcat-with-jvmExporter-AllinOne.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-wtih-jvmExporter
namespace: default
labels:
app: tomcat-with-jvmExporter
spec:
replicas: 1
selector:
matchLabels:
app: tomcat-with-jvmExporter
template:
metadata:
labels:
app: tomcat-with-jvmExporter
spec:
nodeSelector:
kubernetes.io/hostname: docker49
restartPolicy: Always
containers:
- name: tomcat-with-jvmExporter
image: tomcat:9.0.54-jdk8-openjdk
imagePullPolicy: Always
env:
# JVM env
- name: JAVA_OPTS
value: "-Xms500m -Xmx500m -javaagent:/jmx_prometheus_javaagent-0.17.0.jar=5555:/jmx/prometheus-jmx-config.yaml"
#value: "-Xms500m -Xmx500m"
volumeMounts:
- mountPath: /jmx_prometheus_javaagent-0.17.0.jar
name: jmx-prometheus-javaagent
- mountPath: /jmx
name: prometheus-jmx-config
resources:
limits:
cpu: 2000m
memory: "4096Mi"
requests:
cpu: 2000m
memory: "4096Mi"
volumes:
- name: jmx-prometheus-javaagent
hostPath:
path: /tmp/jmx_prometheus_javaagent-0.17.0.jar
- configMap:
name: prometheus-jmx-config
name: prometheus-jmx-config
---
apiVersion: v1
kind: Service
metadata:
name: tomcat-with-jvmExporter-svc
namespace: default
labels:
app: tomcat-with-jvmExporter-svc
annotations:
prometheus.io/port: "5555"
prometheus.io/jvm: "true"
spec:
type: NodePort
ports:
- port: 8080
name: tomcat-with-jvmExporter-svc
targetPort: 30899
nodePort: 30899
- name: jmx-metrics
port: 5555
protocol: TCP
selector:
app: tomcat-with-jvmExporter
---
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-jmx-config
namespace: default
data:
prometheus-jmx-config.yaml: |
#lowercaseOutputLabelNames: true
#lowercaseOutputName: true
rules:
- pattern: ".*"
# - pattern: 'Catalina<type=GlobalRequestProcessor, name=\"(\w+-\w+)-(\d+)\"><>(\w+):'
# name: tomcat_$3_total
# labels:
# port: "$2"
# protocol: "$1"
# help: Tomcat global $3
# type: COUNTER
# - pattern: 'Catalina<j2eeType=Servlet, WebModule=//([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), name=([-a-zA-Z0-9+/$%~_-|!.]*), J2EEApplication=none, J2EEServer=none><>(requestCount|maxTime|processingTime|errorCount):'
# name: tomcat_servlet_$3_total
# labels:
# module: "$1"
# servlet: "$2"
# help: Tomcat servlet $3 total
# type: COUNTER
# - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount):'
# name: tomcat_threadpool_$3
# labels:
# port: "$2"
# protocol: "$1"
# help: Tomcat threadpool $3
# type: GAUGE
# - pattern: 'Catalina<type=Manager, host=([-a-zA-Z0-9+&@#/%?=~_|!:.,;]*[-a-zA-Z0-9+&@#/%=~_|]), context=([-a-zA-Z0-9+/$%~_-|!.]*)><>(processingTime|sessionCounter|rejectedSessions|expiredSessions):'
# name: tomcat_session_$3_total
# labels:
# context: "$2"
# host: "$1"
# help: Tomcat session $3 total
# type: COUNTER
2.4 启动并检查服务
启动服务：
kubectl apply -f tomcat-with-jvmExporter-AllinOne.yaml
检查服务：
kubectl get svc -n default
检查JVM Exporter是否已获取JVM监控数据，此处docker49 ip：192.168.6.6，JVM Exporter对外nodeExporter为30566
curl 192.168.6.6:30566/metrics
如果返回以下数据，说明已成功获取监控数据：
jvm_threads_daemon 36.0
# HELP jvm_threads_peak Peak thread count of a JVM
# TYPE jvm_threads_peak gauge
jvm_threads_peak 80.0
# HELP jvm_threads_started_total Started thread count of a JVM
# TYPE jvm_threads_started_total counter
jvm_threads_started_total 155322.0
# HELP jvm_threads_deadlocked Cycles of JVM-threads that are in deadlock waiting to acquire object monitors or ownable synchronizers
# TYPE jvm_threads_deadlocked gauge
jvm_threads_deadlocked 0.0
# HELP jvm_threads_deadlocked_monitor Cycles of JVM-threads that are in deadlock waiting to acquire object monitors
# TYPE jvm_threads_deadlocked_monitor gauge
jvm_threads_deadlocked_monitor 0.0
# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="TIMED_WAITING",} 22.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 12.0
jvm_threads_state{state="WAITING",} 41.0
# HELP jvm_classes_currently_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_currently_loaded gauge
jvm_classes_currently_loaded 18332.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 18478.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 146.0
# HELP jmx_config_reload_failure_created Number of times configuration have failed to be reloaded.
# TYPE jmx_config_reload_failure_created gauge
jmx_config_reload_failure_created 1.655778604558E9
# HELP jmx_config_reload_success_created Number of times configuration have successfully been reloaded.
# TYPE jmx_config_reload_success_created gauge
jmx_config_reload_success_created 1.655778604539E9
# HELP jvm_memory_pool_allocated_bytes_created Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
# TYPE jvm_memory_pool_allocated_bytes_created gauge
jvm_memory_pool_allocated_bytes_created{pool="Code Cache",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="PS Eden Space",} 1.655778606951E9
jvm_memory_pool_allocated_bytes_created{pool="PS Old Gen",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="PS Survivor Space",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="Compressed Class Space",} 1.655778606954E9
jvm_memory_pool_allocated_bytes_created{pool="Metaspace",} 1.655778606954E9
0x03 配置Prometheus获取监控个数据
修改prometheus.yml配置文件，新增JVM Exporter数据
- job_name: "jvm-ucm-config"
honor_labels: true
static_configs:
- targets: ["192.168.6.6:30566"]
重新加载Prometheus配置文件
kill -HUP $pid
0x04 Grafana导入JVM监控面板
推荐面板id：11278
需要修改面板变量，根据以上配置，变量设置为2，job+instance
0x05 参考
