Prometheus监控案例-tomcat、mysql、redis、haproxy、nginx
ingress-nginx-controller程序自身可以提供指标数据,默认通过端口10254/metrics提供指标数据,在部署ingress-nginx-controller时候需要在容器上定义10254端口,并在ingress-nginx-controller对应的Service资源上也定义10254端口。基于tomcat官方镜像,重新制作一个镜像,将tomcat-exporter和tom
监控tomcat
tomcat自身并不能提供监控指标数据,需要借助第三方exporter实现:https://github.com/nlighten/tomcat_exporter
构建镜像
基于tomcat官方镜像,重新制作一个镜像,将tomcat-exporter和tomcat整合到一起。Ddockerfile如下:
FROM tomcat:8.5.73
LABEL maintainer="admin@163.com"
ADD server.xml /usr/local/tomcat/conf/
RUN mkdir -p /data/tomcat/webapps
ADD myapp /data/tomcat/webapps/myapp
#添加tomcat-expoter相关文件
ADD metrics.war /data/tomcat/webapps
ADD simpleclient-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_common-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_hotspot-0.8.0.jar /usr/local/tomcat/lib/
ADD simpleclient_servlet-0.8.0.jar /usr/local/tomcat/lib/
ADD tomcat_exporter_client-0.0.17.jar /usr/local/tomcat/lib/
EXPOSE 8080 8443
ADD run_tomcat.sh /usr/bin/
CMD ["/usr/bin/run_tomcat.sh"]
myapp/目录下是一个测试页面index.jsp
root@master-01:~/dockerfile/tomcat# cat myapp/index.jsp
tomcat myapp
tomcat配置文件server.xml中需要指定app路径,并且打开war包的自动解压和部署功能
root@master-01:~/dockerfile/tomcat# cat server.xml |grep /data/tomcat
<Host name="localhost" appBase="/data/tomcat/webapps" unpackWARs="true" autoDeploy="true">
run_tomcat.sh是启动tomcat脚本
root@master-01:~/dockerfile/tomcat# cat run_tomcat.sh
#!/bin/bash
/usr/local/tomcat/bin/catalina.sh start
tail -f /etc/hosts
执行构建,上传到私有仓库
docker build -t harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73 .
docker push harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73
运行一个测试容器,验证metrics接口是否可以访问
docker run -d --name tomcat -p 8080:8080 harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73
部署tomcat
以Deployment方式运行tomcat
apiVersion: apps/v1
kind: Deployment
metadata:
name: tomcat-deployment
namespace: default
spec:
selector:
matchLabels:
app: tomcat
replicas: 1
template:
metadata:
labels:
app: tomcat
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: "8080"
spec:
containers:
- name: tomcat
image: harbor-server.linux.io/n70/tomcat-with-exporter:8.5.73
imagePullPolicy: Always
ports:
- containerPort: 8080
securityContext:
privileged: true
---
kind: Service
apiVersion: v1
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '8080'
name: tomcat-service
spec:
selector:
app: tomcat
ports:
- nodePort: 31080
port: 80
protocol: TCP
targetPort: 8080
type: NodePort
将此yaml文件创建到集群上,然后等待Pod就绪
配置Prometheus采集数据
如果是部署在集群内部的Prometheus,建议使用角色为pod或endpoints的基于API Server的服务发现来发现target并监控。具体job配置可以参考之前的文章《Prometheus服务发现》
如果是集群外的Prometheus,添加一个job,内容如下:
- job_name: tomcat-exporter
static_configs:
- targets: ["192.168.122.12:31080"]
配置修改完成后,重启Prometheus。然后通过Prometheus界面查看target状态
在Grafana上导入模板,查看监控数据。模板地址:https://github.com/nlighten/tomcat_exporter/blob/master/dashboard/example.json
监控mysql
Prometheus官方提供了mysql-expoter用于采集mysql指标数据,项目地址:https://github.com/prometheus/mysqld_exporter
部署mysql
直接以apt方式安装mysql
apt -y install mariadb-server
mysql
mysql授权
创建一个账户并授权,之后mysql-exporter会使用此账户来访问mysql查询相关信息
> CREATE USER 'mysql-exporter'@'localhost' IDENTIFIED BY 'Admin@123' WITH MAX_USER_CONNECTIONS 3;
> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysql-exporter'@'localhost';
mysql -u mysql-exporter -p -h localhost; #验证权限
部署mysql-exporter
下载安装包
wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.14.0/mysqld_exporter-0.14.0.linux-amd64.tar.gz
tar xf mysqld_exporter-0.14.0.linux-amd64.tar.gz
mv mysqld_exporter-0.14.0.linux-amd64/mysqld_exporter /usr/bin/
mysqld_exporter -h #查看帮助信息
准备mysql-exporter配置文件
root@prometheus-node-02:~# mkdir /etc/mysqld-exporter
root@prometheus-node-02:~# cat /etc/mysqld-exporter/.my.cnf
[client]
#指定连接mysql的账户和密码
user = mysql-exporter
password = Admin@123
准备service文件
root@prometheus-node-02:~# cat /lib/systemd/system/mysql-exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target
[Service]
ExecStart=/usr/bin/mysqld_exporter --config.my-cnf=/etc/mysql-exporter/.my.cnf --web.listen-address=192.168.122.23:9104
[Install]
WantedBy=multi-user.target
启动服务
systemctl daemon-reload
systemctl start mysql-exporter.service
systemctl status mysql-exporter.service
systemctl enable mysql-exporter.service
访问mysql-exporter指标接口测试
配置Prometheus采集数据
编辑Prometheus配置,添加一个job,内容如下:
- job_name: mysql-exporter
static_configs:
- targets: ["192.168.122.23:9104"]
配置修改完成后,重启Prometheus。然后通过Prometheus界面查看target状态
Grafana导入模板,查看监控数据。模板地址:https://github.com/prometheus/mysqld_exporter/blob/main/mysqld-mixin/dashboards/mysql-overview.json
监控redis
redis-exporter项目地址:https://github.com/oliver006/redis_exporter
部署redis&redis-exporter
将redis和redis-exporter运行在同一个Pod中,然后以Deployment方式部署Pod
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-deploy
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
annotations:
prometheus.io/scrape: "true"
pometheus.io/port: "9121"
spec:
containers:
- name: redis
image: redis
ports:
- containerPort: 6379
resources:
requests:
cpu: 100m
memory: 100Mi
- name: redis-exporter
image: oliver006/redis_exporter
ports:
- containerPort: 9121
resources:
requests:
cpu: 100m
memory: 100Mi
kind: Service
apiVersion: v1
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: "9121"
name: redis-exporter-service
spec:
selector:
app: redis
ports:
- nodePort: 31082
name: prom
port: 9121
protocol: TCP
targetPort: 9121
type: NodePort
---
kind: Service
apiVersion: v1
metadata:
name: redis-redis-service
spec:
selector:
app: redis
ports:
- nodePort: 31081
name: redis
port: 6379
protocol: TCP
targetPort: 6379
type: NodePort
将yaml文件创建到集群上,等待Pod就绪
通过 任意节点:31082/metrics 访问redis-exporter指标接口测试
配置Prometheus采集数据
编辑Prometheus配置,添加job,内容如下:
- job_name: redis-exporter
static_configs:
- targets: ["192.168.122.13:31082"]
配置修改完成后,重启Prometheus。然后通过Prometheus界面查看target状态
Grafana导入模板,查看监控数据。模板ID:14615
监控Haproxy
Prometheus官方提供了haproxy-exporter用于采集haproxy指标数据,项目地址:https://github.com/prometheus/haproxy_exporter
部署haproxy-exporter
下载安装包
wget https://github.com/prometheus/haproxy_exporter/releases/download/v0.15.0/haproxy_exporter-0.15.0.linux-amd64.tar.gz
tar xf haproxy_exporter-0.15.0.linux-amd64.tar.gz
mv haproxy_exporter-0.15.0.linux-amd64/haproxy_exporter /usr/bin/
haproxy_exporter -h #查看帮助信息
haproxy-exporter可以通过两种方式连接haproxy来获取数据:
- 通过socket文件连接haproxy获取数据,这种方式只能获取本机的haproxy数据
- 通过haproxy状态页连接haproxy获取数据,这种方式可以获取本机或远程主机的数据,但需要在haproxy配置文件中开启状态页面配置
启动hapeoxy-exporter
#如果通过haproxy状态页获取数据,运行下面命令,其中haadmin:123456表示访问状态页的用户名密码
haproxy_exporter --haproxy.scrape-uri="http://haadmin:123456@127.0.0.1:9999/haproxy-status;csv" &
#如果通过socket文件获取数据,运行下面命令
haproxy_exporter --haproxy.scrape-uri="unix:/run/haproxy/admin.sock" &
访问haproxy-exporter指标接口验证,默认端口9101
配置Prometheus采集数据
编辑Prometheus配置添加job,内容如下:
- job_name: haproxy-exporter
static_configs:
- targets: ["192.168.122.14:9101", "192.168.122.15:9101"]
配置修改完成后,重启Prometheus。然后通过Prometheus界面查看target状态
Grafana导入模板,查看监控数据。模板ID: 2428
监控nginx
nginx-exporter项目地址:https://github.com/hnlq715/nginx-vts-exporter
此nginx-exporter依赖一个nginx模块nginx-module-vts(https://github.com/vozlt/nginx-module-vts),所以需要编译安装nginx,并加载此模块。
nginx-module-vts模块会收集nginx指标数据,然后提供一个状态页面,nginx-exporter通过此状态页获取指标数据,再将数据转换为Prometheus需要的数据格式提供给Prometheus。
部署nginx
下载nginx源码包和nginx-module-vts模块
wget http://nginx.org/download/nginx-1.22.1.tar.gz
wget https://github.com/vozlt/nginx-module-vts/archive/refs/tags/v0.2.1.tar.gz
tar xf nginx-module-vts-0.2.1.tar.gz -C /usr/local/src/
tar xvf nginx-1.22.1.tar.gz
cd nginx-1.22.1/
./configure --prefix=/usr/local/nginx \
--with-http_ssl_module \
--with-http_v2_module \
--with-http_realip_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--with-pcre \
--with-file-aio \
--with-stream \
--with-stream_ssl_module \
--with-stream_realip_module \
--add-module=/usr/local/src/nginx-module-vts-0.2.1/
修改nginx配置文件
http {
vhost_traffic_status_zone; #启用状态页
...
server {
...
location /status { #状态页面location配置
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
}
}
启动nginx
/usr/local/nginx/sbin/nginx -t
/usr/local/nginx/sbin/nginx
访问状态页测试
还可以用json格式输出数据
部署nginx-exporter
wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
tar xf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
cp nginx-vts-exporter-0.10.3.linux-amd64/nginx-vts-exporter /usr/bin/
nginx-vts-exporter -nginx.scrape_uri http://192.168.122.23/status/format/json & #启动nginx-exporter
访问nginx-exporter指标接口测试
配置Prometheus采集数据
编辑Prometheus配置,添加Job,内容如下:
- job_name: nginx-exporter
static_configs:
- targets: ["192.168.122.23:9913"]
配置修改完成后,重启Prometheus。然后通过Prometheus界面查看target状态
Grafana导入模板,查看数据。模板地址:https://github.com/hnlq715/nginx-vts-exporter/blob/master/dashboard/nginx-vts-exporter.json
监控ingress-nginx-controller
ingress-nginx-controller程序自身可以提供指标数据,默认通过端口10254/metrics提供指标数据,在部署ingress-nginx-controller时候需要在容器上定义10254端口,并在ingress-nginx-controller对应的Service资源上也定义10254端口
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8443
name: webhook
protocol: TCP
- name: metrics
containerPort: 10254
protocol: TCP
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.3.1
name: ingress-nginx-controller
namespace: ingress-nginx
spec:
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
- name: metrics
port: 10254
targetPort: 10254
protocol: TCP
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: NodePort
之后就可以通过nodePort访问ingress-nginx-controller的指标接口
然后在Prometheus中配置中添加Job,内容如下
- job_name: nginx-ingress-controller
static_configs:
- targets: ["192.168.122.11:38144"]
配置修改完成后,重启Prometheus。通过Prometheus界面查看target状态
Grafana导入模板,查看监控数据。模板ID:9614
开放原子开发者工作坊旨在鼓励更多人参与开源活动,与志同道合的开发者们相互交流开发经验、分享开发心得、获取前沿技术趋势。工作坊有多种形式的开发者活动,如meetup、训练营等,主打技术交流,干货满满,真诚地邀请各位开发者共同参与!
更多推荐
所有评论(0)