搜索中...
🔍

未找到相关结果

Akemi

recording rules与使用consul kv管理

Kubernetes Prometheus 监控
字数统计: 1.3k阅读时长: 6 min
2026/08/04

recording rules的核心优势，是将海量数据进行提前计算
比如cpu mem使用率这类基础的数据，如果节点只有10 50台可能无所谓，查询的时候感觉不出区别，但如果有500台，那是否使用recording rules的差异就会相当明显

编写recording rules——使用prometheusRule CRD

旨在说明如何进行recording rules配置编写与使用，这种方式也是kube-prometheus-stack使用的方式，是使用Operator的原生方式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: custom-recording-rules
  namespace: monitoring
  labels:
    release: prometheus                # 匹配 Operator 的 ruleSelector
spec:
  groups:
    # 节点级聚合
    - name: node_recording_rules 
      interval: 30s
      rules:
        # 1. 节点 CPU 使用率（5分钟均值，0~1）
        - record: node:cpu_usage:avg_rate5m
          expr: |
            avg by (instance) (
              rate(node_cpu_seconds_total{mode!="idle"}[5m])
            )

        # 2. 节点内存使用率（0~1）
        - record: node:mem_usage:ratio
          expr: |
            1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

        # 3. 节点磁盘使用率（0~1，排除临时文件系统）
        - record: node:disk_usage:ratio
          expr: |
            1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
              / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"}

        # 4. 节点网络接收速率（bytes/s）
        - record: node:network_receive:rate5m
          expr: |
            sum by (instance) (
              rate(node_network_receive_bytes_total{device!="lo"}[5m])
            )
            
    # pod级聚合
    - name: container_recording_rules
      interval: 30s
      rules:
        # 1. 容器 CPU 使用（cores，0.5 = 半个核心）
        - record: container:cpu_usage:rate5m
          expr: |
            sum by (namespace, pod, container) (
              rate(container_cpu_usage_seconds_total{container!=""}[5m])
            )

        # 2. 容器内存工作集（bytes）
        - record: container:mem_working_set:bytes
          expr: |
            sum by (namespace, pod, container) (
              container_memory_working_set_bytes{container!=""}
            )

        # 3. 容器 CPU 限流比例（被限流的时间占比，越高越卡）
        - record: container:cpu_throttled:ratio
          expr: |
            sum by (namespace, pod, container) (
              rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m])
            )
            /
            sum by (namespace, pod, container) (
              rate(container_cpu_cfs_periods_total{container!=""}[5m])
            )

        # 4. 容器内存使用占 limit 的比例（0~1，接近1就快OOM了）
        - record: container:mem_limit_usage:ratio
          expr: |
            container_memory_working_set_bytes{container!=""}
            /
            kube_pod_container_resource_limits{resource="memory"}

查看prometheusd规则是否生效

使用recording rules

直接在grafana中进行查询，带上查询条件就可以查询到，甚至可以服用到其他的告警规则中
红框是这个版本的grafana可以直接解析聚合的告警，老版本好像没有这个功能

管理recording rules——集成consul

consul tempalte渲染prometheusRule，化整为零

准备consul kv目录结构

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
目录结构
consul kv
└── record-rule/
    ├── node/          # 节点级指标
    ├── pod/           # 容器级指标
    └── cluster/       # 集群级指标

cluster级别聚合告警：
- name: cluster_recording_rules
  interval: 30s
  rules:
    # apiserver 每秒请求速率
    - expr: |
        sum by (verb) (
        rate(apiserver_request_total{code!~"5.."}[5m])
        )
      record: cluster:apiserver_request_rate:sum_rate5m
    # apiserver 5xx 错误率
    - expr: |
        sum by (verb) (
        rate(apiserver_request_total{code=~"5.."}[5m])
        )
        /
        sum by (verb) (
        rate(apiserver_request_total[5m])
        )
      record: cluster:apiserver_error_rate:ratio_rate5m
    # 过去1小时 etcd leader 切换次数
    - expr: |
        sum(
        increase(etcd_server_leader_changes_seen_total[1h])
        )
      record: cluster:etcd_leader_changes:sum_increase1h
    # etcd WAL fsync 的 P99 延迟
    - expr: |
        histogram_quantile(0.99,
        rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])
        )
      record: cluster:etcd_wal_fsync_p99:histogram_quantile5m

准备consul-template模板

根据上面我写的这个consul kv的格式写，感觉其实也不难，因为其实和helm的go语言渲染的方式其实也差不多
创建对应格式的PrometheusRule模板
{{- range $item := tree “record-rule/“ }} → 循环，遍历record-rule/下的文件
{{ $item.Value | indent 2 }} → 取当前values的值，且增加两格缩进
{{- end }} → 循环结束

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: custom-recording-rules
  namespace: monitoring
  labels:
    release: prometheus
spec:
  groups:
{{- range $item := tree "record-rule/" }}
{{ $item.Value | indent 2 }}
{{- end }}

创建consul-template

这个tempalte要带kubectl命令，所以从bitnami/kubectl:latest做init容器复制过来，自己打镜像太麻烦了，搞一个通用的yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
---
# RBAC: 授权 consul-template 管理 PrometheusRule
apiVersion: v1
kind: ServiceAccount
metadata:
  name: consul-template-prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: consul-template-prometheus
rules:
  - apiGroups: ["monitoring.coreos.com"]
    resources: ["prometheusrules"]
    verbs: ["get", "list", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: consul-template-prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: consul-template-prometheus
subjects:
  - kind: ServiceAccount
    name: consul-template-prometheus
    namespace: monitoring
---
# 模板内容
apiVersion: v1
kind: ConfigMap
metadata:
  name: consul-template-prometheus-tmpl
  namespace: monitoring
data:
  recording-rules.yml.ctmpl: |
    apiVersion: monitoring.coreos.com/v1
    kind: PrometheusRule
    metadata:
      name: custom-recording-rules
      namespace: monitoring
      labels:
        release: prometheus
    spec:
      groups:
    {{- range $item := tree "record-rule/" }}
    {{ $item.Value | indent 4 }}
    {{- end }}
---
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: consul-template-prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: consul-template-prometheus
  template:
    metadata:
      labels:
        app: consul-template-prometheus
    spec:
      serviceAccountName: consul-template-prometheus
      shareProcessNamespace: true
      initContainers:
        # 从 bitnami/kubectl 拷贝 kubectl 到共享 volume
        - name: kubectl-init
          image: bitnami/kubectl:latest
          command: ["cp", "/opt/bitnami/kubectl/bin/kubectl", "/kubectl-bin/kubectl"]
          volumeMounts:
            - name: kubectl-bin
              mountPath: /kubectl-bin
      containers:
        # consul-template: 监听 KV 变化，渲染模板，执行 kubectl apply
        - name: consul-template
          image: hashicorp/consul-template:0.39.0
          command:
            - consul-template
            - -consul-addr=consul-consul-server.consul.svc.cluster.local:8500 # 指定consul server的地址
            - -template=/etc/ct-templates/recording-rules.yml.ctmpl:/etc/rendered/recording-rules.yml:/kubectl-bin/kubectl apply -f /etc/rendered/recording-rules.yml # 模板以及对应的apply的文件
            - -log-level=info
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 50m
              memory: 64Mi
          volumeMounts:
            - name: ct-templates
              mountPath: /etc/ct-templates
            - name: rendered
              mountPath: /etc/rendered
            - name: kubectl-bin
              mountPath: /kubectl-bin
      volumes:
        - name: ct-templates
          configMap:
            name: consul-template-prometheus-tmpl
        - name: rendered
          emptyDir: {}
        - name: kubectl-bin
          emptyDir: {}
          
          
kubectl apply -f consul-template.yaml
serviceaccount/consul-template-prometheus created
clusterrole.rbac.authorization.k8s.io/consul-template-prometheus created
clusterrolebinding.rbac.authorization.k8s.io/consul-template-prometheus created
configmap/consul-template-prometheus-tmpl created
deployment.apps/consul-template-prometheus created

测试consul kv管理record-rule

1
2
3
首先要把我之前prometheusRule CRD的那个删了，不然这里体现不出来
cd victoria-metrics/charts/victoria-metrics-cluster/
kubectl delete -f node_recording_rules.yaml


原文作者：王盛

原文链接：https://akemi.zj.cn/2026/08/04/Prometheus-Recording-Rules-Consul-KV/

发表日期：August 4th 2026, 8:50:00 pm

更新日期：August 4th 2026, 8:50:59 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

📚 相关文章推荐

2026-04-08 Prometheus使用process-exporter #Prometheus #监控 2026-07-09 使用blackbox-exporter #Prometheus #监控 2026-06-01 VictoriaMetrics从零到告警邮件实战 #Prometheus #监控
CATALOG
  1. 1. 编写recording rules——使用prometheusRule CRD
  2. 2. 使用recording rules
  3. 3. 管理recording rules——集成consul
    1. 3.1. 准备consul kv目录结构
    2. 3.2. 准备consul-template模板
    3. 3.3. 创建consul-template
    4. 3.4. 测试consul kv管理record-rule
Total : 352
2026
2025
2024
2023
自动化 web服务 k8s Docker Kubernetes DaemonSet Linux服务 前端 存储 Consul VictoriaMetrics Promxy 虚拟化 网络 Python 容器 Dify Prometheus 数据库 PVE EVE-NG EFK Elasticsearch ELK logstash filebeat elasticsearch kibana Kibana GeoIP 数据清洗 Fluentd Fluent Bit etcd 运维 安全 纳甲六爻 易经 六爻 六神 AI 占卜 GPU虚拟化 HAMi Milvus Helm LangChain DNS CoreDNS 数据备份 MoneyPrinter 容器技术 实战 监控 Rsyslog 审计日志 Blackbox Exporter Meilisearch 博客优化 iflow OpenClaw iFlow 故障处理 容器运行时 测试