前言Prometheus 评估告警规则后触发告警但怎么发、发给谁、怎么避免告警风暴这些是 Alertmanager 的职责。本篇详解 Alertmanager 的路由、分组、抑制、静默四大核心功能。一、Alertmanager 架构Prometheus Server ├── 评估 Alerting Rules └── 触发的告警 → 发送给 Alertmanager │ ▼ ┌───────────────┐ │ Alertmanager │ │ │ │ 1. 去重 │ ← 相同告警合并 │ 2. 分组 │ ← 同类告警打包 │ 3. 路由 │ ← 按标签分发到不同渠道 │ 4. 抑制 │ ← 高级告警抑制低级告警 │ 5. 静默 │ ← 维护期间不告警 │ 6. 重试 │ ← 发送失败自动重试 └───────┬───────┘ │ ┌─────────────┼─────────────┐ ▼ ▼ ▼ Slack/钉钉 邮件 Webhook二、安装部署二进制wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz tar xzf alertmanager-0.27.0.linux-amd64.tar.gz cd alertmanager-0.27.0.linux-amd64 sudo cp alertmanager amtool /usr/local/bin/ sudo mkdir -p /etc/alertmanager /data/alertmanagerSystemd# /etc/systemd/system/alertmanager.service [Unit] DescriptionAlertmanager Afternetwork.target [Service] Typesimple ExecStart/usr/local/bin/alertmanager \ --config.file/etc/alertmanager/alertmanager.yml \ --storage.path/data/alertmanager \ --web.listen-address0.0.0.0:9093 \ --cluster.listen-address0.0.0.0:9094 Restarton-failure [Install] WantedBymulti-user.target高可用集群# 节点 1 alertmanager \ --config.filealertmanager.yml \ --cluster.listen-address0.0.0.0:9094 \ --cluster.peeram-2:9094 \ --cluster.peeram-3:9094 # 节点 2 alertmanager \ --config.filealertmanager.yml \ --cluster.listen-address0.0.0.0:9094 \ --cluster.peeram-1:9094 \ --cluster.peeram-3:9094 # 节点 3 alertmanager \ --config.filealertmanager.yml \ --cluster.listen-address0.0.0.0:9094 \ --cluster.peeram-1:9094 \ --cluster.peeram-2:9094三、配置文件详解# alertmanager.yml global: resolve_timeout: 5m # 告警恢复后的等待时间 # SMTP 邮件配置 smtp_smarthost: smtp.example.com:587 smtp_from: alertmanagerexample.com smtp_auth_username: alertmanagerexample.com smtp_auth_password: password smtp_require_tls: true # Slack slack_api_url: https://hooks.slack.com/services/xxx # 模板 templates: - /etc/alertmanager/templates/*.tmpl # 路由树形结构 route: group_by: [alertname, cluster, severity] group_wait: 30s # 同组首次告警等待时间 group_interval: 5m # 同组后续告警间隔 repeat_interval: 4h # 重复发送间隔 receiver: default # 默认接收者 # 子路由 routes: # 严重告警 → 钉钉 电话 - matchers: - severitycritical receiver: dingtalk-critical group_wait: 10s repeat_interval: 1h # 子路由 routes: - matchers: - servicepayment receiver: payment-team # 警告告警 → 钉钉 - matchers: - severitywarning receiver: dingtalk-warning repeat_interval: 4h # 信息告警 → 邮件 - matchers: - severityinfo receiver: email repeat_interval: 24h # 接收者 receivers: - name: default webhook_configs: - url: http://dingtalk-webhook/alert send_resolved: true - name: dingtalk-critical webhook_configs: - url: http://dingtalk-webhook/critical send_resolved: true - name: dingtalk-warning webhook_configs: - url: http://dingtalk-webhook/warning send_resolved: true - name: email email_configs: - to: ops-teamexample.com send_resolved: true - name: payment-team webhook_configs: - url: http://payment-dingtalk/alert send_resolved: true # 抑制规则 inhibit_rules: # 严重告警抑制同服务警告告警 - source_matchers: - severitycritical target_matchers: - severitywarning equal: [cluster, service] # 集群宕机抑制所有该集群告警 - source_matchers: - alertnameClusterDown target_matchers: - cluster~. equal: [cluster]配置验证amtool check-config /etc/alertmanager/alertmanager.yml # Checking alertmanager.yml # SUCCESS: 5 rules found四、告警分组详解分组机制不分组 10:00:00 instancenode1 CPU 90% 10:00:05 instancenode2 CPU 90% 10:00:10 instancenode3 CPU 90% → 3 条独立通知 分组后group_by: [alertname] 10:00:00 CPU 90% (3 个实例) → 1 条通知含 3 个实例信息分组参数route: group_by: [alertname, cluster, severity] group_wait: 30s # 首次等待收到第一条告警后等 30s收齐同组告警 group_interval: 5m # 同组后续第二批告警至少间隔 5 分钟 repeat_interval: 4h # 重复发送未恢复的告警每 4 小时再发一次时间线 t0 第一条告警进入group_wait 30s t30s 发送第一批3条之后 group_interval 5m t2m 新增告警不立即发等 group_interval 到期 t5m 发送第二批新增告警 t4h 重复发送第一批仍未恢复五、路由匹配matchers 语法# 精确匹配 matchers: - severitycritical # 正则匹配 matchers: - service~payment|order # 不匹配 matchers: - severity!info # 存在 matchers: - cluster # 不存在 matchers: - !cluster路由树示例route: receiver: default routes: # 第一层按环境分发 - matchers: [envproduction] receiver: prod-default routes: # 第二层按严重程度 - matchers: [severitycritical] receiver: prod-critical - matchers: [severitywarning] receiver: prod-warning - matchers: [envstaging] receiver: staging-default # catch-all - matchers: [env~.] receiver: other-default告警匹配流程 告警进入 → 从根路由开始 → 匹配子路由 continuefalse → 向下匹配 → 匹配子路由 continuetrue → 匹配但继续向下 → 首个匹配的叶子路由 → 发送到对应 receivercontinue 字段routes: - matchers: [servicepayment] receiver: payment-team continue: true # 匹配后继续匹配后续路由 - matchers: [severitycritical] receiver: on-call # payment 的 critical 告警会同时发给两个 receiver六、抑制规则Inhibition机制抑制 当 A 告警触发时自动抑制 B 告警不发送 B source 触发抑制的告警 target 被抑制的告警 equal 相同标签才抑制实战示例inhibit_rules: # 1. 节点宕机 → 抑制该节点上的所有告警 - source_matchers: - alertnameNodeDown target_matchers: - alertname~HighCPU|HighMemory|DiskFull equal: [instance] # 2. 数据库宕机 → 抑制依赖数据库的应用告警 - source_matchers: - alertnameMySQLDown target_matchers: - alertname~AppError|SlowResponse equal: [service] # 3. 机房断网 → 抑制该机房所有告警 - source_matchers: - alertnameNetworkOutage target_matchers: - alertname~. equal: [datacenter] # 4. K8s 节点 NotReady → 抑制该节点 Pod 告警 - source_matchers: - alertnameKubernetesNodeNotReady target_matchers: - alertname~PodCrashLooping|PodNotReady equal: [node] # 5. critical 抑制 warning同级服务 - source_matchers: - severitycritical target_matchers: - severitywarning equal: [cluster, service]七、静默Silence创建静默# 命令行创建 amtool silence add \ --comment 维护窗口升级数据库 \ --duration 2h \ alertnameMySQLHighConnections \ instancedb-prod-1 # 按标签匹配静默 amtool silence add \ --comment 换机房 \ --duration 6h \ --author admin \ clusterbeijing # 查看活跃静默 amtool silence query # 取消静默 amtool silence expire silence-idAPI 创建curl -X POST http://localhost:9093/api/v2/silences \ -H Content-Type: application/json \ -d { matchers: [ {name: alertname, value: Maintenance, isRegex: false} ], startsAt: 2024-01-15T22:00:00Z, endsAt: 2024-01-16T02:00:00Z, createdBy: ops-team, comment: 计划维护窗口 }Web UI# 访问 Alertmanager Web UI http://localhost:9093 # 功能 # - Silence → 创建静默 # - Status → 查看告警状态 # - 查看/取消活跃静默八、通知模板# /etc/alertmanager/templates/dingtalk.tmpl {{ define dingtalk.default.title -}} [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} {{- end }} {{ define dingtalk.default.content -}} {{ range .Alerts }} **告警名称**: {{ .Labels.alertname }} **严重级别**: {{ .Labels.severity }} **服务**: {{ .Labels.service }} **实例**: {{ .Labels.instance }} **状态**: {{ .Status }} **开始时间**: {{ .StartsAt.Format 2006-01-02 15:04:05 }} {{ if .EndsAt }} **结束时间**: {{ .EndsAt.Format 2006-01-02 15:04:05 }} {{ end }} **详情**: {{ .Annotations.summary }} --- {{ end }} {{- end }}# receiver 中引用模板 receivers: - name: dingtalk-critical webhook_configs: - url: http://dingtalk-webhook/critical send_resolved: true # 部分 Webhook 实现支持模板引用九、钉钉/企微集成钉钉 Webhookreceivers: - name: dingtalk webhook_configs: - url: https://oapi.dingtalk.com/robot/send?access_tokenxxx send_resolved: truePrometheusAlert 转发推荐# PrometheusAlert 是一个通知转发中间件 # 支持钉钉、企微、飞书、短信、电话 docker run -d \ --name prometheusalert \ -p 8080:8080 \ -e DD_URLhttps://oapi.dingtalk.com/robot/send?access_tokenxxx \ feiyu5637/prometheusalert# Alertmanager 发给 PrometheusAlert receivers: - name: dingtalk webhook_configs: - url: http://prometheusalert:8080/prometheusalert?typeddtpldd send_resolved: true - name: phone webhook_configs: - url: http://prometheusalert:8080/prometheusalert?typephonephone13800138000要点回顾功能说明配置位置分组同类告警合并发送group_by路由按标签分发到不同渠道routes抑制高级告警抑制低级告警inhibit_rules静默维护期间临时屏蔽amtool/API模板自定义通知格式templates去重多 Prometheus 副本去重自动group_by 控制什么算同类group_wait/group_interval 控制节奏路由树从上到下匹配continuetrue 时继续向下抑制规则用 equal 字段确保只抑制相关告警静默用 amtool 或 Web UI 管理支持定时自动过期钉钉/企微推荐通过 PrometheusAlert 转发下一篇预告Alertmanager 配好了下一篇【Prometheus·告警篇】告警规则Recording Rules 与 Alerting Rules 最佳实践将讲解如何在 Prometheus 中定义告警规则。