模拟告警

修改告警规则值(写入值小于100）

[root@iZ2ze4v2sdd3v6w7w8doecZ prometheus]# vim rules.yml
- name: disk write latency
  rules:
  - alert: 磁盘写入延迟
    expr: rate(node_disk_write_time_seconds_total[1m]) / rate(node_disk_writes_completed_total[1m]) < 100
    for: 1m
    labels:
      name: disk
      severity: Critical
    annotations:
      summary: "unusual disk write latency (instance {{ $labels.instance }})"
      description: "磁盘写入延迟大于100毫秒"
      value: "{{ $value }}%"

重新加载

[root@iZ2ze4v2sdd3v6w7w8doecZ prometheus]# systemctl reload prometheus

坐等邮件告警，如果收不到告警重启一下alertmanager即可。

[root@iZ2ze4v2sdd3v6w7w8doecZ alertmanager]# systemctl restart alertmanager.service

报警邮件

改回之前的>100，重新加载服务。

[root@iZ2ze4v2sdd3v6w7w8doecZ prometheus]# systemctl reload prometheus

恢复之后的邮件提醒