监控系统操作手册

本文档包含完整的部署说明和所有配置文件内容,只需阅读此文档即可完成全部部署操作。

一、架构概览

本地机房                                    云端 Debian 12
┌──────────────────────────────┐       ┌──────────────────────────────────────┐
│ Grafana Alloy (每台服务器)    │       │ 现有 Nginx (TLS:443, 已有证书)        │
│                              │       │  │  /etc/nginx/conf.d/monitor.conf   │
│ · 系统指标 (unix/win)        │       │  │                                   │
│ · MySQL 指标 (Alloy内置采集) │       │  ├─► monitor-prometheus :9090        │
│ · JMX Exporter localhost抓取 │       │  │   prom/prometheus:v3.13.0         │
│ · 安全日志 (auth/event)     │       │  │                                   │
│ · 应用日志                   │       │  ├─► monitor-loki :3100              │
│                              │       │  │   grafana/loki:3.7.2              │
│         │                    │       │  │                                   │
│         │ HTTP               │       │  ├─► monitor-grafana :3000           │
│         ▼                    │       │  │   grafana/grafana:13.1.0         │
│ ┌──────────────────────┐     │       │  │                                   │
│ │ 本地 Nginx 中转       │     │       │  └─► monitor-alertmanager :9093     │
│ │ (Ubuntu 22, root     │──── HTTPS ──▶│      prom/alertmanager:v0.33.0     │
│ │  Docker, :80, 无证书) │     │       │      (告警推送企业微信)              │
│ └──────────────────────┘     │       └──────────────────────────────────────┘
└──────────────────────────────┘        (全通过 shared-proxy-network 互通)

关键设计:

  • 采集端统一 full 配置: Debian/Windows 各一个 full 配置文件, 分块设计, 默认只启用系统指标 + 安全日志, MySQL/Java/Nginx 按需取消注释启用
  • MySQL 指标: Grafana Alloy 内置 prometheus.exporter.mysql 在 MySQL 本机 localhost 采集
  • JMX Exporter: Java 服务的 JMX 指标由 JMX Exporter javaagent 暴露, Alloy 本地抓取
  • Nginx 指标: 通过独立 nginx-prometheus-exporter 采集 stub_status,Alloy 抓取 Prometheus 格式指标(Alloy 无内置 nginx exporter)
  • 本地 Nginx: Ubuntu 22 上 root Docker 部署,内网 HTTP (:80),统一代理所有 Alloy 推送到云端 HTTPS
  • 云端 Nginx: 单一入口,同时接收推送 (/api/v1/write, /loki/api/v1/push) 和对外提供 Grafana Web
  • Docker 用户: 云端 rootless (容器 0:0 → 宿主机 100000),本地 root 模式 (无 remap)

二、密码体系说明

监控系统涉及 3 个独立的密码/认证

#用途配置文件使用者
1Grafana Web 登录密码docker-compose.ymlGF_SECURITY_ADMIN_PASSWORD人通过浏览器登录 Grafana
2Prometheus admin 密码prometheus/web.ymladmin 用户Grafana 连接 Prometheus 数据源 (内部服务间)
3Prometheus prometheus 密码prometheus/web.ymlprometheus 用户各 Alloy 采集端 remote_write 推送指标
4Nginx basic auth 密码nginx/.htpasswdloki 用户仅 Loki push 入口(Loki 无内置认证,由 Nginx 提供)

注意:Prometheus 指标推送(/api/v1/write)不走 Nginx basic auth,由 Prometheus 自身的 web.yml 认证,Nginx 仅透明转发。

Grafana 浏览器登录 → GF_SECURITY_ADMIN_PASSWORD (密码1,初始 admin/xxx)
Grafana 查询指标   → Prometheus basic_auth admin:密码2
Alloy 推送指标     → Prometheus remote_write prometheus:密码3 → Nginx 透传至 Prometheus web.yml 验证
Alloy 推送日志     → Loki push loki:密码4 (同一密码) → Nginx auth_basic 验证后转发至 Loki

密码 3 和密码 4 可以设为相同值以简化管理,但认证层不同:Prometheus 是自己的 web.yml,Loki 是 Nginx auth_basic

密码 2 的特殊情况:由于 Prometheus basic auth 是”全有或全无”机制(无 localhost 豁免),Prometheus 自监控(scrape 自身 /metrics)也需要在 prometheus.yml 的 scrape job 中显式提供 admin 的明文密码。详情见 5.7 节


三、组件清单与下载

3.1 Docker 镜像 (云端 Debian 12)

组件镜像用途
Prometheusprom/prometheus:v3.13.0指标存储与告警
Lokigrafana/loki:3.7.2日志存储与告警
Grafanagrafana/grafana:13.1.0可视化仪表盘
Alertmanagerprom/alertmanager:v0.33.0告警路由

3.2 采集端组件 (各服务器)

组件适用系统下载地址
Grafana Alloy v1.17.1Debian 12 / Ubuntu 22https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-linux-amd64.zip
Grafana Alloy v1.17.1Windows Server 2016+https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-installer-windows-amd64.exe
JMX Exporter 1.6.0Java 8+ 服务https://github.com/prometheus/jmx_exporter/releases/download/v1.6.0/jmx_prometheus_javaagent-1.6.0.jar
JMX Exporter 0.17.1 (Java 6)Java 6/7 服务https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent_java6/0.17.1/jmx_prometheus_javaagent_java6-0.17.1.jar
Nginx Prometheus Exporter 1.5.1Nginx 指标采集nginx/nginx-prometheus-exporter:1.5.1 (Docker) / https://github.com/nginx/nginx-prometheus-exporter/releases

版本说明: JMX Exporter 提供两个版本——1.6.0(当前最新,Java 8+)和 0.17.1 Java 6 构建(兼容 Java 6/7,Maven Central 分发)。两者规则配置语法兼容,下方 jmx_exporter.yaml 同时适用两个版本。1.1.0 起 jar 仅通过 GitHub Releases 分发。Grafana Alloy 最低要求 Windows Server 2016。

Nginx 指标采集说明: Grafana Alloy 没有内置 prometheus.exporter.nginx 组件(与 Apache 的 prometheus.exporter.apache 不同),Nginx 指标需通过独立的 nginx-prometheus-exporter 采集,再由 Alloy 的 prometheus.scrape 抓取。


四、Docker 镜像导出 (离线部署)

在可联网机器上执行:

# 拉取
docker pull prom/prometheus:v3.13.0
docker pull grafana/loki:3.7.2
docker pull grafana/grafana:13.1.0
docker pull prom/alertmanager:v0.33.0
docker pull nginx:stable-alpine
docker pull nginx/nginx-prometheus-exporter:1.5.1
 
# 一键导出
docker save -o monitor-images.tar \
    prom/prometheus:v3.13.0 \
    grafana/loki:3.7.2 \
    grafana/grafana:13.1.0 \
    prom/alertmanager:v0.33.0 \
    nginx:stable-alpine \
    nginx/nginx-prometheus-exporter:1.5.1

生产服务器导入:

docker load -i monitor-images.tar
docker images | grep -E "prom/prometheus|grafana/(loki|grafana)|prom/alertmanager|nginx"

4.1 采集端离线下载

组件URL
Grafana Alloy v1.17.1 Linux amd64https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-linux-amd64.zip
Grafana Alloy v1.17.1 Windows 安装器https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-installer-windows-amd64.exe
JMX Exporter 1.6.0https://github.com/prometheus/jmx_exporter/releases/download/v1.6.0/jmx_prometheus_javaagent-1.6.0.jar
Nginx Prometheus Exporter 1.5.1https://github.com/nginx/nginx-prometheus-exporter/releases/tag/v1.5.1

五、云端服务器部署 (Debian 12)

5.1 目录结构

在云端服务器上创建以下目录结构:

mkdir -p /opt/monitor/{prometheus/{data,rules},loki/{data,rules},grafana/{data,provisioning/{datasources,dashboards}},alertmanager/data,nginx/verify}

5.2 网络说明

monitor 服务通过 shared-proxy-network 与现有 Nginx 互通。Nginx 在 location 中直接用容器名代理(proxy_pass http://monitor-prometheus:9090),不是 127.0.0.1

云端 Nginx 同时处理:

  1. 接收 Alloy 推送/api/v1/write → Prometheus, /loki/api/v1/push → Loki
  2. 对外 Web 访问/ → Grafana UI

5.3 Rootless Docker 权限设置

UID 映射: 容器 0 → 宿主机 100000
# 确认映射关系
grep ^$(whoami) /etc/subuid
# 示例: dockeruser:100000:65536
 
# 创建数据目录并设置权限
chown -R 100000:100000 /opt/monitor/prometheus/data
chown -R 100000:100000 /opt/monitor/loki/data
chown -R 100000:100000 /opt/monitor/grafana/data
chown -R 100000:100000 /opt/monitor/alertmanager/data

5.4 生成认证密码

apt update && apt install apache2-utils -y
 
# 生成 admin 用户的 bcrypt 哈希 (Grafana 连接 Prometheus 用)
htpasswd -nbB admin '<ADMIN_PASSWORD>' | cut -d: -f2
 
# 生成 prometheus 用户的 bcrypt 哈希 (Alloy remote_write 用)
htpasswd -nbB prometheus '<PROMETHEUS_PASSWORD>' | cut -d: -f2

5.5 预先替换占位符

在创建以下配置文件之前,先确定并记录所有需要替换的值:

.env 文件变量docker-compose.yml 自动读取):

变量说明
DOMAIN实际域名,如 monitor.yourdomain.com
GRAFANA_PASSWORDGrafana Web 登录密码

配置文件占位符

占位符说明出现位置
<ADMIN_PASSWORD>admin 用户明文密码grafana datasources
<PROMETHEUS_PASSWORD>prometheus 用户明文密码Alloy 配置
<BCRYPT_HASH_ADMIN>admin 的 bcrypt 哈希prometheus/web.yml, alertmanager/web.yml
<BCRYPT_HASH_PROMETHEUS>prometheus 的 bcrypt 哈希prometheus/web.yml
<YOUR_CORP_ID>企业微信企业 IDalertmanager.yml
<YOUR_API_SECRET>企业微信应用 Secretalertmanager.yml
<YOUR_AGENT_ID>企业微信应用 Agent IDalertmanager.yml

5.5.1 .env (环境变量)

文件路径: /opt/monitor/.env

docker-compose.yml 自动读取此文件,部署前替换为实际值:

DOMAIN=monitor.yourdomain.com
GRAFANA_PASSWORD=<你的 Grafana 登录密>

5.6 docker-compose.yml

文件路径: /opt/monitor/docker-compose.yml

services:
  prometheus:
    image: prom/prometheus:v3.13.0
    container_name: monitor-prometheus
    restart: unless-stopped
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    deploy:
      resources:
        limits:
          cpus: "0.5"
          memory: 512M
          pids: 100
        reservations:
          memory: 256M
    user: "0:0"
    environment:
      - TZ=Asia/Shanghai
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/web.yml:/etc/prometheus/web.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - ./prometheus/data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=15d"
      - "--web.enable-remote-write-receiver"
      - "--web.config.file=/etc/prometheus/web.yml"
      - "--web.enable-admin-api"
      - "--web.external-url=https://${DOMAIN}/prometheus/"
      - "--web.route-prefix=/prometheus/"
    networks:
      - monitor
      - proxy
 
  loki:
    image: grafana/loki:3.7.2
    container_name: monitor-loki
    restart: unless-stopped
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    deploy:
      resources:
        limits:
          cpus: "0.3"
          memory: 256M
          pids: 100
        reservations:
          memory: 128M
    user: "0:0"
    volumes:
      - ./loki/local-config.yaml:/etc/loki/local-config.yaml:ro
      - ./loki/rules:/etc/loki/rules:ro
      - ./loki/data:/loki
    command:
      - "-config.file=/etc/loki/local-config.yaml"
      - "-log.level=warn"
    networks:
      - monitor
      - proxy
 
  grafana:
    image: grafana/grafana:13.1.0
    container_name: monitor-grafana
    restart: unless-stopped
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
      - GF_SERVER_ROOT_URL=https://${DOMAIN}
      - GF_USERS_ALLOW_SIGN_UP=false
      - GF_AUTH_ANONYMOUS_ENABLED=false
      - TZ=Asia/Shanghai
    deploy:
      resources:
        limits:
          cpus: "0.2"
          memory: 256M
          pids: 100
        reservations:
          memory: 128M
    user: "0:0"
    volumes:
      - ./grafana/data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    networks:
      - monitor
      - proxy
 
  alertmanager:
    image: prom/alertmanager:v0.33.0
    container_name: monitor-alertmanager
    restart: unless-stopped
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    deploy:
      resources:
        limits:
          cpus: "0.1"
          memory: 64M
          pids: 50
        reservations:
          memory: 32M
    user: "0:0"
    environment:
      - TZ=Asia/Shanghai
    volumes:
      - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - ./alertmanager/web.yml:/etc/alertmanager/web.yml:ro
      - ./alertmanager/data:/alertmanager
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"
      - "--storage.path=/alertmanager"
      - "--web.config.file=/etc/alertmanager/web.yml"
      - "--web.external-url=https://${DOMAIN}/alertmanager/"
      - "--log.level=debug"
    networks:
      - monitor
      - proxy
 
networks:
  monitor:
    driver: bridge
  proxy:
    name: shared-proxy-network
    external: true

5.7 prometheus/prometheus.yml

文件路径: /opt/monitor/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s
 
alerting:
  alertmanagers:
    - path_prefix: /alertmanager/
      basic_auth:
        username: admin
        password: <YOUR_ADMIN_PASSWORD>
      static_configs:
        - targets:
            - alertmanager:9093
 
rule_files:
  - "/etc/prometheus/rules/*.yml"
 
scrape_configs:
  # =========================================================================
  # Prometheus 自监控
  #
  # 由于同时启用了:
  #   1. --web.route-prefix=/prometheus/  → /metrics 被重写为 /prometheus/metrics
  #   2. --web.config.file=web.yml        → basic auth 全有或全无,无 localhost 豁免
  # 自抓取必须显式指定 metrics_path 和 basic_auth。
  #
  # basic_auth 中的 password 必须是明文,不能用 web.yml 中的 bcrypt hash。
  # 官方讨论见:https://groups.google.com/g/prometheus-users/c/3Rn-2bAbusI
  # =========================================================================
  - job_name: "prometheus"
    metrics_path: "/prometheus/metrics"
    basic_auth:
      username: admin
      password: <ADMIN_PASSWORD>  # web.yml 中 admin 对应的明文密码,不是 bcrypt hash
    static_configs:
      - targets: ["localhost:9090"]
        labels:
          instance: "monitor-prometheus"

5.8 prometheus/web.yml

文件路径: /opt/monitor/prometheus/web.yml

# ==========================================
# Prometheus HTTP basic_auth 用户
#
# admin:      Grafana 连接 Prometheus 数据源时使用 (内部服务间认证)
# prometheus: Alloy 采集端 remote_write 推送时使用 (跨网络认证)
# ==========================================
basic_auth_users:
  admin: <BCRYPT_HASH_ADMIN>
  prometheus: <BCRYPT_HASH_PROMETHEUS>

5.9 prometheus/rules/alerts.yml

文件路径: /opt/monitor/prometheus/rules/alerts.yml

groups:
  - name: system
    rules:
      - alert: HighCPUUsage
        expr: |
          100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU 使用率超过 80%"
          description: '当前值 {{ printf "%.1f" $value }}%'
 
      - alert: HighCPUUsageCritical
        expr: |
          100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "CPU 使用率超过 95%"
          description: '当前值 {{ printf "%.1f" $value }}%'
      - alert: HighMemoryUsage
        expr: |
          (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "内存使用率超过 90%"
          description: '当前值 {{ printf "%.1f" $value }}%'
 
      - alert: DiskFull
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "磁盘剩余空间不足 10%"
          description: '剩余空间 {{ printf "%.1f" $value }}%'
 
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "实例已下线"
          description: "服务不可达,请排查"
 
      - alert: HighDiskIO
        expr: |
          rate(node_disk_io_time_seconds_total[5m]) * 100 > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘 IO 使用率过高"
          description: '当前值 {{ printf "%.1f" $value }}%'
 
  - name: windows_system
    rules:
      # https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.cpu.md
      - alert: WindowsHighCPUUsage
        expr: |
          100 - (avg by(instance) (irate(windows_cpu_time_total{mode="idle"}[2m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU 使用率超过 80%"
          description: '当前值 {{ printf "%.1f" $value }}%'
 
      - alert: WindowsHighCPUUsageCritical
        expr: |
          100 - (avg by(instance) (irate(windows_cpu_time_total{mode="idle"}[2m])) * 100) > 95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "CPU 使用率超过 95%"
          description: '当前值 {{ printf "%.1f" $value }}%'
 
      # https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.memory.md
      - alert: WindowsHighMemoryUsage
        expr: |
          100 - 100 * (windows_memory_physical_free_bytes / windows_memory_physical_total_bytes) > 96
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "内存使用率超过 96%"
          description: '当前值 {{ printf "%.1f" $value }}%'
 
      # https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.logical_disk.md
      - alert: WindowsDiskFull
        expr: |
          100 - 100 * (windows_logical_disk_free_bytes / windows_logical_disk_size_bytes) > 90
          and windows_logical_disk_size_bytes > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "磁盘剩余空间不足 10% ({{ $labels.volume }})"
          description: '当前使用率 {{ printf "%.1f" $value }}%'
 
      # 官方 Disk Activity 公式
      - alert: WindowsHighDiskIO
        expr: |
          (
            rate(windows_logical_disk_read_seconds_total[2m])
            + rate(windows_logical_disk_write_seconds_total[2m])
          )
          /
          (
            rate(windows_logical_disk_read_seconds_total[2m])
            + rate(windows_logical_disk_write_seconds_total[2m])
            + rate(windows_logical_disk_idle_seconds_total[2m])
          ) > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "磁盘 IO 使用率超过 80% ({{ $labels.volume }})"
          description: '当前值 {{ printf "%.1f" $value }}'

5.10 loki/local-config.yaml

文件路径: /opt/monitor/loki/local-config.yaml

auth_enabled: false
 
server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  http_server_read_timeout: 60s
  http_server_write_timeout: 60s
 
common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules-temp
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
 
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
 
storage_config:
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache
  filesystem:
    directory: /loki/chunks
 
compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: filesystem
 
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: 168h
  max_entries_limit_per_query: 10000
  split_queries_by_interval: 1h
  ingestion_rate_mb: 8
  ingestion_burst_size_mb: 16
  per_stream_rate_limit: 32M
  per_stream_rate_limit_burst: 64M
  max_cache_freshness_per_query: "10m"
 
query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 50
 
ruler:
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  rule_path: /loki/rules-temp
  alertmanager_url: http://alertmanager:9093/alertmanager
  alertmanager_client:
    basic_auth_username: admin
    basic_auth_password: <YOUR_ADMIN_PASSWORD>
  ring:
    kvstore:
      store: inmemory
  enable_api: true
 
analytics:
  reporting_enabled: false

5.11 loki/rules/fake/security.yml

文件路径: /opt/monitor/loki/rules/fake/security.yml

注意: Loki 单租户模式(auth_enabled: false)下,默认租户 ID 为 fake。规则文件必须放在 <ruler_storage_directory>/fake/ 子目录下,否则规则不会被加载。

groups:
  - name: security
    rules:
      - alert: SSHBruteForce
        expr: |
          sum(rate({job="auth"} |~ "Failed password" [5m])) by (instance) > 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "SSH 暴力破解检测"
          description: "过去 5 分钟内 SSH 登录失败超过 10 次"
 
      - alert: SSHRootLogin
        expr: |
          sum(rate({job="auth"} |~ "Accepted .* for root" [5m])) by (instance) > 0
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "检测到 root 直接 SSH 登录"
          description: "root 用户通过 SSH 直接登录系统,建议使用普通用户并 sudo"
 
      - alert: SSHLoginSuccess
        expr: |
          sum by (host, group, job, login_user, source_ip, geoip_country_name, geoip_city_name) (
            count_over_time({job="auth", login_user!=""} [1m])
          ) > 0
        for: 0m
        labels:
          severity: login
        annotations:
          summary: "SSH 登录成功"
          description: "用户 {{ $labels.login_user }} 从 {{ $labels.source_ip }} ({{ $labels.geoip_country_name }} {{ $labels.geoip_city_name }}) 登录"
 
      - alert: WindowsLoginFailure
        expr: |
          sum(rate({job="windows_security", event_id="4625"} [5m])) by (host) > 20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Windows 登录失败过多"
          description: "过去 5 分钟内 EventID 4625 超过 20 次"
 
      - alert: WindowsLoginSuccess
        expr: |
          sum by (host, group, job, login_user, source_ip, logon_type, logon_type_name, geoip_country_name, geoip_city_name) (
            count_over_time({job="windows_security", event_id="4624", login_user!=""} [1m])
          ) > 0
        for: 0m
        labels:
          severity: login
        annotations:
          summary: "Windows 登录成功"
          description: "用户 {{ $labels.login_user }} 从 {{ $labels.source_ip }} 登录 ({{ $labels.geoip_country_name }} {{ $labels.geoip_city_name }})"
 
      - alert: WindowsLoginSuccessAnomaly
        expr: |
          sum(rate({job="windows_security", event_id="4624"} [5m])) by (host) > 50
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Windows 登录次数异常增多"
          description: "过去 5 分钟内 EventID 4624 超过 50 次"

5.12 alertmanager/alertmanager.yml

文件路径: /opt/monitor/alertmanager/alertmanager.yml

# ==========================================
# Alertmanager 配置
# 告警通过企业微信应用消息推送
#
# 前置条件 (在企业微信管理后台获取):
#   corp_id:    企业 ID  → 管理后台 → "我的企业" → "企业信息"
#   agent_id:   应用 AgentID → 管理后台 → "应用管理" → "应用" → 点进自建应用
#   api_secret: 应用 Secret  → 同上页面, Secret 字段
#   to_user:    接收人账号  → 管理后台 → "通讯录" → 点成员 → "账号"字段, 多个用 | 分隔
#
# 官方文档:
#   基本概念:       https://developer.work.weixin.qq.com/document/path/90665
#   创建自建应用:    https://developer.work.weixin.qq.com/document/path/90487
#   管理后台:        https://work.weixin.qq.com/
# ==========================================
 
global:
  resolve_timeout: 2m
  wechat_api_url: "https://qyapi.weixin.qq.com/cgi-bin/"
  wechat_api_corp_id: "<YOUR_CORP_ID>"
  wechat_api_secret: "<YOUR_API_SECRET>"
 
route:
  receiver: "wechat-warning"
  group_by: ["alertname", "severity", "instance", "job", "group"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
 
  routes:
    - matchers:
        - severity="critical"
      receiver: "wechat-critical"
      group_wait: 10s
    - matchers:
        - severity="warning"
      receiver: "wechat-warning"
    - matchers:
        - severity="login"
      receiver: "wechat-login"
      group_by: ["alertname", "group", "host", "job", "login_user", "source_ip"]
      group_wait: 5s
      group_interval: 1s
      repeat_interval: 5m
    - matchers:
        - severity="info"
      receiver: "wechat-info"
 
receivers:
  - name: "wechat-critical"
    wechat_configs:
      - send_resolved: true
        agent_id: "<YOUR_AGENT_ID>"
        to_user: "<YOUR_USER_ID>"
        message_type: "markdown"
        message: |
          {{ range .Alerts }}
          {{ if eq .Status "resolved" }}
          ## <font color="info">[RESOLVED] {{ .Labels.alertname }} 已恢复</font>
          {{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
          {{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
          {{ end }}> 分组: {{ .Labels.group }}
          > 主机: {{ .Labels.host }}
          > 任务: {{ .Labels.job }}
          > 时间: {{ .EndsAt.Local.Format "2006-01-02 15:04:05" }}
          {{ else }}
          ## <font color="warning">[CRITICAL] {{ .Annotations.summary }}</font>
          {{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
          {{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
          {{ end }}> 分组: {{ .Labels.group }}
          > 主机: {{ .Labels.host }}
          > 任务: {{ .Labels.job }}
          > 描述: {{ .Annotations.description }}
          > 时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
          {{ end }}
          {{ end }}
 
  - name: "wechat-warning"
    wechat_configs:
      - send_resolved: true
        agent_id: "<YOUR_AGENT_ID>"
        to_user: "<YOUR_USER_ID>"
        message_type: "markdown"
        message: |
          {{ range .Alerts }}
          {{ if eq .Status "resolved" }}
          ## <font color="info">[RESOLVED] {{ .Labels.alertname }} 已恢复</font>
          {{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
          {{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
          {{ end }}> 分组: {{ .Labels.group }}
          > 主机: {{ .Labels.host }}
          > 任务: {{ .Labels.job }}
          > 时间: {{ .EndsAt.Local.Format "2006-01-02 15:04:05" }}
          {{ else }}
          ## <font color="warning">[WARNING] {{ .Annotations.summary }}</font>
          {{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
          {{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
          {{ end }}> 分组: {{ .Labels.group }}
          > 主机: {{ .Labels.host }}
          > 任务: {{ .Labels.job }}
          > 描述: {{ .Annotations.description }}
          > 时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
          {{ end }}
          {{ end }}
 
  - name: "wechat-info"
    wechat_configs:
      - send_resolved: true
        agent_id: "<YOUR_AGENT_ID>"
        to_user: "<YOUR_USER_ID>"
        message_type: "markdown"
        message: |
          {{ range .Alerts }}
          {{ if eq .Status "resolved" }}
          ## <font color="info">[RESOLVED] {{ .Labels.alertname }} 已恢复</font>
          {{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
          {{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
          {{ end }}> 分组: {{ .Labels.group }}
          > 主机: {{ .Labels.host }}
          > 任务: {{ .Labels.job }}
          > 时间: {{ .EndsAt.Local.Format "2006-01-02 15:04:05" }}
          {{ else }}
          ## <font color="info">[INFO] {{ .Annotations.summary }}</font>
          {{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
          {{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
          {{ end }}> 分组: {{ .Labels.group }}
          > 主机: {{ .Labels.host }}
          > 任务: {{ .Labels.job }}
          > 描述: {{ .Annotations.description }}
          > 时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
          {{ end }}
          {{ end }}
 
  - name: "wechat-login"
    wechat_configs:
      - send_resolved: false
        agent_id: "<YOUR_AGENT_ID>"
        to_user: "<YOUR_USER_ID>"
        message_type: "markdown"
        message: |
          {{ range .Alerts }}
          ## <font color="warning">[LOGIN] {{ .Annotations.summary }}</font>
          > 分组: {{ .Labels.group }}
          > 主机: {{ .Labels.host }}
          > 任务: {{ .Labels.job }}
          > 账号: {{ .Labels.login_user }}
          > 来源IP: {{ .Labels.source_ip }}
          {{ if .Labels.geoip_country_name }}> 位置: {{ .Labels.geoip_country_name }} {{ .Labels.geoip_city_name }}
          {{ end }}{{ if .Labels.logon_type }}> 登录类型: {{ .Labels.logon_type_name }} (ID:{{ .Labels.logon_type }})
          {{ end }}> 时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
          {{ end }}

5.12.1 alertmanager/web.yml

文件路径: /opt/monitor/alertmanager/web.yml

Alertmanager 的 HTTP basic_auth 配置,与 Prometheus web.yml 共享同一套密码体系(bcrypt 哈希)。

basic_auth_users:
  admin: <BCRYPT_HASH>

访问 Alertmanager Web UI: 通过 https://yourdomain.com/alertmanager/ 访问,认证由 Alertmanager 自身的 basic auth 处理(admin 用户)。


5.13 grafana/provisioning/datasources/datasources.yml

文件路径: /opt/monitor/grafana/provisioning/datasources/datasources.yml

apiVersion: 1
 
# ==========================================
# 数据源配置
#
# Prometheus 中 basicAuthUser: admin 对应 prometheus/web.yml 中的 admin 用户
# basicAuthPassword 是 admin 的明文密码 (非 bcrypt 哈希)
#
# Grafana 自身的 Web 登录密码由 docker-compose.yml 中 GF_SECURITY_ADMIN_PASSWORD 设置
# ==========================================
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    basicAuth: true
    basicAuthUser: admin
    secureJsonData:
      basicAuthPassword: "<ADMIN_PASSWORD>"
    isDefault: true
    editable: false
 
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    editable: false

5.14 grafana/provisioning/dashboards/dashboards.yml

文件路径: /opt/monitor/grafana/provisioning/dashboards/dashboards.yml

apiVersion: 1
 
providers:
  - name: "default"
    orgId: 1
    folder: ""
    type: file
    disableDeletion: false
    updateIntervalSeconds: 60
    options:
      # 将 dashboard JSON 文件放入此目录后自动加载
      # 推荐导入: Node Exporter Full (ID: 1860), MySQL Overview (ID: 7362)
      # 下载 JSON: https://grafana.com/grafana/dashboards/<ID>
      path: /etc/grafana/provisioning/dashboards

5.15 nginx/nginx.conf (云端 Nginx 反代)

文件路径: /opt/monitor/nginx/nginx.conf

此文件需 include 到现有云端 Nginx 的配置中。该文件包含两层入口设计:

  • 443 (TLS) — 公网入口,server_name 为公网域名,供外部 Alloy / 浏览器访问
  • 80 (HTTP) — 内网入口,server_namemonitor-proxy.local,供云端同一 VPC 内的 Alloy 使用,避免流量出公网回环

TLS 证书由现有 Nginx 全局配置提供,内网入口无需证书。

# ==========================================
# 监控 Nginx 配置 (include 到现有云端 Nginx)
# 路径: /etc/nginx/conf.d/monitor.conf
# 前提: 现有 Nginx 已处理 TLS 证书,此文件只负责路由
# 前提: Nginx 容器需加入 shared-proxy-network 才能解析容器名
#
# 两层入口设计:
#   443 (TLS) — 公网入口,server_name 为公网域名
#   80        — 内网入口,server_name 为 monitor-proxy.local
#               供云端同一 VPC 内的 Alloy 使用,避免流量出公网回环
# ==========================================
 
# 使用 Docker 内置 DNS 进行运行时解析,避免启动时因上游容器未就绪而失败
resolver 127.0.0.11 valid=300s ipv6=off;
 
# ==========================================
# 公网入口 (TLS) — 外部 Alloy / 浏览器访问
# ==========================================
server {
    listen 443 ssl;
    server_name monitor.yourdomain.com;
 
    # 全局代理配置,所有 location 自动继承
    client_body_buffer_size 10M;
    proxy_http_version 1.1;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
 
    # Prometheus remote_write 入口 (Alloy 推送指标)
    # Prometheus Web UI (调试用,需配合 --web.external-url 和 --web.route-prefix)
    set $prometheus_upstream "monitor-prometheus:9090";
 
    # 注意:Prometheus 设置了 --web.route-prefix=/prometheus/,
    # 所以 /api/v1/write 的实际内部路由为 /prometheus/api/v1/write,
    # proxy_pass 尾部带路径来补上 /prometheus/ 前缀
    location /api/v1/write {
        proxy_pass http://$prometheus_upstream/prometheus/api/v1/write;
        client_max_body_size 10M;
    }
 
    location /prometheus/ {
        proxy_pass http://$prometheus_upstream;
    }
 
    # Loki 日志 push 入口 (Alloy 推送日志)
    # Loki 自身不支持内置 basic auth,由 Nginx auth_basic 提供认证
    # .htpasswd 需包含 loki 用户 (可与 prometheus 用户共用同一个密码)
    set $loki_upstream "monitor-loki:3100";
 
    location /loki/api/v1/push {
        auth_basic "Loki";
        auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
        proxy_pass http://$loki_upstream;
        client_max_body_size 20M;
    }
 
    # Loki API (调试用)
    # 注意:Loki 自身不支持内置 basic auth,需要通过 Nginx auth_basic 提供认证
    # 注意:proxy_pass 使用变量时不做 URI 替换,需用 rewrite 去掉 /loki/ 前缀
    location /loki/ {
        auth_basic "Loki";
        auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
        rewrite ^/loki/(.*) /$1 break;
        proxy_pass http://$loki_upstream;
    }
 
    # Alertmanager Web UI (查看活跃告警)
    set $alertmanager_upstream "monitor-alertmanager:9093";
 
    location /alertmanager/ {
        proxy_pass http://$alertmanager_upstream;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
 
    # 企业微信可信域名验证文件 (WW_verify_*.txt)
    # 将验证文件放到 verify/ 目录并挂载到容器
    location /WW_verify_ {
        root /etc/nginx/conf.d/verify;
    }
 
    # Grafana Web UI
    set $grafana_upstream "monitor-grafana:3000";
 
    location / {
        proxy_pass http://$grafana_upstream;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
 
# ==========================================
# 内网入口 (HTTP) — 供云端 VPC 内 Alloy 推送使用
# 通过 monitor-proxy.local 访问,走内网不出公网
# DNS 解析方式:
#   - Linux: /etc/hosts 添加 "<内网IP> monitor-proxy.local"
#   - Windows: C:\Windows\System32\drivers\etc\hosts 相同操作
#   - 云服务商内网 DNS(如阿里云 PrivateZone 等,可能收费)
# ==========================================
server {
    listen 80;
    server_name monitor-proxy.local;
 
    # 只允许内网 IP 访问
    allow 127.0.0.1;
    allow 10.0.0.0/8;
    allow 172.16.0.0/12;
    allow 192.168.0.0/16;
    deny all;
 
    client_body_buffer_size 10M;
    proxy_http_version 1.1;
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
    set $prometheus_upstream "monitor-prometheus:9090";
    set $loki_upstream "monitor-loki:3100";
 
    # Prometheus remote_write 入口 (Alloy 推送指标)
    location /api/v1/write {
        proxy_pass http://$prometheus_upstream/prometheus/api/v1/write;
        client_max_body_size 10M;
    }
 
    # Loki 日志 push 入口 (Alloy 推送日志)
    # 内网环境可按需启用 basic_auth:
    #   auth_basic "Loki";
    #   auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
    location /loki/api/v1/push {
        proxy_pass http://$loki_upstream;
        client_max_body_size 20M;
    }
 
    # Nginx 自身指标 (按需取消注释)
    # 前提: Nginx 已开启 stub_status 模块
    # location /nginx_status {
    #     stub_status on;
    #     access_log off;
    # }
}

挂载到现有 Nginx:

# 方法1: 在现有 Nginx 的 docker-compose 中挂载 (推荐)
# volumes:
#   - /opt/monitor/nginx/nginx.conf:/etc/nginx/conf.d/monitor.conf:ro
#   - /opt/monitor/nginx/verify:/etc/nginx/conf.d/verify:ro
 
# 方法2: 在现有 Nginx http{} 块中 include
# include /etc/nginx/conf.d/monitor.conf;

注意: nginx.conf 中定义了 server { listen 443 ssl; ... } 块。如果现有 Nginx 已有监听 443 的 server,需要将 location 块合并到现有 server 中,而非整体 include。内网 server 块(listen 80)可直接 include,互不冲突。

Nginx 容器需加入 shared-proxy-network 才能解析 monitor-prometheus 等容器名。resolver 127.0.0.11 使用 Docker 内置 DNS 做运行时解析,配合 set + 变量 proxy_pass 避免 nginx 启动时因上游容器未就绪而报 host not found in upstream

访问 Prometheus Web UI: 部署后通过 https://yourdomain.com/prometheus/ 访问。认证由 Prometheus 自身的 web.yml basic auth 处理(admin 用户),无需在 Nginx 额外配置。Prometheus 需要知道自己在子路径下,已在 docker-compose.yml 中配置 --web.external-url--web.route-prefix

访问 Loki API (调试用): 通过 https://yourdomain.com/loki/ 访问。Loki 自身不支持内置 basic auth,由 Nginx auth_basic 提供认证。需要创建 .htpasswd 文件:

# 生成 Loki 推送专用用户 (Alloy 推送日志认证)
htpasswd -c /opt/monitor/nginx/.htpasswd loki
 
# 如需其他用户 (如 Loki Web UI 访问):
htpasswd /opt/monitor/nginx/.htpasswd admin
 
# 在现有 Nginx 的 docker-compose 中挂载:
#   - /opt/monitor/nginx/.htpasswd:/etc/nginx/conf.d/.htpasswd:ro
 
nginx -s reload

注意: .htpasswd 中的 loki 用户用于 Alloy 推送日志(Nginx auth_basic 验证),与 Prometheus web.ymlprometheus 用户是两个独立的认证层。密码可以相同。

内网入口说明: 新增的 server { listen 80; server_name monitor-proxy.local; } 块专供云端同一 VPC 内 Alloy 使用,配置思路参考第 6.1 节 nginx-local.conf(同样监听 80 端口 + monitor-proxy.local)。区别在于 6.1 的本地 Nginx 将请求转发到公网云端,而此处的云端 Nginx 内网入口直接转发到本地的 Docker 后端服务。VPC 内 Alloy 的 endpoint 应设为 http://monitor-proxy.local/api/v1/writehttp://monitor-proxy.local/loki/api/v1/pushmonitor-proxy.local 通过 /etc/hosts 或内网 DNS 解析到云端 Nginx 的内网 IP。

5.16 启动 monitor 服务

cd /opt/monitor
docker load -i monitor-images.tar
docker compose up -d
docker compose ps

六、本地 Nginx 中转 (Ubuntu 22, root Docker)

6.1 nginx/nginx-local.conf

文件路径: /opt/monitor/nginx/nginx-local.conf

# ==========================================
# 本地 Nginx 中转配置 (数据中心侧)
# 路径: /etc/nginx/sites-available/monitor-proxy
# 作用: 汇总本地所有 Alloy 的推送,统一转发到云端
# 证书: 不需要,纯内网 HTTP → 云端 HTTPS
# ==========================================
 
server {
    listen 80;
    server_name monitor-proxy.local;
 
    # 只允许内网 IP 访问
    allow 127.0.0.1;
    allow 10.0.0.0/8;
    allow 172.16.0.0/12;
    allow 192.168.0.0/16;
    deny all;
 
    client_max_body_size 20M;
    client_body_buffer_size 10M;
    proxy_http_version 1.1;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_ssl_server_name on;
    proxy_ssl_protocols TLSv1.2 TLSv1.3;
 
    # 转发 Alloy remote_write 到云端 Prometheus
    location /api/v1/write {
        proxy_pass https://monitor.yourdomain.com;
    }
 
    # 转发 Alloy log push 到云端 Loki
    location /loki/api/v1/push {
        proxy_pass https://monitor.yourdomain.com;
    }
 
    # Nginx 自身指标 (按需取消注释)
    # 前提: Nginx 已开启 stub_status 模块
    # location /nginx_status {
    #     stub_status on;
    #     access_log off;
    # }
}

6.2 nginx/docker-compose-nginx-local.yml

文件路径: /opt/monitor/nginx/docker-compose-nginx-local.yml

# ==========================================
# 本地 Nginx 中转代理 (Ubuntu 22, root Docker)
# 用途: 汇总本地所有 Alloy 推送,统一转发到云端
# 部署: docker compose -f docker-compose-nginx-local.yml up -d
# ==========================================
services:
  nginx-proxy:
    image: nginx:stable-alpine
    container_name: monitor-proxy-nginx
    restart: unless-stopped
    ports:
      - "80:80"
    volumes:
      - ./nginx-local.conf:/etc/nginx/conf.d/default.conf:ro
      - ./nginx-logs:/var/log/nginx
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
      - CHOWN
      - SETGID
      - SETUID
    security_opt:
      - no-new-privileges:true
 
# ==========================================
# 权限说明:
# Root Docker 模式,无 userns-remap
# nginx:stable-alpine 容器内以 nginx 用户 (UID 101) 运行
# 日志目录需可写:
#   mkdir -p nginx-logs && chown 101:101 nginx-logs
# 配置文件只需要可读 (默认 644 即可)
# ==========================================

6.3 部署

# 在 monitor/nginx/ 目录
mkdir -p nginx-logs
chown 101:101 nginx-logs    # nginx:stable-alpine 容器内 UID 101
 
# 修改 nginx-local.conf 中的云端域名
#   monitor.yourdomain.com → 实际云端域名
 
docker compose -f docker-compose-nginx-local.yml up -d

6.4 是否需要证书?

不需要。本地 Nginx 只监听内网 HTTP (:80),Alloy 通过 HTTP 推送。本地 Nginx 作为 HTTPS 客户端代理到云端(云端已有 TLS 证书)。

6.5 Alloy 端点选择

所有 Alloy 配置中两个端点 URL 需填写,loki.writebasic_auth 中 username 为 loki(对应云端 Nginx .htpasswd 中的 loki 用户):

场景端点
有本地 Nginx(数据中心)http://<本地Nginx_IP>/api/v1/write/loki/api/v1/push
云端 VPC 内(同 VPC 不同机器)http://monitor-proxy.local/api/v1/write/loki/api/v1/push
直连云端(公网)https://monitor.yourdomain.com/api/v1/write/loki/api/v1/push

6.6 日志轮转

Nginx 自身不内置 log rotate,需配合系统工具做日志轮转。以下提供 Linux 和 Windows 两种方案。

Linux(logrotate)

适用于本地 Nginx 中转(mount nginx-logs:/var/log/nginx)和云端 Nginx:

# /etc/logrotate.d/monitor-nginx
/opt/monitor/nginx/nginx-logs/*.log {
    daily
    rotate 7
    maxsize 20M
    missingok
    notifempty
    compress
    delaycompress
    sharedscripts
    postrotate
        docker exec monitor-proxy-nginx nginx -s reopen 2>/dev/null || true
    endscript
}

云端 Nginx 若不在此 compose,将 docker exec 替换为对应的容器名。 daily 保证每天至少 rotate 一次(即使未达到 maxsize,避免 error.log 长期不转)。

验证:

logrotate -d /etc/logrotate.d/monitor-nginx   # dry-run
logrotate -f /etc/logrotate.d/monitor-nginx   # 手动执行一次

Windows(bat + 任务计划)

适用于 Windows 上部署的 Nginx。注意:Windows nginx 不支持 Unix 信号,但 nginx -s reopen 命令行参数可正常工作,关闭旧 access.log 的 handle 并创建新文件。

@echo off
set nginx_dir=<nginx_dir>
set log_dir=%nginx_dir%\logs
set current_date=%date:~0,4date:~8,2%
 
:: rename 日志 → nginx 仍往旧 handle 写
move %log_dir%\access.log %log_dir%\access_%current_date%.log
move %log_dir%\error.log %log_dir%\error_%current_date%.log
:: reopen → nginx 创建新 access.log / error.log,切换 handle
call %nginx_dir%\nginx -p %nginx_dir%\ -s reopen
:: 清理 90 天前的归档
forfiles /p %log_dir% /m access_*.log -d -90 /c "cmd /c del /f @path" 2>nul
forfiles /p %log_dir% /m error_*.log -d -90 /c "cmd /c del /f @path" 2>nul
@echo on

注意%date% 格式依赖系统区域设置(中文 Windows 正常,英文系统需调整截取位置)。

创建计划任务(每天 23:59:56 执行):

schtasks /create /tn "NginxLogRotate" /tr "C:\scripts\rotate-nginx-log.bat" /sc daily /st 23:59

PowerShell 版更健壮(Get-Date 不依赖区域设置),如有需要可询问。


七、采集端部署 (各被监控服务器)

7.1 Debian 12 / Ubuntu 22

安装 Grafana Alloy

# 离线安装 (指定版本 v1.17.1)
wget https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-linux-amd64.zip
unzip alloy-linux-amd64.zip
sudo cp alloy-linux-amd64 /usr/local/bin/alloy
sudo chmod +x /usr/local/bin/alloy
 
# 创建 systemd service
sudo tee /etc/systemd/system/alloy.service << 'EOF'
[Unit]
Description=Grafana Alloy
Wants=network-online.target
After=network-online.target
 
[Service]
ExecStart=/usr/local/bin/alloy run /etc/alloy/config.alloy
Restart=on-failure
RestartSec=5s
 
[Install]
WantedBy=multi-user.target
EOF
 
sudo mkdir -p /etc/alloy

配置文件: config-debian-full.alloy

文件路径: monitor/alloy/config-debian-full.alloy (仓库中) → /etc/alloy/config.alloy (目标服务器)

标签设计

Alloy 将数据 PUSH 到云端,标签完全在采集端处理:

  • 全局标签 (external_labels): host(自动取主机名) + group(环境标识) — 由 prometheus.remote_writeloki.write 统一添加
  • 按源标签 (jobappinstance): 在各自采集模块中定义 — 独立 prometheus.scrape 设置 job_name,日志在 loki.source.file targets 中内联
数据源hostgroupjobapp/instance
Node 指标hostnamelocal/cloudnode
Systemd 指标hostnamelocal/cloudsystemd
MySQL 指标hostnamelocal/cloudmysqlinstance=
Java JMX 指标hostnamelocal/cloudjavainstance=
Nginx 指标hostnamelocal/cloudnginxinstance=
系统日志hostnamelocal/cloudauth/syslog/kernel
Java 应用日志hostnamelocal/cloudjavainstance=
MySQL 日志hostnamelocal/cloudmysqlinstance=
Nginx 日志hostnamelocal/cloudnginxinstance=
部署变量

通过 declare "cfg" 定义可复用配置,改顶部 5 个 cfg 块即可,下方自动引用:

declare "cfg" {
    argument "value" { }
    export "value" { value = argument.value.value }
}
 
cfg "endpoint_prom" { value = "https://monitor.yourdomain.com/api/v1/write" }
cfg "endpoint_loki" { value = "https://monitor.yourdomain.com/loki/api/v1/push" }
cfg "password_prom" { value = "<PROM_PASSWORD>" }
cfg "password_loki" { value = "<LOKI_PASSWORD>" }
cfg "group"         { value = "local" }
cfg "hostname"      { value = constants.hostname }  // 或改为自定义标识
配置项说明
endpoint_prom / endpoint_loki修改域名,或用本地 Nginx / 云端 VPC 内网中转地址
password_prom对应 prometheus/web.ymlprometheus 用户的明文密码
password_loki对应云端 Nginx .htpasswdloki 用户的明文密码
group环境标识: local / cloud / test / prod

host 标签使用 constants.hostname 自动取主机名,如需自定义改为字符串。

完整配置
// ==========================================
// Grafana Alloy - Debian 12 / Ubuntu 22 完整配置
// ==========================================
// 安装: 离线下载 alloy-linux-amd64.zip (见手册四章)
// 配置路径: /etc/alloy/config.alloy
// 启动: sudo systemctl enable --now alloy
//
// ==========================================
// 部署变量 — 修改 cfg 块即可, 下方自动引用
// ==========================================
// 有本地 Nginx 中转  → http://monitor-proxy.local/...
// 云端 VPC 内直连    → http://monitor-proxy.local/...
//                     (需将 monitor-proxy.local 通过内网 DNS 或 hosts 指向云端 Nginx 内网 IP)
// 云端公网直连       → https://monitor.yourdomain.com/...
 
declare "cfg" {
    argument "value" { }
    export "value" { value = argument.value.value }
}
 
cfg "endpoint_prom" { value = "https://monitor.yourdomain.com/api/v1/write" }
cfg "endpoint_loki" { value = "https://monitor.yourdomain.com/loki/api/v1/push" }
cfg "password_prom" { value = "<PROM_PASSWORD>" }
cfg "password_loki" { value = "<LOKI_PASSWORD>" }
cfg "group"         { value = "local" }
cfg "hostname"      { value = constants.hostname }  // 或改为自定义标识
 
// ==========================================
// [启用] 系统指标 (替代 node_exporter)
// ==========================================
 
prometheus.exporter.unix "node" {
    disable_collectors = ["ipvs", "btrfs", "infiniband", "xfs", "zfs"]
    filesystem {
        fs_types_exclude     = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
        mount_points_exclude = "^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)"
    }
    netdev {
        device_exclude = "^(veth.*|cali.*|[a-f0-9]{15})$"
    }
}
 
discovery.relabel "node" {
    targets = prometheus.exporter.unix.node.targets
    rule {
        target_label = "instance"
        replacement  = cfg.hostname.value
    }
    rule {
        target_label = "job"
        replacement  = "node"
    }
}
 
prometheus.scrape "node" {
    scrape_interval = "15s"
    targets         = discovery.relabel.node.output
    forward_to      = [prometheus.remote_write.default.receiver]
}
 
// ==========================================
// [启用] Systemd 服务状态
// ==========================================
 
prometheus.exporter.unix "systemd" {
    set_collectors = ["systemd"]
    systemd {
        unit_include = ".+"
        unit_exclude = ".+\\.(automount|device|mount|scope|slice)"
        enable_restarts = true
        start_time = true
    }
}
 
discovery.relabel "systemd" {
    targets = prometheus.exporter.unix.systemd.targets
    rule {
        target_label = "instance"
        replacement  = cfg.hostname.value
    }
    rule {
        target_label = "job"
        replacement  = "systemd"
    }
}
 
prometheus.scrape "systemd" {
    scrape_interval = "15s"
    targets         = discovery.relabel.systemd.output
    forward_to      = [prometheus.remote_write.default.receiver]
}
 
// ==========================================
// [启用] 安全日志 (auth.log)
// ==========================================
// syslog 和 journald 输出格式一致,共用一套处理器
// 有 auth.log 用 file 源,无 rsyslog 时改用 journal 源
 
loki.source.file "auth_log" {
    targets = [
        {__path__ = "/var/log/auth.log", job = "auth"},
    ]
    forward_to = [loki.process.auth_filter.receiver]
}
 
// loki.source.journal "auth_journal" {
//     matches = "SYSLOG_IDENTIFIER=sshd"
//     forward_to = [loki.process.auth_filter.receiver]
//     labels = {
//         job = "auth",
//     }
// }
 
loki.process "auth_filter" {
    forward_to = [loki.write.default.receiver]
 
    // 提取 SSH 登录成功信息: 时间 + 用户 + 来源IP
    // syslog 行有前缀:   Jul 23 17:17:23 hostname sshd[pid]: Accepted ...
    // journal 行无前缀:  Accepted publickey for ztjt from 1.2.3.4 ...
    stage.regex {
        expression = `(?:(?P<login_time>[A-Z][a-z]{2}\s+\d+\s+\d{2}:\d{2}:\d{2})\s+\S+\s+\S+(?:\[\d+\]):\s+)?Accepted (?:password|publickey|keyboard-interactive) for (?P<login_user>[^ ]+) from (?P<source_ip>[0-9a-fA-F:.]+)`
    }
 
    // 将 syslog 时间戳 Jul 23 17:17:23 转为 2026-07-23 17:17:23
    // journal 无 login_time,用当前处理时间(与登录时间误差秒级)
    stage.template {
        source   = "login_time"
        template = `{{ if .login_time }}{{ printf "%d-%s" now.Year (date "01-02 15:04:05" (toDate "Jan _2 15:04:05" .login_time)) }}{{ else }}{{ date "2006-01-02 15:04:05" now }}{{ end }}`
    }
 
    // GeoIP 地理位置查询 (离线 .mmdb)
    stage.geoip {
        source  = "source_ip"
        db      = "/etc/alloy/GeoLite2-City.mmdb"
        db_type = "city"
    }
 
    // SSH 登录事件统一为 info
    stage.template {
        source   = "detected_level"
        template = "info"
    }
 
    stage.structured_metadata {
        values = {
            detected_level          = "",
            login_time              = "",
            geoip_location_latitude  = "",
            geoip_location_longitude = "",
        }
    }
 
    stage.labels {
        values = {
            login_user         = "",
            source_ip          = "",
            geoip_city_name    = "",
            geoip_country_name = "",
        }
    }
}
 
// ==========================================
// [启用] 系统日志 (syslog/kern.log)
// ==========================================
 
loki.source.file "syslog" {
    targets = [
        {__path__ = "/var/log/syslog",   job = "syslog"},
        {__path__ = "/var/log/kern.log", job = "kernel"},
    ]
    forward_to = [loki.write.default.receiver]
}
 
 
// ==========================================
// [禁用] MySQL — 按需取消注释启用
// 前置: 创建监控用户
//   CREATE USER 'exporter'@'localhost' IDENTIFIED BY '<PASSWORD>';
//   GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
// ==========================================
 
// prometheus.exporter.mysql "mysql" {
//     data_source_name  = "exporter:<PASSWORD>@(localhost:3306)/"
//     enable_collectors = ["mysql.user", "engine_innodb_status", "info_schema.processlist", "info_schema.tables", "perf_schema.eventsstatements", "perf_schema.tableiowaits", "perf_schema.indexiowaits", "binlog_size"]
// }
 
// discovery.relabel "mysql" {
//     targets = prometheus.exporter.mysql.mysql.targets
//     rule {
//         target_label = "instance"
//         replacement  = "<MYSQL_NAME>"    // 替换为 MySQL 实例名, 如 mysql-prod
//     }
//     rule {
//         target_label = "job"
//         replacement  = "mysql"
//     }
//     rule {
//         target_label = "env"
//         replacement  = "test"             // test 或 prod
//     }
// }
 
// prometheus.scrape "mysql" {
//     scrape_interval = "15s"
//     targets         = discovery.relabel.mysql.output
//     forward_to      = [prometheus.remote_write.default.receiver]
// }
 
// loki.source.file "mysql_log" {
//     targets = [
//         {__path__ = "/var/log/mysql/error.log",     instance = "<MYSQL_NAME>"},
//         {__path__ = "/var/log/mysql/slow-query.log", instance = "<MYSQL_NAME>"},
//     ]
//     forward_to = [loki.process.mysql.receiver]
// }
//
// loki.process "mysql" {
//     forward_to = [loki.write.default.receiver]
//
//     stage.static_labels {
//         values = { job = "mysql" }
//     }
//
//     // 解析 MySQL 错误日志格式
//     // 支持: [Note] [MY-012345] [InnoDB] msg / [Note] InnoDB: msg / [ERROR] msg
//     stage.regex {
//         expression = `^\S+\s+\d+\s+\[(?P<raw_level>\w+)\]\s+(?:\[(?P<error_code>[\w-]+)\]\s+)?(?:\[?(?P<subsystem>\w+)\]?(?::\s+|\s+))?(?P<msg>.*)`
//     }
//
//     stage.template {
//         source   = "detected_level"
//         template = `{{- if eq .raw_level "Note" -}}info{{- else if eq .raw_level "System" -}}info{{- else if eq .raw_level "Warning" -}}warn{{- else if eq .raw_level "ERROR" -}}error{{- end -}}`
//     }
//
//     stage.structured_metadata {
//         values = {
//             detected_level = "",
//             error_code     = "",
//         }
//     }
//
//     stage.labels {
//         values = {
//             subsystem = "",
//         }
//     }
// }
 
 
// ==========================================
// [禁用] Java JMX 指标 — 按需取消注释启用
// 前置: JMX Exporter 1.6.0 已通过 javaagent 挂载到 JVM
//   java -javaagent:jmx_prometheus_javaagent-1.6.0.jar=9404:jmx_exporter.yaml -jar app.jar
// ==========================================
 
// prometheus.scrape "java" {
//     scrape_interval = "15s"
//     job_name        = "java"
//     targets = [
//         {__address__ = "localhost:9404", instance = "gateway", env = "test"},
//         // {__address__ = "localhost:9405", instance = "oa",      env = "test"},
//         // {__address__ = "localhost:9406", instance = "hr",      env = "prod"},
//     ]
//     forward_to = [prometheus.remote_write.default.receiver]
// }
 
 
// ==========================================
// [禁用] Java 应用日志 — 按需取消注释启用
// 格式: YYYY-MM-DD HH:MM:SS.mmm [IP] [threadID] [0] [] LEVEL class - message
// ==========================================
 
// loki.source.file "java_logs" {
//     targets = [
//         {__path__ = "/opt/gateway/logs/*.log", instance = "gateway", env = "test"},
//         {__path__ = "/opt/oa/logs/*.log",      instance = "oa",      env = "test"},
//         {__path__ = "/opt/hr/logs/*.log",       instance = "hr",      env = "prod"},
//     ]
//     forward_to = [loki.process.java.receiver]
// }
//
// loki.process "java" {
//     forward_to = [loki.write.default.receiver]
//
//     stage.static_labels {
//         values = { job = "java" }
//     }
//
//     stage.regex {
//         expression = `^(?P<timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d+)\s+\[(?P<client_ip>[^\]]+)\]\s+\[(?P<thread_id>\d+)\]\s+\[\d+\]\s+\[\]\s+(?P<level>\w+)\s+(?P<class>\S+)\s+-\s+(?P<msg>.*)$`
//     }
//
//     stage.template {
//         source   = "detected_level"
//         template = `{{- if eq .level "DEBUG" -}}debug{{- else if eq .level "INFO" -}}info{{- else if eq .level "WARN" -}}warn{{- else if eq .level "ERROR" -}}error{{- else -}}{{ .level | ToLower }}{{- end -}}`
//     }
//
//     stage.structured_metadata {
//         values = {
//             detected_level = "",
//             class          = "",
//             client_ip      = "",
//             thread_id      = "",
//         }
//     }
// }
 
 
// ==========================================
// [禁用] Nginx 指标 + 日志 — 按需取消注释启用
// 前提: 部署 nginx-prometheus-exporter (Alloy 无内置 nginx exporter)
//   Docker 方式: nginx/nginx-prometheus-exporter:1.5.1
//     监听 :9113, 抓取 nginx stub_status
//     需与 Nginx 在同一 Docker 网络 (如 shared-proxy-network)
//     命令: docker run -d --name nginx-exporter \
//             --network shared-proxy-network \
//             -p 127.0.0.1:9113:9113 \
//             nginx/nginx-prometheus-exporter:1.5.1 \
//             --nginx.scrape-uri=http://<nginx容器名>/nginx_status
//
// 多实例: 每个实例单独跑一个 exporter, 不同端口
//
// access log 格式 (main):
//   $remote_addr - $remote_user [$time_iso8601] "$request" $status
//   $body_bytes_sent "$http_referer" "$http_user_agent"
//   "$http_x_forwarded_for" rt=$request_time uct="$upstream_connect_time"
//   uht="$upstream_header_time" urt="$upstream_response_time"
// error log 格式 (不可自定义):
//   YYYY/MM/DD HH:MM:SS [level] pid#tid: message, client: IP, server: name,
//   request: "METHOD /path HTTP/1.1", host: "hostname"
// ==========================================
 
// prometheus.scrape "nginx" {
//     scrape_interval = "15s"
//     targets = [
//         {__address__ = "127.0.0.1:9113", instance = "proxy", job = "nginx"},
//         // {__address__ = "127.0.0.1:9114", instance = "backend", job = "nginx"},
//     ]
//     forward_to = [prometheus.remote_write.default.receiver]
// }
 
// loki.source.file "nginx_access" {
//     targets = [
//         {__path__ = "/var/log/nginx/frontend/access.log", instance = "frontend"},
//         // {__path__ = "/var/log/nginx/backend/access.log",  instance = "backend"},
//     ]
//     forward_to = [loki.process.nginx_access.receiver]
// }
//
// loki.process "nginx_access" {
//     forward_to = [loki.write.default.receiver]
//
//     stage.static_labels {
//         values = { job = "nginx" }
//     }
//
//     stage.regex {
//         expression = `^(?P<remote_addr>\S+)\s+-\s+(?P<remote_user>\S+)\s+\[(?P<time_iso8601>[^\]]+)\]\s+"(?P<request>[^"]*)"\s+(?P<status>\d+)\s+(?P<body_bytes_sent>\d+)\s+"(?P<http_referer>[^"]*)"\s+"(?P<http_user_agent>[^"]*)"\s+"(?P<http_x_forwarded_for>[^"]*)"\s+rt=(?P<request_time>[\d.]+)\s+uct="(?P<upstream_connect_time>[^"]*)"\s+uht="(?P<upstream_header_time>[^"]*)"\s+urt="(?P<upstream_response_time>[^"]*)"`
//     }
//
//     stage.regex {
//         source     = "request"
//         expression = `^(?P<method>\S+)\s+(?P<request_uri>\S+)\s+\S+$`
//     }
//
//     stage.template {
//         source   = "detected_level"
//         template = "info"
//     }
//
//     stage.structured_metadata {
//         values = {
//             detected_level           = "",
//             request_time             = "",
//             upstream_response_time   = "",
//             body_bytes_sent          = "",
//             remote_addr              = "",
//             http_referer             = "",
//             http_user_agent          = "",
//             http_x_forwarded_for     = "",
//             request_uri              = "",
//         }
//     }
//
//     stage.labels {
//         values = {
//             status = "",
//             method = "",
//         }
//     }
// }
//
// loki.source.file "nginx_error" {
//     targets = [
//         {__path__ = "/var/log/nginx/frontend/error.log", instance = "frontend"},
//         // {__path__ = "/var/log/nginx/backend/error.log",  instance = "backend"},
//     ]
//     forward_to = [loki.process.nginx_error.receiver]
// }
//
// loki.process "nginx_error" {
//     forward_to = [loki.write.default.receiver]
//
//     stage.static_labels {
//         values = { job = "nginx" }
//     }
//
//     stage.regex {
//         expression = `^\d{4}/\d{2}/\d{2}\s+\d{2}:\d{2}:\d{2}\s+\[(?P<level>\w+)\]\s+\d+#\d+:\s+(?P<msg>.+?)(?:,\s*client:\s*(?P<client>[^,]+))?(?:,\s*server:\s*(?P<server>[^,]+))?(?:,\s*request:\s*"(?P<request>[^"]*)")?(?:,\s*host:\s*"(?P<http_host>[^"]*)")?\s*$`
//     }
//
//     stage.template {
//         source   = "detected_level"
//         template = `{{- if eq .level "debug" -}}debug{{- else if eq .level "info" -}}info{{- else if eq .level "notice" -}}info{{- else if eq .level "warn" -}}warn{{- else if eq .level "error" -}}error{{- else if eq .level "crit" -}}critical{{- else if eq .level "alert" -}}error{{- else if eq .level "emerg" -}}fatal{{- end -}}`
//     }
//
//     stage.structured_metadata {
//         values = {
//             detected_level = "",
//             client         = "",
//             server         = "",
//             request        = "",
//             http_host      = "",
//         }
//     }
// }
 
 
// ==========================================
// 推送配置 (引用 cfg 变量, 无需修改)
// ==========================================
 
// --- 推送日志到 Loki ---
loki.write "default" {
    external_labels = {
        host  = cfg.hostname.value,
        group = cfg.group.value,
    }
    endpoint {
        url = cfg.endpoint_loki.value
        basic_auth {
            username = "loki"
            password = cfg.password_loki.value
        }
    }
}
 
// --- 推送指标到 Prometheus ---
prometheus.remote_write "default" {
    external_labels = {
        host  = cfg.hostname.value,
        group = cfg.group.value,
    }
    endpoint {
        url = cfg.endpoint_prom.value
        basic_auth {
            username = "prometheus"
            password = cfg.password_prom.value
        }
    }
}

Alloy 语法提示: 带 = 的 argument(如 external_labels = { ... })必须写在不带 = 的 block(如 endpoint { ... })之前,否则会报语法错误。

部署步骤
sudo cp alloy/config-debian-full.alloy /etc/alloy/config.alloy
# 修改文件头部 cfg 块中的域名、密码、group
sudo vim /etc/alloy/config.alloy
sudo systemctl restart alloy
sudo systemctl status alloy

如需使用环境变量,将配置变量替换为 sys.env("VAR")


7.2 Windows Server 2016+

安装 Grafana Alloy

  1. 下载 alloy-installer-windows-amd64.exe
  2. 管理员运行安装(或静默安装 alloy-installer-windows-amd64.exe /S)→ 自动安装为 Windows Service

配置文件: config-windows-full.alloy

文件路径: monitor/alloy/config-windows-full.alloy (仓库中) → C:\Program Files\GrafanaLabs\Alloy\config.alloy (目标服务器)

部署变量

与 Debian 版一致,修改文档头部 4 个变量即可。标签策略见 7.1 节标签设计表格。

完整配置
// ==========================================
// Grafana Alloy - Windows Server 2016+ 完整配置
// ==========================================
// 安装: 管理员运行 alloy-installer-windows-amd64.exe /S
// 配置路径: C:\Program Files\GrafanaLabs\Alloy\config.alloy
// 服务管理: sc.exe start/stop/query "Grafana Alloy"
//
// ==========================================
// 部署变量 — 修改 cfg 块即可, 下方自动引用
// ==========================================
// 有本地 Nginx 中转  → http://monitor-proxy.local/...
// 云端 VPC 内直连    → http://monitor-proxy.local/...
//                     (需将 monitor-proxy.local 通过内网 DNS 或 hosts 指向云端 Nginx 内网 IP)
// 云端公网直连       → https://monitor.yourdomain.com/...
 
declare "cfg" {
    argument "value" { }
    export "value" { value = argument.value.value }
}
 
cfg "endpoint_prom" { value = "https://monitor.yourdomain.com/api/v1/write" }
cfg "endpoint_loki" { value = "https://monitor.yourdomain.com/loki/api/v1/push" }
cfg "password_prom" { value = "<PROM_PASSWORD>" }
cfg "password_loki" { value = "<LOKI_PASSWORD>" }
cfg "group"         { value = "local" }
cfg "hostname"      { value = constants.hostname }  // 或改为自定义标识
 
// ==========================================
// [启用] Windows 系统指标 (替代 windows_exporter)
// ==========================================
// 注意: cs、logon 收集器已在 Alloy 中移除,不再可用
 
prometheus.exporter.windows "win" {
    enabled_collectors = ["cpu", "logical_disk", "memory", "net", "os", "process", "service", "system", "tcp"]
    service {
        include = "(mysql.*|filezilla-server)"
    }
}
 
discovery.relabel "win_metrics" {
    targets = prometheus.exporter.windows.win.targets
    rule {
        target_label = "instance"
        replacement  = cfg.hostname.value
    }
    rule {
        target_label = "job"
        replacement  = "node"
    }
}
 
prometheus.scrape "node" {
    scrape_interval = "15s"
    targets         = discovery.relabel.win_metrics.output
    forward_to      = [prometheus.remote_write.default.receiver]
}
 
// ==========================================
// [启用] Windows 安全事件日志 (登录成功 4624 / 失败 4625)
// ==========================================
 
loki.source.windowsevent "security" {
    eventlog_name = "Security"
    xpath_query   = "*[System[(EventID=4624 or EventID=4625)]]"
    use_incoming_timestamp = true
    bookmark_path = "C:\\ProgramData\\GrafanaLabs\\Alloy\\bookmark-security.xml"
    forward_to    = [loki.process.win_security.receiver]
    labels = {
        job = "windows_security",
    }
}
 
loki.process "win_security" {
    forward_to = [loki.write.default.receiver]
 
    stage.json {
        expressions = {
            source     = "source",
            event_id   = "event_id",
            level      = "levelText",
            time       = "timeCreated",
            message    = "message",
            event_data = "event_data",
        }
    }
 
    // 从 XML 格式 event_data 提取登录用户、来源IP、登录类型
    stage.regex {
        source     = "event_data"
        expression = `<Data Name='TargetUserName'>(?P<login_user>[^<]*)</Data>`
    }
 
    stage.regex {
        source     = "event_data"
        expression = `<Data Name='IpAddress'>(?P<source_ip>[^<]*)</Data>`
    }
 
    stage.regex {
        source     = "event_data"
        expression = `<Data Name='LogonType'>(?P<logon_type>[^<]*)</Data>`
    }
 
    // 过滤系统账号和网络会话噪音,只保留真实用户登录
    stage.drop {
        source      = "login_user"
        expression  = "^(DWM-\\d+|UMFD-\\d+|.*\\$|ANONYMOUS LOGON)$"
        drop_counter_reason = "system_account"
    }
 
    stage.drop {
        source      = "logon_type"
        expression  = "^(3|4|5)$"
        drop_counter_reason = "service_logon"
    }
 
    // GeoIP 地理位置查询 (离线 .mmdb)
    stage.geoip {
        source  = "source_ip"
        db      = "C:\\Program Files\\GrafanaLabs\\Alloy\\GeoLite2-City.mmdb"
        db_type = "city"
    }
 
    // 格式化 login_time 为 yyyy-MM-dd HH:mm:ss
    stage.template {
        source   = "login_time"
        template = `{{ date "2006-01-02 15:04:05" (toDate "2006-01-02T15:04:05Z07:00" .time) }}`
    }
 
    // Security 事件无数字 level,用 levelText(中文)映射
    // 信息→info  警告→warn  错误→error  关键→critical  详细→debug
    stage.template {
        source   = "detected_level"
        template = `{{- if eq .level "信息" -}}info{{- else if eq .level "警告" -}}warn{{- else if eq .level "错误" -}}error{{- else if eq .level "关键" -}}critical{{- else if eq .level "详细" -}}debug{{- end -}}`
    }
 
    // 将数字 logon_type 映射为可读名称
    stage.template {
        source   = "logon_type_name"
        template = `{{- $t := .logon_type }}{{- if eq $t "2" -}}Interactive{{- else if eq $t "3" -}}Network{{- else if eq $t "4" -}}Batch{{- else if eq $t "5" -}}Service{{- else if eq $t "7" -}}Unlock{{- else if eq $t "8" -}}NetworkCleartext{{- else if eq $t "9" -}}NewCredentials{{- else if eq $t "10" -}}RemoteInteractive{{- else if eq $t "11" -}}CachedInteractive{{- else -}}Unknown{{- end -}}`
    }
 
    stage.structured_metadata {
        values = {
            detected_level          = "",
            login_time              = "",
            geoip_location_latitude  = "",
            geoip_location_longitude = "",
        }
    }
 
    stage.labels {
        values = {
            event_id         = "",
            level            = "",
            login_user       = "",
            source_ip        = "",
            logon_type       = "",
            logon_type_name  = "",
            geoip_city_name  = "",
            geoip_country_name = "",
        }
    }
}
 
// ==========================================
// [启用] Windows 应用事件日志
// ==========================================
 
loki.source.windowsevent "application" {
    eventlog_name = "Application"
    xpath_query   = "*"
    use_incoming_timestamp = true
    bookmark_path = "C:\\ProgramData\\GrafanaLabs\\Alloy\\bookmark-application.xml"
    forward_to    = [loki.process.win_application.receiver]
    labels = {
        job = "windows_application",
    }
}
 
loki.process "win_application" {
    forward_to = [loki.write.default.receiver]
 
    stage.json {
        expressions = {
            source     = "source",
            event_id   = "event_id",
            level      = "level",
            level_text = "levelText",
            time       = "timeCreated",
            message    = "message",
            event_data = "event_data",
        }
    }
 
    // 将数字 level 转为字符串,确保后续比较不会因类型问题失败
    stage.template {
        source   = "level_str"
        template = `{{ printf "%v" .level }}`
    }
 
    // 将 level 映射为 Loki 标准 log level
    // Windows: 1=Critical 2=Error 3=Warning 4=Information 5=Verbose
    stage.template {
        source   = "detected_level"
        template = `{{- if eq .level_str "1" -}}critical{{- else if eq .level_str "2" -}}error{{- else if eq .level_str "3" -}}warn{{- else if eq .level_str "4" -}}info{{- else if eq .level_str "5" -}}debug{{- else -}}unknown{{- end -}}`
    }
 
    stage.structured_metadata {
        values = {
            detected_level = "",
            event_id       = "",
            source         = "",
        }
    }
 
    stage.labels {
        values = {
            level = "",
        }
    }
}
 
 
// ==========================================
// [禁用] MySQL — 按需取消注释启用
// 前置: 创建监控用户
//   CREATE USER 'exporter'@'localhost' IDENTIFIED BY '<PASSWORD>';
//   GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
// ==========================================
 
// prometheus.exporter.mysql "mysql" {
//     data_source_name  = "exporter:<PASSWORD>@(localhost:3306)/"
//     enable_collectors = ["mysql.user", "engine_innodb_status", "info_schema.processlist", "info_schema.tables", "perf_schema.eventsstatements", "perf_schema.tableiowaits", "perf_schema.indexiowaits", "binlog_size"]
// }
 
// discovery.relabel "mysql" {
//     targets = prometheus.exporter.mysql.mysql.targets
//     rule {
//         target_label = "instance"
//         replacement  = "<MYSQL_NAME>"    // 替换为 MySQL 实例名, 如 mysql-prod
//     }
//     rule {
//         target_label = "job"
//         replacement  = "mysql"
//     }
//     rule {
//         target_label = "env"
//         replacement  = "test"             // test 或 prod
//     }
// }
 
// prometheus.scrape "mysql" {
//     scrape_interval = "15s"
//     targets         = discovery.relabel.mysql.output
//     forward_to      = [prometheus.remote_write.default.receiver]
// }
 
// loki.source.file "mysql_log" {
//     targets = [
//         {__path__ = "C:\\ProgramData\\MySQL\\MySQL Server 8.0\\Data\\*.err", instance = "<MYSQL_NAME>"},
//     ]
//     forward_to = [loki.process.mysql.receiver]
// }
//
// loki.process "mysql" {
//     forward_to = [loki.write.default.receiver]
//
//     stage.static_labels {
//         values = { job = "mysql" }
//     }
//
//     // 解析 MySQL 错误日志格式
//     // 支持: [Note] [MY-012345] [InnoDB] msg / [Note] InnoDB: msg / [ERROR] msg
//     stage.regex {
//         expression = `^\S+\s+\d+\s+\[(?P<raw_level>\w+)\]\s+(?:\[(?P<error_code>[\w-]+)\]\s+)?(?:\[?(?P<subsystem>\w+)\]?(?::\s+|\s+))?(?P<msg>.*)`
//     }
//
//     stage.template {
//         source   = "detected_level"
//         template = `{{- if eq .raw_level "Note" -}}info{{- else if eq .raw_level "System" -}}info{{- else if eq .raw_level "Warning" -}}warn{{- else if eq .raw_level "ERROR" -}}error{{- end -}}`
//     }
//
//     stage.structured_metadata {
//         values = {
//             detected_level = "",
//             error_code     = "",
//         }
//     }
//
//     stage.labels {
//         values = {
//             subsystem = "",
//         }
//     }
// }
 
 
// ==========================================
// [禁用] Java JMX 指标 — 按需取消注释启用
// 前置: JMX Exporter 1.6.0 已通过 javaagent 挂载到 JVM
//   -javaagent:C:\jmx_exporter\jmx_prometheus_javaagent-1.6.0.jar=9404:C:\jmx_exporter\jmx_exporter.yaml
// ==========================================
 
// prometheus.scrape "java" {
//     scrape_interval = "15s"
//     job_name        = "java"
//     targets = [
//         {__address__ = "localhost:9404", instance = "gateway", env = "test"},
//         // {__address__ = "localhost:9405", instance = "oa",      env = "test"},
//         // {__address__ = "localhost:9406", instance = "hr",      env = "prod"},
//     ]
//     forward_to = [prometheus.remote_write.default.receiver]
// }
 
 
// ==========================================
// [禁用] Java 应用日志 — 按需取消注释启用
// 格式: YYYY-MM-DD HH:MM:SS.mmm [IP] [threadID] [0] [] LEVEL class - message
// ==========================================
 
// loki.source.file "java_logs" {
//     targets = [
//         {__path__ = "C:\\gateway\\logs\\*.log", instance = "gateway", env = "test"},
//         {__path__ = "C:\\oa\\logs\\*.log",      instance = "oa",      env = "test"},
//         {__path__ = "C:\\hr\\logs\\*.log",       instance = "hr",      env = "prod"},
//     ]
//     forward_to = [loki.process.java.receiver]
// }
//
// loki.process "java" {
//     forward_to = [loki.write.default.receiver]
//
//     stage.static_labels {
//         values = { job = "java" }
//     }
//
//     stage.regex {
//         expression = `^(?P<timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d+)\s+\[(?P<client_ip>[^\]]+)\]\s+\[(?P<thread_id>\d+)\]\s+\[\d+\]\s+\[\]\s+(?P<level>\w+)\s+(?P<class>\S+)\s+-\s+(?P<msg>.*)$`
//     }
//
//     stage.template {
//         source   = "detected_level"
//         template = `{{- if eq .level "DEBUG" -}}debug{{- else if eq .level "INFO" -}}info{{- else if eq .level "WARN" -}}warn{{- else if eq .level "ERROR" -}}error{{- else -}}{{ .level | ToLower }}{{- end -}}`
//     }
//
//     stage.structured_metadata {
//         values = {
//             detected_level = "",
//             class          = "",
//             client_ip      = "",
//             thread_id      = "",
//         }
//     }
// }
 
 
// ==========================================
// [禁用] Nginx 指标 + 日志 — 按需取消注释启用
// 前提: 部署 nginx-prometheus-exporter (Alloy 无内置 nginx exporter)
//   下载: https://github.com/nginx/nginx-prometheus-exporter/releases
//   运行: nginx-prometheus-exporter.exe --nginx.scrape-uri=http://localhost/nginx_status
//
// 多实例: 每个实例单独跑一个 exporter, 不同端口
//
// access log 格式 (main):
//   $remote_addr - $remote_user [$time_iso8601] "$request" $status
//   $body_bytes_sent "$http_referer" "$http_user_agent"
//   "$http_x_forwarded_for" rt=$request_time uct="$upstream_connect_time"
//   uht="$upstream_header_time" urt="$upstream_response_time"
// error log 格式 (不可自定义):
//   YYYY/MM/DD HH:MM:SS [level] pid#tid: message, client: IP, server: name,
//   request: "METHOD /path HTTP/1.1", host: "hostname"
// ==========================================
 
// prometheus.scrape "nginx" {
//     scrape_interval = "15s"
//     targets = [
//         {__address__ = "localhost:9113", instance = "proxy", job = "nginx"},
//     ]
//     forward_to = [prometheus.remote_write.default.receiver]
// }
 
// loki.source.file "nginx_access" {
//     targets = [
//         {__path__ = "C:\\nginx\\frontend\\logs\\access.log", instance = "frontend"},
//         // {__path__ = "C:\\nginx\\backend\\logs\\access.log",  instance = "backend"},
//     ]
//     forward_to = [loki.process.nginx_access.receiver]
// }
//
// loki.process "nginx_access" {
//     forward_to = [loki.write.default.receiver]
//
//     stage.static_labels {
//         values = { job = "nginx" }
//     }
//
//     stage.regex {
//         expression = `^(?P<remote_addr>\S+)\s+-\s+(?P<remote_user>\S+)\s+\[(?P<time_iso8601>[^\]]+)\]\s+"(?P<request>[^"]*)"\s+(?P<status>\d+)\s+(?P<body_bytes_sent>\d+)\s+"(?P<http_referer>[^"]*)"\s+"(?P<http_user_agent>[^"]*)"\s+"(?P<http_x_forwarded_for>[^"]*)"\s+rt=(?P<request_time>[\d.]+)\s+uct="(?P<upstream_connect_time>[^"]*)"\s+uht="(?P<upstream_header_time>[^"]*)"\s+urt="(?P<upstream_response_time>[^"]*)"`
//     }
//
//     stage.regex {
//         source     = "request"
//         expression = `^(?P<method>\S+)\s+(?P<request_uri>\S+)\s+\S+$`
//     }
//
//     stage.template {
//         source   = "detected_level"
//         template = "info"
//     }
//
//     stage.structured_metadata {
//         values = {
//             detected_level           = "",
//             request_time             = "",
//             upstream_response_time   = "",
//             body_bytes_sent          = "",
//             remote_addr              = "",
//             http_referer             = "",
//             http_user_agent          = "",
//             http_x_forwarded_for     = "",
//             request_uri              = "",
//         }
//     }
//
//     stage.labels {
//         values = {
//             status = "",
//             method = "",
//         }
//     }
// }
//
// loki.source.file "nginx_error" {
//     targets = [
//         {__path__ = "C:\\nginx\\frontend\\logs\\error.log", instance = "frontend"},
//         // {__path__ = "C:\\nginx\\backend\\logs\\error.log",  instance = "backend"},
//     ]
//     forward_to = [loki.process.nginx_error.receiver]
// }
//
// loki.process "nginx_error" {
//     forward_to = [loki.write.default.receiver]
//
//     stage.static_labels {
//         values = { job = "nginx" }
//     }
//
//     stage.regex {
//         expression = `^\d{4}/\d{2}/\d{2}\s+\d{2}:\d{2}:\d{2}\s+\[(?P<level>\w+)\]\s+\d+#\d+:\s+(?P<msg>.+?)(?:,\s*client:\s*(?P<client>[^,]+))?(?:,\s*server:\s*(?P<server>[^,]+))?(?:,\s*request:\s*"(?P<request>[^"]*)")?(?:,\s*host:\s*"(?P<http_host>[^"]*)")?\s*$`
//     }
//
//     stage.template {
//         source   = "detected_level"
//         template = `{{- if eq .level "debug" -}}debug{{- else if eq .level "info" -}}info{{- else if eq .level "notice" -}}info{{- else if eq .level "warn" -}}warn{{- else if eq .level "error" -}}error{{- else if eq .level "crit" -}}critical{{- else if eq .level "alert" -}}error{{- else if eq .level "emerg" -}}fatal{{- end -}}`
//     }
//
//     stage.structured_metadata {
//         values = {
//             detected_level = "",
//             client         = "",
//             server         = "",
//             request        = "",
//             http_host      = "",
//         }
//     }
// }
 
 
// ==========================================
// 推送配置 (引用 cfg 变量, 无需修改)
// ==========================================
 
// --- 推送日志到 Loki ---
loki.write "default" {
    external_labels = {
        host  = cfg.hostname.value,
        group = cfg.group.value,
    }
    endpoint {
        url = cfg.endpoint_loki.value
        basic_auth {
            username = "loki"
            password = cfg.password_loki.value
        }
    }
}
 
// --- 推送指标到 Prometheus ---
prometheus.remote_write "default" {
    external_labels = {
        host  = cfg.hostname.value,
        group = cfg.group.value,
    }
    endpoint {
        url = cfg.endpoint_prom.value
        basic_auth {
            username = "prometheus"
            password = cfg.password_prom.value
        }
    }
}

Alloy 语法提示: 带 = 的 argument(如 external_labels = { ... })必须写在不带 = 的 block(如 endpoint { ... })之前,否则会报语法错误。

部署步骤
# 复制配置
copy config-windows-full.alloy "C:\Program Files\GrafanaLabs\Alloy\config.alloy"
# 修改文件头部 cfg 块中的域名、密码、group
 
# 启动服务
sc.exe start "Grafana Alloy"
sc.exe query "Grafana Alloy"

注意: Windows 的认证配置与 Debian 版完全一致。


7.3 Java 服务 JMX Exporter

启用 Java 模块时的前置步骤:

# === JMX Exporter 1.6.0 (Java 8+) ===
# Linux
curl -LO https://github.com/prometheus/jmx_exporter/releases/download/v1.6.0/jmx_prometheus_javaagent-1.6.0.jar
# 编写 JMX 规则配置文件到 /opt/jmx_exporter/jmx_exporter.yaml (参考下方配置)
# JVM 启动参数: -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-1.6.0.jar=9404:/opt/jmx_exporter/jmx_exporter.yaml
 
# Windows (PowerShell)
curl -LO https://github.com/prometheus/jmx_exporter/releases/download/v1.6.0/jmx_prometheus_javaagent-1.6.0.jar
# 编写 JMX 规则配置文件到 C:\jmx_exporter\jmx_exporter.yaml (参考下方配置)
# JVM 启动参数: -javaagent:C:\jmx_exporter\jmx_prometheus_javaagent-1.6.0.jar=9404:C:\jmx_exporter\jmx_exporter.yaml
 
# === JMX Exporter 0.17.1 Java 6 构建 (Java 6/7) ===
# Linux
curl -LO https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent_java6/0.17.1/jmx_prometheus_javaagent_java6-0.17.1.jar
# JVM 启动参数: -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent_java6-0.17.1.jar=9404:/opt/jmx_exporter/jmx_exporter.yaml
 
# Windows
curl -LO https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent_java6/0.17.1/jmx_prometheus_javaagent_java6-0.17.1.jar
# JVM 启动参数: -javaagent:C:\jmx_exporter\jmx_prometheus_javaagent_java6-0.17.1.jar=9404:C:\jmx_exporter\jmx_exporter.yaml

JMX Exporter 规则配置参考 (jmx_exporter.yaml):

---
lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:
  - pattern: 'com.jolbox.bonecp<type=BoneCP[^,]*><>(TotalFree|TotalLeased|TotalCreatedConnections):'
    name: bonecp_$1
    help: BoneCP connection pool $1
    type: GAUGE

说明: JVM 基础指标(内存/GC/线程/类加载/CPU)由 JMX Exporter 内置自动导出,无需在 rules 中声明。以上仅额外采集 BoneCP 连接池状态。如果无需连接池监控,可设为 rules: []

Tomcat 用户: 如果 Catalina MBean 可用(Tomcat 7+ 且 JMX 正常),可参考 官方 examples/tomcat.yml 追加 Servlet/ThreadPool/Session 规则。

JMX Exporter 不是独立 Service,随 JVM 启停。同机 Alloy 通过 localhost:9404 抓取。上方 jmx_exporter.yaml 兼容 0.17.x 和 1.6.0 两个版本。


7.4 MySQL 监控账号创建

启用 MySQL 模块时,在 MySQL 中创建监控用户(Alloy 用此账号连接):

CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'password';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';

将配置中 data_source_name = "exporter:password@(localhost:3306)/" 的密码替换为实际值。


7.5 配置模块启用速查

两个 full 配置文件结构相同,通过注释/取消注释控制各模块:

模块默认状态启用方法
系统指标 (node/systemd)启用无需操作
系统日志 (auth/syslog)启用无需操作
安全事件日志 (Windows)启用无需操作
MySQL 指标 + 日志禁用取消 4 个注释块: exporter + discovery + scrape + loki.source.file
Java JMX 指标禁用取消 prometheus.scrape "java" 块注释
Java 应用日志禁用取消 loki.source.file "java_logs" 块注释
Nginx 指标 + 日志禁用先部署 nginx-prometheus-exporter,再取消 prometheus.scrape "nginx"loki.source.file "nginx_access"loki.source.file "nginx_error" 注释

说明: 每个模块现在是自包含的 4 段式结构(exporter → discovery.relabel → prometheus.scrape → loki.source.file),每个 prometheus.scrape 有独立的 job_name


八、采集说明

8.1 标签策略

Alloy 在采集端(PUSH 侧)完成全部标签处理,服务端(Prometheus/Loki)不参与:

Alloy 采集端
├── 顶部配置变量 (改一处,全局生效)
│   ├── hostname = constants.hostname  (或 "app_local", 改 cfg 块即可)
│   └── group    = "local"
│
├── external_labels (全局,同机所有数据一致)
│   ├── host  = hostname
│   └── group = group
│
├── 指标标签 (独立 prometheus.scrape,各自 job_name)
│   ├── job=node,      instance=hostname        → node_exporter metrics
│   ├── job=systemd,   instance=hostname        → systemd 服务状态
│   ├── job=mysql,     instance=<MYSQL_NAME>    → MySQL exporter metrics
│   ├── job=java,      instance=<SERVICE_NAME>  → JMX Exporter metrics
│   └── job=nginx,     instance=<NGINX_NAME>    → nginx stub_status metrics
│
└── 日志标签 (loki.source.file targets 内联)
    ├── job=auth/syslog/kernel                    → 系统日志 (host 已足够)
    ├── job=mysql,     instance=<MYSQL_NAME>      → MySQL 日志
    ├── job=java,      instance=<SERVICE_NAME>    → Java 应用日志
    └── job=nginx,     instance=<NGINX_NAME>      → Nginx 日志

设计原则:

  • host + group = 全局(external_labels),同机所有数据一致
  • job = 数据源类型(prometheus.scrape.job_name 或 loki.source.file.job)
  • instance = 具体实例标识(Prometheus 标准标签,所有指标和可能多实例的日志都带)
  • 单实例日志(auth/syslog/kernel)不加 instancehost 已唯一标识

8.2 采集端分工

环境配置文件默认采集可选扩展
Debian 12 / Ubuntu 22config-debian-full.alloy系统指标 + systemd + 系统日志 (auth/syslog/kernel)MySQL 指标+日志 / Java JMX+日志 / Nginx 指标+日志
Windows Server 2016+config-windows-full.alloy系统指标 + Windows 事件日志 (安全+应用)MySQL 指标+日志 / Java JMX+日志 / Nginx 指标+日志

8.3 MySQL 采集

MySQL 指标由 Grafana Alloy 内置的 prometheus.exporter.mysql 在本机 localhost 采集,不需要独立的 mysqld-exporter 容器。同时支持错误日志和慢查询日志采集。

MySQL 8 (localhost:3306)
    ↑ DSN
Grafana Alloy (prometheus.exporter.mysql) ──→ 指标 → 云端 Prometheus
Grafana Alloy (loki.source.file)           ──→ 日志 → 云端 Loki

8.4 Nginx 采集

Grafana Alloy 没有内置 prometheus.exporter.nginx 组件,Nginx 指标需通过独立的 nginx-prometheus-exporter 采集:

Nginx (stub_status)
    ↑ scrape
nginx-prometheus-exporter (Docker/二进制, :9113)
    ↑ prometheus.scrape
Grafana Alloy                                ──→ 指标 → 云端 Prometheus
Grafana Alloy (loki.source.file)             ──→ 日志 → 云端 Loki

Docker 部署 (推荐,与 Nginx 同一 compose):

nginx-exporter:
  image: nginx/nginx-prometheus-exporter:1.5.1
  container_name: monitor-nginx-exporter
  restart: unless-stopped
  command: ["--nginx.scrape-uri=http://nginx/nginx_status"]
  ports:
    - "127.0.0.1:9113:9113"
  networks:
    - proxy-network   # 与 Nginx 同一 Docker 网络
  cap_drop:
    - ALL
  security_opt:
    - no-new-privileges:true
  privileged: false
  deploy:
    resources:
      limits:
        cpus: "0.1"
        memory: 32M
        pids: 20
      reservations:
        memory: 16M

注意: location /nginx_status 不要写 allow/deny,继承 server 级规则即可(Docker 容器间流量源 IP 不是 127.0.0.1)。


九、告警说明

9.1 告警链路

  • Prometheus 指标告警 (prometheus/rules/alerts.yml) — Linux + Windows 双平台(CPU/内存/磁盘/下线/IO)
  • Loki 安全日志告警 (loki/rules/fake/security.yml) — SSH 爆破/Windows 登录异常/登录提醒
  • 告警发送到 Alertmanager → 按严重级别路由到企业微信推送
严重级别路由接收器推送行为
criticalwechat-critical推送告警 + 恢复通知
warningwechat-warning推送告警 + 恢复通知
loginwechat-login推送登录提醒,不推恢复
infowechat-info推送通知,不推恢复
  • 可在 Alertmanager Web UI (https://monitor.yourdomain.com/alertmanager/) 查看活跃告警

9.2 企业微信配置前置条件

告警通过企业微信自建应用的消息接口推送,部署前需在企业微信管理后台完成以下准备:

第一步:获取企业 ID (corp_id)

  • 登录 企业微信管理后台
  • 进入 “我的企业” → “企业信息” → “企业ID” (即 corp_id)
  • 填入 alertmanager/alertmanager.yml<YOUR_CORP_ID>

第二步:创建自建应用,获取 AgentID 和 Secret

  • 管理后台 → “应用管理” → “应用” → “自建” → “创建应用”
  • 设置应用名称 (如 “监控告警”)、应用 logo、可见范围 (选择接收告警的部门)
  • 创建完成后进入应用详情页,可以看到:
    • AgentId → 填入 <YOUR_AGENT_ID>
    • Secret → 填入 <YOUR_API_SECRET> (点击查看,仅显示一次,务必保存)

第三步:获取接收人账号 (to_user)

  • 管理后台 → “通讯录” → 点击具体成员 → 字段名 “账号”
  • 支持多人,用 | 分隔,如 zhangsan|lisi
  • 填入 alertmanager.ymlto_user 字段

第四步:配置应用可信 IP

  • 在自建应用详情页 → “企业可信IP” → 添加云端服务器的公网 IP
  • 企业微信会校验调用来源 IP,不配置将导致消息发送失败

第五步:配置可信域名

  • 自建应用详情页 → “网页授权及 JS-SDK” → “设置可信域名”
  • 填入监控域名 (如 monitor.yourdomain.com),下载验证文件 (如 WW_verify_<VERIFICATION_FILE>.txt)
  • 将验证文件放到 /opt/monitor/nginx/verify/ 目录下
  • 确保 Nginx 容器挂载了该目录(见 5.15 节 volumes 配置)
  • Nginx 已预置 /WW_verify_ 路径直接返回静态文件,无需额外配置
  • 在企业微信管理后台点击”确定”完成域名校验

注意: 可信域名配置是可选的,仅当应用需要 OAuth 网页授权时才必须设置。 Alertmanager 仅使用服务端 API 发送消息,不依赖网页授权,理论上可不配置可信域名。 但部分企业微信版本的管理后台可能强制要求,届时按上述步骤配置即可。

官方文档:

9.3 修改后重启

# 修改 alertmanager.yml 后重启生效
cd /opt/monitor
docker compose restart alertmanager

十、验证清单

10.1 云端服务

docker compose ps
curl -s http://localhost:9090/-/healthy
curl -s http://localhost:3100/ready
curl -s http://localhost:3000/api/health
curl -s http://localhost:9093/-/healthy

10.2 采集端

# Debian/Ubuntu
sudo systemctl status alloy
sudo journalctl -u alloy -n 30 --no-pager
curl -s http://localhost:12345/metrics | head -5
 
# Windows
sc.exe query "Grafana Alloy"

10.3 端到端

  1. 浏览器 https://monitor.yourdomain.com → Grafana 登录页 (密码1)
  2. Explore → Prometheus → 查 up → 看到所有实例
  3. Explore → Loki → 查 {job="auth"} → 看到安全日志
  4. 导入仪表盘: Node Exporter Full (ID: 1860), MySQL Overview (ID: 7362)
    • 方式1: Grafana UI → Dashboards → Import → 输入 ID
    • 方式2: 下载 JSON 放入 grafana/provisioning/dashboards/ 目录自动加载

十一、日常运维

# 云端重启
docker compose restart prometheus
docker compose restart loki
 
# 云端日志
docker compose logs -f --tail=100 prometheus
 
# 采集端重载
sudo systemctl reload alloy                     # Linux
sc.exe stop "Grafana Alloy" && sc.exe start "Grafana Alloy"  # Windows
 
# 配置语法检查
sudo alloy run /etc/alloy/config.alloy --dry-run