监控系统操作手册
本文档包含完整的部署说明和所有配置文件内容,只需阅读此文档即可完成全部部署操作。
一、架构概览
本地机房 云端 Debian 12
┌──────────────────────────────┐ ┌──────────────────────────────────────┐
│ Grafana Alloy (每台服务器) │ │ 现有 Nginx (TLS:443, 已有证书) │
│ │ │ │ /etc/nginx/conf.d/monitor.conf │
│ · 系统指标 (unix/win) │ │ │ │
│ · MySQL 指标 (Alloy内置采集) │ │ ├─► monitor-prometheus :9090 │
│ · JMX Exporter localhost抓取 │ │ │ prom/prometheus:v3.13.0 │
│ · 安全日志 (auth/event) │ │ │ │
│ · 应用日志 │ │ ├─► monitor-loki :3100 │
│ │ │ │ grafana/loki:3.7.2 │
│ │ │ │ │ │
│ │ HTTP │ │ ├─► monitor-grafana :3000 │
│ ▼ │ │ │ grafana/grafana:13.1.0 │
│ ┌──────────────────────┐ │ │ │ │
│ │ 本地 Nginx 中转 │ │ │ └─► monitor-alertmanager :9093 │
│ │ (Ubuntu 22, root │──── HTTPS ──▶│ prom/alertmanager:v0.33.0 │
│ │ Docker, :80, 无证书) │ │ │ (告警推送企业微信) │
│ └──────────────────────┘ │ └──────────────────────────────────────┘
└──────────────────────────────┘ (全通过 shared-proxy-network 互通)
关键设计:
- 采集端统一 full 配置: Debian/Windows 各一个 full 配置文件, 分块设计, 默认只启用系统指标 + 安全日志, MySQL/Java/Nginx 按需取消注释启用
- MySQL 指标: Grafana Alloy 内置
prometheus.exporter.mysql在 MySQL 本机 localhost 采集 - JMX Exporter: Java 服务的 JMX 指标由 JMX Exporter javaagent 暴露, Alloy 本地抓取
- Nginx 指标: 通过独立
nginx-prometheus-exporter采集 stub_status,Alloy 抓取 Prometheus 格式指标(Alloy 无内置 nginx exporter) - 本地 Nginx: Ubuntu 22 上 root Docker 部署,内网 HTTP (:80),统一代理所有 Alloy 推送到云端 HTTPS
- 云端 Nginx: 单一入口,同时接收推送 (
/api/v1/write,/loki/api/v1/push) 和对外提供 Grafana Web - Docker 用户: 云端 rootless (容器
0:0→ 宿主机100000),本地 root 模式 (无 remap)
二、密码体系说明
监控系统涉及 3 个独立的密码/认证:
| # | 用途 | 配置文件 | 使用者 |
|---|---|---|---|
| 1 | Grafana Web 登录密码 | docker-compose.yml → GF_SECURITY_ADMIN_PASSWORD | 人通过浏览器登录 Grafana |
| 2 | Prometheus admin 密码 | prometheus/web.yml → admin 用户 | Grafana 连接 Prometheus 数据源 (内部服务间) |
| 3 | Prometheus prometheus 密码 | prometheus/web.yml → prometheus 用户 | 各 Alloy 采集端 remote_write 推送指标 |
| 4 | Nginx basic auth 密码 | nginx/.htpasswd → loki 用户 | 仅 Loki push 入口(Loki 无内置认证,由 Nginx 提供) |
注意:Prometheus 指标推送(
/api/v1/write)不走 Nginx basic auth,由 Prometheus 自身的web.yml认证,Nginx 仅透明转发。
Grafana 浏览器登录 → GF_SECURITY_ADMIN_PASSWORD (密码1,初始 admin/xxx)
Grafana 查询指标 → Prometheus basic_auth admin:密码2
Alloy 推送指标 → Prometheus remote_write prometheus:密码3 → Nginx 透传至 Prometheus web.yml 验证
Alloy 推送日志 → Loki push loki:密码4 (同一密码) → Nginx auth_basic 验证后转发至 Loki
密码 3 和密码 4 可以设为相同值以简化管理,但认证层不同:Prometheus 是自己的
web.yml,Loki 是 Nginxauth_basic。密码 2 的特殊情况:由于 Prometheus basic auth 是”全有或全无”机制(无 localhost 豁免),Prometheus 自监控(scrape 自身
/metrics)也需要在prometheus.yml的 scrape job 中显式提供admin的明文密码。详情见 5.7 节。
三、组件清单与下载
3.1 Docker 镜像 (云端 Debian 12)
| 组件 | 镜像 | 用途 |
|---|---|---|
| Prometheus | prom/prometheus:v3.13.0 | 指标存储与告警 |
| Loki | grafana/loki:3.7.2 | 日志存储与告警 |
| Grafana | grafana/grafana:13.1.0 | 可视化仪表盘 |
| Alertmanager | prom/alertmanager:v0.33.0 | 告警路由 |
3.2 采集端组件 (各服务器)
| 组件 | 适用系统 | 下载地址 |
|---|---|---|
| Grafana Alloy v1.17.1 | Debian 12 / Ubuntu 22 | https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-linux-amd64.zip |
| Grafana Alloy v1.17.1 | Windows Server 2016+ | https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-installer-windows-amd64.exe |
| JMX Exporter 1.6.0 | Java 8+ 服务 | https://github.com/prometheus/jmx_exporter/releases/download/v1.6.0/jmx_prometheus_javaagent-1.6.0.jar |
| JMX Exporter 0.17.1 (Java 6) | Java 6/7 服务 | https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent_java6/0.17.1/jmx_prometheus_javaagent_java6-0.17.1.jar |
| Nginx Prometheus Exporter 1.5.1 | Nginx 指标采集 | nginx/nginx-prometheus-exporter:1.5.1 (Docker) / https://github.com/nginx/nginx-prometheus-exporter/releases |
版本说明: JMX Exporter 提供两个版本——1.6.0(当前最新,Java 8+)和 0.17.1 Java 6 构建(兼容 Java 6/7,Maven Central 分发)。两者规则配置语法兼容,下方
jmx_exporter.yaml同时适用两个版本。1.1.0 起 jar 仅通过 GitHub Releases 分发。Grafana Alloy 最低要求 Windows Server 2016。Nginx 指标采集说明: Grafana Alloy 没有内置
prometheus.exporter.nginx组件(与 Apache 的prometheus.exporter.apache不同),Nginx 指标需通过独立的nginx-prometheus-exporter采集,再由 Alloy 的prometheus.scrape抓取。
四、Docker 镜像导出 (离线部署)
在可联网机器上执行:
# 拉取
docker pull prom/prometheus:v3.13.0
docker pull grafana/loki:3.7.2
docker pull grafana/grafana:13.1.0
docker pull prom/alertmanager:v0.33.0
docker pull nginx:stable-alpine
docker pull nginx/nginx-prometheus-exporter:1.5.1
# 一键导出
docker save -o monitor-images.tar \
prom/prometheus:v3.13.0 \
grafana/loki:3.7.2 \
grafana/grafana:13.1.0 \
prom/alertmanager:v0.33.0 \
nginx:stable-alpine \
nginx/nginx-prometheus-exporter:1.5.1生产服务器导入:
docker load -i monitor-images.tar
docker images | grep -E "prom/prometheus|grafana/(loki|grafana)|prom/alertmanager|nginx"4.1 采集端离线下载
| 组件 | URL |
|---|---|
| Grafana Alloy v1.17.1 Linux amd64 | https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-linux-amd64.zip |
| Grafana Alloy v1.17.1 Windows 安装器 | https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-installer-windows-amd64.exe |
| JMX Exporter 1.6.0 | https://github.com/prometheus/jmx_exporter/releases/download/v1.6.0/jmx_prometheus_javaagent-1.6.0.jar |
| Nginx Prometheus Exporter 1.5.1 | https://github.com/nginx/nginx-prometheus-exporter/releases/tag/v1.5.1 |
五、云端服务器部署 (Debian 12)
5.1 目录结构
在云端服务器上创建以下目录结构:
mkdir -p /opt/monitor/{prometheus/{data,rules},loki/{data,rules},grafana/{data,provisioning/{datasources,dashboards}},alertmanager/data,nginx/verify}5.2 网络说明
monitor 服务通过 shared-proxy-network 与现有 Nginx 互通。Nginx 在 location 中直接用容器名代理(proxy_pass http://monitor-prometheus:9090),不是 127.0.0.1。
云端 Nginx 同时处理:
- 接收 Alloy 推送 —
/api/v1/write→ Prometheus,/loki/api/v1/push→ Loki - 对外 Web 访问 —
/→ Grafana UI
5.3 Rootless Docker 权限设置
UID 映射: 容器 0 → 宿主机 100000
# 确认映射关系
grep ^$(whoami) /etc/subuid
# 示例: dockeruser:100000:65536
# 创建数据目录并设置权限
chown -R 100000:100000 /opt/monitor/prometheus/data
chown -R 100000:100000 /opt/monitor/loki/data
chown -R 100000:100000 /opt/monitor/grafana/data
chown -R 100000:100000 /opt/monitor/alertmanager/data5.4 生成认证密码
apt update && apt install apache2-utils -y
# 生成 admin 用户的 bcrypt 哈希 (Grafana 连接 Prometheus 用)
htpasswd -nbB admin '<ADMIN_PASSWORD>' | cut -d: -f2
# 生成 prometheus 用户的 bcrypt 哈希 (Alloy remote_write 用)
htpasswd -nbB prometheus '<PROMETHEUS_PASSWORD>' | cut -d: -f25.5 预先替换占位符
在创建以下配置文件之前,先确定并记录所有需要替换的值:
.env 文件变量(docker-compose.yml 自动读取):
| 变量 | 说明 |
|---|---|
DOMAIN | 实际域名,如 monitor.yourdomain.com |
GRAFANA_PASSWORD | Grafana Web 登录密码 |
配置文件占位符:
| 占位符 | 说明 | 出现位置 |
|---|---|---|
<ADMIN_PASSWORD> | admin 用户明文密码 | grafana datasources |
<PROMETHEUS_PASSWORD> | prometheus 用户明文密码 | Alloy 配置 |
<BCRYPT_HASH_ADMIN> | admin 的 bcrypt 哈希 | prometheus/web.yml, alertmanager/web.yml |
<BCRYPT_HASH_PROMETHEUS> | prometheus 的 bcrypt 哈希 | prometheus/web.yml |
<YOUR_CORP_ID> | 企业微信企业 ID | alertmanager.yml |
<YOUR_API_SECRET> | 企业微信应用 Secret | alertmanager.yml |
<YOUR_AGENT_ID> | 企业微信应用 Agent ID | alertmanager.yml |
5.5.1 .env (环境变量)
文件路径: /opt/monitor/.env
docker-compose.yml 自动读取此文件,部署前替换为实际值:
DOMAIN=monitor.yourdomain.com
GRAFANA_PASSWORD=<你的 Grafana 登录密码>5.6 docker-compose.yml
文件路径: /opt/monitor/docker-compose.yml
services:
prometheus:
image: prom/prometheus:v3.13.0
container_name: monitor-prometheus
restart: unless-stopped
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
deploy:
resources:
limits:
cpus: "0.5"
memory: 512M
pids: 100
reservations:
memory: 256M
user: "0:0"
environment:
- TZ=Asia/Shanghai
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./prometheus/web.yml:/etc/prometheus/web.yml:ro
- ./prometheus/rules:/etc/prometheus/rules:ro
- ./prometheus/data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--storage.tsdb.retention.time=15d"
- "--web.enable-remote-write-receiver"
- "--web.config.file=/etc/prometheus/web.yml"
- "--web.enable-admin-api"
- "--web.external-url=https://${DOMAIN}/prometheus/"
- "--web.route-prefix=/prometheus/"
networks:
- monitor
- proxy
loki:
image: grafana/loki:3.7.2
container_name: monitor-loki
restart: unless-stopped
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
deploy:
resources:
limits:
cpus: "0.3"
memory: 256M
pids: 100
reservations:
memory: 128M
user: "0:0"
volumes:
- ./loki/local-config.yaml:/etc/loki/local-config.yaml:ro
- ./loki/rules:/etc/loki/rules:ro
- ./loki/data:/loki
command:
- "-config.file=/etc/loki/local-config.yaml"
- "-log.level=warn"
networks:
- monitor
- proxy
grafana:
image: grafana/grafana:13.1.0
container_name: monitor-grafana
restart: unless-stopped
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
environment:
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
- GF_SERVER_ROOT_URL=https://${DOMAIN}
- GF_USERS_ALLOW_SIGN_UP=false
- GF_AUTH_ANONYMOUS_ENABLED=false
- TZ=Asia/Shanghai
deploy:
resources:
limits:
cpus: "0.2"
memory: 256M
pids: 100
reservations:
memory: 128M
user: "0:0"
volumes:
- ./grafana/data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning:ro
networks:
- monitor
- proxy
alertmanager:
image: prom/alertmanager:v0.33.0
container_name: monitor-alertmanager
restart: unless-stopped
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
deploy:
resources:
limits:
cpus: "0.1"
memory: 64M
pids: 50
reservations:
memory: 32M
user: "0:0"
environment:
- TZ=Asia/Shanghai
volumes:
- /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime:ro
- ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
- ./alertmanager/web.yml:/etc/alertmanager/web.yml:ro
- ./alertmanager/data:/alertmanager
command:
- "--config.file=/etc/alertmanager/alertmanager.yml"
- "--storage.path=/alertmanager"
- "--web.config.file=/etc/alertmanager/web.yml"
- "--web.external-url=https://${DOMAIN}/alertmanager/"
- "--log.level=debug"
networks:
- monitor
- proxy
networks:
monitor:
driver: bridge
proxy:
name: shared-proxy-network
external: true5.7 prometheus/prometheus.yml
文件路径: /opt/monitor/prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- path_prefix: /alertmanager/
basic_auth:
username: admin
password: <YOUR_ADMIN_PASSWORD>
static_configs:
- targets:
- alertmanager:9093
rule_files:
- "/etc/prometheus/rules/*.yml"
scrape_configs:
# =========================================================================
# Prometheus 自监控
#
# 由于同时启用了:
# 1. --web.route-prefix=/prometheus/ → /metrics 被重写为 /prometheus/metrics
# 2. --web.config.file=web.yml → basic auth 全有或全无,无 localhost 豁免
# 自抓取必须显式指定 metrics_path 和 basic_auth。
#
# basic_auth 中的 password 必须是明文,不能用 web.yml 中的 bcrypt hash。
# 官方讨论见:https://groups.google.com/g/prometheus-users/c/3Rn-2bAbusI
# =========================================================================
- job_name: "prometheus"
metrics_path: "/prometheus/metrics"
basic_auth:
username: admin
password: <ADMIN_PASSWORD> # web.yml 中 admin 对应的明文密码,不是 bcrypt hash
static_configs:
- targets: ["localhost:9090"]
labels:
instance: "monitor-prometheus"5.8 prometheus/web.yml
文件路径: /opt/monitor/prometheus/web.yml
# ==========================================
# Prometheus HTTP basic_auth 用户
#
# admin: Grafana 连接 Prometheus 数据源时使用 (内部服务间认证)
# prometheus: Alloy 采集端 remote_write 推送时使用 (跨网络认证)
# ==========================================
basic_auth_users:
admin: <BCRYPT_HASH_ADMIN>
prometheus: <BCRYPT_HASH_PROMETHEUS>5.9 prometheus/rules/alerts.yml
文件路径: /opt/monitor/prometheus/rules/alerts.yml
groups:
- name: system
rules:
- alert: HighCPUUsage
expr: |
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "CPU 使用率超过 80%"
description: '当前值 {{ printf "%.1f" $value }}%'
- alert: HighCPUUsageCritical
expr: |
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 95
for: 2m
labels:
severity: critical
annotations:
summary: "CPU 使用率超过 95%"
description: '当前值 {{ printf "%.1f" $value }}%'
- alert: HighMemoryUsage
expr: |
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
for: 5m
labels:
severity: warning
annotations:
summary: "内存使用率超过 90%"
description: '当前值 {{ printf "%.1f" $value }}%'
- alert: DiskFull
expr: |
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
for: 5m
labels:
severity: critical
annotations:
summary: "磁盘剩余空间不足 10%"
description: '剩余空间 {{ printf "%.1f" $value }}%'
- alert: InstanceDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "实例已下线"
description: "服务不可达,请排查"
- alert: HighDiskIO
expr: |
rate(node_disk_io_time_seconds_total[5m]) * 100 > 80
for: 5m
labels:
severity: warning
annotations:
summary: "磁盘 IO 使用率过高"
description: '当前值 {{ printf "%.1f" $value }}%'
- name: windows_system
rules:
# https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.cpu.md
- alert: WindowsHighCPUUsage
expr: |
100 - (avg by(instance) (irate(windows_cpu_time_total{mode="idle"}[2m])) * 100) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "CPU 使用率超过 80%"
description: '当前值 {{ printf "%.1f" $value }}%'
- alert: WindowsHighCPUUsageCritical
expr: |
100 - (avg by(instance) (irate(windows_cpu_time_total{mode="idle"}[2m])) * 100) > 95
for: 2m
labels:
severity: critical
annotations:
summary: "CPU 使用率超过 95%"
description: '当前值 {{ printf "%.1f" $value }}%'
# https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.memory.md
- alert: WindowsHighMemoryUsage
expr: |
100 - 100 * (windows_memory_physical_free_bytes / windows_memory_physical_total_bytes) > 96
for: 5m
labels:
severity: warning
annotations:
summary: "内存使用率超过 96%"
description: '当前值 {{ printf "%.1f" $value }}%'
# https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.logical_disk.md
- alert: WindowsDiskFull
expr: |
100 - 100 * (windows_logical_disk_free_bytes / windows_logical_disk_size_bytes) > 90
and windows_logical_disk_size_bytes > 0
for: 5m
labels:
severity: critical
annotations:
summary: "磁盘剩余空间不足 10% ({{ $labels.volume }})"
description: '当前使用率 {{ printf "%.1f" $value }}%'
# 官方 Disk Activity 公式
- alert: WindowsHighDiskIO
expr: |
(
rate(windows_logical_disk_read_seconds_total[2m])
+ rate(windows_logical_disk_write_seconds_total[2m])
)
/
(
rate(windows_logical_disk_read_seconds_total[2m])
+ rate(windows_logical_disk_write_seconds_total[2m])
+ rate(windows_logical_disk_idle_seconds_total[2m])
) > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "磁盘 IO 使用率超过 80% ({{ $labels.volume }})"
description: '当前值 {{ printf "%.1f" $value }}'5.10 loki/local-config.yaml
文件路径: /opt/monitor/loki/local-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
grpc_server_max_recv_msg_size: 104857600
grpc_server_max_send_msg_size: 104857600
http_server_read_timeout: 60s
http_server_write_timeout: 60s
common:
instance_addr: 127.0.0.1
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules-temp
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2024-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
storage_config:
tsdb_shipper:
active_index_directory: /loki/tsdb-index
cache_location: /loki/tsdb-cache
filesystem:
directory: /loki/chunks
compactor:
working_directory: /loki/compactor
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
delete_request_store: filesystem
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
retention_period: 168h
max_entries_limit_per_query: 10000
split_queries_by_interval: 1h
ingestion_rate_mb: 8
ingestion_burst_size_mb: 16
per_stream_rate_limit: 32M
per_stream_rate_limit_burst: 64M
max_cache_freshness_per_query: "10m"
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 50
ruler:
storage:
type: local
local:
directory: /etc/loki/rules
rule_path: /loki/rules-temp
alertmanager_url: http://alertmanager:9093/alertmanager
alertmanager_client:
basic_auth_username: admin
basic_auth_password: <YOUR_ADMIN_PASSWORD>
ring:
kvstore:
store: inmemory
enable_api: true
analytics:
reporting_enabled: false5.11 loki/rules/fake/security.yml
文件路径: /opt/monitor/loki/rules/fake/security.yml
注意: Loki 单租户模式(
auth_enabled: false)下,默认租户 ID 为fake。规则文件必须放在<ruler_storage_directory>/fake/子目录下,否则规则不会被加载。
groups:
- name: security
rules:
- alert: SSHBruteForce
expr: |
sum(rate({job="auth"} |~ "Failed password" [5m])) by (instance) > 10
for: 5m
labels:
severity: critical
annotations:
summary: "SSH 暴力破解检测"
description: "过去 5 分钟内 SSH 登录失败超过 10 次"
- alert: SSHRootLogin
expr: |
sum(rate({job="auth"} |~ "Accepted .* for root" [5m])) by (instance) > 0
for: 1m
labels:
severity: warning
annotations:
summary: "检测到 root 直接 SSH 登录"
description: "root 用户通过 SSH 直接登录系统,建议使用普通用户并 sudo"
- alert: SSHLoginSuccess
expr: |
sum by (host, group, job, login_user, source_ip, geoip_country_name, geoip_city_name) (
count_over_time({job="auth", login_user!=""} [1m])
) > 0
for: 0m
labels:
severity: login
annotations:
summary: "SSH 登录成功"
description: "用户 {{ $labels.login_user }} 从 {{ $labels.source_ip }} ({{ $labels.geoip_country_name }} {{ $labels.geoip_city_name }}) 登录"
- alert: WindowsLoginFailure
expr: |
sum(rate({job="windows_security", event_id="4625"} [5m])) by (host) > 20
for: 5m
labels:
severity: warning
annotations:
summary: "Windows 登录失败过多"
description: "过去 5 分钟内 EventID 4625 超过 20 次"
- alert: WindowsLoginSuccess
expr: |
sum by (host, group, job, login_user, source_ip, logon_type, logon_type_name, geoip_country_name, geoip_city_name) (
count_over_time({job="windows_security", event_id="4624", login_user!=""} [1m])
) > 0
for: 0m
labels:
severity: login
annotations:
summary: "Windows 登录成功"
description: "用户 {{ $labels.login_user }} 从 {{ $labels.source_ip }} 登录 ({{ $labels.geoip_country_name }} {{ $labels.geoip_city_name }})"
- alert: WindowsLoginSuccessAnomaly
expr: |
sum(rate({job="windows_security", event_id="4624"} [5m])) by (host) > 50
for: 5m
labels:
severity: warning
annotations:
summary: "Windows 登录次数异常增多"
description: "过去 5 分钟内 EventID 4624 超过 50 次"5.12 alertmanager/alertmanager.yml
文件路径: /opt/monitor/alertmanager/alertmanager.yml
# ==========================================
# Alertmanager 配置
# 告警通过企业微信应用消息推送
#
# 前置条件 (在企业微信管理后台获取):
# corp_id: 企业 ID → 管理后台 → "我的企业" → "企业信息"
# agent_id: 应用 AgentID → 管理后台 → "应用管理" → "应用" → 点进自建应用
# api_secret: 应用 Secret → 同上页面, Secret 字段
# to_user: 接收人账号 → 管理后台 → "通讯录" → 点成员 → "账号"字段, 多个用 | 分隔
#
# 官方文档:
# 基本概念: https://developer.work.weixin.qq.com/document/path/90665
# 创建自建应用: https://developer.work.weixin.qq.com/document/path/90487
# 管理后台: https://work.weixin.qq.com/
# ==========================================
global:
resolve_timeout: 2m
wechat_api_url: "https://qyapi.weixin.qq.com/cgi-bin/"
wechat_api_corp_id: "<YOUR_CORP_ID>"
wechat_api_secret: "<YOUR_API_SECRET>"
route:
receiver: "wechat-warning"
group_by: ["alertname", "severity", "instance", "job", "group"]
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers:
- severity="critical"
receiver: "wechat-critical"
group_wait: 10s
- matchers:
- severity="warning"
receiver: "wechat-warning"
- matchers:
- severity="login"
receiver: "wechat-login"
group_by: ["alertname", "group", "host", "job", "login_user", "source_ip"]
group_wait: 5s
group_interval: 1s
repeat_interval: 5m
- matchers:
- severity="info"
receiver: "wechat-info"
receivers:
- name: "wechat-critical"
wechat_configs:
- send_resolved: true
agent_id: "<YOUR_AGENT_ID>"
to_user: "<YOUR_USER_ID>"
message_type: "markdown"
message: |
{{ range .Alerts }}
{{ if eq .Status "resolved" }}
## <font color="info">[RESOLVED] {{ .Labels.alertname }} 已恢复</font>
{{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
{{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
{{ end }}> 分组: {{ .Labels.group }}
> 主机: {{ .Labels.host }}
> 任务: {{ .Labels.job }}
> 时间: {{ .EndsAt.Local.Format "2006-01-02 15:04:05" }}
{{ else }}
## <font color="warning">[CRITICAL] {{ .Annotations.summary }}</font>
{{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
{{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
{{ end }}> 分组: {{ .Labels.group }}
> 主机: {{ .Labels.host }}
> 任务: {{ .Labels.job }}
> 描述: {{ .Annotations.description }}
> 时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
{{ end }}
{{ end }}
- name: "wechat-warning"
wechat_configs:
- send_resolved: true
agent_id: "<YOUR_AGENT_ID>"
to_user: "<YOUR_USER_ID>"
message_type: "markdown"
message: |
{{ range .Alerts }}
{{ if eq .Status "resolved" }}
## <font color="info">[RESOLVED] {{ .Labels.alertname }} 已恢复</font>
{{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
{{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
{{ end }}> 分组: {{ .Labels.group }}
> 主机: {{ .Labels.host }}
> 任务: {{ .Labels.job }}
> 时间: {{ .EndsAt.Local.Format "2006-01-02 15:04:05" }}
{{ else }}
## <font color="warning">[WARNING] {{ .Annotations.summary }}</font>
{{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
{{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
{{ end }}> 分组: {{ .Labels.group }}
> 主机: {{ .Labels.host }}
> 任务: {{ .Labels.job }}
> 描述: {{ .Annotations.description }}
> 时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
{{ end }}
{{ end }}
- name: "wechat-info"
wechat_configs:
- send_resolved: true
agent_id: "<YOUR_AGENT_ID>"
to_user: "<YOUR_USER_ID>"
message_type: "markdown"
message: |
{{ range .Alerts }}
{{ if eq .Status "resolved" }}
## <font color="info">[RESOLVED] {{ .Labels.alertname }} 已恢复</font>
{{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
{{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
{{ end }}> 分组: {{ .Labels.group }}
> 主机: {{ .Labels.host }}
> 任务: {{ .Labels.job }}
> 时间: {{ .EndsAt.Local.Format "2006-01-02 15:04:05" }}
{{ else }}
## <font color="info">[INFO] {{ .Annotations.summary }}</font>
{{ if .Labels.instance }}> 实例: {{ .Labels.instance }}
{{ end }}{{ if .Labels.env }}> 环境: {{ .Labels.env }}
{{ end }}> 分组: {{ .Labels.group }}
> 主机: {{ .Labels.host }}
> 任务: {{ .Labels.job }}
> 描述: {{ .Annotations.description }}
> 时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
{{ end }}
{{ end }}
- name: "wechat-login"
wechat_configs:
- send_resolved: false
agent_id: "<YOUR_AGENT_ID>"
to_user: "<YOUR_USER_ID>"
message_type: "markdown"
message: |
{{ range .Alerts }}
## <font color="warning">[LOGIN] {{ .Annotations.summary }}</font>
> 分组: {{ .Labels.group }}
> 主机: {{ .Labels.host }}
> 任务: {{ .Labels.job }}
> 账号: {{ .Labels.login_user }}
> 来源IP: {{ .Labels.source_ip }}
{{ if .Labels.geoip_country_name }}> 位置: {{ .Labels.geoip_country_name }} {{ .Labels.geoip_city_name }}
{{ end }}{{ if .Labels.logon_type }}> 登录类型: {{ .Labels.logon_type_name }} (ID:{{ .Labels.logon_type }})
{{ end }}> 时间: {{ .StartsAt.Local.Format "2006-01-02 15:04:05" }}
{{ end }}5.12.1 alertmanager/web.yml
文件路径: /opt/monitor/alertmanager/web.yml
Alertmanager 的 HTTP basic_auth 配置,与 Prometheus web.yml 共享同一套密码体系(bcrypt 哈希)。
basic_auth_users:
admin: <BCRYPT_HASH>访问 Alertmanager Web UI: 通过
https://yourdomain.com/alertmanager/访问,认证由 Alertmanager 自身的 basic auth 处理(admin用户)。
5.13 grafana/provisioning/datasources/datasources.yml
文件路径: /opt/monitor/grafana/provisioning/datasources/datasources.yml
apiVersion: 1
# ==========================================
# 数据源配置
#
# Prometheus 中 basicAuthUser: admin 对应 prometheus/web.yml 中的 admin 用户
# basicAuthPassword 是 admin 的明文密码 (非 bcrypt 哈希)
#
# Grafana 自身的 Web 登录密码由 docker-compose.yml 中 GF_SECURITY_ADMIN_PASSWORD 设置
# ==========================================
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
basicAuth: true
basicAuthUser: admin
secureJsonData:
basicAuthPassword: "<ADMIN_PASSWORD>"
isDefault: true
editable: false
- name: Loki
type: loki
access: proxy
url: http://loki:3100
editable: false5.14 grafana/provisioning/dashboards/dashboards.yml
文件路径: /opt/monitor/grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
- name: "default"
orgId: 1
folder: ""
type: file
disableDeletion: false
updateIntervalSeconds: 60
options:
# 将 dashboard JSON 文件放入此目录后自动加载
# 推荐导入: Node Exporter Full (ID: 1860), MySQL Overview (ID: 7362)
# 下载 JSON: https://grafana.com/grafana/dashboards/<ID>
path: /etc/grafana/provisioning/dashboards5.15 nginx/nginx.conf (云端 Nginx 反代)
文件路径: /opt/monitor/nginx/nginx.conf
此文件需 include 到现有云端 Nginx 的配置中。该文件包含两层入口设计:
- 443 (TLS) — 公网入口,
server_name为公网域名,供外部 Alloy / 浏览器访问 - 80 (HTTP) — 内网入口,
server_name为monitor-proxy.local,供云端同一 VPC 内的 Alloy 使用,避免流量出公网回环
TLS 证书由现有 Nginx 全局配置提供,内网入口无需证书。
# ==========================================
# 监控 Nginx 配置 (include 到现有云端 Nginx)
# 路径: /etc/nginx/conf.d/monitor.conf
# 前提: 现有 Nginx 已处理 TLS 证书,此文件只负责路由
# 前提: Nginx 容器需加入 shared-proxy-network 才能解析容器名
#
# 两层入口设计:
# 443 (TLS) — 公网入口,server_name 为公网域名
# 80 — 内网入口,server_name 为 monitor-proxy.local
# 供云端同一 VPC 内的 Alloy 使用,避免流量出公网回环
# ==========================================
# 使用 Docker 内置 DNS 进行运行时解析,避免启动时因上游容器未就绪而失败
resolver 127.0.0.11 valid=300s ipv6=off;
# ==========================================
# 公网入口 (TLS) — 外部 Alloy / 浏览器访问
# ==========================================
server {
listen 443 ssl;
server_name monitor.yourdomain.com;
# 全局代理配置,所有 location 自动继承
client_body_buffer_size 10M;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
# Prometheus remote_write 入口 (Alloy 推送指标)
# Prometheus Web UI (调试用,需配合 --web.external-url 和 --web.route-prefix)
set $prometheus_upstream "monitor-prometheus:9090";
# 注意:Prometheus 设置了 --web.route-prefix=/prometheus/,
# 所以 /api/v1/write 的实际内部路由为 /prometheus/api/v1/write,
# proxy_pass 尾部带路径来补上 /prometheus/ 前缀
location /api/v1/write {
proxy_pass http://$prometheus_upstream/prometheus/api/v1/write;
client_max_body_size 10M;
}
location /prometheus/ {
proxy_pass http://$prometheus_upstream;
}
# Loki 日志 push 入口 (Alloy 推送日志)
# Loki 自身不支持内置 basic auth,由 Nginx auth_basic 提供认证
# .htpasswd 需包含 loki 用户 (可与 prometheus 用户共用同一个密码)
set $loki_upstream "monitor-loki:3100";
location /loki/api/v1/push {
auth_basic "Loki";
auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
proxy_pass http://$loki_upstream;
client_max_body_size 20M;
}
# Loki API (调试用)
# 注意:Loki 自身不支持内置 basic auth,需要通过 Nginx auth_basic 提供认证
# 注意:proxy_pass 使用变量时不做 URI 替换,需用 rewrite 去掉 /loki/ 前缀
location /loki/ {
auth_basic "Loki";
auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
rewrite ^/loki/(.*) /$1 break;
proxy_pass http://$loki_upstream;
}
# Alertmanager Web UI (查看活跃告警)
set $alertmanager_upstream "monitor-alertmanager:9093";
location /alertmanager/ {
proxy_pass http://$alertmanager_upstream;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# 企业微信可信域名验证文件 (WW_verify_*.txt)
# 将验证文件放到 verify/ 目录并挂载到容器
location /WW_verify_ {
root /etc/nginx/conf.d/verify;
}
# Grafana Web UI
set $grafana_upstream "monitor-grafana:3000";
location / {
proxy_pass http://$grafana_upstream;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
# ==========================================
# 内网入口 (HTTP) — 供云端 VPC 内 Alloy 推送使用
# 通过 monitor-proxy.local 访问,走内网不出公网
# DNS 解析方式:
# - Linux: /etc/hosts 添加 "<内网IP> monitor-proxy.local"
# - Windows: C:\Windows\System32\drivers\etc\hosts 相同操作
# - 云服务商内网 DNS(如阿里云 PrivateZone 等,可能收费)
# ==========================================
server {
listen 80;
server_name monitor-proxy.local;
# 只允许内网 IP 访问
allow 127.0.0.1;
allow 10.0.0.0/8;
allow 172.16.0.0/12;
allow 192.168.0.0/16;
deny all;
client_body_buffer_size 10M;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
set $prometheus_upstream "monitor-prometheus:9090";
set $loki_upstream "monitor-loki:3100";
# Prometheus remote_write 入口 (Alloy 推送指标)
location /api/v1/write {
proxy_pass http://$prometheus_upstream/prometheus/api/v1/write;
client_max_body_size 10M;
}
# Loki 日志 push 入口 (Alloy 推送日志)
# 内网环境可按需启用 basic_auth:
# auth_basic "Loki";
# auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
location /loki/api/v1/push {
proxy_pass http://$loki_upstream;
client_max_body_size 20M;
}
# Nginx 自身指标 (按需取消注释)
# 前提: Nginx 已开启 stub_status 模块
# location /nginx_status {
# stub_status on;
# access_log off;
# }
}挂载到现有 Nginx:
# 方法1: 在现有 Nginx 的 docker-compose 中挂载 (推荐)
# volumes:
# - /opt/monitor/nginx/nginx.conf:/etc/nginx/conf.d/monitor.conf:ro
# - /opt/monitor/nginx/verify:/etc/nginx/conf.d/verify:ro
# 方法2: 在现有 Nginx http{} 块中 include
# include /etc/nginx/conf.d/monitor.conf;注意:
nginx.conf中定义了server { listen 443 ssl; ... }块。如果现有 Nginx 已有监听 443 的 server,需要将 location 块合并到现有 server 中,而非整体 include。内网 server 块(listen 80)可直接 include,互不冲突。
Nginx 容器需加入 shared-proxy-network 才能解析 monitor-prometheus 等容器名。resolver 127.0.0.11 使用 Docker 内置 DNS 做运行时解析,配合 set + 变量 proxy_pass 避免 nginx 启动时因上游容器未就绪而报 host not found in upstream。
访问 Prometheus Web UI: 部署后通过
https://yourdomain.com/prometheus/访问。认证由 Prometheus 自身的web.ymlbasic auth 处理(admin用户),无需在 Nginx 额外配置。Prometheus 需要知道自己在子路径下,已在docker-compose.yml中配置--web.external-url和--web.route-prefix。
访问 Loki API (调试用): 通过
https://yourdomain.com/loki/访问。Loki 自身不支持内置 basic auth,由 Nginxauth_basic提供认证。需要创建.htpasswd文件:# 生成 Loki 推送专用用户 (Alloy 推送日志认证) htpasswd -c /opt/monitor/nginx/.htpasswd loki # 如需其他用户 (如 Loki Web UI 访问): htpasswd /opt/monitor/nginx/.htpasswd admin # 在现有 Nginx 的 docker-compose 中挂载: # - /opt/monitor/nginx/.htpasswd:/etc/nginx/conf.d/.htpasswd:ro nginx -s reload注意:
.htpasswd中的loki用户用于 Alloy 推送日志(Nginxauth_basic验证),与 Prometheusweb.yml的prometheus用户是两个独立的认证层。密码可以相同。
内网入口说明: 新增的
server { listen 80; server_name monitor-proxy.local; }块专供云端同一 VPC 内 Alloy 使用,配置思路参考第 6.1 节nginx-local.conf(同样监听 80 端口 +monitor-proxy.local)。区别在于 6.1 的本地 Nginx 将请求转发到公网云端,而此处的云端 Nginx 内网入口直接转发到本地的 Docker 后端服务。VPC 内 Alloy 的 endpoint 应设为http://monitor-proxy.local/api/v1/write和http://monitor-proxy.local/loki/api/v1/push,monitor-proxy.local通过/etc/hosts或内网 DNS 解析到云端 Nginx 的内网 IP。
5.16 启动 monitor 服务
cd /opt/monitor
docker load -i monitor-images.tar
docker compose up -d
docker compose ps六、本地 Nginx 中转 (Ubuntu 22, root Docker)
6.1 nginx/nginx-local.conf
文件路径: /opt/monitor/nginx/nginx-local.conf
# ==========================================
# 本地 Nginx 中转配置 (数据中心侧)
# 路径: /etc/nginx/sites-available/monitor-proxy
# 作用: 汇总本地所有 Alloy 的推送,统一转发到云端
# 证书: 不需要,纯内网 HTTP → 云端 HTTPS
# ==========================================
server {
listen 80;
server_name monitor-proxy.local;
# 只允许内网 IP 访问
allow 127.0.0.1;
allow 10.0.0.0/8;
allow 172.16.0.0/12;
allow 192.168.0.0/16;
deny all;
client_max_body_size 20M;
client_body_buffer_size 10M;
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_ssl_server_name on;
proxy_ssl_protocols TLSv1.2 TLSv1.3;
# 转发 Alloy remote_write 到云端 Prometheus
location /api/v1/write {
proxy_pass https://monitor.yourdomain.com;
}
# 转发 Alloy log push 到云端 Loki
location /loki/api/v1/push {
proxy_pass https://monitor.yourdomain.com;
}
# Nginx 自身指标 (按需取消注释)
# 前提: Nginx 已开启 stub_status 模块
# location /nginx_status {
# stub_status on;
# access_log off;
# }
}6.2 nginx/docker-compose-nginx-local.yml
文件路径: /opt/monitor/nginx/docker-compose-nginx-local.yml
# ==========================================
# 本地 Nginx 中转代理 (Ubuntu 22, root Docker)
# 用途: 汇总本地所有 Alloy 推送,统一转发到云端
# 部署: docker compose -f docker-compose-nginx-local.yml up -d
# ==========================================
services:
nginx-proxy:
image: nginx:stable-alpine
container_name: monitor-proxy-nginx
restart: unless-stopped
ports:
- "80:80"
volumes:
- ./nginx-local.conf:/etc/nginx/conf.d/default.conf:ro
- ./nginx-logs:/var/log/nginx
cap_drop:
- ALL
cap_add:
- NET_BIND_SERVICE
- CHOWN
- SETGID
- SETUID
security_opt:
- no-new-privileges:true
# ==========================================
# 权限说明:
# Root Docker 模式,无 userns-remap
# nginx:stable-alpine 容器内以 nginx 用户 (UID 101) 运行
# 日志目录需可写:
# mkdir -p nginx-logs && chown 101:101 nginx-logs
# 配置文件只需要可读 (默认 644 即可)
# ==========================================6.3 部署
# 在 monitor/nginx/ 目录
mkdir -p nginx-logs
chown 101:101 nginx-logs # nginx:stable-alpine 容器内 UID 101
# 修改 nginx-local.conf 中的云端域名
# monitor.yourdomain.com → 实际云端域名
docker compose -f docker-compose-nginx-local.yml up -d6.4 是否需要证书?
不需要。本地 Nginx 只监听内网 HTTP (:80),Alloy 通过 HTTP 推送。本地 Nginx 作为 HTTPS 客户端代理到云端(云端已有 TLS 证书)。
6.5 Alloy 端点选择
所有 Alloy 配置中两个端点 URL 需填写,loki.write 的 basic_auth 中 username 为 loki(对应云端 Nginx .htpasswd 中的 loki 用户):
| 场景 | 端点 |
|---|---|
| 有本地 Nginx(数据中心) | http://<本地Nginx_IP>/api/v1/write 和 /loki/api/v1/push |
| 云端 VPC 内(同 VPC 不同机器) | http://monitor-proxy.local/api/v1/write 和 /loki/api/v1/push |
| 直连云端(公网) | https://monitor.yourdomain.com/api/v1/write 和 /loki/api/v1/push |
6.6 日志轮转
Nginx 自身不内置 log rotate,需配合系统工具做日志轮转。以下提供 Linux 和 Windows 两种方案。
Linux(logrotate)
适用于本地 Nginx 中转(mount nginx-logs:/var/log/nginx)和云端 Nginx:
# /etc/logrotate.d/monitor-nginx
/opt/monitor/nginx/nginx-logs/*.log {
daily
rotate 7
maxsize 20M
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
docker exec monitor-proxy-nginx nginx -s reopen 2>/dev/null || true
endscript
}云端 Nginx 若不在此 compose,将
docker exec替换为对应的容器名。daily保证每天至少 rotate 一次(即使未达到maxsize,避免error.log长期不转)。
验证:
logrotate -d /etc/logrotate.d/monitor-nginx # dry-run
logrotate -f /etc/logrotate.d/monitor-nginx # 手动执行一次Windows(bat + 任务计划)
适用于 Windows 上部署的 Nginx。注意:Windows nginx 不支持 Unix 信号,但 nginx -s reopen 命令行参数可正常工作,关闭旧 access.log 的 handle 并创建新文件。
@echo off
set nginx_dir=<nginx_dir>
set log_dir=%nginx_dir%\logs
set current_date=%date:~0,4date:~8,2%
:: rename 日志 → nginx 仍往旧 handle 写
move %log_dir%\access.log %log_dir%\access_%current_date%.log
move %log_dir%\error.log %log_dir%\error_%current_date%.log
:: reopen → nginx 创建新 access.log / error.log,切换 handle
call %nginx_dir%\nginx -p %nginx_dir%\ -s reopen
:: 清理 90 天前的归档
forfiles /p %log_dir% /m access_*.log -d -90 /c "cmd /c del /f @path" 2>nul
forfiles /p %log_dir% /m error_*.log -d -90 /c "cmd /c del /f @path" 2>nul
@echo on注意:
%date%格式依赖系统区域设置(中文 Windows 正常,英文系统需调整截取位置)。
创建计划任务(每天 23:59:56 执行):
schtasks /create /tn "NginxLogRotate" /tr "C:\scripts\rotate-nginx-log.bat" /sc daily /st 23:59PowerShell 版更健壮(
Get-Date不依赖区域设置),如有需要可询问。
七、采集端部署 (各被监控服务器)
7.1 Debian 12 / Ubuntu 22
安装 Grafana Alloy
# 离线安装 (指定版本 v1.17.1)
wget https://github.com/grafana/alloy/releases/download/v1.17.1/alloy-linux-amd64.zip
unzip alloy-linux-amd64.zip
sudo cp alloy-linux-amd64 /usr/local/bin/alloy
sudo chmod +x /usr/local/bin/alloy
# 创建 systemd service
sudo tee /etc/systemd/system/alloy.service << 'EOF'
[Unit]
Description=Grafana Alloy
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/usr/local/bin/alloy run /etc/alloy/config.alloy
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
EOF
sudo mkdir -p /etc/alloy配置文件: config-debian-full.alloy
文件路径: monitor/alloy/config-debian-full.alloy (仓库中) → /etc/alloy/config.alloy (目标服务器)
标签设计
Alloy 将数据 PUSH 到云端,标签完全在采集端处理:
- 全局标签 (
external_labels):host(自动取主机名) +group(环境标识) — 由prometheus.remote_write和loki.write统一添加 - 按源标签 (
job、app、instance): 在各自采集模块中定义 — 独立prometheus.scrape设置job_name,日志在loki.source.filetargets 中内联
| 数据源 | host | group | job | app/instance |
|---|---|---|---|---|
| Node 指标 | hostname | local/cloud | node | — |
| Systemd 指标 | hostname | local/cloud | systemd | — |
| MySQL 指标 | hostname | local/cloud | mysql | instance= |
| Java JMX 指标 | hostname | local/cloud | java | instance= |
| Nginx 指标 | hostname | local/cloud | nginx | instance= |
| 系统日志 | hostname | local/cloud | auth/syslog/kernel | — |
| Java 应用日志 | hostname | local/cloud | java | instance= |
| MySQL 日志 | hostname | local/cloud | mysql | instance= |
| Nginx 日志 | hostname | local/cloud | nginx | instance= |
部署变量
通过 declare "cfg" 定义可复用配置,改顶部 5 个 cfg 块即可,下方自动引用:
declare "cfg" {
argument "value" { }
export "value" { value = argument.value.value }
}
cfg "endpoint_prom" { value = "https://monitor.yourdomain.com/api/v1/write" }
cfg "endpoint_loki" { value = "https://monitor.yourdomain.com/loki/api/v1/push" }
cfg "password_prom" { value = "<PROM_PASSWORD>" }
cfg "password_loki" { value = "<LOKI_PASSWORD>" }
cfg "group" { value = "local" }
cfg "hostname" { value = constants.hostname } // 或改为自定义标识| 配置项 | 说明 |
|---|---|
endpoint_prom / endpoint_loki | 修改域名,或用本地 Nginx / 云端 VPC 内网中转地址 |
password_prom | 对应 prometheus/web.yml 中 prometheus 用户的明文密码 |
password_loki | 对应云端 Nginx .htpasswd 中 loki 用户的明文密码 |
group | 环境标识: local / cloud / test / prod |
host标签使用constants.hostname自动取主机名,如需自定义改为字符串。
完整配置
// ==========================================
// Grafana Alloy - Debian 12 / Ubuntu 22 完整配置
// ==========================================
// 安装: 离线下载 alloy-linux-amd64.zip (见手册四章)
// 配置路径: /etc/alloy/config.alloy
// 启动: sudo systemctl enable --now alloy
//
// ==========================================
// 部署变量 — 修改 cfg 块即可, 下方自动引用
// ==========================================
// 有本地 Nginx 中转 → http://monitor-proxy.local/...
// 云端 VPC 内直连 → http://monitor-proxy.local/...
// (需将 monitor-proxy.local 通过内网 DNS 或 hosts 指向云端 Nginx 内网 IP)
// 云端公网直连 → https://monitor.yourdomain.com/...
declare "cfg" {
argument "value" { }
export "value" { value = argument.value.value }
}
cfg "endpoint_prom" { value = "https://monitor.yourdomain.com/api/v1/write" }
cfg "endpoint_loki" { value = "https://monitor.yourdomain.com/loki/api/v1/push" }
cfg "password_prom" { value = "<PROM_PASSWORD>" }
cfg "password_loki" { value = "<LOKI_PASSWORD>" }
cfg "group" { value = "local" }
cfg "hostname" { value = constants.hostname } // 或改为自定义标识
// ==========================================
// [启用] 系统指标 (替代 node_exporter)
// ==========================================
prometheus.exporter.unix "node" {
disable_collectors = ["ipvs", "btrfs", "infiniband", "xfs", "zfs"]
filesystem {
fs_types_exclude = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
mount_points_exclude = "^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)"
}
netdev {
device_exclude = "^(veth.*|cali.*|[a-f0-9]{15})$"
}
}
discovery.relabel "node" {
targets = prometheus.exporter.unix.node.targets
rule {
target_label = "instance"
replacement = cfg.hostname.value
}
rule {
target_label = "job"
replacement = "node"
}
}
prometheus.scrape "node" {
scrape_interval = "15s"
targets = discovery.relabel.node.output
forward_to = [prometheus.remote_write.default.receiver]
}
// ==========================================
// [启用] Systemd 服务状态
// ==========================================
prometheus.exporter.unix "systemd" {
set_collectors = ["systemd"]
systemd {
unit_include = ".+"
unit_exclude = ".+\\.(automount|device|mount|scope|slice)"
enable_restarts = true
start_time = true
}
}
discovery.relabel "systemd" {
targets = prometheus.exporter.unix.systemd.targets
rule {
target_label = "instance"
replacement = cfg.hostname.value
}
rule {
target_label = "job"
replacement = "systemd"
}
}
prometheus.scrape "systemd" {
scrape_interval = "15s"
targets = discovery.relabel.systemd.output
forward_to = [prometheus.remote_write.default.receiver]
}
// ==========================================
// [启用] 安全日志 (auth.log)
// ==========================================
// syslog 和 journald 输出格式一致,共用一套处理器
// 有 auth.log 用 file 源,无 rsyslog 时改用 journal 源
loki.source.file "auth_log" {
targets = [
{__path__ = "/var/log/auth.log", job = "auth"},
]
forward_to = [loki.process.auth_filter.receiver]
}
// loki.source.journal "auth_journal" {
// matches = "SYSLOG_IDENTIFIER=sshd"
// forward_to = [loki.process.auth_filter.receiver]
// labels = {
// job = "auth",
// }
// }
loki.process "auth_filter" {
forward_to = [loki.write.default.receiver]
// 提取 SSH 登录成功信息: 时间 + 用户 + 来源IP
// syslog 行有前缀: Jul 23 17:17:23 hostname sshd[pid]: Accepted ...
// journal 行无前缀: Accepted publickey for ztjt from 1.2.3.4 ...
stage.regex {
expression = `(?:(?P<login_time>[A-Z][a-z]{2}\s+\d+\s+\d{2}:\d{2}:\d{2})\s+\S+\s+\S+(?:\[\d+\]):\s+)?Accepted (?:password|publickey|keyboard-interactive) for (?P<login_user>[^ ]+) from (?P<source_ip>[0-9a-fA-F:.]+)`
}
// 将 syslog 时间戳 Jul 23 17:17:23 转为 2026-07-23 17:17:23
// journal 无 login_time,用当前处理时间(与登录时间误差秒级)
stage.template {
source = "login_time"
template = `{{ if .login_time }}{{ printf "%d-%s" now.Year (date "01-02 15:04:05" (toDate "Jan _2 15:04:05" .login_time)) }}{{ else }}{{ date "2006-01-02 15:04:05" now }}{{ end }}`
}
// GeoIP 地理位置查询 (离线 .mmdb)
stage.geoip {
source = "source_ip"
db = "/etc/alloy/GeoLite2-City.mmdb"
db_type = "city"
}
// SSH 登录事件统一为 info
stage.template {
source = "detected_level"
template = "info"
}
stage.structured_metadata {
values = {
detected_level = "",
login_time = "",
geoip_location_latitude = "",
geoip_location_longitude = "",
}
}
stage.labels {
values = {
login_user = "",
source_ip = "",
geoip_city_name = "",
geoip_country_name = "",
}
}
}
// ==========================================
// [启用] 系统日志 (syslog/kern.log)
// ==========================================
loki.source.file "syslog" {
targets = [
{__path__ = "/var/log/syslog", job = "syslog"},
{__path__ = "/var/log/kern.log", job = "kernel"},
]
forward_to = [loki.write.default.receiver]
}
// ==========================================
// [禁用] MySQL — 按需取消注释启用
// 前置: 创建监控用户
// CREATE USER 'exporter'@'localhost' IDENTIFIED BY '<PASSWORD>';
// GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
// ==========================================
// prometheus.exporter.mysql "mysql" {
// data_source_name = "exporter:<PASSWORD>@(localhost:3306)/"
// enable_collectors = ["mysql.user", "engine_innodb_status", "info_schema.processlist", "info_schema.tables", "perf_schema.eventsstatements", "perf_schema.tableiowaits", "perf_schema.indexiowaits", "binlog_size"]
// }
// discovery.relabel "mysql" {
// targets = prometheus.exporter.mysql.mysql.targets
// rule {
// target_label = "instance"
// replacement = "<MYSQL_NAME>" // 替换为 MySQL 实例名, 如 mysql-prod
// }
// rule {
// target_label = "job"
// replacement = "mysql"
// }
// rule {
// target_label = "env"
// replacement = "test" // test 或 prod
// }
// }
// prometheus.scrape "mysql" {
// scrape_interval = "15s"
// targets = discovery.relabel.mysql.output
// forward_to = [prometheus.remote_write.default.receiver]
// }
// loki.source.file "mysql_log" {
// targets = [
// {__path__ = "/var/log/mysql/error.log", instance = "<MYSQL_NAME>"},
// {__path__ = "/var/log/mysql/slow-query.log", instance = "<MYSQL_NAME>"},
// ]
// forward_to = [loki.process.mysql.receiver]
// }
//
// loki.process "mysql" {
// forward_to = [loki.write.default.receiver]
//
// stage.static_labels {
// values = { job = "mysql" }
// }
//
// // 解析 MySQL 错误日志格式
// // 支持: [Note] [MY-012345] [InnoDB] msg / [Note] InnoDB: msg / [ERROR] msg
// stage.regex {
// expression = `^\S+\s+\d+\s+\[(?P<raw_level>\w+)\]\s+(?:\[(?P<error_code>[\w-]+)\]\s+)?(?:\[?(?P<subsystem>\w+)\]?(?::\s+|\s+))?(?P<msg>.*)`
// }
//
// stage.template {
// source = "detected_level"
// template = `{{- if eq .raw_level "Note" -}}info{{- else if eq .raw_level "System" -}}info{{- else if eq .raw_level "Warning" -}}warn{{- else if eq .raw_level "ERROR" -}}error{{- end -}}`
// }
//
// stage.structured_metadata {
// values = {
// detected_level = "",
// error_code = "",
// }
// }
//
// stage.labels {
// values = {
// subsystem = "",
// }
// }
// }
// ==========================================
// [禁用] Java JMX 指标 — 按需取消注释启用
// 前置: JMX Exporter 1.6.0 已通过 javaagent 挂载到 JVM
// java -javaagent:jmx_prometheus_javaagent-1.6.0.jar=9404:jmx_exporter.yaml -jar app.jar
// ==========================================
// prometheus.scrape "java" {
// scrape_interval = "15s"
// job_name = "java"
// targets = [
// {__address__ = "localhost:9404", instance = "gateway", env = "test"},
// // {__address__ = "localhost:9405", instance = "oa", env = "test"},
// // {__address__ = "localhost:9406", instance = "hr", env = "prod"},
// ]
// forward_to = [prometheus.remote_write.default.receiver]
// }
// ==========================================
// [禁用] Java 应用日志 — 按需取消注释启用
// 格式: YYYY-MM-DD HH:MM:SS.mmm [IP] [threadID] [0] [] LEVEL class - message
// ==========================================
// loki.source.file "java_logs" {
// targets = [
// {__path__ = "/opt/gateway/logs/*.log", instance = "gateway", env = "test"},
// {__path__ = "/opt/oa/logs/*.log", instance = "oa", env = "test"},
// {__path__ = "/opt/hr/logs/*.log", instance = "hr", env = "prod"},
// ]
// forward_to = [loki.process.java.receiver]
// }
//
// loki.process "java" {
// forward_to = [loki.write.default.receiver]
//
// stage.static_labels {
// values = { job = "java" }
// }
//
// stage.regex {
// expression = `^(?P<timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d+)\s+\[(?P<client_ip>[^\]]+)\]\s+\[(?P<thread_id>\d+)\]\s+\[\d+\]\s+\[\]\s+(?P<level>\w+)\s+(?P<class>\S+)\s+-\s+(?P<msg>.*)$`
// }
//
// stage.template {
// source = "detected_level"
// template = `{{- if eq .level "DEBUG" -}}debug{{- else if eq .level "INFO" -}}info{{- else if eq .level "WARN" -}}warn{{- else if eq .level "ERROR" -}}error{{- else -}}{{ .level | ToLower }}{{- end -}}`
// }
//
// stage.structured_metadata {
// values = {
// detected_level = "",
// class = "",
// client_ip = "",
// thread_id = "",
// }
// }
// }
// ==========================================
// [禁用] Nginx 指标 + 日志 — 按需取消注释启用
// 前提: 部署 nginx-prometheus-exporter (Alloy 无内置 nginx exporter)
// Docker 方式: nginx/nginx-prometheus-exporter:1.5.1
// 监听 :9113, 抓取 nginx stub_status
// 需与 Nginx 在同一 Docker 网络 (如 shared-proxy-network)
// 命令: docker run -d --name nginx-exporter \
// --network shared-proxy-network \
// -p 127.0.0.1:9113:9113 \
// nginx/nginx-prometheus-exporter:1.5.1 \
// --nginx.scrape-uri=http://<nginx容器名>/nginx_status
//
// 多实例: 每个实例单独跑一个 exporter, 不同端口
//
// access log 格式 (main):
// $remote_addr - $remote_user [$time_iso8601] "$request" $status
// $body_bytes_sent "$http_referer" "$http_user_agent"
// "$http_x_forwarded_for" rt=$request_time uct="$upstream_connect_time"
// uht="$upstream_header_time" urt="$upstream_response_time"
// error log 格式 (不可自定义):
// YYYY/MM/DD HH:MM:SS [level] pid#tid: message, client: IP, server: name,
// request: "METHOD /path HTTP/1.1", host: "hostname"
// ==========================================
// prometheus.scrape "nginx" {
// scrape_interval = "15s"
// targets = [
// {__address__ = "127.0.0.1:9113", instance = "proxy", job = "nginx"},
// // {__address__ = "127.0.0.1:9114", instance = "backend", job = "nginx"},
// ]
// forward_to = [prometheus.remote_write.default.receiver]
// }
// loki.source.file "nginx_access" {
// targets = [
// {__path__ = "/var/log/nginx/frontend/access.log", instance = "frontend"},
// // {__path__ = "/var/log/nginx/backend/access.log", instance = "backend"},
// ]
// forward_to = [loki.process.nginx_access.receiver]
// }
//
// loki.process "nginx_access" {
// forward_to = [loki.write.default.receiver]
//
// stage.static_labels {
// values = { job = "nginx" }
// }
//
// stage.regex {
// expression = `^(?P<remote_addr>\S+)\s+-\s+(?P<remote_user>\S+)\s+\[(?P<time_iso8601>[^\]]+)\]\s+"(?P<request>[^"]*)"\s+(?P<status>\d+)\s+(?P<body_bytes_sent>\d+)\s+"(?P<http_referer>[^"]*)"\s+"(?P<http_user_agent>[^"]*)"\s+"(?P<http_x_forwarded_for>[^"]*)"\s+rt=(?P<request_time>[\d.]+)\s+uct="(?P<upstream_connect_time>[^"]*)"\s+uht="(?P<upstream_header_time>[^"]*)"\s+urt="(?P<upstream_response_time>[^"]*)"`
// }
//
// stage.regex {
// source = "request"
// expression = `^(?P<method>\S+)\s+(?P<request_uri>\S+)\s+\S+$`
// }
//
// stage.template {
// source = "detected_level"
// template = "info"
// }
//
// stage.structured_metadata {
// values = {
// detected_level = "",
// request_time = "",
// upstream_response_time = "",
// body_bytes_sent = "",
// remote_addr = "",
// http_referer = "",
// http_user_agent = "",
// http_x_forwarded_for = "",
// request_uri = "",
// }
// }
//
// stage.labels {
// values = {
// status = "",
// method = "",
// }
// }
// }
//
// loki.source.file "nginx_error" {
// targets = [
// {__path__ = "/var/log/nginx/frontend/error.log", instance = "frontend"},
// // {__path__ = "/var/log/nginx/backend/error.log", instance = "backend"},
// ]
// forward_to = [loki.process.nginx_error.receiver]
// }
//
// loki.process "nginx_error" {
// forward_to = [loki.write.default.receiver]
//
// stage.static_labels {
// values = { job = "nginx" }
// }
//
// stage.regex {
// expression = `^\d{4}/\d{2}/\d{2}\s+\d{2}:\d{2}:\d{2}\s+\[(?P<level>\w+)\]\s+\d+#\d+:\s+(?P<msg>.+?)(?:,\s*client:\s*(?P<client>[^,]+))?(?:,\s*server:\s*(?P<server>[^,]+))?(?:,\s*request:\s*"(?P<request>[^"]*)")?(?:,\s*host:\s*"(?P<http_host>[^"]*)")?\s*$`
// }
//
// stage.template {
// source = "detected_level"
// template = `{{- if eq .level "debug" -}}debug{{- else if eq .level "info" -}}info{{- else if eq .level "notice" -}}info{{- else if eq .level "warn" -}}warn{{- else if eq .level "error" -}}error{{- else if eq .level "crit" -}}critical{{- else if eq .level "alert" -}}error{{- else if eq .level "emerg" -}}fatal{{- end -}}`
// }
//
// stage.structured_metadata {
// values = {
// detected_level = "",
// client = "",
// server = "",
// request = "",
// http_host = "",
// }
// }
// }
// ==========================================
// 推送配置 (引用 cfg 变量, 无需修改)
// ==========================================
// --- 推送日志到 Loki ---
loki.write "default" {
external_labels = {
host = cfg.hostname.value,
group = cfg.group.value,
}
endpoint {
url = cfg.endpoint_loki.value
basic_auth {
username = "loki"
password = cfg.password_loki.value
}
}
}
// --- 推送指标到 Prometheus ---
prometheus.remote_write "default" {
external_labels = {
host = cfg.hostname.value,
group = cfg.group.value,
}
endpoint {
url = cfg.endpoint_prom.value
basic_auth {
username = "prometheus"
password = cfg.password_prom.value
}
}
}Alloy 语法提示: 带
=的 argument(如external_labels = { ... })必须写在不带=的 block(如endpoint { ... })之前,否则会报语法错误。
部署步骤
sudo cp alloy/config-debian-full.alloy /etc/alloy/config.alloy
# 修改文件头部 cfg 块中的域名、密码、group
sudo vim /etc/alloy/config.alloy
sudo systemctl restart alloy
sudo systemctl status alloy如需使用环境变量,将配置变量替换为
sys.env("VAR")。
7.2 Windows Server 2016+
安装 Grafana Alloy
- 下载
alloy-installer-windows-amd64.exe - 管理员运行安装(或静默安装
alloy-installer-windows-amd64.exe /S)→ 自动安装为 Windows Service
配置文件: config-windows-full.alloy
文件路径: monitor/alloy/config-windows-full.alloy (仓库中) → C:\Program Files\GrafanaLabs\Alloy\config.alloy (目标服务器)
部署变量
与 Debian 版一致,修改文档头部 4 个变量即可。标签策略见 7.1 节标签设计表格。
完整配置
// ==========================================
// Grafana Alloy - Windows Server 2016+ 完整配置
// ==========================================
// 安装: 管理员运行 alloy-installer-windows-amd64.exe /S
// 配置路径: C:\Program Files\GrafanaLabs\Alloy\config.alloy
// 服务管理: sc.exe start/stop/query "Grafana Alloy"
//
// ==========================================
// 部署变量 — 修改 cfg 块即可, 下方自动引用
// ==========================================
// 有本地 Nginx 中转 → http://monitor-proxy.local/...
// 云端 VPC 内直连 → http://monitor-proxy.local/...
// (需将 monitor-proxy.local 通过内网 DNS 或 hosts 指向云端 Nginx 内网 IP)
// 云端公网直连 → https://monitor.yourdomain.com/...
declare "cfg" {
argument "value" { }
export "value" { value = argument.value.value }
}
cfg "endpoint_prom" { value = "https://monitor.yourdomain.com/api/v1/write" }
cfg "endpoint_loki" { value = "https://monitor.yourdomain.com/loki/api/v1/push" }
cfg "password_prom" { value = "<PROM_PASSWORD>" }
cfg "password_loki" { value = "<LOKI_PASSWORD>" }
cfg "group" { value = "local" }
cfg "hostname" { value = constants.hostname } // 或改为自定义标识
// ==========================================
// [启用] Windows 系统指标 (替代 windows_exporter)
// ==========================================
// 注意: cs、logon 收集器已在 Alloy 中移除,不再可用
prometheus.exporter.windows "win" {
enabled_collectors = ["cpu", "logical_disk", "memory", "net", "os", "process", "service", "system", "tcp"]
service {
include = "(mysql.*|filezilla-server)"
}
}
discovery.relabel "win_metrics" {
targets = prometheus.exporter.windows.win.targets
rule {
target_label = "instance"
replacement = cfg.hostname.value
}
rule {
target_label = "job"
replacement = "node"
}
}
prometheus.scrape "node" {
scrape_interval = "15s"
targets = discovery.relabel.win_metrics.output
forward_to = [prometheus.remote_write.default.receiver]
}
// ==========================================
// [启用] Windows 安全事件日志 (登录成功 4624 / 失败 4625)
// ==========================================
loki.source.windowsevent "security" {
eventlog_name = "Security"
xpath_query = "*[System[(EventID=4624 or EventID=4625)]]"
use_incoming_timestamp = true
bookmark_path = "C:\\ProgramData\\GrafanaLabs\\Alloy\\bookmark-security.xml"
forward_to = [loki.process.win_security.receiver]
labels = {
job = "windows_security",
}
}
loki.process "win_security" {
forward_to = [loki.write.default.receiver]
stage.json {
expressions = {
source = "source",
event_id = "event_id",
level = "levelText",
time = "timeCreated",
message = "message",
event_data = "event_data",
}
}
// 从 XML 格式 event_data 提取登录用户、来源IP、登录类型
stage.regex {
source = "event_data"
expression = `<Data Name='TargetUserName'>(?P<login_user>[^<]*)</Data>`
}
stage.regex {
source = "event_data"
expression = `<Data Name='IpAddress'>(?P<source_ip>[^<]*)</Data>`
}
stage.regex {
source = "event_data"
expression = `<Data Name='LogonType'>(?P<logon_type>[^<]*)</Data>`
}
// 过滤系统账号和网络会话噪音,只保留真实用户登录
stage.drop {
source = "login_user"
expression = "^(DWM-\\d+|UMFD-\\d+|.*\\$|ANONYMOUS LOGON)$"
drop_counter_reason = "system_account"
}
stage.drop {
source = "logon_type"
expression = "^(3|4|5)$"
drop_counter_reason = "service_logon"
}
// GeoIP 地理位置查询 (离线 .mmdb)
stage.geoip {
source = "source_ip"
db = "C:\\Program Files\\GrafanaLabs\\Alloy\\GeoLite2-City.mmdb"
db_type = "city"
}
// 格式化 login_time 为 yyyy-MM-dd HH:mm:ss
stage.template {
source = "login_time"
template = `{{ date "2006-01-02 15:04:05" (toDate "2006-01-02T15:04:05Z07:00" .time) }}`
}
// Security 事件无数字 level,用 levelText(中文)映射
// 信息→info 警告→warn 错误→error 关键→critical 详细→debug
stage.template {
source = "detected_level"
template = `{{- if eq .level "信息" -}}info{{- else if eq .level "警告" -}}warn{{- else if eq .level "错误" -}}error{{- else if eq .level "关键" -}}critical{{- else if eq .level "详细" -}}debug{{- end -}}`
}
// 将数字 logon_type 映射为可读名称
stage.template {
source = "logon_type_name"
template = `{{- $t := .logon_type }}{{- if eq $t "2" -}}Interactive{{- else if eq $t "3" -}}Network{{- else if eq $t "4" -}}Batch{{- else if eq $t "5" -}}Service{{- else if eq $t "7" -}}Unlock{{- else if eq $t "8" -}}NetworkCleartext{{- else if eq $t "9" -}}NewCredentials{{- else if eq $t "10" -}}RemoteInteractive{{- else if eq $t "11" -}}CachedInteractive{{- else -}}Unknown{{- end -}}`
}
stage.structured_metadata {
values = {
detected_level = "",
login_time = "",
geoip_location_latitude = "",
geoip_location_longitude = "",
}
}
stage.labels {
values = {
event_id = "",
level = "",
login_user = "",
source_ip = "",
logon_type = "",
logon_type_name = "",
geoip_city_name = "",
geoip_country_name = "",
}
}
}
// ==========================================
// [启用] Windows 应用事件日志
// ==========================================
loki.source.windowsevent "application" {
eventlog_name = "Application"
xpath_query = "*"
use_incoming_timestamp = true
bookmark_path = "C:\\ProgramData\\GrafanaLabs\\Alloy\\bookmark-application.xml"
forward_to = [loki.process.win_application.receiver]
labels = {
job = "windows_application",
}
}
loki.process "win_application" {
forward_to = [loki.write.default.receiver]
stage.json {
expressions = {
source = "source",
event_id = "event_id",
level = "level",
level_text = "levelText",
time = "timeCreated",
message = "message",
event_data = "event_data",
}
}
// 将数字 level 转为字符串,确保后续比较不会因类型问题失败
stage.template {
source = "level_str"
template = `{{ printf "%v" .level }}`
}
// 将 level 映射为 Loki 标准 log level
// Windows: 1=Critical 2=Error 3=Warning 4=Information 5=Verbose
stage.template {
source = "detected_level"
template = `{{- if eq .level_str "1" -}}critical{{- else if eq .level_str "2" -}}error{{- else if eq .level_str "3" -}}warn{{- else if eq .level_str "4" -}}info{{- else if eq .level_str "5" -}}debug{{- else -}}unknown{{- end -}}`
}
stage.structured_metadata {
values = {
detected_level = "",
event_id = "",
source = "",
}
}
stage.labels {
values = {
level = "",
}
}
}
// ==========================================
// [禁用] MySQL — 按需取消注释启用
// 前置: 创建监控用户
// CREATE USER 'exporter'@'localhost' IDENTIFIED BY '<PASSWORD>';
// GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';
// ==========================================
// prometheus.exporter.mysql "mysql" {
// data_source_name = "exporter:<PASSWORD>@(localhost:3306)/"
// enable_collectors = ["mysql.user", "engine_innodb_status", "info_schema.processlist", "info_schema.tables", "perf_schema.eventsstatements", "perf_schema.tableiowaits", "perf_schema.indexiowaits", "binlog_size"]
// }
// discovery.relabel "mysql" {
// targets = prometheus.exporter.mysql.mysql.targets
// rule {
// target_label = "instance"
// replacement = "<MYSQL_NAME>" // 替换为 MySQL 实例名, 如 mysql-prod
// }
// rule {
// target_label = "job"
// replacement = "mysql"
// }
// rule {
// target_label = "env"
// replacement = "test" // test 或 prod
// }
// }
// prometheus.scrape "mysql" {
// scrape_interval = "15s"
// targets = discovery.relabel.mysql.output
// forward_to = [prometheus.remote_write.default.receiver]
// }
// loki.source.file "mysql_log" {
// targets = [
// {__path__ = "C:\\ProgramData\\MySQL\\MySQL Server 8.0\\Data\\*.err", instance = "<MYSQL_NAME>"},
// ]
// forward_to = [loki.process.mysql.receiver]
// }
//
// loki.process "mysql" {
// forward_to = [loki.write.default.receiver]
//
// stage.static_labels {
// values = { job = "mysql" }
// }
//
// // 解析 MySQL 错误日志格式
// // 支持: [Note] [MY-012345] [InnoDB] msg / [Note] InnoDB: msg / [ERROR] msg
// stage.regex {
// expression = `^\S+\s+\d+\s+\[(?P<raw_level>\w+)\]\s+(?:\[(?P<error_code>[\w-]+)\]\s+)?(?:\[?(?P<subsystem>\w+)\]?(?::\s+|\s+))?(?P<msg>.*)`
// }
//
// stage.template {
// source = "detected_level"
// template = `{{- if eq .raw_level "Note" -}}info{{- else if eq .raw_level "System" -}}info{{- else if eq .raw_level "Warning" -}}warn{{- else if eq .raw_level "ERROR" -}}error{{- end -}}`
// }
//
// stage.structured_metadata {
// values = {
// detected_level = "",
// error_code = "",
// }
// }
//
// stage.labels {
// values = {
// subsystem = "",
// }
// }
// }
// ==========================================
// [禁用] Java JMX 指标 — 按需取消注释启用
// 前置: JMX Exporter 1.6.0 已通过 javaagent 挂载到 JVM
// -javaagent:C:\jmx_exporter\jmx_prometheus_javaagent-1.6.0.jar=9404:C:\jmx_exporter\jmx_exporter.yaml
// ==========================================
// prometheus.scrape "java" {
// scrape_interval = "15s"
// job_name = "java"
// targets = [
// {__address__ = "localhost:9404", instance = "gateway", env = "test"},
// // {__address__ = "localhost:9405", instance = "oa", env = "test"},
// // {__address__ = "localhost:9406", instance = "hr", env = "prod"},
// ]
// forward_to = [prometheus.remote_write.default.receiver]
// }
// ==========================================
// [禁用] Java 应用日志 — 按需取消注释启用
// 格式: YYYY-MM-DD HH:MM:SS.mmm [IP] [threadID] [0] [] LEVEL class - message
// ==========================================
// loki.source.file "java_logs" {
// targets = [
// {__path__ = "C:\\gateway\\logs\\*.log", instance = "gateway", env = "test"},
// {__path__ = "C:\\oa\\logs\\*.log", instance = "oa", env = "test"},
// {__path__ = "C:\\hr\\logs\\*.log", instance = "hr", env = "prod"},
// ]
// forward_to = [loki.process.java.receiver]
// }
//
// loki.process "java" {
// forward_to = [loki.write.default.receiver]
//
// stage.static_labels {
// values = { job = "java" }
// }
//
// stage.regex {
// expression = `^(?P<timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\.\d+)\s+\[(?P<client_ip>[^\]]+)\]\s+\[(?P<thread_id>\d+)\]\s+\[\d+\]\s+\[\]\s+(?P<level>\w+)\s+(?P<class>\S+)\s+-\s+(?P<msg>.*)$`
// }
//
// stage.template {
// source = "detected_level"
// template = `{{- if eq .level "DEBUG" -}}debug{{- else if eq .level "INFO" -}}info{{- else if eq .level "WARN" -}}warn{{- else if eq .level "ERROR" -}}error{{- else -}}{{ .level | ToLower }}{{- end -}}`
// }
//
// stage.structured_metadata {
// values = {
// detected_level = "",
// class = "",
// client_ip = "",
// thread_id = "",
// }
// }
// }
// ==========================================
// [禁用] Nginx 指标 + 日志 — 按需取消注释启用
// 前提: 部署 nginx-prometheus-exporter (Alloy 无内置 nginx exporter)
// 下载: https://github.com/nginx/nginx-prometheus-exporter/releases
// 运行: nginx-prometheus-exporter.exe --nginx.scrape-uri=http://localhost/nginx_status
//
// 多实例: 每个实例单独跑一个 exporter, 不同端口
//
// access log 格式 (main):
// $remote_addr - $remote_user [$time_iso8601] "$request" $status
// $body_bytes_sent "$http_referer" "$http_user_agent"
// "$http_x_forwarded_for" rt=$request_time uct="$upstream_connect_time"
// uht="$upstream_header_time" urt="$upstream_response_time"
// error log 格式 (不可自定义):
// YYYY/MM/DD HH:MM:SS [level] pid#tid: message, client: IP, server: name,
// request: "METHOD /path HTTP/1.1", host: "hostname"
// ==========================================
// prometheus.scrape "nginx" {
// scrape_interval = "15s"
// targets = [
// {__address__ = "localhost:9113", instance = "proxy", job = "nginx"},
// ]
// forward_to = [prometheus.remote_write.default.receiver]
// }
// loki.source.file "nginx_access" {
// targets = [
// {__path__ = "C:\\nginx\\frontend\\logs\\access.log", instance = "frontend"},
// // {__path__ = "C:\\nginx\\backend\\logs\\access.log", instance = "backend"},
// ]
// forward_to = [loki.process.nginx_access.receiver]
// }
//
// loki.process "nginx_access" {
// forward_to = [loki.write.default.receiver]
//
// stage.static_labels {
// values = { job = "nginx" }
// }
//
// stage.regex {
// expression = `^(?P<remote_addr>\S+)\s+-\s+(?P<remote_user>\S+)\s+\[(?P<time_iso8601>[^\]]+)\]\s+"(?P<request>[^"]*)"\s+(?P<status>\d+)\s+(?P<body_bytes_sent>\d+)\s+"(?P<http_referer>[^"]*)"\s+"(?P<http_user_agent>[^"]*)"\s+"(?P<http_x_forwarded_for>[^"]*)"\s+rt=(?P<request_time>[\d.]+)\s+uct="(?P<upstream_connect_time>[^"]*)"\s+uht="(?P<upstream_header_time>[^"]*)"\s+urt="(?P<upstream_response_time>[^"]*)"`
// }
//
// stage.regex {
// source = "request"
// expression = `^(?P<method>\S+)\s+(?P<request_uri>\S+)\s+\S+$`
// }
//
// stage.template {
// source = "detected_level"
// template = "info"
// }
//
// stage.structured_metadata {
// values = {
// detected_level = "",
// request_time = "",
// upstream_response_time = "",
// body_bytes_sent = "",
// remote_addr = "",
// http_referer = "",
// http_user_agent = "",
// http_x_forwarded_for = "",
// request_uri = "",
// }
// }
//
// stage.labels {
// values = {
// status = "",
// method = "",
// }
// }
// }
//
// loki.source.file "nginx_error" {
// targets = [
// {__path__ = "C:\\nginx\\frontend\\logs\\error.log", instance = "frontend"},
// // {__path__ = "C:\\nginx\\backend\\logs\\error.log", instance = "backend"},
// ]
// forward_to = [loki.process.nginx_error.receiver]
// }
//
// loki.process "nginx_error" {
// forward_to = [loki.write.default.receiver]
//
// stage.static_labels {
// values = { job = "nginx" }
// }
//
// stage.regex {
// expression = `^\d{4}/\d{2}/\d{2}\s+\d{2}:\d{2}:\d{2}\s+\[(?P<level>\w+)\]\s+\d+#\d+:\s+(?P<msg>.+?)(?:,\s*client:\s*(?P<client>[^,]+))?(?:,\s*server:\s*(?P<server>[^,]+))?(?:,\s*request:\s*"(?P<request>[^"]*)")?(?:,\s*host:\s*"(?P<http_host>[^"]*)")?\s*$`
// }
//
// stage.template {
// source = "detected_level"
// template = `{{- if eq .level "debug" -}}debug{{- else if eq .level "info" -}}info{{- else if eq .level "notice" -}}info{{- else if eq .level "warn" -}}warn{{- else if eq .level "error" -}}error{{- else if eq .level "crit" -}}critical{{- else if eq .level "alert" -}}error{{- else if eq .level "emerg" -}}fatal{{- end -}}`
// }
//
// stage.structured_metadata {
// values = {
// detected_level = "",
// client = "",
// server = "",
// request = "",
// http_host = "",
// }
// }
// }
// ==========================================
// 推送配置 (引用 cfg 变量, 无需修改)
// ==========================================
// --- 推送日志到 Loki ---
loki.write "default" {
external_labels = {
host = cfg.hostname.value,
group = cfg.group.value,
}
endpoint {
url = cfg.endpoint_loki.value
basic_auth {
username = "loki"
password = cfg.password_loki.value
}
}
}
// --- 推送指标到 Prometheus ---
prometheus.remote_write "default" {
external_labels = {
host = cfg.hostname.value,
group = cfg.group.value,
}
endpoint {
url = cfg.endpoint_prom.value
basic_auth {
username = "prometheus"
password = cfg.password_prom.value
}
}
}Alloy 语法提示: 带
=的 argument(如external_labels = { ... })必须写在不带=的 block(如endpoint { ... })之前,否则会报语法错误。
部署步骤
# 复制配置
copy config-windows-full.alloy "C:\Program Files\GrafanaLabs\Alloy\config.alloy"
# 修改文件头部 cfg 块中的域名、密码、group
# 启动服务
sc.exe start "Grafana Alloy"
sc.exe query "Grafana Alloy"注意: Windows 的认证配置与 Debian 版完全一致。
7.3 Java 服务 JMX Exporter
启用 Java 模块时的前置步骤:
# === JMX Exporter 1.6.0 (Java 8+) ===
# Linux
curl -LO https://github.com/prometheus/jmx_exporter/releases/download/v1.6.0/jmx_prometheus_javaagent-1.6.0.jar
# 编写 JMX 规则配置文件到 /opt/jmx_exporter/jmx_exporter.yaml (参考下方配置)
# JVM 启动参数: -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-1.6.0.jar=9404:/opt/jmx_exporter/jmx_exporter.yaml
# Windows (PowerShell)
curl -LO https://github.com/prometheus/jmx_exporter/releases/download/v1.6.0/jmx_prometheus_javaagent-1.6.0.jar
# 编写 JMX 规则配置文件到 C:\jmx_exporter\jmx_exporter.yaml (参考下方配置)
# JVM 启动参数: -javaagent:C:\jmx_exporter\jmx_prometheus_javaagent-1.6.0.jar=9404:C:\jmx_exporter\jmx_exporter.yaml
# === JMX Exporter 0.17.1 Java 6 构建 (Java 6/7) ===
# Linux
curl -LO https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent_java6/0.17.1/jmx_prometheus_javaagent_java6-0.17.1.jar
# JVM 启动参数: -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent_java6-0.17.1.jar=9404:/opt/jmx_exporter/jmx_exporter.yaml
# Windows
curl -LO https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent_java6/0.17.1/jmx_prometheus_javaagent_java6-0.17.1.jar
# JVM 启动参数: -javaagent:C:\jmx_exporter\jmx_prometheus_javaagent_java6-0.17.1.jar=9404:C:\jmx_exporter\jmx_exporter.yamlJMX Exporter 规则配置参考 (jmx_exporter.yaml):
---
lowercaseOutputLabelNames: true
lowercaseOutputName: true
rules:
- pattern: 'com.jolbox.bonecp<type=BoneCP[^,]*><>(TotalFree|TotalLeased|TotalCreatedConnections):'
name: bonecp_$1
help: BoneCP connection pool $1
type: GAUGE说明: JVM 基础指标(内存/GC/线程/类加载/CPU)由 JMX Exporter 内置自动导出,无需在
rules中声明。以上仅额外采集 BoneCP 连接池状态。如果无需连接池监控,可设为rules: []。Tomcat 用户: 如果 Catalina MBean 可用(Tomcat 7+ 且 JMX 正常),可参考 官方
examples/tomcat.yml追加 Servlet/ThreadPool/Session 规则。JMX Exporter 不是独立 Service,随 JVM 启停。同机 Alloy 通过
localhost:9404抓取。上方jmx_exporter.yaml兼容 0.17.x 和 1.6.0 两个版本。
7.4 MySQL 监控账号创建
启用 MySQL 模块时,在 MySQL 中创建监控用户(Alloy 用此账号连接):
CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'password';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'localhost';将配置中
data_source_name = "exporter:password@(localhost:3306)/"的密码替换为实际值。
7.5 配置模块启用速查
两个 full 配置文件结构相同,通过注释/取消注释控制各模块:
| 模块 | 默认状态 | 启用方法 |
|---|---|---|
| 系统指标 (node/systemd) | 启用 | 无需操作 |
| 系统日志 (auth/syslog) | 启用 | 无需操作 |
| 安全事件日志 (Windows) | 启用 | 无需操作 |
| MySQL 指标 + 日志 | 禁用 | 取消 4 个注释块: exporter + discovery + scrape + loki.source.file |
| Java JMX 指标 | 禁用 | 取消 prometheus.scrape "java" 块注释 |
| Java 应用日志 | 禁用 | 取消 loki.source.file "java_logs" 块注释 |
| Nginx 指标 + 日志 | 禁用 | 先部署 nginx-prometheus-exporter,再取消 prometheus.scrape "nginx"、loki.source.file "nginx_access" 和 loki.source.file "nginx_error" 注释 |
说明: 每个模块现在是自包含的 4 段式结构(exporter → discovery.relabel → prometheus.scrape → loki.source.file),每个
prometheus.scrape有独立的job_name。
八、采集说明
8.1 标签策略
Alloy 在采集端(PUSH 侧)完成全部标签处理,服务端(Prometheus/Loki)不参与:
Alloy 采集端
├── 顶部配置变量 (改一处,全局生效)
│ ├── hostname = constants.hostname (或 "app_local", 改 cfg 块即可)
│ └── group = "local"
│
├── external_labels (全局,同机所有数据一致)
│ ├── host = hostname
│ └── group = group
│
├── 指标标签 (独立 prometheus.scrape,各自 job_name)
│ ├── job=node, instance=hostname → node_exporter metrics
│ ├── job=systemd, instance=hostname → systemd 服务状态
│ ├── job=mysql, instance=<MYSQL_NAME> → MySQL exporter metrics
│ ├── job=java, instance=<SERVICE_NAME> → JMX Exporter metrics
│ └── job=nginx, instance=<NGINX_NAME> → nginx stub_status metrics
│
└── 日志标签 (loki.source.file targets 内联)
├── job=auth/syslog/kernel → 系统日志 (host 已足够)
├── job=mysql, instance=<MYSQL_NAME> → MySQL 日志
├── job=java, instance=<SERVICE_NAME> → Java 应用日志
└── job=nginx, instance=<NGINX_NAME> → Nginx 日志
设计原则:
host+group= 全局(external_labels),同机所有数据一致job= 数据源类型(prometheus.scrape.job_name 或 loki.source.file.job)instance= 具体实例标识(Prometheus 标准标签,所有指标和可能多实例的日志都带)- 单实例日志(auth/syslog/kernel)不加
instance,host已唯一标识
8.2 采集端分工
| 环境 | 配置文件 | 默认采集 | 可选扩展 |
|---|---|---|---|
| Debian 12 / Ubuntu 22 | config-debian-full.alloy | 系统指标 + systemd + 系统日志 (auth/syslog/kernel) | MySQL 指标+日志 / Java JMX+日志 / Nginx 指标+日志 |
| Windows Server 2016+ | config-windows-full.alloy | 系统指标 + Windows 事件日志 (安全+应用) | MySQL 指标+日志 / Java JMX+日志 / Nginx 指标+日志 |
8.3 MySQL 采集
MySQL 指标由 Grafana Alloy 内置的 prometheus.exporter.mysql 在本机 localhost 采集,不需要独立的 mysqld-exporter 容器。同时支持错误日志和慢查询日志采集。
MySQL 8 (localhost:3306)
↑ DSN
Grafana Alloy (prometheus.exporter.mysql) ──→ 指标 → 云端 Prometheus
Grafana Alloy (loki.source.file) ──→ 日志 → 云端 Loki
8.4 Nginx 采集
Grafana Alloy 没有内置 prometheus.exporter.nginx 组件,Nginx 指标需通过独立的 nginx-prometheus-exporter 采集:
Nginx (stub_status)
↑ scrape
nginx-prometheus-exporter (Docker/二进制, :9113)
↑ prometheus.scrape
Grafana Alloy ──→ 指标 → 云端 Prometheus
Grafana Alloy (loki.source.file) ──→ 日志 → 云端 Loki
Docker 部署 (推荐,与 Nginx 同一 compose):
nginx-exporter:
image: nginx/nginx-prometheus-exporter:1.5.1
container_name: monitor-nginx-exporter
restart: unless-stopped
command: ["--nginx.scrape-uri=http://nginx/nginx_status"]
ports:
- "127.0.0.1:9113:9113"
networks:
- proxy-network # 与 Nginx 同一 Docker 网络
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
privileged: false
deploy:
resources:
limits:
cpus: "0.1"
memory: 32M
pids: 20
reservations:
memory: 16M注意:
location /nginx_status不要写allow/deny,继承 server 级规则即可(Docker 容器间流量源 IP 不是 127.0.0.1)。
九、告警说明
9.1 告警链路
- Prometheus 指标告警 (
prometheus/rules/alerts.yml) — Linux + Windows 双平台(CPU/内存/磁盘/下线/IO) - Loki 安全日志告警 (
loki/rules/fake/security.yml) — SSH 爆破/Windows 登录异常/登录提醒 - 告警发送到 Alertmanager → 按严重级别路由到企业微信推送
| 严重级别 | 路由接收器 | 推送行为 |
|---|---|---|
| critical | wechat-critical | 推送告警 + 恢复通知 |
| warning | wechat-warning | 推送告警 + 恢复通知 |
| login | wechat-login | 推送登录提醒,不推恢复 |
| info | wechat-info | 推送通知,不推恢复 |
- 可在 Alertmanager Web UI (
https://monitor.yourdomain.com/alertmanager/) 查看活跃告警
9.2 企业微信配置前置条件
告警通过企业微信自建应用的消息接口推送,部署前需在企业微信管理后台完成以下准备:
第一步:获取企业 ID (corp_id)
- 登录 企业微信管理后台
- 进入 “我的企业” → “企业信息” → “企业ID” (即 corp_id)
- 填入
alertmanager/alertmanager.yml的<YOUR_CORP_ID>
第二步:创建自建应用,获取 AgentID 和 Secret
- 管理后台 → “应用管理” → “应用” → “自建” → “创建应用”
- 设置应用名称 (如 “监控告警”)、应用 logo、可见范围 (选择接收告警的部门)
- 创建完成后进入应用详情页,可以看到:
- AgentId → 填入
<YOUR_AGENT_ID> - Secret → 填入
<YOUR_API_SECRET>(点击查看,仅显示一次,务必保存)
- AgentId → 填入
第三步:获取接收人账号 (to_user)
- 管理后台 → “通讯录” → 点击具体成员 → 字段名 “账号”
- 支持多人,用
|分隔,如zhangsan|lisi - 填入
alertmanager.yml中to_user字段
第四步:配置应用可信 IP
- 在自建应用详情页 → “企业可信IP” → 添加云端服务器的公网 IP
- 企业微信会校验调用来源 IP,不配置将导致消息发送失败
第五步:配置可信域名
- 自建应用详情页 → “网页授权及 JS-SDK” → “设置可信域名”
- 填入监控域名 (如
monitor.yourdomain.com),下载验证文件 (如WW_verify_<VERIFICATION_FILE>.txt) - 将验证文件放到
/opt/monitor/nginx/verify/目录下 - 确保 Nginx 容器挂载了该目录(见 5.15 节 volumes 配置)
- Nginx 已预置
/WW_verify_路径直接返回静态文件,无需额外配置 - 在企业微信管理后台点击”确定”完成域名校验
注意: 可信域名配置是可选的,仅当应用需要 OAuth 网页授权时才必须设置。 Alertmanager 仅使用服务端 API 发送消息,不依赖网页授权,理论上可不配置可信域名。 但部分企业微信版本的管理后台可能强制要求,届时按上述步骤配置即可。
官方文档:
- 基本概念介绍 — corp_id / agent_id / secret / 部门 ID 的定义
- 简易教程 - 创建自建应用 — 图文步骤
- 企业微信管理后台
9.3 修改后重启
# 修改 alertmanager.yml 后重启生效
cd /opt/monitor
docker compose restart alertmanager十、验证清单
10.1 云端服务
docker compose ps
curl -s http://localhost:9090/-/healthy
curl -s http://localhost:3100/ready
curl -s http://localhost:3000/api/health
curl -s http://localhost:9093/-/healthy10.2 采集端
# Debian/Ubuntu
sudo systemctl status alloy
sudo journalctl -u alloy -n 30 --no-pager
curl -s http://localhost:12345/metrics | head -5
# Windows
sc.exe query "Grafana Alloy"10.3 端到端
- 浏览器
https://monitor.yourdomain.com→ Grafana 登录页 (密码1) - Explore → Prometheus → 查
up→ 看到所有实例 - Explore → Loki → 查
{job="auth"}→ 看到安全日志 - 导入仪表盘: Node Exporter Full (ID: 1860), MySQL Overview (ID: 7362)
- 方式1: Grafana UI → Dashboards → Import → 输入 ID
- 方式2: 下载 JSON 放入
grafana/provisioning/dashboards/目录自动加载
十一、日常运维
# 云端重启
docker compose restart prometheus
docker compose restart loki
# 云端日志
docker compose logs -f --tail=100 prometheus
# 采集端重载
sudo systemctl reload alloy # Linux
sc.exe stop "Grafana Alloy" && sc.exe start "Grafana Alloy" # Windows
# 配置语法检查
sudo alloy run /etc/alloy/config.alloy --dry-run