使用 Grafana Loki + Grafana 实现日志监控

昨天小红书刷到最近曝出一个严重的 Linux 本地提权漏洞 Copy Fail，当时心想补漏洞的话还要重启服务器耽误大家实验进度，实验室的同学应该没什么坏心眼吧。结果第二天做日常维护的时候用到 su 命令，发现怎么没输密码就进 root 了。心想坏了，肯定是有人利用了昨天的漏洞。查了一下系统日志，发现果然有其他用户提权到了 root，再一查用户命令历史，果然是执行了昨天曝出来的 Copy Fail PoC。哎，看来还是要多长个心眼。把这位用户锁定之后，觉得还是要加强防护，需要有一个完善的日志监控系统。毕竟这次对方只是忘记清理痕迹，下次如果遇到更强大的对手可能就永远不会发现自己的服务器已经被人攻破了。

在一番调查之后，我了解到目前主流的日志监控系统主要是 Datadog、ELK Stack 和 Grafana Loki + Grafana。其中 Datadog 和 ELK Stack 有些功能过剩，于是决定部署 Grafana Loki + Grafana。

auditd

auditd 负责收集内核事件，发送到 audispd。

  1. 安装 auditd：

    sudo apt install auditd

  2. 配置审计规则：

    sudo curl -o /etc/audit/rules.d/audit.rules https://raw.githubusercontent.com/Neo23x0/auditd/master/audit.rules  # 下载 Neo23x0 规则
sudo augenrules --load  # 加载规则（不用管报错）
sudo systemctl restart auditd  # 重启 auditd
sudo auditctl -l  # 验证规则已加载

  3. 查看审计日志：

    sudo cat /var/log/audit/audit.log

Laurel

Laurel 负责将多行审计事件转换成 JSON。

sudo apt install clang llvm-dev libauparse-dev libacl1-dev  # 安装依赖
git clone https://github.com/threathunters-io/laurel.git
cd laurel
cargo build --release  # 构建 laurel
sudo install -m755 target/release/laurel /usr/local/sbin/laurel  # 安装 laurel 二进制文件
sudo useradd --system --home-dir /var/log/laurel --create-home _laurel  # 创建 _laurel 用户
sudo mkdir /etc/laurel
sudo cp etc/laurel/config.toml /etc/laurel/config.toml  # 复制配置文件
sudo cp etc/audit/plugins.d/laurel.conf /etc/audit/plugins.d/laurel.conf  # 复制 audit 插件配置
sudo systemctl restart auditd  # 重启 auditd 使其加载 laurel

Grafana Alloy

Alloy 是通用遥测收集器，可以同时采集日志、指标、链路和性能分析。

  1. 安装 Alloy：

    curl -fsSL https://apt.grafana.com/gpg.key | \
gpg --dearmor | \
sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null && \
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | \
sudo tee /etc/apt/sources.list.d/grafana.list && \
sudo apt update && \
sudo apt install alloy

  2. 配置 Alloy：

    sudoedit /etc/alloy/config.alloy

    // File Discovery
local.file_match "laurel" {
	path_targets = [{
		__path__ = "/var/log/laurel/audit.log",
		job      = "laurel",
	}]
}

local.file_match "auth" {
	path_targets = [{
		__path__ = "/var/log/auth.log",
		job      = "auth",
	}]
}

local.file_match "sssd" {
	path_targets = [{
		__path__ = "/var/log/sssd/*.log",
		job      = "sssd",
	}]
}

// Laurel Audit Logs - Extract key fields
loki.source.file "laurel" {
	targets    = local.file_match.laurel.targets
	forward_to = [loki.process.laurel_json.receiver]
}

loki.process "laurel_json" {
	// Extract audit ID
	stage.json {
		expressions = {
			audit_id = "ID",
		}
	}

	// Extract numeric fields (uid, auid, pid, ses)
	stage.json {
		expressions = {
			uid  = "USER_CMD.uid",
			auid = "USER_CMD.auid",
			ses  = "USER_CMD.ses",
			pid  = "USER_CMD.pid",
		}
	}

	// Extract from CRED_REFR events
	stage.json {
		expressions = {
			uid  = "CRED_REFR.uid",
			auid = "CRED_REFR.auid",
			ses  = "CRED_REFR.ses",
			pid  = "CRED_REFR.pid",
		}
	}

	// Extract from other event types
	stage.json {
		expressions = {
			uid  = "USER_START.uid",
			auid = "USER_START.auid",
			ses  = "USER_START.ses",
			pid  = "USER_START.pid",
		}
	}

	// Set labels from extracted fields
	stage.labels {
		values = {
			audit_id = "audit_id",
			auid     = "auid",
			uid      = "uid",
		}
	}

	stage.static_labels {
		values = {
			job = "laurel",
		}
	}

	forward_to = [loki.write.loki.receiver]
}

// Auth Logs
loki.source.file "auth" {
	targets    = local.file_match.auth.targets
	forward_to = [loki.process.auth_filter.receiver]
}

loki.process "auth_filter" {
	stage.drop {
		source              = ""
		expression          = ".*Connection closed by authenticating user.*"
		drop_counter_reason = "noisy_connection_closed"
	}

	stage.static_labels {
		values = {
			job = "auth",
		}
	}

	forward_to = [loki.write.loki.receiver]
}

// SSSD Logs
loki.source.file "sssd" {
	targets    = local.file_match.sssd.targets
	forward_to = [loki.process.sssd_filter.receiver]
}

loki.process "sssd_filter" {
	stage.static_labels {
		values = {
			job = "sssd",
		}
	}

	forward_to = [loki.write.loki.receiver]
}

// Loki Output
loki.write "loki" {
	endpoint {
		url = "http://localhost:3100/loki/api/v1/push"
	}
}

  3. 重启 Alloy：

    sudo systemctl enable alloy
sudo systemctl restart alloy

参见：Install Grafana Alloy on Linux | Grafana Alloy documentation

Grafana Loki

Loki 负责存储和索引日志。

  1. 安装 Loki：

    sudo apt install loki

  2. 配置 Loki：

    sudoedit /etc/loki/config.yml

    auth_enabled: false

ingester:
  chunk_idle_period: 3m
  max_chunk_age: 1h
  wal:
    enabled: true
    dir: /var/lib/loki/wal
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1

limits_config:
  allow_structured_metadata: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

server:
  http_listen_port: 3100
  log_level: info

storage_config:
  boltdb_shipper:
    active_index_directory: /var/lib/loki/index
    cache_location: /var/lib/loki/boltdb-cache
  filesystem:
    directory: /var/lib/loki/chunks

chunk_store_config:

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

compactor:
  working_directory: /var/lib/loki/compactor

  3. 创建数据目录：

    sudo mkdir -p /var/lib/loki/{index,chunks,wal,compactor}
sudo chown -R loki:nogroup /var/lib/loki

  4. 重启 Loki：

    sudo systemctl enable alloy
sudo systemctl restart loki

  5. 等待 15s 后，验证：

    $ curl http://localhost:3100/ready
ready

Grafana OSS

Grafana OSS 负责查询 Loki，可视化和告警。

  1. 安装：

    sudo apt install grafana

  2. 启动：

    sudo systemctl enable --now grafana-server

  3. 访问 http://localhost:3000，用户名: admin, 密码: admin

参见：Install Grafana on Debian or Ubuntu | Grafana documentation

添加数据源

  1. 访问 http://localhost:3000，登录（admin / admin）

  2. Connections > Data Sources > Add data source

  3. 选择 Loki

  4. 配置：

    Name: Loki
URL: http://localhost:3100

  5. 点击 Save & Test

  6. 测试查询：

    1. Explore

    2. 选择数据源：Loki

    3. 在 Code 模式查询框输入：

      {job="auth"} |= `sudo` |= `COMMAND`

    4. 点 Run Query，你应该能看到 sudo 命令的日志。

  7. 点击 Add to dashboard 添加到仪表板。

posted @ 2026-05-02 04:17  Undefined443  阅读(6)  评论(0)    收藏  举报