使用loki+ mtail + grafana + prometheus server分析应用问题

loki 是一个方便的类似prometheus 的log 系统，mtail 是一个方便的日志提取工具，
可以暴露为http 服务——支持导出prometheus metrics

环境准备

docker-compose 文件

 
version: "3"

services:

  nginx-log:

    build: ./

    ports:

    - "8090:80"

    - "3903:3903"

    volumes:

    - "./examples/linecount.mtail:/progs/linecount.mtail"

    - $PWD:/etc/promtail

  loki:

    image: grafana/loki:master

    ports:

      - "3100:3100"

    volumes:

      - $PWD:/etc/loki

    command: -config.file=/etc/loki/loki-local-config.yaml

  promtail:

    image: grafana/promtail:make-images-static-26a87c9

    volumes:

      - $PWD:/etc/promtail

      - ./log:/var/log

    command: 

      -config.file=/etc/promtail/promtail-docker-config.yaml

  grafana:

    image: grafana/grafana:master

    ports:

      - "3000:3000"

    environment:

    - "GF_EXPLORE_ENABLED=true"

  prometheus:

    image: prom/prometheus

    volumes:

    - "./prometheus.yml:/etc/prometheus/prometheus.yml"

    ports:

    - "9090:9090"

dockerfile 说明
dockerfile 主要是一个简单的openresty 服务，集成了mtail+loki,dockerfile 使用multi stage 处理

 
FROM dalongrong/mtail as builder

FROM grafana/promtail:make-images-static-26a87c9 as promtail

​

FROM openresty/openresty:alpine

ENV TINI_VERSION v0.18.0

RUN apk add --update \

    && apk add --no-cache tini

ADD entrypoint.sh /entrypoint.sh

ADD mtail.sh /mtail.sh

ADD promtail.sh /promtail.sh

COPY nginx.conf usr/local/openresty/nginx/conf/

COPY --from=builder /usr/bin/mtail /usr/bin/

COPY --from=promtail /usr/bin/promtail /usr/bin/

EXPOSE 80 3903

ENTRYPOINT ["/sbin/tini","-s", "--", "/entrypoint.sh"]

启动脚本(entrypoint)

#!/bin/sh

sh mtail.sh

sh promtail.sh

exec /usr/local/openresty/bin/openresty -g "daemon off;"

​

mtail.sh:

​

#!/bin/sh

nohup /usr/bin/mtail -logtostderr -progs /progs/linecount.mtail -logs /var/log/error.log & 

​

promtail.sh:

​

#!/bin/sh

nohup /usr/bin/promtail -config.file=/etc/promtail/promtail-docker-config2.yaml &

prometheus server 配置
prometheus.yml 文件

 
scrape_configs:

  - job_name: openresty-metrics

    metrics_path: /metrics

    static_configs:

      - targets: ['nginx-log:3903']

loki 配置
主要是agent 以及server

 
promtail-docker-config2.yaml:

server:

  http_listen_port: 0

  grpc_listen_port: 0

​

positions:

  filename: /tmp/positions.yaml

​

client:

  url: http://loki:3100/api/prom/push

​

scrape_configs:

- job_name: system

  entry_parser: raw

  static_configs:

  - targets:

      - localhost

    labels:

      job: varlogs-nginx

      __path__: /var/log

mtail 配置

# Copyright 2011 Google Inc. All Rights Reserved.

# This file is available under the Apache license.

​

counter line_count

​

/$/ {

  line_count++

}

​