asp.net core项目集成OpenTelemetry输出到OpenObserve进行可观测性处理

一、安装OpenObserve和Otel-collector

使用docker-compose 安装

记得替换下USER_MAIL xxx@xx.com和PASSWORD xxx
这里解释下为啥要用named volume来映射openobserve的data目录，因为目前版本(0.7.2)在windows的docker desktop里使用会有bug，导致stream无法正确写入文件持久化，一直在生成新的stream文件，而使用named volume则正常，详情请见issue

openobserve占用了5080（http）、5081（grpc）两个端口
otel-collector占用了5078和5079两个端口来作为对外提供服务的入口，后边的配置会有解释，其他端口有注释这里就不赘述了

新建一个openobserve目录，新建docker-compose.yaml文件，内容如下

services:
  main:
    image: openobserve/openobserve
    volumes:
      - ob-v:/data
    ports:
      - 5080:5080
      - 5081:5081
    environment:
      - ZO_DATA_DIR=/data
      - ZO_ROOT_USER_EMAIL=xxx@xx.com
      - ZO_ROOT_USER_PASSWORD=xxxx
      - TZ=Asia/Shanghai
    restart: always
  otel-collector:
    image: otel/opentelemetry-collector-contrib
    restart: always
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - 1888:1888 # pprof extension
      - 8888:8888 # Prometheus metrics exposed by the Collector
      - 8889:8889 # Prometheus exporter metrics
      - 13133:13133 # health_check extension
      - 5078:5078 # OTLP   gRPC receiver
      - 5079:5079 # OTLP/2 gRPC receiver
      #- 55679:55679 # zpages extension

volumes:
  ob-v: {}

otel-collector-config配置

然后在当前目录新建otel-collector-config.yaml内容如下
指定了两个端点，5079和5078，让测试环境和生产环境走不同的流水线处理。otel-collector-config配置参考文档
记得替换下边的myapp为自己的项目名称

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:5079
  otlp/2:
    protocols:
      grpc:
        endpoint: 0.0.0.0:5078

processors:
  batch:

exporters:
  otlp:
    endpoint: main:5081
    headers:
      Authorization: "Basic xxxxxx=="
      organization: myapp_test
      stream-name: default
    tls:
      insecure: true
  otlp/2:
    endpoint: main:5081
    headers:
      Authorization: "Basic xxxxxx=="
      organization: myapp_production
      stream-name: default
    tls:
      insecure: true

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]
    traces/2:
      receivers: [otlp/2]
      processors: [batch]
      exporters: [otlp/2]
    metrics/2:
      receivers: [otlp/2]
      processors: [batch]
      exporters: [otlp/2]
    logs/2:
      receivers: [otlp/2]
      processors: [batch]
      exporters: [otlp/2]

然后使用命令行运行docker-compose up -d

浏览器访问 服务器ip:5080, 用户名和密码是上边设置的邮箱和密码
点击左侧采集--Trace（OpenTelemetry）复制其中的Authorization: 后边的部分，替换otel-collector-config.yaml中的Basic xxxxxx==

最后执行docker-compose restart重启容器完成配置

二、asp.net core项目增加OpenTelemetry导出

引用OpenTelemetry Nuget包

在web api项目的csproj文件中增加以下nuget包引用

    <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.EntityFrameworkCore" Version="1.12.0-beta.1" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.12.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.12.0" />

在appsettings.json中配置启用开关

  "OpenTelemetry": {
    "ServiceName": "UserService",
    "EndPoint": "http://192.168.1.101:5079",
    "Enable": true
  }

其中ServiceName是该web api项目名称，EndPoint是otel-collector的一个grpc端点
比如测试环境的项目使用了5079端口，那生成环境就改为5078端口，如果还有演示环境、预发布环境之类的就继续在配置里增加端口

代码中配置启用Logging Metrics Traces

配置Logging导出

var builder = WebApplication.CreateBuilder(args);
_ = bool.TryParse(builder.Configuration["OpenTelemetry:Enable"], out var otelEnable);
var serviceName = builder.Configuration["OpenTelemetry:ServiceName"] ?? "";
builder.Services.AddLogging(logging =>
{
    if (otelEnable)
    {
        logging.AddOpenTelemetry(options =>
        {
            options.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(serviceName));
            options.AddConsoleExporter();
        });
    }
});

配置 Tracing和Metrics导出

以下tracing配置限制了只记录/api开头的请求，并记录X-Forwarded-For请求头或RemoteIpAddress作为client_ip. X-Forwarded-For用于配置反向代理后的客户端ip获取，一般用于正式环境。

if (otelEnable)
    builder.Services
        .AddOpenTelemetry()
        .ConfigureResource(resourceBuilder => resourceBuilder
            .AddService(serviceName, serviceInstanceId: Environment.MachineName))
        .WithTracing(traceBuilder => traceBuilder
                .SetSampler(new AlwaysOnSampler())
                .AddHttpClientInstrumentation()
                .AddAspNetCoreInstrumentation(options =>
                {
                    options.RecordException = true;
                    //只记录api
                    options.Filter = context => context.Request.Path.Value?.StartsWith("/api") ?? false;
                    // 增加client_ip到Activity.Tags
                    options.EnrichWithHttpRequest = (activity, request) =>
                    {
                        var clientIp = request.Headers["X-Forwarded-For"].FirstOrDefault() ?? request.HttpContext.Connection.RemoteIpAddress?.ToString();
                        if (!string.IsNullOrEmpty(clientIp)) activity.SetTag("client_ip", clientIp);
                    };
                })
#if DEBUG
                .AddConsoleExporter()
#else
                .AddOtlpExporter(o =>
                {
                    o.Endpoint = new Uri(otelCollectorEndpoint);
                })
#endif
        )
        .WithMetrics(metricsBuilder => metricsBuilder
                .AddRuntimeInstrumentation()
                .AddAspNetCoreInstrumentation()
#if DEBUG
            .AddConsoleExporter()
#else
                .AddOtlpExporter(o =>
                {
                    o.Endpoint = new Uri(otelCollectorEndpoint);
                })
#endif
        );

最后项目运行起来在OpenObserve的数据流中有对应数据就算成功了

禁用HealthCheck的Tracing Metrics

禁用tracing：
在AddAspNetCoreInstrumentation（options => 方法中配置Filter，请求的Path不以/health开头

options.Filter = context => context.Request.Path.Value?.StartsWith("/health") is not true;

禁用metrics： 直接调用DisableHttpMetrics

app.MapHealthChecks("/health").DisableHttpMetrics();

日志的话因为是属于http请求日志，暂时没有太好的办法单独禁用掉健康检查端点的日志。