使用OpenSearch Observability Stack官方示例分析AI agent可观测性实现的实践
本文经过AI辅助编写，请谨慎参考
OpenSearch Observability Stack 是 OpenSearch 项目下的一站式可观测性平台，其核心定位为：完全遵循 Apache 2.0 协议，无厂商锁定风险。所有数据摄入走 OTLP 协议，遵循 OTel Semantic Conventions，并且是第一个将 AI Agent 追踪、评估作为核心能力的开源可观测性平台
本文使用官方的示例启动测试环境，总体架构如下
flowchart TB
subgraph L1["第一层：应用层"]
A1["weather-agent<br/>:8000"] --- A2["events-agent<br/>:8002"] --- A3["travel-planner<br/>:8003"] --- A4["canary"]
L1_DESC["职责：埋点、生成 Span/Log/Metric"]
end
subgraph L2["第二层：接收层"]
OTel["OTel Collector<br/>gRPC :4317 | HTTP :4318<br/>metrics :8888"] --- L2_DESC["职责：接收、批处理、路由遥测数据"]
end
subgraph L3["第三层：处理层"]
DP["Data Prepper<br/>gRPC :21890 | HTTP :21892"] --- L3_DESC["职责：Trace 处理、Service Map 生成、写入存储"]
end
subgraph L4["第四层：存储层"]
OS["OpenSearch<br/>:9200"] --- Cortex["Cortex/Prometheus<br/>:9090"] --- L4_DESC["职责：持久化存储、索引、查询"]
end
subgraph L5["第五层：展示层"]
OSD["OpenSearch Dashboards<br/>:5601"] --- L5_DESC["职责：可视化 UI、Agent Traces 插件、告警管理"]
end
L1 -->|"OTLP gRPC/HTTP"| L2
L2 -->|"OTLP"| L3
L3 -->|"HTTPS"| OS
L3 -->|"Remote Write"| Cortex
OS -->|"PPL Query"| OSD
Cortex -->|"PromQL"| OSD
style L1 fill:#e1f5fe
style L2 fill:#fff3e0
style L3 fill:#f3e5f5
style L4 fill:#e8f5e9
style L5 fill:#fce4ec
为了理解整个系统的工作方式，追踪一条 Trace 从产生到展示的完整路径如下
sequenceDiagram
autonumber
participant User as 用户请求
participant App as weather-agent<br/>(应用层)
participant SDK as OTel SDK<br/>observe/enrich
participant Collector as OTel Collector<br/>(接收层)
participant Prepper as Data Prepper<br/>(处理层)
participant OS as OpenSearch<br/>(存储层)
participant Dash as Dashboards<br/>(展示层)
User->>App: POST /invoke
activate App
Note over App,SDK: 1. 应用层：创建 Span
App->>SDK: with observe("Weather Assistant")
SDK-->>App: gen_ai.operation.name = invoke_agent
SDK-->>App: gen_ai.request.model = gpt-4
SDK-->>App: gen_ai.usage.input_tokens = 150
SDK-->>App: gen_ai.usage.output_tokens = 75
Note over App,Collector: 2. OTLP gRPC 导出
App->>Collector: OTLPSpanExporter.batch_export()
Note over Collector: receivers.otlp.grpc.endpoint: :4317
Note over Collector: processors: [resourcedetection, transform, batch]
Note over Collector: exporters.otlp/opensearch: data-prepper
Note over Collector,Prepper: 3. Data Prepper 路由
Collector->>Prepper: OTLP → otlp-pipeline
Prepper->>Prepper: route(TRACE) → otel-traces-pipeline
Prepper->>Prepper: traces-raw-pipeline + service-map-pipeline
Note over Prepper,OS: 4. 写入 OpenSearch
Prepper->>OS: PUT /otel-v1-apm-span-000001/_doc/{spanId}
Note over OS: {<br/> "traceId": "4bf92f35...",<br/> "name": "invoke_agent",<br/> "attributes.gen_ai.agent.name": "Weather Asst"<br/>}
Note over OS,Dash: 5. 可视化查询
OS-->>Dash: PPL Query Results
Note over Dash: Agent Traces UI<br/>• Traces 列表<br/>• Span Tree<br/>• DAG Flow<br/>• Timeline
App-->>User: 响应结果
deactivate App
应用层——遥测数据的产生
OTEL SDK 初始化
应用要产生遥测数据，首先需要初始化 OpenTelemetry SDK。这涉及到三个核心概念
| 概念 | 职责 | 类比 |
|---|---|---|
| TracerProvider | 创建和管理 Tracer，配置导出器 | 类似日志系统的 LoggerProvider |
| MeterProvider | 创建和管理 Meter，配置指标导出 | 类似 Metrics Registry |
| LoggerProvider | 管理日志记录器的导出 | 类似日志 Handler |
examples/plain-agents/weather-agent/main.py中 register() 一行替代约 30 行手动配置：
- TracerProvider 创建
- OTLPSpanExporter 配置
- BatchSpanProcessor 添加
- trace.set_tracer_provider() 注册
from opentelemetry import trace, metrics
from opentelemetry.trace import SpanKind, Status, StatusCode
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry._logs import set_logger_provider
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import OTLPLogExporter
# OpenSearch Gen-AI Observability SDK
from opensearch_genai_observability_sdk_py import Op, enrich, observe, register
def setup_telemetry(service_name: str, service_version: str, otlp_endpoint: str):
# 使用 opensearch-genai-observability-sdk-py 简化配置
register(
endpoint=f"grpc://{otlp_endpoint.replace('http://', '')}",
service_name=service_name,
service_version=service_version,
)
# Metrics 需要手动配置（SDK 仅处理 Tracing）
resource = Resource.create({
"service.name": service_name,
"service.version": service_version,
"deployment.environment": "development"
})
# 创建 OTLP 指标导出器
otlp_metric_exporter = OTLPMetricExporter(
endpoint=otlp_endpoint, # 指向 OTel Collector
insecure=True # 开发环境不使用 TLS
)
# 配置周期性导出（每 2 秒导出一次）
metric_reader = PeriodicExportingMetricReader(
otlp_metric_exporter,
export_interval_millis=2000
)
meter_provider = MeterProvider(
resource=resource,
metric_readers=[metric_reader]
)
metrics.set_meter_provider(meter_provider)
meter = metrics.get_meter(__name__)
# Logging 初始化
logger_provider = LoggerProvider(resource=resource)
otlp_log_exporter = OTLPLogExporter(endpoint=otlp_endpoint, insecure=True)
logger_provider.add_log_record_processor(BatchLogRecordProcessor(otlp_log_exporter))
set_logger_provider(logger_provider)
# 将 OTel LoggingHandler 附加到 Python logging
handler = LoggingHandler(level=logging.INFO, logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
return meter, logging.getLogger(__name__)
配置示例：.env 文件中的端点配置
# OpenTelemetry Collector 端点配置
OTEL_COLLECTOR_HOST=otel-collector
OTEL_COLLECTOR_PORT_GRPC=4317
OTEL_COLLECTOR_PORT_HTTP=4318
# 应用通过环境变量获取端点
# OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
Span 的创建与属性设置
Span 是 Trace 的基本单位，代表一个操作单元。在 Gen-AI 场景下，OpenTelemetry 定义了专门的语义约定（Semantic Conventions）：
| 操作类型 | gen_ai.operation.name | 含义 |
|---|---|---|
| Agent 调用 | invoke_agent |
Agent 编排步骤 |
| LLM 调用 | chat |
Chat Completion 调用 |
| 工具执行 | execute_tool |
工具函数调用 |
| 向量嵌入 | embeddings |
向量嵌入操作 |
| 数据检索 | retrieval |
RAG 检索操作 |
使用 @observe 装饰器创建 Span
# examples/plain-agents/weather-agent/main.py
from opentelemetry.trace import SpanKind
from opensearch_genai_observability_sdk_py import Op, enrich, observe
class WeatherAgent:
def invoke(self, user_message: str, conversation_id: str, fault=None) -> str:
# 创建 invoke_agent Span
# observe() 是 SDK 提供的上下文管理器，自动处理：
# 1. Span 创建
# 2. Span 结束时的导出
# 3. 错误捕获和状态设置
with observe(self.agent_name, op=Op.INVOKE_AGENT, kind=SpanKind.CLIENT) as span:
# 设置 Gen-AI 语义约定属性
# enrich() 批量设置 gen_ai.* 属性，替代手动 set_attribute() 调用
enrich(
# 模型信息
model=self.model, # gen_ai.request.model
provider=provider, # gen_ai.provider.name
# Agent 信息
agent_id=self.agent_id, # gen_ai.agent.id
agent_description=self.agent_description,
# 会话信息
session_id=conversation_id, # gen_ai.conversation.id
# 生成参数
temperature=0.7,
max_tokens=1024,
# 上下文信息
system_instructions=json.dumps(system_instructions),
tool_definitions=self.tools, # gen_ai.tool.definitions
input_messages=input_messages, # gen_ai.input.messages
)
llm_response = call_llm(self.model, messages, self.tools)
# 设置响应属性
enrich(
response_id=llm_response["id"],
finish_reason=llm_response["choices"][0]["finish_reason"],
input_tokens=llm_response["usage"]["prompt_tokens"],
output_tokens=llm_response["usage"]["completion_tokens"],
)
# 执行工具调用
if tool_call:
tool_result = self.execute_tool(tool_name, tool_args, tool_call_id)
# 设置最终输出
enrich(output_messages=[
{"role": "assistant", "parts": [{"type": "text", "content": final_response}]}
])
return final_response
工具执行的追踪
execute_tool 方法的实现
# examples/plain-agents/weather-agent/main.py
import json
from typing import Dict
from opensearch_genai_observability_sdk_py import Op, observe
def execute_tool(self, tool_name: str, arguments: Dict, tool_call_id: str, fault=None) -> Dict:
"""
执行工具调用，创建 execute_tool Span
"""
# 创建 execute_tool Span
with observe(tool_name, op=Op.EXECUTE_TOOL) as span:
# 设置工具类型和调用 ID
span.set_attribute("gen_ai.tool.type", "function")
span.set_attribute("gen_ai.tool.call.id", tool_call_id)
# 设置工具定义（从预定义的工具列表中获取）
tool_def = next((t for t in self.tools if t["function"]["name"] == tool_name), None)
if tool_def:
span.set_attribute("gen_ai.tool.description", tool_def["function"]["description"])
# 记录调用参数
span.set_attribute("gen_ai.tool.call.arguments", json.dumps(arguments))
# 执行工具逻辑
if tool_name == "get_current_weather":
result = get_weather(arguments["location"])
elif tool_name == "get_forecast":
result = get_forecast(arguments["location"], arguments.get("days", 3))
# 记录执行结果
span.set_attribute("gen_ai.tool.call.result", json.dumps(result))
return result
OTel Collector 接收到的 Span 数据如下
# otel-collector 日志输出
2026-05-21T04:04:14.123Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 3}
Span #0
Trace ID : 4bf92f3577b34da6a3ce929d0e0e4736
Span ID : 00f067aa0ba902b7
Parent :
Name : invoke_agent
Kind : Client
Start : 2026-05-21 04:04:14.123 +0000 UTC
End : 2026-05-21 04:04:15.890 +0000 UTC
Attributes:
gen_ai.operation.name: invoke_agent
gen_ai.agent.name: Travel Planner
gen_ai.request.model: gpt-4.1-mini
gen_ai.usage.input_tokens: 150
gen_ai.usage.output_tokens: 75
Trace Context 传播
在微服务或多 Agent 架构中，一次请求可能跨越多个服务。为了将不同服务产生的 Span 串联成一条完整的 Trace，需要在服务间传递 Trace Context。W3C TraceContext 标准定义了 traceparent Header：
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
│ │ │ │
│ │ │ └── flags (01 = sampled)
│ │ └── parent-span-id
│ └── trace-id (全局唯一，32位十六进制)
└── version (固定为 00)
HTTP 客户端自动传播 Trace Context
# examples/plain-agents/multi-agent-planner/orchestrator/main.py
import httpx
from opentelemetry.trace import SpanKind
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opensearch_genai_observability_sdk_py import Op, observe
# HTTPXClientInstrumentor 自动为所有 HTTP 请求添加追踪：
# 1. 创建 CLIENT Span
# 2. 注入 traceparent Header
# 3. 记录 HTTP 状态码
HTTPXClientInstrumentor().instrument()
async def invoke_sub_agents(request: PlanRequest):
"""调用子 Agent"""
async with httpx.AsyncClient(timeout=30.0) as client:
# 调用 weather-agent
# HTTPX Instrumentor 自动：
# 1. 从当前 Context 提取 Trace ID
# 2. 创建新的 Span（parent-id 指向当前 Span）
# 3. 在 HTTP Headers 中注入 traceparent
with observe("weather-agent", op=Op.INVOKE_AGENT, kind=SpanKind.CLIENT) as span:
span.set_attribute("gen_ai.agent.name", "weather-agent")
resp = await client.post(
f"{WEATHER_AGENT_URL}/invoke",
json={"message": f"What's the weather in {request.destination}?"}
)
# Headers 中自动包含：
# traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-<new-span-id>-01
FastAPI 服务端提取 Trace Context
# examples/plain-agents/weather-agent/server.py
from fastapi import FastAPI
from opentelemetry.instrumentation.asgi import OpenTelemetryMiddleware
# 创建 FastAPI 应用
inner_app = FastAPI(title="Weather Agent API")
# OTel 中间件
# OpenTelemetryMiddleware 自动处理：
# 1. 从 HTTP Headers 提取 traceparent
# 2. 恢复上级 Span 的 Context
# 3. 为当前 HTTP 请求创建 SERVER Span
# 4. 设置 http.method, http.url, http.status_code 等属性
app = OpenTelemetryMiddleware(inner_app)
日志输出：跨服务的 Trace Context 传播
# travel-planner 日志
04:04:14.123 [trace-id: 4bf92f35...] Creating invoke_agent span (Travel Planner)
# HTTP request to weather-agent
04:04:14.156 [trace-id: 4bf92f35...] Injecting traceparent: 00-4bf92f35...-abc123-01
# weather-agent 日志
04:04:14.178 [trace-id: 4bf92f35...] Extracted traceparent from headers
04:04:14.178 [trace-id: 4bf92f35...] Creating invoke_agent span (Weather Assistant)
Metrics 的生成与导出
Metrics 是聚合的数值数据，适合记录计数、延迟、资源使用等。与 Traces 的每个 Span 独立记录不同，Metrics 会进行时间窗口聚合。
Token 使用量 Metrics示例代码
# examples/plain-agents/weather-agent/main.py
import time
from opentelemetry import metrics
class WeatherAgent:
def __init__(self, meter, logger):
# 创建 Counter
# Counter 是只能递增的指标，适合计数场景
self.token_counter = meter.create_counter(
name="gen_ai.client.token.usage",
description="Number of tokens used in LLM operations",
unit="token"
)
# 创建 Histogram
# Histogram 用于记录分布数据，自动计算分位数
self.operation_duration = meter.create_histogram(
name="gen_ai.client.operation.duration",
description="Duration of agent operations",
unit="s"
)
def invoke(self, user_message: str, conversation_id: str, fault=None) -> str:
# ... 执行 Agent 逻辑 ...
# 记录 Token 消耗
self.token_counter.add(
llm_response["usage"]["prompt_tokens"],
attributes={
"gen_ai.operation.name": "invoke_agent",
"gen_ai.provider.name": "openai",
"gen_ai.request.model": self.model,
"gen_ai.response.model": llm_response["model"],
"gen_ai.token.type": "input", # 区分输入/输出 Token
}
)
# 记录操作延迟
duration = time.time() - start_time
self.operation_duration.record(
duration,
attributes={
"gen_ai.operation.name": "invoke_agent",
"gen_ai.provider.name": "openai",
}
)
Prometheus 抓取配置
# docker-compose/prometheus/prometheus.yml
global:
scrape_interval: 60s
evaluation_interval: 60s
# OTLP 配置：接收来自 OTel Collector 的 Metrics
otlp:
keep_identifying_resource_attributes: true
promote_resource_attributes:
- service.instance.id
- service.name
- service.namespace
- service.version
# Gen-AI 语义约定属性
- gen_ai.agent.id
- gen_ai.agent.name
- gen_ai.provider.name
- gen_ai.request.model
scrape_configs:
- job_name: 'otel-collector'
static_configs:
- targets: ['otel-collector:8888']
scrape_interval: 10s
接收层——OTel Collector
OTel Collector 是遥测数据的"交通枢纽"，其核心职责是：
- 接收：从多种来源接收数据（OTLP、Prometheus、Jaeger 等）
- 处理：对数据进行转换、批处理、过滤
- 导出：将数据发送到多种后端（OpenSearch、Prometheus、Jaeger 等）
完整的 Collector 架构配置
# docker-compose/otel-collector/config.yaml
# ==================== Receivers: 数据接收器 ====================
receivers:
otlp:
protocols:
# gRPC 接收器 — 高性能二进制协议
grpc:
endpoint: 0.0.0.0:4317
# HTTP 接收器 — 易调试，支持浏览器
http:
endpoint: 0.0.0.0:4318
cors:
allowed_origins:
- "http://*"
- "https://*"
# 自监控：抓取 Collector 自身的 Metrics
prometheus/self:
config:
scrape_configs:
- job_name: otel-collector
static_configs:
- targets: ["localhost:8888"]
# ==================== Processors: 数据处理器 ====================
processors:
# 1. 内存限制 — 防止 OOM
memory_limiter:
check_interval: 5s
limit_percentage: 80 # 内存超过 80% 开始丢弃
spike_limit_percentage: 25 # 允许 25% 突发增长
# 2. 批处理 — 提高传输效率
batch:
timeout: 10s # 10 秒强制发送
send_batch_size: 1024 # 累积 1024 条发送
# 3. 资源检测 — 添加环境信息
resourcedetection:
detectors: [env, system]
# 4. 数据转换 — 处理字段冲突
transform:
error_mode: ignore
trace_statements:
- context: span
statements:
# 扁平化嵌套属性，避免 OpenSearch mapping 冲突
- set(attributes["db_system_name"], attributes["db.system.name"])
where attributes["db.system.name"] != nil
- delete_key(attributes, "db.system.name")
# ==================== Exporters: 数据导出器 ====================
exporters:
# 调试导出器 — 输出到日志
debug:
verbosity: detailed
# OTLP 导出器 — 发送到 Data Prepper
otlp/opensearch:
endpoint: "data-prepper:21890"
tls:
insecure: true
# Prometheus Remote Write — 发送到 Cortex
prometheusremotewrite/cortex:
endpoint: "http://prometheus:9090/api/v1/push"
resource_to_telemetry_conversion:
enabled: true # 将 Resource Attributes 转为 Metric Labels
# ==================== Service: 管道定义 ====================
service:
pipelines:
# Traces 管道
traces:
receivers: [otlp]
processors: [resourcedetection, memory_limiter, transform, batch]
exporters: [otlp/opensearch, debug]
# Metrics 管道
metrics:
receivers: [otlp, prometheus/self]
processors: [resourcedetection, memory_limiter, batch]
exporters: [prometheusremotewrite/cortex, debug]
# Logs 管道
logs:
receivers: [otlp]
processors: [resourcedetection, memory_limiter, transform, batch]
exporters: [otlp/opensearch, debug]
Transform Processor 工作原理
Transform Processor 是 Collector 中最强大的处理器之一，它使用 OTTL（OpenTelemetry Transformation Language）对数据进行转换。
为什么需要 Transform？因为 OpenSearch 在处理嵌套属性时会将其解析为对象，导致 Mapping 冲突。例如：
原始数据:
db.system.name: "postgresql" # OpenSearch 解析为 db.system.name 对象
后续数据:
db_system_name: "mysql" # 已经存在 db 对象，无法设置为字符串
# 结果: Mapping conflict!
Transform 规则详解
# docker-compose/otel-collector/config.yaml
transform:
error_mode: ignore # 遇到错误继续处理，不中断
trace_statements:
- context: span
statements:
# ==================== 扁平化嵌套属性 ====================
# 将 db.system.name 重命名为 db_system_name
- set(attributes["db_system_name"], attributes["db.system.name"])
where attributes["db.system.name"] != nil
- delete_key(attributes, "db.system.name")
# 将 code.function.name 重命名为 code_function_name
- set(attributes["code_function_name"], attributes["code.function.name"])
where attributes["code.function.name"] != nil
- delete_key(attributes, "code.function.name")
# ==================== URL 路径标准化 ====================
# 将动态路径参数标准化为模板
# /api/products/123 → /api/products/{productId}
- replace_match(name, "GET /api/products/*", "GET /api/products/{productId}")
# 移除 URL 查询参数（避免路径爆炸）
- replace_pattern(name, "\\?.*", "")
Pipeline 路由逻辑
Pipeline 定义了数据的流转路径：Receivers → Processors → Exporters。一个 Receiver 可以发送到多个 Pipeline，一个 Pipeline 也可以有多个 Exporter。
# 多个 Pipeline 共享同一个 Receiver
service:
pipelines:
traces:
receivers: [otlp] # OTLP 接收
processors: [memory_limiter, batch] # 处理
exporters: [otlp/opensearch, debug] # 导出到 Data Prepper 和日志
metrics:
receivers: [otlp, prometheus/self] # OTLP + 自抓取
processors: [memory_limiter, batch]
exporters: [prometheusremotewrite/cortex, debug] # 导出到 Cortex
Collector 处理日志
# otel-collector 启动日志
2026-05-21T04:00:00.000Z info service/telemetry.go:94 Setting up own telemetry...
2026-05-21T04:00:00.001Z info service/telemetry.go:124 Serving Prometheus metrics {"address": ":8888"}
# 接收到 Trace 数据
2026-05-21T04:04:14.123Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "otlp/opensearch", "spans": 3}
# 处理过程中应用 Transform
2026-05-21T04:04:14.124Z debug transform Applied 5 transformation statements
# 批处理发送
2026-05-21T04:04:24.130Z info batch_processor Exported batch {"items": 156, "bytes": 24567}
处理层——Data Prepper
Data Prepper 的核心功能
Data Prepper 是 OpenSearch 项目下的数据预处理引擎，专门为可观测性数据设计。其核心功能包括：
- Trace 处理：解析 OTLP Span 格式，提取关键信息
- Service Map 生成：从 Trace 中推导服务依赖关系
- RED Metrics 计算：计算 Rate（请求速率）、Errors（错误率）、Duration（延迟）
Pipeline 架构
# docker-compose/data-prepper/pipelines.template.yaml
# ==================== 主路由 Pipeline ====================
# 接收所有 OTLP 数据，按类型分流
otlp-pipeline:
delay: 10
source:
otlp:
port: 21890 # 接收来自 OTel Collector 的数据
ssl: false
# 路由规则：根据事件类型分发
route:
- logs: 'getEventType() == "LOG"'
- traces: 'getEventType() == "TRACE"'
# 分发到子 Pipeline
sink:
- pipeline:
name: "otel-logs-pipeline"
routes: ["logs"]
- pipeline:
name: "otel-traces-pipeline"
routes: ["traces"]
# ==================== Logs 处理 Pipeline ====================
otel-logs-pipeline:
workers: 5
delay: 10
source:
pipeline:
name: "otlp-pipeline"
processor:
# 复制 time 字段到 @timestamp（OpenSearch 默认时间字段）
- copy_values:
entries:
- from_key: "time"
to_key: "@timestamp"
sink:
- opensearch:
hosts: ["${OPENSEARCH_PROTOCOL}://${OPENSEARCH_HOST}:${OPENSEARCH_PORT}"]
username: "${OPENSEARCH_USER}"
password: "${OPENSEARCH_PASSWORD}"
insecure: true
# OpenSearch 内置索引类型，自动管理索引生命周期
index_type: log-analytics-plain
# ==================== Traces 处理 Pipeline ====================
otel-traces-pipeline:
delay: 100
source:
pipeline:
name: "otlp-pipeline"
# Trace 数据分流：原始存储 + Service Map 生成
sink:
- pipeline:
name: "traces-raw-pipeline"
- pipeline:
name: "service-map-pipeline"
# ==================== Raw Trace 存储 ====================
traces-raw-pipeline:
source:
pipeline:
name: "otel-traces-pipeline"
processor:
# otel_traces 处理器：解析 OTLP 格式
- otel_traces:
sink:
- opensearch:
hosts: ["${OPENSEARCH_PROTOCOL}://${OPENSEARCH_HOST}:${OPENSEARCH_PORT}"]
username: "${OPENSEARCH_USER}"
password: "${OPENSEARCH_PASSWORD}"
insecure: true
# 索引类型：trace-analytics-plain-raw
index_type: trace-analytics-plain-raw
# ==================== Service Map 生成 ====================
service-map-pipeline:
delay: 100
source:
pipeline:
name: "otel-traces-pipeline"
processor:
# 核心处理器：从 Trace 中提取服务拓扑
- otel_apm_service_map:
group_by_attributes: [telemetry.sdk.language]
window_duration: 10s # 10 秒窗口聚合
sink:
# 1. Service Map 数据 → OpenSearch
- opensearch:
hosts: ["${OPENSEARCH_PROTOCOL}://${OPENSEARCH_HOST}:${OPENSEARCH_PORT}"]
username: "${OPENSEARCH_USER}"
password: "${OPENSEARCH_PASSWORD}"
insecure: true
index_type: otel-v2-apm-service-map
# 2. RED Metrics → Cortex
- pipeline:
name: "service-metrics-cortex-pipeline"
Service Map 生成原理
Service Map（服务拓扑图）展示了服务间的调用关系。Data Prepper 通过分析 Trace 中的 Span 关系来推导：
- 服务节点：从
resource.attributes.service.name提取 - 调用边：从 Span 的
parentSpanId和serviceName关系推导 - RED 指标：统计每个服务的请求速率、错误率、延迟
处理流程：
输入: Trace 数据流
│
├── Span A (service: travel-planner, parentSpanId: "")
├── Span B (service: weather-agent, parentSpanId: A.id)
└── Span C (service: events-agent, parentSpanId: A.id)
│
▼
otel_apm_service_map 处理器
│
├── 提取服务节点: [travel-planner, weather-agent, events-agent]
├── 推导调用关系: travel-planner → weather-agent, travel-planner → events-agent
└── 计算 RED 指标: rate=10/s, error=0.1%, p99=250ms
│
▼
输出:
├── Service Map 文档 (OpenSearch)
└── RED Metrics (Cortex)
OpenSearch 索引结构
Data Prepper 使用 OpenSearch 内置的索引类型，自动管理索引生命周期：
| 索引类型 | 索引名称模式 | 数据内容 |
|---|---|---|
log-analytics-plain |
logs-otel-v1-000001 |
Log 记录 |
trace-analytics-plain-raw |
otel-v1-apm-span-000001 |
Trace Span |
otel-v2-apm-service-map |
otel-v2-apm-service-map-000001 |
Service Map |
索引创建和写入
# 检查 OpenSearch 索引
$ curl -k -u admin:My_password_123!@# https://localhost:9200/_cat/indices?v
health status index docs.count store.size
green open otel-v1-apm-span-000001 156 1.2mb
green open logs-otel-v1-000001 89 256kb
green open otel-v2-apm-service-map-000001 12 64kb
索引文档结构示例：
// otel-v1-apm-span-000001 文档
{
"traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
"spanId": "00f067aa0ba902b7",
"parentSpanId": "",
"name": "invoke_agent",
"kind": "CLIENT",
"startTime": "2026-05-21T04:04:14.123Z",
"endTime": "2026-05-21T04:04:15.890Z",
"durationInNanos": 1767000000,
"status.code": "OK",
"status.message": "",
// Gen-AI 语义约定属性
"attributes.gen_ai.operation.name": "invoke_agent",
"attributes.gen_ai.agent.name": "Travel Planner",
"attributes.gen_ai.agent.id": "travel-planner-001",
"attributes.gen_ai.request.model": "gpt-4.1-mini",
"attributes.gen_ai.usage.input_tokens": 150,
"attributes.gen_ai.usage.output_tokens": 75,
// 服务信息
"resource.attributes.service.name": "travel-planner",
"resource.attributes.service.version": "1.0.0"
}
存储层——OpenSearch 与 Cortex
OpenSearch 的角色
OpenSearch 是 Trace 和 Log 数据的存储后端，提供：
- 全文检索：对 Span 属性进行高效搜索
- 聚合分析：统计 Token 用量、延迟分布等
- 时序索引：按时间滚动创建索引，便于数据管理
OpenSearch 服务定义
# docker-compose.local-opensearch.yml
services:
opensearch:
image: ${OPENSEARCH_DOCKER_REPO}/opensearch:${OPENSEARCH_VERSION}
container_name: opensearch
environment:
- cluster.name=observability-stack-cluster
- node.name=observability-stack-node
- discovery.type=single-node
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
- "OPENSEARCH_INITIAL_ADMIN_PASSWORD=admin" # 初始密码
volumes:
- opensearch-data:/usr/share/opensearch/data
ports:
- "9200:9200"
- "9600:9600" # 性能分析器端口
healthcheck:
test: curl -s -k -u admin:${OPENSEARCH_PASSWORD} https://localhost:9200/_cluster/health | grep -E '"status":"(green|yellow)"'
start_period: 30s
interval: 5s
timeout: 10s
retries: 30
索引生命周期管理
OpenSearch 使用 ISM（Index State Management）管理索引生命周期：
- Rollover：当索引达到大小或时间阈值时滚动
- Delete：自动删除过期数据
ISM 策略通过初始化脚本设置
# .env 文件
ISM_RETENTION_DAYS=7 # 数据保留 7 天
# 初始化脚本创建的 ISM 策略
# otel-v1-apm-span-* 索引：rollover at 50GB/24h, delete after 7 days
# logs-otel-v1-* 索引：rollover at 50GB/24h, delete after 7 days
展示层——OpenSearch Dashboards
observabilityDashboards 插件
OpenSearch Dashboards 通过 observabilityDashboards 插件提供专门的 Agent Traces UI。这个插件是预装在 opensearchstaging/opensearch-dashboards 镜像中的。
检查已安装插件
$ docker exec opensearch-dashboards bin/opensearch-dashboards-plugin list
customImportMapDashboards@3.7.0.0
mlCommonsDashboards@3.7.0.0
observabilityDashboards@3.7.0.0 # ← 可观测性插件
queryInsightsDashboards@3.7.0.0
searchRelevanceDashboards@3.7.0.0
securityDashboards@3.7.0.0
插件结构：
/usr/share/opensearch-dashboards/plugins/observabilityDashboards/
├── server/
│ ├── routes/ # API 路由
│ │ ├── traces.ts # Trace 查询 API
│ │ └── service_map.ts
│ ├── services/ # 业务逻辑
│ └── saved_objects/ # 保存对象管理
├── common/ # 前端共享代码
└── target/ # 编译后的前端资源
配置启用 Agent Traces UI
opensearch_dashboards.yml
# docker-compose/opensearch-dashboards/opensearch_dashboards.template.yml
# ==================== Explore UI 配置 ====================
explore.enabled: true
explore.discoverTraces.enabled: true # Trace 发现
explore.discoverMetrics.enabled: true # Metrics 发现
explore.agentTraces.enabled: true # ⭐ Agent Traces UI
# ==================== 可观测性插件配置 ====================
observability.alertManager.enabled: true # Alert Manager UI
# ==================== 工作区配置 ====================
workspace.enabled: true
data_source.enabled: true
# ==================== UI 设置覆盖 ====================
uiSettings.overrides.home:useNewHomePage: true
uiSettings.overrides.query:enhancements:enabled: true
uiSettings.overrides.explore:experimental: true
Agent Traces UI 功能
Agent Trace 列表
-
显示时间、Agent 名称、状态、延迟、Token 数
-
Span 树层级展示 Span 父子关系，可展开/折叠
服务拓扑图
trace和日志关联
prometheus指标接入
PPL 查询示例
PPL（Piped Processing Language）是 OpenSearch 的管道式查询语言，语法简洁，适合探索性分析。
浙公网安备 33010602011771号