Go 微服务可观测性trace metrics logs 三件套联动可观测性是系统健康监控的根本。本文讲清三大支柱 实际联动做法。一、可观测性三件套Metrics定点采样监控变化趋势Logs事件记录Trace请求链路追踪三者结合 完整可观测性。二、OTel 跨度OTel span 包含trace_idspan_idparent_spanattributes / events三、Metrics 设计维度例子类型Counterhttp_requests_totalCounterGaugequeue_depthGaugeHistogramhttp_durationHistogram四、Logs 结构化slog.Info(user login,user_id,42,ip,1.1.1.1)五、关联起来日志中带 trace_idslog.Info(user login,trace_id,trace.SpanFromContext(ctx).SpanContext().TraceID().String(),)Grafana Loki 支持{trace_idxxx}查询。六、Metrics 触发 TraceifrequestCount.MustGet()threshold{capture.Trace:true}七、OTel Prometheusimport(go.opentelemetry.io/otel/exporters/prometheusgo.opentelemetry.io/otel/sdk/metric)exp,_:prometheus.New()meterProvider:sdkmetric.NewMeterProvider(sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exp)))otel.SetMeterProvider(meterProvider)八、OpenTelemetry SDK 概览模块作用otelAPIotel/sdkSDKotel/exporters/jaegerJaeger exportotel/exporters/otlpOTLP grpc / http九、JAEGER 后端Jaeger collector ES 后端collector:receivers:[otlp]processors:[batch]exporters:[jaeger,kafka]十、Loki 收集日志backend:lokilabels:[job,level,trace_id]十一、Prometheus labels 与 cardinalityrequestCount.WithLabelValues(GET,/users,200).Inc()labels 不要过多否则 cardinality 爆炸。十二、Trace 与 Metrics 跨界OTel 自动生成 R.E.D 指标requests / errors / duration。十三、企业实战告警引擎-alert:HighLatencyexpr:histogram_quantile(0.99,sum(rate(http_duration_seconds_bucket[5m])) by (le))0.5labels:{severity:critical}十四、踩坑清单trace_id 不打印就关联不上cardinality 高导致 TSDB 内存爆metric 名随版本乱改十五、未来eBPF 提供内核级 metricsAI 监控自动归因十六、总结与展望三件套不是孤立。Logs 给事件Metrics 给信号Trace 给链路。协同才能构成完整可观测。十七、参考文献OpenTelemetry 文档Grafana 文档