It exists to answer two questions, reliably and quickly, for any service at any time:
What is going on with our app? (normal operating behavior, request flow, throughput)
Is something wrong? (degradation, errors, bottlenecks, failures — and why)
Observability is not a dashboard — it is the ability to ask arbitrary questions about system behavior after the fact, without having to ship new code to answer them.
Pillars of Observability
Observability rests on three complementary signal types. None of them alone answers both core questions — they are used together.
Logs - What exactly happened, in detail, for one event
character: High context, high volume
Traces - Where time was spent across a single request's journey
character: Causal, request-scoped
Metrics - How the system behaves in aggregate, over time
character: Low context, cheap to query at scale
Worked example
A single incident, seen through all three signals:
logs: order #123: large veggie pizza burned at 8:05 PM due to oven failure.
traces: order #123 took 30 mins: 5 min prep -> 20 min cooking (delay) -> 5 min delivery.
metrics: sold 50 pizzas/hour (average cook time: 8 minutes).
The log tells us precisely what happened to order #123 and why (oven failure).
The trace shows where the 30 minutes went, and that cooking — normally ~8 minutes per the metric baseline — ballooned to 20.
The metric gives the baseline (8 min average cook time, 50 pizzas/hour) that lets us recognize the trace's 20-minute cook stage as anomalous in the first place.
This is the value of observability: logs and traces give context to the anomaly that metrics first surface. Metrics tell you something is wrong and roughly where; traces tell you which stage; logs tell you the root cause.
Why Observability (Not Just Monitoring)
Traditional monitoring answers known questions with pre-built dashboards ("is CPU high?"). Observability lets us answer unknown questions after the fact ("why did checkout latency spike for this specific customer segment at 8:05 PM, and only for orders touching the oven-scheduling service?"). It provides the context needed to find issues and bottlenecks in systems we didn't specifically build a dashboard for in advance.
OpenTelemetry?
OTel is an open, vendor-neutral standard (CNCF project) for generating, collecting, and exporting telemetry. It exists to solve two problems:
Vendor lock-in — instrument once with OTel SDKs/APIs, then export to whatever backend you choose (Grafana stack today, something else tomorrow) without re-instrumenting application code.
Unification — logs, metrics, and traces share a common data model and context, so they can be correlated instead of living in three disconnected systems
What is a trace?
A trace represents the end-to-end journey of a single request through the system, composed of one or more spans. Each span represents a unit of work (a service call, a DB query, a queue hop) with a start time, duration, and its own attributes. Spans are linked in a parent-child hierarchy, so a trace shows not just "the request took 30 minutes" but the causal breakdown of where those 30 minutes went (prep -> cooking -> delivery, in the example above).
Sampling
Capturing 100% of traces at scale is expensive and often unnecessary. Sampling decides which traces to keep.
Head sampling — the decision to keep or drop a trace is made at the start of the request (e.g., "keep 10% of all traces"), before the outcome is known. Cheap, but may discard the interesting (slow/errored) traces along with the boring ones.
Tail sampling — the decision is made after the trace completes, so it can preferentially keep traces that were slow, errored, or otherwise interesting, and discard routine ones. More expensive (requires buffering), but far more useful for incident investigation.
Guidance: default to tail sampling for production services where investigating rare, slow, or failed requests matters more than uniform coverage.
Metrics
Metrics are numerical, low-context, and fast to query even at high cardinality of time — this is what makes them suitable for always-on dashboards and alerting, unlike logs or traces which are expensive to scan broadly. Metrics are time-series data: a value, tagged with attributes, recorded at a point in time.
metrics type:
Counter - Ever increasing values(eg: total request served, total errors)
Gauge - Values that fluctuate up and down (e.g., current memory usage, active connections)
Histogram - Distribution of values within predefined buckets (e.g., request latency distribution)
UpDownCounter - Values that can increase or decrease by arbitrary amounts (e.g., queue depth, in-flight requests)
Metric model
Every metric consists of:
Name — a descriptive, hierarchical name following OTel semantic conventions, e.g.
http.server.request_count.Attributes — key-value pairs that add context (e.g.,
http.method=GET,http.status_code=500,service.name=order-api). Attributes let you slice and filter a metric without needing separate metrics per dimension.
Correlating Logs, Traces, and Metrics
The trace ID is the join key across all three signals. A request enters the system, gets a trace ID and a root span ID; every span, log line, and (via exemplars) metric sample emitted during that request carries or points back to that trace ID. This section shows the mechanics, with Go/Gin examples matching the backend-api stack.
8.2 Getting the trace ID and span into a request (Go / Gin)
Wrap the OTel otelgin middleware so every incoming request automatically starts a span, and pull the trace/span ID out of the request context wherever you need it — most importantly, in your logger.
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/trace"
"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.New()
// otelgin starts a span per request and injects it into c.Request.Context()
r.Use(otelgin.Middleware("platform-api"))
r.GET("/pipelines/:id/status", getPipelineStatus)
r.Run()
}
func getPipelineStatus(c *gin.Context) {
ctx := c.Request.Context()
span := trace.SpanFromContext(ctx)
traceID := span.SpanContext().TraceID().String()
spanID := span.SpanContext().SpanID().String()
logger.Info("fetching pipeline status",
"trace_id", traceID,
"span_id", spanID,
"pipeline_id", c.Param("id"),
)
// ... handler logic
}
8.3 Structured logs carrying trace context
Every log line must include trace_id and span_id as structured fields (not embedded in the message string) so Loki can index and filter on them directly.
import (
"log/slog"
"go.opentelemetry.io/otel/trace"
)
func logWithTraceContext(ctx context.Context, logger *slog.Logger, msg string, args ...any) {
span := trace.SpanFromContext(ctx)
sc := span.SpanContext()
baseArgs := []any{
"trace_id", sc.TraceID().String(),
"span_id", sc.SpanID().String(),
}
logger.Info(msg, append(baseArgs, args...)...)
}
// usage inside a handler or service method:
logWithTraceContext(ctx, logger, "order burned",
"order_id", "123",
"reason", "oven_failure",
)
// -> {"msg":"order burned","trace_id":"4bf92f...","span_id":"00f067...","order_id":"123","reason":"oven_failure"}
With the OTel Collector's loki exporter (or a Promtail/Loki pipeline stage), this trace_id field is automatically parsed and used to derive Loki's "Related traces" link back into Tempo — no manual correlation needed in Grafana.