An API can return a successful response and still feel broken. A visitor clicks, waits, clicks again, and eventually leaves. Meanwhile, the server's error counter stays flat.
To investigate that situation, start with a question that names the behavior: where does the time go between receiving a request and returning its response? That question gives instrumentation a purpose.
Choose the signal for the question
Metrics summarize behavior across many operations. Traces describe the path of individual operations through spans. Logs capture events that can help explain what happened at a particular point. OpenTelemetry provides instrumentation and telemetry collection; a compatible backend stores and lets you explore the data. Its observability primer introduces these complementary signals.
For a small API, a useful investigation might start like this:
| Question | Evidence to inspect |
|---|---|
| Did the slowdown affect many requests? | Request duration distribution and traffic volume |
| Which step was slow in one affected request? | A trace containing its child spans |
| What happened inside that step? | Relevant events or correlated logs |
A dashboard that answers only the first question can confirm a problem while leaving its cause unresolved.
Read one request carefully
Consider this illustrative trace, with invented durations rather than benchmark results:
GET /api/projects 620 ms authenticate 12 ms database: select 35 ms fetch preview metadata 540 ms render response 18 ms
The external metadata request deserves attention before the database query. That is a lead, not a diagnosis. The remote service might be slow, a connection might be waiting for a free socket, or your application might be retrying a failed call.
Compare a few affected requests with ordinary ones. Look for the same operation repeatedly consuming time. Check whether child spans overlap: concurrent durations cannot simply be added to calculate the parent's elapsed time. Uninstrumented gaps also matter; a short child span does not account for everything happening around it.
This discipline prevents a familiar debugging mistake: optimizing the component you know best because it is the easiest one to change.
Keep the trace connected
When a request crosses a service boundary, context propagation carries the information needed to relate downstream work to its caller. For HTTP, W3C Trace Context commonly travels in the traceparent header. Instrumentation libraries often handle injection and extraction; custom transports may require explicit propagation. See OpenTelemetry's context propagation guide.
If the API and metadata service appear as unrelated traces, investigate that boundary before adding more spans. Check that the relevant instrumentation is active, the transport preserves the headers, and the receiving service extracts the context. A trace identifier in a log is only helpful when it refers to the same request you are investigating.
Keep sensitive data out of propagated baggage. Credentials and personal information can travel to downstream systems through it; the same guide explains this risk. Prefer operation names and carefully selected technical attributes over complete request payloads.
Make the collection useful enough to keep
Instrumentation has a cost, and retaining every trace may be unnecessary. Sampling selects which traces to retain. Head sampling decides early; tail sampling can use information available later in a trace. These choices have different operational tradeoffs, described in OpenTelemetry's sampling documentation.
Sampling also limits your conclusions. An absent trace does not prove that a request never happened. Record the collection settings alongside an investigation, particularly when comparing two environments or a before-and-after change.
For a first implementation, choose one important endpoint. Confirm that a request produces a connected trace, simulate a slow dependency in a local environment, and verify that the evidence points to it. Then make a targeted change and compare the same workload again.
A useful observability setup should make the next debugging decision easier. Start with one request you can explain from beginning to end, then expand the coverage where questions remain.