Skip to content

LearnOperate

Observability

Instrumenting a system with traces, metrics and logs so you can answer new questions about its behavior, such as why one search took four seconds, without shipping new code.

In DocSlurpGlossary entry

On this page

Observability is being able to explain what a running system is doing from the data it emits, including for questions you didn’t anticipate. It rests on three kinds of telemetry:

  • Metrics: numbers aggregated over time, such as request count, queue depth, and latency percentiles. Cheap to keep and good for dashboards and alerts.
  • Traces: the path of one request through the system, as a tree of timed spans. A search trace might show 15 database calls, an embedding call and a rerank, each with its duration.
  • Logs: individual events with context, such as an error message or a usage record.

They answer different questions. An alert fires because p99 search latency rose (metric). You open a slow trace from that window and see one database query waiting 3 seconds for a pool connection (trace). The log line next to it, carrying the same trace ID, shows which project and query it was. Without the shared ID, each signal is a separate haystack.

Standards

OpenTelemetry is the vendor-neutral standard for producing all three, and W3C Trace Context (the traceparent header) carries the trace ID across services, so a request from the API into a background worker stays one trace.

In DocSlurp

The server and worker export traces, metrics and logs through OpenTelemetry when an OTLP endpoint is configured, so any compatible backend can receive them; AI calls can also be sent to Langfuse. A Prometheus-format /metrics endpoint reports queue depth, oldest queued run age, p50 and p95 stage durations, and spend by provider and model. Search responses include per-phase timings.

The gotcha

Telemetry contains customer data: queries, filenames, error text. Decide what may appear in spans and logs, keep raw error details in internal telemetry rather than user-facing responses, and scope access to the backend like any other production data store.

In DocSlurp

In DocSlurp

Server and worker export OpenTelemetry traces, metrics and logs over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set, and can send AI spans to Langfuse (core/packages/shared/src/telemetry.ts). /metrics publishes queue depth, stage-duration percentiles and spend in Prometheus format (core/server/src/routes/health.ts).

Sources

Jump to