Skip to content

LearnOperate

Latency percentiles

p50 · p95 · p99 · Tail latency

Describing response times by rank instead of average: p95 is the time that 95% of requests finished within, so the slow tail stays visible.

In DocSlurpIllustrated explainer

On this page
Histogram of 100 request latencies with mean and percentile markersIllustrative data. Of 100 requests, 5 take the slow path. Mean 122.6 ms, p50 61 ms, p95 148 ms, p99 1,320 ms. Most requests cluster between 24 and 148 milliseconds; the slow ones sit far to the right, between 1,000 and 1,500 milliseconds.02505007501,0001,2501,500response time (ms)25 ms buckets · requests per bucketp50 61 msmean 122.6 msp95 148 msp99 1,320 ms
The mean hides the tail; p99 shows it. With 5 slow requests in 100, the median is 61 ms and p95 is still 148 ms, but p99 is 1,320 ms and the mean, 122.6 ms, is a time almost no request took. Slide toward 10 and p95 jumps too; slide to 0 or 1 and p99 drops back. Data is illustrative.

The problem: a healthy average and angry users

Your search dashboard says the average response time is 123 ms. That sounds fine. Meanwhile people complain that search “sometimes hangs for a second”. Both are true. The average blended a hundred requests into one number, and the number describes almost none of them.

Take the hundred requests behind it (illustrative numbers, as in the figure). Ninety-five took between 24 and 148 ms. Five took over a second, because they missed a cache and waited on a cold embedding call. The mean, 122.6 ms, is double what a typical request took, and far below what the slow ones took. Nobody experienced 123 ms.

Percentiles rank instead of averaging

Sort the requests from fastest to slowest. The p-th percentile is the time below which p% of them finished.

  • p50 (the median) is what a typical request sees.
  • p95 is what the slowest 1 in 20 requests see or worse.
  • p99 is the slowest 1 in 100.

With the nearest-rank method and n requests, p is the value at position ⌈p/100 × n⌉ in the sorted list. With 100 requests that is simply the 50th, 95th and 99th value. For the example:

Slow requests out of 100 Mean p50 p95 p99
0 63.7 ms 60 ms 105 ms 128 ms
1 74.9 ms 60 ms 109 ms 148 ms
2 87.5 ms 60 ms 113 ms 1,180 ms
5 122.6 ms 61 ms 148 ms 1,320 ms
10 181.8 ms 62 ms 1,240 ms 1,410 ms

Read down the columns. The median hardly moves: it can’t see the tail at all. The mean drifts up steadily and can’t tell you whether that is everyone getting a bit slower or a few requests getting much slower. p95 looks healthy until more than 5% of requests are slow, then jumps. p99 catches the tail as soon as 2 requests in 100 hit it.

A percentile only sees a tail fatter than its own cutoff. That is why teams track several at once, and why the figure lets you add slow requests and watch which markers move.

Why the tail matters more than it looks

One user action often makes many requests. A chat turn that runs three searches, each with a 1% chance of being slow, has about a 3% chance of feeling slow (1 − 0.99³ = 0.0297). A page that fans out to 100 backends, each slow 1% of the time, is slow 63% of the time (1 − 0.99¹⁰⁰ ≈ 0.634). Dean and Barroso call this the tail at scale: in a fan-out system, the p99 of a part becomes the median of the whole.

The tail also concentrates your heaviest users. The person with the largest project, the longest query or the coldest cache lives at p99.

Measuring them correctly

  • Don’t average percentiles. The mean of ten servers’ p99 values is not the fleet’s p99. Keep the underlying distribution, usually as a histogram with fixed buckets, and compute percentiles from the merged histogram. OpenTelemetry histograms exist for this reason.
  • Say the window and the count. “p99 over the last 5 minutes, 40 requests” is a guess: the 99th value of 40 is the slowest request. You need hundreds of samples before p99 means much.
  • Measure where users wait. Time from the client, or at least at the edge, includes queueing. A server that only times its own handler misses requests waiting for a connection or a slot, which is often where the tail lives.
  • Break the total into phases. A p99 search of 1.3 s is a symptom. Per-phase timings (embedding, vector lookup, keyword lookup, reranking) show which phase owns the tail.

Where it breaks

  • Percentiles hide the shape between them. p50 and p99 say nothing about whether the slow requests form one cluster or a long smear. Look at the histogram itself when you investigate.
  • Bucket resolution limits precision. A histogram with a 250–500 ms bucket can only say p95 is “somewhere in there”.
  • Coordinated omission. A load test that waits for each response before sending the next one sends fewer requests during a stall, so the stall is under-counted. Generate load on a fixed schedule instead.

When to use them

Use p50 for “what is normal”, p95 or p99 for alerts and service-level objectives, and the maximum only as a hint for investigation. Report percentiles alongside throughput: a system can keep p99 low by refusing work, which is backpressure doing its job, or by quietly serving fewer requests.

In DocSlurp

In DocSlurp

Search responses carry per-phase timings such as embed, vector and text (core/server/src/search/timings.ts). The /metrics endpoint exposes p50 and p95 pipeline stage durations (core/server/src/routes/health.ts), and request and dependency durations are exported as OpenTelemetry histograms (core/packages/shared/src/telemetry.ts).

Sources

Jump to