Skip to content

LearnOperate

Throughput

How much work a system completes per unit of time, such as pages ingested per hour or searches served per second, as opposed to how long one item takes.

In DocSlurpGlossary entry

On this page

Throughput is work completed per unit of time: pages ingested per hour, searches served per second, embeddings computed per minute. Latency is how long one item takes. They are different questions, and improving one often hurts the other.

A worker that processes one 10-page document at a time, 30 seconds each, has a throughput of 20 pages per minute and a latency of 30 seconds per document. Run four documents at once on the same machine and, if CPU allows, throughput approaches 80 pages per minute. Each document may now take 40 seconds because they compete for CPU and memory. More work gets done; each piece waits longer.

Little’s law ties the three together for any stable system: items in progress = throughput × time per item. With 4 documents in progress and 40 seconds each, throughput is 4 / 40 = 0.1 documents per second, or 6 documents (60 pages) per minute. To raise throughput without raising latency, you need more capacity, not just more items in flight.

The gotcha

Throughput has a ceiling set by the slowest shared resource: CPU, memory, a database connection pool, or a model provider’s rate limit. Past that point, adding concurrency only grows queues and latency. A healthy system measures where the ceiling is and applies backpressure before it, rather than accepting work it can’t finish. For ingestion, report throughput as pages per minute with its scope and concurrency stated.

In DocSlurp

In DocSlurp

Each worker processes several documents at once (WORKER_DOC_CONCURRENCY, by default one per two cores, at most 4) across twice as many activity slots, and backs off under memory or CPU pressure (core/worker/src/concurrency.ts, core/worker/src/pipeline/adaptive-tuning.ts).

Sources

Jump to