Skip to content

LearnGenerate & answer

Streaming responses

Server-sent events · Token streaming

Sending a model’s answer to the client piece by piece as it is generated, usually over server-sent events, instead of waiting for the whole response.

In DocSlurpGlossary entry

On this page

A streaming response sends a model’s output as it is produced instead of all at once. A language model generates one token at a time, so a 300-word answer exists in pieces for several seconds before it is complete. Streaming forwards each piece immediately.

The usual transport is server-sent events (SSE): one long HTTP response with Content-Type: text/event-stream, made of data: lines separated by blank lines.

data: {"choices":[{"delta":{"content":"The London office"}}]}

data: {"choices":[{"delta":{"content":" opens in Q2 2027"}}]}

data: [DONE]

The client appends each delta to the screen. The total generation time doesn’t change; what changes is time to first token. A reader starts reading after a fraction of a second instead of staring at a spinner for the whole generation, and can stop a bad answer early.

Streams can also carry progress before text: “searching”, “reading 3 passages”, “writing”. For a RAG answer that spends seconds retrieving before the model writes anything, those status events are often worth more than the token stream.

The gotcha

Anything you only know at the end can’t be streamed ahead of the text. Citations resolved after generation, usage totals and final error states arrive in the last events, so clients must handle a stream that breaks halfway: partial text, then an error. Proxies that buffer responses also defeat streaming, which is why servers send headers such as X-Accel-Buffering: no. Always keep a non-streaming endpoint as a fallback.

In DocSlurp

In DocSlurp

/v1/chat/completions with stream: true sends text deltas as they arrive, citations in the last chunk, then [DONE] (core/server/src/routes/chat-completions.ts). GET /v1/chat/stream sends a status event, then the whole cited answer as one message event (core/server/src/routes/search.ts). POST /v1/chat is the non-streaming fallback.

Sources

Jump to