Skip to content

LearnGenerate & answer

Temperature

Sampling temperature

A sampling setting that divides a model’s token scores before softmax: low values let the likeliest token dominate, high values flatten the choice.

In DocSlurpGlossary entry

On this page

Temperature controls how a model chooses among candidate next tokens. The model outputs a score (a logit) for each token; sampling divides every score by the temperature T, then applies softmax to turn scores into probabilities.

Take three candidates with logits 2, 1 and 0:

T Divided logits Probabilities
0.5 4, 2, 0 0.867, 0.117, 0.016
1 2, 1, 0 0.665, 0.245, 0.090
2 1, 0.5, 0 0.506, 0.307, 0.186

Lower T sharpens the distribution toward the top token; higher T flattens it. As T approaches 0 the top token always wins, which is why “temperature 0” is treated as greedy decoding.

For extraction, classification and grounded answers, low temperature is the usual choice: you want the same input to give the same output. For brainstorming or varied phrasings, higher temperature helps. Related settings, top-p and top-k sampling, cut off the unlikely tail instead of reshaping the whole distribution.

The gotcha

Low temperature buys consistency, not accuracy. If the model’s top token is wrong, temperature 0 repeats that wrong answer every time, which can make an error look like a fact. Even temperature 0 is not perfectly deterministic on most hosted APIs: batching and floating-point order can change results between runs. And not every endpoint honors the parameter; some ignore it silently, so check that the setting you pass is applied.

In DocSlurp

In DocSlurp

Vision extraction and output moderation set temperature 0 (core/worker/src/media/vision.ts, core/server/src/services/chat-output-moderation.ts). Chat uses the provider’s default, and /v1/chat/completions rejects a temperature parameter as unsupported (core/server/src/routes/chat-completions.ts).

Sources

Jump to