Attention
Self-attention · Scaled dot-product attention
The transformer operation that lets each token build its representation as a weighted mix of every other token, with weights computed from dot products.
Background conceptGlossary entry
On this page
Attention is the step inside a transformer where each token updates its vector using information from the other tokens. It is how “office” in “open the London office” learns which office is meant.
Each token’s vector is turned into three new vectors: a query (what this token is looking for), a key (what it offers to others) and a value (what it passes on if chosen). For one token, attention works like this:
- Take the dot product of its query with every token’s key. Higher means a better match.
- Divide by √d (d is the key length) and apply softmax, which turns the scores into positive weights that sum to 1.
- Output the weighted average of all the value vectors.
A tiny example. Suppose that, after scaling, “office” scores 2, 1 and 0 against the keys of “London”, “the” and “open”. Softmax gives:
e² = 7.389 e¹ = 2.718 e⁰ = 1 sum = 11.107
weights: 7.389/11.107 = 0.665 2.718/11.107 = 0.245 1/11.107 = 0.090
So the new vector for “office” is 66.5% “London”, 24.5% “the” and 9.0% “open”. In practice each layer runs many of these in parallel (“heads”), each free to learn a different kind of relationship.
The gotchas: every token attends to every other token, so the work grows with the square of the input length; that is the root of context window limits and of the cost of long prompts. And attention weights are not explanations. A high weight shows where information flowed in one head of one layer, not why the model produced an answer.
In DocSlurp
Background concept
Every embedding and language model DocSlurp calls is a transformer built on attention. DocSlurp does not implement attention itself; it calls the models through providers or Transformers.js (core/worker/src/stages/embedding-runtime.ts).