Skip to content

LearnRepresent

Attention

Self-attention · Scaled dot-product attention

The transformer operation that lets each token build its representation as a weighted mix of every other token, with weights computed from dot products.

Background conceptGlossary entry

On this page

Attention is the step inside a transformer where each token updates its vector using information from the other tokens. It is how “office” in “open the London office” learns which office is meant.

Each token’s vector is turned into three new vectors: a query (what this token is looking for), a key (what it offers to others) and a value (what it passes on if chosen). For one token, attention works like this:

  1. Take the dot product of its query with every token’s key. Higher means a better match.
  2. Divide by √d (d is the key length) and apply softmax, which turns the scores into positive weights that sum to 1.
  3. Output the weighted average of all the value vectors.

A tiny example. Suppose that, after scaling, “office” scores 2, 1 and 0 against the keys of “London”, “the” and “open”. Softmax gives:

e² = 7.389   e¹ = 2.718   e⁰ = 1     sum = 11.107
weights: 7.389/11.107 = 0.665   2.718/11.107 = 0.245   1/11.107 = 0.090

So the new vector for “office” is 66.5% “London”, 24.5% “the” and 9.0% “open”. In practice each layer runs many of these in parallel (“heads”), each free to learn a different kind of relationship.

The gotchas: every token attends to every other token, so the work grows with the square of the input length; that is the root of context window limits and of the cost of long prompts. And attention weights are not explanations. A high weight shows where information flowed in one head of one layer, not why the model produced an answer.

In DocSlurp

Background concept

Every embedding and language model DocSlurp calls is a transformer built on attention. DocSlurp does not implement attention itself; it calls the models through providers or Transformers.js (core/worker/src/stages/embedding-runtime.ts).

Sources

Jump to