Transformer
The neural network architecture behind modern language and embedding models: stacked layers of attention and small feed-forward networks over a sequence of tokens.
Background conceptGlossary entry
On this page
A transformer is the neural network architecture behind almost every modern language model and embedding model. Introduced in 2017, it processes a sequence of tokens in layers. Each layer does two things: attention, where every token gathers information from the other tokens, then a small feed-forward network applied to each token separately. A model stacks anywhere from 6 of these layers (all-MiniLM-L6-v2) to more than 100 (the largest language models).
Each token starts as a vector looked up from the vocabulary and leaves the stack as a vector that reflects its context. The “margin” in “operating margin” and the “margin” in “the page margin” start identical and end up different.
What happens to those output vectors depends on the model’s job:
- Embedding models (encoders such as BERT, BGE, E5) pool the token vectors into one embedding, for example by averaging them.
- Rerankers (cross-encoders) read a query and passage together and turn the output into one relevance score.
- Generative models (decoders, the large language models behind chat) use the last token’s vector to predict the next token, append it and repeat.
Attention compares every token with every other, so cost grows with the square of the sequence length: twice the tokens is roughly four times the attention work. That is one reason models have a fixed input limit and long inputs are slow and expensive.
The gotcha: “transformer” names an architecture, not a capability. Two transformer models of similar size can differ completely in what they’re good at, depending on training data and objective. An embedding model can’t answer questions, and a chat model’s internal vectors are not good search embeddings without further training.
In DocSlurp
Background concept
DocSlurp calls transformer models rather than training them: hosted embedding and chat models, and local Transformers.js embedding models whose token vectors are mean- or CLS-pooled into one vector (core/worker/src/stages/embedding-runtime.ts).