What Is a Transformer? The Definitive Guide to the Architecture Behind Modern AI

What is a transformer in AI?

A transformer is a neural network architecture that uses attention to model the relationships between the elements of a sequence, processing many of them in parallel rather than strictly one after another. Where earlier models read text one word at a time, a transformer can look across the whole input and learn, for each token, which other tokens are most relevant to it. That design choice is what makes the architecture both powerful and fast to train.

The name refers to the architecture, not to any one model. GPT, Claude, Gemini, Llama, and BERT are all specific models built on the transformer architecture, in the same way that many different cars are built on the same chassis. When people say a model is a "transformer-based large language model," they mean it uses this architecture as its underlying design. The "T" in GPT stands for transformer.

Transformers were originally built for machine translation, converting a sentence in one language into another. The researchers quickly found the architecture generalized far beyond translation. Today transformers are the dominant design for foundation models across text generation, image recognition, speech processing, and multimodal systems, and attention-based components now appear in scientific machine learning as well. The architecture turned out to be a general-purpose engine for finding structure in sequences of almost any kind.

In short: a transformer is the neural network design that uses attention to figure out what depends on what in a sequence, and it is the engine underneath the current era of AI.

Why were transformers invented?

Before transformers, the leading architectures for language were recurrent neural networks (RNNs) and their more capable variant, long short-term memory networks (LSTMs). These models read a sentence the way a person reads aloud, one word after another, carrying a running summary of everything seen so far. This sequential design had two crippling weaknesses.

The first was parallelism. Because each recurrent step depended on the previous hidden state, the sequence could not be fully parallelized across positions during training. More hardware could accelerate the individual operations, but it could not remove that sequential dependency, and as datasets grew to billions of words the bottleneck became unaffordable.

The second was memory. In a long sentence, the influence of an early word on a much later word had to survive being passed through every intermediate step, and in practice it faded. RNNs struggled to connect "The cat that the dog chased across the yard was black" because the subject and its description were too many steps apart. This is the long-range dependency problem.

The transformer addressed both. By replacing recurrence with attention, it removed the strict sequential chain, so training parallelizes across many positions at once. And because attention can connect a token directly to distant tokens, the computational path between two related words is short regardless of how many words sit between them. The word "black" can relate to "cat" without passing through every word in between. Removing recurrence is the pivotal move the paper's title captures: attention is all you need.

How does self-attention work?

Self-attention is the mechanism that lets each token in a sequence gather information from the other tokens that are most relevant to it, producing a new, context-aware representation of that token. It is the defining mechanism of the transformer, working alongside feed-forward layers, residual connections, normalization, embeddings, and positional information to build the model's understanding.

The word "self" means the mechanism operates within a single sequence: every token attends to the other tokens in the same input. This is how a model turns a flat list of word vectors into representations that reflect context. The same word gets a different representation depending on the sentence around it, which is why a transformer can tell the two meanings of "bank" apart.

Consider the sentence "the trophy did not fit in the suitcase because it was too big." A human knows "it" refers to the trophy. Self-attention gives the model a mechanism for representing relationships like this, allowing information from the relevant parts of the sequence to influence the representation of each token. The mechanism does not hand us a tidy explanation of the model's reasoning, and attention weights should not be read as a literal account of what the model is thinking, but the effect is that the representation of "it" is shaped by the tokens the model has learned are relevant to it.

In short: self-attention rebuilds each token's meaning as a blend of the other tokens in the sequence, which is how a transformer becomes sensitive to context.

What are query, key, and value?

Query, key, and value are the three roles every token plays inside attention, and they are how the mechanism actually computes relevance. For each token, the model produces these three vectors by multiplying the token's embedding by three separate sets of learned weights. The query represents what this token is looking for, the key represents what each token offers, and the value represents the information a token will contribute if it is attended to. The common analogy is a library search: the query is your request, the keys are the labels on the books, and the values are the contents of the books that match.

The mechanism combines them in four steps. First, the model compares one token's query against every token's key by taking their dot product, producing a raw relevance score for each pair. Second, it divides those scores by the square root of the key dimension, a scaling step that keeps the numbers in a stable range so training does not break. Third, it passes the scaled scores through a softmax function, turning them into weights that add up to one, a clean distribution of attention across the tokens. Fourth, it uses those weights to take a weighted sum of the value vectors, producing the token's new representation.

Written compactly, the whole operation is the formula the 2017 paper made famous: Attention(Q, K, V) equals softmax of (Q times K transpose, divided by the square root of d_k), times V. The division by the square root of the key dimension matters more than it looks. Without it, large dot products would push the softmax into a region where its gradients are almost zero and learning would stall.

In short: the query, key, and value let attention score how relevant every token is to every other token, and the scaled dot-product formula turns those scores into each token's new, context-aware vector.

What is multi-head attention?

Multi-head attention runs several attention operations in parallel, each with its own learned query, key, and value weights, so the model can capture several kinds of relationship at once. A single attention calculation can only emphasize one pattern at a time, but language is richer than that. In one sentence, a word relates to others grammatically, semantically, and by topic simultaneously.

Each parallel attention operation is called a head, and different heads can specialize. One head may come to track subject and verb agreement, another may link pronouns to the nouns they refer to, and another may follow the overall topic. The original transformer used eight heads. Their outputs are combined and projected back together, giving the model a multi-dimensional view of context that no single head could produce.

In short: multi-head attention lets a transformer attend to several types of relationship in parallel, which is what gives it a rich, layered sense of context.

What is positional encoding?

Positional encoding is the signal a transformer adds to each token so the model knows the order of the sequence. Because a transformer processes tokens in parallel rather than in sequence, attention on its own is blind to order. Without help, "the dog bit the man" and "the man bit the dog" would look identical to it.

Positional encoding fixes this by adding a distinct positional signal to each token's embedding before attention runs. The original design used sine and cosine waves of different frequencies, a pattern that gives each position a unique fingerprint and lets the model reason about relative distances. Modern transformers use several approaches to represent position, including learned positional embeddings, relative-position methods, and rotary positional embeddings (RoPE), the last of which is now common in large language models because it extends gracefully to long contexts. The scheme has evolved, but the purpose is unchanged: give the model a sense of order that attention alone cannot provide.

In short: positional encoding tells an order-blind attention mechanism which token came where, and modern models use several schemes to do it.

What is a transformer block?

A transformer block is the repeating unit that a transformer stacks to build its full depth, and it is the most useful way to picture the architecture. Each block combines the components already described into a fixed sequence of operations, and a complete model is simply many of these blocks stacked one on top of another.

Inside a block, the data flows through a consistent path: the input passes through a normalization step, then multi-head self-attention, then a residual connection that adds the block's input back to its output, then another normalization step, then a feed-forward neural network that transforms each token's representation further, and finally another residual connection. The two supporting techniques here, residual connections and layer normalization, are what keep the signal stable as it flows through dozens of blocks, and without them deep transformers would be very hard to train.

The original 2017 transformer stacked six of these blocks in its encoder and six in its decoder. Modern large models stack many more, often dozens or over a hundred. Each block builds a richer, more abstract representation on top of the one below it, which is how depth turns simple token vectors into a deep understanding of language.

How does a transformer process text?

A transformer converts input text into an output prediction through a consistent pipeline that ties every component together. Following the full path from raw text to prediction makes the architecture concrete.

1. Tokenization: The input text is split into tokens, which are words or fragments of words, giving the model a sequence of discrete units to process.
2. Embedding: Each token is converted into a vector, a list of numbers that represents its meaning in mathematical form. This is the same idea covered in our guide to embeddings.
3. Positional encoding: A positional signal is added to each embedding so the model knows the order of the tokens.
4. Transformer blocks: The sequence passes through a stack of transformer blocks. In each block, multi-head self-attention blends in context from across the sequence, and a feed-forward network transforms each token further, with normalization and residual connections keeping the signal stable.
5. Output: For a text model, the final layer produces a set of scores over the entire vocabulary, one per possible next token, which are turned into probabilities.

The key thing to notice is that meaning is built up gradually. Early blocks capture simple, local relationships, and later blocks combine those into abstract, sentence-level and document-level understanding. By the time the sequence reaches the final layer, each token's representation reflects the entire context around it.

In short: a transformer turns text into tokens, tokens into vectors, then refines those vectors block by block using attention until the final layer can score what comes next.

How does a transformer generate text?

A transformer generates text one token at a time in a loop, a process called autoregressive generation. It does not write a sentence in one shot. It predicts the single most suitable next token, adds it to the input, and runs the entire pipeline again to predict the following one.

The loop runs like this: the prompt is turned into tokens and embeddings, passed through the transformer blocks, and the final layer produces logits, which are raw scores for every token in the vocabulary. A softmax function converts those logits into a probability distribution. A decoding step then selects the next token from that distribution, and a setting called temperature controls how much randomness is allowed, with low temperature making the model pick high-probability tokens for predictable output and higher temperature allowing more varied, creative choices. The chosen token is appended to the sequence, and the whole process repeats to produce the next token, continuing until the model emits a stop signal or reaches a length limit.

This autoregressive loop is why generative models such as GPT and Claude are called decoder-only transformers, and it is also why they generate text at a steady, token-by-token pace rather than instantly.

In short: a transformer generates text by repeatedly predicting the next token from a probability distribution, appending it, and running the pipeline again, one token at a time.

What are the three types of transformer architectures?

The original transformer had two halves, an encoder that reads and understands input and a decoder that generates output. Later research found that each half is useful on its own, which produced three architecture families. Knowing which family a model belongs to explains what it is good at.

The most important trend is that decoder-only models have come to dominate generation. Many of the most widely used generative language models, including OpenAI's GPT series and Meta's Llama, use decoder-style transformer architectures, while encoder-only models such as BERT remain important workhorses for search, classification, and representation. Encoder-decoder models still excel where one text is converted into another, such as translation.

In short: encoder-only models understand, decoder-only models generate, and encoder-decoder models convert one text into another, with decoder-only now the leading design for generative language models.

Transformer vs LLM: what is the difference?

A transformer is an architecture, and a large language model (LLM) is a specific kind of model built using that architecture. The two words are often used loosely as if they mean the same thing, but keeping them distinct clears up a lot of confusion.

The relationship runs one way more than the other. Almost every large language model in common use is transformer-based, but the transformer is also used well beyond language, for images, audio, and multimodal systems. So the transformer is the general design, and the LLM is one prominent product of it. A model can be a transformer without being a language model, as a Vision Transformer is.

In short: the transformer is the architecture and the large language model is a model built on it, so all mainstream LLMs are transformers but not all transformers are LLMs.

How is a transformer different from RNNs and LSTMs?

Transformers replaced a lineage of sequence models, and the contrast explains why the switch was so decisive.

The transformer's key advantage was combining highly parallelizable training with short computational paths between distant positions. RNNs and LSTMs had a form of memory but could not parallelize across positions. CNNs could parallelize but connected distant positions only indirectly, through many stacked layers. The transformer offered both parallel training and direct, short paths between any two tokens, which is the property the original paper emphasized when it set out to remove recurrence.

In short: the transformer won because it trained in parallel and kept the path between distant tokens short, which earlier sequence models could not do together.

How have transformers evolved since 2017?

The transformer's most consequential property is that it improved as it was made larger and trained on more data, but the story since 2017 is not only about size, and by 2026 it is much less about size than it once was.

The early years were a scaling story. BERT arrived in 2018 with 340 million parameters and showed the encoder half was powerful on its own. GPT followed with the decoder half. GPT-2 in 2019 and GPT-3 in 2020, with 175 billion parameters, then demonstrated that scaling transformers produced large jumps in capability. Google's PaLM reached 540 billion parameters in 2022. GPT-4's parameter count has been estimated at roughly 1.8 trillion by external researchers such as Epoch AI, though OpenAI never officially disclosed it.

That transparency has since faded. The 2026 Stanford HAI AI Index notes that reported frontier-model parameter counts have remained around the trillion scale while disclosure of model specifications has declined, so precise sizes for the newest systems are increasingly difficult to verify. Just as importantly, more parameters stopped being the main lever of progress. Frontier capability now comes increasingly from post-training methods, higher-quality and curated data, inference-time computation where a model spends more compute reasoning through a problem, and architectural efficiency, rather than from raw parameter growth alone.

The costs of the scaling era were real. The 2026 Stanford AI Index estimates the training emissions of xAI's Grok 4 at 72,816 tons of carbon-equivalent, compared with roughly 5,000 tons estimated for GPT-4 two years earlier, though these figures apply to specific frontier training runs and are not representative of transformers in general. Several forces now push the field beyond simply adding parameters: the finite supply of high-quality human text, compute and energy costs, memory bandwidth, inference economics, and diminishing returns. The Stanford index also notes that synthetic data has not replaced real data in pretraining, which is why data quality and post-training have become such important drivers of capability.

What are the limitations of transformers?

• Quadratic attention cost. Standard self-attention compares every token with every other token, so its cost grows with the square of the sequence length. Doubling the context does not double the work, it quadruples it. This is the single biggest obstacle to processing very long inputs efficiently, and the target of most efficiency research.
• Enormous compute and energy demand. Training frontier transformers requires vast clusters of specialized chips and consumes energy at industrial scale, which concentrates the ability to build them in a handful of well-funded organizations and carries a significant carbon cost.
• Data hunger. Transformers need very large training datasets to reach their potential, and the supply of high-quality human text is finite and being consumed quickly.
• Limited interpretability. The learned weights and attention patterns are not directly human-readable, which makes it hard to explain exactly why a model produced a given output, a serious concern in regulated settings.
• No built-in grounding. A transformer predicts likely text from patterns, not verified truth, which is why it can produce fluent but false output. This is the root of the hallucination problem and the reason techniques like retrieval-augmented generation exist.

What is challenging the transformer architecture?

The quadratic cost of attention has motivated a wave of research, some that makes transformers more efficient and some that proposes to change how sequence modeling works.

Within the transformer family, several techniques extend its reach. FlashAttention is an input-output-aware algorithm that reorganizes the attention computation to use fast on-chip memory, cutting memory use sharply and speeding up training without changing the result. Mixture of Experts (MoE) takes a different angle: instead of activating the entire network for every token, it routes each token to a small subset of specialized "expert" sub-networks, so total parameter counts can grow without a proportional rise in per-token compute. This approach underpins several long-context systems. Sparse and sub-quadratic attention methods approximate full attention by having each token attend to only the most relevant others, with recent techniques reporting large speedups at long context lengths.

Beyond the transformer, a genuine architectural challenger has emerged. Mamba and related state space models replace attention with a mechanism whose cost grows linearly rather than quadratically with sequence length. In the original Mamba experiments, the authors reported roughly five times higher inference throughput than comparable transformers and demonstrated scaling to sequences of around one million elements. Those results established state space models as an important alternative for long-sequence workloads, though they do not imply that Mamba universally outperforms transformers. The emerging pattern is hybrid rather than winner-take-all: designs such as Jamba interleave transformer and state-space layers to capture the strengths of both. Attention may not be all you need forever, but for now it remains the core of the field.

Where are transformers used beyond text?

The transformer began in language, but its reach now extends across many domains of AI, which is the clearest evidence of how general the architecture is.

In computer vision, the Vision Transformer (ViT) splits an image into patches and treats each patch like a token, applying the same self-attention mechanism to recognize objects and scenes, and it now rivals or beats the convolutional networks that dominated vision for a decade. In audio, transformers drive modern speech recognition and generation. In multimodal AI, a single transformer-based model can process text, images, and audio together, which is how a system can look at a photograph and describe it or read a chart and answer questions about it. Attention-based components have also become important in scientific machine learning, including systems for protein structure prediction, though those are specialized architectures that combine attention with other components rather than plain transformers.

This cross-domain success is why the transformer is considered one of the most important ideas in the history of machine learning. An architecture built to translate French into English turned out to be a general engine for modeling structure in many kinds of data.

In short: transformers now underpin vision, speech, and multimodal AI and appear in scientific models, not just language, which is why they are treated as general-purpose infrastructure.

Why do transformers matter?

The transformer is the architectural foundation beneath most of modern AI, and nearly every concept in this Learn series traces back to it. A large language model is a large model, usually a transformer, trained to generate and understand language. Embeddings are produced by transformer layers converting tokens into vectors. Retrieval-augmented generation, tool use, and AI agents are all patterns built on top of transformer-based models. For today's dominant foundation-model stack, the transformer is the layer underneath much of the technology built above it.

Its importance is also a matter of record. The 2026 Stanford HAI AI Index reports that industry produced more than 90 percent of notable frontier models in 2025, and that performance on demanding benchmarks has climbed steeply, with one leading coding benchmark rising from around 60 percent to near 100 percent in a single year. The frontier models behind those gains are transformers or close variants. The architecture described in an eight-page paper in 2017 became, within a decade, the common design language of the field.

A brief timeline of the transformer

Frequently asked questions

Sources and further reading

1. Vaswani, A. et al. Attention Is All You Need. NeurIPS, 2017. arxiv.org/abs/1706.03762
2. Stanford Institute for Human-Centered AI. 2026 AI Index Report. Stanford HAI, 2026. hai.stanford.edu/ai-index
3. Epoch AI. Machine Learning Trends: Model Size, Compute, and Data. 2026. epoch.ai
4. Devlin, J. et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018. arxiv.org/abs/1810.04805
5. Dao, T. et al. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. 2022. arxiv.org/abs/2205.14135
6. Gu, A. and Dao, T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. 2023. arxiv.org/abs/2312.00752
7. Dosovitskiy, A. et al. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. 2020. arxiv.org/abs/2010.11929
8. IBM. What is a Transformer Model? IBM Think, 2026. ibm.com
9. NVIDIA. What Is a Transformer Model? NVIDIA Blog. blogs.nvidia.com
10. Amatriain, X. et al. Transformer Models: An Introduction and Catalog. 2023. arxiv.org/abs/2302.07730