transformer

From RNNs to attention

Before 2017, sequence models like RNNs and LSTMs read text one token at a time, left to right. That made long-range connections slow to learn and hard to parallelize. The transformer, introduced in the paper “Attention Is All You Need” by researchers at Google, dropped recurrence and convolution entirely and used attention instead.

Self-attention lets the model compare every token with every other token directly and assign each pair a relevance weight. In the phrase “the animal did not cross the street because it was too tired,” attention is what links it back to animal rather than street. Multi-head attention runs several of these comparisons at the same time, each learning a different kind of relationship (syntax, meaning, position).

Because all positions are handled together instead of in sequence, transformers map cleanly onto GPUs and TPUs. Google reported the design sped up training by up to an order of magnitude versus recurrent models, which is what made it practical to scale to billions of parameters.

The original design pairs an encoder with a decoder. Most of today’s large language models (GPT, Claude, Gemini) are decoder-only transformers that predict the next token one step at a time. The same architecture powers the AI systems that now answer search queries directly, which is why the concept matters well beyond machine learning teams.