Transformer

The Transformer is a neural architecture that uses self-attention to model relationships among tokens, forming the backbone of most modern LLMs.

It is the model design that lets each word look at other words to decide what matters in context.

Transformers replace recurrence with self-attention and feed-forward blocks, enabling parallel training and long-range dependency modeling. Variants include encoder-only, decoder-only, and encoder–decoder designs.

Understanding Transformers clarifies why modern LLMs scale and how attention relates to context windows and compute cost.

• Language modeling and translation
• Vision Transformers
• Multimodal fusion architectures

• Strong sequence modeling
• Parallelizable training
• Flexible transfer across tasks

• Quadratic attention cost with sequence length (mitigated by variants)
• Large models need significant compute

• Not every neural net is a Transformer
• Attention maps are not complete explanations of model behavior

Are Transformers only for text?

No. Transformers are used in vision, speech, and multimodal models as well.

• Attention Is All You Need (arXiv)

• Vaswani et al., 2017