Definition
A neural network architecture that organizes sequence information processing through attention mechanisms and feed-forward transformations.
The original Transformer of 2017 centered on attention, avoiding recurrence or convolution as the main structure for sequence modeling, and was evaluated on machine translation. Earlier work had already introduced attention mechanisms; their origin should not be attributed to the same paper.
Later models use different encoder, decoder, or combined forms. The Transformer architecture is a distinct concept from the training scale and tasks of large language models.
Principles and impact
Problems solved
- Improves information exchange in sequence models and the parallelism of training.
Principles
- Uses attention to aggregate information across input positions, together with positional representations and feed-forward transformations.
Improvements
Not recorded
Significance
- Became an important architecture for many modern language models and models of other modalities.
History and applications
People
Ashish Vaswani、Noam Shazeer
Organizations
Not recorded
Events
Not recorded
Applications
Machine translation、Language modeling
Limitations
- The computation and memory required by standard full attention grow with sequence length; the architecture itself does not guarantee factual accuracy.
Upstream and downstream relations
Upstream
Sources: Attention Is All You Need
Downstream
Sources: Language Models are Few-Shot LearnersAttention Is All You Need