Definition
A method for computing parameter gradients in differentiable networks by applying the chain rule to propagate derivatives from the output toward earlier layers.
The 1986 paper by Rumelhart, Hinton and Williams demonstrated error backpropagation and the learning of hidden representations. The year marks a representative paper; it does not attribute the origins of reverse-mode differentiation or the chain rule to that paper.
Backpropagation computes gradients, while the way parameters are updated depends on the optimization method. It is not itself a network architecture, and it does not guarantee that training reaches a global optimum.
Principles and impact
Problems solved
- Efficiently computes how large numbers of parameters in a multilayer differentiable model affect the loss.
Principles
- Applies the chain rule backward along computational dependencies, reusing intermediate derivatives.
Improvements
Not recorded
Significance
- Made end-to-end gradient-based training of complex networks more practical.
History and applications
People
David E. Rumelhart、Geoffrey E. Hinton、Ronald J. Williams
Organizations
Not recorded
Events
Not recorded
Applications
Training multilayer networks
Limitations
- Requires differentiable computational paths and is affected by gradient issues, numerical stability and optimization difficulties.
Upstream and downstream relations
Upstream
No direct upstream relations recorded
Downstream
Sources: Learning Representations by Back-propagating Errors
Sources: Learning Representations by Back-propagating ErrorsImageNet Classification with Deep Convolutional Neural Networks