We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Enhanced gradient learning for deep neural networks.
- Authors
Yan, Ming; Yang, Jianxi; Chen, Cen; Zhou, Joey Tianyi; Pan, Yi; Zeng, Zeng
- Abstract
Deep neural networks have achieved great success in both computer vision and natural language processing tasks. How to improve the gradient flows is crucial in training very deep neural networks. To address this challenge, a gradient enhancement approach is proposed through constructing the short circuit neural connections. The proposed short circuit is a unidirectional neural connection that back propagates the sensitivities rather than gradients in neural networks from the deep layers to the shallow layers. Moreover, the short circuit is further formulated as a gradient truncation operation in its connecting layers, which can be plugged into the backbone models without introducing extra training parameters. Extensive experiments demonstrate that the deep neural networks, with the help of short circuit connection, gain a large margin of improvement over the baselines on both computer vision and natural language processing tasks. The work provides the promising solution to the low‐resource scenarios, such as, intelligence transport systems of computer vision, question answering of natural language processing.
- Subjects
NEURAL circuitry; ARTIFICIAL neural networks; COMPUTER vision; NATURAL language processing; SHORT circuits
- Publication
IET Image Processing (Wiley-Blackwell), 2022, Vol 16, Issue 2, p365
- ISSN
1751-9659
- Publication type
Article
- DOI
10.1049/ipr2.12353