We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Fast deep learning with tight frame wavelets.
- Authors
Cao, Haitao
- Abstract
The cost function gradient vanishing or exploding problem and slow convergence speed are key issues when training deep neural networks (DNNs). In this paper, we investigate the forward and backward propagation processes of DNN training and explore the properties of the activation function and derivative function (ADF) employed. The outputs' distribution of ADF with near-zero mean is proposed to reduce gradient problems. Additionally, the constant energy transfer of propagating data in the training process is also proposed to speed up convergence further. Based on wavelet frame theory, we derive a novel ADF, i.e., tight frame wavelet activation function (TFWAF) and tight frame wavelet derivative function (TFWDF) of the Mexican hat wavelet, to stabilize and accelerate DNN training. The nonlinearity of wavelet functions can strengthen the learning capacity of DNN models, while the sparse property of wavelets derived can reduce the overfitting problem and enhance the robustness of models. Experiments demonstrate that the proposed method stabilizes the DNN training process and accelerates convergence.
- Subjects
ARTIFICIAL neural networks; DEEP learning; DERIVATIVES (Mathematics); COST functions; WAVELETS (Mathematics); ENERGY transfer
- Publication
Neural Computing & Applications, 2024, Vol 36, Issue 9, p4885
- ISSN
0941-0643
- Publication type
Article
- DOI
10.1007/s00521-023-09260-y