We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Correlations of Cross-Entropy Loss in Machine Learning.
- Authors
Connor, Richard; Dearle, Alan; Claydon, Ben; Vadicamo, Lucia
- Abstract
Cross-entropy loss is crucial in training many deep neural networks. In this context, we show a number of novel and strong correlations among various related divergence functions. In particular, we demonstrate that, in some circumstances, (a) cross-entropy is almost perfectly correlated with the little-known triangular divergence, and (b) cross-entropy is strongly correlated with the Euclidean distance over the logits from which the softmax is derived. The consequences of these observations are as follows. First, triangular divergence may be used as a cheaper alternative to cross-entropy. Second, logits can be used as features in a Euclidean space which is strongly synergistic with the classification process. This justifies the use of Euclidean distance over logits as a measure of similarity, in cases where the network is trained using softmax and cross-entropy. We establish these correlations via empirical observation, supported by a mathematical explanation encompassing a number of strongly related divergence functions.
- Subjects
ARTIFICIAL neural networks; EUCLIDEAN distance; LOGITS; DEEP learning
- Publication
Entropy, 2024, Vol 26, Issue 6, p491
- ISSN
1099-4300
- Publication type
Article
- DOI
10.3390/e26060491