We found a match
Your institution may have rights to this item. Sign in to continue.
- Title
Commonalities and variations in emotion representation across modalities and brain regions.
- Authors
Kiyokawa, Hiroaki; Hayashi, Ryusuke
- Abstract
Humans express emotions through various modalities such as facial expressions and natural language. However, the relationships between emotions expressed through different modalities and their correlations with neural activities remain uncertain. Here, we aimed to unveil some of these uncertainties by investigating the similarity of emotion representations across modalities and brain regions. First, we represented various emotion categories as multi-dimensional vectors derived from visual (face), linguistic, and visio-linguistic data, and used representational similarity analysis to compare these modalities. Second, we examined the linear transferability of emotion representation from other modalities to the visual modality. Third, we compared the representational structure derived in the first step with those from brain activities across 360 regions. Our findings revealed that emotion representations share commonalities across modalities with modality-type dependent variations, and they can be linearly mapped from other modalities to the visual modality. Additionally, emotion representations in uni-modalities showed relatively higher similarity with specific brain regions, while multi-modal emotion representation was most similar to representations across the entire brain region. These findings suggest that emotional experiences are represented differently across various brain regions with varying degrees of similarity to different modality types, and that they may be multi-modally conveyable in visual and linguistic domains.
- Subjects
FACIAL expression &; emotions (Psychology); FACIAL expression; DEEP learning; NATURAL languages; EMOTIONS
- Publication
Scientific Reports, 2024, Vol 14, Issue 1, p1
- ISSN
2045-2322
- Publication type
Article
- DOI
10.1038/s41598-024-71690-y