We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
A Multimodal Late Fusion Framework for Physiological Sensor and Audio-Signal-Based Stress Detection: An Experimental Study and Public Dataset.
- Authors
Xefteris, Vasileios-Rafail; Dominguez, Monica; Grivolla, Jens; Tsanousa, Athina; Zaffanela, Francesco; Monego, Martina; Symeonidis, Spyridon; Diplaris, Sotiris; Wanner, Leo; Vrochidis, Stefanos; Kompatsiaris, Ioannis
- Abstract
Stress can be considered a mental/physiological reaction in conditions of high discomfort and challenging situations. The levels of stress can be reflected in both the physiological responses and speech signals of a person. Therefore the study of the fusion of the two modalities is of great interest. For this cause, public datasets are necessary so that the different proposed solutions can be comparable. In this work, a publicly available multimodal dataset for stress detection is introduced, including physiological signals and speech cues data. The physiological signals include electrocardiograph (ECG), respiration (RSP), and inertial measurement unit (IMU) sensors equipped in a smart vest. A data collection protocol was introduced to receive physiological and audio data based on alterations between well-known stressors and relaxation moments. Five subjects participated in the data collection, where both their physiological and audio signals were recorded by utilizing the developed smart vest and audio recording application. In addition, an analysis of the data and a decision-level fusion scheme is proposed. The analysis of physiological signals includes a massive feature extraction along with various fusion and feature selection methods. The audio analysis comprises a state-of-the-art feature extraction fed to a classifier to predict stress levels. Results from the analysis of audio and physiological signals are fused at a decision level for the final stress level detection, utilizing a machine learning algorithm. The whole framework was also tested in a real-life pilot scenario of disaster management, where users were acting as first responders while their stress was monitored in real time.
- Subjects
MACHINE learning; FEATURE selection; FEATURE extraction; MULTIMODAL user interfaces; INTRUSION detection systems (Computer security); INTELLIGENT sensors; EMERGENCY management
- Publication
Electronics (2079-9292), 2023, Vol 12, Issue 23, p4871
- ISSN
2079-9292
- Publication type
Article
- DOI
10.3390/electronics12234871