We found a match
Your institution may have rights to this item. Sign in to continue.
- Title
Pushing the limits of remote RF sensing by reading lips under the face mask.
- Authors
Hameed, Hira; Usman, Muhammad; Tahir, Ahsen; Hussain, Amir; Abbas, Hasan; Cui, Tie Jun; Imran, Muhammad Ali; Abbasi, Qammer H.
- Abstract
The problem of Lip-reading has become an important research challenge in recent years. The goal is to recognise speech from lip movements. Most of the Lip-reading technologies developed so far are camera-based, which require video recording of the target. However, these technologies have well-known limitations of occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based technologies are not useful for multi-modal hearing aids in the coronavirus (COVID-19) environment, where face masks have become a norm. This paper aims to solve the fundamental limitations of camera-based systems by proposing a radio frequency (RF) based Lip-reading framework, having an ability to read lips under face masks. The framework employs Wi-Fi and radar technologies as enablers of RF sensing based Lip-reading. A dataset comprising of vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train machine learning (ML) and deep learning (DL) models. A high classification accuracy of 95% is achieved on the Wi-Fi data utilising neural network (NN) models. Moreover, similar accuracy is achieved by VGG16 deep learning model on the collected radar-based dataset. This study aims to solve the fundamental limitations of camera-based systems by proposing a contactless and privacy-preserving radio frequency-based lip-reading framework, which can recognise lip movements under face masks.
- Subjects
LIPREADING; MEDICAL masks; REMOTE sensing; WIRELESS Internet; DEEP learning; RADIO frequency; BISTATIC radar; DIGITAL cameras
- Publication
Nature Communications, 2022, Vol 12, Issue 1, p1
- ISSN
2041-1723
- Publication type
Article
- DOI
10.1038/s41467-022-32231-1