We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Utilizing gestures to improve sentence boundary detection.
- Authors
Lei Chen; Harper, Mary
- Abstract
n accurate estimation of sentence units (SUs) in spontaneous speech is important for (1) helping listeners to better understand speech content and (2) supporting other natural language processing tasks that require sentence information. There has been much research on automatic SU detection; however, most previous studies have only used lexical and prosodic cues, but have not used nonverbal cues, e.g., gesture. Gestures play an important role in human conversations, including providing semantic content, expressing emotional status, and regulating conversational structure. Given the close relationship between gestures and speech, gestures may provide additional contributions to automatic SU detection. In this paper, we have investigated the use of gesture cues for enhancing the SU detection. Particularly, we have focused on: (1) collecting multimodal data resources involving gestures and SU events in human conversations, (2) analyzing the collected data sets to enrich our knowledge about co-occurrence of gestures and SUs, and (3) building statistical models for detecting SUs using speech and gestural cues. Our data analyses suggest that some gesture patterns influence a word boundary's probability of being an SU. On the basis of the data analyses, a set of novel gestural features were proposed for SU detection. A combination of speech and gestural features was found to provide more accurate SU predictions than using only speech features in discriminative models. Findings in this paper support the view that human conversations are processes involving multimodal cues, and so they are more effectively modeled using information from both verbal and nonverbal channels.
- Subjects
NATURAL language processing; AUTOMATIC speech recognition; NONVERBAL communication; SIGN language; NONVERBAL ability; NONVERBAL learning; ORAL communication; VISUAL literacy; PROSODIC analysis (Linguistics)
- Publication
Multimedia Tools & Applications, 2011, Vol 51, Issue 3, p1035
- ISSN
1380-7501
- Publication type
Article
- DOI
10.1007/s11042-009-0436-z