We found a match
Your institution may have rights to this item. Sign in to continue.
- Title
Lessons Learned from the Usability Evaluation of a Simulated Patient Dialogue System.
- Authors
Campillos-Llanos, Leonardo; Thomas, Catherine; Bilinski, Éric; Neuraz, Antoine; Rosset, Sophie; Zweigenbaum, Pierre
- Abstract
Simulated consultations through virtual patients allow medical students to practice history-taking skills. Ideally, applications should provide interactions in natural language and be multi-case, multi-specialty. Nevertheless, few systems handle or are tested on a large variety of cases. We present a virtual patient dialogue system in which a medical trainer types new cases and these are processed without human intervention. To develop it, we designed a patient record model, a knowledge model for the history-taking task, and a termino-ontological model for term variation and out-of-vocabulary words. We evaluated whether this system provided quality dialogue across medical specialities (n = 18), and with unseen cases (n = 29) compared to the cases used for development (n = 6). Medical evaluators (students, residents, practitioners, and researchers) conducted simulated history-taking with the system and assessed its performance through Likert-scale questionnaires. We analysed interaction logs and evaluated system correctness. The mean user evaluation score for the 29 unseen cases was 4.06 out of 5 (very good). The evaluation of correctness determined that, on average, 74.3% (sd = 9.5) of replies were correct, 14.9% (sd = 6.3) incorrect, and in 10.7% the system behaved cautiously by deferring a reply. In the user evaluation, all aspects scored higher in the 29 unseen cases than in the 6 seen cases. Although such a multi-case system has its limits, the evaluation showed that creating it is feasible; that it performs adequately; and that it is judged usable. We discuss some lessons learned and pivotal design choices affecting its performance and the end-users, who are primarily medical students.
- Subjects
COMPUTER simulation; MEDICAL history taking; MEDICAL education; MEDICAL technology; MEDICAL quality control; RESEARCH funding; ARTIFICIAL intelligence; NATURAL language processing; DESCRIPTIVE statistics; SURVEYS; USER-centered system design
- Publication
Journal of Medical Systems, 2021, Vol 45, Issue 7, p1
- ISSN
0148-5598
- Publication type
Article
- DOI
10.1007/s10916-021-01737-4