We found a match
Your institution may have rights to this item. Sign in to continue.
- Title
Detecting rater bias using a person-fit statistic: a Monte Carlo simulation study.
- Authors
Aubin, André-Sébastien; St-Onge, Christina; Renaud, Jean-Sébastien
- Abstract
Introduction With the Standards voicing concern for the appropriateness of response processes, we need to explore strategies that would allow us to identify inappropriate rater response processes. Although certain statistics can be used to help detect rater bias, their use is complicated by either a lack of data about their actual power to detect rater bias or the difficulty related to their application in the context of health professions education. This exploratory study aimed to establish the worthiness of pursuing the use of lz to detect rater bias. Methods We conducted a Monte Carlo simulation study to investigate the power of a specific detection statistic, that is: the standardized likelihood lz person-fit statistics (PFS). Our primary outcome was the detection rate of biased raters, namely: raters whom we manipulated into being either stringent (giving lower scores) or lenient (giving higher scores), using the lz statistic while controlling for the number of biased raters in a sample (6 levels) and the rate of bias per rater (6 levels). Results Overall, stringent raters (M = 0.84, SD = 0.23) were easier to detect than lenient raters (M = 0.31, SD = 0.28). More biased raters were easier to detect then less biased raters (60% bias: 62, SD = 0.37; 10% bias: 43, SD = 0.36). Discussion The PFS lz seems to offer an interesting potential to identify biased raters. We observed detection rates as high as 90% for stringent raters, for whom we manipulated more than half their checklist. Although we observed very interesting results, we cannot generalize these results to the use of PFS with estimated item/station parameters or real data. Such studies should be conducted to assess the feasibility of using PFS to identify rater bias.
- Subjects
MONTE Carlo method; SIMULATION methods &; models; BIAS (Law); MEDICAL personnel; APPROPRIATENESS (Ethics)
- Publication
Perspectives on Medical Education, 2018, Vol 7, Issue 2, p83
- ISSN
2212-2761
- Publication type
Article
- DOI
10.1007/s40037-017-0391-8