We found a match
Your institution may have access to this item. Find your institution then sign in to continue.
- Title
Peer Review in Agricultural Education: Interrater Reliability of Manuscript Reviews for the 2014 National Agricultural Education Research Conference.
- Authors
Shoulders, Catherine W.; Johnson, Donald M.; Flowers, Jim
- Abstract
This study analyzed 336 peer reviews of 112 manuscripts submitted for possible presentation at the 2014 National Agricultural Education Research Conference (NAERC). There were scoring errors on 6.8% of the reviews; the most frequent errors were failure to record a score or assigning a score above the range of points possible for one or more of the review criteria. The coefficient of variation (CV) for same-paper ratings of manuscript quality ranged from 3.94% to 96.43% with a mean of 19.65% (SD = 15.22%). The interrater reliability for same-paper evaluations of manuscript quality was .15. The CVs for reviewer same-paper reject-accept recommendations ranged from 0.00% to 98.97% with a mean of 38.80% (SD = 22.72%). The interrater reliability for same-paper reject-accept recommendations was .09. On 82 (73.21%) manuscripts all three reviewers agreed on the manuscript's relevance to agricultural education. Mean manuscript quality scores explained 68.3% of the variance in mean reject-accept recommendations leaving 31.7% of the variance unexplained. There were no significant (p > .05) differences by academic rank in ratings of manuscript quality, reject-accept recommendations, or assessments of relevance to agricultural education. The authors offered recommendations for improvement of the NAERC manuscript review process.
- Subjects
AGRICULTURAL education; PEER review of teachers; INTER-observer reliability; COEFFICIENTS (Statistics); ACADEMIC titles (Higher education)
- Publication
Journal of Agricultural Education, 2015, Vol 56, Issue 3, p121
- ISSN
1042-0541
- Publication type
Article
- DOI
10.5032/jae.2015.03121