TY - JOUR
T1 - Evaluating the Impact of Careless Responding on Aggregated-Scores
T2 - To Filter Unmotivated Examinees or Not?
AU - Rios, Joseph A.
AU - Guo, Hongwen
AU - Mao, Liyang
AU - Liu, Ou Lydia
N1 - Publisher Copyright:
© 2017, Copyright © Eductional Testing Service.
PY - 2017/1/2
Y1 - 2017/1/2
N2 - When examinees' test-taking motivation is questionable, practitioners must determine whether careless responding is of practical concern and if so, decide on the best approach to filter such responses. As there has been insufficient research on these topics, the objectives of this study were to: a) evaluate the degree of underestimation in the true mean when careless responses are present, and b) compare the effectiveness of two filtering procedures in purifying biased aggregated-scores. Results demonstrated that: a) the true mean was underestimated by around 0.20 SDs if the total amount of careless responses exceeded 6.25%, 12.5%, and 12.5% for easy, moderately difficult, and difficult tests, respectively, and b) listwise deleting data from unmotivated examinees artificially inflated the true mean by as much as.42 SDs when ability was related to careless responding. Findings from this study have implications for when and how practitioners should handle careless responses for group-based low-stakes assessments.
AB - When examinees' test-taking motivation is questionable, practitioners must determine whether careless responding is of practical concern and if so, decide on the best approach to filter such responses. As there has been insufficient research on these topics, the objectives of this study were to: a) evaluate the degree of underestimation in the true mean when careless responses are present, and b) compare the effectiveness of two filtering procedures in purifying biased aggregated-scores. Results demonstrated that: a) the true mean was underestimated by around 0.20 SDs if the total amount of careless responses exceeded 6.25%, 12.5%, and 12.5% for easy, moderately difficult, and difficult tests, respectively, and b) listwise deleting data from unmotivated examinees artificially inflated the true mean by as much as.42 SDs when ability was related to careless responding. Findings from this study have implications for when and how practitioners should handle careless responses for group-based low-stakes assessments.
KW - careless responding
KW - low-stakes testing
KW - motivation
KW - rapid guessing
KW - test taking
KW - validity
UR - http://www.scopus.com/inward/record.url?scp=84990213011&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84990213011&partnerID=8YFLogxK
U2 - 10.1080/15305058.2016.1231193
DO - 10.1080/15305058.2016.1231193
M3 - Article
AN - SCOPUS:84990213011
SN - 1530-5058
VL - 17
SP - 74
EP - 104
JO - International Journal of Testing
JF - International Journal of Testing
IS - 1
ER -