TY - JOUR
T1 - Item Pool Quality Control in Educational Testing
T2 - Change Point Model, Compound Risk, and Sequential Detection
AU - Chen, Yunxiao
AU - Lee, Yi Hsuan
AU - Li, Xiaoou
N1 - Publisher Copyright:
© 2021 The Author(s).
PY - 2022/6
Y1 - 2022/6
N2 - In standardized educational testing, test items are reused in multiple test administrations. To ensure the validity of test scores, the psychometric properties of items should remain unchanged over time. In this article, we consider the sequential monitoring of test items, in particular, the detection of abrupt changes to their psychometric properties, where a change can be caused by, for example, leakage of the item or change of the corresponding curriculum. We propose a statistical framework for the detection of abrupt changes in individual items. This framework consists of (1) a multistream Bayesian change point model describing sequential changes in items, (2) a compound risk function quantifying the risk in sequential decisions, and (3) sequential decision rules that control the compound risk. Throughout the sequential decision process, the proposed decision rule balances the trade-off between two sources of errors, the false detection of prechange items, and the nondetection of postchange items. An item-specific monitoring statistic is proposed based on an item response theory model that eliminates the confounding from the examinee population which changes over time. Sequential decision rules and their theoretical properties are developed under two settings: the oracle setting where the Bayesian change point model is completely known and a more realistic setting where some parameters of the model are unknown. Simulation studies are conducted under settings that mimic real operational tests.
AB - In standardized educational testing, test items are reused in multiple test administrations. To ensure the validity of test scores, the psychometric properties of items should remain unchanged over time. In this article, we consider the sequential monitoring of test items, in particular, the detection of abrupt changes to their psychometric properties, where a change can be caused by, for example, leakage of the item or change of the corresponding curriculum. We propose a statistical framework for the detection of abrupt changes in individual items. This framework consists of (1) a multistream Bayesian change point model describing sequential changes in items, (2) a compound risk function quantifying the risk in sequential decisions, and (3) sequential decision rules that control the compound risk. Throughout the sequential decision process, the proposed decision rule balances the trade-off between two sources of errors, the false detection of prechange items, and the nondetection of postchange items. An item-specific monitoring statistic is proposed based on an item response theory model that eliminates the confounding from the examinee population which changes over time. Sequential decision rules and their theoretical properties are developed under two settings: the oracle setting where the Bayesian change point model is completely known and a more realistic setting where some parameters of the model are unknown. Simulation studies are conducted under settings that mimic real operational tests.
KW - compound decision
KW - item preknowledge
KW - multistream data
KW - sequential change point detection
KW - standardized testing
KW - test security
UR - http://www.scopus.com/inward/record.url?scp=85121336908&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85121336908&partnerID=8YFLogxK
U2 - 10.3102/10769986211059085
DO - 10.3102/10769986211059085
M3 - Article
AN - SCOPUS:85121336908
SN - 1076-9986
VL - 47
SP - 322
EP - 352
JO - Journal of Educational and Behavioral Statistics
JF - Journal of Educational and Behavioral Statistics
IS - 3
ER -