Statistical and practical significance of item misfit in educational testing

检验模型匹配被认为是项目反应理论(IRT)建模的重要步骤，因为模型匹配是从估计参数中提取有效推理的必要前提(Wainer&Thissen，1987)。Hambleton和Hahn(2005)建议了几个步骤来评估模型匹配，包括(A)项目匹配统计的计算和(B)关于测试结果的不匹配的后果的调查。在教育大规模评估中，采用了不同的项目匹配指数。其中许多统计数据显示出严重的局限性，例如缺乏关于测试统计数据分布的理论证据(梁，威尔斯和汉布尔顿，2014)。由于缺乏确定准确分界值的方法论基础，使得关于物品适合性的决定相当武断。不足为奇的是，比较统计数据的研究表明，统计数据得出了相互矛盾的结论，即哪些项目显示出统计不匹配，因此应该被排除在测试之外(例如，见Chon，Lee，&Ansley，2013)。关于项目匹配决定的第二个问题涉及这样一个事实，即在确定项目排除时没有考虑项目不匹配的实际意义(即对相关测试结果的影响)。这在很大程度上是因为，到目前为止，还没有容易使用的方法来评估实际意义。拟议项目的目的是：(1)就教育大规模评估中相对常见和较新的项目匹配统计数据的表现为从业人员制定指导方针；(2)制定评价项目不匹配的实际意义的标准。为了实现第一个研究目标，将进行模拟研究，调查可能影响Fit统计量性能的因素(即其I类错误率和功率)。这些因素包括(I)样本量，(ii.)不匹配与项目参数之间的相互作用，(Iii)违反范本的类型(四)不匹配的大小，(V)数据中不匹配项目的数量，以及(Vi)数据中缺失值的数量和类型。关于不匹配的大小，我们计划提出不同的效果大小测量方法，以区分小、中、大项目不匹配。第二个主要目标是开发方法来确定与与低风险教育测试相关的结果不匹配的实际意义，包括(I)分析胜任力与协变量之间的关系，以及(Ii)随着时间的推移进行能力比较。我们将使用经验数据(例如，来自PISA或NEP)来验证我们的发现并说明我们的方法。

Testing model fit is considered an important step in item response theory (IRT) modeling, since model fit is a necessary prerequisite for drawing valid inferences from estimated parameters (Wainer & Thissen, 1987). Hambleton and Hahn (2005) suggest several steps to evaluate model fit, including (a) the calculation of item fit statistics and (b) investigating the consequences of misfit with regard to test outcomes. In educational large-scale assessments, various item fit indices are employed. Many of these statistics show severe limitations such as lack of theoretical proof about the distribution of the test statistic (Liang, Wells, & Hambleton, 2014). This lack of methodological basis for determining accurate cut-off values makes decisions regarding item fit rather arbitrary. Unsurprisingly, studies comparing the statistics have demonstrated that the statistics result in contradictory conclusions regarding which items show statistical misfit and should hence be excluded from the test (see, e.g., Chon, Lee, & Ansley, 2013). A second issue regarding decisions about item fit concerns the fact that practical significance of item misfit (i.e., effects on relevant test outcomes) is not taken into account when determining item exclusions. This is mostly due the fact that, thus far, no readily usable method for evaluating practical significance exists. The proposed project aims to (1) establish guidelines for practitioners about the performance of relatively common as well as more recent item fit statistics in educational large-scale assessments and (2) develop criteria for evaluating practical significance of item misfit. Simulation studies will be conducted to pursue the first research objective, investigating possible influencing factors on the fit statistics` performance (i.e., their Type I error rates and power). These factors include (i.) sample size, (ii.) interaction between misfit and item parameters, (iii.) type of model violation (iv.) size of misfit, (v.) amount of misfitting items in the data, and (vi.) amount and type of missing values in the data. Regarding the size of misfit, we plan to propose different effect size measures that allow distinguishing between small, medium, and large item misfit. The second major objective is to develop methods to determine practical significance of misfit for outcomes that are relevant in low-stakes educational testing, including (i.) analyses on relationships between competence and covariates, and (ii.) competence comparisons over time. We will use empirical data (e.g., from PISA or NEPS) to validate our findings and to illustrate our methods.