Statistical and practical significance of item misfit in educational testing

在项目反应理论（IRT）建模中，模型拟合被认为是一个重要的步骤，因为模型拟合是从估计参数中得出有效推论的必要前提（Wainer & Thissen，1987）。Hambleton和Hahn（2005）提出了几个评估模型拟合的步骤，包括（a）项目拟合统计量的计算和（B）调查与测试结果相关的失配后果。在教育大规模评估中，会使用各种项目拟合指数。许多这些统计量显示出严重的局限性，例如缺乏关于检验统计量分布的理论证明（Liang，威尔斯，& Hambleton，2014）。由于缺乏确定准确截止值的方法学基础，因此关于项目适合性的决定相当武断。毫不奇怪，比较统计数据的研究表明，统计数据导致关于哪些项目显示统计失配并因此应从测试中排除的矛盾结论（参见，例如，Chon，Lee，& Ansley，2013）.关于关于项匹配的决定的第二个问题涉及项不匹配的实际意义（即，对相关测试结果的影响）。这主要是因为，迄今为止，还没有现成可用的方法来评估实际意义。该项目的目的是（1）为从业人员制定指导方针，以了解教育大规模评估中相对常见的以及最近的项目匹配统计数据的表现;（2）制定评估项目不匹配的实际意义的标准。将进行模拟研究以实现第一个研究目标，调查拟合统计性能的可能影响因素（即，类型I错误率和功率）。这些因素包括（i.）样本量，（二）失配和项目参数之间的相互作用，（iii.）违反模型的类型（四）错配的大小，（v.）数据中的失配项的量，以及（vi.）数据中缺失值的数量和类型。关于错配的大小，我们计划提出不同的效应大小措施，允许区分小，中，大项目错配。第二个主要目标是开发方法来确定与低风险教育测试相关的结果的不匹配的实际意义，包括（i）分析能力和协变量之间的关系，以及（ii.）时间上的能力对比。我们将使用经验数据（例如，从比萨或NEPS），以验证我们的发现，并说明我们的方法。

Testing model fit is considered an important step in item response theory (IRT) modeling, since model fit is a necessary prerequisite for drawing valid inferences from estimated parameters (Wainer & Thissen, 1987). Hambleton and Hahn (2005) suggest several steps to evaluate model fit, including (a) the calculation of item fit statistics and (b) investigating the consequences of misfit with regard to test outcomes. In educational large-scale assessments, various item fit indices are employed. Many of these statistics show severe limitations such as lack of theoretical proof about the distribution of the test statistic (Liang, Wells, & Hambleton, 2014). This lack of methodological basis for determining accurate cut-off values makes decisions regarding item fit rather arbitrary. Unsurprisingly, studies comparing the statistics have demonstrated that the statistics result in contradictory conclusions regarding which items show statistical misfit and should hence be excluded from the test (see, e.g., Chon, Lee, & Ansley, 2013). A second issue regarding decisions about item fit concerns the fact that practical significance of item misfit (i.e., effects on relevant test outcomes) is not taken into account when determining item exclusions. This is mostly due the fact that, thus far, no readily usable method for evaluating practical significance exists. The proposed project aims to (1) establish guidelines for practitioners about the performance of relatively common as well as more recent item fit statistics in educational large-scale assessments and (2) develop criteria for evaluating practical significance of item misfit. Simulation studies will be conducted to pursue the first research objective, investigating possible influencing factors on the fit statistics` performance (i.e., their Type I error rates and power). These factors include (i.) sample size, (ii.) interaction between misfit and item parameters, (iii.) type of model violation (iv.) size of misfit, (v.) amount of misfitting items in the data, and (vi.) amount and type of missing values in the data. Regarding the size of misfit, we plan to propose different effect size measures that allow distinguishing between small, medium, and large item misfit. The second major objective is to develop methods to determine practical significance of misfit for outcomes that are relevant in low-stakes educational testing, including (i.) analyses on relationships between competence and covariates, and (ii.) competence comparisons over time. We will use empirical data (e.g., from PISA or NEPS) to validate our findings and to illustrate our methods.