权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

Statistical and practical significance of item misfit in educational testing

教育测试中项目失配的统计意义和实际意义

基本信息

批准号：
388489847
负责人：
Dr. Carmen Köhler
金额：
--
依托单位：
DIPF I Leibniz-Institut für Bildungsforschung und Bildungsinformation
依托单位国家：
德国
项目类别：
Research Grants
财政年份：
2017
资助国家：
德国
起止时间：
2016-12-31 至 2020-12-31
项目状态：
已结题

来源：
https://gepris.dfg.de/gepris/projekt/388489847?language=en
关键词：
Statistical practical significance item misfit

项目摘要

Testing model fit is considered an important step in item response theory (IRT) modeling, since model fit is a necessary prerequisite for drawing valid inferences from estimated parameters (Wainer & Thissen, 1987). Hambleton and Hahn (2005) suggest several steps to evaluate model fit, including (a) the calculation of item fit statistics and (b) investigating the consequences of misfit with regard to test outcomes. In educational large-scale assessments, various item fit indices are employed. Many of these statistics show severe limitations such as lack of theoretical proof about the distribution of the test statistic (Liang, Wells, & Hambleton, 2014). This lack of methodological basis for determining accurate cut-off values makes decisions regarding item fit rather arbitrary. Unsurprisingly, studies comparing the statistics have demonstrated that the statistics result in contradictory conclusions regarding which items show statistical misfit and should hence be excluded from the test (see, e.g., Chon, Lee, & Ansley, 2013). A second issue regarding decisions about item fit concerns the fact that practical significance of item misfit (i.e., effects on relevant test outcomes) is not taken into account when determining item exclusions. This is mostly due the fact that, thus far, no readily usable method for evaluating practical significance exists. The proposed project aims to (1) establish guidelines for practitioners about the performance of relatively common as well as more recent item fit statistics in educational large-scale assessments and (2) develop criteria for evaluating practical significance of item misfit. Simulation studies will be conducted to pursue the first research objective, investigating possible influencing factors on the fit statistics` performance (i.e., their Type I error rates and power). These factors include (i.) sample size, (ii.) interaction between misfit and item parameters, (iii.) type of model violation (iv.) size of misfit, (v.) amount of misfitting items in the data, and (vi.) amount and type of missing values in the data. Regarding the size of misfit, we plan to propose different effect size measures that allow distinguishing between small, medium, and large item misfit. The second major objective is to develop methods to determine practical significance of misfit for outcomes that are relevant in low-stakes educational testing, including (i.) analyses on relationships between competence and covariates, and (ii.) competence comparisons over time. We will use empirical data (e.g., from PISA or NEPS) to validate our findings and to illustrate our methods.

测试模型拟合被认为是项目反应理论（IRT）建模的重要步骤，因为模型拟合是从估计参数中得出有效推论的必要前提（Wainer & Thissen, 1987）。Hambleton和Hahn（2005）提出了评估模型拟合的几个步骤，包括(a)计算项目拟合统计数据和(b)调查不拟合对测试结果的影响。在教育大规模评估中，采用了多种项目契合度指标。其中许多统计数据显示出严重的局限性，例如缺乏关于检验统计量分布的理论证明（Liang, Wells, & Hambleton, 2014）。由于缺乏确定准确临界值的方法学基础，使得有关项目拟合的决定相当武断。不出所料，比较统计数据的研究表明，统计数据导致了关于哪些项目显示统计不匹配的矛盾结论，因此应该从测试中排除（参见，例如，Chon, Lee, & Ansley, 2013）。关于项目拟合决策的第二个问题是，在确定项目排除时，没有考虑到项目不拟合的实际意义（即对相关测试结果的影响）。这主要是由于到目前为止，还没有现成可用的方法来评估实际意义。拟议的项目旨在(1)为从业者建立关于在教育大规模评估中相对常见的以及最近的项目拟合统计性能的指导方针；(2)制定评估项目不拟合的实际意义的标准。为了实现第一个研究目标，将进行仿真研究，调查可能影响拟合统计性能的因素（即它们的I型错误率和功率）。这些因素包括（i.）样本量，（ii.）错拟和项目参数之间的相互作用，（iii.）模型违反的类型（iv.）错拟的大小，（v.）数据中错拟项目的数量，以及（vi.）数据中缺失值的数量和类型。关于错配的大小，我们计划提出不同的效应大小测量，以区分小、中、大项目错配。第二个主要目标是开发方法，以确定与低风险教育测试相关的结果的不匹配的实际意义，包括(i)对能力和协变量之间关系的分析，以及（ii）随时间的能力比较。我们将使用经验数据（例如，来自PISA或NEPS）来验证我们的发现并说明我们的方法。