The Trade-Off between Model Fit, Invariance, and Validity: The Case of PISA Science Assessments
The Trade-Off between Model Fit, Invariance, and Validity: The Case of PISA Science Assessments
复制标题
DOI：
10.1080/08957347.2020.1732384
复制
发表时间：
2020-02-29
影响因子：
1.5
通讯作者：
Andrich, David
中科院分区：
文献类型：
--
作者：
El Masri, Yasmine H.;Andrich, David
在大规模的教育评估中，通常要求测试由在要比较的组中不变地发挥作用的项目组成。尽管努力确保物品建造阶段的不变性，但由于一系列原因(包括物品的安全)，往往有必要说明物品的不同功能。这通常需要在保留项目而不考虑其DIF、删除项目或通过为每个组创建不同的项目来解析(拆分)项目之间进行选择。这些选项涉及模型拟合和项目参数不变性之间的权衡，每个选项可能是有效的，取决于DIF的来源是否与被评估的变量相关。我们认为，做出选择需要仔细分析统计DIF及其实质来源。我们通过使用Rasch模型分析三个国家(英国、法国和约旦)的PISA 2006年的科学数据来说明我们的论点，该模型用于分析所有PISA 2006年的数据。我们确定了具有真实DIF的各个国家的项目，并检查了当这些项目被消除、解决或保留时，对模型拟合度、不变性以及跨国比较的有效性的影响。
In large-scale educational assessments, it is generally required that tests are composed of items that function invariantly across the groups to be compared. Despite efforts to ensure invariance in the item construction phase, for a range of reasons (including the security of items) it is often necessary to account for differential item functioning (DIF) of items post hoc. This typically requires a choice among retaining an item as it is despite its DIF, deleting the item, or resolving (splitting) an item by creating a distinct item for each group. These options involve a trade-off between model fit and the invariance of item parameters, and each option could be valid depending on whether or not the source of DIF is relevant or irrelevant to the variable being assessed. We argue that making a choice requires a careful analysis of statistical DIF and its substantive source. We illustrate our argument by analyzing PISA 2006 science data of three countries (UK, France and Jordan) using the Rasch model, which was the model used for the analyses of all PISA 2006 data. We identify items with real DIF across countries and examine the implications for model fit, invariance, and the validity of cross-country comparisons when these items are either eliminated, resolved or retained.