Instructional Sensitivity of Test Items in Educational Testing

学生在教育考试中的表现通常归因于或多或少有效的教学。这意味着，作为结果衡量标准的标准化考试在实践上是敏感的，即对考试项目的反应实际上受到学生所接受的指导的影响。然而，这一假设很少被实证研究。为了验证这一假设，不同的操作教学敏感性已被提出。对于拟议的项目，我们建立在Naumann，Hochweber和Hartig（2013）之前的工作基础上，他们提出了一个纵向多级DIF模型，将联合收割机两种现有的但独立的方法结合起来，以评估项目的教学敏感性。该模型结合了随着时间的推移和教室之间的差异性能的变化，允许全球和差分敏感性的区别。拟议的项目有四个主要目标，其中前两个主要是方法性的：（1）进一步发展和扩展LML-DIF模型，（2）检查参数估计的质量和条件。另外两个目标与课堂评估测试开发的实质性问题有关：（3）验证教学敏感性的统计指标，以及（4）开发测试和项目水平上的教学敏感性分类标准。实现这些目标的一揽子工作可分为模型开发、模拟研究、现有数据集的次级分析和在瑞士学校进行的实证研究。关于模型开发，我们的目标是通过以下方式扩展可访问的研究问题的范围：a）在测试水平上更明确地纳入教学敏感性，B）纳入预测变量，c）纳入不同的项目歧视，d）对多分类反应的概括，e）以及对趋势数据的应用。模拟研究将支持模型开发，旨在更深入地了解模型参数估计的质量和条件（例如所需的样本量）。现有的测试数据的二次分析将有助于提供“高”和“低”的教学敏感性的标准，尽可能多的合适的数据集应用教学敏感性模型。此外，我们的目标是验证教学敏感性的统计指标，通过检查它们与项目和课堂水平的预测。计划进行额外的二次分析，以测试新开发的教学敏感性模型在实证应用。由于现有的数据集缺乏几个变量，我们的研究问题，我们正计划收集新的测试数据在圣加仑州，瑞士的实证研究。本研究的重点将是教学内容的详细测量以及测试项目与教学关系的详尽评级。

Students' performance in educational tests is commonly attributed to more or less effective teaching. This implies that standardized tests used as outcome measures are instructionally sensitive, i.e. responses to test items are actually affected by instruction the students have received. However, this assumption is rarely investigated empirically. To investigate this assumption empirically, different operationalizations of instructional sensitivity have been proposed. For the proposed project we are building on previous work by Naumann, Hochweber, and Hartig (2013), who proposed a longitudinal multilevel-DIF model to combine two existing yet independent approaches to evaluate items' instructional sensitivity. The model incorporates of changes in performance over time and differences between classrooms, allowing the distinction of global and differential sensitivity. The proposed project aims at four major objectives, the first two of them being primarily of methodological nature: (1) the further development and extension of the LML-DIF model, and (2) the examination of quality and conditions of parameter estimation. Two further objectives are related to substantive questions of classroom assessment test development: (3) the validation of the statistical indicators for instructional sensitivity, and (4) the development of criteria to classify instructional sensitivity on test and item level. The work packages to achieve these goals can be grouped into model development, simulation studies, secondary analyses of existing data sets, and an empirical study conducted at Swiss schools. With respect to model development, we aim at extending the range of accessible research questions by a) a more explicit incorporation of instructional sensitivity on the test level, b) the incorporation of predictor variables, c) the incorporation of varying item discriminations, d) the generalization to polytomous responses, e) and the application to trend data. Simulation studies will support the model development and aim at gaining a deeper knowledge of the quality and conditions (e.g. required sample sizes) of model parameter estimation. Secondary analyses of existing test data will serve to provide criteria for "high" and "low" instructional sensitivity by applying instructional sensitivity models to as many suitable data sets as possible. Furthermore, we aim to validate the statistical indicators for instructional sensitivity by examining their relations with item and classroom level predictors. Additional of secondary analyses are planned to test newly developed instructional sensitivity models in empirical applications. Since available datasets lack several variables desirable for our research questions, we are planning to collect new test data in an empirical study in the Canton St. Gallen, Switzerland. The focus of this study will be to detailed measures of instructional content and exhaustive ratings of test items regarding their relation to instruction.