The Subjective and Objective Interface of Bias Detection on language tests

测试有效性的预测是在任务，项目或测试内容中缺乏偏见。众所周知，考生的母语、生活经历和更广泛社区的社会化实践等因素可能有助于在个人背景和考试内容之间注入微妙的相互作用。当测试候选人的性别进一步与这些因素相互作用时，项目偏见影响测试表现的可能性就会增加。测试设计者面临的一个困境是如何主动筛选测试内容中可能存在的偏见。在许多情况下，传统做法依赖于审查小组的主观意见来发现可能有偏见的项目。在过去的二十年里，这种做法已经与项目偏差诊断软件的增加相媲美。很少有研究比较这两种方法的相对准确性和成本效用。本研究正是做了这样的比较。本研究采用四通道20题阅读理解测验，对日本五所大学的825名高中生和大学生进行分层抽样。抽样包括468名女学生的焦点组相比，357名男性英语作为外语学习者的参考组。测试的文章和项目也给了一个由97名在职和职前英语教师组成的小组，对潜在的性别偏见进行主观评分。然后，使用同时项目偏倚分析、Mantel-Haenszel Delta方法和逻辑回归，对实际项目反应的结果进行经验性检查，以获得ditl+ eeps项目功能的证据。主观和客观方法的一致性分析表明，主观筛选的偏见高估了实际项目的偏见的程度。项目偏差检测的成本效益的方法的影响进行了讨论。

Test validity is predicted on there being a lack of bias in tasks, items, or test content. It is well known that factors such as test candidates' mother tongue, life experiences and socialization practices of the wider community may serve to inject subtle interactions between individuals' background and the test content. When the gender of the test candidate interacts further with these factors, the potential for item bias to influence test performances grows. A dilemma faced by test designers concerns how they can proactively screen test content for possible sources of bias. Conventional practices in many contexts rely on the subjective opinion of review panels in detecting potentially biased items. In the last two decades, this practice has been rivaled by the increased availability of item bias diagnostic software. Few studies have compared the relative accuracy and cost utility of the two approaches. The present study makes just that comparison. A fourpassage, 20 item reading comprehension test was given to a stratified sample of 825 high school students and college undergraduates at five Japanese institutions. The sampling included a focus group of 468 female students compared to a reference group of 357 male English as a foreign language learners. The test passages and items were also given to a panel of 97 in-service and pre-service EFL teachers for subjective ratings of potential gender bias. The results of the actual item responses were then empirically checked for evidence of ditl+erential item functioning using Simultaneous Item Bias analysis, the Mantel-Haenszel Delta method, and logistic regression. Concordance analyses of the subjective and objective methods suggest that subjective screening of bias overestimates the extent of actual item bias. Implications for cost-effective approaches to item bias detection are discussed.