The Subjective and Objective Interface of Bias Detection on language tests

测试效度是在任务、项目或测试内容中缺乏偏差的情况下预测的。众所周知，考生的母语、生活经历和更广泛社区的社交实践等因素可能会在个人背景和考试内容之间注入微妙的相互作用。当测试候选人的性别与这些因素进一步相互作用时，项目偏见影响测试表现的可能性就会增加。测试设计人员面临的一个困境是，他们如何主动筛选测试内容，以寻找可能的偏见来源。在许多情况下，传统做法依靠审查小组的主观意见来发现可能存在偏见的项目。在过去的二十年里，这种做法已经被越来越多的项目偏见诊断软件所取代。很少有研究比较这两种方法的相对准确性和成本效用。目前的研究正是做了这样的比较。对日本五所大学的825名高中生和大学生进行了一项四段式、20项的阅读理解测试。抽样包括一个由468名女学生组成的焦点小组，而参照组是357名作为外语学习英语的男学生。测试的段落和项目也交给了一个由97名在职和职前英语教师组成的小组，对潜在的性别偏见进行主观评分。实际项目反应的结果，然后使用同时项目偏差分析，Mantel-Haenszel方法和逻辑回归，经验性地检查ditl+erential项目功能的证据。主观和客观方法的一致性分析表明，主观筛选偏倚高估了实际项目偏倚的程度。讨论了项目偏差检测的成本效益方法的含义。

Test validity is predicted on there being a lack of bias in tasks, items, or test content. It is well known that factors such as test candidates' mother tongue, life experiences and socialization practices of the wider community may serve to inject subtle interactions between individuals' background and the test content. When the gender of the test candidate interacts further with these factors, the potential for item bias to influence test performances grows. A dilemma faced by test designers concerns how they can proactively screen test content for possible sources of bias. Conventional practices in many contexts rely on the subjective opinion of review panels in detecting potentially biased items. In the last two decades, this practice has been rivaled by the increased availability of item bias diagnostic software. Few studies have compared the relative accuracy and cost utility of the two approaches. The present study makes just that comparison. A fourpassage, 20 item reading comprehension test was given to a stratified sample of 825 high school students and college undergraduates at five Japanese institutions. The sampling included a focus group of 468 female students compared to a reference group of 357 male English as a foreign language learners. The test passages and items were also given to a panel of 97 in-service and pre-service EFL teachers for subjective ratings of potential gender bias. The results of the actual item responses were then empirically checked for evidence of ditl+erential item functioning using Simultaneous Item Bias analysis, the Mantel-Haenszel Delta method, and logistic regression. Concordance analyses of the subjective and objective methods suggest that subjective screening of bias overestimates the extent of actual item bias. Implications for cost-effective approaches to item bias detection are discussed.