The Subjective and Objective Interface of Bias Detection on language tests

语言测试偏差检测的主客观界面

• 批准号: 15520366
• 负责人: OKABE Junko
• 金额: $ 1.22万
• 依托单位: Aichi Prefectural University
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (C)
• 财政年份: 2003
• 资助国家: 日本
• 起止时间: 2003 至 2004
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-15520366/
• 关键词: gender bias; test; DIF; schemata; subjectivity; gender bias

Test validity is predicted on there being a lack of bias in tasks, items, or test content. It is well known that factors such as test candidates' mother tongue, life experiences and socialization practices of the wider community may serve to inject subtle interactions between individuals' background and the test content. When the gender of the test candidate interacts further with these factors, the potential for item bias to influence test performances grows. A dilemma faced by test designers concerns how they can proactively screen test content for possible sources of bias. Conventional practices in many contexts rely on the subjective opinion of review panels in detecting potentially biased items. In the last two decades, this practice has been rivaled by the increased availability of item bias diagnostic software. Few studies have compared the relative accuracy and cost utility of the two approaches. The present study makes just that comparison. A fourpassage, 20 item reading comprehension test was given to a stratified sample of 825 high school students and college undergraduates at five Japanese institutions. The sampling included a focus group of 468 female students compared to a reference group of 357 male English as a foreign language learners. The test passages and items were also given to a panel of 97 in-service and pre-service EFL teachers for subjective ratings of potential gender bias. The results of the actual item responses were then empirically checked for evidence of ditl+erential item functioning using Simultaneous Item Bias analysis, the Mantel-Haenszel Delta method, and logistic regression. Concordance analyses of the subjective and objective methods suggest that subjective screening of bias overestimates the extent of actual item bias. Implications for cost-effective approaches to item bias detection are discussed.

测试效度是在任务、项目或测试内容缺乏偏见的情况下预测的。众所周知，考生的母语、生活经历和更广泛社区的社交实践等因素可能会在个人背景和考试内容之间注入微妙的互动。当应试者的性别与这些因素进一步相互作用时，项目偏差影响测试成绩的可能性就会增加。测试设计者面临的一个两难境地是，他们如何主动筛选测试内容，以寻找可能的偏见来源。在许多情况下，传统做法依赖于审查小组在发现可能存在偏见的项目时的主观意见。在过去的二十年里，这种做法已经被越来越多的项目偏向诊断软件所取代。很少有研究比较这两种方法的相对准确性和成本效用。目前的研究正是对此进行了比较。对日本五所高校的825名高中生和本科生进行了四道题、20道题的分层抽样调查。样本包括468名女性学生组成的焦点组，与357名男性英语学习者作为外语学习者的参照组进行比较。测试段落和项目也被交给了一个由97名在职和任职前的英语教师组成的小组，以对潜在的性别偏见进行主观评级。然后，使用同步项目偏差分析、Mantel-Haenszel Delta方法和Logistic回归对实际项目反应的结果进行经验性检验，以寻找双项目+事件项目功能的证据。主观和客观方法的一致性分析表明，主观筛选偏差高估了实际项目偏差的程度。讨论了具有成本效益的项目偏向检测方法的含义。