Statistical and Psychometric Methods for Measuring the Extent to Which Culturally Responsive Assessments Reduce Cultural Bias

这个研究项目将开发一套统计工具来分析来自文化响应性评估的数据。对于标准化评估通过其内容、设计和使用使社会不公正永久化的担忧，文化响应性评估已经成为一种潜在的解决方案。然而，很少有真正的文化响应性评估存在，并且缺乏经过验证的统计方法来分析从中获得的数据。本项目将使用有待开发的工具和项目人员收集的数据来衡量文化响应性评估在多大程度上减少了文化偏见。科学产品将通过社交媒体、研讨会和同行评议期刊的出版物进行传播。开源软件的开发将使这些工具更容易被更广泛的受众使用。该项目将让研究生参与这项研究，调查人员将在他们的项目中包括来自代表性不足群体的学生。这个项目的发现和成果将对社会、行为和经济科学领域的学者有用。该项目还将对社会产生影响，因为它将提供一套新的工具来促进社会正义。这一研究项目将为学者们提供新的工具，以便从文化反应性评估中得出公平的分数。项目团队首先将研究几种统计模型和方法，以分析来自文化响应性评估的数据，并使用模拟数据证明这些方法的有效性。然后，项目组将在一个由研究人员和实践者组成的多学科团队的帮助下制定若干文化响应性评估。这些评估将对数千名从亚马逊土耳其机械或多产等平台获得的考生进行管理。将使用新开发的模型和方法来分析数据。分析的重点是衡量标准化考试中文化偏见的类型和程度，并确定对文化有反应的评估在多大程度上产生更公平和公平的结果。该项目将制定可公开获得的原则和准则，以便编制对文化有反应的评估和用户友好的软件以及随附的手册，以执行本项目制定的方法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This research project will develop a set of statistical tools to analyze data from culturally responsive assessments. Culturally responsive assessments have emerged as a potential solution to the concern that standardized assessments perpetuate social injustice through their content, design, and use. However, few true culturally responsive assessments exist, and there is a lack of proven statistical methods to analyze data from them. This project will use the tools to be developed and data collected by the project personnel to measure the extent to which culturally responsive assessments reduce cultural bias. Scientific products will be disseminated via social media, workshops, and publications in peer-reviewed journals. Open-source software will be developed to make the tools more accessible to a broader audience. The project will engage graduate students in conducting this research, and the investigators will include students from underrepresented groups in their project. The findings and products originating from this project will be useful for scholars across the social, behavioral, and economic sciences. The project also will have implications for society as it will provide a new set of tools to promote social justice.This research project will provide scholars with new tools for producing fair scores from culturally responsive assessments. The project team first will examine several statistical models and approaches to analyzing data from culturally responsive assessments and demonstrate the effectiveness of these approaches using simulated data. The project team then will develop several culturally responsive assessments with the help of a multidisciplinary team of researchers and practitioners. These assessments will be administered to several thousand examinees obtained from a platform such as Amazon Mechanical Turk or Prolific. The newly developed models and approaches will be used to analyze the data. The analyses will focus on measuring the type and extent of cultural bias in standardized tests and determining the extent to which culturally responsive assessments produce more fair and equitable results. The project will produce publicly available principles and guidelines for preparing culturally responsive assessments and user-friendly software and accompanying manual to implement the approaches developed in this project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.