WTG: Beyond the black box: understanding the use of algorithmic risk assessments in the juvenile justice system

WTG：超越黑匣子：了解算法风险评估在少年司法系统中的使用

• 批准号: 2244705
• 负责人: Kelly Murphy
• 金额: $ 45万
• 依托单位: Child Trends Inc
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-02-15 至 2025-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2244705&HistoricalAwards=false
• 关键词: WTG; Beyond; black; box; understanding

Research has consistently shown that youth, who come into contact with the juvenile justice system, are less likely to re-offend when the intensity of the intervention that they receive (e.g., placement in a secure residential facility vs. participation in community-based services) is aligned with their relative risk for reoffending (Lipsey, 2009). Accordingly, juvenile justice systems throughout the nation have adopted research-based, juvenile risk and needs assessment tools that can be used to inform youths’ case plans. These tools, many of which are algorithmically based, are used by decision makers, such as judges, to estimate youths’ risk for reoffending as well as identify factors that may make reoffense more (or less) likely. Although research has generally shown that risk assessment tools are more accurate and less biased than the unstructured judgements of decision makers, some research has shown that these tools have the potential to perpetuate and mask racial disparities and other historical inequities (Goal et al., 2021; St. John et al., 2020). Research on risk assessments has been limited, however, by the fact that it has almost exclusively focused on the tools themselves—rarely examining how decision makers use the tools in practice. This limited focus is problematic because in practice, these tools are intended to provide information to decision makers rather than replace them. Thus, biases can potentially be introduced not just in the risk assessments themselves but also in decisions on whether to accept or reject the assessment's recommendation. To address this issue, this study will examine how decision makers use risk assessments. Importantly, we will examine how different contextual factors affect the use of the tool as well as how the use of the tool affects youth outcomes. A key goal of this study will be to develop policy and practice guidance to promote more effective and equitable use of risk assessments. To achieve the study aims, we will partner with a juvenile justice agency in a Southern state—which implements a statewide juvenile risk and needs assessment tool—to conduct a three-year mixed-methods study. We will conduct primary data collection, including a survey and focus groups, with judges and analyze secondary data from the state juvenile justice and education agencies, law enforcement, and the American Community Survey. Secondary data will include variables such as youths’ sociodemographic characteristics (e.g., race/ethnicity, age, and sex); risk assessment scores; case histories (e.g., offense severity); disposition received (e.g., probation, commitment); recidivism (12-month re-adjudication); and educational outcomes (e.g., high school completion) as well as community characteristics, such as the presence of racial disparities in policing, median household income, and the percent of residents that are people of color. These data allow us to directly identify cases in which the disposition a youth received differs from what the risk score recommended. We will use data from June 2017 to July 2023 (with sensitivity analyses performed to assess how inclusion of data from the pandemic affects study results), which we anticipate will include approximately 23,000 youth records. We will use regression analysis and multi-level modeling to answer our research questions—assessing both (1) the circumstances that increase the likelihood a decision maker will override the risk assessment; and (2) how use (or non-use) of risk assessments’ recommendations affect youths’ educational outcomes and likelihood of reoffending. We will supplement administrative data with the primary data collection to gather more in-depth data on how judges’ use and perceive the use of risk assessments. We will present the study findings in multiple ways, so they are accessible to both practitioners and researchers, including formats such as a webinar, conference presentation, and journal articles.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

研究一直表明，与少年司法系统接触的青年，如果他们受到的干预强度（例如，安置在安全的住宅设施与参与社区服务相比）与他们重新犯罪的相对风险相一致（Lipsey，2009年）。因此，全国各地的少年司法系统采用了以研究为基础的少年风险和需求评估工具，可用于为青少年的案件计划提供信息。这些工具，其中许多是基于算法的，由决策者，如法官，用来估计青少年的重新犯罪的风险，以及确定可能使重新犯罪的可能性更大（或更小）的因素。虽然研究普遍表明，风险评估工具比决策者的非结构化判断更准确，偏见更少，但一些研究表明，这些工具有可能使种族差异和其他历史不平等永久化并掩盖这些差异和其他历史不平等（Goal等人，2021; St. John等人，2020年）。然而，对风险评估的研究受到了限制，因为它几乎完全集中在工具本身，很少研究决策者如何在实践中使用这些工具。这种有限的重点是有问题的，因为在实践中，这些工具的目的是向决策者提供信息，而不是取代他们。因此，不仅在风险评估本身，而且在决定是否接受或拒绝评估建议时，都可能产生偏见。为了解决这个问题，本研究将探讨决策者如何使用风险评估。重要的是，我们将研究不同的背景因素如何影响工具的使用，以及工具的使用如何影响青年的成果。这项研究的一个关键目标是制定政策和实践指南，以促进更有效和公平地使用风险评估。为了实现研究的目标，我们将与一个少年司法机构在南方国家的合作伙伴，它实现了全州范围内的青少年风险和需求评估工具，进行为期三年的混合方法研究。我们将进行主要的数据收集，包括调查和焦点小组，与法官和分析从国家少年司法和教育机构，执法和美国社区调查的二级数据。次级数据将包括青年的社会人口特征等变量（例如，种族/民族、年龄和性别）;风险评估评分;病历（例如，罪行严重性）;收到的处置（例如，缓刑，承诺）;累犯（12个月重新判决）;和教育成果（例如，高中毕业）以及社区特征，如警务中存在的种族差异，家庭收入中位数以及有色人种居民的百分比。这些数据使我们能够直接识别青少年接受的处置与风险评分建议不同的情况。我们将使用2017年6月至2023年7月的数据（进行敏感性分析，以评估纳入大流行数据如何影响研究结果），我们预计将包括约23，000份青年记录。我们将使用回归分析和多层次建模来回答我们的研究问题，评估（1）增加决策者推翻风险评估的可能性的情况;以及（2）如何使用（或不使用）风险评估的建议影响青少年的教育成果和重新犯罪的可能性。我们将用原始数据收集补充行政数据，以收集关于法官如何使用和如何看待风险评估使用情况的更深入数据。我们将以多种方式展示研究成果，以便从业者和研究人员都能获得，包括网络研讨会、会议演示和期刊文章等形式。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。