FuzzE: Fuzzy Fairness Evaluation of Offensive Language Classifiers on African-American English
FuzzE: Fuzzy Fairness Evaluation of Offensive Language Classifiers on African-American English
复制标题
DOI：
10.1609/aaai.v34i01.5434
复制
发表时间：
2020-04
期刊：
影响因子：
--
通讯作者：
Anthony Rios
中科院分区：
文献类型：
--
作者：
Anthony Rios
社交媒体上的仇恨言论和攻击性语言十分猖獗。机器学习提供了一种大规模缓和脏话的方法。然而，目前的研究大多集中在整体表现上。模型在用少数民族方言写的文本上可能表现不佳。例如，仇恨言论分类器可能会对用非裔美国人白话英语（AAVE）写的推文产生更多的误报。为了衡量这些问题，我们需要用AAVE和标准美国英语（SAE）编写的文本。不幸的是，及时整理所有语言风格的数据是一项挑战，尤其是当我们受到特定问题、社交媒体平台或资源有限的限制时。在本文中，我们回答了这个问题，“当少数民族方言语言不存在于特定数据集中时，我们如何评估分类器的性能？”具体来说，我们提出了一种名为fuzzze的自动公平性模糊测试工具，该工具使用仅包含SAE文本的数据集来量化应用于AAVE文本的文本分类器的公平性。总的来说，我们发现我们的技术返回的公平性估计与真实的ground-truth AAVE文本的使用适度相关。警告：本手稿中有冒犯性语言。
Hate speech and offensive language are rampant on social media. Machine learning has provided a way to moderate foul language at scale. However, much of the current research focuses on overall performance. Models may perform poorly on text written in a minority dialectal language. For instance, a hate speech classifier may produce more false positives on tweets written in African-American Vernacular English (AAVE). To measure these problems, we need text written in both AAVE and Standard American English (SAE). Unfortunately, it is challenging to curate data for all linguistic styles in a timely manner—especially when we are constrained to specific problems, social media platforms, or by limited resources. In this paper, we answer the question, “How can we evaluate the performance of classifiers across minority dialectal languages when they are not present within a particular dataset?” Specifically, we propose an automated fairness fuzzing tool called FuzzE to quantify the fairness of text classifiers applied to AAVE text using a dataset that only contains text written in SAE. Overall, we find that the fairness estimates returned by our technique moderately correlates with the use of real ground-truth AAVE text. Warning: Offensive language is displayed in this manuscript.