A Comprehensive Study of Gender Bias in Chemical Named Entity Recognition Models
A Comprehensive Study of Gender Bias in Chemical Named Entity Recognition Models
复制标题
DOI：
10.48550/arxiv.2212.12799
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Xingmeng Zhao;A. Niazi;Anthony Rios
中科院分区：
文献类型：
--
作者：
Xingmeng Zhao;A. Niazi;Anthony Rios
化学命名实体识别（NER）模型用于许多下游任务，从药物不良反应识别到药物流行病学。然而，目前尚不清楚这些模型是否适用于所有人。性能差异可能会造成潜在的伤害，而不是预期的好处。本文评估了化学NER系统中与性别相关的性能差异。我们开发了一个框架，用于测量化学NER模型中的性别偏见，使用合成数据和来自Reddit的超过92,405个单词的新注释语料库，其中包含自我识别的性别信息。我们对多个生物医学NER模型的评估显示出明显的偏差。例如，合成数据表明，女性名字经常被错误地归类为化学品，尤其是在提到品牌名称时。此外，我们在两个数据集中观察到女性和男性相关数据之间的性能差异。许多系统无法检测诸如节育之类的避孕措施。我们的研究结果强调了化学NER模型中的偏差，敦促从业者在下游应用中考虑这些偏差。
Chemical named entity recognition (NER) models are used in many downstream tasks, from adverse drug reaction identification to pharmacoepidemiology. However, it is unknown whether these models work the same for everyone. Performance disparities can potentially cause harm rather than the intended good. This paper assesses gender-related performance disparities in chemical NER systems. We develop a framework for measuring gender bias in chemical NER models using synthetic data and a newly annotated corpus of over 92,405 words with self-identified gender information from Reddit. Our evaluation of multiple biomedical NER models reveals evident biases. For instance, synthetic data suggests that female names are frequently misclassified as chemicals, especially when it comes to brand name mentions. Additionally, we observe performance disparities between female- and male-associated data in both datasets. Many systems fail to detect contraceptives such as birth control. Our findings emphasize the biases in chemical NER models, urging practitioners to account for these biases in downstream applications.