Out of One, Many: Using Language Models to Simulate Human Samples
Out of One, Many: Using Language Models to Simulate Human Samples
复制标题
DOI：
10.1017/pan.2023.2
复制
发表时间：
2023-02-21
影响因子：
5.4
通讯作者：
Wingate, David
中科院分区：
文献类型：
--
作者：
Argyle, Lisa P. P.;Busby, Ethan C. C.;Wingate, David
我们提出并探索了将语言模型作为社会科学研究中特定人类亚群的有效代理进行研究的可能性。人工智能工具的实际和研究应用有时会受到有问题的偏见（例如种族主义或性别歧视）的限制，这些偏见通常被视为模型的统一属性。我们证明，这样的工具（GPT-3 语言模型）中的“算法偏差”既是细粒度的，又与人口统计相关，这意味着适当的调节将使其能够准确地模拟来自各种人类亚组的响应分布。我们将此属性称为算法保真度，并探讨其在 GPT-3 中的范围。我们根据在美国进行的多项大型调查中真实人类参与者的数千个社会人口背景故事来调整模型，从而创建“硅样本”。然后，我们比较硅样本和人体样本，以证明 GPT-3 中包含的信息远远超出了表面相似性。它是微妙的、多方面的，反映了表征人类态度的思想、态度和社会文化背景之间复杂的相互作用。我们认为，具有足够算法保真度的语言模型构成了一种新颖而强大的工具，可以促进跨学科对人类和社会的理解。
We propose and explore the possibility that language models can be studied as effective proxies for specific human subpopulations in social science research. Practical and research applications of artificial intelligence tools have sometimes been limited by problematic biases (such as racism or sexism), which are often treated as uniform properties of the models. We show that the "algorithmic bias" within one such tool-the GPT-3 language model-is instead both fine-grained and demographically correlated, meaning that proper conditioning will cause it to accurately emulate response distributions from a wide variety of human subgroups. We term this property algorithmic fidelity and explore its extent in GPT-3. We create "silicon samples" by conditioning the model on thousands of sociodemographic backstories from real human participants in multiple large surveys conducted in the United States. We then compare the silicon and human samples to demonstrate that the information contained in GPT-3 goes far beyond surface similarity. It is nuanced, multifaceted, and reflects the complex interplay between ideas, attitudes, and sociocultural context that characterize human attitudes. We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines.