Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour
Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour
复制标题
更大的章鱼是否仍然会放大报告偏差？
DOI：
10.48550/arxiv.2209.12786
复制
发表时间：
2022
影响因子：
2.3
通讯作者：
Nigel Collier
中科院分区：
文献类型：
--
作者：
Fangyu Liu;Julian Martin Eisenschlos;Jeremy R. Cole;Nigel Collier
在原始文本上训练的语言模型（LM）无法直接访问物理世界。Gordon and货车Durme（2013）指出，语言模块可能因此受到报告偏见的影响：文本很少报告共同的事实，而是关注情况的不寻常方面。如果LM只接受文本语料库的训练，并天真地记住局部共现统计数据，那么它们自然会对物理世界产生偏见。虽然先前的研究已经反复验证了较小规模的LM（例如，RoBERTa，GPT-2）放大了报告偏倚，但当模型按比例放大时，这种趋势是否会持续仍是未知数。我们从颜色的角度研究了PaLM和GPT-3等大型语言模型（LLM）中的报告偏差。具体来说，我们向LLM查询对象的典型颜色，这是一种简单的基于感知的物理常识。令人惊讶的是，我们发现LLM在确定物体的典型颜色方面明显优于较小的LM，并且更紧密地跟踪人类的判断，而不是过度拟合存储在文本中的表面图案。这表明，非常大的语言模型本身就能够克服某些类型的报告偏见，其特点是当地共同出现。
Language models (LMs) trained on raw texts have no direct access to the physical world. Gordon and Van Durme (2013) point out that LMs can thus suffer from reporting bias: texts rarely report on common facts, instead focusing on the unusual aspects of a situation. If LMs are only trained on text corpora and naively memorise local co-occurrence statistics, they thus naturally would learn a biased view of the physical world. While prior studies have repeatedly verified that LMs of smaller scales (e.g., RoBERTa, GPT-2) amplify reporting bias, it remains unknown whether such trends continue when models are scaled up. We investigate reporting bias from the perspective of colour in larger language models (LLMs) such as PaLM and GPT-3. Specifically, we query LLMs for the typical colour of objects, which is one simple type of perceptually grounded physical common sense. Surprisingly, we find that LLMs significantly outperform smaller LMs in determining an object’s typical colour and more closely track human judgments, instead of overfitting to surface patterns stored in texts. This suggests that very large models of language alone are able to overcome certain types of reporting bias that are characterized by local co-occurrences.