Like a Baby: Visually Situated Neural Language Acquisition
Like a Baby: Visually Situated Neural Language Acquisition
复制标题
DOI：
10.18653/v1/p19-1506
复制
发表时间：
2018-05
期刊：
影响因子：
--
通讯作者：
Alexander Ororbia;A. Mali;M. Kelly;D. Reitter
中科院分区：
文献类型：
--
作者：
Alexander Ororbia;A. Mali;M. Kelly;D. Reitter
我们研究了视觉上下文在训练神经语言模型以执行下一个单词预测方面的好处。引入了一种多模态神经架构，即使在测试中没有视觉上下文可用时，该架构的表现也优于仅接受语言训练的同等神经架构，困惑度降低了2%。在语言建模框架中微调预训练的最先进的双向语言模型（BERT）的嵌入，可以获得3.5%的改进。在没有测试的情况下，使用视觉上下文进行训练的优势在不同的语言（英语，德语和西班牙语）和不同的模型（GRU，LSTM，Delta-RNN以及使用BERT嵌入的模型）中是鲁棒的。因此，语言模型在像婴儿一样学习时表现得更好，即在多模态环境中。这一发现与情境认知理论相一致：语言与其物理语境密不可分。
We examine the benefits of visual context in training neural language models to perform next-word prediction. A multi-modal neural architecture is introduced that outperform its equivalent trained on language alone with a 2% decrease in perplexity, even when no visual context is available at test. Fine-tuning the embeddings of a pre-trained state-of-the-art bidirectional language model (BERT) in the language modeling framework yields a 3.5% improvement. The advantage for training with visual context when testing without is robust across different languages (English, German and Spanish) and different models (GRU, LSTM, Delta-RNN, as well as those that use BERT embeddings). Thus, language models perform better when they learn like a baby, i.e, in a multi-modal environment. This finding is compatible with the theory of situated cognition: language is inseparable from its physical context.