Improving language models, inspired by the brain

点击翻译按钮获取中文摘要

Language models (LMs) have markedly improved over the past 10 years, due in large part to the deep learning neural networks models they now use. These learned models can be fed a text snippet and generate words to meaningfully finish the thought. Current models can generate text that is so human-like that it can be difficult to differentiate from text written by people. These models have written segments of articles for the New Yorker and can answer some types of reading comprehension questions with accuracy at or above the accuracy of the average person. In fact, these models are so accurate, they are sometimes able to generate novel text that is better than what a human could generate. Remarkably, one such model generated the previous sentence. However, as convincing as current LMs are, they remain imperfect and sub-optimal in certain scenarios. For example, LMs perform poorly when tasks require inference, generalization, or commonsense knowledge. They also require huge amounts of data to train. Contrast this to young children who require only a fraction of the data our models do, and who are master generalizers and inference-makers. Where is that disconnect? Why are children so masterful with language in ways that our current models simply cannot replicate? Through this proposed research plan, I will study patterns in language development and usage, and use them as inspiration for improving computational LMs. I will study language acquisition and the onset of semantic composition in infants, the prioritization of different kinds of information in children, and the meta-cognitive processes governing language usage in fluent adults. Each of these points in the lifespan bring into focus different aspects of human language learning and usage, and novel ways of understanding and improving computational language learning. The impact of this work is multi-faceted. Firstly, these findings will allow me to create more accurate but also more efficient language models. More efficient LMs will train faster, with less data. This will improve multiple downstream usages of language models like dialogue generation and sentiment classification. Because more efficient models of language also require fewer resources to train, my work will impact the carbon footprint of the current natural language processing community. Secondly, my proposed research will contribute to our understanding of how the brain acquires and develops language, feed back into the basic neuroscience of learning, and help inform literacy education research. The impacts of my research will be as interdisciplinary as the research program itself.