Developing inquisitive, model-based agents for reinforcement learning

自然的智能体，像动物一样，从一生的经验中学习。大多数人工学习系统不会。新生儿一出生就疯狂地学习：试图控制他们的肌肉抽搐，理解他们的视觉输入。这些知识在一生中不断地被重用和提炼。我们当前的人工智能（AI）系统非常适合于系统决策与这些决策的效用之间具有明确因果关系的问题。游向鲨鱼会造成生命损失。射击外星飞船会增加分数。然而，在决策结果明显延迟的问题中，学习这种映射就更加困难了。最具挑战性和基本上未解决的人工智能基准问题具有这种延迟后果。对于最先进的系统来说，在每款雅达利游戏上训练相当于30天的时间是很常见的，但在具有延迟结果的游戏中，它的表现仍然远低于人类。***处理延迟结果问题的一种方法是让AI构建自己对世界如何运作的理解，通常称为世界模型。模型对世界的规律进行编码。例如，一个模型可能会编码：(1)当我和一条鲨鱼站在一起时，我决定发射一枚鱼雷，鲨鱼就会消失；(2)如果我站在平台上，我决定跳下去，我就会落到地面上。如果获得这种形式的模型，AI就可以在心理上模拟未来的情况，这些情况可能是由于特定的行为方式而导致的，而无需与世界进行实际交互。就像一个人如果选择一条通往河边的新路径，他可以决定自己可能会在哪里结束一样。我们可以想象选择这条替代路径的结果，而不实际去做，并避免不必要的探索，除非我们认为这样做是有价值的。基于模型的心理模拟可以显著提高学习效率。剩下的问题是系统如何决定如何最好地利用心理模拟。人们经常决定尝试他们从未做过的事情。我们选择参加那些对身心都有挑战，但又不会超出我们能力范围的活动。人类的动机是新奇、好奇和求知，对我们已经知道的事情感到厌倦。将这一想法与模型相结合，可以让人工智能模拟不同的行为方式，选择那些能够减少不确定性和获取新知识的行为方式。有了模型，人工智能就可以产生自己的内部反馈，专注于心理模拟。本研究计划的目标有两个方面：(1)设计表征和学习世界模型的新方法；(2)整合可以指导心理模拟（规划）和探索不确定性和知识获取的机制。

Natural agents, like animals, learn from a life-time of experience. Most artificial learning systems do not. Newborns begin life with a frenzy of learning: attempting to master their muscle twitches and make sense of their visual inputs. This knowledge is continuously reused and refined throughout life. Our current Artificial Intelligence (AI) systems are well-suited to problems with a clear cause and effect relationship between the system's decisions and the utility of those decisions. Swimming into a shark will cause a loss of life. Shooting an alien ship will increase the score. However, in problems where the consequences of a decision are significantly delayed, it is more difficult to learn this mapping. The most challenging and largely unsolved AI benchmark problems feature such delayed consequences. It is common practice for state-of-the-art systems to train for the equivalent of 30 days on each Atari game, and still achieve well-below human performance in games that feature delayed consequences. *** One way to deal with the problem of delayed consequences is for the AI to construct its own understanding of how the world works, usually called a model of the world. A model encodes the regularities of the world. For example a model might encode: (1) when I am lined up with a shark and I decide to fire a torpedo, the shark will disappear, and (2) if I am standing on a platform and I decide to jump down, I will end up on the ground. Given access to a model of this form, an AI can mentally simulate future situations that would result from behaving in particular ways without actually interacting with the world. Just as a human can decide where they might end-up if the took a new path down to the river. We can imagine the outcome of taking this alternative path without physically doing it, and avoid unnecessary exploration unless we decide it is valuable to do so. Model-based mental simulation can dramatically improve the efficiency of learning. *** The remaining question is how does the system decide how to best make use of mental simulation. People often decide to try out things they have never done before. We choose to engage in activities that are mentally and physically challenging, but not beyond our abilities. Humans are motivated by novelty, curiosity and knowledge seeking, and bored by things we already know about. Combining this idea with a model could allow an AI to simulate different ways of behaving, preferring those ways of behaving that result in reduction of uncertainty and the acquisition of new knowledge. With a model, the AI can generate its own internal feedback to focus its mental simulations. The objective of this research program is two fold: (1) to design new approaches for representing and learning models of the world, and (2) to integrate mechanisms that can guide mental simulations (planning) and exploration toward uncertainty and knowledge acquisition.