Developing inquisitive, model-based agents for reinforcement learning

自然的代理人，像动物一样，从一生的经验中学习。大多数人工学习系统都没有。新生儿开始的生活充满了疯狂的学习：试图掌握他们的肌肉抽搐和理解他们的视觉输入。这些知识在一生中不断重复使用和完善。我们目前的人工智能（AI）系统非常适合解决系统决策与这些决策的效用之间存在明确因果关系的问题。游泳时撞上鲨鱼会造成生命损失。射击一艘外星飞船将增加分数。然而，在决策的结果被显著延迟的问题中，学习这种映射更困难。最具挑战性和基本上未解决的AI基准问题具有这种延迟后果。对于最先进的系统来说，通常的做法是在每个Atari游戏上训练相当于30天，并且在具有延迟后果的游戏中仍然达到远低于人类的表现。 处理延迟后果问题的一种方法是人工智能构建自己对世界如何运作的理解，通常称为世界模型。一个模型编码了世界的模式。例如，一个模型可能会编码：（1）当我和一条鲨鱼排成一行，我决定发射鱼雷时，鲨鱼会消失，（2）如果我站在平台上，我决定跳下去，我最终会在地面上。如果能够访问这种形式的模型，人工智能可以在精神上模拟未来的情况，这些情况将由特定的行为方式产生，而无需实际与世界进行交互。就像一个人可以决定他们可能会结束，如果他们采取了一个新的路径下到河。我们可以想象采取这种替代路径而不实际这样做的结果，并避免不必要的探索，除非我们认为这样做是有价值的。基于模型的心理模拟可以显著提高学习效率。 剩下的问题是系统如何决定如何最好地利用心理模拟。人们经常决定尝试他们以前从未做过的事情。我们选择从事那些在精神上和身体上具有挑战性的活动，但不会超出我们的能力。人类的动机是新奇、好奇和求知，而对我们已经知道的事情感到厌倦。将这个想法与模型相结合，可以让人工智能模拟不同的行为方式，更喜欢那些导致减少不确定性和获取新知识的行为方式。有了模型，人工智能可以生成自己的内部反馈，以集中其心理模拟。该研究计划的目标有两个方面：（1）设计新的方法来表示和学习世界的模型，以及（2）整合可以引导心理模拟（规划）和探索不确定性和知识获取的机制。

Natural agents, like animals, learn from a life-time of experience. Most artificial learning systems do not. Newborns begin life with a frenzy of learning: attempting to master their muscle twitches and make sense of their visual inputs. This knowledge is continuously reused and refined throughout life. Our current Artificial Intelligence (AI) systems are well-suited to problems with a clear cause and effect relationship between the system's decisions and the utility of those decisions. Swimming into a shark will cause a loss of life. Shooting an alien ship will increase the score. However, in problems where the consequences of a decision are significantly delayed, it is more difficult to learn this mapping. The most challenging and largely unsolved AI benchmark problems feature such delayed consequences. It is common practice for state-of-the-art systems to train for the equivalent of 30 days on each Atari game, and still achieve well-below human performance in games that feature delayed consequences. One way to deal with the problem of delayed consequences is for the AI to construct its own understanding of how the world works, usually called a model of the world. A model encodes the regularities of the world. For example a model might encode: (1) when I am lined up with a shark and I decide to fire a torpedo, the shark will disappear, and (2) if I am standing on a platform and I decide to jump down, I will end up on the ground. Given access to a model of this form, an AI can mentally simulate future situations that would result from behaving in particular ways without actually interacting with the world. Just as a human can decide where they might end-up if the took a new path down to the river. We can imagine the outcome of taking this alternative path without physically doing it, and avoid unnecessary exploration unless we decide it is valuable to do so. Model-based mental simulation can dramatically improve the efficiency of learning. The remaining question is how does the system decide how to best make use of mental simulation. People often decide to try out things they have never done before. We choose to engage in activities that are mentally and physically challenging, but not beyond our abilities. Humans are motivated by novelty, curiosity and knowledge seeking, and bored by things we already know about. Combining this idea with a model could allow an AI to simulate different ways of behaving, preferring those ways of behaving that result in reduction of uncertainty and the acquisition of new knowledge. With a model, the AI can generate its own internal feedback to focus its mental simulations. The objective of this research program is two fold: (1) to design new approaches for representing and learning models of the world, and (2) to integrate mechanisms that can guide mental simulations (planning) and exploration toward uncertainty and knowledge acquisition.