Developing inquisitive, model-based agents for reinforcement learning

自然因素，就像动物一样，从一生的经验中学习。大多数人工学习系统都不会。新生儿以疯狂的学习开始生活：试图控制他们的肌肉抽动，并理解他们的视觉输入。这些知识在一生中不断地被重复使用和提炼。我们目前的人工智能(AI)系统非常适合处理系统决策和这些决策的效用之间有明确因果关系的问题。游进鲨鱼会造成生命损失。射杀一艘外星飞船会增加分数。然而，在决策结果明显延迟的问题中，学习这种映射更加困难。最具挑战性且基本上尚未解决的人工智能基准问题都具有这种延迟后果。对于最先进的系统来说，通常的做法是在每一场雅达利游戏上进行相当于30天的训练，但在具有延迟结果的游戏中，仍然取得远低于人类的表现。处理延迟后果问题的一种方法是人工智能构建自己对世界如何运作的理解，通常被称为世界模型。模型对世界的规律进行了编码。例如，一个模型可能是这样编码的：(1)当我和一条鲨鱼排成一排，我决定发射一枚鱼雷时，鲨鱼就会消失，(2)如果我站在平台上，我决定跳下去，我最终会倒在地上。在获得这种形式的模型后，人工智能可以在精神上模拟未来的情况，这些情况将由特定的行为方式导致，而不是实际与世界互动。就像人类可以决定，如果他们走一条新的路到河里，他们可能会去哪里一样。我们可以想象在没有实际行动的情况下选择这条替代道路的结果，并避免不必要的探索，除非我们认为这样做是有价值的。基于模型的心理模拟可以极大地提高学习效率。剩下的问题是，系统如何决定如何最好地利用心理模拟。人们经常决定尝试他们以前从未做过的事情。我们选择从事对精神和身体都具有挑战性的活动，但不会超出我们的能力范围。人类受新奇、好奇心和求知的驱使，对我们已知的事物感到厌烦。将这一想法与模型相结合，可以让人工智能模拟不同的行为方式，更喜欢那些能够减少不确定性和获取新知识的行为方式。有了模型，人工智能可以生成自己的内部反馈，以集中其心理模拟。这个研究计划的目标有两个：(1)设计新的方法来表示和学习世界的模型，以及(2)整合能够引导心理模拟(规划)和探索不确定性和知识获取的机制。

Natural agents, like animals, learn from a life-time of experience. Most artificial learning systems do not. Newborns begin life with a frenzy of learning: attempting to master their muscle twitches and make sense of their visual inputs. This knowledge is continuously reused and refined throughout life. Our current Artificial Intelligence (AI) systems are well-suited to problems with a clear cause and effect relationship between the system's decisions and the utility of those decisions. Swimming into a shark will cause a loss of life. Shooting an alien ship will increase the score. However, in problems where the consequences of a decision are significantly delayed, it is more difficult to learn this mapping. The most challenging and largely unsolved AI benchmark problems feature such delayed consequences. It is common practice for state-of-the-art systems to train for the equivalent of 30 days on each Atari game, and still achieve well-below human performance in games that feature delayed consequences. One way to deal with the problem of delayed consequences is for the AI to construct its own understanding of how the world works, usually called a model of the world. A model encodes the regularities of the world. For example a model might encode: (1) when I am lined up with a shark and I decide to fire a torpedo, the shark will disappear, and (2) if I am standing on a platform and I decide to jump down, I will end up on the ground. Given access to a model of this form, an AI can mentally simulate future situations that would result from behaving in particular ways without actually interacting with the world. Just as a human can decide where they might end-up if the took a new path down to the river. We can imagine the outcome of taking this alternative path without physically doing it, and avoid unnecessary exploration unless we decide it is valuable to do so. Model-based mental simulation can dramatically improve the efficiency of learning. The remaining question is how does the system decide how to best make use of mental simulation. People often decide to try out things they have never done before. We choose to engage in activities that are mentally and physically challenging, but not beyond our abilities. Humans are motivated by novelty, curiosity and knowledge seeking, and bored by things we already know about. Combining this idea with a model could allow an AI to simulate different ways of behaving, preferring those ways of behaving that result in reduction of uncertainty and the acquisition of new knowledge. With a model, the AI can generate its own internal feedback to focus its mental simulations. The objective of this research program is two fold: (1) to design new approaches for representing and learning models of the world, and (2) to integrate mechanisms that can guide mental simulations (planning) and exploration toward uncertainty and knowledge acquisition.