Learning and Search in Decision Domains Featuring Large Action Sets and Uncertainty

人工智能（AI）研究已经走过了漫长的道路，创造了挑战人类在决策领域的霸权的系统，如国际象棋，危险，股票交易，以及最近的图像识别，Atari 2600街机游戏和亚洲棋盘游戏Go。相比之下，流行的视频游戏中的人工智能进展缓慢，这些游戏通常具有大的动作空间，实时约束，多个玩家和隐藏信息，在许多情况下，人类专家仍然可以轻松地超越最好的人工智能系统。人类在这些领域的优势可以部分归因于我们简化问题的能力，同时保持解决方案，在不同的抽象层次上搜索（例如，只有在高级解决方案概念似乎不起作用时才查看细节），从观察到的行动中推断意图，并迅速适应对手和合作伙伴。这些方法有助于创建上面列出的强大AI系统。例如，训练策略网络和使用蒙特卡洛搜索来确定良好的低级动作目前还不足以在以大动作空间和由具有微观效果的动作组成的长播放片段为特征的领域中实现人类专家水平的性能。我们建议研究如何更好地整合启发式搜索（它可以通过前瞻性来评估动作的优点）与机器学习相结合，以处理大型组合动作空间、不确定性和智能体合作。主要的长期研究目标是：1）使用深度神经网络从自我游戏中学习分层策略，2）理解启发式搜索与学习策略在前向模型不可用的领域中的作用，3）合作多智能体领域中的学习策略，以及4）用于合作和开发的数据高效代理建模。为了实现这些长期目标，我们从简单的任务开始，包括从人类训练数据中进行监督学习，研究中等规模行动空间领域的强化学习，并将人类合作策略整合到现有的基于搜索的AI系统中。在该提案的目标领域取得实质性进展将对技术和社会产生深远影响。在一个机器可以学习在多智能体环境中表现良好，并可以制定和执行有效的高级行动计划的世界中，我们可能离一般的类人智能只有一步之遥。

Artificial Intelligence (AI) research has come a long way creating systems that challenge human supremacy in decision domains such as Chess, Jeopardy, stock trading, and recently image recognition, Atari 2600 arcade games, and the Asian boardgame Go. By contrast, AI progress in popular video games which often feature large action spaces, real-time constraints, multiple players, and hidden information, has been slow, and in many cases human experts can still easily outperform the best AI systems.The human advantage in these domains can in part be attributed to our abilities to simplify problems while maintaining solutions, to search at different abstraction levels (e.g., looking into details only when high-level solution concepts do not seem to work), to infer intentions from observed actions, and to quickly adjust to opponents and partners. The methods that have been instrumental to creating strong AI systems listed above. For example, training policy networks and using Monte Carlo search to determine good low-level actions are currently not powerful enough to achieve human expert level performance in domains featuring large action spaces and long playing episodes consisting of actions with microscopic effects.To overcome these problems, we propose to investigate how to better integrate heuristic search (which can evaluate the merit of actions by looking ahead) with machine learning to deal with large combinatorial action spaces, uncertainty, and agent cooperation. The main long-term research objectives are: 1) learning hierarchical policies from self-play using deep neural networks, 2) understanding the role of heuristic search vs. learned policies in domains for which forward-models are not available, 3) learning strategies in cooperative multi-agent domains, and 4) data efficient agent modelling for cooperation and exploitation. We approach these long-term goals by starting with simpler tasks involving supervised learning from human training data, studying reinforcement learning in medium-sized action space domains, and integrating human cooperation strategies into existing search-based AI systems.Making substantial progress in the target domains of this proposal will have a profound impact on technology and society. In a world in which machines can learn to perform well in multi-agent settings and can formulate and execute effective high-level action plans, we may just be a step away from general human-like intelligence.