CAREER: Using Imperfect Predictions to Make Good Decisions

当人类和其他动物在世界上航行时，他们在遇到不熟悉的系统，空间和现象时表现出非凡的灵活性，学会预测他们将如何表现，并根据这些预测做出正确的决定。这种能力的关键是，人们不需要做出完全准确或完全详细的预测来做出好的决策。尽管由于我们的自然局限性，我们对未来的预测必然是有缺陷的，但它们仍然足够有用，可以做出合理的决定。相比之下，对于人工智能体来说，不完美的预测往往会导致决策的灾难性失败。许多现有的方法从根本上假设智能体最终会学会做出完美的预测和决策，这在足够丰富，复杂的环境中是不合理的。这项工作考虑的问题，开发人工代理，更清楚，更强大的自己的局限性。能够在真正复杂的环境中更强大和灵活地从经验中学习的代理有可能影响几乎任何随着时间的推移做出决策的应用，例如自主机器人/车辆，个人助理和医疗/法律的决策支持。此外，由于该项目将在一所只招收本科生的文理学院进行，本科生研究人员将在工作中发挥不可或缺的作用。PI还将建立在文科设置的优势，以加强整个计算机科学课程的关键学科特定研究和写作技能的指导。这些技能的明确发展不仅会提高学生的各种职业道路（包括基础研究）的准备，但也符合最佳实践，扩大在学科的参与。该项目研究基于模型的强化学习（MBRL），假设代理具有基本限制，阻止它学习完美的模型或产生最佳计划。中心假设是，在这种情况下，MBRL问题不能分解为单独的模型学习和规划问题，每一个对待对方作为一个理想化的黑盒子。相反，每个组件的优化过程必须了解其在整个体系结构中的作用及其合作伙伴的限制。这项工作的一个关键目标是获得新的模型质量的措施，更紧密地关系到控制性能的真正目标比标准措施的一步预测精度适应监督学习设置。另一个是研究模型学习目标/算法如何适应将使用该模型的特定计划者的限制。此外，控制算法将进行调查，可以有效地利用模型的非均匀质量之间的调解模型为基础的和无模型的知识。最终的目标是将这些原则整合到新的MBRL代理中，这些代理对模型类和/或规划器的限制更加强大，并且能够在过于复杂和高维的环境中取得成功，从而精确建模或解决。

As humans and other animals navigate the world they demonstrate remarkable flexibility in encountering unfamiliar systems, spaces and phenomena, learning to make predictions about how they will behave, and making good decisions based on those predictions. Crucial to this ability is the fact that one does not need to make perfectly accurate or fully detailed predictions to make good decisions. Though, due to our natural limitations, our predictions about the future are necessarily flawed, they are nevertheless sufficiently useful to make reasonable decisions. For artificial agents, in contrast, imperfect predictions often lead to catastrophic failures in decision making. Many existing approaches fundamentally assume that the agent will eventually learn to make perfect predictions and make perfect decisions, which is unreasonable in sufficiently rich, complex environments. This work considers the problem of developing artificial agents that are more aware of and more robust to their own limitations. Agents that can more robustly and flexibly learn from experience in truly complex environments have the potential to impact nearly any application in which decisions are made over time, for instance autonomous robots/vehicles, personal assistants, and medical/legal decision support. Furthermore, as the project will be undertaken at an undergraduate-only liberal arts college, undergraduate researchers will play an integral role in the work. The PI will also build on the strength of the liberal arts setting to enhance instruction of key discipline-specific research and writing skills throughout the Computer Science curriculum. Explicit development of these skills will not only improve students' preparation for a wide variety of career paths (including basic research) but is also aligned with best practices for broadening participation in the discipline. This project studies model-based reinforcement learning (MBRL) under the assumption that the agent has fundamental limitations that prevent it from learning a perfect model or from producing optimal plans. The central hypothesis is that in this context the MBRL problem cannot be decomposed into separate model-learning and planning problems, each treating the other as an idealized black box. Rather the optimization process for each component must be aware of its role in the overall architecture and of the limitations of its partner. One key aim of the work is to derive novel measures of model quality that are more tightly related to the true objective of control performance than standard measures of one-step prediction accuracy adapted from supervised learning settings. Another is to investigate how model learning objectives/algorithms can be adapted to account for the limitations of the specific planner that will use the model. Further, control algorithms will be investigated that can make effective use of models of non-homogeneous quality by mediating between model-based and model-free knowledge. The ultimate goal is to integrate these principles into novel MBRL agents that are significantly more robust to limitations in the model class and/or planner and are able to succeed in environments that are too complex and high-dimensional to be modeled or solved exactly.