RI: Small: Collaborative Research: Hidden Parameter Markov Decision Processes: Exploiting Structure in Families of Tasks

机器学习有可能使许多复杂的、现实生活中的任务自动化。然而，学习算法通常需要从他们被要求解决的每个特定任务中获得大量数据，需要与世界重复交互，每个交互都需要时间和精力。许多现实生活中的学习场景涉及与相似但不相同的任务的重复交互。例如，免疫学家可能会遇到具有不同合并症和潜伏病毒库的艾滋病毒患者——每个人都有类似的疾病，但进展不同，需要个体化治疗；机器人可能需要操纵不同大小和重量的物体，每种物体都需要相似但不完全相同的抓取策略。在这种情况下，将所有任务视为相同的结果会导致表现不佳，但学会将每个任务视为完全不同的任务来解决会花费太长时间。该项目将开发智能代理，它可以利用在解决先前任务时获得的知识，更快地学习相似但不完全相同的新任务。这个项目的主要技术组成部分将在于严格定义相关任务的含义，并生成算法，以利用该定义来实现快速学习。为此，该项目将引入隐参数马尔可夫决策过程，该过程通过一个参数对一系列任务建模，该参数描述了整个家族的变化，但对学习者是隐藏的。该项目将通过学习任务变化模型来研究利用这种结构的方法，然后寻求确定每个特定任务的参数值。计划中的工作将侧重于医疗保健应用程序，在这些应用程序中，相关但不同的任务的家庭是常见的（即每个患者都有独特的特征）。然而，该项目旨在产生适用于许多应用领域的基础学习算法，从机器人到系统设计。这项研究也将被整合到哈佛大学和布朗大学的私人顾问教授的课程中，并在网上提供；在这些班级和他们的研究小组中，pi将包括不同的人群，包括reu。许多现实生活中的学习场景涉及与具有相似但不相同动态的任务的重复交互。例如，免疫学家可能会遇到具有不同合并症和潜伏病毒库的HIV患者；机器人可能要操纵不同大小和重量的物体。这些案例描述了一系列相关的任务，每个任务都很相似，但又不完全相同。智能代理应该能够转移在以前的经验中学习到的知识，以快速解决同一家庭中的新任务。然而，虽然已经开发了许多算法来转移知识，但缺乏任务相关性模型抑制了我们正式理解此类算法的好处或它们所利用的结构的能力。计划中的工作将通过将任务嵌入到捕获实例之间相关变化的低维流形上来对这些场景进行建模。这个流形上的每个位置（代理无法观察到）描述了一个任务实例，形成了在任务族上下文中解决任务的足够统计量。pi的初步工作表明，在解决几个单独的任务实例后，可以学习这样一个流形，并能够为新任务实例快速优化策略。在这些有希望的初步结果的基础上，pi计划：1)通过确定任务集合是否可以通过单个流形或由几个集群组成来建模，从而开发任务族特征的方法；新任务是否属于现有的集群或歧管；如果是，以及是否值得转移。2)规模推理，通过适应机器学习的最新结果来处理大的状态和动作空间。3)使用贝叶斯强化学习算法生成策略，并利用状态和策略表示之间的正式链接。除了合成领域，这些方向的进展将通过pi与这些疾病的世界专家的临床合作，应用于艾滋病、败血症和抑郁症患者的治疗优化问题。

Part 1Machine learning has the potential to automate many complex, real-life tasks. However, learning algorithms typically require a substantial amount of data from each specific task they are asked to solve, requiring repeated interactions with the world, each of which take time and effort. Many real-life learning scenarios involve repeated interactions with tasks that are similar, but not identical. For example, an immunologist may encounter HIV patients with different comorbid conditions and latent viral reservoirs - each has a similar disease but a different progression, requiring individualized treatment; a robot may have to manipulate objects of different size and weight - each requiring similar but not identical grasping strategies. In such cases treating all of the tasks as the same results in poor performance, but learning to solve each as if they were completely different takes far too long. This project will develop intelligent agents that can use knowledge gained when solving prior tasks to much more rapidly learn new tasks that are similar but not quite the same.The principal technical component of this project will lie in rigorously defining what it means for tasks to be related and in producing algorithms for leveraging that definition to enable rapid learning. To do so, the project will introduce the Hidden-Parameter Markov Decision Process, which models a family of tasks through a parameter which describes variation through the family but is hidden from the learner. The project will investigate methods that exploit this structure by learning a model of task variation and then seeking to identify the parameter value for each specific task. The planned work will focus on healthcare applications, where families of related but distinct tasks are common (i.e. each patient will have unique characteristics). However, the project aims to produce foundational learning algorithms applicable to many application areas, ranging from robotics to systems design. This research will also be integrated into the courses taught by the PIs at Harvard and Brown and made available online; the PIs will include a diverse population, including REUs, both in these classes and in their research groups.Part 2Many real-life learning scenarios involve repeated interactions with tasks that have similar, but not identical, dynamics. For example, an immunologist may encounter HIV patients with different comorbid conditions and latent viral reservoirs; a robot may have to manipulate objects of different size and weight. These cases describe a family of related tasks, each of which is similar but not quite the same. An intelligent agent should be able to transfer knowledge learned during previous experiences to rapidly solve new tasks in the same family. However, while many algorithms have been developed to transfer knowledge, the lack of a model of task relatedness inhibits our ability to formally understand the benefits of such algorithms or the structure they exploit.The planned work will model such scenarios by embedding the tasks on a low dimensional manifold that captures relevant variation between instances. Each location on this manifold (unobserved by the agent) describes a task instance, forming a sufficient statistic for solving the task in the context of the task family. Preliminary work by the PIs has shown that it is possible to learn such a manifold after solving just a few individual task instances and enable the rapid optimization of policies for new task instances. Building on these promising initial results, the PIs plan to: 1) Develop methods for task family characterization, by determining whether a collection of tasks can be modeled via a single manifold or consists of several clusters; whether a new task belongs to an existing cluster or manifold; and if so, and whether or not transfer is worthwhile. 2) Scale inference by adapting recent results from machine learning to deal with large state and action spaces. 3) Generate policies using Bayesian reinforcement learning algorithms, and by exploiting formal links between state and policy representations.In addition to synthetic domains, progress on these directions will be applied to problems of treatment optimization for patients with HIV, sepsis, and depression via clinical collaborations that the PIs have with world-experts in these diseases.