RI: Small: Collaborative Research: Hidden Parameter Markov Decision Processes: Exploiting Structure in Families of Tasks

第1部分机器学习有可能使许多复杂的现实任务自动化。然而，学习算法通常需要从他们被要求解决的每一项特定任务中获得大量数据，这需要与世界反复互动，每一次都需要时间和努力。许多现实生活中的学习场景都涉及与相似但不完全相同的任务的重复交互。例如，免疫学家可能会遇到患有不同共病条件和潜在病毒库的艾滋病毒患者--每个人都有类似的疾病，但病情发展不同，需要个体化治疗；机器人可能必须操纵不同大小和重量的物体--每个人都需要类似但不相同的抓取策略。在这种情况下，将所有任务都视为相同的任务会导致表现不佳，但学习解决每一项任务就好像它们完全不同一样需要花费的时间太长了。该项目将开发智能代理，这些代理可以使用在解决先前任务时获得的知识来更快地学习类似但不完全相同的新任务。该项目的主要技术组成部分将在于严格定义相关任务的含义，并产生利用该定义实现快速学习的算法。为此，该项目将引入隐藏参数马尔可夫决策过程，该过程通过一个参数对一系列任务进行建模，该参数描述了整个家族的变化，但对学习者是隐藏的。该项目将通过学习任务变化的模型，然后寻求确定每个特定任务的参数值，来研究利用这种结构的方法。计划的工作将集中在医疗保健应用程序上，其中相关但不同的任务家族是常见的(即每个患者都有独特的特征)。然而，该项目的目标是产生适用于从机器人到系统设计等许多应用领域的基础学习算法。这项研究还将被整合到哈佛大学和布朗大学的PI教授的课程中，并在网上提供；PI将包括不同的人群，包括这些班级和他们的研究小组中的REU。2许多现实生活中的学习情景涉及与具有相似但不相同的动态的任务的重复互动。例如，免疫学家可能会遇到患有不同共病条件和潜在病毒库的艾滋病毒患者；机器人可能不得不操纵不同大小和重量的物体。这些案例描述了一系列相关的任务，每个任务都相似，但并不完全相同。智能代理应该能够将在先前经验中学到的知识转移到同一个家庭中快速解决新任务。然而，虽然已经开发了许多算法来传递知识，但缺乏任务相关性模型阻碍了我们正式理解这些算法的好处或它们所利用的结构的能力。计划中的工作将通过将任务嵌入到低维流形上来建模，该流形捕捉实例之间的相关变化。这个流形上的每个位置(代理没有观察到)描述了一个任务实例，形成了在任务族上下文中求解任务的足够统计信息。PIS的初步工作表明，在仅解决几个单独的任务实例后就可以学习这样的流形，并能够快速优化新任务实例的策略。在这些有希望的初步结果的基础上，PI计划：1)开发任务系列表征的方法，方法是确定一组任务是否可以通过单个流形建模或由几个集群组成；新任务是否属于现有的集群或流形；如果是，以及是否值得转移。2)通过采用机器学习的最新结果来处理大的状态和动作空间来进行规模推理。3)使用贝叶斯强化学习算法生成策略，并利用状态和策略表示之间的正式联系。除了合成域，这些方向的进展将通过PI与世界专家在这些疾病方面的临床合作，应用于HIV、脓毒症和抑郁症患者的治疗优化问题。

Part 1Machine learning has the potential to automate many complex, real-life tasks. However, learning algorithms typically require a substantial amount of data from each specific task they are asked to solve, requiring repeated interactions with the world, each of which take time and effort. Many real-life learning scenarios involve repeated interactions with tasks that are similar, but not identical. For example, an immunologist may encounter HIV patients with different comorbid conditions and latent viral reservoirs - each has a similar disease but a different progression, requiring individualized treatment; a robot may have to manipulate objects of different size and weight - each requiring similar but not identical grasping strategies. In such cases treating all of the tasks as the same results in poor performance, but learning to solve each as if they were completely different takes far too long. This project will develop intelligent agents that can use knowledge gained when solving prior tasks to much more rapidly learn new tasks that are similar but not quite the same.The principal technical component of this project will lie in rigorously defining what it means for tasks to be related and in producing algorithms for leveraging that definition to enable rapid learning. To do so, the project will introduce the Hidden-Parameter Markov Decision Process, which models a family of tasks through a parameter which describes variation through the family but is hidden from the learner. The project will investigate methods that exploit this structure by learning a model of task variation and then seeking to identify the parameter value for each specific task. The planned work will focus on healthcare applications, where families of related but distinct tasks are common (i.e. each patient will have unique characteristics). However, the project aims to produce foundational learning algorithms applicable to many application areas, ranging from robotics to systems design. This research will also be integrated into the courses taught by the PIs at Harvard and Brown and made available online; the PIs will include a diverse population, including REUs, both in these classes and in their research groups.Part 2Many real-life learning scenarios involve repeated interactions with tasks that have similar, but not identical, dynamics. For example, an immunologist may encounter HIV patients with different comorbid conditions and latent viral reservoirs; a robot may have to manipulate objects of different size and weight. These cases describe a family of related tasks, each of which is similar but not quite the same. An intelligent agent should be able to transfer knowledge learned during previous experiences to rapidly solve new tasks in the same family. However, while many algorithms have been developed to transfer knowledge, the lack of a model of task relatedness inhibits our ability to formally understand the benefits of such algorithms or the structure they exploit.The planned work will model such scenarios by embedding the tasks on a low dimensional manifold that captures relevant variation between instances. Each location on this manifold (unobserved by the agent) describes a task instance, forming a sufficient statistic for solving the task in the context of the task family. Preliminary work by the PIs has shown that it is possible to learn such a manifold after solving just a few individual task instances and enable the rapid optimization of policies for new task instances. Building on these promising initial results, the PIs plan to: 1) Develop methods for task family characterization, by determining whether a collection of tasks can be modeled via a single manifold or consists of several clusters; whether a new task belongs to an existing cluster or manifold; and if so, and whether or not transfer is worthwhile. 2) Scale inference by adapting recent results from machine learning to deal with large state and action spaces. 3) Generate policies using Bayesian reinforcement learning algorithms, and by exploiting formal links between state and policy representations.In addition to synthetic domains, progress on these directions will be applied to problems of treatment optimization for patients with HIV, sepsis, and depression via clinical collaborations that the PIs have with world-experts in these diseases.