S&AS: FND: COLLAB: Learning from Stories: Practical Value Alignment and Taskability for Autonomous Systems

在不久的将来，我们可能会看到能力越来越强的自主系统在靠近人类的地方运行，并沉浸在社会中。随着这些系统变得更加复杂，它们将越来越多地与人类互动。随着人与代理人互动的增加，确保自主系统不会对人类造成哪怕是无意的伤害的义务也随之增加。创建不会有意或无意地伤害人类的系统并不是一项容易的任务。这是因为在一个开放的世界里，有无限多的不受欢迎的结果可以实现，因此不可能指示这些系统避免每一种结果。如果无法直接指定所需的行为，则必须学习它。过去学习这些类型行为的方法都集中在从人类例子中学习，但这些方法不太可能规模化。这项研究使用自然语言对行为的解释作为一种可扩展的替代方案，以训练自主代理的安全操作。自然主义的描述包含了大量关于社会文化规范的信息，这使它们成为此类培训的丰富来源。使系统能够更好地理解和学习这样的描述，将使人类操作员能够更自然地指定目标或任务，以便代理完成。本研究探索了通过所需行为的自然语言描述进行学习的概念。该技术使用自然语言解释中包含的程序性知识来帮助训练自主代理。具体地说，这种方法学习了可用于引导自主代理走向与训练所用描述一致的行为的效用函数。为了做到这一点，研究人员将创建能够从自然产生的语料库中提取社会文化规范知识和程序性知识的计算模型。然后，这些模型将被用来创建既符合社会文化规范，又在程序上可信的行为政策。为了进一步确保这些模型能够实际应用，研究人员将使他们的模型能够纳入“环路中的人”，就社会可接受性和适当性而言，就这些学习到的行为政策的质量提供在线反馈。还将调查保障措施，以保护学习的行为政策免受对抗性或恶意培训实例的影响。该奖项反映了NSF的法定使命，并已通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In the near future we are likely to see increasingly-capable autonomous systems operating in proximity to humans and immersed in society. As these systems become more sophisticated, they will interact increasingly with humans. With this increased human-agent interaction comes an increased obligation to ensure that autonomous systems do not cause even unintentional harm to a human. Creating systems that cannot intentionally or unintentionally harm humans in not an easy task. This is because there are infinitely many undesirable outcomes that can be achieved in an open world, making it impossible to instruct these systems to avoid each one. If the desired behavior cannot be directly specified, then it must be learned. Past approaches to learn these types of behaviors have focused on learning from human examples, but these methods are unlikely to scale. This research uses natural language explanations of behavior as a scalable alternative for training autonomous agents for safe operation. Naturalistic descriptions contain vast amounts of information about sociocultural norms, which make them rich sources for such training. Enabling systems to better understand and learn from such descriptions will enable human operators to more naturally specify goals or tasks for the agent to complete.This research explores the concept of learning via natural language descriptions of desired behavior. This technique uses procedural knowledge contained in natural language explanations to help train autonomous agents. Concretely, this approach learns utility functions that can be used to guide autonomous agents towards behaviors that are aligned with the description used for training. To accomplish this, researchers will create computational models capable of extracting both knowledge about sociocultural norms as well as procedural knowledge from naturally occurring corpora. These models will then be used to create behavior policies that are both aligned with sociocultural norms and procedurally plausible. To further ensure that these models can be practically deployed, researchers will enable their models to incorporate a "human in the loop" to provide online feedback about the quality of these learned behavior policies in terms of their social acceptability and appropriateness. Safeguards will also be investigated to protect the learned behavior policies against the effects of adversarial or malicious training examples.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.