S&AS: FND: COLLAB: Learning from Stories: Practical Value Alignment and Taskability for Autonomous Systems

在不久的将来，我们可能会看到越来越强大的自主系统在人类附近运行，并沉浸在社会中。随着这些系统变得越来越复杂，它们将越来越多地与人类互动。随着人类与智能体互动的增加，确保自主系统不会对人类造成无意伤害的义务也在增加。创建不会有意或无意地伤害人类的系统并非易事。这是因为在一个开放的世界中，有无限多的不良结果可以实现，因此不可能指示这些系统避免每一个结果。如果期望的行为不能被直接指定，那么它必须被学习。过去学习这类行为的方法主要集中在从人类的例子中学习，但这些方法不太可能规模化。本研究使用行为的自然语言解释作为训练自主代理安全操作的可扩展替代方案。自然主义描述包含了大量关于社会文化规范的信息，这使它们成为这种训练的丰富来源。使系统能够更好地理解和从这些描述中学习，将使人类操作员能够更自然地指定代理要完成的目标或任务。本研究探讨了通过自然语言描述期望行为来学习的概念。该技术使用自然语言解释中包含的程序知识来帮助训练自主代理。具体地说，这种方法学习效用函数，可以用来指导自主代理的行为与用于训练的描述一致。为了实现这一目标，研究人员将创建能够从自然发生的语料库中提取社会文化规范知识和程序知识的计算模型。然后，这些模型将用于创建既符合社会文化规范又在程序上合理的行为政策。为了进一步确保这些模型可以实际应用，研究人员将使他们的模型纳入“人在循环”，以提供关于这些学习行为政策的质量的在线反馈，即社会可接受性和适当性。还将研究保护措施，以保护学习行为策略免受对抗性或恶意训练示例的影响。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

In the near future we are likely to see increasingly-capable autonomous systems operating in proximity to humans and immersed in society. As these systems become more sophisticated, they will interact increasingly with humans. With this increased human-agent interaction comes an increased obligation to ensure that autonomous systems do not cause even unintentional harm to a human. Creating systems that cannot intentionally or unintentionally harm humans in not an easy task. This is because there are infinitely many undesirable outcomes that can be achieved in an open world, making it impossible to instruct these systems to avoid each one. If the desired behavior cannot be directly specified, then it must be learned. Past approaches to learn these types of behaviors have focused on learning from human examples, but these methods are unlikely to scale. This research uses natural language explanations of behavior as a scalable alternative for training autonomous agents for safe operation. Naturalistic descriptions contain vast amounts of information about sociocultural norms, which make them rich sources for such training. Enabling systems to better understand and learn from such descriptions will enable human operators to more naturally specify goals or tasks for the agent to complete.This research explores the concept of learning via natural language descriptions of desired behavior. This technique uses procedural knowledge contained in natural language explanations to help train autonomous agents. Concretely, this approach learns utility functions that can be used to guide autonomous agents towards behaviors that are aligned with the description used for training. To accomplish this, researchers will create computational models capable of extracting both knowledge about sociocultural norms as well as procedural knowledge from naturally occurring corpora. These models will then be used to create behavior policies that are both aligned with sociocultural norms and procedurally plausible. To further ensure that these models can be practically deployed, researchers will enable their models to incorporate a "human in the loop" to provide online feedback about the quality of these learned behavior policies in terms of their social acceptability and appropriateness. Safeguards will also be investigated to protect the learned behavior policies against the effects of adversarial or malicious training examples.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.