S&AS: FND: COLLAB: Learning from Stories: Practical Value Alignment and Taskability for Autonomous Systems

在不久的将来，我们可能会看到越来越多的自主系统在人类附近运行并融入社会。随着这些系统变得越来越复杂，它们将越来越多地与人类互动。随着人类与智能体互动的增加，确保自主系统不会对人类造成意外伤害的义务也随之增加。创建一个不会有意或无意地伤害人类的系统并不是一件容易的事情。这是因为在一个开放的世界中，有无限多的不希望的结果可以实现，这使得不可能指示这些系统来避免每一个。如果不能直接指定所需的行为，那么必须学习它。过去学习这类行为的方法主要集中在从人类的例子中学习，但这些方法不太可能扩展。这项研究使用自然语言解释的行为作为一个可扩展的替代训练自主代理的安全操作。自然主义的描述包含大量关于社会文化规范的信息，这使它们成为这种培训的丰富来源。使系统能够更好地理解和学习，从这样的描述将使人类操作员更自然地指定目标或任务的代理completen.This研究探讨了学习的概念，通过自然语言描述所需的行为。该技术使用包含在自然语言解释中的程序知识来帮助训练自主代理。具体地说，这种方法学习效用函数，可用于引导自主代理的行为与用于训练的描述一致。为了实现这一目标，研究人员将创建能够从自然发生的语料库中提取有关社会文化规范和程序知识的计算模型。然后，这些模型将被用来创建既符合社会文化规范又在程序上合理的行为政策。为了进一步确保这些模型能够实际部署，研究人员将使他们的模型能够纳入“人在回路中”，以提供关于这些学习行为政策在社会可接受性和适当性方面的质量的在线反馈。还将调查保障措施，以保护学习到的行为策略免受敌对或恶意训练示例的影响。该奖项由计算机信息科学工程局信息和智能系统部门&&以及刺激竞争研究的既定计划（EPSCoR）共同资助该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响进行评估来支持审查标准。

In the near future we are likely to see increasingly-capable autonomous systems operating in proximity to humans and immersed in society. As these systems become more sophisticated, they will interact increasingly with humans. With this increased human-agent interaction comes an increased obligation to ensure that autonomous systems do not cause even unintentional harm to a human. Creating systems that cannot intentionally or unintentionally harm humans in not an easy task. This is because there are infinitely many undesirable outcomes that can be achieved in an open world, making it impossible to instruct these systems to avoid each one. If the desired behavior cannot be directly specified, then it must be learned. Past approaches to learn these types of behaviors have focused on learning from human examples, but these methods are unlikely to scale. This research uses natural language explanations of behavior as a scalable alternative for training autonomous agents for safe operation. Naturalistic descriptions contain vast amounts of information about sociocultural norms, which make them rich sources for such training. Enabling systems to better understand and learn from such descriptions will enable human operators to more naturally specify goals or tasks for the agent to complete.This research explores the concept of learning via natural language descriptions of desired behavior. This technique uses procedural knowledge contained in natural language explanations to help train autonomous agents. Concretely, this approach learns utility functions that can be used to guide autonomous agents towards behaviors that are aligned with the description used for training. To accomplish this, researchers will create computational models capable of extracting both knowledge about sociocultural norms as well as procedural knowledge from naturally occurring corpora. These models will then be used to create behavior policies that are both aligned with sociocultural norms and procedurally plausible. To further ensure that these models can be practically deployed, researchers will enable their models to incorporate a "human in the loop" to provide online feedback about the quality of these learned behavior policies in terms of their social acceptability and appropriateness. Safeguards will also be investigated to protect the learned behavior policies against the effects of adversarial or malicious training examples.This award is jointly funded by the Division of Information and Intelligent Systems in the Directorate for Computer & Information Science & Engineering and the Established Program to Stimulate Competitive Research (EPSCoR) in the Office of Integrative Activities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.