EAGER: Scalable Crowdsourced Reinforcement of Robot Behavior

该项目将为系统开发基础，允许大量志愿者贡献者帮助训练具有不同运动能力的机器人正确响应命令，回答有关如何设计系统以激励人们做出贡献以及人群如何帮助训练机器人的研究问题。 这是一个重要的机器人问题，因为训练这些机器人需要大型的标记数据集，其中机器人正在移动，并且它们的运动被适当地标记;收集志愿者贡献（“众包”）是生成这些数据集的潜在成本和时间有效的方法。 为了测试这一点，研究团队将扩展现有的平台，允许人们围绕视频流进行交谈和协作，以吸引和留住有兴趣帮助训练机器人的参与者。 他们将向志愿者展示随着时间的推移越来越复杂的模拟机器人，要求志愿者提出适当的命令，这些命令可能与特定机器人的结构很好地配合，并向机器人反馈他们执行命令的情况。 虽然一开始机器人只是尝试不同的可能动作，但志愿者的反馈将帮助机器人了解哪些动作有助于它们执行命令，同时为志愿者提供机器人研究的直接经验，这可能会增加他们对机器人的兴趣和接受未来。 为了鼓励长期参与，该团队还将在流中添加一些界面功能，让志愿者能够更多地控制所展示的机器人以及有关其培训工作有效性的信息。 该研究将深入了解机器人学习，产生可用于其他机器人研究的大型数据集，并验证鼓励人们参与众包系统的策略（例如逐渐增加任务的复杂性和展示进展）。 这也是一个重要的一般性问题，因为众包在许多与社会相关的领域都很有用，包括构建开源软件、构建信息资源和支持公民科学。 最后，该团队将为语言学和发展心理学等领域的研究人员提供支持软件和数据，这些领域也对语言习得感兴趣。为此，该团队将利用基于实时视频流平台的现有原型系统。 该系统呈现一个机器人，然后循环通过命令选择期，其中人们键入聊天窗口的内容被解释为潜在的命令，以及命令响应期，其中机器人执行一些动作，聊天输入被解释为机器人学习的积极或消极强化。 最初的试验产生了很多兴趣和有用的学习数据，但贡献迅速下降，因为机器人没有实时学习，也没有改变。 为了改善学习，该团队将整合机器人控制算法，利用人群输入在线学习适当的每命令行为，以及学习量的视觉指标，通过显示志愿者贡献的价值来激励参与。 为了提高多样性，该团队将慢慢发展机器人的形态，增加额外的附件和传感器，以支持探索和学习新命令，同时帮助保持志愿者的兴趣。 除了评估数据本身在支持机器人学习命令方面的有效性外，该团队还将测量贡献的水平，频率和适当性，以评估响应性和机器人复杂性的变化是否实际上增加了人们的参与意愿，以及关于机器人的明显能力如何影响人们选择与他们互动的方式的假设。 长期目标是围绕机器人教学建立一个大型的、持续的社区，帮助进化生物学家、社会科学家、认知科学家、神经科学家、语言学家和机器人学家提出并回答关于人类和机器人共同进化的新假设。

This project will develop the groundwork for systems that allow large numbers of volunteer contributors to help train robots with different movement capabilities to respond correctly to commands, answering research questions both about how to design the systems to motivate people to contribute and about how crowds can help train robots. This is an important robotics question because training these robots requires large, labeled datasets in which the robots are moving and their motions are appropriately labeled; collecting volunteer contributions ("crowdsourcing") is a potentially cost- and time-effective way to generate these kinds of datasets. To test this, the research team will extend an existing platform that allows people to talk and collaborate around video streams in order to attract and retain participants interested in helping train robots. They will present volunteers watching their stream with simulated robots that are increasingly complex over time, asking volunteers both to suggest appropriate commands that might work well with a particular robot's structure and to give the robots feedback about how well they carry out the commands. Although at first the robots will simply try different possible motions, volunteers' feedback will help the robots learn which motions help them carry out commands, while providing the volunteers direct experience with robotics research that might increase their interest in and acceptance of robots in the future. To encourage long-term participation, the team will also add a number of interface features to the stream that allow volunteers more control over the robots being displayed as well as information about the effectiveness of their training efforts. The research will result in insights into robot learning, the production of large datasets that could be used in other robotics research, and the validation of strategies (such as gradually increasing the complexity of tasks and the demonstration of progress) for encouraging people to participate in crowdsourcing systems. This, too, is an important general question, as crowdsourcing is useful in many socially relevant domains, including building open source software, constructing information resources, and supporting citizen science. Finally, the team will make the supporting software and data available to researchers in domains such as linguistics and developmental psychology that are also interested in language acquisition. To do this, the team will leverage an existing prototype system that is based on a live video streaming platform. This system presents a robot, then cycles through a command selection period where the content people type into the chat window is interpreted as potential commands, and a command response period where the robot performs some action and chat input is interpreted as positive or negative reinforcement for robot learning. An initial trial generated much interest and useful data for learning, but contributions rapidly dropped because the robots did not learn in real-time and did not change. To improve learning, the team will integrate robot control algorithms that leverage crowd input to learn appropriate per-command behaviors online, as well as visual indicators of the amount of learning that might incentivize participation through showing the value of volunteers' contributions. To improve variety, the team will slowly evolve the robot's morphology, adding additional appendages and sensors that support the exploration and learning of new commands while helping maintain volunteers' interest. In addition to evaluating the effectiveness of the data itself in supporting robot learning of commands, the team will measure the levels, frequency, and appropriateness of contributions to evaluate whether changes in responsiveness and robot complexity in fact increase people's willingness to participate, as well as hypotheses about how the apparent capabilities of robots affect the ways people choose to interact with them. The long-term goal is to build a large, sustained community around robot teaching that can help evolutionary biologists, social scientists, cognitive scientists, neuroscientists, linguists, and roboticists all pose and answer novel hypotheses around the co-evolution of humans and robots.