CONVER-SE: Conversational Programming for Smart Environments

智能环境旨在智能地响应在其中访问、生活和工作的人的需求。例如，当客厅天黑时，灯就会亮起来，或者当博物馆参观者走近视频展览时，它可以用正确的语言播放。然而，我们缺乏直观的方法让没有技术背景的用户理解和重新配置这种环境的行为，而且公众对自动化环境有相当大的不信任。虽然有一些工具可以让用户在不具备编程知识的情况下查看和更改定义智能环境行为的规则，但除了技术爱好者之外，他们还没有看到广泛的接受。现有工具的一个缺点是它们会将注意力从相关环境中转移开，要求用户将现实世界中的对象转换为基于屏幕的抽象表示。新的编程工具允许用户利用他们对现实世界中对象的理解和引用，可以极大地增加对智能环境的信任和接受。这项研究将调查用户如何在现场理解和描述智能环境行为，并利用这些发现开发更直观的编程工具。例如，一个工具可以让人们简单地说，他们想要一盏灯在天黑时亮起来，然后指向它来识别它。语音接口现在被广泛应用于智能个人助理，但其功能在很大程度上仅限于发出即时命令或设置简单的提醒。在现实中，使用语音界面进行编程任务有很多挑战，而像LAMP这样的理想化交互在现实中一点也不简单。在许多情况下，用于为日常用户设计编程界面的研究是在研究实验室进行的，而不是在真实的家庭或工作场所环境中进行的，被邀请参加设计和评估研究的人通常是大学生或工作人员，或者对技术有现有兴趣或背景的人。这些界面一旦从设计、测试并提供给日常用户的一小部分玩具使用场景中剥离出来，就会出现故障。这项研究调查了使用语音编程的挑战，并评估了缓解这些挑战的方法，包括对话提示、使用手势和接近数据来避免模棱两可，以及提供可以定制的默认行为。在这个项目中，我们主要关注智能家居场景，我们将在真实的家庭环境中进行研究。语音界面在这些场景中越来越多地被使用，但不支持通过语音来查询、调试和交互行为。我们将招募没有编程背景的参与者，包括年长和残疾的用户，他们经常被强调为可以从智能家居技术中受益的人，但很少被纳入这类研究。我们将在人们的家中进行采访，了解他们如何自然地描述智能环境的规则，并考虑到语音、手势和位置。我们将寻找他们所描述的规则中的任何错误或不清楚的元素，并调查研究人员的提示在多大程度上可以帮助他们能够清楚地表达规则。我们还将探索参与者可以在多大程度上定制呈现给他们的默认行为。这些数据将被用来让我们创建一个对话界面，利用人类提示的方法，并在现实世界中对其进行测试。系统的一些元素将由人类研究人员控制，但系统将模拟与智能对话界面交互的体验。这将使我们能够确定在开发功能齐全的对话式编程工具方面可以追求的富有成效的领域，这些工具在博物馆、教育、农业和机器人技术中也可能有用。

Smart environments are designed to react intelligently to the needs of those who visit, live and work in them. For example, the lights can come on when it gets dark in a living room or a video exhibit can play in the correct language when a museum visitor approaches it. However, we lack intuitive ways for users without technical backgrounds to understand and reconfigure the behaviours of such environments, and there is considerable public mistrust of automated environments. Whilst there are tools that let users view and change the rules defining smart environment behaviours without having programming knowledge, they have not seen wide uptake beyond technology enthusiasts. One drawback of existing tools is that they pull attention away from the environment in question, requiring users to translate from real world objects to abstract screen-based representations of them. New programming tools that allow users to harness their understandings of and references to objects in the real world could greatly increase trust and uptake of smart environments.This research will investigate how users understand and describe smart environment behaviours whilst in situ, and use the findings to develop more intuitive programming tools. For example, a tool could let someone simply say that they want a lamp to come on when it gets dark, and point at it to identify it. Speech interfaces are now widely used in intelligent personal assistants, but the functionality is largely limited to issuing immediate commands or setting simple reminders. In reality, there are many challenges with using speech interfaces for programming tasks, and idealised interactions such as the lamp example are not at all simple, in reality. In many cases, research used to design programming interfaces for everyday users is carried out in research labs rather than in the real home or workplace settings, and the people invited to take part in design and evaluation studies are often university students or staff, or people with an existing interest or background in technology. These interfaces often fall down once taken away from the small set of toy usage scenarios in which they have been designed and tested and given to everyday users. This research investigates the challenges with using speech for programming, and evaluates ways to mitigate these challenges, including conversational prompts, use of gesture and proximity data to avoid ambiguity, and providing default behaviours that can be customised. In this project, we focus primarily on smart home scenarios, and we will carry out our studies in real domestic settings. Speech interfaces are increasingly being used in these scenarios, but there is no support for querying, debugging and alternating the behaviours through speech.We will recruit participants with no programming background, including older and disabled users, who are often highlighted as people who could benefit from smart home technology, but rarely included in studies of this sort. We will carry out interviews in people's homes to understand how they naturally describe rules for smart environments, taking into account speech, gesture and location. We will look for any errors or unclear elements in the rules they describe, and investigate how far prompts from researchers can help them to be able to express the rules clearly. We will also explore how far participants can customise default behaviours presented to them. This data will be used to allow us to create a conversational interface that harnesses the approaches that worked with human prompts, and test it in real world settings. Some elements of the system will be controlled by a human researcher, but the system will simulate the experience of interacting with an intelligent conversational interface. This will allow us to identify fruitful areas to pursue in developing fully functional conversational programming tools, which may also be useful in museums, education, agriculture and robotics.