Help, It Looks Confusing
Help, It Looks Confusing
复制标题
求助，看起来很混乱
DOI：
10.1145/3025171.3025176
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Intharah T
中科院分区：
文献类型：
--
作者：
Intharah T
非编程用户应该能够创建自己的定制脚本，为他们执行基于计算机的任务，只需向机器演示如何完成。为此，我们开发了一个通过演示学习的系统原型，称为HILC（帮助，它看起来令人困惑）。用户通过演示任务来训练HILC合成任务脚本，从而生成所需的屏幕截图和相应的鼠标键盘信号。演示结束后，用户回答后续问题。我们提出了一个用户在循环框架，学习生成在图形应用程序的可见元素上执行的动作脚本。虽然纯粹的演示编程仍然不现实，但我们使用定量和定性实验来表明非编程用户愿意并且有效地回答我们系统提出的后续查询。我们对事件和表象的模型出奇地简单，但却能有效地结合起来应对不同程度的监督。在我们的用户研究实验中，最好的基线——Sikuli幻灯片——在大多数测试中都表现不佳。我们提出的方法的原型成功地帮助用户完成了简单的线性任务、复杂的任务（监视、循环和混合）以及跨越多个可执行文件的任务。即使两个系统最终都能完成任务，我们的系统也能在更短的时间内被用户训练和完善。
Non-programming users should be able to create their own customized scripts to perform computer-based tasks for them, just by demonstrating to the machine how it's done. To that end, we develop a system prototype which learns-by-demonstration called HILC (Help, It Looks Confusing). Users train HILC to synthesize a task script by demonstrating the task, which produces the needed screenshots and their corresponding mouse-keyboard signals. After the demonstration, the user answers follow-up questions. We propose a user-in-the-loop framework that learns to generate scripts of actions performed on visible elements of graphical applications. While pure programming-by-demonstration is still unrealistic, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system. Our models of events and appearance are surprisingly simple, but are combined effectively to cope with varying amounts of supervision. The best available baseline, Sikuli Slides, struggled with the majority of the tests in our user study experiments. The prototype with our proposed approach successfully helped users accomplish simple linear tasks, complicated tasks (monitoring, looping, and mixed), and tasks that span across multiple executables. Even when both systems could ultimately perform a task, ours was trained and refined by the user in less time.