RecurBot
RecurBot
复制标题
递归机器人
DOI：
10.1145/3170427.3188532
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Intharah T
中科院分区：
文献类型：
--
作者：
Intharah T
从表面上看，在图形用户界面（GUI）设置中完成任务应该很容易。然而，在实践中，不同的操作看起来很相似，应用程序在操作系统孤岛中运行。我们在GUI动作识别和预测中的目标是帮助用户，至少在完成大量重复的繁琐任务方面。我们提出了一种方法，从一些用户执行的演示学习，然后预测，并最终执行任务中的剩余动作。例如，用户可以向学校家长电子表格中的前三个联系人发送定制的SMS消息;然后我们的系统循环该过程，迭代其余的家长。首先，我们的分析系统将演示分割成离散的循环，其中每次迭代通常包括有意和意外的变化。我们的技术创新方法是一个解决standingmotif-findingoptimization问题，但我们也发现在这些有意的变化视觉模式。第二个挑战是预测后续的GUI操作，推断模式以允许我们的系统预测和执行任务的其余部分。我们验证我们的方法在一个新的数据库的GUI任务，并表明，我们的系统通常（a）收集它需要从短的用户演示，和（B）自动完成任务在不同的GUI情况。
On the surface, task-completion should be easy in graphical user interface (GUI) settings. In practice however, different actions look alike and applications run in operating-system silos. Our aim within GUI action recognition and prediction is to help the user, at least in completing the tedious tasks that are largely repetitive. We propose a method that learns from a few user-performed demonstrations, and then predicts and finally performs the remaining actions in the task. For example, a user can send customized SMS messages to the first three contacts in a school's spreadsheet of parents; then our system loops the process, iterating through the remaining parents.First, our analysis system segments the demonstration into discrete loops, where each iteration usually included both intentional and accidental variations. Our technical innovation approach is a solution to the standingmotif-findingoptimization problem, but we also find visual patterns in those intentional variations. The second challenge is to predict subsequent GUI actions, extrapolating the patterns to allow our system to predict and perform the rest of a task. We validate our approach on a new database of GUI tasks, and show that our system usually (a) gleans what it needs from short user demonstrations, and (b) autocompletes tasks in diverse GUI situations.