CAREER: Modeling User Touch and Motion Behaviors for Adaptive Interfaces in Mobile Devices

在本项目中，PI将研究如何应用移动的交互数据来自动提高移动的用户界面的可用性和可访问性。 这项研究将根据用户的行为来识别用户的能力，从而产生更容易被不同用户社区访问的移动的用户界面（例如，退伍军人），在各种环境（包括情况引起的损害的情况下）。 特别是，PI将探索数据驱动的自适应界面布局的挑战，基于用户行为和视觉注意力在移动的计算时，用户实际上是移动的。 这项工作将涉及目前由PI建议的代表性不足的群体的学生研究人员，并将由从事现实但不同活动的不同人群进行评估。 这将使PI能够发布软件和研究成果，使人们能够设计适当适应目标社区成员能力的界面。 这项研究直接与一项教育计划相联系，该计划旨在为讲座式用户界面课程开发一个学生响应工具，该工具允许学生创建线框界面，设计字体，并在课堂上在触摸屏和移动的设备上绘制可视化效果，这些设备可以显示在房间屏幕上进行讨论和同行反馈。 该工具将在具有不同学生群体的用户界面课程中反复完善，并部署在PI机构计算机科学以外的课程以及其他大学的用户界面课程中。移动的设备上的用户界面不是万能的，也不应该是万能的。 用户的能力可能会有所不同，或者环境的情境可能会带来新的挑战。 就其本质而言，移动的设备被用于许多不同的环境中并且具有不同的姿势。 例如，用户可以双手握住平板电脑，屏幕处于横向方向，以便在床上阅读，偶尔滑动到不同的页面;在其他时间，他们可能推着婴儿车，同时用一只手抓住他们的手机来导航地图应用程序。 因为制造商知道这一点，智能手机和平板电脑，不像台式电脑，可以接受触摸输入和感知运动和方向，这些交互的数据可以被网站和应用程序捕获，以识别特定的用户能力和上下文。 随着时间的推移，大规模收集的用户交互数据将实现界面的个性化，例如通过重塑触摸目标以补偿用户通常点击目标右侧的习惯，通过将重要按钮重新定位到屏幕上更容易访问的位置，或者通过注意用户经常应用的缩放级别来确定理想的文本大小。 因此，这项工作将包括三个研究目标。 第一个目标是研究如何被动地从移动的设备捕获触摸和运动数据，以计算表示用户习惯和错误的度量，因为他们执行触摸交互，并确定从运动和触摸行为的用户的环境上下文。 第二个目标是结合方向和触摸传感器来训练眼睛跟踪模型，使用前置摄像头来检测用户的注意力。 第三个目标是根据前两个目标的调查结果，改进现有接口的可用性和可访问性，例如，通过调整目标的可命中区域、文本大小和界面布局。

In this project the PI will investigate how to apply mobile interaction data to automatically improve the usability and accessibility of mobile user interfaces. The research will identify user abilities based on their behaviors, leading to mobile user interfaces that are more accessible to diverse user communities (e.g., veterans), in a variety of environments (including cases of situation-induced impairments). In particular, the PI will explore the challenges of data-driven adaptive interface layouts based on user behavior and visual attention in mobile computing when the user is actually mobile. The work will involve student researchers from under-represented groups currently advised by the PI, and will be evaluated by different populations engaged in realistic but varied activities. This will allow the PI to release software and research findings that enable people to design interfaces that properly adapt to the abilities of members of the target communities. The research ties directly into an educational plan to develop a student response tool for lecture-style user interface courses, that allows students to create wireframe interfaces, to design typefaces, and to draw visualizations during class on touchscreen and mobile devices, which can be displayed on the room screen for discussion and peer feedback. The tool will be iteratively refined in a user interface course with a diverse student population, and deployed in a course at the PI's institution outside of computer science as well as in user interface courses at other universities. User interfaces on mobile devices are not one-size-fits-all, nor should they be. Users' abilities may differ, or the situational context of the environment can introduce new challenges. By their very nature, mobile devices are used in many different environments and with different postures. For example, users may hold their tablet in both hands with the screen in landscape orientation to read in bed, swiping to different pages occasionally; at other times, they may be pushing a stroller while gripping their phone with one hand to navigate a map application. Because manufacturers know this, smartphones and tablets, unlike desktop computers, can accept touch input and sense both motion and orientation, and data from these interactions can be captured by websites and apps to identify specific user abilities and context. Over time, user interaction data collected at scale will enable personalization of the interface, say by reshaping touch targets to compensate for a user's habit of typically tapping to the right of a target, by relocating important buttons to more accessible locations on the screen, or by determining ideal text size by noting the zoom level a user often applies. Thus, the work will comprise three research objectives. The first objective is to investigate how to passively capture touch and motion data from mobile devices, to compute metrics representing user habits and mistakes as they perform touch interactions, and to determine the environmental context of the user from motion and touch behaviors. The second objective is to incorporate orientation and touch sensors to train an eye tracking model using the front-facing camera to detect the user's attention. The third objective, informed by findings from the first two, is to improve the usability and accessibility of existing interfaces, e.g., by adjusting the hittable area of targets, the text size, and interface layout.