CAREER: Modeling User Touch and Motion Behaviors for Adaptive Interfaces in Mobile Devices

在这个项目中，PI将研究如何应用移动交互数据来自动提高移动用户界面的可用性和可访问性。这项研究将根据用户的行为确定他们的能力，从而在各种环境(包括情景致残情况)下，为不同的用户群体(如退伍军人)提供更容易访问的移动用户界面。特别是，PI将探索当用户实际处于移动状态时，基于用户行为和视觉注意力的数据驱动的自适应界面布局在移动计算中的挑战。这项工作将包括来自目前由国际和平协会建议的代表性不足群体的学生研究人员，并将由从事现实但不同活动的不同人群进行评估。这将允许PI发布软件和研究结果，使人们能够设计适当适应目标社区成员能力的界面。这项研究直接与一项教育计划挂钩，该计划旨在为讲授式的用户界面课程开发一个学生响应工具，该工具允许学生在课堂上创建线框界面、设计字体，并在触摸屏和移动设备上绘制可视化效果，这些可视化效果可以显示在房间屏幕上，供讨论和同行反馈。该工具将在不同学生群体的用户界面课程中反复改进，并部署在PI机构计算机科学以外的课程中，以及其他大学的用户界面课程中。移动设备上的用户界面不是万能的，也不应该是万能的。用户的能力可能不同，或者环境的情景背景可能会带来新的挑战。就其本质而言，移动设备可以在许多不同的环境中以不同的姿势使用。例如，用户可能会双手拿着平板电脑，屏幕处于横向，在床上阅读，偶尔会滑动到不同的页面；其他时候，他们可能会推着婴儿车，同时用一只手握住手机，以浏览地图应用程序。因为制造商知道这一点，智能手机和平板电脑不同于台式电脑，可以接受触摸输入，并感知运动和方向，这些互动的数据可以被网站和应用程序捕获，以识别特定的用户能力和背景。随着时间的推移，按比例收集的用户交互数据将实现界面的个性化，比如通过重塑触摸目标来补偿用户通常点击目标右侧的习惯，通过将重要按钮重新定位到屏幕上更容易访问的位置，或者通过注意用户经常应用的缩放级别来确定理想的文本大小。因此，这项工作将包括三个研究目标。第一个目标是研究如何从移动设备被动地捕获触摸和动作数据，计算代表用户在执行触摸交互时的习惯和错误的度量，并根据动作和触摸行为确定用户的环境背景。第二个目标是结合方向和触摸传感器来训练眼睛跟踪模型，使用前置摄像头来检测用户的注意力。第三个目标是改进现有界面的可用性和可访问性，例如，通过调整目标的可点击区域、文本大小和界面布局。

In this project the PI will investigate how to apply mobile interaction data to automatically improve the usability and accessibility of mobile user interfaces. The research will identify user abilities based on their behaviors, leading to mobile user interfaces that are more accessible to diverse user communities (e.g., veterans), in a variety of environments (including cases of situation-induced impairments). In particular, the PI will explore the challenges of data-driven adaptive interface layouts based on user behavior and visual attention in mobile computing when the user is actually mobile. The work will involve student researchers from under-represented groups currently advised by the PI, and will be evaluated by different populations engaged in realistic but varied activities. This will allow the PI to release software and research findings that enable people to design interfaces that properly adapt to the abilities of members of the target communities. The research ties directly into an educational plan to develop a student response tool for lecture-style user interface courses, that allows students to create wireframe interfaces, to design typefaces, and to draw visualizations during class on touchscreen and mobile devices, which can be displayed on the room screen for discussion and peer feedback. The tool will be iteratively refined in a user interface course with a diverse student population, and deployed in a course at the PI's institution outside of computer science as well as in user interface courses at other universities. User interfaces on mobile devices are not one-size-fits-all, nor should they be. Users' abilities may differ, or the situational context of the environment can introduce new challenges. By their very nature, mobile devices are used in many different environments and with different postures. For example, users may hold their tablet in both hands with the screen in landscape orientation to read in bed, swiping to different pages occasionally; at other times, they may be pushing a stroller while gripping their phone with one hand to navigate a map application. Because manufacturers know this, smartphones and tablets, unlike desktop computers, can accept touch input and sense both motion and orientation, and data from these interactions can be captured by websites and apps to identify specific user abilities and context. Over time, user interaction data collected at scale will enable personalization of the interface, say by reshaping touch targets to compensate for a user's habit of typically tapping to the right of a target, by relocating important buttons to more accessible locations on the screen, or by determining ideal text size by noting the zoom level a user often applies. Thus, the work will comprise three research objectives. The first objective is to investigate how to passively capture touch and motion data from mobile devices, to compute metrics representing user habits and mistakes as they perform touch interactions, and to determine the environmental context of the user from motion and touch behaviors. The second objective is to incorporate orientation and touch sensors to train an eye tracking model using the front-facing camera to detect the user's attention. The third objective, informed by findings from the first two, is to improve the usability and accessibility of existing interfaces, e.g., by adjusting the hittable area of targets, the text size, and interface layout.