CAREER: Modeling User Touch and Motion Behaviors for Adaptive Interfaces in Mobile Devices

在这个项目中，PI将研究如何应用移动交互数据来自动提高移动用户界面的可用性和可访问性。这项研究将根据用户的行为来确定用户的能力，从而使移动用户界面在各种环境（包括情境导致的损伤）中更容易被不同的用户群体（例如，退伍军人）使用。特别是，PI将探索基于用户行为和视觉注意力的数据驱动的自适应界面布局的挑战，当用户实际上是移动的。这项工作将涉及来自目前由PI提供建议的代表性不足的群体的学生研究人员，并将由从事现实但不同活动的不同人群进行评估。这将允许PI发布软件和研究成果，使人们能够设计适合目标社区成员能力的界面。这项研究与一项教育计划直接相关，该计划旨在为授课式用户界面课程开发一种学生响应工具，允许学生在触摸屏和移动设备上创建线框界面，设计字体，并在课堂上绘制可视化图像，这些图像可以显示在房间屏幕上，供讨论和同伴反馈。该工具将在用户界面课程中迭代改进，并在PI所在机构的计算机科学以外的课程以及其他大学的用户界面课程中部署。移动设备上的用户界面不是万能的，也不应该是万能的。用户的能力可能不同，或者环境的情景上下文可能会引入新的挑战。就其本质而言，移动设备可以在许多不同的环境中以不同的姿势使用。例如，用户可能会双手拿着平板电脑，屏幕横着在床上阅读，偶尔滑动到不同的页面；其他时候，他们可能推着婴儿车，一手抓着手机浏览地图应用程序。因为制造商知道这一点，所以与台式电脑不同，智能手机和平板电脑可以接受触摸输入，并感知运动和方向，这些交互的数据可以被网站和应用程序捕获，以识别特定的用户能力和背景。随着时间的推移，大规模收集的用户交互数据将实现界面的个性化，例如通过重塑触摸目标来补偿用户通常在目标右侧点击的习惯，通过将重要按钮重新定位到屏幕上更容易访问的位置，或者通过注意用户经常使用的缩放级别来确定理想的文本大小。因此，这项工作将包括三个研究目标。第一个目标是研究如何被动地从移动设备捕获触摸和运动数据，计算代表用户习惯和错误的指标，因为他们执行触摸交互，并从运动和触摸行为确定用户的环境背景。第二个目标是结合方向和触摸传感器来训练眼动追踪模型，使用前置摄像头来检测用户的注意力。第三个目标，根据前两个的发现，是改善现有界面的可用性和可访问性，例如，通过调整目标的可点击区域、文本大小和界面布局。

In this project the PI will investigate how to apply mobile interaction data to automatically improve the usability and accessibility of mobile user interfaces. The research will identify user abilities based on their behaviors, leading to mobile user interfaces that are more accessible to diverse user communities (e.g., veterans), in a variety of environments (including cases of situation-induced impairments). In particular, the PI will explore the challenges of data-driven adaptive interface layouts based on user behavior and visual attention in mobile computing when the user is actually mobile. The work will involve student researchers from under-represented groups currently advised by the PI, and will be evaluated by different populations engaged in realistic but varied activities. This will allow the PI to release software and research findings that enable people to design interfaces that properly adapt to the abilities of members of the target communities. The research ties directly into an educational plan to develop a student response tool for lecture-style user interface courses, that allows students to create wireframe interfaces, to design typefaces, and to draw visualizations during class on touchscreen and mobile devices, which can be displayed on the room screen for discussion and peer feedback. The tool will be iteratively refined in a user interface course with a diverse student population, and deployed in a course at the PI's institution outside of computer science as well as in user interface courses at other universities. User interfaces on mobile devices are not one-size-fits-all, nor should they be. Users' abilities may differ, or the situational context of the environment can introduce new challenges. By their very nature, mobile devices are used in many different environments and with different postures. For example, users may hold their tablet in both hands with the screen in landscape orientation to read in bed, swiping to different pages occasionally; at other times, they may be pushing a stroller while gripping their phone with one hand to navigate a map application. Because manufacturers know this, smartphones and tablets, unlike desktop computers, can accept touch input and sense both motion and orientation, and data from these interactions can be captured by websites and apps to identify specific user abilities and context. Over time, user interaction data collected at scale will enable personalization of the interface, say by reshaping touch targets to compensate for a user's habit of typically tapping to the right of a target, by relocating important buttons to more accessible locations on the screen, or by determining ideal text size by noting the zoom level a user often applies. Thus, the work will comprise three research objectives. The first objective is to investigate how to passively capture touch and motion data from mobile devices, to compute metrics representing user habits and mistakes as they perform touch interactions, and to determine the environmental context of the user from motion and touch behaviors. The second objective is to incorporate orientation and touch sensors to train an eye tracking model using the front-facing camera to detect the user's attention. The third objective, informed by findings from the first two, is to improve the usability and accessibility of existing interfaces, e.g., by adjusting the hittable area of targets, the text size, and interface layout.