Multimodal natural assistive interfaces to support reading, learning, and access to critical information

“传统的”用户界面正在不断地被基于移动、触摸和语音的交互模式所取代。这些都帮助了我们并增强了我们的能力；然而，移动界面也使许多弱势用户群体边缘化。对于一些人来说，例如低识字率的成年人或视力受限的用户，即使是简单的任务(如阅读书籍、理解用户手册、阅读健康手册)也可能是一场斗争，尽管智能、便携和个人界面可能会提供帮助。 我的研究目标是为努力执行基本活动的边缘化用户提供实际支持，如阅读和获取关键信息。为此，我建议研究和开发基于移动、协作和多模式界面(语音、自然语言和手势)的智能辅助技术。 在我之前的研究中，我广泛研究了使用不完善的自动语音识别(ASR)和自然语言处理(NLP)系统的人为因素，并为促进与信息丰富的媒体和计算设备的自然交互(如手势)做出了贡献。我已经表明，如果这些界面能够很好地适应其用户的需要，就可以向边缘化用户提供急需的帮助。反过来，当用户的需求和限制为其设计和功能提供信息时，实现这种多模式系统的基础技术可以得到改进。我计划继续遵循将交互设计和ASR/NLP研究相结合的相同方法。 该项目的目标是研究基于语音、语言和手势的交互式移动辅助界面的方法和设计，为努力执行基本活动(如阅读、学习和获取关键信息)的用户提供实际支持。我计划研究多模式和协作、半自动和交互式注释和信息提取机制，以帮助用户从电子文档中获取基本知识。这项研究涵盖了广泛的语音和语言媒体，将满足各种边缘化和未得到充分服务的用户群体的需求，从试图了解关键健康信息的低识字率成年人，到努力学习数字素养的成人学习者，以及试图适应当今以信息为中心的社会变化的老年人。

“Traditional” user interfaces are continuously being replaced by mobile, touch, and speech-based interaction paradigms. These have helped us and empowered us; yet mobile interfaces have also marginalized many vulnerable user groups. For some, such as low-literacy adults or users with limited eyesight, even simple tasks (e.g. reading a book, understanding a user manual, comprehending a health pamphlet) can be a struggle despite the potential for assistance afforded by smart, portable, and personal interfaces. The goal of my research is to provide practical support to marginalized users struggling to perform essential activities such as reading and acquiring critical information. For this, I propose to investigate and develop intelligent assistive technologies based on mobile, collaborative, and multimodal interfaces (speech, natural language, and gestural). During my previous research I have extensively studied the human factors of using imperfect Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) systems, and have contributed to facilitating natural interactions (such as gestures) with information-rich media and computing devices. I have shown that such interfaces, if well adapted to the needs of their users, can provide much-needed assistance to marginalized users. In turn, the underlying technology that enables such multimodal systems can improve when its design and functionality is informed by users' needs and limitations. I plan to continue following the same approach of combining interaction design and ASR/NLP research. The goal of this program is to research methods and designs for interactive speech, language, and gesture-based mobile assistive interfaces that provide practical support to users struggling to perform essential activities such as reading, learning, and acquiring critical information. I plan to investigate multimodal and collaborative, semi-automated, and interactive annotation and information extraction mechanisms that will assist users in acquiring essential knowledge from electronic documents. This research, encompassing a wide range of speech and language-enabled media, will address the needs of a diverse set of marginalized and underserved user groups, from low-literacy adults trying to understand critical health information, to adult learners struggling with digital literacy, and to older adults trying to adapt to changes in today's information-centric society.