Multimodal natural assistive interfaces to support reading, learning, and access to critical information

“传统的”用户界面正不断地被移动的、触摸和基于语音的交互范例所取代。这些帮助了我们，赋予了我们力量;然而，移动的界面也使许多弱势用户群体边缘化。对于某些人来说，例如识字率低的成年人或视力有限的用户，即使是简单的任务（例如阅读书籍，理解用户手册，理解健康小册子）也可能是一场斗争，尽管智能，便携式和个人界面提供了潜在的帮助。 我的研究目标是为那些努力进行基本活动（如阅读和获取关键信息）的边缘化用户提供实际支持。为此，我建议研究和开发基于移动的、协作和多模式接口（语音、自然语言和手势）的智能辅助技术。 在我之前的研究中，我广泛研究了使用不完善的自动语音识别（ASR）和自然语言处理（NLP）系统的人为因素，并为促进与信息丰富的媒体和计算设备的自然交互（如手势）做出了贡献。我已经证明，如果这些界面能够很好地适应用户的需求，就可以为边缘化用户提供急需的帮助。反过来，当其设计和功能符合用户的需求和限制时，支持这种多模式系统的基础技术也可以得到改进。我计划继续遵循同样的方法，将交互设计和ASR/NLP研究结合起来。 该计划的目标是研究方法和基于交互式语音，语言和手势的移动的辅助界面的设计，为用户努力执行基本活动，如阅读，学习和获取关键信息提供实际支持。我计划研究多模式和协作，半自动化，交互式注释和信息提取机制，将有助于用户从电子文档中获取必要的知识。这项研究，涵盖了广泛的语音和语言媒体，将满足各种各样的边缘化和服务不足的用户群体的需求，从低识字率的成年人试图了解关键的健康信息，成人学习者与数字扫盲斗争，以及老年人试图适应当今以信息为中心的社会的变化。

“Traditional” user interfaces are continuously being replaced by mobile, touch, and speech-based interaction paradigms. These have helped us and empowered us; yet mobile interfaces have also marginalized many vulnerable user groups. For some, such as low-literacy adults or users with limited eyesight, even simple tasks (e.g. reading a book, understanding a user manual, comprehending a health pamphlet) can be a struggle despite the potential for assistance afforded by smart, portable, and personal interfaces. The goal of my research is to provide practical support to marginalized users struggling to perform essential activities such as reading and acquiring critical information. For this, I propose to investigate and develop intelligent assistive technologies based on mobile, collaborative, and multimodal interfaces (speech, natural language, and gestural). During my previous research I have extensively studied the human factors of using imperfect Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) systems, and have contributed to facilitating natural interactions (such as gestures) with information-rich media and computing devices. I have shown that such interfaces, if well adapted to the needs of their users, can provide much-needed assistance to marginalized users. In turn, the underlying technology that enables such multimodal systems can improve when its design and functionality is informed by users' needs and limitations. I plan to continue following the same approach of combining interaction design and ASR/NLP research. The goal of this program is to research methods and designs for interactive speech, language, and gesture-based mobile assistive interfaces that provide practical support to users struggling to perform essential activities such as reading, learning, and acquiring critical information. I plan to investigate multimodal and collaborative, semi-automated, and interactive annotation and information extraction mechanisms that will assist users in acquiring essential knowledge from electronic documents. This research, encompassing a wide range of speech and language-enabled media, will address the needs of a diverse set of marginalized and underserved user groups, from low-literacy adults trying to understand critical health information, to adult learners struggling with digital literacy, and to older adults trying to adapt to changes in today's information-centric society.