Multimodal natural assistive interfaces to support reading, learning, and access to critical information

“传统的”用户界面正在不断地被移动、触摸和基于语音的交互范例所取代。这些帮助了我们，赋予了我们力量；然而，移动界面也使许多弱势用户群体边缘化。对于一些人，例如低文化水平的成年人或视力有限的用户，即使是简单的任务（例如阅读一本书、理解用户手册、理解健康小册子）也可能是一场斗争，尽管智能、便携式和个人界面可能提供帮助。******我的研究目标是为边缘化用户提供实际支持，以努力执行基本活动，如阅读和获取关键信息。为此，我建议研究和开发基于移动、协作和多模态接口（语音、自然语言和手势）的智能辅助技术。******在我之前的研究中，我广泛研究了使用不完善的自动语音识别（ASR）和自然语言处理（NLP）系统的人为因素，并为促进与信息丰富的媒体和计算设备的自然交互（如手势）做出了贡献。我已经表明，这种界面如果能很好地适应其用户的需要，就能向边缘用户提供急需的援助。反过来，支持这种多模式系统的底层技术可以在其设计和功能根据用户的需求和限制进行改进时得到改进。我计划继续遵循同样的方法，将交互设计与ASR/NLP研究相结合。******这个项目的目标是研究交互式语音、语言和基于手势的移动辅助界面的方法和设计，为难以完成基本活动（如阅读、学习和获取关键信息）的用户提供实际支持。我计划研究多模式和协作、半自动化和交互的注释和信息提取机制，这些机制将帮助用户从电子文档中获取必要的知识。这项研究涵盖了广泛的语音和语言媒体，将满足各种边缘化和服务不足的用户群体的需求，从试图理解关键健康信息的低识字率成年人，到努力掌握数字素养的成人学习者，再到试图适应当今以信息为中心的社会变化的老年人。

“Traditional” user interfaces are continuously being replaced by mobile, touch, and speech-based interaction paradigms. These have helped us and empowered us; yet mobile interfaces have also marginalized many vulnerable user groups. For some, such as low-literacy adults or users with limited eyesight, even simple tasks (e.g. reading a book, understanding a user manual, comprehending a health pamphlet) can be a struggle despite the potential for assistance afforded by smart, portable, and personal interfaces.******The goal of my research is to provide practical support to marginalized users struggling to perform essential activities such as reading and acquiring critical information. For this, I propose to investigate and develop intelligent assistive technologies based on mobile, collaborative, and multimodal interfaces (speech, natural language, and gestural).******During my previous research I have extensively studied the human factors of using imperfect Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) systems, and have contributed to facilitating natural interactions (such as gestures) with information-rich media and computing devices. I have shown that such interfaces, if well adapted to the needs of their users, can provide much-needed assistance to marginalized users. In turn, the underlying technology that enables such multimodal systems can improve when its design and functionality is informed by users' needs and limitations. I plan to continue following the same approach of combining interaction design and ASR/NLP research.******The goal of this program is to research methods and designs for interactive speech, language, and gesture-based mobile assistive interfaces that provide practical support to users struggling to perform essential activities such as reading, learning, and acquiring critical information. I plan to investigate multimodal and collaborative, semi-automated, and interactive annotation and information extraction mechanisms that will assist users in acquiring essential knowledge from electronic documents. This research, encompassing a wide range of speech and language-enabled media, will address the needs of a diverse set of marginalized and underserved user groups, from low-literacy adults trying to understand critical health information, to adult learners struggling with digital literacy, and to older adults trying to adapt to changes in today's information-centric society.