Multimodal natural assistive interfaces to support reading, learning, and access to critical information

点击翻译按钮获取中文摘要

“Traditional” user interfaces are continuously being replaced by mobile, touch, and speech-based interaction paradigms. These have helped us and empowered us; yet mobile interfaces have also marginalized many vulnerable user groups. For some, such as low-literacy adults or users with limited eyesight, even simple tasks (e.g. reading a book, understanding a user manual, comprehending a health pamphlet) can be a struggle despite the potential for assistance afforded by smart, portable, and personal interfaces.******The goal of my research is to provide practical support to marginalized users struggling to perform essential activities such as reading and acquiring critical information. For this, I propose to investigate and develop intelligent assistive technologies based on mobile, collaborative, and multimodal interfaces (speech, natural language, and gestural).******During my previous research I have extensively studied the human factors of using imperfect Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) systems, and have contributed to facilitating natural interactions (such as gestures) with information-rich media and computing devices. I have shown that such interfaces, if well adapted to the needs of their users, can provide much-needed assistance to marginalized users. In turn, the underlying technology that enables such multimodal systems can improve when its design and functionality is informed by users' needs and limitations. I plan to continue following the same approach of combining interaction design and ASR/NLP research.******The goal of this program is to research methods and designs for interactive speech, language, and gesture-based mobile assistive interfaces that provide practical support to users struggling to perform essential activities such as reading, learning, and acquiring critical information. I plan to investigate multimodal and collaborative, semi-automated, and interactive annotation and information extraction mechanisms that will assist users in acquiring essential knowledge from electronic documents. This research, encompassing a wide range of speech and language-enabled media, will address the needs of a diverse set of marginalized and underserved user groups, from low-literacy adults trying to understand critical health information, to adult learners struggling with digital literacy, and to older adults trying to adapt to changes in today's information-centric society.