CAREER: Technology Assisted Conversations

面对面的交谈是人们相互交流的重要方式，但不幸的是，有数百万人患有阻碍正常交谈的疾病。该项目将探索新的实时沟通解决方案，为那些有语言障碍的人，包括那些有身体或认知障碍的人，例如，利用从一个人的对话伙伴那里获得的隐性和显性语境输入。目标是开发技术，改进现有的辅助和替代通信（AAC）设备，帮助人们更快、更流畅地说话。该项目将为会话交互系统的研究扩展资源，可交付成果包括概率文本输入工具包、AAC用户界面和增强现实会话助手。项目成果将包括扩展到新使用场景的灵活、健壮和数据驱动的方法。为了增强其更广泛的影响，该项目将通过外展活动和该工作将创建的在线社区向公众宣传AAC。项目负责人将召集本科生团队开发项目软件，他还将主持一个关于短信背后技术的暑期青年项目，为女性、残疾学生和弱势群体的学生提供奖学金。获得资助的第一年研究机会将进一步帮助留住计算机专业的本科生，尤其是女性。这个项目将探索对话互动系统的设计空间，通过研究两种系统，一种是为使用AAC设备的不会说话的人改善沟通，另一种是为面临其他对话相关挑战的会说话的人改善沟通。上下文敏感的预测算法使用：1)轮到对话伙伴时的语音识别；2)通过说话人身份识别来确定同伴的身份；3)对话状态信息；4)考虑合作伙伴在移动设备上提出的建议。用户研究将调查基于合作伙伴的预测的有效性和用户接受程度。将创建新的方法来评估上下文敏感的AAC接口。将比较训练AAC语言模型对来自现有语料库、模拟AAC用户和实际AAC用户的数据的影响。这项研究将扩展我们关于如何在增强现实中利用会话上下文的知识，它将策划一个由AAC用户贡献的公共测试集。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Face-to-face conversation is an important way in which people communicate with each other, but unfortunately there are millions who suffer from disorders that impede normal conversation. This project will explore new real-time communication solutions for people who face speaking challenges, including those with physical or cognitive disabilities, for example by exploiting implicit and explicit contextual input obtained from a person's conversation partner. The goal is to develop technology that improves upon the Augmentative and Alternative Communication (AAC) devices currently available to help people speak faster and more fluidly. The project will expand the resources for research into conversational interactive systems, the deliverables to include a probabilistic text entry toolkit, AAC user interfaces, and an augmented reality conversation assistant. Project outcomes will include flexible, robust, and data-driven methods that extend to new use scenarios. To enhance its broader impact, the project will educate the public about AAC via outreach events and by the online community the work will create. The PI will assemble teams of undergraduates to develop the project's software, and he will host a summer youth program on the technology behind text messaging, offering scholarships for women, students with disabilities, and students from underrepresented groups. Funded first-year research opportunities will further help retain undergraduates, particularly women, in computing.This project will explore the design space of conversational interactive systems, by investigating both systems that improve communication for non-speaking individuals who use AAC devices and systems that enhance communication for speaking individuals who face other conversation-related challenges. Context-sensitive prediction algorithms that use: 1) speech recognition on the conversation partner's turns; 2) the identity of the partner as determined by speaker identification; 3) dialogue state information; and 4) suggestions made by a partner on a mobile device will be considered. User studies will investigate the effectiveness and user acceptance of partner-based predictions. New methodologies will be created for evaluating context-sensitive AAC interfaces. The impact of training AAC language models on data from existing corpora, from simulated AAC users, and from actual AAC users will be compared. This research will expand our knowledge about how to leverage conversational context in augmented reality, and it will curate a public test set contributed by AAC users.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.