EAGER: Comprehension Assessment via Spoken Dialog

我的科学导师（MyST）是一个为小学生开发的智能虚拟导师，在过去的10年里，在8个科学领域有超过13,000个口语对话。它的目标是评估学生对概念而不是事实的理解，这对于培养学生和未来的STEM劳动力非常重要。这个早期的探索性EAGER项目旨在确定对新语料库的分析是否可以促进MyST的开发和使用，以便教师、课程开发人员和研究人员可以更容易地为新的科学主题开发自动评估。该方法将应用深度学习技术的最新进展，在与虚拟导师的口语对话中评估学生对科学的概念理解。该项目的动机是最近可用的一个适合训练和测试所提议的系统的示例语料库。该研究的成功成果将产生一种新颖、稳健和可移植的方法，用于从学生的回答中提取语义表示，并将其与参考陈述进行比较，以确定概念关系是否得到正确表达。这种方法有可能消除发展基于对话的评估的主要障碍：语法发展或特定主题的培训。这种新颖且未经测试的方法是高风险的，但如果成功，将获得高回报，因为教程开发人员只需要为讨论的每个概念提供一个示例语句，而不必通过语法开发显式指定所有允许的表达方式。这反过来又可以消除口语对话系统广泛发展的障碍，并使用很少或没有训练数据开发新主题的自动评估。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

My Science Tutor (MyST) is an intelligent virtual tutor for elementary school students that has been developed over the last 10 years, with over 13,000 spoken dialog sessions in 8 areas of science. Its goal is to assess student understanding of concepts rather than facts, which is very important to prepare students and the future workforce in STEM. This early-stage, exploratory EAGER project seeks to determine whether the analysis of a new corpora of data could advance the development and use of MyST so that teachers, curriculum developers and researchers could more easily develop automated assessments for new science topics. The approach will apply recent advances in deep learning techniques to assess students' conceptual understanding of science during spoken dialogs with the virtual tutor. The project is motivated by the recent availability of a corpus of examples suitable for training and testing the proposed system. Successful outcomes of the proposed research will result in a novel, robust and portable method for extracting semantic representations from student responses and comparing these to reference statements to determine if conceptual relationships are correctly expressed. The approach has the potential to remove the primary impediments to developing dialog-based assessments: grammar development or topic-specific training. This novel and untested approach is high-risk, but if successful, would have high-payoff, by allowing tutorial developers to only need to provide one example statement for each concept being discussed rather than having to explicitly specify all allowable ways that it could be expressed through development of grammars. This in turn could remove barriers to widespread development of spoken dialog systems and develop automated assessments for new topics using little or no training data.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.