Development and Evaluation of an Automatized Comprehension Assessment Tool

这个项目的目标是开发一个全国可用的全自动网络诊断系统，称为ARCADE(自动阅读理解和诊断评估)，它将能够根据学生的自由反应数据来评估复杂的理解。所使用的技术将以新的方式结合信息提取(IE)技术和先进的心理测量技术，以便为教师和学生提供详细的评估和诊断信息，为改进课堂教学和学习服务。拱廊系统还将为其他调查人员提供可用的数据基础设施。具体地说，对开发新的信息提取技术感兴趣的计算机科学家以及对开发新的理解和评估理论感兴趣的认知科学家和教育工作者将能够访问将在与学生一起使用拱廊的过程中开发的数据库。因此，CARADE系统除了在国家层面上提高课堂教学效率外，还有可能在全国范围内提供资源，促进阅读理解和信息技术研究领域的科学研究的进步。正在进行的具体研究是开发和实证测试ARCARD系统的核心数学算法，以评估科学和文学基础素养中阅读理解的可靠性和有效性。经验数据库将包括学生在小学和初中课堂环境中对开放式探究性问题的回答所产生的自由回答数据。核心拱廊系统使用信息提取和心理测量技术的创新组合来满足关键的教育需求，即评估复杂理解的多个维度的方法。这样的理解维度是由一组特殊的语义网络(称为知识有向图)指定的，这些网络体现了文本和文档中概念之间的意义关系，以及与先前知识和推理的关系。然后定义了一个新的考生行为统计模型，该模型综合了项目反应理论、隐马尔可夫模型、IE技术和知识有向图贡献分析等领域的技术。这种新的统计模型的重要创新之处在于，可以使用蒙特卡罗模拟和计量经济学方法从考生的自由反应数据中直接估计理解的多个维度及其各自的标准误差。此外，使用与IRT中开发的方法类似的方法，这些理解维度的评估可以在给定的等同测试材料家族中被数学证明是可靠的。

The goal of this project is to develop a nationally available fully automated web-based diagnostic system called ARCADE (Automated Reading Comprehension and Diagnostic Evaluation) that will be capable of assessing complex comprehension based upon student free response data. The technology used will be to combine information extraction (IE) technologies and advanced psychometric techniques in novel ways so as to provide detailed assessment and diagnostic information for use by teachers and students in the service of improving classroom instruction and learning. The ARCADE system will also contribute to a data infrastructure useable by other investigators. Specifically, computer scientists interested in the development of new information extraction technology and cognitive scientists and educators interested in the development of new theories of comprehension and assessment would be able to access the database that will be developed in the course of using ARCADE with students. Thus, in addition to improving educational effectiveness in classrooms on a national level, the ARCADE system has the potential to provide a nation-wide resource for facilitating the advancement of scientific research in both the fields of reading comprehension and information technology research.The specific research being pursued is to develop and empirically test the core mathematical algorithms of the ARCADE system with respect to their reliability and validity for assessing reading comprehension in foundational literacy in science and literature. The empirical database will consist of free response data generated by student examinees in grade school and junior high school classroom settings in response to open-ended probe questions. The core ARCADE system employs innovative combinations of information extraction and psychometric techniques to address a critical educational need, namely, ways to assess multiple dimensions of complex comprehension. Such dimensions of comprehension are specified by a set of special semantic networks (called "knowledge digraphs") which embody meaning relations among ideas in texts and documents as well as relations to prior knowledge and inferences. A new statistical model of examinee behavior is then defined which incorporates techniques from the fields of Item Response Theory (IRT), Hidden Markov Model IE technology, and Knowledge Digraph Contribution analysis. The important innovation of this new statistical model is that multiple dimensions of comprehension in conjunction with their respective standard errors can be directly estimated from examinee free response data using Monte Carlo simulation and econometric methods Moreover, using an approach analogous to that developed in IRT , these assessments of comprehension dimensions can be mathematically proven to be reliable across a given family of equivalent testing materials.