Development and Evaluation of an Automatized Comprehension Assessment Tool

该项目的目标是开发一个全国可用的全自动网络诊断系统，称为ARCADE（自动阅读理解和诊断评估），该系统将能够根据学生的自由反应数据评估复杂的理解。所使用的技术将是以新颖的方式将联合收割机信息提取技术和先进的心理测量技术结合起来，以便为教师和学生提供详细的评估和诊断信息，为改善课堂教学和学习服务。ARCADE系统还将有助于建立一个可供其他调查人员使用的数据基础设施。具体而言，对开发新的信息提取技术感兴趣的计算机科学家和对开发新的理解和评估理论感兴趣的认知科学家和教育工作者将能够访问在与学生使用ARCADE的过程中开发的数据库。因此，除了在国家一级提高教室的教育效率外，ARCADE系统有可能为国家提供一个广泛的资源，以促进科学研究的进步，在这两个领域的阅读理解和信息技术研究。具体的研究正在进行的是开发和经验测试的核心数学算法的ARCADE系统方面的可靠性和科学和文学基础素养中阅读理解的有效性评估。实验数据库将由学生考生在小学和初中课堂设置的自由反应数据，以回应开放式的探测问题。核心的ARCADE系统采用创新的信息提取和心理测量技术的组合，以解决关键的教育需求，即评估复杂理解的多个维度的方法。这种理解的维度由一组特殊的语义网络（称为“知识有向图”）指定，这些网络体现了文本和文档中思想之间的意义关系以及与先验知识和推理的关系。然后定义了一个新的考生行为的统计模型，它结合了项目反应理论（IRT），隐马尔可夫模型IE技术，和知识有向图贡献分析领域的技术。这一新的统计模型的重要创新之处在于，可以利用蒙特卡罗模拟和计量经济学方法，从考生自由回答数据中直接估计理解的多个维度及其各自的标准误 此外，使用类似于IRT中开发的方法，这些理解维度的评估可以在数学上证明在给定的等效测试材料家族中是可靠的。

The goal of this project is to develop a nationally available fully automated web-based diagnostic system called ARCADE (Automated Reading Comprehension and Diagnostic Evaluation) that will be capable of assessing complex comprehension based upon student free response data. The technology used will be to combine information extraction (IE) technologies and advanced psychometric techniques in novel ways so as to provide detailed assessment and diagnostic information for use by teachers and students in the service of improving classroom instruction and learning. The ARCADE system will also contribute to a data infrastructure useable by other investigators. Specifically, computer scientists interested in the development of new information extraction technology and cognitive scientists and educators interested in the development of new theories of comprehension and assessment would be able to access the database that will be developed in the course of using ARCADE with students. Thus, in addition to improving educational effectiveness in classrooms on a national level, the ARCADE system has the potential to provide a nation-wide resource for facilitating the advancement of scientific research in both the fields of reading comprehension and information technology research.The specific research being pursued is to develop and empirically test the core mathematical algorithms of the ARCADE system with respect to their reliability and validity for assessing reading comprehension in foundational literacy in science and literature. The empirical database will consist of free response data generated by student examinees in grade school and junior high school classroom settings in response to open-ended probe questions. The core ARCADE system employs innovative combinations of information extraction and psychometric techniques to address a critical educational need, namely, ways to assess multiple dimensions of complex comprehension. Such dimensions of comprehension are specified by a set of special semantic networks (called "knowledge digraphs") which embody meaning relations among ideas in texts and documents as well as relations to prior knowledge and inferences. A new statistical model of examinee behavior is then defined which incorporates techniques from the fields of Item Response Theory (IRT), Hidden Markov Model IE technology, and Knowledge Digraph Contribution analysis. The important innovation of this new statistical model is that multiple dimensions of comprehension in conjunction with their respective standard errors can be directly estimated from examinee free response data using Monte Carlo simulation and econometric methods Moreover, using an approach analogous to that developed in IRT , these assessments of comprehension dimensions can be mathematically proven to be reliable across a given family of equivalent testing materials.