Development and Evaluation of an Automatized Comprehension Assessment Tool

自动化理解评估工具的开发和评估

• 批准号: 0113669
• 负责人: Richard Golden
• 金额: $ 39.38万
• 依托单位: University of Texas at Dallas
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2001
• 资助国家: 美国
• 起止时间: 2001-11-15 至 2005-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0113669&HistoricalAwards=false
• 关键词: Development; Evaluation; Automatized; Comprehension; Assessment

The goal of this project is to develop a nationally available fully automated web-based diagnostic system called ARCADE (Automated Reading Comprehension and Diagnostic Evaluation) that will be capable of assessing complex comprehension based upon student free response data. The technology used will be to combine information extraction (IE) technologies and advanced psychometric techniques in novel ways so as to provide detailed assessment and diagnostic information for use by teachers and students in the service of improving classroom instruction and learning. The ARCADE system will also contribute to a data infrastructure useable by other investigators. Specifically, computer scientists interested in the development of new information extraction technology and cognitive scientists and educators interested in the development of new theories of comprehension and assessment would be able to access the database that will be developed in the course of using ARCADE with students. Thus, in addition to improving educational effectiveness in classrooms on a national level, the ARCADE system has the potential to provide a nation-wide resource for facilitating the advancement of scientific research in both the fields of reading comprehension and information technology research.The specific research being pursued is to develop and empirically test the core mathematical algorithms of the ARCADE system with respect to their reliability and validity for assessing reading comprehension in foundational literacy in science and literature. The empirical database will consist of free response data generated by student examinees in grade school and junior high school classroom settings in response to open-ended probe questions. The core ARCADE system employs innovative combinations of information extraction and psychometric techniques to address a critical educational need, namely, ways to assess multiple dimensions of complex comprehension. Such dimensions of comprehension are specified by a set of special semantic networks (called "knowledge digraphs") which embody meaning relations among ideas in texts and documents as well as relations to prior knowledge and inferences. A new statistical model of examinee behavior is then defined which incorporates techniques from the fields of Item Response Theory (IRT), Hidden Markov Model IE technology, and Knowledge Digraph Contribution analysis. The important innovation of this new statistical model is that multiple dimensions of comprehension in conjunction with their respective standard errors can be directly estimated from examinee free response data using Monte Carlo simulation and econometric methods Moreover, using an approach analogous to that developed in IRT , these assessments of comprehension dimensions can be mathematically proven to be reliable across a given family of equivalent testing materials.

该项目的目标是开发一个全国可用的全自动基于网络的诊断系统，称为ARCADE（自动阅读理解和诊断评估），该系统将能够根据学生的自由回答数据评估复杂的理解能力。所使用的技术将以新颖的方式结合信息提取（IE）技术和先进的心理测量技术，以便为教师和学生提供详细的评估和诊断信息，以改善课堂教学。ARCADE系统还将有助于其他调查人员使用的数据基础设施。具体来说，对开发新的信息提取技术感兴趣的计算机科学家和对开发新的理解和评估理论感兴趣的认知科学家和教育工作者将能够访问将在与学生一起使用ARCADE的过程中开发的数据库。因此，除了在全国范围内提高课堂的教育效率外，ARCADE系统有潜力为促进阅读理解和信息技术研究领域的科学研究的进步提供全国性的资源。正在进行的具体研究是开发和实证测试ARCADE系统的核心数学算法，以评估其在科学和文学基础素养阅读理解方面的可靠性和有效性。实证数据库将包括学生考生在小学和初中课堂环境中对开放式探究问题的自由回答数据。核心ARCADE系统采用信息提取和心理测量技术的创新组合来解决关键的教育需求，即评估复杂理解的多个维度的方法。这些理解维度由一组特殊的语义网络（称为“知识有向图”）来指定，这些网络体现了文本和文档中思想之间的意义关系以及与先验知识和推理的关系。然后定义了一个新的考生行为统计模型，该模型结合了项目反应理论（IRT）、隐马尔可夫模型IE技术和知识有向图贡献分析等领域的技术。这个新统计模型的重要创新之处在于，可以使用蒙特卡罗模拟和计量经济学方法直接从考生的自由反应数据中估计理解的多个维度及其各自的标准误差。此外，使用类似于IRT开发的方法，这些理解维度的评估可以在数学上证明在给定的等效测试材料系列中是可靠的。