EAGER: Collaborative Research: Computer-Aided Response-to-Intervention for Reading Comprehension Disabilities

44个州和哥伦比亚特区通过的共同核心学习标准(CCLS)规定了学生在每个年级结束前应该展示的技能。CCLS强调的一项关键技能是阅读能力，这是学习所有内容领域的先导。在纽约州，3-8年级的学生每年春天都要参加英语语言艺术(ELA)测试，以衡量他们在阅读方面的CCLS成绩。ELA考试既包括多项选择题，也包括基于短文的开放式问题；要做好这项测试，学生应该能够仔细阅读一篇文章，寻找文本证据，并从中得出逻辑推理。为了报告结果，学生正确回答的数量被转换为量表分数；这又被分为四个表现级别：NYS级别1表示远低于熟练程度，NYS级别2表示部分熟练，NYS级别3表示熟练，NYS级别4表示年级标准优异。学校为表现水平为NYS 1级或NYS 2级的学生安排学业干预服务。为了推动那些有可能达不到学业期望的学生的改变，对干预的反应模式旨在根据这些评估结果提供指导。但PIs认为，作为评估结果的单一表现分数通常不足以识别潜在的学习问题，特别是在阅读理解方面。在这个探索性项目中，他们将专注于在词汇层面上发现评估结果中的错误模式，期望这些最终将有助于更好地理解如何将表现不佳的基于文本的分析性阅读评估的原始数据转化为信息丰富且易于理解的结构，以提供有效的阅读理解干预。项目成果将补充目前的评分系统，支持个人层面的诊断，并促进将有类似阅读障碍的学生归入同一干预小组，以优化学校教学资源。该方法还将支持数据驱动的教学框架，使每次考试获得的信息最大化，从而减少每学年的考试次数和教学时数。这是计算机科学家(Tsai)和扫盲教育专家(Zakierski)之间的跨学科合作。Pi Tsai将负责计算机算法开发和数据分析，而Pi Zakierski将负责数据收集和基于识字和教育学结果的拟议方法的评估。该团队将建立一个数据库，其中包含从儿童文学到三年级的词汇属性，来自纽约ELA的评估材料，以及干预记录。他们将每年从大约120名表现在NYS 2级或以下的三年级学生中收集ELA评估材料，用于研究开发和评估。他们将开发一个计算机辅助干预系统，对表现不佳的个人ELA评估材料进行数据挖掘，以发现错误模式，这应该有助于教师识别学生潜在的阅读理解问题，以便制定更有效的教学计划。他们将通过形成性评估和终结性评估来评估成绩，前者包括对教师的问卷调查和对学生在干预期间进行的模拟ELA测试，后者是干预后4月份的真实ELA测试。从计算机科学的角度来看，主要的挑战是数据集的小尺寸。PIS将开发以领域知识为导向的新技术，以便进行有意义的分析，以发现此类情况下的错误模式；如果成功，这种方法将在其他更容易获得“小数据”的情况下打开广泛研究机会的大门。此外，数据挖掘对扫盲教育本身的探索将是一项独特的贡献，因为这两个领域的结合尚未得到研究界的太多关注，还有许多有趣的问题等待使用计算方法来解决。

The Common Core Learning Standards (CCLS), adopted by 44 states and the District of Columbia, define the skills a student should demonstrate by the end of each grade. One key skill emphasized by CCLS is reading ability, which is the precursor for learning in all content areas. In New York State, students in grades 3-8 take an English Language Arts (ELA) test each spring to measure their CCLS achievement in reading. An ELA test contains both multiple choice questions and open-ended questions based on short text passages; to do well, students should be able to read a passage closely for textual evidence and draw logical inferences from it. To report the results, the number of correct student responses is converted into a scale score; this in turn is divided into four performance levels: NYS Level 1 for well below proficient, NYS Level 2 for partially proficient, NYS Level 3 for proficient, and NYS Level 4 for exceptional in grade-level standards. Schools arrange academic intervention services for students whose performance level is either NYS Level 1 or NYS Level 2. To drive change in students who are at risk for not meeting academic expectations, the Response-to-Intervention model aims to deliver instructions as a function of these assessment outcomes. But the PIs argue that a single performance score as the assessment outcome is often insufficient for identifying underlying learning problems, especially in reading comprehension. In this exploratory project they will focus on discovering error patterns in assessment outcomes at the lexical level, in the expectation these will ultimately lead to improved understanding of how the raw data from a pool of underperforming text-based analytic reading assessments can be transformed into an informative and understandable structure for delivery of effective reading comprehension interventions. Project outcomes will complement the current scoring system by supporting diagnosis at an individual level, and by facilitating grouping of students with similar reading disabilities in the same intervention group in order to optimize school teaching resources. The approach will also support a data-driven instruction framework by maximizing the information gain from each test, which can result in fewer tests taken and more hours for teaching per school year.This is an interdisciplinary collaboration between a computer scientist (Tsai) and an expert in literacy education (Zakierski). PI Tsai will be responsible for computer algorithm development and data analysis, whereas PI Zakierski will be in charge of data collection and evaluation of the proposed approach based on findings in literacy and pedagogy. The team will build a database containing words with lexical properties from literature for children up to grade 3, assessment materials from NYS ELAs, and intervention records. They will annually collect ELA assessment materials from a pool of approximately 120 third grade students with performance at NYS Level 2 or below, for both research development and evaluation. They will develop a computer-aided intervention system that performs data-mining on underperforming individual ELA assessment materials to discover error patterns, which should assist the teacher in identifying a student's underlying reading comprehension problems in order to prepare a more effective instruction plan. And they will evaluate the performance by doing both formative and summative assessments, the former to consist of questionnaires for teachers and mock ELA tests for students taken during the period of intervention, and the latter being the real ELA tests in April following the intervention. From the computer science perspective, the main challenge is the small size of the dataset. The PIs will develop new techniques that are domain-knowledge driven for performing meaningful analysis to discover error patterns in such situations; if successful, the approach will open the door to broad research opportunities in other cases where "small data" is easier to come by. In addition, the exploration of data mining on literacy education itself will constitute a unique contribution, since the marriage of the two fields has not yet received much attention from the research community and there are many interesting questions waiting to be addressed using computational approaches.