Collaborative Research: Development of Natural Language Processing Techniques to Improve Students' Revision of Evidence Use in Argument Writing

写作是多学科学习的基础。这是一个关键的过程，通过这个过程，学生们理解思想——尤其是从原始文本中——并将它们运用到展示他们对概念的新理解和提出合理的论点上。认识到议论文写作的重要性，由自然语言处理（NLP）驱动的多种教育技术已经开发出来，以支持学生和教师在这些过程中。然而，这种系统能提高写作技巧的证据并不多，尤其是对年纪较小的学生。其中一个原因是，NLP技术直到最近才成熟到可以提供与学生写作内容相关的反馈。第二个原因是，即使在收到写作反馈后，许多学生也缺乏修改论文所需的战略知识和技能。一种教育技术可以评估学生修改写作的技能，并对他们的修改尝试提供反馈，这将支持这种关键技能的发展，同时不会给教师带来额外的负担。这种技术有可能为新一代的学生做好准备，使他们能够高效地写作和修改议论文，这是他们为未来的教育和工作环境做好准备所需要的技能。为了解决现有写作教育技术的局限性，研究小组将开发一个系统，利用NLP为学生提供关于他们的修订质量的形成性反馈。该团队将1)开发并建立针对证据使用形成性反馈的修订质量新措施的可靠性和有效性，2)使用NLP使用这些措施自动对修订进行评分，3)根据自动修订评分向学生提供形成性反馈，4)评估这些反馈在改善学生写作和课堂修改方面的效用。该团队假设，这样一个系统将提高学生对基于文本的论点写作的反馈信息的实施，导致更成功的修改，最终更成功的写作。对于学习研究人员和教育工作者来说，修订质量措施将提供有关学生如何实施形成性反馈的详细信息。目前很少有总结性或形成性评估提供这类信息。对于技术研究人员来说，自动修改评分将以新颖的方式扩展先前的写作分析研究，例如，通过评估论文草稿之间的修改质量，并将与先前形成性反馈的一致性纳入评估。将开发多种类型的NLP模型来检查模型类型和不同评估维度（如可靠性、透明度和公平性）之间的权衡。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Writing is foundational to learning in multiple disciplines. It is a critical process by which students make sense of ideas – particularly from source texts – and bring them to bear to demonstrate their emerging understanding of concepts and to make sound arguments. Recognizing the importance of argumentative writing, multiple educational technologies driven by natural language processing (NLP) have been developed to support students and teachers in these processes. However, evidence is modest that such systems improve writing skills, and this is especially the case for younger students. One reason is that NLP technologies have only recently matured to the point that it is possible to provide feedback keyed to the content of students’ writing. A second reason is that many students lack the strategic knowledge and skills needed to revise their essays even after receiving writing feedback. An educational technology that assesses students’ skill at revising their writing and that provides feedback on their revision attempts would support the development of this critical skill, while placing no additional burden on teachers. Such a technology has the potential to prepare a new generation of students for productively writing and revising argumentative essays, a skill they will need in order to be prepared for the educational and workplace settings of the future.To address the limitations of existing educational technologies for writing, the research team will develop a system that leverages NLP to provide students with formative feedback on the quality of their revisions. The team will 1) develop and establish the reliability and validity of new measures of revision quality in response to formative feedback on evidence use, 2) use NLP to automate the scoring of revisions using these measures, 3) provide formative feedback to students based on the automated revision scoring, and 4) evaluate the utility of this feedback in improving student writing and revision in classroom settings. The team hypothesizes that such a system will improve students’ implementation of feedback messages on text-based argument writing, leading toward more successful revision and ultimately more successful writing. For learning researchers and educators, the revision quality measures will provide detailed information about how students implement formative feedback. Few summative or formative assessments currently exist that provide this type of information. For technology researchers, the automated revision scoring will extend prior writing analysis research in novel ways, e.g., by assessing the quality of revisions between essay drafts and by incorporating alignment with prior formative feedback into the assessment. Multiple types of NLP models will be developed to examine tradeoffs between model type and differing evaluation dimensions such as reliability, transparency, and fairness.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.