Collaborative Research: Development of Natural Language Processing Techniques to Improve Students' Revision of Evidence Use in Argument Writing

写作是多学科学习的基础。这是一个关键的过程，通过这个过程，学生理解思想-特别是来自源文本的思想-并运用这些思想来展示他们对概念的新理解，并提出合理的论点。认识到议论文写作的重要性，由自然语言处理（NLP）驱动的多种教育技术已经开发出来，以支持学生和教师在这些过程中。然而，证据是适度的，这种系统提高写作技能，这是特别是年轻的学生的情况。一个原因是NLP技术最近才成熟到可以提供针对学生写作内容的反馈。第二个原因是，许多学生缺乏必要的战略知识和技能，即使在收到写作反馈后也要修改他们的论文。一种评估学生修改写作的技能并对他们的修改尝试提供反馈的教育技术将支持这一关键技能的发展，同时不会给教师带来额外的负担。这样的技术有可能为新一代学生有效地写作和修改议论文做好准备，这是他们为未来的教育和工作环境做好准备所需要的技能。为了解决现有写作教育技术的局限性，研究团队将开发一个系统，利用NLP为学生提供关于修改质量的形成性反馈。该团队将1）开发和建立新的修订质量措施的可靠性和有效性，以应对证据使用的形成性反馈，2）使用NLP来自动化使用这些措施的修订评分，3）根据自动修订评分向学生提供形成性反馈，以及4）评估这种反馈在改善学生写作和课堂环境中的修订方面的效用。该团队假设，这样的系统将改善学生对基于文本的论点写作的反馈信息的实施，从而导致更成功的修改，最终更成功的写作。对于学习研究者和教育工作者来说，修订质量测量将提供有关学生如何实施形成性反馈的详细信息。目前很少有总结性或形成性评估提供这类信息。对于技术研究人员来说，自动修订评分将以新的方式扩展先前的写作分析研究，例如，通过评估论文草稿之间的修改质量，并将与先前形成的反馈结合到评估中。将开发多种类型的NLP模型，以检查模型类型和不同评估维度（如可靠性、透明度和公平性）之间的权衡。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估来支持。

Writing is foundational to learning in multiple disciplines. It is a critical process by which students make sense of ideas – particularly from source texts – and bring them to bear to demonstrate their emerging understanding of concepts and to make sound arguments. Recognizing the importance of argumentative writing, multiple educational technologies driven by natural language processing (NLP) have been developed to support students and teachers in these processes. However, evidence is modest that such systems improve writing skills, and this is especially the case for younger students. One reason is that NLP technologies have only recently matured to the point that it is possible to provide feedback keyed to the content of students’ writing. A second reason is that many students lack the strategic knowledge and skills needed to revise their essays even after receiving writing feedback. An educational technology that assesses students’ skill at revising their writing and that provides feedback on their revision attempts would support the development of this critical skill, while placing no additional burden on teachers. Such a technology has the potential to prepare a new generation of students for productively writing and revising argumentative essays, a skill they will need in order to be prepared for the educational and workplace settings of the future.To address the limitations of existing educational technologies for writing, the research team will develop a system that leverages NLP to provide students with formative feedback on the quality of their revisions. The team will 1) develop and establish the reliability and validity of new measures of revision quality in response to formative feedback on evidence use, 2) use NLP to automate the scoring of revisions using these measures, 3) provide formative feedback to students based on the automated revision scoring, and 4) evaluate the utility of this feedback in improving student writing and revision in classroom settings. The team hypothesizes that such a system will improve students’ implementation of feedback messages on text-based argument writing, leading toward more successful revision and ultimately more successful writing. For learning researchers and educators, the revision quality measures will provide detailed information about how students implement formative feedback. Few summative or formative assessments currently exist that provide this type of information. For technology researchers, the automated revision scoring will extend prior writing analysis research in novel ways, e.g., by assessing the quality of revisions between essay drafts and by incorporating alignment with prior formative feedback into the assessment. Multiple types of NLP models will be developed to examine tradeoffs between model type and differing evaluation dimensions such as reliability, transparency, and fairness.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.