I-Corps: Investigating the Commercialization of Peer Review Tools and Writing Analytics

目前指导和评估学生写作的措施虽然耗时且用意良好，但未能为学生提供他们作为作家和同行评审员需要改进的反馈。改善对学生写作的反馈和评估对于美国在全球竞争的努力至关重要。目前，美国的识字排名在过去十年里一直在下降：2012年，国际学生评估计划得出结论，在最新的全球排名研究中，美国的识字率从第10位降至第20位。2013年，美国大学理事会报告称，57%的SAT考生没有资格上大学。ACT发现，31%的高中毕业生未能达到ACT大学准备基准，最新的NAEP写作报告显示，2011年，73%的12年级学生的成绩低于基础或基础，而不是熟练或高级。很少有教育工作者或学生质疑对学生进行质量反馈的重要性？写作。然而，在过去的100年里，研究人员一再发现，教师的评分缺乏有效性和可靠性；教师对试卷的评论往往缺乏有益的、批判性的评论；学生不知道如何批评其他学生的写作，即使他们经常被指派同行评议。而老师呢？共享使用类比法提高了评分者之间的一致分数，尤其是在教师可以练习评分以达成更高水平共识的测试情况下？在真实的学习环境中，教师在同一门课程的各个部分的分数比较发现，教师的分数和评论存在很大差异。这项研究旨在帮助加快文件标记、同行评议、团队项目、ePortfolio和写作评估过程；使教师和学生能够提供更有帮助的、及时的反馈；更好地为学生进行同行评议和团队项目做准备；使高中、学院和大学能够更准确地评估学习情况，提高留存率，并准备认证报告。最终，这项研究的目标是将数字工具概念化，使美国能够完成对认知、人际和个人内部能力的全球测量。在i-Corps计划期间，团队将采访学术利益相关者(写作项目管理员、教师和关心留存问题的大学管理人员和业务利益相关者(技术写作公司、竞争对手和NLP研究人员))，以评估写作分析、学生留存警报、数据可视化和智能辅导系统的模型。例如，团队将询问撰写项目的管理员，他们是否可能采用和使用提供讲师评分者间协议报告的工具？分数与学生的同行评议分数或教师的词汇分析有关？学生呢？书面评论。此外，该团队将与教师和学生举行焦点小组，以确定基于NLP的报告对同行审查的帮助。最终，这项研究将探索第二代数字工具的有效性--即提供写作分析的工具，根据用户对工具的使用向用户反馈信息，例如评论的词法分析、评价者之间一致的报告、分析反馈长度和质量的报告，或者激励质量审查的徽章。在i-Corps计划结束之前，团队将优先考虑允许技术商业化的发展计划和商业计划，My Revisers http://myreviewers.com.

Current measures of coaching and assessing student writing, while time consuming and well intentioned, fail to provide students with the feedback they need to improve as writers and peer reviewers. Improving feedback and assessment of students' writing is critical to U.S. efforts to compete globally. Currently, U.S. rankings for literacy have been declining over the past decade: in 2012, the Programme for International Student Assessment concluded that the U.S. literacy rate fell from 10th to 20th in the latest study on global rankings. In 2013, the College Board reported 57% of SAT takers did not qualify as college ready. The ACT found 31% of high school graduates failed to meet ACT College Readiness Benchmarks, and the most recent NAEP Writing Report determined 73% of 12th graders received scores of Below Basic or Basic as opposed to Proficient or Advanced in 2011. Few educators or students dispute the importance of quality feedback on students? writing. However, over the past 100 years, researchers have repeatedly found that teachers' grades lack validity and reliability; that teachers' comments on papers often lack helpful, critical commentary; and that students do not know how to critique other students' writing even though they are frequently assigned peer-reviews. While teachers? shared use of rubrics has led to improved inter-rater agreement scores?especially in testing situations where teachers can practice scoring to reach stronger levels of consensus?comparisons of teachers' scores across sections of the same course in authentic learning contexts has found great variance in teachers' scores and comments.This research seeks to help expedite document markup, peer-review, team-project, eportfolio, and writing-assessment processes; empower instructors and students to provide more helpful, timely feedback; better prepare students for peer reviews and team projects; and enable high schools, colleges, and universities to more accurately assess learning, improve retention, and prepare accreditation reports. Ultimately, this research aspires to conceptualize digital tools that will enable the U.S. to complete on global measures of cognitive, interpersonal and intrapersonal competencies. During the I-Corps program, the team will interview academic stakeholders (writing program administrators, faculty, and university administrators who are concerned with retention and business stakeholders (technical writing firms, competitors, and NLP researchers)?to evaluate mockups for writing analytics, student-retention alerts, data visualizations, and intelligent tutoring systems. For example, the team will ask writing program administrators if they would be likely to adopt and use tools that provide inter-rater agreement reports on instructors? scores in relation to students' peer-review scores or lexical analysis of the instructors? and students? written commentaries. Additionally, the team will hold focus groups with instructors and students to ascertain the helpfulness of NLP-based reports that vet peer reviews. Ultimately, this research will explore the efficacy of second-generation digital tools?i.e., tools that provide writing analytics that give information back to users based on their use of a tool, such as lexical analysis of comments, reports of inter-rater agreement among reviewers, reports that analyze the length and quality of feedback, or badges that incentivize quality reviews. By the end of the I-Corps program, the team will prioritize a development plan and a business plan that will allow for commercialization of the technology, My Reviewers http://myreviewers.com.