I-Corps: Investigating the Commercialization of Peer Review Tools and Writing Analytics

I-Corps：调查同行评审工具的商业化和撰写分析

• 批准号: 1636511
• 负责人: Joseph Moxley
• 金额: $ 5万
• 依托单位: University of South Florida
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-05-01 至 2017-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1636511&HistoricalAwards=false
• 关键词: Corps; Investigating; Commercialization; Peer; Review

Current measures of coaching and assessing student writing, while time consuming and well intentioned, fail to provide students with the feedback they need to improve as writers and peer reviewers. Improving feedback and assessment of students' writing is critical to U.S. efforts to compete globally. Currently, U.S. rankings for literacy have been declining over the past decade: in 2012, the Programme for International Student Assessment concluded that the U.S. literacy rate fell from 10th to 20th in the latest study on global rankings. In 2013, the College Board reported 57% of SAT takers did not qualify as college ready. The ACT found 31% of high school graduates failed to meet ACT College Readiness Benchmarks, and the most recent NAEP Writing Report determined 73% of 12th graders received scores of Below Basic or Basic as opposed to Proficient or Advanced in 2011. Few educators or students dispute the importance of quality feedback on students? writing. However, over the past 100 years, researchers have repeatedly found that teachers' grades lack validity and reliability; that teachers' comments on papers often lack helpful, critical commentary; and that students do not know how to critique other students' writing even though they are frequently assigned peer-reviews. While teachers? shared use of rubrics has led to improved inter-rater agreement scores?especially in testing situations where teachers can practice scoring to reach stronger levels of consensus?comparisons of teachers' scores across sections of the same course in authentic learning contexts has found great variance in teachers' scores and comments.This research seeks to help expedite document markup, peer-review, team-project, eportfolio, and writing-assessment processes; empower instructors and students to provide more helpful, timely feedback; better prepare students for peer reviews and team projects; and enable high schools, colleges, and universities to more accurately assess learning, improve retention, and prepare accreditation reports. Ultimately, this research aspires to conceptualize digital tools that will enable the U.S. to complete on global measures of cognitive, interpersonal and intrapersonal competencies. During the I-Corps program, the team will interview academic stakeholders (writing program administrators, faculty, and university administrators who are concerned with retention and business stakeholders (technical writing firms, competitors, and NLP researchers)?to evaluate mockups for writing analytics, student-retention alerts, data visualizations, and intelligent tutoring systems. For example, the team will ask writing program administrators if they would be likely to adopt and use tools that provide inter-rater agreement reports on instructors? scores in relation to students' peer-review scores or lexical analysis of the instructors? and students? written commentaries. Additionally, the team will hold focus groups with instructors and students to ascertain the helpfulness of NLP-based reports that vet peer reviews. Ultimately, this research will explore the efficacy of second-generation digital tools?i.e., tools that provide writing analytics that give information back to users based on their use of a tool, such as lexical analysis of comments, reports of inter-rater agreement among reviewers, reports that analyze the length and quality of feedback, or badges that incentivize quality reviews. By the end of the I-Corps program, the team will prioritize a development plan and a business plan that will allow for commercialization of the technology, My Reviewers http://myreviewers.com.

目前的辅导和评估学生写作的措施，而耗时和善意的，未能提供学生的反馈，他们需要提高作为作家和同行评审。改善对学生写作的反馈和评估对美国在全球竞争中的努力至关重要。目前，美国的识字率排名在过去十年中一直在下降：2012年，国际学生评估计划得出结论，在最新的全球排名研究中，美国的识字率从第10位下降到第20位。2013年，美国大学理事会报告说，57%的SAT考生没有资格进入大学。ACT发现31%的高中毕业生未能达到ACT大学准备基准，而最近的NAEP写作报告确定73%的12年级学生在2011年获得低于基本或基本的分数，而不是熟练或高级。很少有教育工作者或学生质疑对学生的质量反馈的重要性？写作然而，在过去的100年里，研究人员一再发现，教师的成绩缺乏有效性和可靠性;教师对论文的评论往往缺乏有用的，批判性的评论;学生不知道如何批评其他学生的写作，即使他们经常被分配同行评审。当教师？共同使用的标题导致提高评分员之间的一致性分数？特别是在测试的情况下，教师可以练习评分，以达到更高的共识水平？通过对同一门课程在真实学习环境中不同部分的教师成绩进行比较，发现教师的成绩和评语存在很大差异，本研究旨在帮助加快文档标记、同行评议、团队项目、电子档案袋和写作评估过程，使教师和学生能够提供更有用、更及时的反馈，更好地为学生的同行评议和团队项目做好准备，使高中、学院和大学能够更准确地评估学习情况，提高保留率，并准备认证报告。最终，这项研究渴望概念化数字化工具，使美国能够完成认知，人际和自我能力的全球措施。在I-Corps项目期间，团队将采访学术利益相关者（写作项目管理人员，教师和关心保留和业务利益相关者的大学管理人员（技术写作公司，竞争对手和NLP研究人员）？评估写作分析、学生保留警报、数据可视化和智能辅导系统的模型。例如，团队将询问编写程序的管理员，他们是否可能采用和使用提供教师评分员间协议报告的工具？分数与学生的同行评审分数或教师的词汇分析有关？和学生？书面评论。此外，该团队将与教师和学生举行焦点小组，以确定基于NLP的报告对同行评审的帮助。最终，本研究将探讨第二代数字化工具的功效？也就是说，提供写作分析的工具，根据用户对工具的使用情况向用户反馈信息，例如评论的词法分析、评论者之间的评分者一致性报告、分析反馈长度和质量的报告或激励质量评论的徽章。在I-Corps项目结束时，该团队将优先考虑一项开发计划和一项商业计划，以实现该技术的商业化。http://myreviewers.com