TRACE - Training Assessment Competences in English as a Second Language

TrACE（英语作为第二语言的培训评估能力）是两个D-A-CH项目的后续行动：ASSET（评估学生的英语文本）和MEWS（测量中学英语写作）。虽然ASSET关注教师在判断英语学生文本时的诊断能力，但MEWS生成了一个大型的真实学习者文本语料库，并由受过训练的人和评估软件进行整体评级。TrACE结合这些项目以实现以下目标：1）为大量真实学生文本语料库生成文本质量的详细评级，通过结合人工评级和机器评分实现尽可能高的客观性（研究1和2）; 2）调查教师、学生、文本、和判断水平决定了在职和在职英语教师对复杂英语学习者语篇的准确评估能力，3）建立一个全面的、免费的在线培训工具，并对大样本的职前英语教师进行了有效性评估（研究4、5和6）。实证研究表明，当评分者使用分析性而不是整体性评估实践时，评分的一致性相对较高（即，将详细的标准应用于学生文本）（Dempsey等人，2009年）的报告。因此，在研究1中，人类评分员将使用ASSET开发的分析标准评估MEWS语料库中的大部分文本，这些标准与学校教师通常评估的文本质量有关。这些评级作为研究2的基础，其中使用自然语言处理技术开发了分析人类分数的自动评分模型。使用大量真实文本语料库将使我们能够在一项研究中将判断准确性的关键决定因素作为连续变量进行研究。研究3将调查教师、学生、文本和判断特征对教师判断准确性的影响（Südkamp等人，2012年）。相同的语料库，然后使用特定的培训措施的有效性进行调查：研究4将探讨教师谁收到反馈，他们的分析文本的判断（与基准比较），在他们的评估变得更准确。研究5和6将为教师提供自动生成的文本质量关键特征的可视化反馈，以确定这是否有助于他们更准确地评估文本（研究5中的词汇和研究6中的论证结构）。基于这些研究的结果，在TrACE中，我们将为职前和在职教师开发一个全面的免费在线培训工具，以培养他们在使用详细的分析标准判断学生文本时的诊断能力。

TrACE (Training Assessment Competencies in English as a Second Language) is a follow-up to two D-A-CH projects: ASSET (Assessing Student’s English Texts) and MEWS (Measuring English Writing at Secondary Level). While ASSET was concerned with teachers’ diagnostic competences when judging English student texts, MEWS generated a large corpus of authentic learner texts with holistic ratings by trained humans and assessment software. TrACE combines these projects to achieve following objectives: 1) Generating detailed ratings of text qualities for a large corpus of authentic student texts, achieving highest possible objectivity by combining human ratings and machine scoring (Studies 1 & 2); 2) Investigating what factors at teacher, student, text, and judgment level determine the ability of preservice and in-service English teachers to assess complex English learner texts accurately by comparing their assessment to human- and machine-made benchmark scores (Study 3);3) Building a comprehensive, free online training tool and evaluating its effectiveness with a large sample of pre-service English teachers (Study 4, 5 & 6).Empirical studies have shown scoring consistency to be relatively high when raters use analytic instead of holistic assessment practices (i.e., apply detailed criteria to student texts) (Dempsey et al., 2009). Therefore, in Study 1, human raters will assess a large portion of texts from the MEWS corpus using analytic criteria developed in ASSET, relating to aspects of textual quality typically assessed by teachers in schools. These ratings serve as the basis for Study 2, in which automatic scoring models for the analytic human scores are developed using natural language processing techniques. Working with a large corpus of authentic texts will allow us to investigate key determinants of judgment accuracy as continuous variables within one single study. Study 3 will investigate the influence of teacher, student, text, and judgments characteristics on teacher judgement accuracy (Südkamp et al., 2012). The same corpus is then used to investigate the effectiveness of specific training measures: Study 4 will examine whether teachers who receive feedback on their analytical text judgments (comparison with benchmarks) become more accurate in their assessment. Studies 5 & 6 will provide teachers with automatically generated visualized feedback on key features of text quality to determine whether this helps them to assess texts more accurately (vocabulary in Study 5 and argumentative structure in Study 6). Based on the results of these studies, in TrACE we will develop a comprehensive, free on-line training tool for pre-service and in-service teachers to develop their diagnostic competence in judging student texts while using detailed analytic criteria.