TRACE - Training Assessment Competences in English as a Second Language

TrACE（英语作为第二语言的培训评估能力）是两个D-A-CH项目的后续：ASSET（学生英语文本评估）和MEWS（中学英语写作测试）。ASSET关注的是教师在判断英语学生文本时的诊断能力，而MEWS生成了大量真实学习者文本的语料库，并由训练有素的人员和评估软件进行整体评分。TrACE将这些项目结合起来以实现以下目标：1)为大量真实学生文本的语料库生成文本质量的详细评级，通过结合人类评级和机器评分来实现尽可能高的客观性（研究1和2）；2)通过将职前和在职英语教师的评估与人工和机器基准分数进行比较，调查教师、学生、文本和判断层面的哪些因素决定了职前和在职英语教师准确评估复杂英语学习者文本的能力（研究3）；3)建立一个全面的、免费的在线培训工具，并通过大量的职前英语教师样本评估其有效性（研究4、5和6）。实证研究表明，当评分者使用分析而不是整体评估实践（即对学生文本应用详细标准）时，评分一致性相对较高（Dempsey et al., 2009）。因此，在研究1中，人类评分员将使用ASSET中开发的分析标准评估MEWS语料库中的大部分文本，这些标准与学校教师通常评估的文本质量有关。这些评分作为研究2的基础，其中使用自然语言处理技术开发了分析人类评分的自动评分模型。使用大量真实文本的语料库将使我们能够在单个研究中作为连续变量调查判断准确性的关键决定因素。研究3将调查教师、学生、文本和判断特征对教师判断准确性的影响（s<s:2> dkamp et al., 2012）。然后使用相同的语料库来调查特定培训措施的有效性：研究4将检查收到关于分析文本判断（与基准比较）的反馈的教师是否在评估中变得更准确。研究5和研究6将为教师提供自动生成的文本质量关键特征的可视化反馈，以确定这是否有助于他们更准确地评估文本（研究5中的词汇和研究6中的论证结构）。基于这些研究的结果，在TrACE中，我们将为职前和在职教师开发一个全面的、免费的在线培训工具，以培养他们在使用详细的分析标准来判断学生文本的诊断能力。

TrACE (Training Assessment Competencies in English as a Second Language) is a follow-up to two D-A-CH projects: ASSET (Assessing Student’s English Texts) and MEWS (Measuring English Writing at Secondary Level). While ASSET was concerned with teachers’ diagnostic competences when judging English student texts, MEWS generated a large corpus of authentic learner texts with holistic ratings by trained humans and assessment software. TrACE combines these projects to achieve following objectives: 1) Generating detailed ratings of text qualities for a large corpus of authentic student texts, achieving highest possible objectivity by combining human ratings and machine scoring (Studies 1 & 2); 2) Investigating what factors at teacher, student, text, and judgment level determine the ability of preservice and in-service English teachers to assess complex English learner texts accurately by comparing their assessment to human- and machine-made benchmark scores (Study 3);3) Building a comprehensive, free online training tool and evaluating its effectiveness with a large sample of pre-service English teachers (Study 4, 5 & 6).Empirical studies have shown scoring consistency to be relatively high when raters use analytic instead of holistic assessment practices (i.e., apply detailed criteria to student texts) (Dempsey et al., 2009). Therefore, in Study 1, human raters will assess a large portion of texts from the MEWS corpus using analytic criteria developed in ASSET, relating to aspects of textual quality typically assessed by teachers in schools. These ratings serve as the basis for Study 2, in which automatic scoring models for the analytic human scores are developed using natural language processing techniques. Working with a large corpus of authentic texts will allow us to investigate key determinants of judgment accuracy as continuous variables within one single study. Study 3 will investigate the influence of teacher, student, text, and judgments characteristics on teacher judgement accuracy (Südkamp et al., 2012). The same corpus is then used to investigate the effectiveness of specific training measures: Study 4 will examine whether teachers who receive feedback on their analytical text judgments (comparison with benchmarks) become more accurate in their assessment. Studies 5 & 6 will provide teachers with automatically generated visualized feedback on key features of text quality to determine whether this helps them to assess texts more accurately (vocabulary in Study 5 and argumentative structure in Study 6). Based on the results of these studies, in TrACE we will develop a comprehensive, free on-line training tool for pre-service and in-service teachers to develop their diagnostic competence in judging student texts while using detailed analytic criteria.