TRACE - Training Assessment Competences in English as a Second Language

TRACE(英语作为第二语言的培训评估能力)是两个D-A-CH项目的后续项目：ASSET(评估学生的英语课文)和MEWS(衡量中学英语写作)。虽然ASSET在评判英语学生课文时关注教师的诊断能力，但MEWS生成了大量真实的学习者课文语料库，并由训练有素的人和评估软件进行全面评级。TRACE结合这些项目来实现以下目标：1)为大量真实的学生课文语料库生成详细的课文质量评分，通过结合人类评分和机器评分来实现尽可能高的客观性(研究1和2)；2)调查教师、学生、课文和判断水平上的哪些因素决定了职前和在职英语教师通过将他们的评估与人和机器制作的基准分数进行比较来准确地评估复杂的英语学习者课文的能力(研究3)；3)建立一个全面的、免费的在线培训工具，并在大样本的职前英语教师中评估其有效性(研究4、5和6)。经验研究表明，当评分者使用分析性评估实践而不是整体评估实践(即对学生文本应用详细标准)时，得分一致性相对较高(Dempsey等人，2009)。因此，在研究1中，人类评分员将使用ASSET中开发的分析标准来评估MEWS语料库中的大部分文本，这些标准涉及通常由学校教师评估的文本质量方面。这些评分是研究2的基础，在研究2中，使用自然语言处理技术开发了分析性人类分数的自动评分模型。使用大量真实文本语料库将使我们能够在一项研究中将判断准确性的关键决定因素作为连续变量进行调查。研究三将考察教师、学生、文本和判断特征对教师判断准确性的影响(S等，2012年)。然后使用相同的语料库来调查具体培训措施的有效性：研究4将检查收到对其分析性文本判断的反馈(与基准进行比较)的教师是否在评估中变得更准确。研究5和6将为教师提供自动生成的关于语篇质量关键特征的可视化反馈，以确定这是否有助于他们更准确地评估语篇(研究5中的词汇和研究6中的议论结构)。基于这些研究的结果，在TRACE中，我们将为职前和在职教师开发一个全面的、免费的在线培训工具，以培养他们在使用详细的分析标准判断学生课文时的诊断能力。

TrACE (Training Assessment Competencies in English as a Second Language) is a follow-up to two D-A-CH projects: ASSET (Assessing Student’s English Texts) and MEWS (Measuring English Writing at Secondary Level). While ASSET was concerned with teachers’ diagnostic competences when judging English student texts, MEWS generated a large corpus of authentic learner texts with holistic ratings by trained humans and assessment software. TrACE combines these projects to achieve following objectives: 1) Generating detailed ratings of text qualities for a large corpus of authentic student texts, achieving highest possible objectivity by combining human ratings and machine scoring (Studies 1 & 2); 2) Investigating what factors at teacher, student, text, and judgment level determine the ability of preservice and in-service English teachers to assess complex English learner texts accurately by comparing their assessment to human- and machine-made benchmark scores (Study 3);3) Building a comprehensive, free online training tool and evaluating its effectiveness with a large sample of pre-service English teachers (Study 4, 5 & 6).Empirical studies have shown scoring consistency to be relatively high when raters use analytic instead of holistic assessment practices (i.e., apply detailed criteria to student texts) (Dempsey et al., 2009). Therefore, in Study 1, human raters will assess a large portion of texts from the MEWS corpus using analytic criteria developed in ASSET, relating to aspects of textual quality typically assessed by teachers in schools. These ratings serve as the basis for Study 2, in which automatic scoring models for the analytic human scores are developed using natural language processing techniques. Working with a large corpus of authentic texts will allow us to investigate key determinants of judgment accuracy as continuous variables within one single study. Study 3 will investigate the influence of teacher, student, text, and judgments characteristics on teacher judgement accuracy (Südkamp et al., 2012). The same corpus is then used to investigate the effectiveness of specific training measures: Study 4 will examine whether teachers who receive feedback on their analytical text judgments (comparison with benchmarks) become more accurate in their assessment. Studies 5 & 6 will provide teachers with automatically generated visualized feedback on key features of text quality to determine whether this helps them to assess texts more accurately (vocabulary in Study 5 and argumentative structure in Study 6). Based on the results of these studies, in TrACE we will develop a comprehensive, free on-line training tool for pre-service and in-service teachers to develop their diagnostic competence in judging student texts while using detailed analytic criteria.