Investigating the extent to which speech rater judgement relates to objective speech measures

2022年，我们完成了从日本评级机构收集数据的工作，并提交了申请，要求允许收集加拿大公立学校教师评级机构的反馈。我们还开始对迄今收集的数据进行定量分析。初步研究结果表明，L1日本英语教师评为L1日本L2英语扬声器样本为低得分的所有类别，除了整体的可理解性，但教学经验并没有影响L1日本L2英语扬声器样本评级。然而，对于L1泰国L2英语样本，评分员L1只影响音段发音，我们观察到教学经验的积极影响，与L1日本L2英语学生评分样本显着高于他们的英语教师同行。此外，日本教师和学生报告显着较低的信心，他们的能力，以评价英语演讲比英语L1或L2英语教师在日本。此外，关于评估演讲的具体方面，流利性是最容易的，而语法和词汇往往是最具挑战性的。对于以英语为第二语言的人来说，他们对自己的正确语法和词汇知识缺乏信心，而对于以英语为第一语言的人来说，困难在于根据样本长度准确判断语法和词汇的复杂性。在词汇分析方面，我们观察到自我评估的词汇分数和一些词汇测量之间存在相关性，但没有发现外部评分者的评分之间存在相关性。

In 2022 we concluded collecting data from Japan-based raters and submitted applications for permission to collect responses from Canada-based teacher raters at public schools. We also began quantitative analysis of data collected so far. Initial findings suggest that the L1 Japanese teachers of English rated the L1 Japanese L2 English speaker samples as low scoring for all categories aside from overall comprehensibility, but teaching experience did not influence L1 Japanese L2 English speaker sample ratings. However, for the L1 Thai L2 English samples, rater L1 only influenced segmental pronunciation and we observed a positive influence of teaching experience, with the L1 Japanese L2 English students rating samples significantly higher than their English teacher counterparts. In addition, both Japanese teachers and students report significantly lower confidence in their ability to rate English speech than English L1 or L2 English teachers in Japan. Furthermore, regarding evaluating specific aspects of speech, fluency was the easiest, while grammar and lexis were often the most challenging. For English L2 speakers, there was a lack of confidence in their own knowledge of correct grammar and vocabulary, while for English L1 speakers, the difficulty was in judging the complexity of the grammar and lexis accurately based on the sample length. In terms of vocabulary analysis, we observed that there was correlation between self-assessment vocabulary scores and some lexical measures, but no correlations were found for ratings by outside raters.