Investigating the extent to which speech rater judgement relates to objective speech measures

2022年，我们完成了从日本评分员那里收集数据的工作，并向加拿大公立学校教师评分员提交了收集反馈的申请。我们也开始对目前收集到的数据进行定量分析。最初的研究结果表明，除了整体可理解性外，L1日语英语教师认为L1日语L2英语说话者样本在所有类别中得分都很低，但教学经验并不影响L1日语L2英语说话者样本的评分。然而，对于母语为泰国语的L2英语样本来说，较好的L1只影响分段发音，我们观察到教学经验的积极影响，母语为日本语的L2英语学生的评分样本显著高于他们的英语教师。此外，日本教师和学生都报告说，他们对英语演讲能力的信心明显低于日本的英语一级或二级英语教师。此外，在评估语言的具体方面时，流利度是最容易的，而语法和词汇往往是最具挑战性的。对于第二语言的人来说，他们对自己掌握正确的语法和词汇缺乏信心，而对于第一语言的人来说，困难在于根据样本长度准确判断语法和词汇的复杂程度。在词汇分析方面，我们观察到自我评价词汇得分与某些词汇测量指标之间存在相关性，但与外部评分者的评分之间没有相关性。

In 2022 we concluded collecting data from Japan-based raters and submitted applications for permission to collect responses from Canada-based teacher raters at public schools. We also began quantitative analysis of data collected so far. Initial findings suggest that the L1 Japanese teachers of English rated the L1 Japanese L2 English speaker samples as low scoring for all categories aside from overall comprehensibility, but teaching experience did not influence L1 Japanese L2 English speaker sample ratings. However, for the L1 Thai L2 English samples, rater L1 only influenced segmental pronunciation and we observed a positive influence of teaching experience, with the L1 Japanese L2 English students rating samples significantly higher than their English teacher counterparts. In addition, both Japanese teachers and students report significantly lower confidence in their ability to rate English speech than English L1 or L2 English teachers in Japan. Furthermore, regarding evaluating specific aspects of speech, fluency was the easiest, while grammar and lexis were often the most challenging. For English L2 speakers, there was a lack of confidence in their own knowledge of correct grammar and vocabulary, while for English L1 speakers, the difficulty was in judging the complexity of the grammar and lexis accurately based on the sample length. In terms of vocabulary analysis, we observed that there was correlation between self-assessment vocabulary scores and some lexical measures, but no correlations were found for ratings by outside raters.