Very-short-answer questions: reliability, discrimination and acceptability
Very-short-answer questions: reliability, discrimination and acceptability
复制标题
DOI：
10.1111/medu.13504
复制
发表时间：
2018-04-01
影响因子：
6
通讯作者：
Meeran, Karim
中科院分区：
文献类型：
--
作者：
Sam, Amir H.;Field, Samantha M.;Meeran, Karim
上下文单一最佳答案问题（SBAQ）已被广泛用于测试知识，因为它们易于标记并表现出高可靠性。然而，SBAQ 因受到提示的影响而受到批评。 目标我们使用了一种新颖的评估工具，可以促进开放式简答题 (VSAQ) 的高效评分。我们将 VSAQ 与 SBAQ 在可靠性、区分度和学生表现方面进行了比较，并评估了 VSAQ 的可接受性。方法医学生被随机分为以 VSAQ 和 SBAQ 格式（第 1 组，n = 155）或相反（第 2 组，n = 144）形式进行的 60 个问题评估。 VSAQ 通过平板电脑提供；答案经过计算机标记，随后由两名审查员进行审查。使用项目反应理论估计整个能力范围内的测量标准误差 (SEM)。结果所有学生的机器标记问题平均每个问题的复习时间为 1 分 36 秒。 VSAQ 具有高可靠性（α：0.91），SEM 显着低于 SBAQ（p < 0.001），并且平均项目-总点双列相关性较高（p < 0.001）。 VSAQ 分数显着低于 SBAQ 分数 (p < 0.001)。 VSAQ 和 SBAQ 之间的分数差异在第 2 组中有所减弱。尽管 80.4% 的学生认为 VSAQ 更难，但 69.2% 的学生认为它们更真实。结论 VSAQ 格式表现出高可靠性和辨别力，项目被认为更真实。 SBAQ 格式与显着的提示相关。目前的结果表明VSAQ格式具有较高的有效性。作者提出了一种新的简答题评分方法，并表现出较高的可靠性、辨别力和可接受性
ContextSingle-best-answer questions (SBAQs) have been widely used to test knowledge because they are easy to mark and demonstrate high reliability. However, SBAQs have been criticised for being subject to cueing.ObjectivesWe used a novel assessment tool that facilitates efficient marking of open-ended very-short-answer questions (VSAQs). We compared VSAQs with SBAQs with regard to reliability, discrimination and student performance, and evaluated the acceptability of VSAQs.MethodsMedical students were randomised to sit a 60-question assessment administered in either VSAQ and then SBAQ format (Group 1, n = 155) or the reverse (Group 2, n = 144). The VSAQs were delivered on a tablet; responses were computer-marked and subsequently reviewed by two examiners. The standard error of measurement (SEM) across the ability spectrum was estimated using item response theory.ResultsThe review of machine-marked questions took an average of 1 minute, 36 seconds per question for all students. The VSAQs had high reliability (alpha: 0.91), a significantly lower SEM than the SBAQs (p < 0.001) and higher mean item-total point biserial correlations (p < 0.001). The VSAQ scores were significantly lower than the SBAQ scores (p < 0.001). The difference in scores between VSAQs and SBAQs was attenuated in Group 2. Although 80.4% of students found the VSAQs more difficult, 69.2% found them more authentic.ConclusionsThe VSAQ format demonstrated high reliability and discrimination and items were perceived as more authentic. The SBAQ format was associated with significant cueing. The present results suggest the VSAQ format has a higher degree of validity.The authors present a novel method of marking short-answer questions and demonstrate high reliability, discrimination and acceptability