Development of knowledge level-estimating system for multiform multiple-choice questions

多种选择题知识水平评估系统的开发

• 批准号: 15590464
• 负责人: KIMURA Naofumi
• 金额: $ 0.7万
• 依托单位: Jikei University School of Medicine
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Scientific Research (C)
• 财政年份: 2003
• 资助国家: 日本
• 起止时间: 2003 至 2004
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-15590464/
• 关键词: multiple-choice questions; knowledge level; evaluation; expected correct answer probability; probability of correct answer; multiform MCQs; difficulty index; probability of incidental correct answer; MCQ

Multiple-choice questions(MCQs) have been extensively used in the national examinations for licensing physicians and other professionals, and this form of exam is often considered one of the most objective ways to test examinee's acquired knowledge. However, MCQs examinations have some problems of their own. One of the problems is that any question can be incidentally answered correctly at various rates, depending on the question frame (i.e.the number of choices, the number of correct answers, and the combinations of correct answers). In the case of MCQs such as choosing one or more correct answers out of multiple choices, the expected probability of a correct answer (expected P) is given as a multi-dimensional and multi-term equation for the knowledge level (q), which is the probability of discriminating correct and incorrect items. Applying this equation formula for various question types, we have developed a computer program which inversely estimates the q value (knowledge level) from the raw score rate (actual P value) in the examinations. This program is capable of treating up to three thousand multiform MCQs, which vary from choosing one correct answer out of 2-26 choices to choosing two correct answers out of 3-26 choices. To substantiate this program, we analyzed the results of authentic examinations that were used to determine college students' promotion. The analysis revealed that the mean values of the raw score rates fluctuated depending on the different types of MCQs ; however, the mean values of the estimated q values were almost constant independent of question types. Furthermore, the distribution curve of estimated q values also showed a similar pattern despite the difference in question types. This program makes it possible to estimate the actual level of examinees' knowledge, and at the same time, to evaluate the difficulty of MCQs, unaffected by their raw scores or question types.

多项选择题（MCQ）已被广泛用于全国执业医师和其他专业人员的考试中，这种考试形式通常被认为是测试考生所掌握知识的最客观方法之一。然而，MCQs考试本身也存在一些问题。其中一个问题是，任何问题都可以以不同的比率被附带地正确回答，这取决于问题框架（即，选择的数量、正确答案的数量以及正确答案的组合）。在MCQ的情况下，例如从多个选择中选择一个或多个正确答案，正确答案的期望概率（期望P）被给出为知识水平（q）的多维和多项方程，其是区分正确和不正确项目的概率。将此公式应用于各种题型，我们开发了一个计算机程序，可以从考试中的原始得分率（实际P值）反推q值（知识水平）。该程序能够处理多达3000种不同形式的MCQ，从2-26个选项中选择一个正确答案到3-26个选项中选择两个正确答案不等。为了证实这一计划，我们分析了用于确定大学生晋升的真实考试结果。分析表明，原始得分率的平均值波动取决于不同类型的MCQ，但是，估计的q值的平均值几乎是恒定的独立的问题类型。此外，尽管问题类型不同，估计q值的分布曲线也显示出类似的模式。该程序可以估计考生的实际知识水平，同时评估MCQ的难度，而不受原始分数或问题类型的影响。