Development of an Instructional Design Evaluation Survey for Postgraduate Medical E-Learning: Content Validation Study
Development of an Instructional Design Evaluation Survey for Postgraduate Medical E-Learning: Content Validation Study
复制标题
DOI：
10.2196/13921
复制
发表时间：
2019-08-09
影响因子：
7.4
通讯作者：
Scheele, Fedde
中科院分区：
文献类型：
--
作者：
de Leeuw, Robert Adrianus;Westerman, Michiel;Scheele, Fedde
背景：电子学习在研究生医学教育中已经占据了稳固的地位。虽然 10 年前它还很有前景，但现在它已经有了明确的定位，并且显然会继续存在。然而，评估研究生医学电子学习（PGMeL）的效果并对其进行改进可能很复杂。虽然电子学习的学习目标得到了评估，但没有工具可以评估 PGMeL 的教学设计。这样的评估工具可以按照欧洲医学教育协会（AMEE）的七步流程来开发。该过程的前 5 个步骤之前通过文献综述、焦点小组讨论和国际德尔菲研究进行。 目标：本研究将继续步骤 6 和 7，并回答研究问题：经过内容验证的 PGMeL 评估调查对电子学习的创建者有用、可理解且具有附加价值吗？方法：本研究分为五个阶段：从 37 个项目创建调查（A 阶段）；测试可读性和问题解释（B 阶段）；调整、重写和翻译调查（C 阶段）；收集三个 PGMeL 模块的已完成调查（D 阶段）；与电子学习作者举行焦点小组讨论（E 阶段）。 E 阶段首先介绍 D 阶段的评估结果，然后进行小组讨论。这项研究有四组参与者。 A 组和 B 组是有经验的 PGMeL 最终用户，参与了 B 阶段。C 组是进行电子学习并被要求在 D 阶段完成调查的用户。D 组是上述电子学习模块的作者。结果：我们从 36 个项目的列表中制定了研究生医学电子学习评估调查 (MEES)。 7 名住院医师参加了 B 期小组讨论：4 项解释不同，3 项不可读，2 项重复。 B阶段的项目被重写，经过调整，理解正确。 MEES 被翻译成荷兰语并再次进行试点测试。所有项目都很清楚并且被正确理解。用于评估的 MEES 版本包含 3 个积极领域（动机、学习增强剂和现实世界翻译）和 2 个消极领域（障碍和学习阻碍），这些领域中有 36 个项目，5 个 1 到 10 的李克特量表问题，以及 5 个要求参与者在每个领域给出自己的评论的开放性问题。 2018 年 7 月至 11 月对三个电子学习模块进行了评估。荷兰模块、欧洲 OB/GYN（妇产科）模块和全球提供的外科模块总共收到了 158 份回复。最后进行了3场焦点小组讨论，共有10人参加。实用性受到好评，可理解性好，附加值高。有四个项目需要作者额外解释，并根据他们的要求编写了创作者手册。结论：MEES 是第一个评估 PGMeL 教学设计的调查，是按照 AMEE 的所有 7 个步骤构建的。这项研究完成了调查的设计，并向作者展示了其有用性和附加价值。它以最终的公开调查结束，其中包括创作者手册。我们简要讨论了所需的回复数量，并得出结论：越多越好；然而，最终我们必须利用现有的资源。下一步将是看看是否可以通过使用 MEES 来衡量改进，并继续研究不同语言和文化群体的最终可理解性。
Background: E-Learning has taken a firm place in postgraduate medical education. Whereas 10 years ago it was promising, it now has a definite niche and is clearly here to stay. However, evaluating the effect of postgraduate medical e-learning (PGMeL) and improving upon it can be complicated. While the learning aims of e-learning are evaluated, there are no instruments to evaluate the instructional design of PGMeL. Such an evaluation instrument may be developed by following the Association for Medical Education in Europe (AMEE) 7-step process. The first 5 steps of this process were previously performed by literature reviews, focus group discussion, and an international Delphi study.Objective: This study will continue with steps 6 and 7 and answer the research question: Is a content-validated PGMeL evaluation survey useful, understandable, and of added value for creators of e-learning?Methods: There are five phases in this study: creating a survey from 37 items (phase A); testing readability and question interpretation (phase B); adjusting, rewriting, and translating surveys (phase C); gathering completed surveys from three PGMeL modules (phase D); and holding focus group discussions with the e-learning authors (phase E). Phase E was carried out by presenting the results of the evaluations from phase D, followed by a group discussion. There are four groups of participants in this study. Groups A and B are experienced end users of PGMeL and participated in phase B. Group C are users who undertook e-learning and were asked to complete the survey in phase D. Group D are the authors of the e-learning modules described above.Results: From a list of 36 items, we developed a postgraduate Medical E-Learning Evaluation Survey (MEES). Seven residents participated in the phase B group discussion: 4 items were interpreted differently, 3 were not readable, and 2 items were double. The items from phase B were rewritten and, after adjustment, understood correctly. The MEES was translated into Dutch and again pilot-tested. All items were clear and were understood correctly. The MEES version used for the evaluation contained 3 positive domains (motivation, learning enhancers, and real-world translation) and 2 negative domains (barriers and learning discouragers), with 36 items in those domains, 5 Likert scale questions of 1 to 10, and 5 open questions asking participants to give their own comments in each domain. Three e-learning modules were evaluated from July to November 2018. There were a total of 158 responses from a Dutch module, a European OB/GYN (obstetrics and gynecology) module, and a surgical module offered worldwide. Finally, 3 focus group discussions took place with a total of 10 participants. Usefulness was much appreciated, understandability was good, and added value was high. Four items needed additional explanation by the authors, and a Creators' Manual was written at their request.Conclusions: The MEES is the first survey to evaluate the instructional design of PGMeL and was constructed following all 7 steps of the AMEE. This study completes the design of the survey and shows its usefulness and added value to the authors. It finishes with a final, publicly available survey that includes a Creators' Manual. We briefly discuss the number of responses needed and conclude that more is better; in the end, however, one has to work with what is available. The next steps would be to see whether improvement can be measured by using the MEES and continue to work on the end understandability in different languages and cultural groups.