EXP: Assessing 'Complex Epistemic Performance' in Online Learning Environments

网络学习和未来学习技术计划为支持展望学习技术的未来并推动我们了解人们如何在技术丰富的环境中学习的努力提供资金。网络学习探索(EXP)项目设计和构建新的学习技术，以探索它们的生存能力，了解有效使用它们的挑战，并研究它们促进学习的潜力。该项目将开发在线软件工具，以评估并向交流复杂科学或技术信息的学习者提供反馈。这里的“复杂认知表现”是指报告或案例研究中的知识表征，它不仅涉及可能产生正确答案的事实和理论概念，而且还涉及属于纪律或专业判断问题的论点、解释和结论。将利用医学和兽医学的临床案例研究来开发和测试这些工具。医学生将撰写具体的人和动物患病案例的分析，整理证据，并根据这些证据做出诊断。同行将提出“第二意见”，然后进行修改。学生将得到人类反馈和机器反馈的结合。这个项目的主要技术创新将是开发和测试机器学习算法，为学习者提供有用的反馈并支持教师评估。该软件将根据病例目标和临床评估标准分析学生创建的病例。它将逐项将审查文本与评分标准进行比较。由用户(学生和教师)分配的标准评分和总体评分将通过有监督的机器学习过程来训练软件。通过这种方式，该软件将能够对新文本进行逐步更准确的评估，并评价同行评议的质量。该软件还将使用无监督的机器学习技术，突出显示可能需要调查的尚未分类的模式--换句话说，它将要求学生和教师解释可能感兴趣但他们可能没有注意到的反应模式。该项目的一个关键研究成果将是机器支持和机器中介的形成性评估过程是否以及如何改善科学和相关专业的学习结果，如大学水平的医学教育。在这个项目中开发的算法还可以在科学和技术努力的广泛领域中具有更广泛的适用性。

The Cyberlearning and Future Learning Technologies Program funds efforts that support envisioning the future of learning technologies and advance what we know about how people learn in technology-rich environments. Cyberlearning Exploration (EXP) Projects design and build new kinds of learning technologies in order to explore their viability, to understand the challenges to using them effectively, and to study their potential for fostering learning. This project will develop online software tools to assess and offer feedback to learners communicating complex scientific or technical information. "Complex epistemic performance" here refers to knowledge representations in reports or case studies which involve not only facts and theoretical concepts that might produce correct answers but also arguments, interpretations and conclusions that are matters of disciplinary or professional judgment. Development and testing of these tools will take place using clinical case studies in medicine and veterinary medicine. Medical students will write analyses of specific cases of sick people and animals, marshaling evidence and making diagnoses based on this evidence. Peers will offer "second opinions", followed by revision. Students will receive a combination of human feedback and machine feedback.The principal technical innovation in this project will be the development and testing of machine learning algorithms that offer useful feedback to learners and support instructor assessment. The software will analyze student-created cases in relation to the case objectives and a clinical evaluation rubric. It will compare review text item by item to the rubric criteria. The rubric-criterion ratings and overall ratings assigned by users (students and instructors) will train the software via a process of supervised machine learning. In this way, the software will be able to make progressively more accurate assessments of new texts, as well as evaluate the quality of peer reviews. The software will also use unsupervised machine learning techniques, highlighting as-yet unclassified patterns that may warrant investigation -- in other words, it will ask students and instructors to interpret patterns of response that may be of interest but which they may not have noticed. One key research outcome of this project will be whether and how machine-supported and machine-mediated formative assessment processes improve learning outcomes in science and related professions, as exemplified in university-level medical education. The algorithms developed in this project could also have broader applicability in a wide range of areas of scientific and technical endeavor.