EXP: Assessing 'Complex Epistemic Performance' in Online Learning Environments

“网络学习和未来学习技术计划”资助的工作旨在支持展望学习技术的未来，并推进我们对人们如何在技术丰富的环境中学习的了解。网络学习探索（EXP）项目设计和构建新的学习技术，以探索其可行性，了解有效使用它们所面临的挑战，并研究它们促进学习的潜力。该项目将开发在线软件工具，对学习者交流复杂的科学或技术信息进行评估并提供反馈。这里的“复杂认知表现”指的是报告或案例研究中的知识表现，不仅涉及可能产生正确答案的事实和理论概念，还涉及涉及学科或专业判断问题的论证、解释和结论。这些工具的开发和测试将通过医学和兽医学的临床案例研究进行。医学生将撰写对特定病例的分析，整理证据并根据这些证据做出诊断。同行们将提供“第二意见”，然后进行修改。学生将收到人工反馈和机器反馈的结合。该项目的主要技术创新将是开发和测试机器学习算法，为学习者提供有用的反馈，并支持教师评估。该软件将根据病例目标和临床评估标准分析学生创建的病例。它将逐项比较审查文本与标题标准。用户（学生和教师）分配的规则标准评级和总体评级将通过监督机器学习的过程来训练软件。通过这种方式，该软件将能够逐步对新文本做出更准确的评估，并评估同行评议的质量。该软件还将使用无监督的机器学习技术，突出显示可能需要调查的尚未分类的模式——换句话说，它将要求学生和教师解释他们可能感兴趣但可能没有注意到的反应模式。该项目的一个关键研究成果将是机器支持和机器介导的形成性评估过程是否以及如何改善科学和相关专业的学习成果，如大学水平的医学教育。在这个项目中开发的算法也可以在科学和技术努力的广泛领域具有更广泛的适用性。

The Cyberlearning and Future Learning Technologies Program funds efforts that support envisioning the future of learning technologies and advance what we know about how people learn in technology-rich environments. Cyberlearning Exploration (EXP) Projects design and build new kinds of learning technologies in order to explore their viability, to understand the challenges to using them effectively, and to study their potential for fostering learning. This project will develop online software tools to assess and offer feedback to learners communicating complex scientific or technical information. "Complex epistemic performance" here refers to knowledge representations in reports or case studies which involve not only facts and theoretical concepts that might produce correct answers but also arguments, interpretations and conclusions that are matters of disciplinary or professional judgment. Development and testing of these tools will take place using clinical case studies in medicine and veterinary medicine. Medical students will write analyses of specific cases of sick people and animals, marshaling evidence and making diagnoses based on this evidence. Peers will offer "second opinions", followed by revision. Students will receive a combination of human feedback and machine feedback.The principal technical innovation in this project will be the development and testing of machine learning algorithms that offer useful feedback to learners and support instructor assessment. The software will analyze student-created cases in relation to the case objectives and a clinical evaluation rubric. It will compare review text item by item to the rubric criteria. The rubric-criterion ratings and overall ratings assigned by users (students and instructors) will train the software via a process of supervised machine learning. In this way, the software will be able to make progressively more accurate assessments of new texts, as well as evaluate the quality of peer reviews. The software will also use unsupervised machine learning techniques, highlighting as-yet unclassified patterns that may warrant investigation -- in other words, it will ask students and instructors to interpret patterns of response that may be of interest but which they may not have noticed. One key research outcome of this project will be whether and how machine-supported and machine-mediated formative assessment processes improve learning outcomes in science and related professions, as exemplified in university-level medical education. The algorithms developed in this project could also have broader applicability in a wide range of areas of scientific and technical endeavor.