Assessing Complex Collaborative STEM Learning at Scale with Epistemic Network Analysis

本项目是根据EHR核心研究（ECR）项目公告NSF 15-509提交的。STEM教育基础研究ECR项目为重要、广泛和持久的关键研究领域提供资金。EHR寻求有助于在以下重点领域综合、建立和/或扩大研究基础的提案：STEM学习、STEM学习环境、STEM劳动力发展和扩大STEM参与。ECR项目的特点是强调积累有力的证据，为以下方面的努力提供信息：(a)理解，(b)建立理论来解释，(c)提出干预（和创新）措施，以解决STEM兴趣、教育、学习和参与方面的持续挑战。在这个项目中，研究人员将开发一种统计分析技术来衡量人们如何学习。在NSF的资助下，团队成员创建了认知网络分析（ENA），这是一种在科学、技术、工程和数学领域创建复杂和协作思维网络模型的技术。来自19所大学的40多名研究人员正在使用ENA来回答学习科学、认知神经科学、工程教育、环境科学教育、医学和外科教育以及科学史等领域的广泛研究问题。拟议的研究和开发将创建一个在线工具包，允许研究人员上传音频、视频、文本或日志文件数据，自动转录音频数据，使用有监督的自然语言处理工具开发和验证自动代码，并生成ENA模型。这将使研究人员能够分析人们如何学习的数据，而不需要同时具备自动转录、数据分割、编码和网络建模方面的专业知识。它还将使利用在线学习工具目前产生的大量数据进行学习分析成为可能，从而大大扩展了学习研究的能力。在这个项目中，研究团队将通过开发和扩展一种基于理论的统计分析技术，利用网络分析来模拟复杂和协作的STEM思维（CCST），对科学、技术、工程和数学（STEM）的学习进行基础研究。在NSF的资助下，团队成员创建了认知网络分析（ENA），这是一种创建CCST动态模型的技术。CCST的数据通常有两种形式：教室和工作场所互动的视频和音频记录，或在线互动的日志文件。音频或视频形式的数据必须转录。必须对记录或日志文件中的文本数据进行编码或注释，以指示数据中存在哪些CCST元素，以及这些元素位于何处。使用现有的工具，研究人员必须手动完成这些步骤，或者使用自己的技术进行自动转录和编码。然而，许多熟练的CCST研究人员并不同时是自动转录、自动识别、验证和代码应用以及网络分析等科学方面的专家。拟议的研究和开发将创建ENAlysis，这是一个在线工具包，提供从原始数据到最终结果的无缝自动化分析管道。因此，该项目将开发创新的方法来衡量STEM学习，并扩大使用一种经过验证的、基于理论的技术来分析CCST。结果将扩大使用强大的学习分析技术来模拟CCST，这将显著改善对STEM思维和学习的评估，并为政策和实践提供信息。

This project was submitted in response to EHR Core Research (ECR) program announcement NSF 15-509. The ECR program of fundamental research in STEM education provides funding in critical research areas that are essential, broad and enduring. EHR seeks proposals that will help synthesize, build and/or expand research foundations in the following focal areas: STEM learning, STEM learning environments, STEM workforce development, and broadening participation in STEM. The ECR program is distinguished by its emphasis on the accumulation of robust evidence to inform efforts to (a) understand, (b) build theory to explain, and (c) suggest interventions (and innovations) to address persistent challenges in STEM interest, education, learning, and participation.In this project, researchers will develop a statistical analysis technique for measuring how people learn. With prior NSF funding, members of the team created epistemic network analysis (ENA), a technique for creating network models of complex and collaborative thinking in science, technology, engineering, and mathematics. ENA is being used by more than 40 researchers at 19 universities to answer a wide range of research questions in learning sciences, cognitive neuroscience, engineering education, environmental science education, medical and surgical education, and history of science. The proposed research and development will create an online toolkit that lets researchers upload audio, video, text, or log-file data, automatically transcribe the audio data, develop and validate automated codes using supervised natural language processing tools, and produce ENA models. This will make it possible for researchers analyze data on how people learn without requiring simultaneous expertise in automated transcription, data segmentation, coding, and network modeling. It will also make it possible to conduct analyses of learning using the large volumes of data that are currently generated by online learning tools, significantly expanding capacity for research on learning.In this project, the research team will conduct fundamental research on learning in science, technology, engineering, and mathematics (STEM) by developing and extending a theory-based statistical analysis technique for using network analysis to model complex and collaborative STEM thinking (CCST). With prior NSF funding, members of the team created epistemic network analysis (ENA), a technique for creating dynamic models of CCST. Data on CCST typically come in one of two forms: Video and audio recordings of interactions from classrooms and workplaces, or log files from online interactions. Data in audio or video form must be transcribed. Text data from recordings or log files have to be coded, or annotated to indicate what elements of CCST are present in the data, and where those elements are located. With existing tools, researchers must complete these steps by hand, or use their own techniques for automated transcription and coding. However, many skilled CCST researchers are not simultaneously experts in the sciences of automated transcription, automated identification, validation, and application of codes, and network analysis. The proposed research and development will create ENAlysis, an online toolkit that provides a seamless, automated analysis pipeline from raw data to final results. The project will thus develop innovative methods for measuring STEM learning and expand access to a proven, theory-based technique for analyzing CCST. The result will be expanded use of powerful learning analytic techniques to model CCST, which will significantly improve assessment of STEM thinking and learning and inform policy and practice.