EAGER: Natural Language Processing for Teaching and Research in Engineering Education

EAGER：用于工程教育教学和研究的自然语言处理

• 批准号: 2107008
• 负责人: Andrew Katz
• 金额: $ 29.96万
• 依托单位: Virginia Polytechnic Institute and State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-03-01 至 2025-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2107008&HistoricalAwards=false
• 关键词: EAGER; Natural; Language; Processing; Teaching

In ecosystems that form professional engineers, community members produce text through many activities such as end-of-semester feedback to instructors, transcripts of instruction, open-ended survey items, and interviews. In each case, there is abundant text available to educators and researchers that could provide insight into how we form engineers. Unfortunately, while these texts have the potential to provide novel insights, traditional analytic techniques do not scale well. Time investments, bias, interrater reliability, and intrarater reliability each present significant challenges. To address this problem, we aim to develop and characterize approaches for human-in-the-loop (HITL) natural language processing (NLP) systems to augment human analysis, facilitating and enhancing the work of one person (or team). Such systems can help reduce the amount of time needed to analyze texts by grouping similar texts together. The human user can utilize these groupings for further analysis and identify meanings in ways only a human could. The system will also improve consistency by analyzing across the entire collection of texts simultaneously and grouping similar items together. This is in contrast with a single person or a team that would analyze responses sequentially, creating the potential for inconsistencies across time. We will accomplish this work in three phases. In Phase 1, we will conduct a series of experiments to test potential system configurations. The goal will be to identify optimal components and parameter settings for four of the steps in the proposed pipeline. We will use datasets from (i) students’ written responses to an instrument for assessing their systems thinking and (ii) students’ responses to open-ended course feedback surveys. We will measure performance based on consistency of thematic clusters, using standard metrics for homogeneity in text clustering and classification tasks. In Phase 2, we will study system performance on a series of five datasets. These datasets will come from multiple sources: extant NSF-funded projects, longitudinal data from the Virginia Tech College of Engineering, current data in engineering courses, and freshly collected data from online outlets. These represent important areas of the broader ecosystem that supports how we form future engineers. We will test the system for thematic clusters, employing similar metrics as in Phase 1 to identify potential inconsistencies in how different datasets are handled. We will specifically look for homogeneity of texts within a cluster and shared semantic meaning. We will also update the original system designs in the event of systematic differences (e.g., longer texts require a different system configuration). For Phase 3, we will study how it can affect human performance. Since we anticipate significant improvements in human efficiency and consistency, it is important to conduct analyses that can accurately assess the veracity of that proposition. These studies will assess the HITL aspect of this process since many relevant applications of the system will require additional interpretation of the raw output. To accomplish this, we will collect data on differences in human performance when analyzing 1,500 student responses with and without the system’s assistance. We will look at differences when (a) one person alone codes the data and when (b) a team of three researchers codes the data (i.e., we will have two studies: one person with vs one person without and team with vs team without). We will measure differences in coding (whether different themes emerge), reliability (how consistently similar texts are grouped together), time needed to code the data, and differential treatment of student responses associated with student group characteristics. We will host all code on public repositories and notebooks for easy access, copying, and application by other engineering education researchers and teachers along with any new datasets, where appropriate.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在形成专业工程师的生态系统中，社区成员通过许多活动产生文本，例如学期末对讲师的反馈、教学记录、开放式调查项目和访谈。在每种情况下，教育工作者和研究人员都可以获得大量的文本，这些文本可以为我们如何形成工程师提供见解。不幸的是，虽然这些文本有可能提供新颖的见解，但传统的分析技术并不能很好地扩展。时间投入、偏差、判读器间可靠性和判读器内部可靠性都是重大的挑战。为了解决这个问题，我们的目标是开发和表征人类在环（HITL）自然语言处理（NLP）系统的方法，以增强人类分析，促进和增强一个人（或团队）的工作。这样的系统可以通过将相似的文本分组在一起来帮助减少分析文本所需的时间。人类用户可以利用这些分组进行进一步分析，并以只有人类才能做到的方式识别含义。该系统还将通过同时分析整个文本集合并将相似的项目分组在一起来提高一致性。这与一个人或一个团队将依次分析响应形成对比，这会在时间上产生不一致的可能性。我们将分三个阶段完成这项工作。在第一阶段，我们将进行一系列的实验来测试潜在的系统配置。目标是为拟议管道中的四个步骤确定最佳组件和参数设置。我们将使用以下数据集：(i)学生对评估其系统思维的工具的书面回复，以及（ii）学生对开放式课程反馈调查的回复。我们将基于主题聚类的一致性来衡量性能，在文本聚类和分类任务中使用标准的同质性指标。在第二阶段，我们将在一系列五个数据集上研究系统性能。这些数据集将来自多个来源：现有的nsf资助项目、弗吉尼亚理工大学工程学院的纵向数据、工程课程的当前数据，以及从在线渠道收集的最新数据。这些代表了支持我们如何培养未来工程师的更广泛生态系统的重要领域。我们将为主题集群测试系统，采用与第一阶段类似的指标来识别不同数据集处理方式的潜在不一致性。我们将特别寻找一个集群内文本的同质性和共享的语义。如果系统存在差异（例如，较长的文本需要不同的系统配置），我们也将更新原始系统设计。在第三阶段，我们将研究它如何影响人类的表现。由于我们期望在人类效率和一致性方面有重大的改进，因此进行能够准确评估该命题的准确性的分析是很重要的。这些研究将评估这一过程的HITL方面，因为该系统的许多有关应用将需要对原始输出进行额外的解释。为了实现这一目标，我们将在分析1500名学生在有和没有系统帮助的情况下的反应时，收集有关人类表现差异的数据。当(a)一个人单独编码数据和(b)一个由三个研究人员组成的团队编码数据时，我们将查看差异（即，我们将有两个研究：一个人与一个人没有，团队与团队没有）。我们将衡量编码方面的差异（是否出现不同的主题）、可靠性（将相似的文本分组在一起的一致性如何）、编码数据所需的时间，以及与学生群体特征相关的学生回答的差异处理。我们将在公共存储库和笔记本上托管所有代码，以便其他工程教育研究人员和教师在适当的情况下轻松访问，复制和应用任何新的数据集。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

WIP: Faculty Use of Metaphors When Discussing Assessment

WIP：教师在讨论评估时使用隐喻

• 发表时间: 2023
• 期刊: Annual Conference and Exposition of the American Society for Engineering Education
• 作者: Ross, A;Katz, A;Matusovich, H;Chew, K.
• 通讯作者: Chew, K.

Students’ Feedback About Their Experiences in EPICS Using Natural Language Processing

• DOI: 10.1109/fie56618.2022.9962557
• 发表时间: 2022-10
• 期刊: 2022 IEEE Frontiers in Education Conference (FIE)
• 作者: Isil Anakok;J. Woods;Mark V. Huerta;Jared Schoepf;H. Murzi;Andrew Katz

Exploring the Impact of Engineering Projects in Community Service on Students' Perspectives About Engineering as a Major

探索社区服务中的工程项目对学生对工程专业的看法的影响

• 发表时间: 2023
• 期刊: Frontiers in education
• 影响因子: 2.3
• 作者: Anakok, I;Huerta, M;Katz, A.
• 通讯作者: Katz, A.

Board 65: Work in Progress: Using Natural Language Processing to Facilitate Scoring of Scenario-Based Assessments

Board 65：正在进行的工作：使用自然语言处理促进基于场景的评估评分

• 发表时间: 2023
• 期刊: Annual Conference and Exposition of the American Society for Engineering Education
• 作者: Norris, M;Taimoory, H;Katz, A;Grohs, J.
• 通讯作者: Grohs, J.

Understanding First-year Engineering Students’ Perceptions of Working with Real Stakeholders on a Design Project: A PBL Approach

了解一年级工科学生对与真正的利益相关者合作设计项目的看法：PBL 方法

• DOI: 10.1109/fie56618.2022.9962395
• 发表时间: 2022
• 期刊: IEEE Frontiers in Education Conference
• 作者: Murzi, Homero;Fielding, Lydia;Huerta, Mark;Ortega Alvarez, Juan D.;James, Matthew;Katz, Andrew;Grohs, Jacob
• 通讯作者: Grohs, Jacob