EAGER: Artificial Intelligence to Understand Engineering Cultural Norms

人工智能（AI）工具在各行各业的迅速采用表明，这些工具除了用于与学生互动外，还将用于生成有关工程教育和职业的内容。像ChatGPT这样的人工智能工具已经适应了多个行业，从真实的房地产到新闻业，相关的人工智能技术预计将对几乎每个行业产生深远的影响。这种前所未有的采用带来了无数的担忧，包括使用人工智能工具取代人类互动的影响，从而贬低人类互动。该项目将探索像ChatGPT这样的人工智能文本生成工具如何描绘工程中的压力和心理健康。通过研究如何描绘工程并将结果与先前记录的访谈进行比较，该项目将确定如何利用这些人工智能工具进行定性研究。了解工程文化规范和期望，特别是围绕压力和心理健康，对于解决长期存在的与群体代表性不足和学生保留率低有关的工程挑战至关重要。由于其探索性，该项目与EAGER计划相匹配，因为AI文本生成工具尚未用于数据生成，尽管它们在数据分析中的使用越来越多。此外，该项目具有潜在的变革性，有可能引入新的研究方法。最后，该项目将为更大的工程教育社区提供使用AI工具的培训和资源。该项目将探索使用大型语言模型（LLM）来生成定性数据集。该项目是探索性的，其目标是使用LLM作为研究工具，以了解与压力和心理健康相关的工程文化规范，并将通过工程文化框架进行了解。从LLM产生的结果将与来自定性数据集产生的关于工程文化的学生访谈的主题进行比较。随着开源LLM的快速增长，研究人员已经并将越来越多地继续使用LLM和其他AI工具进行研究。这就迫切需要确定最佳做法，以产生高质量的数据和分析。为了解决这个问题，我们的项目提出了一个整体的研究问题：LLM如何用于生成数据的定性工程教育研究研究工程文化？该研究将制定如何利用LLM为文化研究产生高质量数据集的指导方针，以丰富现有的定性方法并克服这些方法的局限性。该项目将专门研究工程文化中对心理健康的文化理解，并将LLM生成的数据集与工程文化中有关压力和心理健康的学生访谈进行比较。研究，以了解由法学硕士提炼的知识将有助于研究理解感知规范和假设的压力和心理健康在工程教育和职业生涯。在定性研究中利用LLM有可能为定性研究提供新的协同方法。该项目将为工程教育研究社区开发资源，以便在定性研究中使用人工智能。最终，拟议的工作将奠定基础，以了解如何LLM可以最好地用于定性EER和从事工程教育研究人员在人工智能革命。这个奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

The rapid adoption of artificial intelligence (AI) tools across industries demonstrates that these tools will be used to generate content about engineering education and careers, in addition to being used to interact with students. AI tools like ChatGPT have been adapted to several industries, from real estate to journalism, and related AI technologies are predicted to have a profound impact on nearly every industry. This unprecedented adoption brings a myriad of concerns, including the impact of using AI tools to replace and thus devalue human interactions. This project will explore how AI text generation tools like ChatGPT portray stress and mental health in engineering. By examining how engineering is portrayed and comparing results to previously recorded interviews, the project will identify how these AI tools can be leveraged for qualitative research studies. Understanding engineering cultural norms and expectations, particularly around stress and mental health, is critical in addressing longstanding challenges in engineering related to chronic underrepresentation of groups and low student retention. The project is a fit with the EAGER program due to its exploratory nature, as AI text generation tools have not been used for data generation, despite their increasing use in data analysis. Further, the project is potentially transformational with its potential to introduce new research methodologies. Finally, the project will provide training and resources to the larger engineering education community to use AI tools.This project will explore the use of large language models (LLMs) to generate qualitative datasets. The project is exploratory in its goal to use LLMs as a research tool to understand engineering cultural norms related to stress and mental health and will be informed by the Engineering Culture framework. Results generated from LLMs will be compared with themes derived from qualitative datasets generated from student interviews about engineering culture. With the rapidly increasing availability of open source LLMs, researchers have, and will increasingly continue, to use LLMs and other AI tools for research. This has created a critical and urgent need to determine best practices to yield high quality data and analyses. To address this, our project asks the overall research question: How can LLMs be used to generate data for qualitative engineering education research studying engineering culture? The study will develop guidelines for how LLMs can be leveraged to produce quality datasets for cultural studies that enrich existing qualitative methods and overcome limitations of these methods. The project will specifically examine cultural understandings of mental health in engineering culture and compare LLM-generated datasets to student interviews about stress and mental health in engineering culture. Research to understand the knowledge distilled by LLMs will contribute to research on understanding perceived norms and assumptions about stress and mental health in engineering education and careers. Leveraging LLMs in qualitative research has the potential to enable new and synergistic methods for qualitative research. The project will develop resources for the engineering education research community to use AI in qualitative research. Ultimately, the proposed work will lay the groundwork to understand how LLMs can be best used in qualitative EER and engage engineering education researchers in the AI Revolution.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.