Research: Faculty Assessment Mental Models in Engineering Education

培养工程师的过程是一个迭代的过程，需要反馈来指示发展进度并确定需要改进的地方。反馈的主要来源是评估，它在工程教育中扮演着许多角色：向学生发出信号，告诉他们对一个概念理解的是什么，不理解的是什么；向教师反馈学生的概念理解，以及他们自己的教学方法可能有效或不有效的地方；并向管理人员和未来雇主提供评估学生能力的信息。尽管评估在工程师的形成中起着关键作用，但目前尚不清楚教员（即通常设计和实施这些评估的个人）如何考虑这一相关的信号机制。由于教职员工通常在课程决策方面拥有自主权，了解他们对评估的看法对于为未来在工程教育中推广多样化和改进的评估方法奠定基础至关重要。为了更好地了解教师是如何思考和做出评估决定的，我们设计了一个三阶段的研究，使用访谈、调查和自然语言处理技术，从不同的教师样本中收集大量数据，这些教师无疑对学生和评估有不同的看法。本研究的结果将包括表征教师评估的心理模型，以及这些模型如何为教学决策提供信息。在发展这些结果的过程中，我们还将识别评估实施中潜在的偏见、误解和有问题的系统性模式。通过该项目产生的知识将为更好的教师培训和政策提供信息，以促进这一重要领域的工程师培养。为了描述教师的心理模型，以及他们如何为评估决策提供信息，我们将进行一项分三个阶段、多方法的研究。在探索性访谈的基础上，第一阶段将绘制出教师可能与评估相关的心理模型的图景。第二阶段将借鉴最初的访谈，并增加经验抽样方法来扩展这一映射，将这些心理模型与教师在一个学期的课程评估中做出的决定联系起来。第三阶段将使用一项调查来扩大研究样本，以便更广泛地支持关于美国工程教育生态系统中与评估相关的心理模型和决策的工程教员人口的推断性陈述。这项研究将在智力价值方面做出几项重要贡献。首先，我们的研究将确定教师参与评估相关决策的一系列心理模型，提供对工程教育现状的看法。其次，在这些发现的基础上，我们将阐明这些核心教学决策与教师运用的更深层次观点之间的联系。第三，从方法论的角度来看，我们将实施结合经验抽样方法（ESM）和自然语言处理（NLP）的突破性方法，并为工程教育领域的其他研究人员提供一个模型。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The process of forming engineers is an iterative one that requires feedback to indicate developmental progress and identify areas for improvement. A primary source of feedback comes from assessment, which can play many roles in engineering education: a signal to students of what they do and do not understand about a concept; feedback to instructors about students’ conceptual understanding as well as what may or may not be working regarding their own teaching approaches; and information to administrators and prospective employers evaluating students’ abilities. Although assessments function as a linchpin in the formation of engineers, it is unclear how faculty members - i.e., the individuals typically designing and implementing these assessments - think about this pertinent signaling mechanism. Because faculty members often have autonomy in making course decisions, understanding how they think about assessments is essential to establish the foundation on future efforts in promoting diverse and improved assessment approaches in engineering education. To better understand how faculty think about and make decisions on assessment, we have designed a three-phase study that uses interviews, surveys, and natural language processing techniques to gather extensive data from a diverse sample of faculty who will undoubtedly have diverse views on students and assessment. The outcomes of this study will include characterizing faculty mental models of assessment and how those models inform instructional decisions. In developing these outcomes, we will also identify potential biases, misconceptions, and problematic, systemic patterns in assessment implementation. The knowledge generated through this project will inform better faculty training and policies to advance this vital area in the formation of engineers.To characterize faculty mental models and how they inform decision-making regarding assessment, we will engage in a three-phased, multi-method study. Drawing on exploratory interviews, Phase 1 will map the landscape of mental models that faculty members might have related to assessment. Phase 2 will draw on the initial interviews and add experience sampling methods to expand this mapping to connect those mental models with decisions made by faculty related to assessment over the course of a semester. Phase 3 will use a survey to expand the study sample in order to support inferential statements about the population of engineering faculty members in US engineering education ecosystems more broadly with regard to mental models and decisions related to assessment. This study will make several important contributions regarding intellectual merit. First, our study will identify a range of mental models that faculty engage in assessment-related decision-making, providing a view of the current state in engineering education. Second, building on these findings, we will illuminate the connections between these central instructional decisions and deeper perspectives that faculty members wield. Third, from a methodological perspective, we will implement ground-breaking methods that combine experience sampling methods (ESM) and natural language processing (NLP) and provide a model for other researchers in engineering education to do the same.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.