Research: Faculty Assessment Mental Models in Engineering Education

形成工程师的过程是一个反复的过程，需要反馈来指示开发进度并确定需要改进的领域。反馈的主要来源来自评估，它可以在工程教育中发挥许多作用：向学生发出他们对概念的理解和不理解的信号;向教师反馈学生的概念理解以及他们自己的教学方法可能或可能不起作用;以及向管理人员和潜在雇主提供评估学生能力的信息。 虽然评估是工程师培养的关键，但目前还不清楚教师是如何工作的，通常设计和实施这些评估的个人-考虑这种相关的信号机制。由于教师往往有自主权，在作出课程的决定，了解他们如何看待评估是必不可少的，以建立在未来的努力，促进多样化和改进的评估方法在工程教育的基础。为了更好地了解教师如何思考和做出评估决策，我们设计了一个三阶段的研究，使用访谈，调查和自然语言处理技术，从不同的教师样本中收集大量数据，这些教师无疑对学生和评估有不同的看法。 本研究的结果将包括教师心理模型的评估和这些模型如何告知教学决策的特点。 在开发这些成果的过程中，我们还将识别评估实施中的潜在偏见、误解和有问题的系统模式。通过这个项目产生的知识将为更好的教师培训和政策提供信息，以推进工程师形成中的这一重要领域。为了表征教师的心理模型以及它们如何为评估决策提供信息，我们将进行三阶段，多方法的研究。 利用探索性访谈，第一阶段将绘制教师可能与评估相关的心理模型的景观。第二阶段将利用最初的访谈，并增加经验抽样方法，以扩大这种映射，将这些心理模型与教师在一个学期的课程中做出的与评估相关的决定联系起来。第3阶段将使用一项调查，以扩大研究样本，以支持推理声明的人口工程教师在美国工程教育生态系统更广泛的心理模型和决策有关的评估。这项研究将在智力价值方面做出几项重要贡献。 首先，我们的研究将确定一系列的心理模型，教师从事评估相关的决策，提供了一个在工程教育的现状。其次，在这些发现的基础上，我们将阐明这些中央教学决策和教师运用的更深层次的观点之间的联系。第三，从方法论的角度来看，我们将实施结合联合收割机经验抽样方法（ESM）和自然语言处理（NLP）的突破性方法，并为工程教育领域的其他研究人员提供一个模式。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

The process of forming engineers is an iterative one that requires feedback to indicate developmental progress and identify areas for improvement. A primary source of feedback comes from assessment, which can play many roles in engineering education: a signal to students of what they do and do not understand about a concept; feedback to instructors about students’ conceptual understanding as well as what may or may not be working regarding their own teaching approaches; and information to administrators and prospective employers evaluating students’ abilities. Although assessments function as a linchpin in the formation of engineers, it is unclear how faculty members - i.e., the individuals typically designing and implementing these assessments - think about this pertinent signaling mechanism. Because faculty members often have autonomy in making course decisions, understanding how they think about assessments is essential to establish the foundation on future efforts in promoting diverse and improved assessment approaches in engineering education. To better understand how faculty think about and make decisions on assessment, we have designed a three-phase study that uses interviews, surveys, and natural language processing techniques to gather extensive data from a diverse sample of faculty who will undoubtedly have diverse views on students and assessment. The outcomes of this study will include characterizing faculty mental models of assessment and how those models inform instructional decisions. In developing these outcomes, we will also identify potential biases, misconceptions, and problematic, systemic patterns in assessment implementation. The knowledge generated through this project will inform better faculty training and policies to advance this vital area in the formation of engineers.To characterize faculty mental models and how they inform decision-making regarding assessment, we will engage in a three-phased, multi-method study. Drawing on exploratory interviews, Phase 1 will map the landscape of mental models that faculty members might have related to assessment. Phase 2 will draw on the initial interviews and add experience sampling methods to expand this mapping to connect those mental models with decisions made by faculty related to assessment over the course of a semester. Phase 3 will use a survey to expand the study sample in order to support inferential statements about the population of engineering faculty members in US engineering education ecosystems more broadly with regard to mental models and decisions related to assessment. This study will make several important contributions regarding intellectual merit. First, our study will identify a range of mental models that faculty engage in assessment-related decision-making, providing a view of the current state in engineering education. Second, building on these findings, we will illuminate the connections between these central instructional decisions and deeper perspectives that faculty members wield. Third, from a methodological perspective, we will implement ground-breaking methods that combine experience sampling methods (ESM) and natural language processing (NLP) and provide a model for other researchers in engineering education to do the same.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.