Research: Faculty Assessment Mental Models in Engineering Education

形成工程师的过程是一个迭代的过程，需要反馈来表明发展进展并确定需要改进的领域。反馈的主要来源来自评估，评估可以在工程教育中发挥多种作用：向学生发出信号，表明他们对一个概念的理解；向教师反馈学生对概念的理解，以及关于他们自己的教学方法，哪些可能奏效，哪些可能不起作用；以及向管理人员和潜在雇主提供评估学生能力的信息。尽管评估在工程师的形成中起着关键作用，但尚不清楚教职员工--即通常设计和实施这些评估的个人--是如何看待这一相关的信号机制的。由于教职员工通常在制定课程决策方面拥有自主权，因此了解他们对评估的看法对于为未来在工程教育中促进多样化和改进评估方法的努力奠定基础是至关重要的。为了更好地了解教师如何思考和做出评估决策，我们设计了一个分三个阶段的研究，使用访谈、调查和自然语言处理技术从不同的教师样本中收集广泛的数据，这些教师无疑会对学生和评估有不同的看法。这项研究的结果将包括表征教师的评估心理模型，以及这些模型如何影响教学决策。在制定这些结果时，我们还将确定评估实施中的潜在偏见、误解和有问题的系统性模式。通过这个项目产生的知识将为更好的教师培训和政策提供信息，以促进这一重要领域的工程师的形成。为了表征教师心理模型及其如何为评估决策提供信息，我们将进行一项分三个阶段、多方法的研究。在探索性访谈的基础上，第一阶段将绘制出教职员工可能与评估相关的心理模型的图景。第二阶段将利用最初的访谈，并增加经验抽样方法，以扩展这种映射，将这些心理模型与教师在一学期课程中做出的与评估有关的决策联系起来。第三阶段将使用一项调查来扩大研究样本，以支持关于美国工程教育生态系统中更广泛的关于与评估相关的心理模型和决策的工程学教师人数的推论。这项研究将在智力价值方面做出几项重要贡献。首先，我们的研究将确定教师参与与评估相关的决策的一系列心理模型，为工程教育的现状提供一个视角。其次，在这些发现的基础上，我们将阐明这些核心教学决策与教职员工拥有的更深层次观点之间的联系。第三，从方法论的角度来看，我们将实施将经验抽样方法(ESM)和自然语言处理(NLP)相结合的开创性方法，并为工程教育中的其他研究人员提供同样的模式。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The process of forming engineers is an iterative one that requires feedback to indicate developmental progress and identify areas for improvement. A primary source of feedback comes from assessment, which can play many roles in engineering education: a signal to students of what they do and do not understand about a concept; feedback to instructors about students’ conceptual understanding as well as what may or may not be working regarding their own teaching approaches; and information to administrators and prospective employers evaluating students’ abilities. Although assessments function as a linchpin in the formation of engineers, it is unclear how faculty members - i.e., the individuals typically designing and implementing these assessments - think about this pertinent signaling mechanism. Because faculty members often have autonomy in making course decisions, understanding how they think about assessments is essential to establish the foundation on future efforts in promoting diverse and improved assessment approaches in engineering education. To better understand how faculty think about and make decisions on assessment, we have designed a three-phase study that uses interviews, surveys, and natural language processing techniques to gather extensive data from a diverse sample of faculty who will undoubtedly have diverse views on students and assessment. The outcomes of this study will include characterizing faculty mental models of assessment and how those models inform instructional decisions. In developing these outcomes, we will also identify potential biases, misconceptions, and problematic, systemic patterns in assessment implementation. The knowledge generated through this project will inform better faculty training and policies to advance this vital area in the formation of engineers.To characterize faculty mental models and how they inform decision-making regarding assessment, we will engage in a three-phased, multi-method study. Drawing on exploratory interviews, Phase 1 will map the landscape of mental models that faculty members might have related to assessment. Phase 2 will draw on the initial interviews and add experience sampling methods to expand this mapping to connect those mental models with decisions made by faculty related to assessment over the course of a semester. Phase 3 will use a survey to expand the study sample in order to support inferential statements about the population of engineering faculty members in US engineering education ecosystems more broadly with regard to mental models and decisions related to assessment. This study will make several important contributions regarding intellectual merit. First, our study will identify a range of mental models that faculty engage in assessment-related decision-making, providing a view of the current state in engineering education. Second, building on these findings, we will illuminate the connections between these central instructional decisions and deeper perspectives that faculty members wield. Third, from a methodological perspective, we will implement ground-breaking methods that combine experience sampling methods (ESM) and natural language processing (NLP) and provide a model for other researchers in engineering education to do the same.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.