Capacity Assessment, Tracking, & Enhancement through Network Analysis: Developing a Tool to Inform Capacity Building Efforts in Complex STEM Education Systems

该研究与开发(R&A；D)项目旨在通过推动开发一种创新的工具和方法来记录和评估高等教育中的科学、技术、工程和数学(STEM)项目的能力建设，从而服务于国家利益。通过网络分析进行能力评估、跟踪和增强(Catena)工具基于系统方法来了解复杂的组织。五种能力模式为理解能力建设提供了一个框架，为假设、指标/措施的概念化、研究问题、数据收集、数据分析、解释、实施和结果的使用提供了信息。五种能力模式的层面包括系统行为者以下方面的能力：(A)承诺和参与；(B)执行任务和产生成果；(C)联系和吸引支持；(D)适应和自我更新；(E)平衡多样性和一致性。研发工作将社会网络分析(SNA)与其他定性信息相结合，以努力衡量教育系统的转型能力。混合方法研究旨在产生以下成果：(1)了解在教育中的互动和网络特征中表现出来的能力方面；(2)获取这些能力方面的工具；(3)以图论为基础的SNA数据收集和分析过程，以捕获紧急系统特征。该项目围绕三个阶段的研究和开发计划进行设计：第一阶段的目的是确定参与提高STEM教育能力的关键参与者之间的显著互动。第二阶段旨在将第一阶段的调查结果纳入调查发展。在第三阶段，项目组将在整个工程学院试行这项调查，并编制支持文件供其使用。评估和传播活动得到一个外部评估小组和一个外部机构改革、国民账户体系和质量方法专家咨询委员会的支持，该项目的目标是满足国家对评估和通报系统性、体制性变化的方法的需要，以便为高等教育的长期、可持续的变化提供持续的能力建设。根据总体目标，该项目将在研究的三个连续阶段调查三个研究问题：(RQ1)在五种能力模型定义的每种能力中，教职员工和研究生之间的哪种类型的互动可以促进STEM教育能力？(RQ2)如何通过设计Catena仪器来定量测量通过回答RQ1观察到的每种类型的交互作用？(RQ3)在测试在回答RQ2中创建的Catena工具后创建的网络中，哪些网络拓扑对评估STEM教育能力具有重要意义？研发工作整合了一系列方法的定量和定性数据，如社交网络分析、调查和访谈，以开发Catena工具，然后研究能力建设。度量的例子包括集群系数、平均路径长度、中间度中心度和度分布。NSF IUSE：EDU计划支持研究和开发项目，以提高所有学生的STEM教育的有效性。通过其机构和社区转型轨道，该项目支持高等教育机构和学科社区改造和改进STEM教育的努力。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This research and development (R&D) project aims to serve the national interest by advancing the development of an innovative tool and approach to document and assess capacity building in science, technology, engineering, and mathematics (STEM) programs in higher education. The Capacity Assessment, Tracking, and Enhancement through Network Analysis (CATENA) tool is based on a systems approach to understand complex organizations. The Five Capabilities Model of Capacity provides a framework for understanding capacity building that informs assumptions, conceptualization of metrics/measures, research questions, data collection, data analysis, interpretation, implementation, and use of findings. The dimensions of the Five Capacities Model include the capabilities of system actors to (a) commit and engage; (b) carry out tasks and produce results; (c) relate and attract support; (d) adapt and self-renew; and (e) balance diversity with coherence. The R&D effort integrates Social Network Analysis (SNA) with additional qualitative information in an effort to measure the capacity of an educational system for transformation. The mixed methods study is designed to yield the following outcomes: (1) An understanding of facets of capacity that are manifest in interactions and network features within education; (2) an instrument to capture these facets of capacity; and (3) a well-documented SNA data collection and analysis process rooted in graph theory to capture emergent system features. The project is designed around a three-phase research and development plan: The purpose of Phase 1 is to define salient interactions among key actors involved in advancing STEM educational capacity. Phase Two is designed to incorporate findings from Phase 1 into survey development. During Phase 3, the project team will pilot the survey across a College of Engineering and develop supporting documentation for its use. Evaluation and dissemination activities are supported by an external evaluation team and an external advisory board of experts in institutional change, SNA, and qualitative methods.The goal of the project is to address a national need for methodologies to evaluate and inform systemic, institutional change in a way that foregrounds continuous capacity building for long-term, sustainable change in higher education. In accordance with the overarching goal, the project will investigate three research questions across the three sequential phases of the study: (RQ1) What types of interactions among faculty, staff, and graduate students advance STEM education capacity among each of the capabilities defined by the Five Capabilities Model? (RQ2) How can each of the types of interaction observed through answering RQ1 be measured quantitatively through the design of the CATENA instrument? (RQ3) What network topologies emerge as significant for evaluating STEM education capacity in the networks created from piloting the CATENA instrument created in answering RQ2? The R&D effort integrates quantitative and qualitative data from a broad range of methodologies, such as social network analysis, surveys, and interviews to develop the CATENA tool and then to study capacity building. Examples of metrics include clustering coefficients, average path length, betweenness centrality, and degree distribution. The NSF IUSE: EDU Program supports research and development projects to improve the effectiveness of STEM education for all students. Through its Institutional and Community Transformation track, the project supports efforts to transform and improve STEM education across institutions of higher education and disciplinary communities.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.