Evaluating STEM Scouts: The Design of a Comprehensive Evaluation Plan and Feasibility Study

推进非正式STEM学习（AISL）计划旨在推进非正式环境中STEM学习的设计和开发的新方法和基于证据的理解。非正式STEM教育面临的一个直接挑战是缺乏高质量的项目评估。高质量的评估是什么样子的，如何才能完成？高质量的评估必须从良好的评估计划开始。此外，尽管一些大型青年服务组织采用了STEM，但关于STEM结果与社会情绪学习（SEL）之间相互作用的研究有限。这个试点和可行性项目涉及STEM教育、校外时间（OST）项目、社会情感学习和评估方面的专业研究人员、从业者和项目开发人员之间的合作。STEM童子军的参与者被分成大约10-20个男孩和女孩的“实验室”，每个小组有3个或更多的领导者，并经常与当地学校合作。STEM童军的使命是帮助年轻人通过体验式活动和与STEM专业人士的互动来探索他们的好奇心，从而培养他们的品格和STEM技能。在先前nsf资助工作的基础上，该项目扩展了系统评估协议（SEP）的实施，SEP是一种基于证据的评估计划的三阶段方法，涉及(a)准备，(b)模型开发和(c)评估计划开发。该计划对其他青年服务机构也有启示，这些机构也肩负着支持STEM和青年社会情感发展的双重使命。该项目的目标是：(1)为STEM童子军发展一个详细的变化理论，强调STEM结果与积极青年发展/SEL结果之间的假设相互作用；(2)试点对SEP的三个新增强（系统映射、生态系统建模和模型验证）；(3)确定开展全国性STEM童子军研究的可行性。该项目将利用研究-实践伙伴关系进化评估方法和增强的SEP。由STEM童子军领导和项目研究人员组成的工作组将通过SEP的初始阶段工作，生成利益相关者地图、逻辑模型和路径模型（项目如何运作的因果描述）。有目的地从全国各地选出十个STEM童子军实验室，目标是多样化的代表，将参加焦点小组。使用由工作组生成的逻辑和路径模型，每个实验室将完成一个系统映射活动，以确定实验室存在的更大的系统（例如，利益相关者和决策者）。焦点小组参与者将审查和讨论他们的地图。通过生态系统建模，他们将确定最重要的结果，关键联系，以及他们可能对模型进行的修改，并讨论该模型如何反映他们的STEM童子军经验。将编写一份编码词典来分析焦点小组的记录；研究结果将为模型验证提供信息，并将用于评估是否需要对增强的SEP进行更改。假设增强的SEP将使工作组能够更好地了解阻碍或支持计划的因素，并评估可行性和成功。项目结果将传播给评估者和研究界、OST项目提供者/开发者和公众。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The Advancing Informal STEM Learning (AISL) program seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. An immediate challenge for informal STEM education is the lack of high quality program evaluation. What does high-quality evaluation look like and how can it be accomplished? High-quality evaluation necessarily begins with good evaluation planning. In addition, although several large-scale youth-serving organizations embrace STEM, there is limited research on the interaction between STEM outcomes and social-emotional learning (SEL). This pilot and feasibility project involves a collaboration between researchers with expertise in STEM education, out-of-school time (OST) programs, social-emotional learning, and evaluation, as well as practitioners and program developers. STEM Scouts participants are grouped into "labs" of approximately 10-20 boys and girls with 3 or more leaders per group and often collaborate with local schools. The mission of STEM Scouts is to help young people grow in character and STEM skills as they explore their curiosity using experiential activities and interaction with STEM professionals. Building on prior NSF-funded work, this project expands the implementation of the Systems Evaluation Protocol (SEP), a three-stage evidence-based approach to evaluation planning involving (a) Preparation, (b) Model Development, and (c) Evaluation Plan Development. The proposed project has implications for other youth-serving organizations that also embrace a dual mission of supporting STEM and the social-emotional development of youth.The goals of this project are to: (1) develop a detailed theory of change for STEM Scouts that highlights the hypothesized interactions between STEM outcomes and Positive Youth Development/SEL outcomes; (2) pilot three new enhancements to the SEP (System Mapping, Ecosystem Modeling, and Model Validation); and (3) determine the feasibility of conducting a national scale STEM Scouts study. The project will utilize a research-practice partnership Evolutionary Evaluation approach and the enhanced SEP. A working group of STEM Scouts leadership and project researchers will work through initial stages of the SEP to generate a stakeholder map, logic model and pathway model (a causal depiction of how the program works). Ten STEM Scout Labs purposively selected from across the country with the goal of diverse representation will participate in focus groups. Using logic and pathway models generated by the working group, each Lab will complete a System Mapping activity to identify the larger system in which the Lab exists (e.g., stakeholders and decision-makers). Focus group participants will review and discuss their maps. Through Ecosystem Modeling they will identify the most important outcomes, key connections, and revisions they might make to their model, and discuss how the model reflects their STEM Scouts experiences. A coding dictionary will be developed to analyze focus group transcripts; findings will inform Model Verification and will be used to assess whether changes need to be made to the enhanced SEP. It is hypothesized the enhanced SEP will enable the working group to better understand factors hindering or enabling program and evaluation feasibility and success. Project findings will be disseminated to the evaluator and research community, OST program providers/developers and the public.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.