Strategic Outcome Progressions Conference: Exploring a Framework for Measuring Informal Education Outcomes and Institutional Impact

存在各种措施来评估非正式科学、技术、工程或数学（STEM）教育规划的学习成果，特别是针对青年受众。尽管从业人员、研究人员和评估人员对开发和实施共享的、通用的STEM措施和框架越来越感兴趣，但非正式的STEM学习（ISL）机构（如博物馆、动物园、水族馆、植物园等）在设计和评估教育计划时面临着独特的挑战。由于这些机构所强调的主题范围广泛，节目设计力求达到年龄、能力、文化背景、兴趣等各不相同的观众。参与程度也各不相同，例如，参与者可以选择参加哪些展览，以及在每个展览上花多长时间。考虑到这种计划的差异，教育者如何以适当的学习成果为目标？结果在多大程度上反映了活动的复杂性？规划的成功如何与机构目标相联系？一个解决方案可能是建立一个成果进展框架。“进展”是在结果类别（例如，兴趣，态度，知识，技能，行为，科学资本等）中，可以映射到编程的顺序系列效应，范围从短时间的简单内容（例如，走廊推车演示可以触发情境兴趣）到长时间的复杂内容（例如，青年实习可以产生良好的个人兴趣）。STEM教育工作者、评估人员和研究人员将在ISL机构工作或与ISL机构合作，共同考虑成果进展框架的效用和可行性。有了这个框架，STEM学习机构的教育工作者可能会更好地理解他们的编程的影响，以及如何为此进行设计。这项工作将由COSI研究和评估中心与国家自然历史博物馆的教育、推广和游客服务办公室以及PEAR研究所共同承担。这次为期数天的虚拟会议的主要目的是为成果进展框架的未来发展提供信息。这些进展将反映和适应现有的和共同的评价措施所针对的结果。该框架将为非正式教育工作者和评估人员提供(1)一种使结果预期与规划深度和剂量相一致的方法；(2)不同深度编程可实现的结果选择；(3)在整个机构教育规划的全部组合中规划全校范围内的目标成果的战略工具；(4)一个记录教育部门在所有可能结果范围内的不同结果的工具。为了创造丰富和知情的批判性讨论，会议将包括代表每个成果类别的研究顾问专家；代表不同节目的从业人员；评估理论家熟悉非正式学习机构的广泛需求。一个在文化响应和公平的项目制定和评估方面具有专业知识的会议小组委员会将确保议程具有灵活性，以产生紧急的公平和包容的对话。与会者将审查和促进这些分类的合理性和实用性以及其中的进展。这些见解和输入将为框架作为有效和可靠的工具的未来开发提供信息。可交付成果包括一份总结会议结果的论文和一个描述框架及其使用建议的相关网络研讨会。这些成果将通过informalscience.org网站和其他专业场所广泛分享。本次会议由推进非正式STEM学习（AISL）计划资助，该计划旨在推进非正式环境中STEM学习的设计和开发的新方法和基于证据的理解。这包括提供多种途径，扩大STEM学习经验的获取和参与，推进非正式环境中STEM学习的创新研究和评估，以及发展参与者对深度学习的理解。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Various measures exist to assess learning outcomes from informal science, technology, engineering, or math (STEM) education programming, especially for youth audiences. Although there has been growing interest in the field among practitioners, researchers and evaluators for the development and implementation of shared, common STEM measures and frameworks, informal STEM learning (ISL) institutions (e.g., museums, zoos, aquaria, botanic gardens, and the like) face a unique challenge when they design and evaluate educational programs. By nature of the broad-ranging topic areas these institutions emphasize, programming seeks to reach audiences that vary in age, ability, cultural background, interest, and more. Participation also varies--for example, participants choose which in a series of exhibits they attend and how much time they spend at each. Given this planned-for variance, how do educators target appropriate learning outcomes? To what extent do outcomes map to activity complexity? How does programming success link to institutional goals? One solution may be a framework of outcome progressions. “Progressions” are within outcome categories (e.g., interest, attitude, knowledge, skills, behavior, science capital, etc.), the ordinal series of effects that can be mapped to programming that ranges from short duration with simple content (e.g. a hallway cart demonstration can trigger situational interest) to extended duration with complex content (e.g. a youth internship can generate well-developed personal interest). STEM educators, evaluators and researchers working in or with ISL institutions will come together to consider the utility and feasibility of an outcome progressions framework. With this framework STEM learning institution educators may be better equipped to understand the impact of their programming and how to design for it. This work will be undertaken by the COSI Center for Research and Evaluation in conjunction with the Office of Education, Outreach, and Visitor Services of the National Museum of Natural History and the PEAR Institute.The primary aim of this multi-day virtual convening is to inform future development of the framework for outcome progressions. These progressions will reflect and accommodate outcomes targeted by existing and shared evaluation measures. The framework will provide informal educators and evaluators with (1) a way of aligning outcome expectations with programming depth and dosage; (2) outcome choices achievable with various depths of programming; (3) a strategic tool for planning institution-wide targeted outcomes across a full portfolio of institution educational programming; and (4) a tool for documenting an education department’s varied outcomes across the full range of possible outcomes. To create rich and informed critical discussion, the conference will involve research consultant experts representing each outcome category; practitioners representing diverse programming; and evaluation theorists familiar with the broad needs of informal learning institutions. A conference subcommittee with expertise in culturally responsive and equitable program development and evaluation will ensure the agenda contains flexibility to generate emergent equitable and inclusive conversations. Conference participants will review and contribute to the soundness and utility of these categorizations as well as the progressions within them. These insights and inputs will inform future development of the framework as a valid and reliable tool. Deliverables include a paper summarizing the findings from the convenings and a related webinar that describes the framework and recommendations for its use. These will be shared broadly via the informalscience.org website and other professional venues. This conference is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.