Strategic Outcome Progressions Conference: Exploring a Framework for Measuring Informal Education Outcomes and Institutional Impact

有各种措施可以评估非正式科学、技术、工程或数学(STEM)教育方案的学习成果，特别是针对青年受众。尽管从业者、研究人员和评估人员对制定和实施共享、共同的STEM措施和框架越来越感兴趣，但非正式STEM学习(ISL)机构(如博物馆、动物园、水族馆、植物园等)在设计和评估教育项目时面临着独特的挑战。根据这些机构强调的广泛主题领域的性质，节目试图接触到年龄、能力、文化背景、兴趣等方面不同的观众。参与者的参与也各不相同--例如，参与者在一系列展览中选择参加哪些展览，以及在每个展览上花费的时间。考虑到这种有计划的差异，教育工作者如何确定适当的学习结果？结果在多大程度上映射到活动的复杂性？方案编制的成功如何与机构目标相联系？一种解决方案可能是制定一个成果进展框架。“进展”属于结果类别(例如，兴趣、态度、知识、技能、行为、科学资本等)，是可以映射到编程的有序系列效果，从具有简单内容的短期持续时间(例如，走廊手推车演示可以触发情景兴趣)到具有复杂内容的较长持续时间(例如，青年实习可以产生良好的个人兴趣)。在ISL机构工作或与ISL机构合作的STEM教育者、评估者和研究人员将聚集在一起，考虑成果进展框架的实用性和可行性。有了这一框架，STEM学习机构的教育工作者可能会更好地了解其方案的影响以及如何为其设计方案。这项工作将由COSI研究和评估中心与国家自然历史博物馆和PEAR研究所的教育、外联和游客服务处一起进行。这一为期多天的虚拟会议的主要目的是为成果进展框架的未来发展提供信息。这些进展将反映和适应现有和共同评价措施所针对的结果。该框架将为非正式教育工作者和评估者提供：(1)使成果预期与方案拟订深度和剂量保持一致的方法；(2)通过不同方案拟订深度可实现的成果选择；(3)在整个机构教育方案拟订组合中规划全机构目标成果的战略工具；(4)记录教育部门所有可能成果的不同成果的工具。为了开展内容丰富、见多识广的批判性讨论，会议将有代表每个成果类别的研究顾问专家、代表不同方案编制的实践者以及熟悉非正式学习机构广泛需求的评价理论家参加。在文化回应和公平方案制定和评估方面拥有专门知识的会议小组委员会将确保议程包含灵活性，以产生紧急、公平和包容的对话。与会者将审查这些分类的合理性和实用性以及其中的进展情况，并作出贡献。这些见解和投入将为框架作为一个有效和可靠的工具的未来发展提供信息。交付成果包括一份总结召集会议结果的文件，以及一次介绍框架和使用建议的相关网络研讨会。这些信息将通过informalcience.org网站和其他专业场所广泛分享。本次会议由推进非正式STEM学习计划(AISL)资助，该计划旨在促进对非正式环境中STEM学习的设计和发展的新方法和基于证据的理解。这包括为扩大获得和参与STEM学习经验提供多种途径，在非正式环境中推进对STEM学习的创新研究和评估，以及加深参与者对更深层次学习的理解。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Various measures exist to assess learning outcomes from informal science, technology, engineering, or math (STEM) education programming, especially for youth audiences. Although there has been growing interest in the field among practitioners, researchers and evaluators for the development and implementation of shared, common STEM measures and frameworks, informal STEM learning (ISL) institutions (e.g., museums, zoos, aquaria, botanic gardens, and the like) face a unique challenge when they design and evaluate educational programs. By nature of the broad-ranging topic areas these institutions emphasize, programming seeks to reach audiences that vary in age, ability, cultural background, interest, and more. Participation also varies--for example, participants choose which in a series of exhibits they attend and how much time they spend at each. Given this planned-for variance, how do educators target appropriate learning outcomes? To what extent do outcomes map to activity complexity? How does programming success link to institutional goals? One solution may be a framework of outcome progressions. “Progressions” are within outcome categories (e.g., interest, attitude, knowledge, skills, behavior, science capital, etc.), the ordinal series of effects that can be mapped to programming that ranges from short duration with simple content (e.g. a hallway cart demonstration can trigger situational interest) to extended duration with complex content (e.g. a youth internship can generate well-developed personal interest). STEM educators, evaluators and researchers working in or with ISL institutions will come together to consider the utility and feasibility of an outcome progressions framework. With this framework STEM learning institution educators may be better equipped to understand the impact of their programming and how to design for it. This work will be undertaken by the COSI Center for Research and Evaluation in conjunction with the Office of Education, Outreach, and Visitor Services of the National Museum of Natural History and the PEAR Institute.The primary aim of this multi-day virtual convening is to inform future development of the framework for outcome progressions. These progressions will reflect and accommodate outcomes targeted by existing and shared evaluation measures. The framework will provide informal educators and evaluators with (1) a way of aligning outcome expectations with programming depth and dosage; (2) outcome choices achievable with various depths of programming; (3) a strategic tool for planning institution-wide targeted outcomes across a full portfolio of institution educational programming; and (4) a tool for documenting an education department’s varied outcomes across the full range of possible outcomes. To create rich and informed critical discussion, the conference will involve research consultant experts representing each outcome category; practitioners representing diverse programming; and evaluation theorists familiar with the broad needs of informal learning institutions. A conference subcommittee with expertise in culturally responsive and equitable program development and evaluation will ensure the agenda contains flexibility to generate emergent equitable and inclusive conversations. Conference participants will review and contribute to the soundness and utility of these categorizations as well as the progressions within them. These insights and inputs will inform future development of the framework as a valid and reliable tool. Deliverables include a paper summarizing the findings from the convenings and a related webinar that describes the framework and recommendations for its use. These will be shared broadly via the informalscience.org website and other professional venues. This conference is funded by the Advancing Informal STEM Learning (AISL) program, which seeks to advance new approaches to, and evidence-based understanding of, the design and development of STEM learning in informal environments. This includes providing multiple pathways for broadening access to and engagement in STEM learning experiences, advancing innovative research on and assessment of STEM learning in informal environments, and developing understandings of deeper learning by participants.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.