Emerging Research- Empirical- Development of an Observation Tool for the Informal Science Field:Refinement, Field-testing and Establishment of Psychometric Properties

本研究项目计划在校外（OST）环境中完善、实地测试和建立正式评估工具——STEM学习成功维度（DOS）的心理测量特性。借助该工具，可以对编程的各个维度进行正式观察，并对非正式科学进行学习。该项目的目标是评估该工具的心理测量属性，主要是其结构效度，因此该工具可以广泛用于评估节目质量。第一阶段将专注于迭代地完善和修订用于STEM学习的DOS，建立在现有协议的基础上。该仪器的可靠性分析将使用教育工作者的样本进行。在第二阶段，学生和教师的自我报告调查将提供额外的数据，这些数据将被分析以支持该工具的进一步验证。随机分配的两名观察员对100名OST教育工作者和他们的学生进行了两次观察，这将有助于全面分析该工具，并在第二阶段为观察员制定培训计划。该项目将对OST项目的质量和评估产生重大影响。在地方一级，DOS将使项目的从业者和管理者能够提高项目质量和专业发展。该项目将产生一个有效的工具，可以作为评估OST STEM项目的标准，以提高项目质量，并比较不同地点的项目。因此，对OST - STEM项目进行更大规模的研究将成为可能，有助于共享知识库。该项目将通过互联网向项目工作人员、研究人员和评估人员提供培训，并将为该仪器的使用提供支持，从而使该仪器得到广泛使用。培训支助将为非正式科学学习领域的方案发展和评价建立能力。

This research project plans to refine, field test and establish the psychometric properties of a formal assessment tool, Dimensions of Sucess (DOS) for STEM Learning, in out-of-school time (OST) settings. With the help of this tool, formal observations on various dimensions of programming and learning of informal science can be conducted. The goal of the project is to assess the tool's psychometric properties, primarily its construct validity, so the instrument can be used widely to assess the program quality. Phase 1 will focus on iteratively refining and revising the DOS for STEM learning, building on existing experience with the protocol. Reliability analysis on the instrument will be condusted using a sample of educators. In Phase 2, student and teacher self-report surveys will provide additional data, which will be analyzed to support the further validation of the instrument. Observations of 100 OST educators with their students on two occasions by two randomly assigned observers will allow for full analysis of the tool and for the development of the training program for observers during Phase 2. The project will have significant implications for the quality and evaluation of OST programs. At the local level, DOS will enable practitioners and administrators of programs to improve program quality and professional development. The project will yield a valid instrument that can serve as a standard for assessing OST STEM programs to improve program quality and to compare programs across sites. Thus, larger studies of OST STEM programs will be possible, contributing to the shared knowledge base. The program will offer training via the internet to program staff, researchers and evaluators, and will provide support for the use of the instrument, so that the instrument can be used widely. The training support will build capacity for program development and evaluation in the field of informal science learning.