Crowd-Sourced Online Nexus for Developing Assessments of Middle-school Physical Science Disciplinary Core Ideas

该项目将开发和测试一个基于网络的平台，以提高科学教室中教师管理的测试质量。它借鉴了课堂教师的知识，同时采用了严格的统计方法，用于标准化评估的创建和验证。内容重点是在下一代科学标准（NGSS）的6-8年级物理科学学科的核心思想。现在，教师估计有20%的时间花在评估上，但在创建评估时，可供利用的工具相对较少。随着时间的推移，他们学会从现有的课程材料和教科书中改编项目。另一方面，标准化评估开发人员具有专家项目编写者、长开发周期、大量且多样化的学生群体和复杂的心理测量工具的优势。该项目结合了这两种方法，利用教师贡献他们最好的项目，然后立即使用众包主题进行试点。心理测量学分析产生的项目质量的措施，然后“反馈”项目参与教师的改进。通过这种方式，将利用教师输入构建大型试题库，每个试题具有：适当的阅读水平、NGSS对齐、科学准确性、适当的难度、高统计区分度以及性别、种族或民族的可忽略差异。参与该项目对缺乏正式评估培训的教师有潜在的好处，使参与者熟悉NGSS和高质量测试开发的要素。该项目将评估一个新的协作系统的优点，用于开发和验证高质量的测试项目和评估工具。它将衡量教师在何种程度上可以产生有效的项目，并改善现有的项目表现出有问题的问题时，给予严格的心理测量措施，估计项目质量的指导。它将建立在早期的研究表明，一个成年人，众包样本工程以及作为一个初始代理6-8年级的科学学生，允许项目质量非常快速的反馈（通常是一夜之间），与项目反应理论计算用于建立项目难度，项目歧视，猜测水平，和差异项目功能（性别和种族/民族偏见）。此外，计算出的错误概念强度、科学正确性、阅读水平以及与NGSS的匹配度等测量指标，将有助于指导教师的修改。贝叶斯无效分析的使用将“分类”项目，最大限度地减少昂贵的测试项目时，认为不可能满足项目的质量标准，降低成本。现场测试与6-8年级学生的大样本将提供对项目质量的最终检查。项目的开发将比标准化测试开发所用的方法便宜得多。两对（化学和物理的公开发布和安全）的评估工具将被构建，并免费提供给科学教师用于课堂使用，并由教育研究人员和课程开发人员。一个系统，提供项目质量的快速反馈可能会改变大学教学和评估专业发展的机会。从选定的响应开始时，（多项选择）项目，该项目将能够实现更大的各种格式在未来，纳入自动化的方法，因为他们成为可用的。发现研究preK-12计划（DRK-12）旨在显着提高学习和科学，技术，工程和数学（STEM）的preK-12学生和教师的教学，通过研究和开发创新资源、模式和工具。DRK-12项目中的项目建立在STEM教育的基础研究以及为拟议项目提供理论和经验依据的先前研究和开发工作的基础上。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project will develop and test a web-based platform to increase the quality of teacher-administered tests in science classrooms. It draws on classroom teacher knowledge while employing the rigorous statistical methods used in standardized assessment creation and validation. The content focus is on the disciplinary core ideas for grades 6-8 physical science in the Next Generation Science Standards (NGSS). Teachers now spend an estimate 20% of their time in assessment, yet have relatively few tools to draw upon when creating them. Over time, they learn to adapt items from available curriculum materials and textbooks. On the other hand, standardized assessment developers have the benefit of expert item writers, long development cycles, a large and diverse student population, and sophisticated psychometric tools. This project combines these two approaches, drawing upon teachers to contribute their best items, then immediately piloting them using crowdsourced subjects. Psychometric analysis generates measures of item quality and then “recycles” items to participating teachers for improvement. In this way, a large test item bank will be constructed utilizing teacher input with each item possessing: appropriate reading levels, NGSS alignment, scientific accuracy, appropriate difficulty, high statistical discrimination, and negligible difference by gender, race, or ethnicity. Involvement in this project has potential benefits for teachers lacking formal training in assessment, familiarizing participants with the NGSS, and with the elements of high-quality test development.The project will gauge the merits of a novel collaborative system for the development and validation of high-quality test items and assessment instruments. It will measure the degree to which teachers can generate effective items and improve existing items exhibiting problematic issues when given the guidance of rigorous psychometric measures that estimate item quality. It will build on earlier research showing that an adult, crowd-sourced sample works well as an initial proxy for grade 6-8 science students, allowing for extremely rapid feedback on item quality (often overnight), with item response theory computation used to establish item difficulty, item discrimination, guessing levels, and differential item functioning (gender and racial/ethnicity bias). In addition, computed measures of misconception strength, scientific correctness, reading level, and match to the NGSS will help to guide revision by teachers. Use of Bayesian futility analysis will “triage” items, minimizing costly testing of items when deemed unlikely to meet item quality criteria, lowering costs. Field testing with a large sample of grade 6-8 students will provide a final check on item quality. Items will be developed much more inexpensively than by methods used for standardized test development. Two pairs (public-release and secure for chemistry and physics) of assessment instruments will be constructed and be freely available to science teachers for classroom use and by education researchers and curriculum developers. A system that provides quick feedback on item quality could potentially transform university instruction and professional development opportunities in assessment. While starting with selected response (multiple-choice) items, the project will be able to implement a larger variety of formats in the future, incorporating automated approaches as they become available.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.