Crowd-Sourced Online Nexus for Developing Assessments of Middle-school Physical Science Disciplinary Core Ideas

该项目将开发和测试一个基于网络的平台，以提高教师在科学课堂上进行的考试的质量。它利用课堂教师的知识，同时使用标准化评估、创建和验证中使用的严格统计方法。内容重点是下一代科学标准(NGSS)中6-8年级物理科学的学科核心理念。教师现在估计有20%的时间花在评估上，但在制作这些课程时，可供参考的工具相对较少。随着时间的推移，他们学会了从现有的课程材料和教科书中改编内容。另一方面，标准化评估开发人员拥有专业的项目编写者、较长的开发周期、大量且多样化的学生群体以及复杂的心理测量工具。这个项目结合了这两种方法，让老师贡献他们最好的项目，然后立即使用众包科目进行试点。心理测量学分析产生项目质量的测量，然后将项目回收给参与的教师进行改进。通过这种方式，将利用教师的输入来构建一个大型试题库，每个试题具有：适当的阅读水平、NGSS对齐、科学准确性、适当的难度、高度的统计学差别以及可以忽略不同性别、种族或民族的差异。参与这个项目对缺乏正规评估培训的教师有潜在的好处，让参与者熟悉NGSS，并了解高质量测试开发的要素。该项目将衡量一个新的协作系统的优点，以开发和验证高质量的测试项目和评估工具。它将衡量教师在接受评估项目质量的严格心理测量措施的指导下，能够在多大程度上产生有效的项目并改进现有的问题项目。它将建立在早期研究的基础上，该研究表明，成人、众包样本可以很好地作为6-8年级理科学生的初始替代，允许对项目质量进行极快的反馈(通常是一夜之间)，项目反应理论计算用于确定项目难度、项目辨别、猜测水平和不同的项目功能(性别和种族/民族偏见)。此外，计算错误概念强度、科学正确性、阅读水平以及与NGSS的匹配度将有助于指导教师进行修改。使用贝叶斯无效性分析将对物品进行分类，最大限度地减少在被认为不太可能满足物品质量标准时对物品进行昂贵的测试，从而降低成本。对6-8年级学生进行大样本的实地测试将对项目质量进行最终检查。试题的开发将比标准化测试开发的方法便宜得多。将建造两对评估工具(公开发布和化学和物理的安全)，供科学教师课堂使用以及教育研究人员和课程制定者免费使用。一个对项目质量提供快速反馈的系统可能会潜在地改变大学教学和评估中的专业发展机会。该项目从选定的答案(多项选择)开始，未来将能够实施更多种类的格式，并在可用时纳入自动化方法。探索研究预科-12课程(DRK-12)旨在通过研究和开发创新资源、模型和工具，显著提高预科-12年级学生和教师在科学、技术、工程和数学(STEM)方面的学习和教学。DRK-12计划中的项目建立在STEM教育的基础研究和先前的研究和开发工作的基础上，为拟议的项目提供了理论和经验上的证明。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project will develop and test a web-based platform to increase the quality of teacher-administered tests in science classrooms. It draws on classroom teacher knowledge while employing the rigorous statistical methods used in standardized assessment creation and validation. The content focus is on the disciplinary core ideas for grades 6-8 physical science in the Next Generation Science Standards (NGSS). Teachers now spend an estimate 20% of their time in assessment, yet have relatively few tools to draw upon when creating them. Over time, they learn to adapt items from available curriculum materials and textbooks. On the other hand, standardized assessment developers have the benefit of expert item writers, long development cycles, a large and diverse student population, and sophisticated psychometric tools. This project combines these two approaches, drawing upon teachers to contribute their best items, then immediately piloting them using crowdsourced subjects. Psychometric analysis generates measures of item quality and then “recycles” items to participating teachers for improvement. In this way, a large test item bank will be constructed utilizing teacher input with each item possessing: appropriate reading levels, NGSS alignment, scientific accuracy, appropriate difficulty, high statistical discrimination, and negligible difference by gender, race, or ethnicity. Involvement in this project has potential benefits for teachers lacking formal training in assessment, familiarizing participants with the NGSS, and with the elements of high-quality test development.The project will gauge the merits of a novel collaborative system for the development and validation of high-quality test items and assessment instruments. It will measure the degree to which teachers can generate effective items and improve existing items exhibiting problematic issues when given the guidance of rigorous psychometric measures that estimate item quality. It will build on earlier research showing that an adult, crowd-sourced sample works well as an initial proxy for grade 6-8 science students, allowing for extremely rapid feedback on item quality (often overnight), with item response theory computation used to establish item difficulty, item discrimination, guessing levels, and differential item functioning (gender and racial/ethnicity bias). In addition, computed measures of misconception strength, scientific correctness, reading level, and match to the NGSS will help to guide revision by teachers. Use of Bayesian futility analysis will “triage” items, minimizing costly testing of items when deemed unlikely to meet item quality criteria, lowering costs. Field testing with a large sample of grade 6-8 students will provide a final check on item quality. Items will be developed much more inexpensively than by methods used for standardized test development. Two pairs (public-release and secure for chemistry and physics) of assessment instruments will be constructed and be freely available to science teachers for classroom use and by education researchers and curriculum developers. A system that provides quick feedback on item quality could potentially transform university instruction and professional development opportunities in assessment. While starting with selected response (multiple-choice) items, the project will be able to implement a larger variety of formats in the future, incorporating automated approaches as they become available.The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering and mathematics (STEM) by preK-12 students and teachers, through research and development of innovative resources, models and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.