Development and Application of a Multilevel Evaluation Procedure for Examining State and School Educational Contexts with NAEP

国家教育进步评估(NAEP)数学考试根据每个测试项目衡量一个潜在的熟练程度(例如，一个数学领域)的假设来提供总分。调查人员先前的工作表明，可以从NAEP考试中挖掘出更多信息，可用于教学层面的政策制定，如学校的政策制定。使用NAEP项目反应数据，研究人员提出了一个双重目标：(1)了解学校和国家对学生数学成绩的影响；(2)改进数学教育评估的测量和统计分析技术。相当一部分的研究将集中于创建测量和统计技术(目标二)，这将在目标一分析的进展中得到证明。调查人员将使用项目反应理论(IRT)和分层线性建模(HLM)分析来产生他们自己的基于统计的NAEP项目分类，当与现有的(官方)基于NAEP概念的分类系统相比，应该同时揭示可以在学校和州一级归因于的信息。这将允许研究人员使用IRT和HLM中的数据及其(误差)方差来检查学校在任何数量的预先选择的相关政策特征上的不同之处。然而，要做到这一点，需要研究人员建立新的统计工具，并将要求以假设的形式对这一前提进行检验。因此，这项可行性研究的结果将决定是否可以使用一项为评估国家(仅)进步而设计的测试来解释学校的相对数学优势。这一方法可用于使用全国教育计划和其他可能的大规模评估数据来评估学校一级的政策倡议。

The National Assessment of Educational Progress (NAEP) mathematics exam renders aggregate scores based on the assumption that each test item measures a single underlying proficiency (e.g., an area of mathematics). Prior work of the investigators suggests that there may be more information that can be mined from the NAEP exam that may be used for policymaking at the instructional level, such as policymaking for schools). Using the NAEP item response data, the investigators propose a two-fold goal: (1) to understand school and state influences on student mathematics achievement, and (2) to improve measurement and statistical analysis techniques for the evaluation of mathematics education. A considerable portion of the research will focus on the creation of measurement and statistical techniques (goal two) that will be demonstrated in the advancement of goal one analyses.The investigators will use Item Response Theory (IRT) and Hierarchical Linear Modeling (HLM) analyses to produce their own statistically-based classification of NAEP items that, when contrasted with the existing (official) NAEP conceptually-based classification system, should reveal information that can be attributed at the school and state levels simultaneously. This will allow the researchers to use the data and their (error) variance--both in IRT and HLM--to examine how schools differ on any number of preselected, relevant policy characteristics. To do this, however, requires that the investigators build new statistical tools and will require that this very premise be tested in the form of a hypothesis. Thus, the results of this feasibility study will determine if a test designed for assessing national (only) progress can be used to explain the relative mathematical strengths of schools. This approach could be used to evaluate policy initiatives focused at the school level using NAEP and potentially other large-scale assessment data.