Development and Application of a Multilevel Evaluation Procedure for Examining State and School Educational Contexts with NAEP

全国教育进步评估（NAEP）数学考试基于每个测试项目衡量单个潜在熟练程度（例如，数学领域）的假设给出总分。研究者先前的工作表明，从NAEP考试中可以挖掘出更多的信息，这些信息可能用于教学层面的政策制定，例如学校的政策制定。利用NAEP项目回答数据，研究者提出了两个目标：(1)了解学校和国家对学生数学成绩的影响；(2)改进数学教育评估的测量和统计分析技术。研究的相当一部分将集中于创造测量和统计技术（目标二），这将在目标一分析的进展中得到证明。研究者将使用项目反应理论（IRT）和层次线性模型（HLM）分析来产生他们自己的基于统计的NAEP项目分类，与现有的（官方的）NAEP基于概念的分类系统相比，应该揭示可以同时归因于学校和州层面的信息。这将使研究人员能够使用IRT和HLM中的数据及其（误差）方差来检查学校在任何数量的预选的相关政策特征上的差异。然而，要做到这一点，需要研究人员建立新的统计工具，并要求以假设的形式对这个前提进行检验。因此，这项可行性研究的结果将决定设计用于评估国家（仅）进步的测试是否可以用来解释学校的相对数学优势。这种方法可用于利用NAEP和其他潜在的大规模评估数据来评估学校层面的政策举措。

The National Assessment of Educational Progress (NAEP) mathematics exam renders aggregate scores based on the assumption that each test item measures a single underlying proficiency (e.g., an area of mathematics). Prior work of the investigators suggests that there may be more information that can be mined from the NAEP exam that may be used for policymaking at the instructional level, such as policymaking for schools). Using the NAEP item response data, the investigators propose a two-fold goal: (1) to understand school and state influences on student mathematics achievement, and (2) to improve measurement and statistical analysis techniques for the evaluation of mathematics education. A considerable portion of the research will focus on the creation of measurement and statistical techniques (goal two) that will be demonstrated in the advancement of goal one analyses.The investigators will use Item Response Theory (IRT) and Hierarchical Linear Modeling (HLM) analyses to produce their own statistically-based classification of NAEP items that, when contrasted with the existing (official) NAEP conceptually-based classification system, should reveal information that can be attributed at the school and state levels simultaneously. This will allow the researchers to use the data and their (error) variance--both in IRT and HLM--to examine how schools differ on any number of preselected, relevant policy characteristics. To do this, however, requires that the investigators build new statistical tools and will require that this very premise be tested in the form of a hypothesis. Thus, the results of this feasibility study will determine if a test designed for assessing national (only) progress can be used to explain the relative mathematical strengths of schools. This approach could be used to evaluate policy initiatives focused at the school level using NAEP and potentially other large-scale assessment data.