Automatic profiling of science assessment items to model item parameters: A natural language processing approach

该提案是为了响应EHR核心研究（ECR）计划公告NSF 19-508而提交的。在STEM教育的基础研究ECR计划提供资金在关键的研究领域是必不可少的，广泛的和持久的。EHR寻求有助于综合，建立和/或扩大以下重点领域研究基础的建议：STEM学习，STEM学习环境，STEM劳动力发展和扩大STEM参与。ECR计划的特点是强调积累强有力的证据，以（a）理解，（B）建立理论来解释，（c）提出干预措施（和创新），以解决STEM兴趣，教育，学习和参与方面的持续挑战。这个EHR核心研究项目调查了自然语言处理（NLP）的使用，以自动识别评估项目的特征，并探索其用于分析学生表现模式的实用性。该项目调查了三个核心类别的评估项目：认知，语言和上下文;并评估它们如何影响学生的测试表现。本研究将有助于中学科学教育中的测试项目开发的科学，通过解决缺乏系统的研究，构建情境化的项目来评估学生的概念理解。这个跨学科的研究项目将加强项目开发的理论和经验基础，考虑到认知，上下文和语言的需求，并有助于改善评估工具和方法，以衡量科学学习中有价值的结构。研究人员将解决四个研究目标：（1）根据文献综述，认知访谈和编码器间可靠性研究定义和操作科学评估项目的目标维度;（2）开发和经验测试机器学习算法，可用于自动分析大量不同的项目集;（3）提供关于项目特征对项目参数和七、八年级学生成绩差异模式可能的主效应和交互效应的证据;（4）对于具有显著影响的变量，开发自动算法的可解释变体，为改进评估项目提供指导。项目成果将包括：i）开发参数化模型，通过利用机器学习纳入大规模数据分析，有助于理解评估项目如何在认知、上下文和语言维度上变化; ii）用于分析情境化评估项目的公开软件工具，以及已分析项目的目录; iii）关于自动分析项目方法的有效性的经验证据，这些方法与K-12科学教育的下一代科学标准框架的愿景相一致;以及iv）描述开发和评估用于内容审查的情境化项目，认知需求分析，该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准。

This proposal was submitted in response to EHR Core Research (ECR) program announcement NSF 19-508. The ECR program of fundamental research in STEM education provides funding in critical research areas that are essential, broad and enduring. EHR seeks proposals that will help synthesize, build and/or expand research foundations in the following focal areas: STEM learning, STEM learning environments, STEM workforce development, and broadening participation in STEM. The ECR program is distinguished by its emphasis on the accumulation of robust evidence to inform efforts to (a) understand, (b) build theory to explain, and (c) suggest interventions (and innovations) to address persistent challenges in STEM interest, education, learning, and participation. This EHR Core Research project investigates the use of natural language processing (NLP) to automatically identify characteristics of assessment items and explore their utility for analyzing patterns of student performance. The project investigates three central categories of assessment items: cognitive, linguistic, and context; and evaluates how they influence students' test performance. This research will contribute to the science of test-item development in secondary science education by addressing the lack of systematic research in constructing contextualized items to assess students' conceptual understanding. This interdisciplinary research project will strengthen the theoretical and empirical basis for item development that accounts for cognitive, contextual and linguistic needs and helps improve assessment tools and methods for measuring valuable constructs in science learning. The investigators will address four research objectives: (1) define and operationalize the targeted dimensions of science assessment items based on literature reviews, cognitive interviews, and inter-coder reliability study; (2) develop and empirically test machine learning algorithms that can be used to automatically profile a large and diverse set of items; (3) provide evidence about the possible main and interaction effects of item characteristics on item parameters and differential patterns of student performance at grades 7 and 8; and (4) for variables with significant effects, develop interpretable variants of the automatic algorithms that provide guidance for improving assessment items. Project outcomes will include: i) development of a parameterized model that contributes to the understanding of how assessment items vary along cognitive, context, and linguistic dimensions by leveraging machine learning to incorporate large scale data analysis; ii) publicly available software tools for profiling contextualized assessment items, together with a catalog of items that have been profiled; iii) empirical evidence about the effectiveness of the automatic approaches for profiling items that align with the vision of the Next Generation Science Standards Framework for K-12 Science Education; and iv) a monograph describing the approach and procedures required to develop and evaluate contextualized items for content review, analysis of cognitive demands, and alignment to standards.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.