Automatic profiling of science assessment items to model item parameters: A natural language processing approach
Automatic profiling of science assessment items to model item parameters: A natural language processing approach
批准号:
1920512
负责人:
Min Li
金额:
$50.0万
依托单位:
依托单位国家:
美国
项目类别:
Standard Grant
财政年份:
2019
资助国家:
美国
项目状态:
已结题
起止时间:
2019-09-01 至 2024-08-31
中文摘要
该提案是针对电子病历核心研究(ECR)计划公告NSF 19-508而提交的。ECR STEM教育基础研究计划为关键的研究领域提供资金,这些领域是基本的、广泛的和持久的。EHR寻求的建议将有助于综合、建立和/或扩大以下重点领域的研究基础:STEM学习、STEM学习环境、STEM劳动力发展和扩大STEM的参与。ECR计划的特点是强调积累可靠的证据,为(A)理解、(B)建立解释理论和(C)建议干预措施(和创新)以应对STEM兴趣、教育、学习和参与方面的持续挑战提供信息。这个EHR核心研究项目调查了自然语言处理(NLP)的使用,以自动识别评估项目的特征,并探索它们在分析学生表现模式方面的效用。该项目调查了三类主要的评估项目:认知、语言和语境;并评估了它们如何影响学生的考试表现。这项研究将有助于中学科学教育中测试项目开发的科学性,解决在构建情境项目以评估学生概念理解方面缺乏系统研究的问题。这一跨学科研究项目将加强考虑认知、背景和语言需求的项目开发的理论和经验基础,并有助于改进评估工具和方法,以衡量科学学习中有价值的结构。研究人员将致力于四个研究目标:(1)根据文献综述、认知访谈和编码者间可靠性研究,确定并实施科学评估项目的目标维度;(2)开发并经验性地测试机器学习算法,该算法可用于自动描述大量且多样化的项目;(3)提供证据,说明项目特征对项目参数和7年级和8年级学生表现差异模式可能的主要和交互影响;以及(4)对于影响显著的变量,开发可解释的自动算法变体,为改进评估项目提供指导。项目成果将包括:i)开发一个参数模型,通过利用机器学习结合大规模数据分析,帮助理解评估项目如何随认知、背景和语言维度的变化而变化;ii)公开可用的软件工具,用于配置情景评估项目,以及已配置的项目目录;iii)关于符合K-12科学教育下一代科学标准框架愿景的项目配置自动方法的有效性的经验证据;以及iv)一本专著,描述为内容审查、认知需求分析和与标准保持一致而开发和评估情景项目所需的方法和程序。该奖项反映了NSF的法定使命,并通过使用基金会的智力优势和更广泛的影响审查标准进行评估,被认为值得支持。
英文摘要
This proposal was submitted in response to EHR Core Research (ECR) program announcement NSF 19-508. The ECR program of fundamental research in STEM education provides funding in critical research areas that are essential, broad and enduring. EHR seeks proposals that will help synthesize, build and/or expand research foundations in the following focal areas: STEM learning, STEM learning environments, STEM workforce development, and broadening participation in STEM. The ECR program is distinguished by its emphasis on the accumulation of robust evidence to inform efforts to (a) understand, (b) build theory to explain, and (c) suggest interventions (and innovations) to address persistent challenges in STEM interest, education, learning, and participation. This EHR Core Research project investigates the use of natural language processing (NLP) to automatically identify characteristics of assessment items and explore their utility for analyzing patterns of student performance. The project investigates three central categories of assessment items: cognitive, linguistic, and context; and evaluates how they influence students' test performance. This research will contribute to the science of test-item development in secondary science education by addressing the lack of systematic research in constructing contextualized items to assess students' conceptual understanding. This interdisciplinary research project will strengthen the theoretical and empirical basis for item development that accounts for cognitive, contextual and linguistic needs and helps improve assessment tools and methods for measuring valuable constructs in science learning. The investigators will address four research objectives: (1) define and operationalize the targeted dimensions of science assessment items based on literature reviews, cognitive interviews, and inter-coder reliability study; (2) develop and empirically test machine learning algorithms that can be used to automatically profile a large and diverse set of items; (3) provide evidence about the possible main and interaction effects of item characteristics on item parameters and differential patterns of student performance at grades 7 and 8; and (4) for variables with significant effects, develop interpretable variants of the automatic algorithms that provide guidance for improving assessment items. Project outcomes will include: i) development of a parameterized model that contributes to the understanding of how assessment items vary along cognitive, context, and linguistic dimensions by leveraging machine learning to incorporate large scale data analysis; ii) publicly available software tools for profiling contextualized assessment items, together with a catalog of items that have been profiled; iii) empirical evidence about the effectiveness of the automatic approaches for profiling items that align with the vision of the Next Generation Science Standards Framework for K-12 Science Education; and iv) a monograph describing the approach and procedures required to develop and evaluate contextualized items for content review, analysis of cognitive demands, and alignment to standards.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
期刊论文(0)
专著(0)
科研奖励(0)
会议论文
Exploring Differences Between Instructors' Exams and How These Differences Produce Scores that Could Inaccurately and Inequitably Represent Student Understanding
-
批准号:1709423
-
项目类别:Standard Grant
-
资助金额:$5.0万
-
财政年份:2017
-
负责人:Min Li
-
依托单位:
Doctoral Dissertation Research: Examining Sequence of Contextualized Items in Science
-
批准号:1461431
-
项目类别:Standard Grant
-
资助金额:$1.6万
-
财政年份:2015
-
负责人:Min Li
-
依托单位:
Collaborative Research: Examining Formative Assessment Practices for English Language Learners in Science Classrooms
-
批准号:1118951
-
项目类别:Standard Grant
-
资助金额:$12.22万
-
财政年份:2011
-
负责人:Min Li
-
依托单位:
S: Identifying Critical Characteristics of Effective Feedback Practices in Science and Mathematics Education
-
批准号:0822373
-
项目类别:Standard Grant
-
资助金额:$25.93万
-
财政年份:2009
-
负责人:Min Li
-
依托单位:
CAREER: Investigating classroom assessment practices in science
-
批准号:0644435
-
项目类别:Continuing Grant
-
资助金额:$67.36万
-
财政年份:2007
-
负责人:Min Li
-
依托单位:
Linking Assessment to Science Achievement
-
批准号:0310360
-
项目类别:Standard Grant
-
资助金额:$28.75万
-
财政年份:2003
-
负责人:Min Li
-
依托单位:
国内基金
海外基金
柴胡类生药鉴定与质量评价的二元条形码系统的研究
-
批准号:30873387
-
项目类别:面上项目
-
资助金额:32.0万元
-
批准年份:2008
-
负责人:晁志
-
依托单位: