SFS-Capacity: Collaborative: Validation of Concept Assessment Tools for Cybersecurity

SFS-Capacity：协作：网络安全概念评估工具的验证

• 批准号: 1820531
• 负责人: Geoffrey Herman
• 金额: $ 19万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2021-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1820531&HistoricalAwards=false
• 关键词: SFS; Capacity; Collaborative; Validation; Concept

The project at the University of Maryland, Baltimore County (UMBC) aims to evaluate two cybersecurity educational assessment tools: the Cybersecurity Concept Inventory (CCI) and the Cybersecurity Curriculum Assessment (CCA). The CCI assesses the quality of instruction of any first course in cybersecurity; the CCA assesses how well a college curriculum prepares graduates for a career in cybersecurity. The evaluation will consist of student interviews, expert reviews, and statistical analyses to examine the efficacy of these tools. In addition, the project will provide initial insights into the efficacy of educational approaches by comparing student performance relative to their instructors? teaching practices. At present, there is not a rigorous, research-based method for measuring the quality of cybersecurity instruction. Validated assessment tools such as the ones in development at UMBC are essential so that educators have trusted methods for discerning whether efforts to improve student preparation are successful. This validation study will complete an essential part of the broader Cybersecurity Assessment Tools project, which will provide rigorous, evidence-based instruments for assessing and evaluating educational practices. To develop evidence for the validity of the CCI?s and CCA?s measurement of student conceptual understanding of cybersecurity, the project will complete six tasks for each assessment tool: (1) conduct at least thirty cognitive interviews of students thinking aloud while answering the draft questions, and use results to improve the validity of the questions; (2) collect and use feedback on the draft questions from at least twenty experts in cybersecurity or cybersecurity education; (3) administer the draft tool to at least 200 students and use psychometric methods of analysis on the results; (4) revise the draft questions in light of tasks 1-3; (5) administer the revised tool to at least 1,000 students and use psychometric analysis on the results; and (6) hold workshops at four cybersecurity education meetings to promote awareness of, and to seek additional feedback on the tools. By following well-established rigorous methods for developing CCIs, this project will help the cybersecurity educational community to develop and improve curricula and instructional methods and materials.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

马里兰州巴尔的摩县大学（UMBC）的项目旨在评估两种网络安全教育评估工具：网络安全概念清单（CCI）和网络安全课程评估（CCA）。CCI评估网络安全的任何第一门课程的教学质量; CCA评估大学课程如何为毕业生在网络安全领域的职业生涯做好准备。评估将包括学生访谈，专家评审和统计分析，以检查这些工具的有效性。此外，该项目将通过比较学生与教师的表现，初步了解教育方法的有效性。教学实践。目前，还没有一个严格的，基于研究的方法来衡量网络安全教学的质量。经过验证的评估工具，如在UMBC开发的工具是必不可少的，使教育工作者有可信的方法来辨别努力，以提高学生的准备是否成功。这项验证研究将完成更广泛的网络安全评估工具项目的重要组成部分，该项目将为评估和评价教育实践提供严格的循证工具。为CCI的有效性提供证据？s和CCA？为了衡量学生对网络安全的概念理解，该项目将为每个评估工具完成六项任务：（1）对学生进行至少三十次认知访谈，在回答问题草稿时大声思考，并使用结果来提高问题的有效性;（2）收集和使用至少二十位网络安全或网络安全教育专家对问题草稿的反馈;（3）向至少200名学生使用工具草稿，并对结果使用心理测量学分析方法;（4）根据任务1-3修改问题草稿;（5）向至少1，000名学生使用修订后的工具，并对结果使用心理测量学分析;以及（6）在四次网络安全教育会议上举办讲习班，以提高对这些工具的认识，并寻求对这些工具的更多反馈。通过遵循完善的开发CCI的严格方法，该项目将帮助网络安全教育社区开发和改进课程、教学方法和材料。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Identifying Core Concepts of Cybersecurity: Results of Two Delphi Processes

• DOI: 10.1109/te.2017.2715174
• 发表时间: 2018-02-01
• 期刊: IEEE TRANSACTIONS ON EDUCATION
• 影响因子: 2.6
• 作者: Parekh, Geet;DeLatte, David;Sherman, Alan T.
• 通讯作者: Sherman, Alan T.

Psychometric Evaluation of the Cybersecurity Concept Inventory

网络安全概念量表的心理测量评估

• DOI: 10.1145/3451346
• 发表时间: 2022
• 期刊: ACM Transactions on Computing Education
• 影响因子: 2.4
• 作者: Poulsen, Seth;Herman, Geoffrey L.;Peterson, Peter A.;Golaszewski, Enis;Gorti, Akshita;Oliva, Linda;Scheponik, Travis;Sherman, Alan T.
• 通讯作者: Sherman, Alan T.

Experiences and lessons learned creating and validating concept inventories for cybersecurity

创建和验证网络安全概念清单的经验和教训

• 发表时间: 2020
• 期刊: National Cyber Summit (NCS
• 作者: Sherman, Alan T.;Herman, Geoffrey L.;Oliva, Linda;Peterson, Peter A.;Golaszewski, Enis;Poulsen, Seth;Scheponik, Travis;Gorti, Akshita
• 通讯作者: Gorti, Akshita

Initial validation of the Cybersecurity Concept Inventory: Pilot testing and expert review

网络安全概念清单的初步验证：试点测试和专家评审

• 发表时间: 2019
• 期刊: Proceedings - Frontiers in Education Conference
• 作者: Offenberger, Spencer;Herman, Geoffrey L.;Peterson, Peter;Sherman, Alan T.;Golaszewski, Enis;Scheponik, Travis;Oliva, Linda
• 通讯作者: Oliva, Linda

Investigating crowdsourcing to generate distractors for multiple-choice assessments

研究众包以产生多项选择评估的干扰因素

• 发表时间: 2019
• 期刊: National Cyber Summit
• 作者: Scheponik, T.;Golaszewski, E.;Herman, G.;Offenberger, S.;Oliva, L.;Peterson, P. A.;Sherman, A. T.
• 通讯作者: Sherman, A. T.