SFS-Capacity: Collaborative: Validation of Concept Assessment Tools for Cybersecurity

马里兰州巴尔的摩县大学（UMBC）和伊利诺伊大学香槟分校（UIUC）建议评估两种网络安全教育评估工具：网络安全概念清单（CCI）和网络安全课程评估（CCA）。CCI评估网络安全的任何第一门课程的教学质量; CCA评估大学课程如何为毕业生在网络安全领域的职业生涯做好准备。评估将包括学生访谈，专家评审和统计分析，以检查这些工具的有效性。此外，该项目将通过比较学生的表现与教师的教学实践，初步了解教育方法的功效。目前，还没有一个严格的，基于研究的方法来衡量网络安全教学的质量。经过验证的评估工具，如在UMBC开发的工具是必不可少的，使教育工作者有可信的方法来辨别努力，以提高学生的准备是否成功。这项验证研究将完成更广泛的网络安全评估工具项目的重要组成部分，该项目将为评估和评价教育实践提供严格的循证工具。为了证明CCI和CCA对学生网络安全概念理解的测量的有效性，该项目将为每个评估工具完成六项任务：（1）对学生进行至少三十次认知访谈，在回答草稿问题时大声思考，并使用结果来提高问题的有效性;（2）收集和使用至少20位网络安全或网络安全教育专家对问题草稿的反馈;（3）向至少200名学生管理工具草稿，并使用心理测量方法分析结果;（4）根据任务1-3修订问题草稿;（5）向至少1，000名学生使用修订后的工具，并对结果进行心理测量分析;（6）在四次网络安全教育会议上举办讲习班，以提高对工具的认识，并寻求对工具的更多反馈。通过遵循成熟的严格方法来开发CCI，该项目将帮助网络安全教育界开发和改进课程、教学方法和材料。该奖项反映了NSF的法定使命，并通过使用基金会的知识产权进行评估而被认为值得支持优点和更广泛的影响审查标准。

The University of Maryland, Baltimore County (UMBC) and University of Illinois at Urbana-Champaign (UIUC) propose to evaluate two cybersecurity educational assessment tools: the Cybersecurity Concept Inventory (CCI) and the Cybersecurity Curriculum Assessment (CCA). The CCI assesses the quality of instruction of any first course in cybersecurity; the CCA assesses how well a college curriculum prepares graduates for a career in cybersecurity. The evaluation will consist of student interviews, expert reviews, and statistical analyses to examine the efficacy of these tools. In addition, the project will provide initial insights into the efficacy of educational approaches by comparing student performance relative to their instructors' teaching practices. At present, there is not a rigorous, research-based method for measuring the quality of cybersecurity instruction. Validated assessment tools such as the ones in development at UMBC are essential so that educators have trusted methods for discerning whether efforts to improve student preparation are successful. This validation study will complete an essential part of the broader Cybersecurity Assessment Tools project, which will provide rigorous, evidence-based instruments for assessing and evaluating educational practices. To develop evidence for the validity of the CCI's and CCA's measurement of student conceptual understanding of cybersecurity, the project will complete six tasks for each assessment tool: (1) conduct at least thirty cognitive interviews of students thinking aloud while answering the draft questions, and use results to improve the validity of the questions; (2) collect and use feedback on the draft questions from at least twenty experts in cybersecurity or cybersecurity education; (3) administer the draft tool to at least 200 students and use psychometric methods of analysis on the results; (4) revise the draft questions in light of tasks 1-3; (5) administer the revised tool to at least 1,000 students and use psychometric analysis on the results; and (6) hold workshops at four cybersecurity education meetings to promote awareness of, and to seek additional feedback on the tools. By following well-established rigorous methods for developing CCIs, this project will help the cybersecurity educational community to develop and improve curricula and instructional methods and materials.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.