SFS-Capacity: Collaborative: Validation of Concept Assessment Tools for Cybersecurity

马里兰大学巴尔的摩县分校(UMBC)和伊利诺伊大学厄巴纳-香槟分校(UIUC)建议评估两种网络安全教育评估工具：网络安全概念清单(CCI)和网络安全课程评估(CCA)。CCI评估任何一门网络安全第一课程的教学质量；CCA评估大学课程对毕业生进入网络安全职业生涯的准备程度。评估将包括学生访谈、专家评审和统计分析，以检验这些工具的有效性。此外，该项目将通过将学生的表现与教师的教学实践进行比较，初步了解教育方法的有效性。目前，还没有一种严谨的、基于研究的方法来衡量网络安全教学的质量。经过验证的评估工具，如密歇根大学正在开发的工具，是必不可少的，这样教育工作者才能有可信的方法来辨别改善学生准备的努力是否成功。这项验证研究将完成更广泛的网络安全评估工具项目的一个重要部分，该项目将为评估和评价教育做法提供严格、循证的工具。为了证明CCI和CCA测量学生对网络安全概念理解的有效性，该项目将为每个评估工具完成六项任务：(1)对回答问题草稿时大声思考的学生进行至少30次认知访谈，并使用结果来提高问题的有效性；(2)收集和使用至少20名网络安全或网络安全教育专家对草稿问题的反馈；(3)向至少200名学生管理草稿工具，并对结果使用心理测量学分析方法；(4)根据任务1-3修改问题草稿；(5)对至少1,000名学生实施经修订的工具，并对结果进行心理测量分析；(6)在四次网络安全教育会议上举办讲习班，以提高对工具的认识，并寻求对工具的更多反馈。通过遵循成熟的严格方法开发CCIS，该项目将帮助网络安全教育界开发和改进课程、教学方法和材料。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The University of Maryland, Baltimore County (UMBC) and University of Illinois at Urbana-Champaign (UIUC) propose to evaluate two cybersecurity educational assessment tools: the Cybersecurity Concept Inventory (CCI) and the Cybersecurity Curriculum Assessment (CCA). The CCI assesses the quality of instruction of any first course in cybersecurity; the CCA assesses how well a college curriculum prepares graduates for a career in cybersecurity. The evaluation will consist of student interviews, expert reviews, and statistical analyses to examine the efficacy of these tools. In addition, the project will provide initial insights into the efficacy of educational approaches by comparing student performance relative to their instructors' teaching practices. At present, there is not a rigorous, research-based method for measuring the quality of cybersecurity instruction. Validated assessment tools such as the ones in development at UMBC are essential so that educators have trusted methods for discerning whether efforts to improve student preparation are successful. This validation study will complete an essential part of the broader Cybersecurity Assessment Tools project, which will provide rigorous, evidence-based instruments for assessing and evaluating educational practices. To develop evidence for the validity of the CCI's and CCA's measurement of student conceptual understanding of cybersecurity, the project will complete six tasks for each assessment tool: (1) conduct at least thirty cognitive interviews of students thinking aloud while answering the draft questions, and use results to improve the validity of the questions; (2) collect and use feedback on the draft questions from at least twenty experts in cybersecurity or cybersecurity education; (3) administer the draft tool to at least 200 students and use psychometric methods of analysis on the results; (4) revise the draft questions in light of tasks 1-3; (5) administer the revised tool to at least 1,000 students and use psychometric analysis on the results; and (6) hold workshops at four cybersecurity education meetings to promote awareness of, and to seek additional feedback on the tools. By following well-established rigorous methods for developing CCIs, this project will help the cybersecurity educational community to develop and improve curricula and instructional methods and materials.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.