Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review

Spencer Offenberger; Geoffrey L. Herman; Peter Peterson; Alan T. Sherman; Enis Golaszewski; Travis Scheponik; Linda Oliva

doi:10.1109/FIE43999.2019.9028652

Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review

Spencer Offenberger, Geoffrey L. Herman, Peter Peterson, Alan T. Sherman, Enis Golaszewski, Travis Scheponik, Linda Oliva

Computer Science (Duluth)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Scopus citations

Abstract

We analyze expert review and student performance data to evaluate the validity of the Cybersecurity Concept Inventory (CCI) for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the CCI, and 142 students from six different institutions took the CCI as a pilot test. The panel reviewed each item of the CCI and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the CCI to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the CCI. This evaluation showed that the CCI is sufficiently reliable for measuring student knowledge of cybersecurity and that the CCI may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the CCI.

Original language	English (US)
Title of host publication	2019 IEEE Frontiers in Education Conference, FIE 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728117461
DOIs	https://doi.org/10.1109/FIE43999.2019.9028652
State	Published - Oct 2019
Event	49th IEEE Frontiers in Education Conference, FIE 2019 - Covington, United States Duration: Oct 16 2019 → Oct 19 2019

Publication series

Name	Proceedings - Frontiers in Education Conference, FIE
Volume	2019-October
ISSN (Print)	1539-4565

Conference

Conference	49th IEEE Frontiers in Education Conference, FIE 2019
Country/Territory	United States
City	Covington
Period	10/16/19 → 10/19/19

Bibliographical note

Publisher Copyright:
© 2019 IEEE.

Keywords

Cybersecurity Assessment Tools (CATS) Project
Cybersecurity Concept Inventory (CCI)
Cybersecurity education
assessment validation

Access

10.1109/FIE43999.2019.9028652

OpenUrl availability

Full text

Cite this

Offenberger, S., Herman, G. L., Peterson, P., Sherman, A. T., Golaszewski, E., Scheponik, T., & Oliva, L. (2019). Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review. In 2019 IEEE Frontiers in Education Conference, FIE 2019 Article 9028652 (Proceedings - Frontiers in Education Conference, FIE; Vol. 2019-October). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/FIE43999.2019.9028652

Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review. / Offenberger, Spencer; Herman, Geoffrey L.; Peterson, Peter et al.
2019 IEEE Frontiers in Education Conference, FIE 2019. Institute of Electrical and Electronics Engineers Inc., 2019. 9028652 (Proceedings - Frontiers in Education Conference, FIE; Vol. 2019-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Offenberger, S, Herman, GL, Peterson, P, Sherman, AT, Golaszewski, E, Scheponik, T & Oliva, L 2019, Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review. in 2019 IEEE Frontiers in Education Conference, FIE 2019., 9028652, Proceedings - Frontiers in Education Conference, FIE, vol. 2019-October, Institute of Electrical and Electronics Engineers Inc., 49th IEEE Frontiers in Education Conference, FIE 2019, Covington, United States, 10/16/19. https://doi.org/10.1109/FIE43999.2019.9028652

Offenberger S, Herman GL, Peterson P, Sherman AT, Golaszewski E, Scheponik T et al. Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review. In 2019 IEEE Frontiers in Education Conference, FIE 2019. Institute of Electrical and Electronics Engineers Inc. 2019. 9028652. (Proceedings - Frontiers in Education Conference, FIE). doi: 10.1109/FIE43999.2019.9028652

@inproceedings{f7648fc3147440699e53d2daf26600e3,

title = "Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review",

abstract = "We analyze expert review and student performance data to evaluate the validity of the Cybersecurity Concept Inventory (CCI) for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the CCI, and 142 students from six different institutions took the CCI as a pilot test. The panel reviewed each item of the CCI and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the CCI to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the CCI. This evaluation showed that the CCI is sufficiently reliable for measuring student knowledge of cybersecurity and that the CCI may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the CCI.",

keywords = "Cybersecurity Assessment Tools (CATS) Project, Cybersecurity Concept Inventory (CCI), Cybersecurity education, assessment validation",

author = "Spencer Offenberger and Herman, {Geoffrey L.} and Peter Peterson and Sherman, {Alan T.} and Enis Golaszewski and Travis Scheponik and Linda Oliva",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 49th IEEE Frontiers in Education Conference, FIE 2019 ; Conference date: 16-10-2019 Through 19-10-2019",

year = "2019",

month = oct,

doi = "10.1109/FIE43999.2019.9028652",

language = "English (US)",

series = "Proceedings - Frontiers in Education Conference, FIE",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2019 IEEE Frontiers in Education Conference, FIE 2019",

}

TY - GEN

T1 - Initial Validation of the Cybersecurity Concept Inventory

T2 - 49th IEEE Frontiers in Education Conference, FIE 2019

AU - Offenberger, Spencer

AU - Herman, Geoffrey L.

AU - Peterson, Peter

AU - Sherman, Alan T.

AU - Golaszewski, Enis

AU - Scheponik, Travis

AU - Oliva, Linda

PY - 2019/10

Y1 - 2019/10

N2 - We analyze expert review and student performance data to evaluate the validity of the Cybersecurity Concept Inventory (CCI) for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the CCI, and 142 students from six different institutions took the CCI as a pilot test. The panel reviewed each item of the CCI and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the CCI to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the CCI. This evaluation showed that the CCI is sufficiently reliable for measuring student knowledge of cybersecurity and that the CCI may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the CCI.

AB - We analyze expert review and student performance data to evaluate the validity of the Cybersecurity Concept Inventory (CCI) for assessing student knowledge of core cybersecurity concepts after a first course on the topic. A panel of 12 experts in cybersecurity reviewed the CCI, and 142 students from six different institutions took the CCI as a pilot test. The panel reviewed each item of the CCI and the overwhelming majority rated every item as measuring appropriate cybersecurity knowledge. We administered the CCI to students taking a first cybersecurity course either online or proctored by the course instructor. We applied classical test theory to evaluate the quality of the CCI. This evaluation showed that the CCI is sufficiently reliable for measuring student knowledge of cybersecurity and that the CCI may be too difficult as a whole. We describe the results of the expert review and the pilot test and provide recommendations for the continued improvement of the CCI.

KW - Cybersecurity Assessment Tools (CATS) Project

KW - Cybersecurity Concept Inventory (CCI)

KW - Cybersecurity education

KW - assessment validation

UR - http://www.scopus.com/inward/record.url?scp=85076112446&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85076112446&partnerID=8YFLogxK

U2 - 10.1109/FIE43999.2019.9028652

DO - 10.1109/FIE43999.2019.9028652

M3 - Conference contribution

AN - SCOPUS:85076112446

T3 - Proceedings - Frontiers in Education Conference, FIE

BT - 2019 IEEE Frontiers in Education Conference, FIE 2019

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 16 October 2019 through 19 October 2019

ER -

Initial Validation of the Cybersecurity Concept Inventory: Pilot Testing and Expert Review

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access

OpenUrl availability

Other files and links

Fingerprint

Cite this