This task proposes a conceptual framework for the monitoring and evaluation of a cybersecurity awareness program. The framework provides guidelines and practical advice on “what to do in each phase” of a cybersecurity awareness program. Moreover, it also answers “what to expect in each phase” i.e., expected outputs and outcomes, after those guidelines and advice are followed. The earlier information can be helpful for designing an effective awareness program, whereas the latter information will more specifically facilitate monitoring and evaluation of the program. In addition, it provides evaluation criteria of two cybersecurity awareness mechanisms, which are posters (simplest and one of the least interactive media that is known to almost everyone), and serious games (highly interactive media that is gaining popularity).
Cybersecurity awareness has existed for a long time, probably as long as cybersecurity itself. However, it still fails to yield the expected outcomes. This could be possible because most cybersecurity awareness programs are limited to a mere delivery of security information (what to do and what not to do) or compliance with standards, policies, and procedures. Indeed, access to security information is an important step, however, this does not guarantee that the information will be absorbed by the audience and more importantly will translate into actions. In order to improve the information absorption and its probability of translating into actions, this study proposes a conceptual framework as well as provides guidelines and practical advice leveraging the knowledge from various disciplines.
- The team should be inclusive with clearly defined roles, responsibilities, and accountabilities for each member. Moreover, it is advisable to have two full-time staff members, but one full-time staff member is a must for CSA. The individual(s) should be equipped with both technical and soft skills, and also be aware of the context.
- The goals should be clear and simple, and its objectives should be SMART (Specific, Measurable, Attainable, Relevant, Time-bound).
- The audience should be grouped preferably based on their beliefs and cybersecurity expertise.
- The program should receive appropriately high priority in terms of support and participation from the leaders, and budget allocation.
- The selected topics should cover threats prevalent to the audience roles and responsibilities, that include both common and new emerging threats.
- The topics relevant to critical security roles and controls, specific to the organization role and risk profile, relevant to critical projects, neglected by the audience, and with resources readily available should get the high priority.
- The message intensiveness or complexities should be adjusted from general to in-depth depending on the audience.
- The message framing should consider human psychological (cognitive, affective, and different biases) and other factors (usability and user experience) that influence the message reception and interpretation by the audience.
- The message delivery methods should be cost-effective; have a broad outreach; support diversity and inclusiveness; be easy and simple to develop, operate, manage, and update; include standardized assessment and feedback features; support information richness; require minimal additional requirements; and interest and motivate the audience.
- The message communication should consider the psychological and other influencing factors that increase the audience’s participation and drive them to practice (or translate into actions) the security knowledge they have learned from the program.
- The enforcement approach used to non-compliance should be a soft approach (mainly using intrinsic incentives) unless a specific need arises for a tough approach.
- The program should be organized periodically, at least once every six months except for responding to new events and situations.
- The lessons learned during the different phases of the program should be properly captured, debriefed, and documented for the effective transfer and use of information.
- The evaluation should measure all four indicators (impact, sustainability, accessibility, and monitoring) to determine the overall effectiveness of the program. Moreover, the measurable parameters selected for each indicator should be economical to gather, consistent to measure, expressible in cardinal number and unit, and contextually specific.
- The program should be adjusted in accordance with the changes in the cybersecurity scenarios. It should also take into consideration the lessons learned and weaknesses identified from monitoring and evaluation.
- Topic (specific)
- Overall information (credible and consistent, complete, up to date)
- Message framing (positive, direct, descriptive)
- Suggestion quality (doable, convenience)
- Content presentation (clarity, conciseness, well-structured, use multiple representations, understandability of the main message)
- Localization
- Style and formatting (visibility of overall message, placement of the main message, color used, typography used, Image used)
- Effectiveness
- Entertainment
- Legal and ethical assessment
- Videos [INSERT LINK]
- Scientific dissemination [INSERT LINK]
| --- | Governance and Capacity Building | Trustworthy Ecosystems of Systems | Trust-Building Blocks | Disruptive Emerging Develpment |
|---|---|---|---|---|
| Asset 1 | ✔️ | - | - | - |
| --- | Collaborative Networks | Education & Training | Certification | Secure Platforms of Platforms | infrastructure Protection | Holistic Data Protection | AI-based Security | Systems Security & Security Lifetime Management | Secure Architectures for Next Generation Communication | Secure Quantum Technologies | Secure AI Systems | Personalized Privacy Protection |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Conceptual Framework & Guidelines | --- | ✔️ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Criteria for Poster Evaluation | --- | ✔️ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Criteria for Serious Game Evaluation | --- | ✔️ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
1 - A. Skarmeta, “D3.1 Common Framework Handbook 1”, CyberSec4Europe, 2019.