Today’s CISOs face many daunting tasks in executing their responsibilities. Executives have pointed to the number of untrained workers (87%) as the greatest risk factor for a security breach. In addition to the ongoing global human capital shortage, CISOs and their HR teams are in a difficult position to acquire security professionals that can be onboarded as hit- the-ground-running members of the team to enhance and accelerate organizational cyber resilience.
With a cyberattack on public and private targets in the USA occurring every 39 seconds and some 359,000 American positions remaining unfilled according to (ISC)2, how are CISOs and their HR teams supposed to acquire and retain qualified, validated human capital capabilities?
Security certifications are plentiful and many more are introduced to the market every year. Choosing suitable candidates based only on their CVs and security certifications has become every CISO’s nightmare and finding a good person to hire may often be compared to finding a needle in the haystack.
Top 10 Abilities
While in the USA the NIST NICE competency framework helps develop pathways and granular assessments in relation to different job roles, in Europe and other parts of the world competency frameworks are shaping up as the proverbial Tower of Babel, away from a unified approach and adding to the chaos of security certifications and overall competence assessment.
Started in 2020 and supported by the European Cyber Security Organization (ECSO) and other government and private organizations, “Top 10 Abilities” is an international initiative for the assessment of observable abilities underpinning key job roles in the cyber security industry today.
Top 10 Abilities does not aim to replace nor to undermine the standard approach of skills assessment based on such competency frameworks as NIST NICE. In fact, Top 10 Abilities is aligned to NIST NICE and focuses on the identification of a more manageable subset of core abilities for each job role.
While competency frameworks continue to provide the granular and fine-grained assessment of specific competencies, Top 10 Abilities focus on the top 10 observable and measurable abilities that will give employers reasonable assurance about the suitability of a person for a specific job role.
Rather than focusing on the entire set of competencies required for such a job, the emphasis is put on observable abilities which can be assessed through realistic scenarios, which simulate typical job- specific tasks.
In 2022, through the consultation of CISOs and security professionals from the wider community, Top 10 Abilities will define abilities and realistic simulation assessment scenarios initially for the following job roles:
• Security / Threat Analyst
• SOC Analyst
• Penetration Tester
• Reverse Engineer
• Computer Forensics Engineer
Assessing observation skills through observation
The most critical certifications issued to people today are based on observation and not on the successful scoring of multiple-choice assessments or any other type of automated scoring technology.
Training activities benefits of the latest threat intelligence inputs.
– Gamified exercises occur in a sandbox environment where the latest threats can be safely detonated in a replica environment according to an engaging slate of roll-out activities and events, designed, built, and deployed to reflect real-life use contexts, to apply any available tools of trade, to engage numbers of users and teams over a meaningful period of time.
– Such a slate of exercises runs on seamless functionalities based on high orchestration and high automation, in order to meet the requirements for cost and time- efficient deployment of resources, participants, events.
– Participants’ performance is full tracked and observed before, during and post exercises, with gathering and analysis of metrics, against a competency framework and other applied criteria.
– This also includes processes and technologies tested in an exercise. All resulting data is captured and made shareable with other internal / external business systems, such as for Learning Management, Human Capital, proof of concepts towards development roadmaps, for policy and regulations compliance, and collaboration and coordination frameworks.
More job roles shall be included based on the survey of inputs from the security community.
The Challenge of Traditional Observation
Traditionally, observation has been an effective assessment method, especially with regards to critical skills where people’s lives are possibly at stake. Typical examples include the examinations required to pass for a driving license, a flying license, a PADI diving certificate, etc.
The need to observe comes from the fact that all such assessments are carried out in the real world, on the road, in the sky or under water, and there is no technology today which can make conclusive evaluations without the involvement of a human observer.
However, the following shortcomings are well known when it comes to observation for skills assessment:
• Scaling – The challenge of using observation is that it does not scale up, or at least it does not scale well. The process of obtaining a driving license presents only one bottleneck: the driving practice test, which needs booking one-on-one with an authorized assessor, who will observe the student driving the car and successfully carrying out the required tasks. The driving instructor can only observe one student at a time and must sit in the same car as the student. Driving simulators have begun being adopted to replace the standard live driving test.
• Fairness – Assessment carried out in the context of a real environment is subject to uncontrollable factors that may negatively affect the complexity of the tasks to be carried out by the person being observed. Imagine a driving test taken on a rainy day compared to the same driving test on a sunny day. The assessment experience can be totally different for different students resulting in unequal and unfair assessments.
• Control – The uncontrollable factors influencing the realism of the assessment environment makes it difficult to control the assessment itself, the tasks to be carried out, the order in which they are supposed to be carried out and more.
• Integrity – specific to the Top 10 Abilities methodology to counteract possible ‘cheating’, the management of repeatable scoring methods for Scale and Fairness aims at preventing users being able to find an answer set of IOCs, and signatures online that they buy and regurgitate.
Observing Cyber Security Skills
When it comes to cyber security, the observation process has not been required, traditionally. The world of cyber security professionals has appeased itself with assessments based on a combination of multiple-choice questions, practice-based questions with specifically expected answers and automatic testing and marking capable of churning large numbers of certified professionals out every day.
Unfortunately, the convergence of the cyber and physical worlds into a cyber- physical world is changing the ways we interact with the physical world and the impact that cyber security has on people and nations’ lives. Cyber-kinetic attacks are becoming more and more real and showing us that any cyber security malpractice can have a devastating effect on our everyday lives.
The ransomware cyberattack on the Colonial Pipeline billing system in Alabama in May 2021 made a huge impact on several neighbouring States, where a large number of filling stations ran out of fuel and gas prices surged within a few days of the attack with gas shortages and price spikes in multiple states over several days. The actual ransom event never transited from the IT business office to the OT network, but the company was so concerned it might that they shut down the pipeline prophylactically to investigate.
The debacle cost Colonial at least $4.4 million, the amount its CEO admitted to paying the hackers. Some of this payment was later recovered by the US Govt and returned to Colonial.
Today, certifications and credentials that enable people to handle the protection and management of cyber- physical systems require more and more observable competencies in order to attest to their real proficiency. A number of technology triggers make it possible today – more than it has been until now – to apply effective observation for assessment in the cyber security industry.
High-Fidelity Cyber Security Simulation Environments
Creating a high-fidelity simulation environment that replicates the systems, security controls and cyberattacks that we use and experience every day on the job has become much simpler and more cost effective than building a simulator emulating a car, roads, the traffic, the weather and other external factors that make up a realistic driving scenario.
To be clear, there are many driving simulators out there but we are referring here to an emulator where its user would not be able to tell the difference between reality and simulation.
Furthermore, a cyber security simulation developed to reproduce reality is accessed through the same browsers, remote desktops and tools that would be used by security professionals in their actual workplace. Once inside the simulation environment, it is impossible to tell the difference: it is the actual Matrix!
The ability to create cost-effective and scalable high-fidelity simulation environments relies on the use of next- generation cyber ranges developed on two core technologies:
• Cloud Technology – Cloud technology has supported the use of high-level orchestration and high-level automation for the creation and management of virtual environments. Today, cyber security simulation environments can be brought up and made available in a few seconds at the click of a button.
• Agent-base Attack simulation – This technology has made it possible to replicate all types of cyberattack, from atomic ones, such as brute- force attacks, up to more complex APTs and attack campaigns.
The opportunity to access a true- to-life simulation environment is the premise upon which we can introduce the observation of cyber security skills. Without a realistic environment, skills assessment is limited to ascertaining atomic skills but not the application of those skills to solve a real problem under pressure.
The figure below illustrates the use of our CYBER RANGES solution for the validation and assessment of professional abilities through the use of high-fidelity simulation environments.
Assessing Top 10 Abilities
With Top 10 Abilities, cyber security competence assessment relies strongly on the use of a suitably featured next-generation cyber range for the development and delivery of realistic
simulation environments. To be clear, typical click-and-play cyber range platforms available today on public clouds simply address atomic skills development but lack the ability to offer realistic simulation environments.
Therefore, those platforms are not suitable to assess Top 10 Abilities.
Furthermore, and equally importantly, the next-generation cyber range used must be equipped with an advanced observer module capable of capturing the outcomes of the assessment, while observing the actions carried out during the assessment and providing formative feedback and scoring mapped to one of the Top 10 Abilities.
Automatic Scoring – Use with Care
There are technology solutions that today offer automatic scoring of a person’s performance in the use of specific tools and/or ability to solve given tasks. This is increasingly considered an effective method to evaluate real skills and to somewhat compensate for observation, since “observation” is taken care of by the scoring automation technology.
The great majority of automatic scoring technology works through the use of following components:
• A testing station – this is the (virtual) machine given to students for carrying out the required tasks.
• An agent – The testing station contains a software agent capable of reading every action and/or keystroke of a user on the testing station and comparing them with the correct solution. In most cases, the agent uses signature to evaluate the correct answers. Therefore, if the user chooses a different course of action or carries out the tasks differently from the prescribed solution, no mark will be assigned, and the test will result in a false negative.
• An attack simulator – this is used to test that the user has been able to successfully configure a device or security solution or other systems by sending probes, connection requests and/or simulated attacks.
An automatic scoring engine can provide a good user experience as it does not require a user to enter answers or flag into a scoring system. Users simply go on about their tasks and the system automatically captures and scores their actions.
From a practice-based perspective though, an automatic scoring engine alone provides no stronger assessment than traditional systems based on users having to enter their answers manually. Only if used within the context of a next- generation cyber range, will automatic scoring be more effective towards the assessment of Top 10 Abilities.
It should also be noted that the assessment value of the Top 10 Abilities approach is enriched by the opportunity of relying on simulation scenarios built on up-to-date data from live and real-world attacks.
A Use Case
An early use case of Top 10 Abilities is the FOUR18 Intelligence TradeCraft SOC Analyst Washington State pilot program sponsored by Workforce Snohomish of Snohomish County, WA in partnership with Edmonds College.
Within 4 months the TradeCraft online internship-based program will prepare a cohort of current students and graduates of Edmonds’ cyber security or computer networking programs with the hands-on skills in threat hunting and cyberattack analysis to enter and thrive in the cyber security workforce as newly minted SOC analysts.
The program is structured to offer students and employers a unique experience by engaging them directly but remotely in analyzing live, real-time threats and attack artifacts through FOUR18’s FOURSightTM platform along with high-fidelity attack simulations and labs run on CYBER RANGES by Silensec.
Instead of the traditional inside- out approach to cyber education, the TradeCraft internship program captivates new analysts to think and learn from the outside-in by making learners deep-dive into the MITRE ATT&CK framework using real threats from the wild surfaced in real- time while they are active.
Through a live collaboration platform, employer mentors, professional coaches, and a community of cyber experts engage with the TradeCraft interns as they hunt these threats together to form bonds and impart skills that translate directly into jobs and skill ratings.
The cohort will also be the first analysts to receive the new Junior SOC Analyst Top 10 Abilities digital badge (TTA-JSOC).
Conclusions
Top 10 Abilities is gaining international momentum as it offers CISOs and HR teams the ability to quickly assess the proficiency of cyber security professionals by means of fast, effective assessments of their competence in realistic environments.
Top 10 Abilities provides a measurable, outcome-driven approach to validating human performance and to better identify training gaps to be filled, thus improving the efficiency of workforce development within an organization.
More information on how to participate in Top 10 Abilities at:
Authors:
Dr. Al Graziano, CEO at CYBER RANGES Quantico Cyber Hub
Mark Jaster, CEO at FOUR18 Intelligence Corp
Dr. Aaron Miller, Director, Cyber Bytes Academy
Click the link to learn more about TOAR
