IT audits have a bad rap for being a “tick-box” activity that only focus on surface-level risks and abstract controls. A recent international study, “The Next Generation Cybersecurity Auditor,” found a significant knowledge gap in 151,000 Big Four IT Auditors’ theoretical knowledge and practical skill. 

The study discovered that IT auditors’ lack of hands-on skill in information technology influences data breach likelihood and technical evidence interpretation for critical infrastructure (power, water, communication, and banking). This study distinguished what the auditors know (declarative knowledge) from what the auditor can do (procedural knowledge) by creating declarative and procedural knowledge assessment inquiries.

For instance, this assessment expounded on common concepts like least privilege and separation of duties via task-based activities. This strategy required the respondent to test their knowledge against specific technologies like Microsoft Server, Amazon Web Services (AWS), Palo Alto firewalls, Kubernetes containers, and Microsoft Azure.

The study’s population consisted of IT auditors, IT professionals, and cybersecurity practitioners. The primary target, IT auditors, either currently or previously, worked in first and second-tier audit firms like Deloitte, PwC, Ernst & Young, KPMG, RSM Tenon, BDO, Grant Thorton, and Smith & Williamson. The study collected survey/assessment responses from 108 IT auditors and 108 subject matter experts (SMEs) and can be generalized to 151,000 IT Auditors in the industry.

The study leveraged the National Institute of Standard and Technology (NIST) ’s National Initiative for Cybersecurity Education (NICE) framework and the Skills Framework for the Information Age (SFIA) to measure IT auditor and SME competence. Specifically, this study utilized NICE’s Security Control Assessor & IT Program Auditor work roles, tasks, and knowledge, skills, and abilities (KSAs).

Let’s take a look at the most significant findings from the study. 

IT Auditors Have Sufficient Book Knowledge, But Inadequate Practical Skills

The results indicate that IT auditors and subject matter experts (SMEs) have sufficient declarative knowledge (book knowledge). IT auditors had an average declarative knowledge score of 25.56 (Level 3 – Apply) on the Skills Framework for Information Age (SFIA) scoring model. The SMEs had an average declarative knowledge score of 33.61 (Level 4 – Ensure).

The Skills Framework for Information Age (SFIA) levels for Autonomy and Knowledge are used to grade technical competency.

SME and IT Auditor SFIA Theoretical Knowledge Scores

Conversely, the IT auditors had inadequate levels of procedural knowledge. For example, IT auditors had an average procedural knowledge score of 19.35 (Level 2 – Assist). In contrast, the SMEs performed better on procedural knowledge questions than IT auditors and achieved 25.19 (Level 3 – Apply). These findings suggest that the current education models in IT certification exams, college curricula, certification boot camps, and training seminars do not provide task-based skills to help implementers and assessors improve their procedural knowledge (demonstrable skills).

SME and IT Auditor SFIA Practical Knowledge Scores

IT Auditors' Proficiency Influences Audit Quality & Data Breach Likelihood

Each declarative and procedural knowledge question was associated with missing safeguards reported in notable breaches like the 2019 Capital One data breach and the 2017 Equifax data breach. Therefore, failure to identify the appropriate safeguards or interpret technical evidence would result in unidentified technical risks and increase the data breach likelihood. These results suggest that the cybersecurity, IT, and audit professions should emphasize virtualized and scenario-based training to equip the next generation of cybersecurity professionals. This strategy will prepare implementers and assessors for emerging technologies such as Industry 4.0 and reduce the likelihood of data breaches.

Significant Gap Between IT Auditors' and SMEs' Declarative and Procedural Knowledge

There was a 17% decline from the IT auditors’ declarative to procedural knowledge. In other words, IT auditors had sufficient theoretical knowledge (Level 3 – Apply), but their ability to apply the same concepts procedurally decreased significantly (Level 2 – Assist). These findings suggest a significant knowledge gap exists between IT auditors’ declarative and procedural knowledge in first and second-tier accounting/audit firms.

IT Auditor’s Knowledge Decline from Theoretical Knowledge to Practical Skill

Likewise, there was a 26% decline from the SMEs’ declarative to procedural knowledge. In other words, SMEs had high levels of theoretical knowledge (Level 4 – Enable), but their ability to apply the same concepts procedurally significantly decreased to Level 3 – Apply. These findings suggest a significant knowledge gap between SMEs’ declarative and procedural knowledge.

SME’s Knowledge Decline From Theoretical Knowledge to Practical Skill

Confidence Levels vs. Actual Performance (The Dunning-Kreuger Effect)

IT Auditors' Confidence Did not Align With Their Performance

The IT auditors possessed high confidence levels in interpreting technical evidence to form conclusions on modern information technology systems’ overall security/risk. Conversely, the auditors’ procedural knowledge score (19.35) did not align with their self-perception (confidence) rating (3.98). These results are congruent with previous findings, demonstrating how auditors had overconfidence in audit tasks, even when those activities fell outside their knowledge, skills, and abilities.

SMEs had above-average confidence levels regarding their ability to interpret technical evidence to form conclusions on modern information technology systems’ overall security/risk. The SMEs had a procedural knowledge score of 25.19 (Level 3 – Apply) and a confidence level of 3.50 (Above-average). Although SMEs’ confidence was more aligned with their ability to audit and interpret evidence than IT auditors, there was no significant alignment between their self-perception (confidence) and procedural knowledge.

Age Influences Over-Confidence

This study discovered that age might negatively affect confidence for both IT auditors and SMEs. For example, both SMEs’ and IT auditors’ confidence increased with age. However, no data suggested that higher confidence levels translated to higher performance.

Certain Certifications Influence Over-Confidence

This study discovered that certifications like the Certified Information System Auditor (CISA) and Certified Information Systems Security Professional (CISSP) were associated with 28 IT auditors who rated their self-perception (confidence) as Very Confident. Conversely, IT auditors that rated themselves as Very Confident were novice performers with an overall score of 40.48 (Level 2 – Apply). In addition, this study also discovered that the CISSP certification was associated with 17 SMEs who rated their self-perception (confidence) as Very Confident. Similarly, SMEs who rated themselves as Very Confident were average performers with an overall score of 57.06 (Level 3 –Apply).

The figure below compares the IT auditors and SMEs’ self-perception (confidence levels) to their theoretical knowledge (declarative) and practical skill (procedural). Proficiency for overall performance occurs at SFIA Responsibility Level 3 – Apply (50 to 70). However, declarative and procedural knowledge proficiency occurs at SFIA Responsibility Level 3 – Apply (21 to 30).

Academic Degrees Do Not Significantly Influence IT Auditors' Performance

Higher academic degrees did not significantly influence IT auditors’ overall performance. Although there were no Level 5 (Ensure) IT auditors in overall performance, four of the nine Level 4 (Enable) auditors held bachelor’s degrees, and the remaining five held master’s degrees. The study also had five IT auditors with doctoral degrees. On average, the doctoral participants achieved Level 2 (Assist). These findings suggest that solely using degrees for IT auditors is a flawed way to measure performance. In contrast to mature fields like healthcare, no state and federal boards require students to obtain licensure after graduation. Most importantly, obtaining degrees does not illuminate the student’s actual grades, and many college courses may curve the class’s scores.

Academic Degrees Do Not Significantly Influence SMEs’ Performance

There were six Level 5 (Ensure) respondents from the SME perspective. Four respondents had master’s degrees, and two held bachelor’s degrees. There were 15 Level 4 (Enable) SMEs. One had no degree with some college credit, three participants had bachelor’s degrees, and the remaining 11 had master’s degrees. Lastly, one SME had a doctorate and achieved Level 3 (Assist). These results indicate that academic degrees do not substantially improve performance for IT auditors or SMEs. This disparity could be explained by the curriculum, tasks, and benchmarks utilized in college courses.

Top Performers Had a Combination of Implementing and Auditing Experience

Individual certifications and licenses did not substantially influence performance. However, certain vendor-specific, cyber, and IT audit certifications potentially improved performance in professionals with hands-on experience implementing and auditing technology. For example, the Level 4 (Enable) IT auditors had both implementing and IT auditing experience. In addition, all Level 4 IT Auditors also held the Certified Information Systems Auditor (CISA) certification and credentials by (ISC)2, CompTIA, EC Council, GIAC, and The Open Group (e.g., TOGAF).

Additional Years of Exposure (Experience) Did Not Improve Performance

Simply utilizing years of experience (exposure) did not yield increased declarative and procedural knowledge performance. The study discovered that using years of experience was too broad and did not consider the hours spent performing job tasks in an eight-hour workday.

This study found that the average employee only spends 2 to 4 hours completing the activities listed in their job description. These findings suggest that although someone may have more years of exposure, they could have less task-based experience than other professionals with fewer years of exposure.

Previous research indicates that reviewing additional years of experience or seniority is a poor proxy for auditing or implementing expertise. Moreover, the world’s top expert on expertise, Anders Ericsson, says that acquiring 10,000 hours or numerous years of exposure in a particular role does not make one an expert.

In fact, the study debunked the years of experience fallacy. The research found that neither age nor additional years of experience influenced declarative or procedural knowledge. However, the study found that the number of times a professional performed a task along with the task quality, capacity, and speed was a more objective way to measure experience versus using time spent on earth or time spent in a certain role. Conversely, employers who utilize years of exposure to hire and promote candidates are not employing objective measures. Instead, enterprises that subjectively review resumes and prioritize years of exposure may hire individuals with novice task performance.

Next-Gen Auditors Can Protect the Nation's Critical Infrastructure

The role of the IT auditor is too broad to be effective in the 33 specialty areas (subdisciplines) of cybersecurity. The rapid introduction of emerging technologies, such as Industry 4.0, will require competent assessors to identify technical risks and recommend appropriate safeguards.

Today, the role of the IT auditor is analogous to a general healthcare provider. For example, the physician performs your routine checkup, and if they discover anything awry, they refer you to a specialist like a surgeon, neurologist, or other types of physicians. In contrast, the industry requires the IT auditor to play the role of the general healthcare provider and numerous specialists.

Most importantly, this study discovered that the lack of vendor-specific certifications and hands-on experience implementing technology potentially influenced the auditor’s evidence interpretation and audit quality.

Creating Specialty Cybersecurity Auditor Roles

The future of the IT auditing profession should create specialty positions and hybrid roles to improve technical competency and audit quality. For example, auditors required to audit artificial intelligence, machine learning, operating systems, networks, and firewalls should have hands-on training and implementation skills in the same technologies. These competencies enable the auditor to identify material misstatements, identify technical risks, communicate with SMEs, and recommend safeguards that reduce data breach likelihood.