ISC2 AI Security Pilot Exam · Free study guide
Domain 2 Study Notes: AI Governance, Ethics, Risk and Compliance
How to use these notes
Domain 2 is where the exam stops asking how AI works and starts asking who decides, who's accountable, and what the law requires. You don't need a governance or legal background to handle it; every term is defined below, and if you've already studied for the IAPP AIGP, much of this will feel familiar. I've followed the ISC2 outline section by section, and each section ends with how I expect it to show up on the exam.
One idea runs through the whole domain, so I'll state it up front: senior management holds final accountability for AI risk. Committees advise, owners operate, and security professionals recommend, but leadership owns the outcome. When an exam question asks who is responsible, start there.
Everything here is my own work, built from public frameworks and regulations. If a reference book frames something differently, trust the book.
2.1 AI governance, frameworks and policies
Core regulations and standards
The first thing to know about these four is what kind of thing each one is, because questions often turn on whether something is a law, a voluntary framework or a certifiable standard.
| Name | What it is | What it does |
|---|---|---|
| EU AI Act | A law, binding in the EU and on anyone placing AI on the EU market | Sorts AI systems by risk and sets obligations for each tier |
| NIST AI RMF | A voluntary US framework | Gives organizations a structured way to manage AI risk |
| ISO/IEC 42001 | An international standard you can be certified against | Defines an AI management system, the way ISO 27001 does for information security |
| GDPR | A law, binding on anyone processing EU residents' personal data | Governs personal data, including data used to train and run AI |
The EU AI Act uses four risk tiers. Unacceptable risk is banned outright; that covers practices like social scoring, manipulative techniques that cause significant harm, untargeted scraping of facial images, and inferring emotions at work or in schools. High risk carries most of the obligations and covers areas like critical infrastructure, education, employment, credit scoring and law enforcement. Limited risk carries transparency duties, so people must know when they're talking to a chatbot or looking at a deepfake. Minimal risk isn't regulated. The Act also separates providers, who build AI systems and carry most of the obligations, from deployers, who use them professionally and carry fewer. General-purpose AI models have their own duties around documentation, copyright and a summary of training content, with extra testing and incident reporting for the largest models.
The timeline matters because it moved. The Act entered into force in August 2024, and prohibitions applied from February 2025. In 2026 the EU adopted the Digital Omnibus on AI, which pushed obligations for stand-alone high-risk systems to December 2, 2027 and for AI embedded in regulated products to August 2, 2028, while the transparency duties stayed on their original August 2026 date.
The NIST AI RMF organizes AI risk management into four functions:
- Govern sets the culture, policies, roles and accountability, and it runs across the other three.
- Map establishes context: what the system is for, who it affects, and what could go wrong.
- Measure analyzes and tracks the risks with tests and metrics.
- Manage prioritizes the risks and acts on them.
NIST pairs the framework with a Playbook of suggested actions and a Generative AI Profile (AI 600-1) for risks specific to GenAI.
ISO/IEC 42001 is the management-system standard: it asks you to set AI policy and objectives, assess risk, apply controls and keep improving, in the same plan-do-check-act cycle as ISO 27001. Its value is that an outside auditor can certify you against it.
GDPR applies whenever personal data is involved. The points that come up most for AI are needing a lawful basis to process the data, running a Data Protection Impact Assessment (DPIA) for high-risk processing, and the right not to be subject to a decision based solely on automated processing when it has legal or similarly significant effects.
AI governance versus IT governance
IT governance grew up around systems that behave the same way every time, so it focuses on assets, change control and availability. AI governance keeps all of that and adds problems IT governance never had to solve. Outputs are probabilistic, so the same input can produce different results. Models change as data changes, so approval at launch isn't enough and monitoring has to continue. Decisions can affect people's rights, so ethics, fairness and explainability enter the picture, along with legal, HR and compliance teams that IT governance rarely involved. And approval happens at the level of each use case, since the same model can be harmless in one use and high-risk in another.
Policies, standards and procedures
The usual hierarchy applies to AI. A policy states management's intent and is mandatory, like "all AI use must be approved before deployment." A standard sets specific mandatory requirements, like "only models from the approved registry may be used with confidential data." A procedure gives the step-by-step method, like how to submit a use case for review. A guideline is recommended practice that isn't mandatory. Typical AI policies cover acceptable use, what data may be used with AI, buying third-party AI, building and deploying models, and disclosing AI use to the people it affects.
Centralized AI use-case inventories
A use-case inventory is a single register of every AI system and use in the organization, recording the owner, purpose, data involved, model or vendor, risk tier, status and next review date. You can't govern what you don't know about, so the inventory is the foundation for risk tiering, regulatory classification, incident response and finding shadow AI.
Organizational roles
- The Chief AI Officer is the executive accountable for AI strategy and governance across the organization.
- AI Champions are people inside business units who promote good practice, help colleagues use AI well, and raise issues early.
- The AI Ethics Officer leads ethical review of AI uses, including fairness and impact on people.
Most organizations also run a cross-functional AI governance council or committee that reviews and approves higher-risk uses, alongside model owners and data owners who are accountable for specific systems.
2.2 Responsible AI principles and ethics
Core principles
Most responsible AI frameworks arrive at a similar list, and the outline uses this one:
- Fairness means the system treats people and groups equitably and doesn't produce unjustified differences in outcomes.
- Reliability and safety mean it performs as intended, including under conditions nobody planned for, and fails in ways that don't hurt people.
- Privacy and security mean it protects the data it uses and holds up against attack.
- Inclusiveness means it works for people across abilities, languages and backgrounds, and those people were considered when it was designed.
- Transparency means people know when AI is involved, what it does and where its limits are.
- Accountability means named people answer for how the system behaves.
Principles only help once they turn into requirements someone can test. "Be fair" does nothing on its own; "approval rates across groups stay within a set threshold, measured every quarter" gives a team something to build and an auditor something to check.
Where bias comes from
Bias in AI means systematic error that favors or disadvantages certain people or groups. It usually enters through the data, long before anyone writes a line of code.
| Type | What happens | Example |
|---|---|---|
| Historical bias | The data faithfully records past discrimination | A hiring model trained on ten years of hires that favored men |
| Representation bias | Some groups are missing or under-sampled | Face recognition trained mostly on lighter-skinned faces |
| Measurement bias | The label is a poor stand-in for what you care about | Using arrests as a measure of crime |
| Proxy variables | A neutral-looking feature carries a protected one | ZIP code tracking race |
| Evaluation bias | The test set doesn't look like real users | A model tested only on native English speakers |
| Automation bias | People over-trust the output | A reviewer approves every recommendation without reading it |
Protected characteristics are the categories anti-discrimination law covers, such as race, sex, age, religion, disability and national origin. Deleting those columns from the training data rarely fixes anything, because proxies carry the same signal back in.
Mitigating bias
There are three points where you can intervene:
- Pre-processing fixes the data: rebalancing or reweighting samples, collecting more data for under-represented groups, and removing or transforming proxies.
- In-processing changes training, for example by adding a fairness constraint to what the model optimizes.
- Post-processing adjusts outputs, for example by setting different decision thresholds. This can raise legal questions of its own, so counsel should be involved.
You measure fairness with metrics. Demographic parity asks whether groups receive positive outcomes at similar rates. Equal opportunity asks whether qualified people in each group are approved at similar rates. Equalized odds asks for both similar true positive rates and similar false positive rates. These metrics conflict, and in most real situations you can't satisfy all of them at once, so picking one is a governance decision that should be documented and shouldn't be left to the data science team alone.
Disparate impact describes a neutral-looking practice that harms a protected group more than others. In the US, the four-fifths rule is a common screen: if one group's selection rate is below 80% of the highest group's rate, the practice needs a closer look. Testing has to continue after launch, because drift can bring bias back.
Transparency and explainability
These three terms get mixed up, and the exam will expect you to keep them apart:
- Transparency is openness about the system as a whole: that AI is in use, what it's for, what data it relies on and what it can't do.
- Explainability is the ability to say why the system produced a particular output. A local explanation covers one decision; a global explanation covers how the model behaves overall.
- Interpretability is how well a person can understand the model's inner workings directly. A small decision tree is interpretable; a large neural network isn't.
For models that aren't interpretable, tools like SHAP and LIME estimate which inputs drove an output. They're approximations, so treat them as evidence rather than proof. Documentation supports transparency too: a model card records a model's purpose, performance across groups and known limitations, and a datasheet does the same for a dataset.
The law sometimes requires explanations. US lenders must give applicants the specific reasons for a credit denial, and "the algorithm decided" doesn't satisfy that. For decisions that cost someone a loan, a job or their benefits, I'd choose the model I can explain, even if it gives up some accuracy.
Systemic impacts and tradeoffs
Some effects only show up at the scale of society:
- Environmental cost. Training and running large models uses a lot of electricity, and data centers use water for cooling. Over a model's life, everyday use can cost more than training did. Picking a smaller model when it's good enough is a sustainability decision.
- Work and skills. Automation can displace jobs and erode the skills of the people who used to do the work.
- Trust in information. Synthetic content at scale makes it harder to know what's real.
- Concentration. A handful of providers control the largest models, which creates dependency for everyone downstream.
Most responsible AI decisions are tradeoffs, and the exam rewards recognizing them:
| You gain | You may give up |
|---|---|
| Accuracy from a complex model | Explainability |
| Privacy from adding noise or removing data | Accuracy and usefulness |
| Fairness on one metric | Fairness on another |
| Speed from full automation | Human judgment and the chance to catch errors |
| Personalization | Privacy |
2.3 AI risk management
Measuring and mapping risk
Risk combines how likely something bad is with how bad it would be: likelihood times impact. You can score it three ways. Qualitative analysis uses ratings like high, medium and low; it's fast but subjective. Quantitative analysis puts money on it, and the classic formula is annualized loss expectancy (ALE), which multiplies the single loss expectancy (SLE), the cost of one incident, by the annualized rate of occurrence (ARO), the number of times a year you expect it to happen. Semi-quantitative analysis sits between the two and uses numeric scores, say 1 to 5, for likelihood and impact.
AI makes the numbers harder. There's little historical loss data, the same model can behave differently on similar inputs, and some harms, like damage to someone's rights or to public trust, don't convert into dollars. Most organizations score AI risk qualitatively or semi-quantitatively and save dollar figures for risks with a clear price, such as regulatory fines or downtime.
Risk mapping ties each risk to its context: the use case, the data, the model, the people affected and the systems it connects to. This is the Map function of the NIST AI RMF. The result usually lands in a risk register, a list of risks with an owner, a score, a treatment and a status for each.
Impact assessments
An impact assessment asks what happens to people and to the organization when the system works as designed and when it fails. You'll see three kinds:
- An algorithmic impact assessment (AIA) evaluates how an automated decision system affects people and their rights, before it's deployed.
- A DPIA is required under GDPR for high-risk processing of personal data.
- A fundamental rights impact assessment (FRIA) is required by the EU AI Act for certain deployers of high-risk systems, including public bodies and organizations using AI for credit scoring or life and health insurance pricing.
The outline names two factors that drive an impact rating up. Blast radius is how far the damage spreads if the system fails or is compromised: how many people it touches, what data it can reach, which systems it connects to and what permissions it holds. A chatbot that reads a public FAQ has a small blast radius; one connected to the HR system with write access has a large one. Autonomous actions matter because a system that only recommends leaves a person between the output and the consequence. A system that acts on its own, sending email, moving money or changing configurations, removes that checkpoint, so an error lands directly.
Risk tiers for use cases
Risk tiering matches the amount of scrutiny to the amount of risk, so a low-stakes summarizer doesn't go through the same process as a hiring model. The outline uses two dimensions: human impact, meaning how much the outcome affects people's rights, safety, money or opportunities, and technical complexity, meaning how complex, autonomous and connected the system is.

I'd let human impact set the floor. A simple scoring rule that screens job applicants deserves more scrutiny than a complex internal tool that summarizes engineering logs, because only one of them can cost someone a job. Higher tiers bring senior approval, deeper testing, bias evaluation, stronger human oversight and more frequent monitoring.
Risk treatment
AI risks get the same four treatments as any other risk:
| Treatment | What it means | AI example |
|---|---|---|
| Avoid | Don't do the risky activity | Deciding not to use AI for parole recommendations |
| Mitigate | Reduce likelihood or impact with controls | Input filtering, output checks, human review, rate limits, least-privilege tool access |
| Transfer | Shift the financial impact to another party | Indemnity terms in a model provider contract, or cyber insurance |
| Accept | Live with the risk, on the record | A risk owner signs off on residual hallucination risk in an internal drafting tool |
Transfer moves the cost, and accountability stays with you. Acceptance has to come from someone with the authority to accept it, in writing, and within the organization's risk appetite, the amount of risk leadership is willing to take on. Whatever remains after controls is residual risk. Some AI risks can't be driven to zero with today's technology; prompt injection has no complete fix, and models still hallucinate. So layered controls followed by a documented acceptance at the right level is normal.
Reassessment triggers
An AI risk assessment goes stale faster than most. Reassess when:
- the model changes, whether it's retrained, fine-tuned or replaced by a new vendor version;
- monitoring detects behavioral drift, meaning outputs, accuracy or fairness metrics have moved;
- a vulnerability is disclosed in the model, a library it depends on or a technique that affects it, and a new jailbreak method counts;
- the system gets a new data source, a new group of users, a new purpose or a new connection to another system;
- an incident or near miss happens; or
- a relevant regulation changes.
Scheduled reviews still happen on top of these, more often for higher tiers.
2.4 Data privacy, IP and regulatory obligations
Regulated data types
| Type | What it is | Common rules |
|---|---|---|
| PII | Personally identifiable information: anything that identifies a person directly, like a name or ID number, or in combination, like birth date plus ZIP code | GDPR, US state privacy laws |
| PHI | Protected health information: health data tied to a person and held by a healthcare provider, insurer or their service providers | HIPAA in the US |
| Financial data | Account and card numbers, credit history and other customer financial information | PCI DSS, GLBA |
| Export-controlled data | Technical data about defense items or dual-use technology | ITAR, EAR in the US |
AI adds new ways for this data to leak. Once regulated data goes into a prompt, a training set or a fine-tune, a model can repeat it in an output, a vendor may keep it in logs, and it may be processed in another country. A few practical rules follow. Sending PHI to an AI vendor requires a business associate agreement (BAA) under HIPAA. Sending export-controlled technical data to an AI service whose servers or staff sit outside the US can count as an export. And the safest data to put in a model is data you never needed in the first place, which is the principle of data minimization.
IP, copyright and fair use
Copyright protects original expression, and training a model involves copying works. In the US, fair use is a legal defense that courts weigh on four factors: the purpose of the use, including whether it turns the work to a new purpose; the nature of the work; how much was used; and the effect on the market for the original. Courts are still working out how fair use applies to AI training, and early rulings have gone different ways depending on facts like how the data was obtained and whether the AI competes with the original. In the EU, a text and data mining exception allows training unless the rights holder has opted out, and the AI Act requires general-purpose model providers to respect those opt-outs.
The output side has questions too. The US Copyright Office's position is that material generated by AI without meaningful human authorship can't be copyrighted. Models and datasets also come with licenses that may forbid commercial use or certain applications, and those terms bind you. And trade secrets lose protection when employees paste confidential code or plans into a public chatbot that keeps the input.
Data derivation and regulatory contamination
Derived data is anything created from other data: embeddings, features, summaries, synthetic data and the trained model itself. Derived data can inherit the obligations of its source. A model fine-tuned on customer records may memorize and repeat them, and an embedding of a medical note is still built from PHI.
Regulatory contamination is what happens when regulated data enters a larger system and pulls the whole thing into scope. One export-controlled design document dropped into a company-wide RAG index can make the index, and everyone's access to it, an export control problem. A handful of patient records in a general training set brings HIPAA along. The controls are to classify data before it's ingested, keep regulated data in separate pipelines, and track data lineage, the record of where data came from and everything it flowed into.
Retention and deletion
A retention schedule for AI has to cover more than the source data. It also covers training sets, prompts and outputs, logs, vector stores and old model versions. Many AI providers keep prompts and outputs for a period, for abuse monitoring, unless the contract says otherwise.
Deletion is where AI gets awkward. Under GDPR's right to erasure, a person can ask you to delete their data. Removing a record from a database is easy, and removing it from a vector store is manageable if you tracked which entries came from it. Removing its influence from trained model weights is hard, and today that usually means retraining. Plan for deletion before training, because afterward your options shrink. A legal hold overrides normal deletion when litigation is expected.
Consent and public data
Under GDPR, consent must be specific, informed and freely given, and consent for one purpose doesn't stretch to another. Someone who agreed to let you process their data to deliver a service didn't agree to have it train your model; that's the principle of purpose limitation. Consent is one of six lawful bases under GDPR, and organizations often rely on another, legitimate interests, for training, which requires a documented balancing test.
Public data isn't free to use. Personal data posted publicly is still personal data under GDPR, and scraping it can also breach a site's terms of service or copyright. A facial recognition company that scraped billions of public photos has been fined by several European data protection authorities.
2.5 Third-party and supply chain risk
Due diligence on AI providers
Most organizations will buy far more AI than they build, so this section matters even if you never train a model. Third-party due diligence is the review you do before you rely on an outside provider. For AI, the questions go beyond the usual security questionnaire:
- Will the provider train on our data, and how long do they keep prompts and outputs?
- Where is data processed and stored, and which subprocessors (the provider's own vendors) touch it?
- What independent assurance do they have, such as a SOC 2 Type II report, ISO 27001 or ISO/IEC 42001 certification?
- Is there a model card or system card showing evaluations, known limitations and safety testing?
- How do they handle model updates, deprecations and incidents, and what's their track record?
Provider controls are the safeguards the vendor operates on your behalf: access control, encryption, tenant isolation, content filtering, abuse monitoring and logging. Due diligence checks that those controls exist and work, and the evidence for that comes from an audit report instead of a sales deck.
Contracts, SLAs and notification rights
You have the most bargaining power before you sign, so the contract is where AI risk gets transferred or limited. Terms to look for:
- Security SLAs that commit to uptime, response times and how fast vulnerabilities get fixed.
- Incident notification rights that require the provider to tell you about a breach or model incident within a set time. Under GDPR, a processor must notify the controller without undue delay, and contracts usually put a number on it.
- A commitment not to train on your data, plus data residency and deletion when the contract ends.
- Advance notice before the model behind the service changes or is retired.
- Right to audit, or at least the right to receive independent audit reports.
- Indemnity if the provider's outputs infringe someone's IP.
- An exit plan, so you can get your data and configurations out.
SBOM and AIBOM
A software bill of materials (SBOM) lists every software component in a system and its version, in standard formats like SPDX or CycloneDX. An AI bill of materials (AIBOM), also called an ML-BOM, extends that to AI: the base model and its version and source, the datasets used for training and tuning, the libraries and frameworks, and the licenses for each. CycloneDX and SPDX both support AI components now.
The point of a BOM is speed. When someone announces a backdoored model, a poisoned dataset or a vulnerable library, the BOM tells you within minutes where you're exposed. AI supply chains have their own threats: model files on public hubs that run malicious code when loaded, packages with names one letter off from popular ones, and datasets someone has tampered with. Verifying hashes, pulling models only from approved sources and preferring file formats that can't execute code all reduce that risk.
Shared responsibility
The shared responsibility model divides security duties between you and the provider, and the line moves depending on how you consume AI:
| Layer | Self-hosted open model | Cloud AI platform | AI API | SaaS with built-in AI |
|---|---|---|---|---|
| Governance, approved uses, who has access | You | You | You | You |
| The data you put in | You | You | You | You |
| Application, prompts, output handling | You | You | You | Mostly vendor; settings are yours |
| Model selection, tuning, weight security | You | Shared | Vendor | Vendor |
| Infrastructure and hosting | You | Vendor | Vendor | Vendor |
Read the top two rows twice. However much you hand off, the use case, your data and accountability for the outcome stay with you.
Trusted execution environments and confidential computing
Encryption has long covered data at rest and in transit. Confidential computing covers the third state, data in use, by processing it inside a trusted execution environment (TEE): an isolated, hardware-protected area of a processor where code and data stay encrypted in memory and hidden from the operating system, the hypervisor and the cloud provider's administrators. Before you send data in, attestation gives you cryptographic proof that the expected code is running inside a genuine TEE. CPUs from Intel and AMD offer TEEs, and recent NVIDIA data center GPUs extend the idea to AI workloads.
For AI, TEEs let you run inference on sensitive data in someone else's cloud, protect valuable model weights from theft, and let several parties train on combined data without seeing each other's records. They have limits: a flaw in the code running inside the TEE is still a flaw, researchers have demonstrated side-channel attacks against some TEEs, and there's a performance cost.
2.6 Human oversight, accountability and transparency
Levels of human oversight
- Human-in-the-loop (HITL) means a person must approve each decision before it takes effect. The AI recommends and the human decides.
- Human-on-the-loop (HOTL) means the AI acts on its own while a person monitors it and can step in or shut it down.
- Human-out-of-the-loop means no person is involved while the system runs, and people only review results afterward.

The right level depends on the stakes and on whether a decision can be reversed. A wrong spam filter decision costs someone a minute in their junk folder; a wrong credit denial can cost them a home. The EU AI Act requires high-risk systems to be designed for effective human oversight, including the ability to disregard or reverse an output and to stop the system.
The word effective is doing the work there. A reviewer who approves 400 AI recommendations an hour is a rubber stamp, and that's automation bias in action. Meaningful oversight needs four things: the authority to override, enough information to judge the output, enough time to think, and training on what the system gets wrong.
Intervention, override and sign-off authority
Decide before launch who holds each kind of authority, and write it down:
- Intervention authority lets someone pause or correct the system while it's running, including a kill switch that stops it entirely.
- Override authority lets someone replace an AI decision with their own.
- Sign-off authority is the formal approval to deploy a system, make a material change or accept a risk.
Set thresholds too, like requiring human sign-off on any automated refund above a set amount, and define the escalation path when a reviewer isn't sure. Authority should sit with the person who answers for the outcome. Separation of duties applies here as everywhere else, so the team that built a model shouldn't be the one that approves it for production.
Detecting shadow AI
Shadow AI is AI used without the organization's knowledge or approval: personal chatbot accounts used for work, browser extensions, AI features switched on inside existing software, and API keys bought on a corporate card. The risk is that company data flows to services with no contract, no security review and no retention terms.
You can find it in several places:
- proxy, firewall and DNS logs showing traffic to AI services;
- a cloud access security broker (CASB) and data loss prevention tools that flag sensitive data pasted into web forms;
- expense reports and card statements showing AI subscriptions;
- OAuth grants and SaaS management tools showing which apps people connected to company accounts; and
- endpoint and browser extension inventories.
Blocking alone tends to push shadow AI further out of sight. In my experience the approach that works pairs detection with an approved alternative and a simple way to request new tools, so people have a reason to come forward.
Governance artifacts
Auditors and regulators work from evidence, and these are the records that provide it:
- Model documentation covers intended use, training data, evaluation results, known limitations and version history, often in the form of a model card or system card.
- Decision logs record governance decisions, such as what was approved, rejected or approved with conditions, by whom, when and why. For AI systems, they also mean logs of individual automated decisions detailed enough to reconstruct and explain them later. The EU AI Act requires high-risk systems to log events automatically and deployers to keep those logs for at least six months.
- Exception records document any case where a use case went ahead without meeting a standard: what the gap is, why it was allowed, the compensating controls, who approved it and when the exception expires. An exception without an expiry date tends to become permanent.
Labeling AI interactions and synthetic content
People have a right to know when they're dealing with AI. Under Article 50 of the EU AI Act, which has applied since August 2, 2026:
- people must be told when they're interacting with an AI system, unless it's obvious;
- providers of systems that generate audio, images, video or text must mark the output as AI-generated in a machine-readable way;
- deployers who publish a deepfake, meaning AI-generated or altered content that looks like real people, places or events and could pass as authentic, must disclose that it's artificial; and
- AI-generated text published to inform the public on matters of public interest must be disclosed, unless a person reviewed it and took editorial responsibility.
The techniques include visible labels, watermarking, which hides a detectable signal inside the content, and content credentials based on the C2PA standard, which attach signed information about where content came from. Each has a weakness; metadata gets stripped when files are re-saved, and watermarks can be degraded by editing. Layering them works better than relying on one.
Internal and external audits
An internal audit is performed by an independent function inside the organization. An external audit is performed by an outside party, offers more independence, and is required for certifications like ISO/IEC 42001 and by some laws. New York City, for example, requires an independent bias audit every year for automated tools used in hiring and promotion decisions.
The outline lists four things AI audits look at:
- Compliance: does the system meet the laws and internal policies that apply to it?
- Bias: do outcomes differ unfairly across groups?
- Drift: has performance moved since the system was approved?
- Business outcomes: is the system delivering the value it was approved for?
That last one gets overlooked. A system that doesn't deliver its promised benefit is carrying risk for nothing, and retiring it is a legitimate audit finding. As in any audit, auditors shouldn't review their own work.
2.7 AI workforce readiness and culture
Skills assessment and gap analysis
A skills assessment measures what people know and can do today. A gap analysis compares that against what each role needs, and the difference tells you where to train, where to hire and where to bring in outside help. Assessment methods range from self-ratings, which are cheap but run high, to manager reviews and practical tests, like asking a developer to spot the prompt injection risk in a design. Start with the gaps in roles that make high-impact decisions, because that's where a missing skill causes the most harm.
The EU AI Act made this a legal duty. Since February 2, 2025, providers and deployers must take measures to give their staff a sufficient level of AI literacy, matched to their role and to the AI systems they work with.
Role-based training
One course for everyone wastes time for some people and leaves others underprepared. Role-based training matches content to what each group does:
| Role | What they need |
|---|---|
| All staff | What AI does well and badly, the acceptable use policy, which data never goes into AI tools, how to spot AI-generated phishing and deepfakes, how to report a concern |
| Executives and the board | Risk appetite for AI, regulatory exposure, accountability, and the questions to ask before approving a use case |
| Developers and data scientists | Secure AI development, prompt injection, data poisoning, testing and bias evaluation |
| Security teams | AI-specific threats, monitoring AI systems, AI incident response and red teaming |
| Human reviewers in HITL roles | How the model fails, automation bias, and when and how to override |
| Procurement and legal | Vendor due diligence, contract terms and regulatory classification |
Measuring effectiveness
Completion rates tell you who clicked through the course and say little about whether anyone learned anything. The Kirkpatrick model is a common way to think about training results at four levels: reaction (did people find it useful), learning (did their knowledge change), behavior (do they act differently on the job) and results (did the organization's risk go down). Most programs stop at the first level. Better AI metrics sit at levels three and four:
- assessment scores before and after training;
- results of phishing and deepfake simulations;
- the number of shadow AI tools found, which should fall over time;
- the number of use cases submitted through the approved intake process, which should rise as people use the front door; and
- how quickly AI incidents and near misses get reported.
Building a risk-aware culture
A risk-aware culture is one where people notice AI risks and raise them without fear. You can see it when employees report AI mistakes and near misses, when leaders follow the same rules they set (the tone at the top), when honest mistakes are handled without blame, and when AI Champions in each department are the first call for questions.
Governance speed is part of culture too. If an approval takes three months, people will route around it, and the organization will have built its own shadow AI problem. A fast path for low-risk use cases keeps people coming through the front door.
Practice questions
I wrote these in the ISC2 style, so more than one answer will often look reasonable. Pick the BEST answer, and when you're torn, remember the order of priorities: the safety of people first, then legal obligations, then the organization's mission.
- An organization plans to deploy an AI system that screens job applicants in several countries, including EU member states. What should the security professional recommend FIRST?
- A. Pursue ISO/IEC 42001 certification for the organization
- B. Determine how the system is classified under each applicable regulation
- C. Adopt the NIST AI RMF as the governing framework
- D. Commission a penetration test of the screening system
- A hospital's AI triage tool performs measurably worse for one patient population. Clinicians say the tool saves them significant time. What is the BEST course of action?
- A. Keep using the tool and add a disclaimer to its outputs
- B. Retrain the model with more data at the next scheduled release
- C. Require clinician review of every triage output while the disparity is investigated and corrected
- D. Remove demographic attributes from the model's inputs
- Who should accept the residual risk of a high-risk AI use case?
- A. The senior leader who owns the business process
- B. The AI governance committee that reviewed the use case
- C. The data science team that built the model
- D. The chief information security officer
- A provider of a generative AI service the organization depends on announces it will replace the model behind the service next month. What should the organization do?
- A. Rely on the provider's release notes, since the provider owns the model under the shared responsibility model
- B. Trigger a reassessment of the affected use cases, including testing the new model against the original acceptance criteria
- C. Request the provider's latest SOC 2 report
- D. Update the AIBOM and continue operations
- Employees in several departments are using public generative AI tools for work. What should the security professional do FIRST?
- A. Block all known AI domains at the firewall
- B. Begin disciplinary action against employees using unapproved tools
- C. Identify which tools are in use and what data is flowing to them
- D. Buy an enterprise AI license for all staff
- An organization wants to fine-tune a model on support transcripts that customers provided while getting help with their accounts. What is the PRIMARY concern?
- A. Whether the transcripts are encrypted at rest
- B. Whether the fine-tuned model will be accurate enough
- C. Whether the transcripts contain copyrighted material
- D. Whether there's a lawful basis for the new use and it's compatible with the original purpose
- A team wants to add engineering documents to a RAG system that all employees can query. Some of the documents contain export-controlled technical data. What is the BEST action?
- A. Keep the export-controlled documents out of the shared index and handle them in a segregated system with appropriate access controls
- B. Add all the documents and display a warning banner in the interface
- C. Add all the documents and log every query for review
- D. Encrypt the vector database
- An AI agent issues customer refunds without human approval. Monitoring shows a steep rise in unusually large refunds. What should happen FIRST?
- A. Retrain the agent on recent transaction data
- B. Update the agent's model card with the new behavior
- C. Notify the model provider
- D. Use intervention authority to pause or limit the agent's refund capability while the cause is investigated
- A city plans to deploy an AI system that adjusts traffic signal timing automatically. Which oversight approach is MOST appropriate?
- A. Human-out-of-the-loop, because signal changes happen too fast for review
- B. Human-on-the-loop, with operators who monitor the system and can override it or revert to fixed timing
- C. Human-in-the-loop, with an operator approving every signal change
- D. No ongoing oversight, as long as the vendor certifies the system
- In negotiations with an AI provider, which contract term BEST supports the organization's ability to respond to a security incident affecting its data?
- A. Incident notification within a defined time frame
- B. Indemnity for intellectual property claims
- C. A 99.9% uptime commitment
- D. Data residency in a specific region
- Which training approach is MOST likely to reduce AI-related risk across an organization?
- A. An annual AI awareness module required for all employees
- B. Training for developers only, since they build the AI systems
- C. Role-based training measured by changes in on-the-job behavior
- D. A signed acknowledgment of the AI acceptable use policy
- An AI system denies an applicant's loan, and the applicant asks why. Which capability matters MOST?
- A. Transparency that AI was used in the process
- B. Interpretability of the full model architecture
- C. Published accuracy metrics for the model
- D. An explanation of the factors behind this specific decision
Show the answer key
- B. Classification comes first because it decides which obligations apply. Hiring is a high-risk area under the EU AI Act, and that drives everything else. The NIST AI RMF and ISO/IEC 42001 are voluntary and useful, but neither tells you what the law requires.
- C. People's health is at stake, so the immediate step is a control that protects patients now. Waiting for the next release (B) leaves them exposed, a disclaimer (A) moves the burden onto busy clinicians, and removing demographic inputs (D) usually doesn't remove the bias because proxies carry it.
- A. Risk acceptance belongs to the senior leader who owns the process and has the authority to accept it. The committee, the builders and the CISO advise.
- B. A model change is a reassessment trigger. Under shared responsibility, the use case and its outcomes stay yours. A SOC 2 report (C) and an AIBOM update (D) are fine steps, but neither tells you how the new model behaves in your use case.
- C. You can't choose a proportionate response until you know the scope. Blocking (A) and discipline (B) tend to push use out of sight, and buying licenses (D) may be part of the fix later.
- D. Purpose limitation: data collected to help a customer with their account wasn't collected to train a model. Encryption (A) matters, and it doesn't make an incompatible use lawful.
- A. Regulatory contamination: one export-controlled document can pull the whole shared index into scope. Banners, logs and encryption don't stop the data from being disclosed to people who aren't allowed to see it.
- D. Contain first. The agent is causing harm right now, and intervention authority exists for this moment. Investigation, retraining and documentation follow.
- B. Public safety is involved, so a human must be able to intervene and the system needs a safe fallback. Approving every change (C) is impossible at that speed, and removing humans entirely (A, D) leaves no one to catch a failure.
- A. You can't respond to an incident you don't know about. The other terms matter for other reasons.
- C. Role-based training matched to what people do, and measured by behavior instead of completion, is what changes risk. A signed acknowledgment (D) proves someone signed something.
- D. The applicant needs a local explanation of their own decision, and US lending law requires specific reasons for a denial. Transparency (A) alone isn't enough, and full interpretability (B) isn't required to give one.
Sources
- ISC2 AI security certification outline
- EU AI Act explorer and high-level summary
- Digital Omnibus on AI timeline changes: 2026 client alerts from Gibson Dunn and DLA Piper
- NIST AI Risk Management Framework and the Generative AI Profile, NIST AI 600-1
- ISO/IEC 42001:2023
- GDPR full text
- EEOC Uniform Guidelines, 29 CFR Part 1607 (four-fifths rule)
- CFPB Circular 2022-03 on adverse action notices and complex algorithms
- US Copyright Office: Copyright and Artificial Intelligence
- CycloneDX ML-BOM
- Confidential Computing Consortium
- C2PA content credentials
- NYC automated employment decision tools law
- OWASP AI Exchange