ISC2 AI Security Pilot Exam · Free study guide
Domain 1 Study Notes: AI Concepts and Security Fundamentals
How to use these notes
I'm writing these notes while I study for the ISC2 AI security pilot exam, and I'm sharing them because the official guidance right now is an outline and a list of 23 books. Domain 1 is the vocabulary domain; every later domain assumes you can tell these building blocks apart and name the threats and controls that go with them. I've followed the ISC2 outline section by section, and each section ends with how I expect it to show up on the exam. The 12 practice questions at the bottom are written the way ISC2 writes, so more than one answer will look right.
Everything here is my own work, built from public frameworks. If a reference book frames something differently, trust the book, since that's what the question writers check against.
1.1 AI concepts and the AI ecosystem
AI types and evolution
Narrow AI does one job well, like a spam filter or an image classifier; general AI that matches people across every task is still hypothetical. The path to today ran from rule-based expert systems, where people wrote the if-then logic by hand, to machine learning, where the rules come from data, then to deep learning and its many-layered neural networks, and now to generative AI.
When I think about the AI ecosystem, I think about everything around the model: the data sources, the training pipeline, compute, model hubs and APIs, the apps on top, and every vendor in between. Each of those is attack surface.
ML paradigms
| Paradigm | Learns from | Typical use | What an attacker targets |
|---|---|---|---|
| Supervised | Labeled examples, each input paired with the right answer | Fraud and malware classifiers | The labels: flipped or poisoned labels teach the wrong answer |
| Unsupervised | Unlabeled data, finding clusters and outliers on its own | Anomaly detection, segmentation | The baseline: slow, patient activity can shift what counts as normal |
| Reinforcement | Rewards and penalties from acting in an environment | Robotics, game play, tuning LLMs with human feedback (RLHF) | The reward: a system can learn to score well without doing the job, which is called reward hacking |
Two more terms come up often. Semi-supervised learning mixes a small amount of labeled data with a lot of unlabeled data, and self-supervised learning makes its own labels from the data; that's how LLMs pretrain, by predicting the next token across enormous amounts of text.
Deep learning and evolutionary computation
Deep learning trains layered neural networks by nudging their weights to reduce error, a process called gradient descent with backpropagation. Evolutionary computation takes a different route: genetic algorithms breed a population of candidate solutions through selection, mutation and crossover, without any gradients, and they show up mostly in optimization and search problems.
Neural network architectures
CNNs (convolutional neural networks) slide small filters across an input to find local patterns, which is why they became the standard for images and video.
RNNs (recurrent neural networks), including LSTMs, read a sequence one step at a time and carry memory forward; they handled time series and early language models but lose track over long sequences.
Transformers use attention to weigh every token against every other token at once, so they train in parallel and scale well. LLMs and most multimodal models are built on them.
AI hardware
GPUs do massive parallel math and dominate both training and inference, while TPUs are Google's chips built specifically for tensor math. Memory is often the real ceiling on how large a model or a context can get. Each of these has a security side: shared GPU clusters raise isolation questions, model weights sitting in memory are a high-value asset, and compute costs enough that exhausting it is a real attack.
1.2 Generative AI and modern architectures
I'd spend the most Domain 1 time here, because these terms come back in every other domain. For each piece, learn what it does and where untrusted content gets in.
GenAI, language and multimodal models
Generative AI produces new text, images, audio or code. LLMs are transformer models trained on huge volumes of text, and multimodal models take in or produce more than one kind of data, such as text with images. A foundation model is a large general-purpose model that others adapt for narrower jobs, so a flaw in the foundation travels to everything built on it.
Prompt engineering
The system prompt sets the model's role and rules, and the user prompt is the request. Zero-shot prompting gives no examples, few-shot prompting gives a handful, and chain-of-thought prompting asks the model to reason step by step. Here's the security point I'd hold onto: anyone who can reach the model can try to override or leak the system prompt, so keep secrets and access decisions out of it.
Retrieval-Augmented Generation (RAG)
When a question comes in, a RAG app searches a knowledge store, usually a vector database, for relevant passages, adds them to the prompt, and has the model answer from them. That keeps answers current and citable without retraining. It also opens doors you should be able to name: instructions hidden in a retrieved document (indirect prompt injection), retrieval that ignores the user's own access rights and leaks data across permission lines, and a knowledge base someone has poisoned.
Fine-tuning
Full fine-tuning updates every weight in the model. Parameter-Efficient Fine-Tuning (PEFT) updates only a small part, and LoRA, the most common PEFT method, freezes the base model and trains small add-on matrices called adapters. Tuning gets cheap that way, but adapters are small files that someone can swap or tamper with, fine-tuning data is a path for poisoning, and tuning can weaken a model's safety training. My rule of thumb is RAG for knowledge that changes and fine-tuning for behavior, tone and format.
Agents and agentic AI
An agent is a model that plans and acts in a loop, calling tools like APIs, code runners or browsers to reach a goal; agentic AI is the broader pattern of systems acting with some autonomy, often several agents working together. Agent-to-agent (A2A) protocols let agents hand work to each other, and the Model Context Protocol (MCP) connects models to tools and data. Session persistence and memory let an agent carry context forward within a session or across many, which also means a poisoned memory keeps doing damage long after the attack. The core agent risk is excessive agency, where an agent holds more permission or autonomy than its task needs.
NLP building blocks
Natural language processing breaks text into tokens and works with ideas like part-of-speech tagging, sentiment analysis and named entity recognition, which finds names, places and account numbers. Classic NLP also normalizes words through stemming and lemmatization. Named entity recognition matters most for security people, since it's how many tools spot personal data in prompts and outputs.
Embeddings and context windows
An embedding is a list of numbers that captures meaning, so similar text lands close together. The context window is the most tokens a model can consider at once, and it has to hold the system prompt, the chat history, any retrieved documents and the reply. When it overflows, older content falls out, and models tend to pay less attention to whatever sits in the middle of a long context.
Token management
Tokens are word pieces; in English, one token runs about four characters. Cost, rate limits and context limits are all counted in tokens, so capping tokens per request and per user controls spend and abuse at the same time. Leave usage uncapped and you invite denial of wallet, where an attacker runs up your bill.
Vector databases
These store embeddings and return the closest matches to a query. I'd treat them like any sensitive data store, because embeddings can sometimes be turned back into something close to the original text; they need access controls, tenant separation and integrity checks.
1.3 AI limitations and performance
AI fails in plenty of ways without anyone attacking it, and the exam wants to know you'd notice.
Reliability limits
Hallucination is confident, fluent output that's false, like a made-up court case or a software package that doesn't exist. Attackers register those invented package names, so a developer who installs whatever the model suggested ends up installing malware.
Non-determinism means the same prompt can return different answers, mostly because of sampling settings like temperature, which makes testing, reproducing a bug and auditing a decision much harder.
Uncertainty is the gap between how sure a model sounds and how right it is, since models don't reliably know what they don't know.
Drift
| Type | What changes | Example |
|---|---|---|
| Data drift | The inputs look different from the training data | A fraud model trained before a new payment method launched |
| Model or concept drift | The link between inputs and the right answer changes | Attackers change tactics, so yesterday's "normal" now hides attacks |
| Behavioral drift | A hosted model starts acting differently | A vendor updates the model behind an API and your tested prompts stop working |
You catch drift by watching outputs against a baseline, and you fix it by retraining, re-testing or rolling back. A model update or a measured drift is also a standard trigger for a fresh risk assessment.
Model collapse
When models train on content that earlier models produced, quality slides over generations; rare cases vanish first, and outputs turn bland and less accurate. The defense is knowing where your training data came from, labeling synthetic data, and keeping a supply of human-made data on hand.
Evaluation metrics and benchmarks
For classifiers, I'd know four metrics cold:
Accuracy is the share of all answers that are right. It misleads on lopsided data, since a detector that calls everything benign scores 99.9% when attacks are rare.
Precision is how much of what got flagged was real; when it drops, analysts drown in false positives.
Recall is how much of what was real got caught; when it drops, attacks slip through.
F1 balances precision and recall in a single score.
For LLMs, evaluation mixes public benchmarks like MMLU for general knowledge, task-specific test sets, safety and red-team tests, human review, and a second model acting as judge. Watch for benchmark contamination; if test questions leaked into training, the score is inflated.
Operational modes
With human-in-the-loop (HITL), a person approves each decision before it takes effect. With human-on-the-loop, the system acts while a person monitors and can step in, and a fully automated system has no person in the path at all. Match the mode to impact and reversibility, so denying a loan or deleting data calls for HITL. On the human side, the trap is automation bias, where reviewers approve whatever the model says.
1.4 Data management and privacy in AI
A model is only as trustworthy as its data, so I treat data as a quality problem and a security problem at once. Domain 3 goes deeper; here you need the vocabulary.
Data types and collection
Structured data lives in rows and columns. Semi-structured data, like JSON or logs, has some shape but no fixed schema, and unstructured data such as free text, email, images and audio has none at all. Most AI training data is unstructured, which is exactly where hidden personal data and hidden instructions tend to live. Collection parameters are the decisions you make up front about source, volume, refresh rate, consent and permitted purpose, and the principle behind all of them is to collect only what the purpose needs.
Data lifecycle stages
Acquisition: where the data comes from, and under what license and consent.
Extraction: pulling it from systems or parsing documents.
Cleaning: removing duplicates, fixing errors, handling outliers, and stripping personal data you don't need.
Transformation: normalizing formats, building features and tokenizing text.
Labeling: people or tools marking the right answers. I'd flag labelers as an insider risk as well as a quality risk, since one bad labeler can poison a dataset.
Storage and versioning, training, retention and disposal come after that. Each stage needs its own controls, and a question may ask you to place a control at the right one.
Data quality
The outline names accuracy, completeness and relevance. I'd add consistency, timeliness and representativeness, because data that underrepresents a group produces a model that performs worse for that group, and that's how bias gets in.
Data provenance
Provenance is the record of where data came from, who owned it and every change since; lineage traces that same record through your pipeline. You need it to verify integrity, prove licensing and consent, investigate suspected poisoning, answer regulators and reproduce a model. Datasheets, hashes and signed manifests are the usual tools.
Privacy preservation techniques
| Technique | What it does | Reversible? | Watch for |
|---|---|---|---|
| Tokenization | Swaps a sensitive value for a random token; a secure vault maps it back | Yes, through the vault | The vault becomes the crown jewel |
| Masking | Hides all or part of a value, like showing only the last four digits | Usually not | Partial masks can still identify someone when combined with other fields |
| k-anonymity | Generalizes or suppresses details so each record matches at least k-1 others | No | If everyone in a group shares the same sensitive value, it still leaks |
| Synthetic data | Generates new records with the same statistical patterns | No direct link | A generator that overfits can reproduce real records, and rare cases may vanish |
Pseudonymized data, where identifiers are replaced but can be re-linked, is still personal data under GDPR, while truly anonymized data falls outside it. Differential privacy adds calibrated noise so no single person's data can be singled out; it belongs to Domain 3, but you should recognize it here.
1.5 AI security fundamentals and threats
Most questions here will describe an attack and ask you to name it. Two details settle almost all of them: when the attack happens, at training time or at inference, and how the malicious content arrives, typed by the user or hidden in something the model reads.
Zero trust applied to AI
The familiar principles of never trusting by default, verifying explicitly, least privilege and assuming breach all carry over, with one twist: I treat the model itself as an untrusted component.
Validate model output before it reaches a database, a shell or a browser.
Treat retrieved documents, tool results and agent memory as untrusted input too.
Authenticate every agent and tool call, and give each agent its own narrowly scoped credentials.
Segment model endpoints and training environments, and monitor them continuously.
Adversarial threats
Prompt injection is input crafted to override the app's instructions. It's direct when the user types it and indirect when it hides in content the model reads, like a web page, an email, a shared document or a tool result. The root cause is that models can't reliably separate instructions from data.
Jailbreaking gets a model to ignore its own safety rules through role-play, encoding tricks or long chains of examples. Prompt injection hijacks what the application is trying to do, while jailbreaking breaks the model's built-in limits, and the two often show up together.
Evasion changes an input at inference time so the model misclassifies it, like small pixel changes that fool an image model or a tweaked binary that slips past an ML malware detector.
Exfiltration gets data out through the model, whether that's personal data in a response, a leaked system prompt, or data smuggled out through a tool call or a link the app renders.
Model inversion and data poisoning
Model inversion reconstructs sensitive features of the training data from the model's outputs. Its relatives are membership inference, which tells whether a specific record was in the training set, and model extraction, which copies a model's behavior through repeated queries. Data poisoning tampers with training, fine-tuning or RAG data to degrade a model or plant a backdoor, a hidden trigger that produces whatever behavior the attacker chose.
| Attack | When it happens | What it hurts most |
|---|---|---|
| Data poisoning, backdoors | Training or fine-tuning, or when the RAG store is loaded | Integrity |
| Evasion | Inference | Integrity |
| Prompt injection, jailbreaking | Inference | Integrity, and confidentiality when data leaks |
| Model inversion, membership inference | Inference | Confidentiality of training data |
| Model extraction | Inference | Confidentiality of the model as intellectual property |
| Resource exhaustion, denial of wallet | Inference | Availability |
Common threat frameworks
OWASP's Top 10 for LLM Applications 2026 ranks application risks, and prompt injection and sensitive information disclosure still hold the top two spots. OWASP also publishes the Top 10 for Agentic Applications and the AI Exchange. I use OWASP for application risks and developer checklists.
NIST's AI 100-2 E2025 is a taxonomy of adversarial ML attacks and mitigations, sorted by system type (predictive or generative), lifecycle stage and attacker goal: availability, integrity, privacy, and misuse for generative AI. I use it for shared vocabulary.
MITRE ATLAS is an ATT&CK-style matrix of adversary tactics and techniques against AI systems, backed by real case studies. I use it to model attacker behavior and to map findings and detections.
1.6 AI controls, safeguards and principles
AI-specific controls are probabilistic, so I layer them on top of ordinary deterministic controls and never let them stand alone.
LLM guardrails
On the way in, check length, format and allowed topics, run classifiers that flag likely prompt injection, redact personal data before it reaches the model, and rate-limit by user and by token. On the way out, validate output against the expected format, run toxicity and harm classifiers, scan for personal data and credentials, check answers against the retrieved sources, encode output before a browser renders it so model text can't turn into script, and block or rewrite risky links.
Guardrails catch a lot and still miss some, because they're models or pattern matches themselves. The controls that hold every time are permissions, allowlists for tools and destinations, and human approval for high-impact actions.
Responsible AI principles
The outline names four, and two of them get confused constantly.
Transparency means being open that AI is in use, what it's for and what data it relies on.
Explainability means being able to say why the model produced a specific output, often with techniques that show which inputs mattered most.
Fairness means outcomes don't differ unjustly across groups.
Bias awareness means knowing where bias enters, whether from unrepresentative data, skewed labels, design choices or the way a system gets deployed.
The NIST AI RMF gives a fuller list of what makes AI trustworthy: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. You'll see that list again in Domain 2.
Model verification techniques
Verification answers two questions: is this the model I think it is, and does it behave the way I expect?
For integrity and provenance, compare cryptographic hashes, check digital signatures on model files, and pull models only from approved registries.
For safe loading, remember that some model file formats, Python's pickle in particular, can run code the moment they load. Scan model files and prefer formats that store only weights, such as safetensors.
For behavior, run test suites, benchmarks, red-team exercises and regression tests before each release and after every update.
Isolation techniques for AI environments
Separate development, test and production, including their data.
Sandbox anything a model can execute, like generated code or agent tool calls, in containers or virtual machines with no standing credentials.
Restrict network egress so a compromised model or agent can't call out freely.
Keep tenants apart in shared GPU clusters and vector databases.
Use private endpoints, and for the most sensitive work, hardware-based confidential computing.
1.7 AI solutions and use cases
Here you weigh a real deployment's benefit against the risk it brings, and the right answer usually enables the use case with controls instead of banning it or waving it through.
AI in business workflows and software development
Copilots for email and documents, customer service chatbots, document summarization and coding assistants are the deployments you'll see most. The risks they add are sensitive data pasted into tools nobody approved (shadow AI), generated code with security flaws or licensing strings attached, hallucinated package names that attackers have already registered, and credentials dropped into prompts. The matching controls are an approved-tools list, enterprise agreements that bar the vendor from training on your data, DLP on prompts, normal code review and scanning for anything generated, and dependency checks before a package gets installed.
Industry use cases
| Industry | Typical uses | What raises the stakes |
|---|---|---|
| Healthcare | Reading medical images, summarizing clinical notes | Protected health information, patient safety, the need to explain decisions to clinicians |
| Finance | Fraud detection, credit decisions, trading | Fair lending rules, model risk management, fraudsters actively trying to evade the model |
| Law enforcement | Facial recognition, crime analytics | Bias, civil liberties, and outright bans: the EU AI Act prohibits building facial recognition databases by scraping images indiscriminately and tightly restricts real-time biometric identification in public spaces |
| Education | Tutoring, advising, proctoring | Student privacy laws, fairness in grading and discipline, minors as users |
AI for security operations
In threat analysis, AI summarizes intelligence, enriches alerts and drafts incident timelines. In vulnerability management, it ranks findings by real risk, reviews code and suggests fixes. In security automation, it drives SOAR playbooks, phishing triage and anomaly detection on user and system behavior.
The catches are real. Attackers will try to poison or evade security models, analysts can start trusting the tool too much, and an agent that can isolate hosts or disable accounts needs tight permissions and human approval for disruptive steps. The AI tool also becomes a target in its own right, and attackers use the same technology to write better phishing and build malware faster.
Practice questions
ISC2 questions test judgment more than recall. Several answers will be technically true, and your job is to pick the one a security leader would choose: human safety first, then the organization's mission and legal obligations, then everything else. Answer all 12 in order without going back, then check the key.
A hospital plans to use an AI model to recommend medication doses, and the recommendations will flow straight into the pharmacy system. What should be the security professional's PRIMARY concern?
A. Protecting the confidentiality of the patient records used in training
B. Making sure a qualified clinician reviews each recommendation before it's acted on
C. Verifying the model vendor's security certifications
D. Measuring the model's accuracy against published benchmarks
Senior leadership wants a generative AI assistant rolled out to every department this quarter. What should the security professional do FIRST?
A. Deploy guardrails that block prompt injection
B. Identify the business use cases and the data each one will touch
C. Run a red-team exercise against the chosen model
D. Publish an acceptable use policy for AI tools
Which statement BEST explains why prompt injection is so hard to prevent completely?
A. Most organizations lack the budget for AI-specific security tools
B. Language models can't reliably tell instructions apart from the data they process
C. Attackers can easily obtain a copy of the system prompt
D. Guardrail classifiers produce too many false positives
A team is choosing a detection model for a network where a missed intrusion could disrupt emergency services. Which characteristic should carry the MOST weight?
A. High precision, to reduce the analysts' workload
B. High recall, so few real intrusions go undetected
C. High overall accuracy across all traffic
D. Low response time
A model's output has grown steadily less reliable over several months, and nobody has changed the model. What is the MOST likely cause?
A. Model collapse
B. Data or concept drift
C. A backdoor planted during training
D. Non-determinism in the sampling settings
Which control gives the GREATEST assurance that an autonomous agent can't cause serious harm if an attacker manipulates it?
A. A system prompt that lists prohibited actions
B. An output classifier that blocks harmful responses
C. Limiting the agent's permissions to only what its task requires
D. Logging every action the agent takes
An organization is deciding whether to use customer personal data to train a model. Which principle should guide the decision FIRST?
A. Data minimization
B. Data quality
C. Model performance
D. Cost of acquisition
A model downloaded from a public repository is headed for production. Which concern is MOST important before anyone loads it?
A. Whether the license permits commercial use
B. Whether the model file could execute malicious code
C. Whether the model meets the accuracy target
D. Whether the model's documentation is complete
A regulator asks a bank how its credit model reached a particular decision. Which principle does the request MOST directly address?
A. Transparency
B. Explainability
C. Fairness
D. Accountability
Which statement BEST describes the security value of data provenance in an AI system?
A. It reduces the cost of storing training data
B. It establishes trust in the origin and integrity of the data a model learns from
C. It improves model accuracy on benchmarks
D. It removes personal data from training sets
The SOC wants an AI agent to isolate hosts automatically when it believes they're compromised, and some of those hosts support patient care. What is the BEST approach?
A. Allow full automation, since speed matters most during an attack
B. Prohibit the agent from taking any action
C. Let the agent gather and enrich evidence on its own, and require human approval before it isolates a host
D. Allow automation only during business hours
Which framework is MOST appropriate for describing the tactics and techniques an adversary might use against an AI system?
A. NIST AI Risk Management Framework
B. MITRE ATLAS
C. ISO/IEC 42001
D. The EU AI Act
Show the answer key
B. Every option has merit, but a wrong dose can kill someone, and human safety outranks confidentiality, vendor assurance and accuracy. A high-impact decision that can't be undone needs a human in the loop.
B. You can't size controls, test the right model or write a useful policy until you know what the assistant will do and what data it will see. Policy (D) is tempting, but it comes after scoping.
B. That's the root cause. The other options describe real problems that sit downstream of it.
B. When a miss could hurt people, recall comes first. Precision protects analyst time, and overall accuracy hides misses when attacks are rare.
B. Gradual decline with no change to the model points to the world changing underneath it. A backdoor fires on a trigger, and collapse comes from training on synthetic data over generations.
C. Least privilege limits damage even when the model is fooled. Prompts and classifiers can be bypassed, and logging only tells you what happened afterward.
A. Privacy law and good practice both start with whether you need the data at all. Quality, performance and cost matter only once that's settled.
B. Licensing matters before deployment, but an unsafe file can compromise the environment the moment it loads, so it comes first.
B. Explainability covers why a specific decision came out the way it did. Transparency is disclosing that AI is used, and accountability is about who answers for the outcome.
B. Provenance is about trust and integrity. It doesn't cut costs, raise accuracy or strip personal data on its own.
C. Isolating a host that supports patient care could put people at risk, so a human approves that step, while automation still speeds up the investigation. Banning the agent outright gives up real value.
B. ATLAS is built like ATT&CK for AI systems. The other three are governance frameworks or regulation.