ISC2 AI Security Pilot Exam · Free study guide

Domain 1 Study Notes: AI Concepts and Security Fundamentals

How to use these notes

I'm writing these notes while I study for the ISC2 AI security pilot exam, and I'm sharing them because the official guidance right now is an outline and a list of 23 books. Domain 1 is the vocabulary domain; every later domain assumes you can tell these building blocks apart and name the threats and controls that go with them. I've followed the ISC2 outline section by section, and each section ends with how I expect it to show up on the exam. The 12 practice questions at the bottom are written the way ISC2 writes, so more than one answer will look right.

Everything here is my own work, built from public frameworks. If a reference book frames something differently, trust the book, since that's what the question writers check against.

1.1 AI concepts and the AI ecosystem

AI types and evolution

Narrow AI does one job well, like a spam filter or an image classifier; general AI that matches people across every task is still hypothetical. The path to today ran from rule-based expert systems, where people wrote the if-then logic by hand, to machine learning, where the rules come from data, then to deep learning and its many-layered neural networks, and now to generative AI.

When I think about the AI ecosystem, I think about everything around the model: the data sources, the training pipeline, compute, model hubs and APIs, the apps on top, and every vendor in between. Each of those is attack surface.

ML paradigms

ParadigmLearns fromTypical useWhat an attacker targets
SupervisedLabeled examples, each input paired with the right answerFraud and malware classifiersThe labels: flipped or poisoned labels teach the wrong answer
UnsupervisedUnlabeled data, finding clusters and outliers on its ownAnomaly detection, segmentationThe baseline: slow, patient activity can shift what counts as normal
ReinforcementRewards and penalties from acting in an environmentRobotics, game play, tuning LLMs with human feedback (RLHF)The reward: a system can learn to score well without doing the job, which is called reward hacking

Two more terms come up often. Semi-supervised learning mixes a small amount of labeled data with a lot of unlabeled data, and self-supervised learning makes its own labels from the data; that's how LLMs pretrain, by predicting the next token across enormous amounts of text.

Deep learning and evolutionary computation

Deep learning trains layered neural networks by nudging their weights to reduce error, a process called gradient descent with backpropagation. Evolutionary computation takes a different route: genetic algorithms breed a population of candidate solutions through selection, mutation and crossover, without any gradients, and they show up mostly in optimization and search problems.

Neural network architectures

AI hardware

GPUs do massive parallel math and dominate both training and inference, while TPUs are Google's chips built specifically for tensor math. Memory is often the real ceiling on how large a model or a context can get. Each of these has a security side: shared GPU clusters raise isolation questions, model weights sitting in memory are a high-value asset, and compute costs enough that exhausting it is a real attack.

1.2 Generative AI and modern architectures

I'd spend the most Domain 1 time here, because these terms come back in every other domain. For each piece, learn what it does and where untrusted content gets in.

GenAI, language and multimodal models

Generative AI produces new text, images, audio or code. LLMs are transformer models trained on huge volumes of text, and multimodal models take in or produce more than one kind of data, such as text with images. A foundation model is a large general-purpose model that others adapt for narrower jobs, so a flaw in the foundation travels to everything built on it.

Prompt engineering

The system prompt sets the model's role and rules, and the user prompt is the request. Zero-shot prompting gives no examples, few-shot prompting gives a handful, and chain-of-thought prompting asks the model to reason step by step. Here's the security point I'd hold onto: anyone who can reach the model can try to override or leak the system prompt, so keep secrets and access decisions out of it.

Retrieval-Augmented Generation (RAG)

When a question comes in, a RAG app searches a knowledge store, usually a vector database, for relevant passages, adds them to the prompt, and has the model answer from them. That keeps answers current and citable without retraining. It also opens doors you should be able to name: instructions hidden in a retrieved document (indirect prompt injection), retrieval that ignores the user's own access rights and leaks data across permission lines, and a knowledge base someone has poisoned.

Fine-tuning

Full fine-tuning updates every weight in the model. Parameter-Efficient Fine-Tuning (PEFT) updates only a small part, and LoRA, the most common PEFT method, freezes the base model and trains small add-on matrices called adapters. Tuning gets cheap that way, but adapters are small files that someone can swap or tamper with, fine-tuning data is a path for poisoning, and tuning can weaken a model's safety training. My rule of thumb is RAG for knowledge that changes and fine-tuning for behavior, tone and format.

Agents and agentic AI

An agent is a model that plans and acts in a loop, calling tools like APIs, code runners or browsers to reach a goal; agentic AI is the broader pattern of systems acting with some autonomy, often several agents working together. Agent-to-agent (A2A) protocols let agents hand work to each other, and the Model Context Protocol (MCP) connects models to tools and data. Session persistence and memory let an agent carry context forward within a session or across many, which also means a poisoned memory keeps doing damage long after the attack. The core agent risk is excessive agency, where an agent holds more permission or autonomy than its task needs.

NLP building blocks

Natural language processing breaks text into tokens and works with ideas like part-of-speech tagging, sentiment analysis and named entity recognition, which finds names, places and account numbers. Classic NLP also normalizes words through stemming and lemmatization. Named entity recognition matters most for security people, since it's how many tools spot personal data in prompts and outputs.

Embeddings and context windows

An embedding is a list of numbers that captures meaning, so similar text lands close together. The context window is the most tokens a model can consider at once, and it has to hold the system prompt, the chat history, any retrieved documents and the reply. When it overflows, older content falls out, and models tend to pay less attention to whatever sits in the middle of a long context.

Token management

Tokens are word pieces; in English, one token runs about four characters. Cost, rate limits and context limits are all counted in tokens, so capping tokens per request and per user controls spend and abuse at the same time. Leave usage uncapped and you invite denial of wallet, where an attacker runs up your bill.

Vector databases

These store embeddings and return the closest matches to a query. I'd treat them like any sensitive data store, because embeddings can sometimes be turned back into something close to the original text; they need access controls, tenant separation and integrity checks.

1.3 AI limitations and performance

AI fails in plenty of ways without anyone attacking it, and the exam wants to know you'd notice.

Reliability limits

Drift

TypeWhat changesExample
Data driftThe inputs look different from the training dataA fraud model trained before a new payment method launched
Model or concept driftThe link between inputs and the right answer changesAttackers change tactics, so yesterday's "normal" now hides attacks
Behavioral driftA hosted model starts acting differentlyA vendor updates the model behind an API and your tested prompts stop working

You catch drift by watching outputs against a baseline, and you fix it by retraining, re-testing or rolling back. A model update or a measured drift is also a standard trigger for a fresh risk assessment.

Model collapse

When models train on content that earlier models produced, quality slides over generations; rare cases vanish first, and outputs turn bland and less accurate. The defense is knowing where your training data came from, labeling synthetic data, and keeping a supply of human-made data on hand.

Evaluation metrics and benchmarks

For classifiers, I'd know four metrics cold:

For LLMs, evaluation mixes public benchmarks like MMLU for general knowledge, task-specific test sets, safety and red-team tests, human review, and a second model acting as judge. Watch for benchmark contamination; if test questions leaked into training, the score is inflated.

Operational modes

With human-in-the-loop (HITL), a person approves each decision before it takes effect. With human-on-the-loop, the system acts while a person monitors and can step in, and a fully automated system has no person in the path at all. Match the mode to impact and reversibility, so denying a loan or deleting data calls for HITL. On the human side, the trap is automation bias, where reviewers approve whatever the model says.

1.4 Data management and privacy in AI

A model is only as trustworthy as its data, so I treat data as a quality problem and a security problem at once. Domain 3 goes deeper; here you need the vocabulary.

Data types and collection

Structured data lives in rows and columns. Semi-structured data, like JSON or logs, has some shape but no fixed schema, and unstructured data such as free text, email, images and audio has none at all. Most AI training data is unstructured, which is exactly where hidden personal data and hidden instructions tend to live. Collection parameters are the decisions you make up front about source, volume, refresh rate, consent and permitted purpose, and the principle behind all of them is to collect only what the purpose needs.

Data lifecycle stages

  1. Acquisition: where the data comes from, and under what license and consent.

  2. Extraction: pulling it from systems or parsing documents.

  3. Cleaning: removing duplicates, fixing errors, handling outliers, and stripping personal data you don't need.

  4. Transformation: normalizing formats, building features and tokenizing text.

  5. Labeling: people or tools marking the right answers. I'd flag labelers as an insider risk as well as a quality risk, since one bad labeler can poison a dataset.

Storage and versioning, training, retention and disposal come after that. Each stage needs its own controls, and a question may ask you to place a control at the right one.

Data quality

The outline names accuracy, completeness and relevance. I'd add consistency, timeliness and representativeness, because data that underrepresents a group produces a model that performs worse for that group, and that's how bias gets in.

Data provenance

Provenance is the record of where data came from, who owned it and every change since; lineage traces that same record through your pipeline. You need it to verify integrity, prove licensing and consent, investigate suspected poisoning, answer regulators and reproduce a model. Datasheets, hashes and signed manifests are the usual tools.

Privacy preservation techniques

TechniqueWhat it doesReversible?Watch for
TokenizationSwaps a sensitive value for a random token; a secure vault maps it backYes, through the vaultThe vault becomes the crown jewel
MaskingHides all or part of a value, like showing only the last four digitsUsually notPartial masks can still identify someone when combined with other fields
k-anonymityGeneralizes or suppresses details so each record matches at least k-1 othersNoIf everyone in a group shares the same sensitive value, it still leaks
Synthetic dataGenerates new records with the same statistical patternsNo direct linkA generator that overfits can reproduce real records, and rare cases may vanish

Pseudonymized data, where identifiers are replaced but can be re-linked, is still personal data under GDPR, while truly anonymized data falls outside it. Differential privacy adds calibrated noise so no single person's data can be singled out; it belongs to Domain 3, but you should recognize it here.

1.5 AI security fundamentals and threats

Most questions here will describe an attack and ask you to name it. Two details settle almost all of them: when the attack happens, at training time or at inference, and how the malicious content arrives, typed by the user or hidden in something the model reads.

Zero trust applied to AI

The familiar principles of never trusting by default, verifying explicitly, least privilege and assuming breach all carry over, with one twist: I treat the model itself as an untrusted component.

Adversarial threats

Model inversion and data poisoning

Model inversion reconstructs sensitive features of the training data from the model's outputs. Its relatives are membership inference, which tells whether a specific record was in the training set, and model extraction, which copies a model's behavior through repeated queries. Data poisoning tampers with training, fine-tuning or RAG data to degrade a model or plant a backdoor, a hidden trigger that produces whatever behavior the attacker chose.

AttackWhen it happensWhat it hurts most
Data poisoning, backdoorsTraining or fine-tuning, or when the RAG store is loadedIntegrity
EvasionInferenceIntegrity
Prompt injection, jailbreakingInferenceIntegrity, and confidentiality when data leaks
Model inversion, membership inferenceInferenceConfidentiality of training data
Model extractionInferenceConfidentiality of the model as intellectual property
Resource exhaustion, denial of walletInferenceAvailability

Common threat frameworks

1.6 AI controls, safeguards and principles

AI-specific controls are probabilistic, so I layer them on top of ordinary deterministic controls and never let them stand alone.

LLM guardrails

On the way in, check length, format and allowed topics, run classifiers that flag likely prompt injection, redact personal data before it reaches the model, and rate-limit by user and by token. On the way out, validate output against the expected format, run toxicity and harm classifiers, scan for personal data and credentials, check answers against the retrieved sources, encode output before a browser renders it so model text can't turn into script, and block or rewrite risky links.

Guardrails catch a lot and still miss some, because they're models or pattern matches themselves. The controls that hold every time are permissions, allowlists for tools and destinations, and human approval for high-impact actions.

Responsible AI principles

The outline names four, and two of them get confused constantly.

The NIST AI RMF gives a fuller list of what makes AI trustworthy: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. You'll see that list again in Domain 2.

Model verification techniques

Verification answers two questions: is this the model I think it is, and does it behave the way I expect?

Isolation techniques for AI environments

1.7 AI solutions and use cases

Here you weigh a real deployment's benefit against the risk it brings, and the right answer usually enables the use case with controls instead of banning it or waving it through.

AI in business workflows and software development

Copilots for email and documents, customer service chatbots, document summarization and coding assistants are the deployments you'll see most. The risks they add are sensitive data pasted into tools nobody approved (shadow AI), generated code with security flaws or licensing strings attached, hallucinated package names that attackers have already registered, and credentials dropped into prompts. The matching controls are an approved-tools list, enterprise agreements that bar the vendor from training on your data, DLP on prompts, normal code review and scanning for anything generated, and dependency checks before a package gets installed.

Industry use cases

IndustryTypical usesWhat raises the stakes
HealthcareReading medical images, summarizing clinical notesProtected health information, patient safety, the need to explain decisions to clinicians
FinanceFraud detection, credit decisions, tradingFair lending rules, model risk management, fraudsters actively trying to evade the model
Law enforcementFacial recognition, crime analyticsBias, civil liberties, and outright bans: the EU AI Act prohibits building facial recognition databases by scraping images indiscriminately and tightly restricts real-time biometric identification in public spaces
EducationTutoring, advising, proctoringStudent privacy laws, fairness in grading and discipline, minors as users

AI for security operations

In threat analysis, AI summarizes intelligence, enriches alerts and drafts incident timelines. In vulnerability management, it ranks findings by real risk, reviews code and suggests fixes. In security automation, it drives SOAR playbooks, phishing triage and anomaly detection on user and system behavior.

The catches are real. Attackers will try to poison or evade security models, analysts can start trusting the tool too much, and an agent that can isolate hosts or disable accounts needs tight permissions and human approval for disruptive steps. The AI tool also becomes a target in its own right, and attackers use the same technology to write better phishing and build malware faster.

Practice questions

ISC2 questions test judgment more than recall. Several answers will be technically true, and your job is to pick the one a security leader would choose: human safety first, then the organization's mission and legal obligations, then everything else. Answer all 12 in order without going back, then check the key.

  1. A hospital plans to use an AI model to recommend medication doses, and the recommendations will flow straight into the pharmacy system. What should be the security professional's PRIMARY concern?

    • A. Protecting the confidentiality of the patient records used in training

    • B. Making sure a qualified clinician reviews each recommendation before it's acted on

    • C. Verifying the model vendor's security certifications

    • D. Measuring the model's accuracy against published benchmarks

  2. Senior leadership wants a generative AI assistant rolled out to every department this quarter. What should the security professional do FIRST?

    • A. Deploy guardrails that block prompt injection

    • B. Identify the business use cases and the data each one will touch

    • C. Run a red-team exercise against the chosen model

    • D. Publish an acceptable use policy for AI tools

  3. Which statement BEST explains why prompt injection is so hard to prevent completely?

    • A. Most organizations lack the budget for AI-specific security tools

    • B. Language models can't reliably tell instructions apart from the data they process

    • C. Attackers can easily obtain a copy of the system prompt

    • D. Guardrail classifiers produce too many false positives

  4. A team is choosing a detection model for a network where a missed intrusion could disrupt emergency services. Which characteristic should carry the MOST weight?

    • A. High precision, to reduce the analysts' workload

    • B. High recall, so few real intrusions go undetected

    • C. High overall accuracy across all traffic

    • D. Low response time

  5. A model's output has grown steadily less reliable over several months, and nobody has changed the model. What is the MOST likely cause?

    • A. Model collapse

    • B. Data or concept drift

    • C. A backdoor planted during training

    • D. Non-determinism in the sampling settings

  6. Which control gives the GREATEST assurance that an autonomous agent can't cause serious harm if an attacker manipulates it?

    • A. A system prompt that lists prohibited actions

    • B. An output classifier that blocks harmful responses

    • C. Limiting the agent's permissions to only what its task requires

    • D. Logging every action the agent takes

  7. An organization is deciding whether to use customer personal data to train a model. Which principle should guide the decision FIRST?

    • A. Data minimization

    • B. Data quality

    • C. Model performance

    • D. Cost of acquisition

  8. A model downloaded from a public repository is headed for production. Which concern is MOST important before anyone loads it?

    • A. Whether the license permits commercial use

    • B. Whether the model file could execute malicious code

    • C. Whether the model meets the accuracy target

    • D. Whether the model's documentation is complete

  9. A regulator asks a bank how its credit model reached a particular decision. Which principle does the request MOST directly address?

    • A. Transparency

    • B. Explainability

    • C. Fairness

    • D. Accountability

  10. Which statement BEST describes the security value of data provenance in an AI system?

    • A. It reduces the cost of storing training data

    • B. It establishes trust in the origin and integrity of the data a model learns from

    • C. It improves model accuracy on benchmarks

    • D. It removes personal data from training sets

  11. The SOC wants an AI agent to isolate hosts automatically when it believes they're compromised, and some of those hosts support patient care. What is the BEST approach?

    • A. Allow full automation, since speed matters most during an attack

    • B. Prohibit the agent from taking any action

    • C. Let the agent gather and enrich evidence on its own, and require human approval before it isolates a host

    • D. Allow automation only during business hours

  12. Which framework is MOST appropriate for describing the tactics and techniques an adversary might use against an AI system?

    • A. NIST AI Risk Management Framework

    • B. MITRE ATLAS

    • C. ISO/IEC 42001

    • D. The EU AI Act

Show the answer key
  1. B. Every option has merit, but a wrong dose can kill someone, and human safety outranks confidentiality, vendor assurance and accuracy. A high-impact decision that can't be undone needs a human in the loop.

  2. B. You can't size controls, test the right model or write a useful policy until you know what the assistant will do and what data it will see. Policy (D) is tempting, but it comes after scoping.

  3. B. That's the root cause. The other options describe real problems that sit downstream of it.

  4. B. When a miss could hurt people, recall comes first. Precision protects analyst time, and overall accuracy hides misses when attacks are rare.

  5. B. Gradual decline with no change to the model points to the world changing underneath it. A backdoor fires on a trigger, and collapse comes from training on synthetic data over generations.

  6. C. Least privilege limits damage even when the model is fooled. Prompts and classifiers can be bypassed, and logging only tells you what happened afterward.

  7. A. Privacy law and good practice both start with whether you need the data at all. Quality, performance and cost matter only once that's settled.

  8. B. Licensing matters before deployment, but an unsafe file can compromise the environment the moment it loads, so it comes first.

  9. B. Explainability covers why a specific decision came out the way it did. Transparency is disclosing that AI is used, and accountability is about who answers for the outcome.

  10. B. Provenance is about trust and integrity. It doesn't cut costs, raise accuracy or strip personal data on its own.

  11. C. Isolating a host that supports patient care could put people at risk, so a human approves that step, while automation still speeds up the investigation. Banning the agent outright gives up real value.

  12. B. ATLAS is built like ATT&CK for AI systems. The other three are governance frameworks or regulation.

Sources