ISC2 AI Security Pilot Exam · Free study guide

Domain 6 Study Notes: AI Threats, Testing, and Lifecycle Management

How to use these notes

Domain 6 ties the other five together. It covers how to think like an attacker (threat modeling), how to build AI with security in every stage (the secure AI development lifecycle and supply chain), and how to prove your defenses work (testing, red teaming, reporting and mitigation). You'll see topics from earlier domains again here, approached from the attacker's side and the tester's side. No testing or red team background is assumed; every term is defined where it first appears. I've followed the ISC2 outline section by section, and each section ends with how I expect it to show up on the exam.

One idea runs through the domain: a control you haven't tested is a hope. AI systems change with every model update, prompt edit and new data source, so testing has to be continuous, and every finding should end as a regression test that runs on every future release.

Everything here is my own work, built from public frameworks, especially MITRE ATLAS, the OWASP GenAI Security Project, NIST's adversarial machine learning taxonomy and the Cloud Security Alliance's agentic AI guidance. If a reference book frames something differently, trust the book.

6.1 AI threat modeling and intelligence

Threat modeling methodologies

Domain 4 introduced threat modeling. The outline names three methodologies, and each fits a different job.

STRIDE is a checklist of six threat categories, applied to each component of a system:

Category AI example
Spoofing Impersonating a user to the model, or a fake agent presenting itself to other agents
Tampering Poisoned training data, altered weights, a modified system prompt
Repudiation Agent actions that no log can tie to an identity
Information disclosure Training data extraction, a leaked system prompt, RAG returning restricted documents
Denial of service Token exhaustion and inputs crafted to burn compute
Elevation of privilege Prompt injection that turns an agent's permissions to the attacker's purpose

PASTA (Process for Attack Simulation and Threat Analysis) is risk-centric and runs in seven stages: define business objectives, define the technical scope, break the application into its parts, analyze threats, analyze vulnerabilities, model attacks (often as attack trees that branch through the ways to reach a goal), and analyze risk and impact. Its strength is tying every threat back to business impact.

MAESTRO (Multi-Agent Environment, Security, Threat, Risk and Outcome) is a threat modeling framework the Cloud Security Alliance published in 2025 for agentic AI. Its premise is that STRIDE and PASTA were built for software that behaves predictably inside clear boundaries, and autonomous agents don't. MAESTRO breaks an agentic system into seven layers, looks for threats at each one, and then looks for threats that travel between layers, like a supply chain compromise in an agent framework that spreads upward or a data leak that cascades across components. Because agents keep changing, it treats threat modeling as a cycle you repeat, never a one-time document.

MAESTRO threat models an agentic system layer by layer
MAESTRO's seven layers: with an example threat at each

A rough rule for choosing: STRIDE for a quick pass over each component, PASTA when the analysis has to line up with business risk, and MAESTRO for agentic and multi-agent systems.

Threat actors, objectives and insider threats

Actor Typical objectives
Nation-states Espionage, stealing model weights and research, influence operations
Cybercriminals Fraud (including deepfake fraud), extortion, reselling access, hijacking compute
Hacktivists Embarrassing an organization, such as jailbreaking its chatbot into saying something offensive
Competitors Copying a model through extraction, stealing intellectual property
Opportunists and researchers Jailbreaking for curiosity, reputation or bug bounties
Malicious insiders Poisoning data, stealing weights or data
Negligent insiders Pasting confidential data into public tools, misconfiguring systems
Non-human actors Compromised agents, malicious tool servers, and AI agents that attackers use to automate attacks

That last row is new and growing. In 2025, an AI developer reported disrupting an espionage campaign in which an AI agent carried out most of the intrusion steps with little human direction.

NIST's adversarial machine learning taxonomy (NIST AI 100-2) groups attacker objectives into availability breakdown, integrity violation and privacy compromise, and adds misuse for generative AI, where the goal is getting the system to produce output its provider intended to prevent. It also describes attacker knowledge: white-box attackers know the model's internals, black-box attackers can only send inputs and see outputs, and gray-box attackers sit in between.

Adversarial ML attacks

Attack When it happens What it does
Evasion At inference Crafted inputs, called adversarial examples, make the model misclassify, like stickers that make a vision system misread a stop sign, or malware tweaked to slip past an ML-based detector
Poisoning During training Corrupted training data changes the model's behavior (Domain 3)
Model extraction At inference Repeated queries let the attacker train a copy (Domain 3)
Model inversion At inference The attacker reconstructs sensitive training data from the model's outputs, like rebuilding a recognizable face from a face recognition model
Backdoors During training A hidden trigger makes the model misbehave on command (Domain 3)

Keep extraction and inversion apart: extraction steals the model, and inversion steals what the model learned about people.

Threats specific to GenAI, RAG, agentic and multimodal systems

Availability attacks

Compute exhaustion uses inputs crafted to make a model work as hard as possible. Token exhaustion stuffs huge contexts into requests, forces maximum-length outputs or traps agents in loops. Denial of wallet turns either one into a financial attack on a pay-per-use service. The controls are the quotas, limits and circuit breakers from Domains 3, 4 and 5.

6.2 Secure AI development lifecycle and change management

The secure AI development lifecycle (AIDLC) builds security into every stage of creating and running an AI system. Domains 3, 4 and 5 covered the pipeline, gates and operations in detail; this section puts them in lifecycle order and adds change management.

Security across the AI/ML pipeline and CI/CD

Stage Security activities
Plan and design Threat model, risk tier, privacy review, build-versus-buy decision
Data Provenance, classification, validation, minimization
Develop and train Isolated environments, reviewed code, dependency scanning, protected secrets
Evaluate Accuracy, bias and adversarial testing; red teaming for higher tiers
Deploy Signing, pipeline gates, staged rollout
Operate Monitoring, incident response, reassessment triggers
Retire Decommissioning, disposal, identity cleanup

The principle is shift left: find problems as early as possible, because a flaw caught in design costs far less than one found in production. NIST's Secure Software Development Framework (SP 800-218) has a companion profile for generative AI and foundation models (SP 800-218A) that maps secure development practices onto AI work.

Environmental control adaptations

Standard environmental controls need adjusting for how AI teams actually work.

AI-specific change management

Change management makes sure every change is requested, assessed, approved, recorded and reversible. In AI, more things count as changes than in traditional software:

Version all of them, so any output can be traced to the exact model, data, index and prompt that produced it. A change that could alter a system's risk tier triggers a fresh risk assessment (Domain 2). Emergency changes get an expedited path, with review after the fact.

Staged rollouts and rollback procedures

Pattern How it works
Canary deployment Send a small share of traffic to the new version, watch the metrics, and widen gradually
Blue-green deployment Run two identical environments and switch all traffic at once, with an instant switch back
Shadow deployment Give the new version a copy of real traffic and compare its outputs without showing them to anyone
A/B testing Compare two versions on live traffic to see which performs better
Feature flags Turn an AI feature on or off without redeploying

A rollback procedure defines what triggers it, who can order it, the exact steps in a runbook and a time target. Keep the previous version ready to take traffic, and remember that pieces must stay compatible: rolling back to an older embedding model means rolling back to an index built with it. Test the rollback before you depend on it.

6.3 AI supply chain, procurement and provenance

Domain 2 covered vendor contracts and Domain 3 covered vetting datasets and models. This section adds the assurance reports you'll be handed, the legal side of acquiring AI, and how to verify that what you received is what you think it is.

Third-party vendor risks, certifications and assessment reports

Assurance What it tells you
SOC 2 Type I An auditor's opinion that controls were designed properly at a single point in time
SOC 2 Type II An auditor's opinion that controls actually operated effectively over a period, usually 6 to 12 months
ISO/IEC 27001 Certification of an information security management system
ISO/IEC 42001 Certification of an AI management system
FedRAMP Authorization for cloud services used by US federal agencies
CSA STAR Cloud Security Alliance's registry of cloud provider security assessments
Pen test and red team reports Results of hands-on testing, often shared as a summary
Model and system cards The provider's own account of evaluations, limitations and safety testing

Type I versus Type II is a classic exam distinction; Type II is the stronger evidence, because it shows controls working over time. Read the scope of any report carefully. A SOC 2 report that excludes the AI service you're buying tells you nothing about it. When a report's period has ended and the next isn't out yet, a bridge letter from the vendor covers the gap.

AI-specific vendor risks include using your data for training, unannounced model changes, undisclosed subprocessors, concentration on a few providers, the vendor's financial health and geopolitical exposure.

Procurement itself is a control. If no purchase order goes through without security, privacy and legal review, AI that arrives through procurement gets vetted, and a lot of shadow AI gets caught at the door.

Pedigree and provenance verification

Pedigree is a component's ancestry: what it was derived from, like the base model, training data and every party that modified it. Provenance is the verifiable record of where it came from and who has handled it. You verify them with:

Know the difference for the exam: a hash proves integrity only, and a signature proves integrity and origin.

AI-BOM for hardware, software and data

The outline extends the AI bill of materials (Domain 2) across three kinds of dependency:

CycloneDX and SPDX both support these component types. An AI-BOM pays off when a GPU driver vulnerability is announced and you need to know which systems use that driver, when legal asks which models have restrictive licenses, and when a regulator asks for documentation. Generate it automatically in the pipeline so it stays current.

6.4 AI security testing and red teaming

The rest of this domain follows a testing engagement from planning to fixes:

A red team engagement ends in fixes and new tests
AI red team engagement: six steps, covered in 6.4 through 6.6

Scope, objectives and test scenarios

Scope sets what gets tested: which systems and components (the model, the RAG pipeline, tools, agents, infrastructure), which environments, and what's explicitly out of bounds. A staging environment that mirrors production is the usual target; testing production needs extra care.

Rules of engagement (RoE) set how testing happens: what testers may do, when, how they handle sensitive or harmful outputs they produce, when they must stop, and who to contact. Get written authorization from the system owner before any testing starts. If the system relies on a third-party model API, check that the provider's terms allow security testing.

Objectives turn worries into questions the test can answer. Can an outside user make the agent send customer data out? Can the chatbot be pushed into giving dangerous advice? How often do the guardrails stop known jailbreaks? Some objectives come from regulation; the EU AI Act requires adversarial testing of general-purpose models with systemic risk.

Test scenarios come from the threat model and the risk tier. Use realistic attacker personas, such as an outside user, a malicious insider or the author of a poisoned document, and map scenarios to MITRE ATLAS techniques and OWASP categories to check coverage.

Adversarial testing for GenAI, RAG and agentic architectures

Architecture What to test
Generative AI Jailbreaks (role-play, encoded text, gradual escalation over many turns, and prompts packed with many fake example dialogues), system prompt extraction, harmful content, sensitive data leakage
RAG Planted documents carrying indirect injection, unauthorized users retrieving restricted documents, index poisoning, manipulated citations
Agentic AI Tool misuse through injection, privilege escalation, excessive agency, goal hijacking, memory poisoning that persists across sessions, attacks spreading between agents, runaway loops, and tricking the human approver with a misleading summary

Many attacks don't land in one message. Test multi-turn conversations, where an attacker escalates gradually, as well as single prompts.

AI red teaming methodologies

AI red teaming is structured adversarial testing that emulates real attackers to find how an AI system fails, covering security failures and harmful behavior alike. It differs from a traditional penetration test in three ways:

Teams combine manual red teaming, where skilled people find novel attack paths, with automated red teaming, where tools generate thousands of attack variations. Domain experts join for specialized harms, such as medical or financial advice. Purple teaming puts red and blue teams together, so every successful attack immediately improves detection. The Cloud Security Alliance's Agentic AI Red Teaming Guide lays out test categories specific to agents.

AI-assisted security tools

AI now helps test AI. Open source tools such as Microsoft's PyRIT, garak and promptfoo automate attacks and score the results. Common techniques include using one model as the attacker to generate attack variations, using another as a judge to decide whether each attack succeeded, fuzzing prompts with random mutations, and letting agents explore attack paths across tools.

These tools have limits. Judge models make mistakes, so people should review a sample of their verdicts. Broad automated coverage doesn't prove a system is safe. Testing tools need least privilege and sandboxing like any other agent. And every finding has to be reproducible before it goes in a report.

6.5 Evaluating, validating and reporting findings

Triage of assessment findings

Findings get ranked on four factors:

Compare two findings. A jailbreak that works 2 times in 100 and produces mildly off-brand text is low priority. An indirect injection that works 9 times in 10 and sends customer data to an outside server is urgent. Severity alone wouldn't tell you that; all four factors together do.

Security vulnerabilities versus model limitations

A security vulnerability is a weakness an attacker can exploit to break a security policy: an injection that triggers an unauthorized tool call, a data leak, or a guardrail bypass that enables prohibited use. A model limitation is a built-in shortcoming of current models that happens without any attacker: hallucinations, inconsistent answers, outdated knowledge, weaker performance in some languages.

The distinction matters because the owners and fixes differ. Vulnerabilities go to security remediation with deadlines, and sometimes disclosure. Limitations go to product design: grounding answers in retrieved sources, showing citations, adding human review, telling users what the system can't do, restricting the use case, and documenting it all in the model card.

The line can move. A limitation becomes a security issue when an attacker can exploit it, like registering package names a coding assistant tends to invent, or when the system's design lets it cause harm, like hallucinated medical advice reaching patients with no review.

Correlating findings with industry frameworks

Map each finding to a MITRE ATLAS technique ID and an OWASP Top 10 for LLM Applications category, and to OWASP's agentic guidance for agent findings. Mapping gives everyone a shared language, shows which categories you haven't tested yet, links each finding to known mitigations and lets auditors and regulators recognize what you found.

The 2025 edition of the OWASP list had ten categories: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. The 2026 update kept prompt injection and sensitive information disclosure at the top. Check OWASP's site for the current list before the exam.

Documenting evidence and reproducibility

Each finding should record:

The test of good documentation is reproducibility: someone else following your steps gets the same result, at roughly the stated rate. Evidence may contain harmful outputs or sensitive data, so store it with restricted access.

Risk communication for different audiences

Audience What they need
Engineers and the SOC Exact reproduction steps, root cause, technical fix, framework IDs
Executives and the board Business impact in plain language, likelihood, cost, the decision needed, options and trade-offs
Legal and compliance Regulatory exposure and any notification implications

For non-technical audiences, lead with the decision you need from them, skip the jargon, and translate numbers into plain terms: "this works about one time in ten." Avoid both alarmism and downplaying; either one costs you credibility the next time.

6.6 Defining, implementing and validating mitigations

Targeted mitigations

Mitigation What it addresses How it works
Adversarial training Evasion and some jailbreaks Training includes adversarial examples, so the model learns to resist them. It costs compute, can reduce accuracy, and only covers attack types you trained on
Provenance tracking Poisoning and supply chain attacks Recording and verifying where every dataset and model came from (Domains 3 and 6.3)
Data validation Poisoning and label flipping Schema checks, statistical tests and outlier detection before data reaches training

Filtering, sandboxing, rate limiting and watermarking

These four general-purpose controls came up in earlier domains. Here's what each one is for:

Control Main threats it reduces Covered in
Filtering inputs and outputs Prompt injection, harmful content, data leakage Domain 4
Sandboxing Malicious code execution, tool misuse Domain 4
Rate limiting Model extraction, denial of service, denial of wallet Domain 3
Watermarking Undisclosed synthetic content, proving model theft Domains 2 and 3

Prioritizing mitigations by risk, feasibility and business context

You'll rarely fix everything at once, so rank mitigations on three things:

Start with quick wins, mitigations that cut a lot of risk for little effort, like removing a tool permission an agent doesn't need. Prefer architectural fixes over detection: taking away a dangerous capability beats trying to catch every misuse of it. When the root cause can't be fixed, as with prompt injection, layer several partial mitigations. And weigh side effects; a filter so strict that it blocks one legitimate request in five will push people toward unapproved tools, creating a new risk.

Validating mitigations with regression testing

A mitigation nobody has tested is an assumption. To validate one:

  1. rerun the original attack and measure its new success rate;
  2. run variants of it, because attackers will mutate it;
  3. run the full regression suite to confirm nothing else broke, including accuracy, fairness and the other guardrails;
  4. measure side effects such as false positives and added latency; and
  5. add the attack and its variants to the permanent test suite.

Continuous monitoring for residual risk

Some risk always remains after mitigation. That residual risk is formally accepted by its owner (Domain 2) and then watched:

That monitoring feeds the next round of threat modeling in 6.1, which closes the loop.

Practice questions

These are my own questions in the ISC2 style. Pick the BEST answer, and when two look right, put the safety of people first, then legal obligations, then the mission.

  1. A team is threat modeling a system where several AI agents call tools and hand tasks to one another. Which methodology is BEST suited?
    • A. STRIDE
    • B. PASTA
    • C. MAESTRO
    • D. A standard vulnerability scan
  2. An attacker sends queries to a face recognition API and reconstructs recognizable images of people whose photos were in the training data. Which attack is this?
    • A. Model extraction
    • B. Model inversion
    • C. Evasion
    • D. Data poisoning
  3. A red team is about to test a customer chatbot built on a third-party model API. What must be in place FIRST?
    • A. Written authorization and rules of engagement, including confirmation that the provider's terms allow testing
    • B. An automated attack tool
    • C. A list of known jailbreak prompts
    • D. A draft report template
  4. A red team finds a jailbreak that succeeded 3 times in 50 attempts. How should it be reported?
    • A. As not reproducible, since it usually fails
    • B. As a critical finding, since any success is a failure
    • C. Only if it works on the first attempt during retesting
    • D. With its success rate and number of attempts, the exact inputs and the model version tested
  5. A health information chatbot sometimes invents medication dosages. No attacker is involved. What is the BEST response?
    • A. Classify it as a security vulnerability and patch the model
    • B. Accept it, since hallucination is a known model limitation
    • C. Stop the chatbot from giving dosage information, route those questions to clinicians, and document the limitation
    • D. Wait for the vendor's next model version
  6. Through an indirect prompt injection, an agent can be made to email customer data to outside addresses. Which mitigation is MOST effective?
    • A. Restrict the agent's email tool to approved internal recipients
    • B. Add a stronger injection-detection filter
    • C. Add a warning to the system prompt
    • D. Increase logging of the agent's email activity
  7. A new input filter has been deployed to block a jailbreak the red team found. What BEST validates it?
    • A. The vendor's confirmation that the filter works
    • B. Rerunning the original attack and variants of it, plus the full regression suite, while measuring false positives
    • C. Monitoring production for a week
    • D. A code review of the filter
  8. The security team is presenting red team findings to the board. What should the presentation lead with?
    • A. The ATLAS technique IDs for each finding
    • B. The full reproduction steps
    • C. Every finding in order of discovery
    • D. The business impact and the decision the board needs to make
  9. An AI vendor provides a SOC 2 Type I report. What is its main limitation?
    • A. It shows controls were designed properly at one point in time, but not that they operated effectively over time
    • B. It covers only physical security
    • C. It's a self-assessment by the vendor
    • D. It applies only to on-premises systems
  10. A team downloads a model file, and its SHA-256 hash matches the value on the publisher's website. What does this confirm?
    • A. The model is free of backdoors
    • B. The file is identical to the one the hash was calculated from
    • C. The publisher is who they claim to be
    • D. The model's license allows commercial use
  11. An engineer wants to change how the RAG system splits documents into chunks. How should the change be handled?
    • A. As a minor configuration tweak with no review
    • B. Only by the data science team, since it doesn't touch the model
    • C. Through change management, versioned and tested like a model change
    • D. Deployed straight to production and monitored
  12. Prompt injection can't be fully eliminated in an agent that reads outside web content. What is the BEST approach?
    • A. Stop using AI agents entirely
    • B. Rely on the model provider's safety training
    • C. Add a system prompt instruction to ignore injected text
    • D. Layer mitigations, have the owner formally accept the residual risk, and monitor for that specific risk
Show the answer key
  1. C. MAESTRO was built for agentic and multi-agent systems, including threats that move between layers. STRIDE and PASTA can help, but they assume more predictable software.
  2. B. Inversion recovers what the model learned about people. Extraction (A) copies the model itself.
  3. A. Testing without written authorization can be illegal and breach the provider's terms. Tools and prompts come later.
  4. D. AI results are probabilistic, so a success rate over stated attempts, with exact inputs and versions, is how you make the finding reproducible. Three in fifty is real and exploitable at scale.
  5. C. No attacker means it's a model limitation, but invented dosages put patients at risk, so the use case gets restricted and people get routed to clinicians. Simply accepting it (B) ignores the safety risk.
  6. A. Removing the dangerous capability is stronger than trying to detect every injection. A filter (B) helps as a layer but will miss some attacks, and a system prompt warning (C) can be overridden.
  7. B. Validation means proving the attack and its variants no longer work and that nothing else broke. Vendor assurance (A) and passive monitoring (C) don't test anything.
  8. D. Boards decide on risk, so give them the impact and the decision. Technical detail belongs in the engineering report.
  9. A. Type I covers design at a point in time; Type II covers operation over a period.
  10. B. A matching hash proves integrity only. It says nothing about backdoors, licensing or who produced the file; a digital signature is what proves origin.
  11. C. Chunking changes what the model retrieves and therefore what it says. Retrieval settings are versioned and go through change management.
  12. D. When a risk can't be eliminated, it gets layered controls, a documented acceptance and monitoring aimed at it. Abandoning agents (A) may not fit the mission, and B and C rely on the model to protect itself.

Sources