ISC2 AI Security Pilot Exam · Free study guide
Domain 6 Study Notes: AI Threats, Testing, and Lifecycle Management
How to use these notes
Domain 6 ties the other five together. It covers how to think like an attacker (threat modeling), how to build AI with security in every stage (the secure AI development lifecycle and supply chain), and how to prove your defenses work (testing, red teaming, reporting and mitigation). You'll see topics from earlier domains again here, approached from the attacker's side and the tester's side. No testing or red team background is assumed; every term is defined where it first appears. I've followed the ISC2 outline section by section, and each section ends with how I expect it to show up on the exam.
One idea runs through the domain: a control you haven't tested is a hope. AI systems change with every model update, prompt edit and new data source, so testing has to be continuous, and every finding should end as a regression test that runs on every future release.
Everything here is my own work, built from public frameworks, especially MITRE ATLAS, the OWASP GenAI Security Project, NIST's adversarial machine learning taxonomy and the Cloud Security Alliance's agentic AI guidance. If a reference book frames something differently, trust the book.
6.1 AI threat modeling and intelligence
Threat modeling methodologies
Domain 4 introduced threat modeling. The outline names three methodologies, and each fits a different job.
STRIDE is a checklist of six threat categories, applied to each component of a system:
| Category | AI example |
|---|---|
| Spoofing | Impersonating a user to the model, or a fake agent presenting itself to other agents |
| Tampering | Poisoned training data, altered weights, a modified system prompt |
| Repudiation | Agent actions that no log can tie to an identity |
| Information disclosure | Training data extraction, a leaked system prompt, RAG returning restricted documents |
| Denial of service | Token exhaustion and inputs crafted to burn compute |
| Elevation of privilege | Prompt injection that turns an agent's permissions to the attacker's purpose |
PASTA (Process for Attack Simulation and Threat Analysis) is risk-centric and runs in seven stages: define business objectives, define the technical scope, break the application into its parts, analyze threats, analyze vulnerabilities, model attacks (often as attack trees that branch through the ways to reach a goal), and analyze risk and impact. Its strength is tying every threat back to business impact.
MAESTRO (Multi-Agent Environment, Security, Threat, Risk and Outcome) is a threat modeling framework the Cloud Security Alliance published in 2025 for agentic AI. Its premise is that STRIDE and PASTA were built for software that behaves predictably inside clear boundaries, and autonomous agents don't. MAESTRO breaks an agentic system into seven layers, looks for threats at each one, and then looks for threats that travel between layers, like a supply chain compromise in an agent framework that spreads upward or a data leak that cascades across components. Because agents keep changing, it treats threat modeling as a cycle you repeat, never a one-time document.

A rough rule for choosing: STRIDE for a quick pass over each component, PASTA when the analysis has to line up with business risk, and MAESTRO for agentic and multi-agent systems.
Threat actors, objectives and insider threats
| Actor | Typical objectives |
|---|---|
| Nation-states | Espionage, stealing model weights and research, influence operations |
| Cybercriminals | Fraud (including deepfake fraud), extortion, reselling access, hijacking compute |
| Hacktivists | Embarrassing an organization, such as jailbreaking its chatbot into saying something offensive |
| Competitors | Copying a model through extraction, stealing intellectual property |
| Opportunists and researchers | Jailbreaking for curiosity, reputation or bug bounties |
| Malicious insiders | Poisoning data, stealing weights or data |
| Negligent insiders | Pasting confidential data into public tools, misconfiguring systems |
| Non-human actors | Compromised agents, malicious tool servers, and AI agents that attackers use to automate attacks |
That last row is new and growing. In 2025, an AI developer reported disrupting an espionage campaign in which an AI agent carried out most of the intrusion steps with little human direction.
NIST's adversarial machine learning taxonomy (NIST AI 100-2) groups attacker objectives into availability breakdown, integrity violation and privacy compromise, and adds misuse for generative AI, where the goal is getting the system to produce output its provider intended to prevent. It also describes attacker knowledge: white-box attackers know the model's internals, black-box attackers can only send inputs and see outputs, and gray-box attackers sit in between.
Adversarial ML attacks
| Attack | When it happens | What it does |
|---|---|---|
| Evasion | At inference | Crafted inputs, called adversarial examples, make the model misclassify, like stickers that make a vision system misread a stop sign, or malware tweaked to slip past an ML-based detector |
| Poisoning | During training | Corrupted training data changes the model's behavior (Domain 3) |
| Model extraction | At inference | Repeated queries let the attacker train a copy (Domain 3) |
| Model inversion | At inference | The attacker reconstructs sensitive training data from the model's outputs, like rebuilding a recognizable face from a face recognition model |
| Backdoors | During training | A hidden trigger makes the model misbehave on command (Domain 3) |
Keep extraction and inversion apart: extraction steals the model, and inversion steals what the model learned about people.
Threats specific to GenAI, RAG, agentic and multimodal systems
- Generative AI: prompt injection, jailbreaks, system prompt leakage, sensitive data disclosure and harmful content. Attackers also exploit hallucinations; when coding assistants invent package names that don't exist, attackers register those names with malicious code, a trick called slopsquatting.
- RAG: indirect injection through documents, knowledge base poisoning, retrieval that ignores permissions, and embedding inversion.
- Agentic systems: tool misuse, goal hijacking, memory poisoning, privilege abuse, rogue agents, and failures that cascade from one agent to the next.
- Multimodal systems, which handle images, audio or video as well as text: instructions hidden in images where people won't notice them, audio carrying hidden commands, adversarial images, and deepfakes used for social engineering.
Availability attacks
Compute exhaustion uses inputs crafted to make a model work as hard as possible. Token exhaustion stuffs huge contexts into requests, forces maximum-length outputs or traps agents in loops. Denial of wallet turns either one into a financial attack on a pay-per-use service. The controls are the quotas, limits and circuit breakers from Domains 3, 4 and 5.
6.2 Secure AI development lifecycle and change management
The secure AI development lifecycle (AIDLC) builds security into every stage of creating and running an AI system. Domains 3, 4 and 5 covered the pipeline, gates and operations in detail; this section puts them in lifecycle order and adds change management.
Security across the AI/ML pipeline and CI/CD
| Stage | Security activities |
|---|---|
| Plan and design | Threat model, risk tier, privacy review, build-versus-buy decision |
| Data | Provenance, classification, validation, minimization |
| Develop and train | Isolated environments, reviewed code, dependency scanning, protected secrets |
| Evaluate | Accuracy, bias and adversarial testing; red teaming for higher tiers |
| Deploy | Signing, pipeline gates, staged rollout |
| Operate | Monitoring, incident response, reassessment triggers |
| Retire | Decommissioning, disposal, identity cleanup |
The principle is shift left: find problems as early as possible, because a flaw caught in design costs far less than one found in production. NIST's Secure Software Development Framework (SP 800-218) has a companion profile for generative AI and foundation models (SP 800-218A) that maps secure development practices onto AI work.
Environmental control adaptations
Standard environmental controls need adjusting for how AI teams actually work.
- Separation: keep development, testing and production apart for models, data and keys alike. Production data doesn't go into development without masking, and experimental notebooks don't run in production accounts.
- Least privilege: data scientists often want broad access to production data for experiments. Give them masked or synthetic data and approved sandboxes instead, and grant access to real data case by case.
- Secure API integration: issue separate model API keys for each environment, never use production keys in development, route model calls through an API gateway, and control outbound network traffic.
AI-specific change management
Change management makes sure every change is requested, assessed, approved, recorded and reversible. In AI, more things count as changes than in traditional software:
- Model versioning: new weights, fine-tunes, adapters or a new provider version.
- Dataset versioning: the data a model was trained or retrained on.
- Retrieval versioning: what's in the RAG index, the embedding model, how documents are split into chunks, and retrieval settings.
- Prompts and guardrails: the system prompt, filters and thresholds.
- Tools and permissions: what an agent can call and do.
Version all of them, so any output can be traced to the exact model, data, index and prompt that produced it. A change that could alter a system's risk tier triggers a fresh risk assessment (Domain 2). Emergency changes get an expedited path, with review after the fact.
Staged rollouts and rollback procedures
| Pattern | How it works |
|---|---|
| Canary deployment | Send a small share of traffic to the new version, watch the metrics, and widen gradually |
| Blue-green deployment | Run two identical environments and switch all traffic at once, with an instant switch back |
| Shadow deployment | Give the new version a copy of real traffic and compare its outputs without showing them to anyone |
| A/B testing | Compare two versions on live traffic to see which performs better |
| Feature flags | Turn an AI feature on or off without redeploying |
A rollback procedure defines what triggers it, who can order it, the exact steps in a runbook and a time target. Keep the previous version ready to take traffic, and remember that pieces must stay compatible: rolling back to an older embedding model means rolling back to an index built with it. Test the rollback before you depend on it.
6.3 AI supply chain, procurement and provenance
Domain 2 covered vendor contracts and Domain 3 covered vetting datasets and models. This section adds the assurance reports you'll be handed, the legal side of acquiring AI, and how to verify that what you received is what you think it is.
Third-party vendor risks, certifications and assessment reports
| Assurance | What it tells you |
|---|---|
| SOC 2 Type I | An auditor's opinion that controls were designed properly at a single point in time |
| SOC 2 Type II | An auditor's opinion that controls actually operated effectively over a period, usually 6 to 12 months |
| ISO/IEC 27001 | Certification of an information security management system |
| ISO/IEC 42001 | Certification of an AI management system |
| FedRAMP | Authorization for cloud services used by US federal agencies |
| CSA STAR | Cloud Security Alliance's registry of cloud provider security assessments |
| Pen test and red team reports | Results of hands-on testing, often shared as a summary |
| Model and system cards | The provider's own account of evaluations, limitations and safety testing |
Type I versus Type II is a classic exam distinction; Type II is the stronger evidence, because it shows controls working over time. Read the scope of any report carefully. A SOC 2 report that excludes the AI service you're buying tells you nothing about it. When a report's period has ended and the next isn't out yet, a bridge letter from the vendor covers the gap.
AI-specific vendor risks include using your data for training, unannounced model changes, undisclosed subprocessors, concentration on a few providers, the vendor's financial health and geopolitical exposure.
Acquisition, data ownership, licensing and legal considerations
- Data ownership: the contract should say who owns your inputs, the outputs, any fine-tuned model and any embeddings built from your data. The usual goal is that you own your data and outputs, and the vendor gets no right to train on them.
- Licensing: open models come under very different licenses. Some use permissive open source licenses like Apache 2.0 or MIT. Others use custom licenses with acceptable use policies, or require a separate agreement above a certain number of monthly users. Legal reviews the license before anyone builds on the model.
- Regulatory role: under the EU AI Act, a deployer can become a provider, with a provider's heavier obligations, by substantially modifying a high-risk system or putting its own name on it. Fine-tuning and rebranding decisions have legal consequences.
- Other legal factors: export controls on advanced chips and some model technology, sector rules like HIPAA, and IP indemnity for outputs.
Procurement itself is a control. If no purchase order goes through without security, privacy and legal review, AI that arrives through procurement gets vetted, and a lot of shadow AI gets caught at the door.
Pedigree and provenance verification
Pedigree is a component's ancestry: what it was derived from, like the base model, training data and every party that modified it. Provenance is the verifiable record of where it came from and who has handled it. You verify them with:
- cryptographic hashing, which proves the artifact hasn't changed;
- digital signatures, which prove who produced it as well as that it hasn't changed;
- signed build attestations (the SLSA and in-toto frameworks), which record how an artifact was built; and
- transparency logs, such as Sigstore's, which keep a public, tamper-evident record of signatures.
Know the difference for the exam: a hash proves integrity only, and a signature proves integrity and origin.
AI-BOM for hardware, software and data
The outline extends the AI bill of materials (Domain 2) across three kinds of dependency:
- Hardware: accelerator models and firmware versions, confidential computing support, and where the hardware is hosted.
- Software: frameworks, libraries, the serving stack, agent frameworks, tools and MCP servers, plus the models themselves with versions, hashes, adapters and licenses.
- Data: training, fine-tuning, evaluation and RAG sources, each with its version and license.
CycloneDX and SPDX both support these component types. An AI-BOM pays off when a GPU driver vulnerability is announced and you need to know which systems use that driver, when legal asks which models have restrictive licenses, and when a regulator asks for documentation. Generate it automatically in the pipeline so it stays current.
6.4 AI security testing and red teaming
The rest of this domain follows a testing engagement from planning to fixes:

Scope, objectives and test scenarios
Scope sets what gets tested: which systems and components (the model, the RAG pipeline, tools, agents, infrastructure), which environments, and what's explicitly out of bounds. A staging environment that mirrors production is the usual target; testing production needs extra care.
Rules of engagement (RoE) set how testing happens: what testers may do, when, how they handle sensitive or harmful outputs they produce, when they must stop, and who to contact. Get written authorization from the system owner before any testing starts. If the system relies on a third-party model API, check that the provider's terms allow security testing.
Objectives turn worries into questions the test can answer. Can an outside user make the agent send customer data out? Can the chatbot be pushed into giving dangerous advice? How often do the guardrails stop known jailbreaks? Some objectives come from regulation; the EU AI Act requires adversarial testing of general-purpose models with systemic risk.
Test scenarios come from the threat model and the risk tier. Use realistic attacker personas, such as an outside user, a malicious insider or the author of a poisoned document, and map scenarios to MITRE ATLAS techniques and OWASP categories to check coverage.
Adversarial testing for GenAI, RAG and agentic architectures
| Architecture | What to test |
|---|---|
| Generative AI | Jailbreaks (role-play, encoded text, gradual escalation over many turns, and prompts packed with many fake example dialogues), system prompt extraction, harmful content, sensitive data leakage |
| RAG | Planted documents carrying indirect injection, unauthorized users retrieving restricted documents, index poisoning, manipulated citations |
| Agentic AI | Tool misuse through injection, privilege escalation, excessive agency, goal hijacking, memory poisoning that persists across sessions, attacks spreading between agents, runaway loops, and tricking the human approver with a misleading summary |
Many attacks don't land in one message. Test multi-turn conversations, where an attacker escalates gradually, as well as single prompts.
AI red teaming methodologies
AI red teaming is structured adversarial testing that emulates real attackers to find how an AI system fails, covering security failures and harmful behavior alike. It differs from a traditional penetration test in three ways:
- Broader targets: harmful content, bias and misuse count as findings alongside technical exploits.
- Probabilistic results: an attack may work once in twenty tries, so red teams repeat attempts and report an attack success rate instead of a simple pass or fail.
- System-level focus: the model is tested together with its prompts, retrieval, tools and guardrails, because that's what attackers face.
Teams combine manual red teaming, where skilled people find novel attack paths, with automated red teaming, where tools generate thousands of attack variations. Domain experts join for specialized harms, such as medical or financial advice. Purple teaming puts red and blue teams together, so every successful attack immediately improves detection. The Cloud Security Alliance's Agentic AI Red Teaming Guide lays out test categories specific to agents.
AI-assisted security tools
AI now helps test AI. Open source tools such as Microsoft's PyRIT, garak and promptfoo automate attacks and score the results. Common techniques include using one model as the attacker to generate attack variations, using another as a judge to decide whether each attack succeeded, fuzzing prompts with random mutations, and letting agents explore attack paths across tools.
These tools have limits. Judge models make mistakes, so people should review a sample of their verdicts. Broad automated coverage doesn't prove a system is safe. Testing tools need least privilege and sandboxing like any other agent. And every finding has to be reproducible before it goes in a report.
6.5 Evaluating, validating and reporting findings
Triage of assessment findings
Findings get ranked on four factors:
- Severity: the worst realistic harm if the finding is exploited.
- Exploitability: how much skill and access an attacker needs, and whether it works remotely with only normal user access.
- Reproducibility: how reliably it works, stated as a success rate over a set number of attempts.
- Business impact: what it means for people, data, money, reputation and regulatory standing.
Compare two findings. A jailbreak that works 2 times in 100 and produces mildly off-brand text is low priority. An indirect injection that works 9 times in 10 and sends customer data to an outside server is urgent. Severity alone wouldn't tell you that; all four factors together do.
Security vulnerabilities versus model limitations
A security vulnerability is a weakness an attacker can exploit to break a security policy: an injection that triggers an unauthorized tool call, a data leak, or a guardrail bypass that enables prohibited use. A model limitation is a built-in shortcoming of current models that happens without any attacker: hallucinations, inconsistent answers, outdated knowledge, weaker performance in some languages.
The distinction matters because the owners and fixes differ. Vulnerabilities go to security remediation with deadlines, and sometimes disclosure. Limitations go to product design: grounding answers in retrieved sources, showing citations, adding human review, telling users what the system can't do, restricting the use case, and documenting it all in the model card.
The line can move. A limitation becomes a security issue when an attacker can exploit it, like registering package names a coding assistant tends to invent, or when the system's design lets it cause harm, like hallucinated medical advice reaching patients with no review.
Correlating findings with industry frameworks
Map each finding to a MITRE ATLAS technique ID and an OWASP Top 10 for LLM Applications category, and to OWASP's agentic guidance for agent findings. Mapping gives everyone a shared language, shows which categories you haven't tested yet, links each finding to known mitigations and lets auditors and regulators recognize what you found.
The 2025 edition of the OWASP list had ten categories: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption. The 2026 update kept prompt injection and sensitive information disclosure at the top. Check OWASP's site for the current list before the exam.
Documenting evidence and reproducibility
Each finding should record:
- the affected system and the exact model, prompt and configuration versions tested;
- the date and the tester;
- the exact inputs, including full multi-turn transcripts and any planted documents;
- settings that affect randomness;
- the observed output, with logs or screenshots;
- the success rate over a stated number of attempts;
- the impact, severity and framework mapping; and
- the recommended mitigation.
The test of good documentation is reproducibility: someone else following your steps gets the same result, at roughly the stated rate. Evidence may contain harmful outputs or sensitive data, so store it with restricted access.
Risk communication for different audiences
| Audience | What they need |
|---|---|
| Engineers and the SOC | Exact reproduction steps, root cause, technical fix, framework IDs |
| Executives and the board | Business impact in plain language, likelihood, cost, the decision needed, options and trade-offs |
| Legal and compliance | Regulatory exposure and any notification implications |
For non-technical audiences, lead with the decision you need from them, skip the jargon, and translate numbers into plain terms: "this works about one time in ten." Avoid both alarmism and downplaying; either one costs you credibility the next time.
6.6 Defining, implementing and validating mitigations
Targeted mitigations
| Mitigation | What it addresses | How it works |
|---|---|---|
| Adversarial training | Evasion and some jailbreaks | Training includes adversarial examples, so the model learns to resist them. It costs compute, can reduce accuracy, and only covers attack types you trained on |
| Provenance tracking | Poisoning and supply chain attacks | Recording and verifying where every dataset and model came from (Domains 3 and 6.3) |
| Data validation | Poisoning and label flipping | Schema checks, statistical tests and outlier detection before data reaches training |
Filtering, sandboxing, rate limiting and watermarking
These four general-purpose controls came up in earlier domains. Here's what each one is for:
| Control | Main threats it reduces | Covered in |
|---|---|---|
| Filtering inputs and outputs | Prompt injection, harmful content, data leakage | Domain 4 |
| Sandboxing | Malicious code execution, tool misuse | Domain 4 |
| Rate limiting | Model extraction, denial of service, denial of wallet | Domain 3 |
| Watermarking | Undisclosed synthetic content, proving model theft | Domains 2 and 3 |
Prioritizing mitigations by risk, feasibility and business context
You'll rarely fix everything at once, so rank mitigations on three things:
- Risk reduction: how much the mitigation actually lowers the likelihood or impact.
- Feasibility: cost, effort, time, available skills, and whether it's technically possible today.
- Business context: effects on users and latency, legal deadlines, and the mission the system serves.
Start with quick wins, mitigations that cut a lot of risk for little effort, like removing a tool permission an agent doesn't need. Prefer architectural fixes over detection: taking away a dangerous capability beats trying to catch every misuse of it. When the root cause can't be fixed, as with prompt injection, layer several partial mitigations. And weigh side effects; a filter so strict that it blocks one legitimate request in five will push people toward unapproved tools, creating a new risk.
Validating mitigations with regression testing
A mitigation nobody has tested is an assumption. To validate one:
- rerun the original attack and measure its new success rate;
- run variants of it, because attackers will mutate it;
- run the full regression suite to confirm nothing else broke, including accuracy, fairness and the other guardrails;
- measure side effects such as false positives and added latency; and
- add the attack and its variants to the permanent test suite.
Continuous monitoring for residual risk
Some risk always remains after mitigation. That residual risk is formally accepted by its owner (Domain 2) and then watched:
- detection rules aimed at the specific residual risk, such as alerting on unusual tool calls that follow retrieval of outside content when injection can't be fully blocked;
- periodic re-testing, because models, prompts and attack techniques all change;
- tracking the attack success rate over time;
- threat intelligence on new techniques; and
- reassessment whenever a trigger from Domain 2 occurs.
That monitoring feeds the next round of threat modeling in 6.1, which closes the loop.
Practice questions
These are my own questions in the ISC2 style. Pick the BEST answer, and when two look right, put the safety of people first, then legal obligations, then the mission.
- A team is threat modeling a system where several AI agents call tools and hand tasks to one another. Which methodology is BEST suited?
- A. STRIDE
- B. PASTA
- C. MAESTRO
- D. A standard vulnerability scan
- An attacker sends queries to a face recognition API and reconstructs recognizable images of people whose photos were in the training data. Which attack is this?
- A. Model extraction
- B. Model inversion
- C. Evasion
- D. Data poisoning
- A red team is about to test a customer chatbot built on a third-party model API. What must be in place FIRST?
- A. Written authorization and rules of engagement, including confirmation that the provider's terms allow testing
- B. An automated attack tool
- C. A list of known jailbreak prompts
- D. A draft report template
- A red team finds a jailbreak that succeeded 3 times in 50 attempts. How should it be reported?
- A. As not reproducible, since it usually fails
- B. As a critical finding, since any success is a failure
- C. Only if it works on the first attempt during retesting
- D. With its success rate and number of attempts, the exact inputs and the model version tested
- A health information chatbot sometimes invents medication dosages. No attacker is involved. What is the BEST response?
- A. Classify it as a security vulnerability and patch the model
- B. Accept it, since hallucination is a known model limitation
- C. Stop the chatbot from giving dosage information, route those questions to clinicians, and document the limitation
- D. Wait for the vendor's next model version
- Through an indirect prompt injection, an agent can be made to email customer data to outside addresses. Which mitigation is MOST effective?
- A. Restrict the agent's email tool to approved internal recipients
- B. Add a stronger injection-detection filter
- C. Add a warning to the system prompt
- D. Increase logging of the agent's email activity
- A new input filter has been deployed to block a jailbreak the red team found. What BEST validates it?
- A. The vendor's confirmation that the filter works
- B. Rerunning the original attack and variants of it, plus the full regression suite, while measuring false positives
- C. Monitoring production for a week
- D. A code review of the filter
- The security team is presenting red team findings to the board. What should the presentation lead with?
- A. The ATLAS technique IDs for each finding
- B. The full reproduction steps
- C. Every finding in order of discovery
- D. The business impact and the decision the board needs to make
- An AI vendor provides a SOC 2 Type I report. What is its main limitation?
- A. It shows controls were designed properly at one point in time, but not that they operated effectively over time
- B. It covers only physical security
- C. It's a self-assessment by the vendor
- D. It applies only to on-premises systems
- A team downloads a model file, and its SHA-256 hash matches the value on the publisher's website. What does this confirm?
- A. The model is free of backdoors
- B. The file is identical to the one the hash was calculated from
- C. The publisher is who they claim to be
- D. The model's license allows commercial use
- An engineer wants to change how the RAG system splits documents into chunks. How should the change be handled?
- A. As a minor configuration tweak with no review
- B. Only by the data science team, since it doesn't touch the model
- C. Through change management, versioned and tested like a model change
- D. Deployed straight to production and monitored
- Prompt injection can't be fully eliminated in an agent that reads outside web content. What is the BEST approach?
- A. Stop using AI agents entirely
- B. Rely on the model provider's safety training
- C. Add a system prompt instruction to ignore injected text
- D. Layer mitigations, have the owner formally accept the residual risk, and monitor for that specific risk
Show the answer key
- C. MAESTRO was built for agentic and multi-agent systems, including threats that move between layers. STRIDE and PASTA can help, but they assume more predictable software.
- B. Inversion recovers what the model learned about people. Extraction (A) copies the model itself.
- A. Testing without written authorization can be illegal and breach the provider's terms. Tools and prompts come later.
- D. AI results are probabilistic, so a success rate over stated attempts, with exact inputs and versions, is how you make the finding reproducible. Three in fifty is real and exploitable at scale.
- C. No attacker means it's a model limitation, but invented dosages put patients at risk, so the use case gets restricted and people get routed to clinicians. Simply accepting it (B) ignores the safety risk.
- A. Removing the dangerous capability is stronger than trying to detect every injection. A filter (B) helps as a layer but will miss some attacks, and a system prompt warning (C) can be overridden.
- B. Validation means proving the attack and its variants no longer work and that nothing else broke. Vendor assurance (A) and passive monitoring (C) don't test anything.
- D. Boards decide on risk, so give them the impact and the decision. Technical detail belongs in the engineering report.
- A. Type I covers design at a point in time; Type II covers operation over a period.
- B. A matching hash proves integrity only. It says nothing about backdoors, licensing or who produced the file; a digital signature is what proves origin.
- C. Chunking changes what the model retrieves and therefore what it says. Retrieval settings are versioned and go through change management.
- D. When a risk can't be eliminated, it gets layered controls, a documented acceptance and monitoring aimed at it. Abandoning agents (A) may not fit the mission, and B and C rely on the model to protect itself.
Sources
- ISC2 AI security certification outline
- MITRE ATLAS
- OWASP GenAI Security Project, including the Top 10 for LLM Applications and agentic AI guidance
- NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- Cloud Security Alliance, MAESTRO threat modeling for agentic AI
- Snyk Labs, overview of MAESTRO
- Cloud Security Alliance, Agentic AI Red Teaming Guide
- NIST SP 800-218A, Secure Software Development Practices for Generative AI and Dual-Use Foundation Models
- SLSA and Sigstore
- CycloneDX ML-BOM
- AICPA SOC 2 overview
- PyRIT, garak and promptfoo