ISC2 AI Security Pilot Exam · Free study guide

Domain 5 Study Notes: Secure AI Operations and Incident Response

How to use these notes

Domain 5 is about running AI systems day to day and handling things when they go wrong. The first half covers operations: readiness, monitoring, detection and vulnerability management. The second half follows an AI incident from triage through containment, investigation, recovery and the lessons afterward. No operations or incident response background is assumed; every term is defined where it first appears. I've followed the ISC2 outline section by section, and each section ends with how I expect it to show up on the exam.

The incident response lifecycle itself isn't new. NIST's incident response guidance, SP 800-61, was revised in 2025 to line up with the NIST Cybersecurity Framework 2.0, and the classic phases still apply: prepare, detect and analyze, contain, eradicate, recover, and learn. What changes with AI is the evidence and the actions. An AI incident may leave no malware and no exploited server behind, only a model that behaves differently, a poisoned document in a knowledge base or an agent that did exactly what an injected instruction told it to.

One idea runs through the domain: you can only investigate what you recorded. Prompts, tool calls, model versions and dataset snapshots have to be captured before an incident, or there will be nothing to reconstruct afterward.

Everything here is my own work, built from public frameworks and guidance. If a reference book frames something differently, trust the book.

5.1 AI SecOps foundations and operational readiness

SecOps (security operations) is the ongoing work of monitoring, detecting and responding to threats. Operational readiness means the people, processes and tools are in place before a system goes live, so the first incident isn't also the first time anyone thought about incidents.

Core security principles in AI operations

Least privilege and separation of duties (Domain 4) apply to the people and tools that operate AI, too. Operators get only the access their job needs. No single person can both change a production model and approve its release, so production changes need two people. Emergency break-glass accounts exist for when normal access fails, and every use of one is alerted and reviewed. Logs that contain prompts are sensitive, so access to them follows need to know.

AI operational roles and responsibilities

Role Responsible for
Model owner The model's behavior, performance and risk; approves releases and accepts residual risk for it
Data owner The datasets the model uses; approves how they're used (Domain 3)
Use-case or product owner The business outcome the AI system delivers
MLOps or platform engineer Running the pipelines, infrastructure and deployments
Data scientist Building, evaluating and retraining models
SecOps analyst Monitoring, detection and incident response
Legal, privacy and compliance Regulatory obligations, including breach notification

A RACI matrix assigns each activity a Responsible party (who does the work), an Accountable party (who owns the outcome), and those Consulted and Informed. Each activity gets exactly one Accountable party. When everyone is accountable, nobody is.

Release requirements

Release requirements are the conditions a model must meet before it goes live, and they also define what normal operation looks like afterward. The outline names two that are specific to AI:

Other common release requirements include evaluation and bias scores above set thresholds, clean security scans, latency targets, a tested rollback, monitoring in place and runbooks written.

Playbooks, runbooks and tabletop exercises

Useful AI tabletop scenarios include poisoned data discovered in a production model, tampered weights found in the model registry, an agent sending data out because of an injected instruction, a deepfaked executive's voice requesting a wire transfer, and an outage at the model provider. The goal is to test decisions: who can pull a model out of service, when legal must be told, and what you'll say to customers.

Configuration management and drift

Configuration management (CM) sets an approved baseline configuration and controls every change to it. For AI, the configuration includes far more than server settings: the model version, system prompt, guardrail settings, model parameters, tool lists and their permissions, RAG sources and index version, rate limits and provider endpoints. Any of them can change the system's behavior.

Automated model updates are a hidden risk. Some providers point an endpoint name like "latest" at whichever version is newest, and some pipelines retrain and redeploy automatically. In production, pin specific versions and send every update through change control and testing.

Configuration drift is the gap that opens over time between the approved baseline and what's actually running, through quick fixes to a system prompt or a filter someone loosened to resolve a complaint. Keep configuration in version control as code, compare running configuration against the baseline automatically, and alert on differences.

5.2 Monitoring, logging and AI security telemetry

Telemetry is the data a system emits about what it's doing. Traditional security telemetry covers logins, network flows and process activity. AI systems need all of that plus a new layer that shows what the model was asked, what it read and what it did.

AI telemetry is masked before it's stored, then correlated
AI telemetry pipeline: sources, masking, SIEM, SOAR and threat intelligence

AI telemetry data

Telemetry What it shows What to watch for
Prompts and completions What people asked and what the model answered Injection attempts, jailbreaks, sensitive data going in or coming out
Retrieval events and embeddings Which documents were retrieved, and for whom Documents outside the person's permissions, a suspicious document retrieved again and again
Tool calls Which tools an agent called, with which parameters, and what came back Unexpected tools, unusual parameters, sudden spikes in volume
API traffic Requests, sources, tokens used, errors and latency Extraction patterns, abuse, denial of wallet
Model and configuration changes Version, system prompt and guardrail changes Unauthorized changes and drift

Every event also needs identity context: which person, which agent, acting on whose behalf, in which session. Standards are emerging here too; OpenTelemetry, the common open standard for system telemetry, has published conventions for recording generative AI activity consistently.

Correlating events across SIEM, SOAR and threat intelligence

A Security Information and Event Management (SIEM) system collects logs from many sources and correlates them to find patterns no single log shows. Picture three events that each look minor: a prompt containing an injection pattern, retrieval of a document from an unusual source, and an agent tool call sending email to an outside domain. Correlated, they're one incident.

Security Orchestration, Automation and Response (SOAR) runs automated playbooks when an alert fires: enrich it with context, throttle the user, revoke an agent's token, open a ticket and page the on-call analyst. Automation buys speed; high-impact decisions still go to a person.

Threat intelligence is information about adversaries and how they operate. For AI, useful feeds include MITRE ATLAS techniques, known jailbreak prompts, hashes of known malicious models, reports of malicious MCP servers and vendor security advisories. It feeds detection rules and adds context to alerts.

Log integrity and evidence documentation

Logs are only useful as evidence if they can be trusted. Send them off the host to central storage quickly, store them write-once, chain or sign them so tampering shows, synchronize clocks so timestamps line up across systems, restrict who can read them and log that access, and keep them as long as legal requirements say. When logs become evidence, document the chain of custody (Domain 3): who collected what, when, how, and the hash of each item.

Privacy-preserving telemetry masking

Here's the tension: investigators need prompts, and prompts are full of personal data. Telemetry masking resolves it by protecting the sensitive parts before storage:

Reversible tokenization is useful here. Masking that destroys what investigators need is its own failure, and reversible tokens let authorized investigators recover the original when an incident calls for it.

Real-time monitoring of behavior, drift and agent actions

None of these alerts work without a baseline. Measure normal behavior first, then alert on departures from it.

5.3 Detecting runtime threats, misuse and compromise

Runtime means while the system is running in production, as opposed to during development or testing. Runtime detection catches attacks as they happen.

Runtime threat detection at each layer

Layer What an attack looks like How to detect it
Inputs and prompts Injection phrases, role-play jailbreaks, encoded payloads, someone rephrasing again and again after refusals Injection classifiers, pattern rules, tracking refusal-then-retry sequences within a session
Retrieval chains Documents that contain instructions, a newly added document suddenly retrieved constantly, content from unapproved sources Scanning content as it's indexed, monitoring retrieval frequency, checking provenance
Outputs Personal data, leaked system prompts, links to unknown domains, data hidden in URLs Output DLP, canary tokens, an allow-list for link domains
Resource consumption Token spikes, very long prompts, repeated maximum-length outputs, many parallel sessions on one key Quotas plus thresholds based on normal usage

Two techniques are worth knowing by name. A canary token is a unique string planted somewhere it should never leave, like a system prompt or a sensitive document. If it ever shows up in an output or anywhere else, you know something leaked. And watch for exfiltration through rendered links: an injected instruction gets the model to output an image link whose web address carries stolen data, and when the user's browser loads the image, the data goes to the attacker. Restricting which domains outputs may link to blocks it.

AI access and agency misuse

The detection method is the same for both: compare each identity's actions against its declared scope and its own history, and alert when an agent does something new.

Secret exposure and tool execution anomalies

Run secret scanning on prompts, outputs, logs and code. A credential appearing in a model's output is a serious finding, because it means a secret was in the model's context where it never belonged (Domain 4). Honeytokens, fake credentials that trigger an alert the moment anyone uses them, catch attackers who harvest secrets.

Tool execution anomalies include unusual sequences, like reading a sensitive file and then calling an outside web address; parameters outside normal ranges; tools called that weren't part of the agent's plan; spikes in failures and retries; commands containing shell control characters; and calls to destinations never seen before.

AI asset and lifecycle integrity

Attackers don't always attack the running system; sometimes they change what it's built from. Watch for:

AI-driven security operations

AI is also a tool for defenders.

5.4 AI vulnerability and asset posture management

Model vulnerability and exposure registers

A vulnerability register lists known weaknesses in AI assets, each with the affected asset, severity, owner, status and due date. AI weaknesses include jailbreak techniques that worked in red teaming, published vulnerabilities in AI frameworks and libraries, insecure configurations and ways around guardrails. Findings come from red team exercises, vendor advisories, the CVE system, MITRE ATLAS case studies and public AI incident and vulnerability databases.

An exposure register records where each asset can be reached from and what it can reach: whether an endpoint faces the internet, which data and tools a model or agent can touch, and who its users are. Exposure decides how much a vulnerability matters.

One difference from traditional vulnerability management: many model weaknesses have no patch. There's no update that makes a model immune to jailbreaks. So the register tracks mitigations and documented risk acceptance as well as fixes.

Prioritizing by context and consequence

CVSS, the common vulnerability scoring system, rates severity in isolation. Real priority depends on context:

Take the same jailbreak weakness in two places. In an internal tool that summarizes public documents, it's a low priority. In an agent that can issue payments, it's urgent. CISA's SSVC (Stakeholder-Specific Vulnerability Categorization) formalizes this kind of decision as a tree built on exploitation status and impact.

Patch management for AI pipelines and open source components

AI stacks pile up dependencies quickly: Python packages, GPU drivers, machine learning frameworks, model serving software, vector databases, orchestration libraries, MCP servers and notebook environments, most of them open source. Good practice:

For the models themselves, "patching" means moving to a provider's fixed version, retraining, or adding guardrails around a weakness that can't be fixed directly.

Compensating controls for legacy models

A legacy model is an older model that can't easily be updated: the vendor no longer supports it, retraining isn't practical, or it was validated for a regulated use and any change would require costly revalidation. When it has to stay in service, compensating controls reduce the risk in other ways:

Pair those controls with a documented risk acceptance that has an expiry date and a retirement plan. Legacy should be a temporary state.

Continuous security reviews and posture metrics

AI Security Posture Management (AI-SPM) tools discover AI assets such as models, agents, endpoints and datasets, check their configurations against policy, find shadow AI and map attack paths. They make review continuous, on top of the scheduled and triggered reviews from Domain 2.

Posture metrics turn that into numbers leadership can track over time:

Area Example metrics
Coverage Share of AI assets inventoried with an owner; share with a model card; share behind guardrails
Vulnerabilities Mean time to fix by severity; open critical findings; findings past their deadline
Detection and response Mean time to detect and to contain AI incidents
Testing Share of releases red-teamed; jailbreak success rate in the latest red team exercise
Hygiene Orphaned non-human identities; unpinned model versions in production; shadow AI tools found

5.5 Responding to and containing AI incidents

The rest of this domain follows an incident from start to finish. Here's the whole lifecycle, with the AI-specific work at each phase:

AI incidents follow the familiar lifecycle, with new actions
AI incident lifecycle: six phases, covered in 5.5 through 5.7

AI-specific triage taxonomy

Triage sorts incoming incidents by type and severity so the right people respond in the right order. A shared taxonomy, a fixed set of incident categories, means everyone describes incidents the same way. The outline names four types:

Type What happened Early signs Typical first containment
Prompt injection Instructions from an attacker, typed directly or hidden in content, changed the system's behavior Odd tool calls, outputs with unexpected links, policy violations Block the source, disable the affected tool, quarantine the document
Jailbreaking A user got around safety restrictions to produce prohibited output Filter hits, refuse-and-retry patterns, harmful output reported Block the user or session, tighten filters, rate limit
Data poisoning Training or knowledge data was altered to change behavior Accuracy drops on certain inputs, trigger-like behavior, unexplained dataset changes Take the model out of service or roll it back, freeze the data
Model theft Weights were stolen, or the model was copied through its API Large outbound transfers, extraction query patterns Revoke access, rotate credentials, rate limit, isolate storage

Most organizations add a few more: sensitive data disclosure, agent misuse, supply chain compromise and denial of service or denial of wallet. Severity depends on harm to people, the data involved and how far the problem reaches.

Multi-disciplinary escalation paths

AI incidents cross team lines faster than most. Decide in advance who gets pulled in and when:

Notification clocks start early, so legal belongs in the escalation path from the start.

AI-specific containment actions

Containment stops the harm from spreading while you work out what happened. AI-specific options include:

Containment trades availability for safety. When people could be harmed, take the system down.

Pipeline suspension and agent termination

Pipeline suspension halts training, fine-tuning and deployment pipelines when poisoning or tampering is suspected, so contaminated artifacts don't spread to other models. Freeze automatic retraining and block promotions until the investigation clears them.

Agent termination stops running agents with the kill switch, revokes their credentials, cancels their scheduled tasks and stops any sub-agents they started. Then check for persistence: an agent under an attacker's influence may have created scheduled jobs, new accounts or API keys. Agents with long-term memory deserve special attention, because an injected instruction stored in memory can come back in every future session. Quarantine that memory.

Preservation of investigative artifacts

Containment can destroy evidence. Rolling back a model overwrites the version you need to examine, and retraining replaces the poisoned data's fingerprints. So preserve first, with hashes and chain of custody:

Capture the most volatile items, like running sessions and memory, first, and place everything under legal hold when litigation or regulatory action is possible.

5.6 Investigation, eradication and forensics

Attack kill chain reconstruction

A kill chain is the sequence of steps an attacker takes from first contact to final impact. Reconstructing it shows how the attack worked, which controls failed and where to add new ones. MITRE ATLAS organizes AI attack techniques by tactic in the same style as ATT&CK, which gives you a common vocabulary for each step. The outline names two kinds of chains to practice on.

A data poisoning chain might run like this: the attacker gains write access to a data source, poisoned samples get collected, they pass validation, the model trains on them, the model is deployed, and later the attacker uses the trigger. For each step, find the evidence and the control that should have stopped it.

An untrusted tool output chain might run like this: an agent calls a web search tool, a result page contains hidden instructions, the model follows them, it calls the email tool with internal data, and the data leaves. The evidence lives in tool call logs, the retrieved content and the model's outputs, which is why 5.2 insisted on logging all three.

Root cause analysis, forensics and timelines

Root cause analysis (RCA) looks past the immediate cause to the underlying ones. In the email example, the immediate cause is that the agent sent data out. The root causes are that the agent held a send permission it didn't need, outputs weren't limited to approved domains, and web content wasn't treated as untrusted. Real incidents usually have several contributing causes. Methods like the five whys and fishbone diagrams help find them.

AI forensics applies investigation techniques to AI artifacts:

A timeline puts every event in order using synchronized timestamps: the first malicious action, detection, containment and every response step. Poisoning attacks can have a long dwell time, the gap between compromise and detection, because the poison enters the data long before anyone notices the model behaving oddly.

Threat actor attribution

Attribution identifies who was behind an attack, using their techniques, infrastructure, artifacts and matches against threat intelligence. It's hard, often lands at low confidence, and many AI incidents come from opportunistic individuals rather than organized groups. Attribution helps you understand intent, anticipate what comes next, support law enforcement and share intelligence. It isn't needed to contain or eradicate, so never delay the response while you work out who did it. State a confidence level with any attribution, and leave formal attribution to law enforcement and intelligence agencies.

Threat eradication

Eradication removes the cause of the incident and everything it touched:

Where you can, rebuild from trusted sources instead of trying to clean a compromised artifact in place.

Safety regression testing before reintroduction

Regression testing re-runs existing tests to confirm a change didn't break something that used to work. Safety regression testing does that for safety and security before a repaired system returns to production: the full jailbreak and injection suite, including the exact attack from the incident, plus bias and accuracy evaluations, backdoor trigger tests and guardrail tests. Compare the results with the pre-incident baseline, get the model owner's sign-off, and bring the system back in stages with closer monitoring than usual. Then add the incident's attack to the permanent test suite, so the same thing gets checked on every future release.

5.7 Recovery, business continuity and post-incident review

Recovery strategies

Restore system prompts, guardrail settings and other configuration from version control too, and rotate every credential the incident may have exposed.

AI circuit breakers and kill switches

A circuit breaker is an automatic control that trips when a threshold is crossed, like an error rate, a cost spike, a burst of policy violations or unusual tool calls, and stops or limits the AI function until a person resets it. If an agent issues more refunds in an hour than it ever has before, the breaker pauses it.

A kill switch is a manual control that lets an authorized person shut an AI system or agent down immediately. It has to work even when the AI system is misbehaving, so it's built outside the AI's control, by revoking credentials or disabling access at the gateway. Test it regularly.

The difference to remember: a circuit breaker is automatic and triggered by thresholds, and a kill switch is a deliberate human decision.

Business continuity, disaster recovery and dependency mapping

Business continuity (BC) keeps critical business functions running during a disruption. Disaster recovery (DR) restores the IT systems behind them. A business impact analysis (BIA) identifies which functions are critical and sets two targets for each: the recovery time objective (RTO), the longest acceptable outage, and the recovery point objective (RPO), the most data loss you can accept, measured in time.

Critical AI dependency mapping traces what each AI system depends on and which business processes depend on it. AI dependencies include the model provider, GPU capacity, the vector database, the embedding model, data pipelines, guardrail services, the identity provider, MCP servers and other tools, and the system prompt and configuration. The map shows single points of failure, like a business process that stops whenever one model provider has an outage.

Two AI-specific traps sit in continuity plans:

Other continuity options include cached responses for common requests, manual procedures that trained staff can actually run, and backups of models, indexes and configuration.

Post-incident reviews and the TTP register

A post-incident review, or lessons-learned session, happens soon after the incident and is blameless, focused on how the system failed rather than who to punish, so people speak openly. It covers what happened, what worked, what didn't and the root causes, and it ends with action items that have owners and due dates. Typical outputs are updated playbooks and runbooks, new detection rules, new test cases, control changes and training.

TTP stands for tactics, techniques and procedures, the way attackers operate. A TTP register catalogs the TTPs seen in your own incidents and in threat intelligence, mapped to MITRE ATLAS or ATT&CK, with the detections and controls for each and any gaps. Update it after every incident, and share what you can through your sector's information sharing and analysis center (ISAC).

Secure decommissioning, disposal and remediation

When a model is retired, whether at end of life or after a compromise:

Keep the records audits require, like the model card and decision logs, even after the weights are gone.

Practice questions

These are my own questions in the ISC2 style. Pick the BEST answer, and when two look right, put the safety of people first, then legal obligations, then the mission.

  1. The team suspects that poisoned data affected a production model that drafts internal marketing copy. They want to roll it back right away. What should happen FIRST?
    • A. Preserve the current model, its configuration and dataset snapshots, with hashes
    • B. Roll back to the previous version
    • C. Start retraining on a new dataset
    • D. Brief the communications team
  2. A hospital's AI dosing assistant starts recommending doses outside normal ranges. What should happen FIRST?
    • A. Preserve the model and logs for investigation
    • B. Stop using the assistant in patient care and switch to the manual process
    • C. Begin root cause analysis
    • D. Determine who attacked the system
  3. An agent with access to customer records sends data to an outside domain right after reading a web page during a research task. Which incident category BEST fits?
    • A. Jailbreaking
    • B. Model theft
    • C. Prompt injection
    • D. Data poisoning
  4. Which mechanism automatically pauses an agent when the number of refunds it issues in an hour crosses a set threshold?
    • A. Kill switch
    • B. Canary token
    • C. Honeytoken
    • D. Circuit breaker
  5. After poisoning is discovered, the team rolls the model back to the previous version, but the malicious behavior continues. What is the MOST likely explanation?
    • A. The previous version was also trained on the poisoned data, because the poison entered long before it was detected
    • B. The rollback wasn't signed
    • C. The model provider pushed an update
    • D. The kill switch failed
  6. In a human-in-the-loop loan review, the reviewers' override rate falls from about 8% to nearly zero over two months, with no change to the model. What should the security professional suspect FIRST?
    • A. The model has improved
    • B. Reviewers have started approving outputs without real review
    • C. An attacker is manipulating the model
    • D. The logging system has failed
  7. During a prompt injection incident, an executive asks the team to identify the attacker before taking any containment action. What is the BEST response?
    • A. Delay containment until the attacker is identified
    • B. Close the incident without attempting attribution
    • C. Contain and eradicate now, and pursue attribution separately with a stated confidence level
    • D. Publicly name the group the team suspects
  8. A model was retrained after a poisoning incident. What must happen before it returns to production?
    • A. Sign-off from the model owner alone
    • B. A successful load test
    • C. A code review of the training pipeline
    • D. A safety regression test suite that includes the attack from the incident
  9. Which telemetry design BEST balances investigation needs and privacy for an AI application?
    • A. Mask personal data before logs reach the SIEM, and keep unmasked prompts in a restricted store with short retention and logged access
    • B. Send all prompts unmasked to the SIEM so investigations are complete
    • C. Disable prompt logging entirely
    • D. Log only request counts and discard all content
  10. The same jailbreak weakness is found in several AI systems. Which should be addressed FIRST?
    • A. An internal tool that summarizes public documents
    • B. An internet-facing agent that can issue payments
    • C. An offline batch model that processes test data
    • D. A model in a development sandbox
  11. An organization's continuity plan names a backup model provider, but no one has tested it with production prompts. What is the BEST action?
    • A. Assume the backup model will behave the same way
    • B. Remove the backup provider from the plan, since it's untested
    • C. Test the backup model with real prompts and guardrails, and update the plan with what you learn
    • D. Increase the quota with the primary provider
  12. A compromised model has been taken out of service. Which remediation step is MOST often overlooked?
    • A. Deleting the model weights
    • B. Revoking the model's API keys
    • C. Marking the model retired in the registry
    • D. Reviewing the decisions and content the compromised model produced
Show the answer key
  1. A. Nobody is in danger from marketing drafts, so preserve the evidence before changing anything. A rollback (B) overwrites the version you need to examine.
  2. B. Patients could be harmed, so stopping the harm comes before everything else, including evidence preservation. Investigation follows once patients are safe.
  3. C. Instructions hidden in the web page redirected the agent. That's indirect prompt injection, and the data leaving is its impact.
  4. D. A circuit breaker trips automatically on a threshold. A kill switch (A) is a manual decision.
  5. A. Poisoning can sit undetected for a long time, so a rollback target must be verified as older than the compromise.
  6. B. With no model change, a collapsing override rate points to automation bias: reviewers have stopped checking. That's a failure of oversight, and oversight is a control.
  7. C. Containment never waits for attribution. Attribution is useful, often uncertain, and a separate track.
  8. D. Safety regression testing, including the incident's attack, has to pass before reintroduction. Owner sign-off still matters, but it comes after the tests.
  9. A. Masking protects privacy in the broadly accessible SIEM, and the restricted store keeps what investigators need. B over-collects, while C and D leave nothing to investigate.
  10. B. Context sets priority. Internet exposure combined with the power to move money makes it the most urgent.
  11. C. Prompts and guardrails tuned for one model can behave differently on another. An untested failover gives false confidence, and the fix is to test it instead of dropping it.
  12. D. Recovery isn't finished while harmful decisions made by the compromised model are still standing.

Sources