ISC2 AI Security Pilot Exam · Free study guide
Domain 4 Study Notes: Secure AI Architecture, Engineering, and IAM
How to use these notes
Domain 4 is the engineering domain. It covers how to design AI systems so they're secure from the start, how to run the infrastructure under them, and how to control who and what can act through them. That last part is the biggest change from traditional security: AI agents are a new kind of identity that can read, decide and act, and most of the hard questions in this domain are about keeping them inside their lane. No engineering or identity background is assumed; every term is defined where it first appears. I've followed the ISC2 outline section by section, and each section ends with how I expect it to show up on the exam.
One idea runs through the whole domain: treat the model as an untrusted component. A model can be manipulated by anything it reads, so it shouldn't be trusted to enforce security decisions. Permissions, validation and approvals belong in the deterministic code and infrastructure around it, where an attacker can't talk their way past them.
Everything here is my own work, built from public guidance, especially the NCSC and CISA Guidelines for Secure AI System Development and the OWASP GenAI Security Project. If a reference book frames something differently, trust the book.
4.1 AI architecture and secure design principles
AI-specific security strategies
The outline names four strategies. None of them is new; what's new is how they apply when one component, the model, can be talked into misbehaving.
- Zero trust means nothing is trusted because of where it sits on the network; every request is authenticated, authorized and encrypted. For AI, that means authenticating every call to the model, the vector database and each tool, and treating model output and retrieved content as untrusted input. An agent running inside your network gets no trust for being there.
- Defense in depth stacks independent layers so one failure doesn't lead to a breach. AI layers include input filtering, a hardened system prompt, the model's own safety training, output filtering, least-privilege tool permissions, human approval for high-impact actions and monitoring. You need the stack because no single layer reliably stops prompt injection.
- Privacy by design (Domain 3) keeps personal data out of prompts, context windows and logs unless it's needed.
- Secure defaults means the system is safe as shipped, before anyone tunes it: tools off until enabled, logging on, data retention minimal, write actions requiring approval and content filters at their strictest.
The NCSC and CISA guidelines organize secure AI development into four stages, secure design, secure development, secure deployment and secure operation and maintenance, and they put the burden on providers to make the secure option the default.
Threat modeling across components
Threat modeling is a structured way to find what can go wrong before you build. Adam Shostack's four questions are a good frame: what are we building, what can go wrong, what are we going to do about it, and did we do a good job? STRIDE is a common checklist for the second question: spoofing, tampering, repudiation, information disclosure, denial of service and elevation of privilege. For AI, walk each component in the architecture:

| Component | Main threats | Example controls |
|---|---|---|
| Inference endpoint | Prompt injection, jailbreaks, model extraction, denial of wallet, sensitive data disclosure | Authentication, rate limits, input and output filtering |
| RAG pipeline | Injected instructions inside retrieved documents, poisoned documents, documents shown to people who lack access | Permission-aware retrieval, content sanitization, approved sources |
| Vector database | Weak access control, data exposure, embedding inversion (rebuilding source text from its embeddings) | Authentication, encryption, tenant separation, the same classification as the source data |
| Tools and plugins | Excessive agency, injected instructions triggering real actions, malicious tool servers | Least privilege, allow-lists, human approval |
| Training and fine-tuning | Poisoning and supply chain attacks | See Domain 3 |
The threat that shows up most in this domain is indirect prompt injection: malicious instructions hidden in content the model reads, like a web page, an email or a shared document, instead of typed by the user. RAG systems and agents read untrusted content constantly, so every document and tool result is a possible attack path. OWASP's Top 10 for LLM Applications and MITRE ATLAS both catalog these threats and are worth browsing.
Architectural trade-offs
The outline names three forces that pull against each other: latency, explainability and cost. Most design choices move one at the expense of another.
| Choice | You gain | You pay |
|---|---|---|
| A larger model | Capability | Cost, latency, harder to explain |
| A smaller or distilled model | Speed, lower cost, option to self-host | Some capability |
| More guardrail layers | Safety | Latency, false positives, cost |
| An interpretable model | Explainability | Possibly accuracy |
| Human approval steps | Control | Speed |
| Response caching | Speed and cost | Stale answers, or one user's answer served to another if the cache ignores permissions |
The risk tier should settle these. A fraud check at checkout needs an answer in milliseconds; a loan decision can take a few seconds and must be explainable.
Build versus buy
AI can be consumed along a spectrum: an AI feature inside a SaaS product, a hosted model API, a managed AI platform in your cloud account, or an open model you run yourself.
| Factor | Hosted (API or SaaS) | Self-managed |
|---|---|---|
| Time to deploy | Days | Months |
| Control over data and location | Limited, set by contract | Full |
| Security responsibility | Shared; the vendor runs the infrastructure | Entirely yours |
| Cost | Pay per use, hard to predict at scale | Hardware and staff up front |
| Model choice and customization | The vendor's models and options | Any open model, full fine-tuning |
| Expertise needed | Modest | GPU operations, MLOps and AI security skills |
| Dependency on a vendor | Higher | Lower |
The deciding factors are usually data sensitivity and residency, regulatory obligations, in-house skills, scale and total cost of ownership. The shared responsibility table in Domain 2 shows how the security duties shift.
Secure integration for tools, APIs and plugins
Agents become useful when they can call tools, and that's where most of the new risk lives.
The Model Context Protocol (MCP) is an open standard, introduced by Anthropic in 2024 and now widely adopted, for connecting AI applications to tools and data. An MCP client inside the AI application connects to MCP servers, which offer tools the model can call. The risks to know:
- Tool poisoning, where a tool's description contains hidden instructions that the model reads and follows;
- a server that changes its tool definitions after you approved it;
- malicious or compromised servers, and local servers that allow command injection on the host;
- servers holding broad tokens, which creates a confused deputy: a component with legitimate authority tricked into using it on someone else's behalf.
Controls include an allow-list of vetted servers, pinned versions, review of tool descriptions, sandboxed local servers, scoped OAuth tokens under MCP's authorization specification, human approval for sensitive calls and a log of every call.
Agent2Agent (A2A) is an open protocol, launched by Google in 2025 and now governed by the Linux Foundation, that lets agents discover each other through Agent Cards, which describe what an agent can do and how to reach it, and then hand tasks to each other. Its risks are spoofed or malicious agents, instructions injected from one agent into another, over-delegation, and data leaking to agents outside your control. Authenticate agents to each other, authorize each task, share only what the task needs, and treat another agent's output as untrusted input.
For any API or plugin, the basics hold: authenticate, give each integration a narrowly scoped token, validate the parameters a model passes to a tool, rate-limit, and never hand the model a user's full credentials.
4.2 Secure AI infrastructure and platforms
Hardening cloud, on-premises and hybrid deployments
Hardening means reducing a system's attack surface: removing what isn't needed and locking down what remains. The basics carry over from any infrastructure. Apply configuration baselines like the CIS Benchmarks, patch, use minimal images, disable unused services, segment networks, encrypt, log, and define infrastructure as code so changes can be reviewed and checked against policy.
AI adds its own soft spots. Data scientists often run notebook servers and distributed computing clusters that were built for convenience inside research labs, with no authentication by default. In 2024, attackers ran a campaign against exposed clusters of Ray, a popular open source framework for scaling AI workloads, to steal credentials and hijack GPUs for cryptocurrency mining. Notebooks, training clusters and model servers should never face the internet without authentication.
- Cloud: least-privilege IAM roles, private network endpoints for model APIs and storage, no public buckets or notebooks, cloud posture management to catch misconfigurations, and keys in a managed KMS.
- On premises: patch GPU drivers and firmware, which get regular security bulletins, and isolate server management interfaces on their own network.
- Hybrid: one identity system and one policy set across environments, encrypted links between them, and a map of where data flows from one to the other.
Isolation and sandboxing
Training and inference workloads should be isolated from each other and from everything else. Separate cloud accounts or projects for development, testing and production, and run containers without root privileges, with read-only file systems and restricted system calls. Limit outbound network access from training environments, so stolen data or model weights can't simply be uploaded somewhere.
Some AI workloads run code nobody reviewed. A code interpreter tool executes whatever code the model writes, and an agent may run scripts based on what it read on a web page. That code belongs in a sandbox: an isolated, short-lived environment with no access to secrets, tight resource limits and no network access, or only to an allow-list. Stronger isolation technologies, like lightweight virtual machines, put a harder wall between untrusted code and the host than standard containers do.
Availability controls against resource exhaustion
AI workloads can exhaust resources fast, by accident or on purpose. OWASP lists this as unbounded consumption in its Top 10 for LLM applications. Controls include:
- rate limits and quotas per user, key and application (see Domain 3);
- compute throttling, which caps GPU time, concurrent jobs or tokens per user or team;
- limits on prompt size, context length and output length, plus request timeouts;
- queues with priorities, so batch jobs can't starve customer-facing inference;
- autoscaling with hard ceilings and budget alerts; and
- a cap on how many steps an agent can take, since an agent stuck in a loop can call tools indefinitely.
Isolated accelerator capacity
Accelerators are the specialized chips, mostly GPUs, that train and run models. They're expensive and often shared, which raises two problems.
The first is confidentiality. GPU memory isn't always cleared between jobs, and in 2024 researchers disclosed a flaw in GPUs from several vendors that let one process read leftover data from another's memory, including fragments of a model's responses. For sensitive workloads, use dedicated GPUs, hardware partitioning that splits one GPU into isolated instances with their own memory, or confidential computing on GPUs that support it.
The second is availability. If training jobs and production inference compete for the same GPUs, a big training run can starve the service customers depend on. Reserved capacity set aside for critical inference solves that.
Physical and environmental security
AI hardware is unusually dense. A rack of modern AI servers can draw many times the power of a traditional server rack and produce so much heat that many data centers now use liquid cooling. The physical controls are familiar, scaled up:
- Power: uninterruptible power supplies, generators and dual power feeds.
- Cooling: redundant cooling, temperature and humidity monitoring, and leak detection where liquid cooling is used.
- Rack security: locked cabinets, cameras and logged access, because GPU servers are valuable theft targets and may hold model weights on local disks.
- Facility access: layered controls such as badges plus biometrics and mantraps.
- Fire suppression: clean-agent gas systems or pre-action sprinklers that protect electronics.
Life safety comes first here, as it always does on ISC2 exams. Some gas suppression systems lower oxygen levels, so they need alarms, evacuation procedures and delays that let people get out before discharge.
4.3 Secure data and model deployment pipelines
Domain 3 covered the model pipeline from a governance angle. This section covers the engineering: how the pipeline itself is built and protected.
Secure CI/CD and model deployment
CI/CD (continuous integration and continuous delivery) is the automated path that builds, tests and ships software. MLOps applies it to machine learning, and LLMOps extends it to LLM applications, where the things being shipped include prompts, RAG indexes, guardrail settings and evaluation sets as well as models and code. All of them change the system's behavior, so all of them go through the pipeline.
Controls for the pipeline:
- source control with protected branches and required code review;
- pipeline definitions stored as code and reviewed like code;
- builds on isolated, short-lived runners that are discarded after each job;
- signed commits and signed build outputs, with build provenance following a framework like SLSA (Supply-chain Levels for Software Artifacts);
- separate, narrowly scoped credentials for each pipeline stage; and
- strict control over who can edit the pipeline, since editing the pipeline is a way to ship anything.
Packaging bundles the model, its inference code and its dependencies, pinned to exact versions and verified by hash, with an AIBOM generated at build time. Containerization puts that package in a container image. Use a minimal base image, reference it by its exact digest instead of a movable tag, scan it for vulnerabilities, never bake secrets into it, and sign it. In Kubernetes, an admission controller can refuse to run any image that isn't signed by your pipeline.
Integrity and confidentiality in ingestion and transformation
Data enters the pipeline through ingestion and gets cleaned and reshaped through transformation. For integrity, authenticate data sources, validate data against its expected schema, scan for malware and personal data, and hash data before and after each transformation. Land new data in a quarantine zone first, and promote it to an approved zone only after it passes checks.
For confidentiality, encrypt data at every step, tokenize sensitive fields during transformation so later stages never see the raw values, run transformation jobs with least privilege, and keep development and production data separate.
Lineage tracking for fine-tuning
Fine-tuning can change a model's behavior a lot with a small amount of data, so it needs tight records. For each fine-tuning run, record the base model's version and hash, the dataset version, the training code version, the settings used, who ran it, the evaluation results and the resulting model's hash.
Two details matter. Models get fine-tuned in chains, base model to department model to team model, and lineage has to follow the whole chain. And many teams fine-tune with adapters, small add-on weight files (LoRA is the common method) that sit on top of a base model; they're models too and need the same tracking. When a fine-tuning dataset later turns out to contain poisoned, personal or unlicensed data, lineage is how you find every model it touched.
Validation checkpoints and pipeline gating
A gate is a check that must pass before the pipeline continues. A typical sequence for a model release:
- data quality and personal data checks;
- evaluation against set thresholds for accuracy, bias and safety, including a red-team test suite and a comparison against the current production model;
- security scans of the model file, container and dependencies;
- signing;
- human approval for higher risk tiers; and
- a staged rollout that rolls back automatically if live metrics cross a threshold.
Common patterns include automated quality gates, policy as code (rules written in a policy language and enforced by the pipeline), manual approval gates and canary analysis gates that compare a new version's live metrics against the old one's. Two rules apply to all of them. Gates fail closed: if a check can't run, the release stops. And the person who can override a gate shouldn't be the person whose release it's blocking.
Data unlearning capabilities
Machine unlearning removes the influence of specific training data from a model that already learned from it. You may need it to honor an erasure request, remove poisoned or unlicensed data, or comply with a court order.
| Approach | How it works | Trade-off |
|---|---|---|
| Exact unlearning | Retrain from scratch without the data | The only certain method, and the most expensive |
| Sharded training | Train separate sub-models on separate slices of data, so only the affected slice needs retraining | Cheaper removal, more complex training |
| Approximate unlearning | Adjust the model's weights to reduce the data's influence | Fast, but hard to prove it worked |
| Remove from RAG | Delete the document from the index and the source | Simple and complete, for data kept outside the weights |
The lesson is architectural: design for unlearning before training. Keep data that may need deleting, like personal data and fast-changing content, in RAG instead of training it into weights, and keep the lineage that shows which data went into which model. Verify any unlearning by testing whether the model still reproduces the removed data.
4.4 AI guardrails and controls
A guardrail is a control that keeps an AI system's inputs, outputs and actions within set limits. Guardrails work in layers, and the most important ones run outside the model, where nothing the model reads can switch them off.

AI input and output handling
- Prompt sanitization cleans inputs before the model sees them: length limits, removal of hidden characters (invisible Unicode characters have been used to smuggle instructions past human reviewers), removal of unneeded markup, and classifiers that score how likely an input is to be an injection attempt. It reduces risk without eliminating it.
- Content filtering runs classifiers on inputs and outputs to catch content the policy forbids, such as violence, self-harm instructions or illegal activity, and keeps a narrow application on topic, so a customer support bot stays a customer support bot.
- Context boundary validation keeps trusted instructions and untrusted data apart inside the model's context. The system prompt, the user's message and retrieved content are labeled and separated, and retrieved or tool content is clearly marked as data to read instead of instructions to follow. It also checks that the context holds only content this user is allowed to see, and that nothing from another user's session leaks in. Models can't reliably tell instructions from data on their own, so the validation has to happen in the code that builds the context.
- Output handling (Domain 3) validates outputs against an expected format, encodes them for wherever they're going, and filters out personal data and secrets.
Architectural guardrails for autonomous behavior
An agent works in a loop: the model decides on an action, a tool runs it, the result goes back to the model, and the loop repeats. The harness is the code that runs that loop and enforces limits around it. Orchestration coordinates multiple steps, models or agents, like a planner agent handing work to worker agents. Guardrails at this level include:
- hard limits on steps, time, tokens and spend for each task;
- a per-task allow-list of tools, with parameters validated before any tool runs;
- human approval before irreversible or high-impact actions, like payments, deletions, external emails and production changes;
- plan review, where the agent proposes a plan and a person or policy engine approves it before execution;
- separate planning from execution, so the component holding permissions checks each step against policy;
- reversibility, using dry runs, staging environments and undo where possible;
- state isolated between sessions; and
- a kill switch, with every action logged.
Functional, performance and load testing
Functional testing checks that the system does what it should and refuses what it should. AI outputs vary from run to run, so tests use evaluation sets with scoring methods such as exact matches, rubrics, or another model doing the grading with human spot checks, and they run more than once to check consistency. Guardrails need their own functional tests: known injection attempts get blocked, filters catch banned content and permissions hold.
Performance testing measures latency, including how long until the first words appear, along with throughput and cost per request at expected volumes. Test with guardrails turned on, since each layer adds time. Load testing runs the system at expected peak traffic and beyond, to confirm that rate limits, quotas, autoscaling and queues behave as designed.
Stress testing and exception handling
Stress testing pushes past normal limits to find the breaking point and see how the system breaks: enormous prompts, full context windows, many agents at once, slow or failing tools, malformed input and an outage at the model provider.
Exception handling validation confirms that failures are safe failures:
- if a guardrail service goes down, requests are blocked instead of passing through unfiltered;
- if a tool times out, the agent stops instead of retrying forever;
- error messages don't reveal system prompts, stack traces or other users' data; and
- a task that fails halfway doesn't leave half-finished actions behind.
A guardrail that fails open is the worst outcome, because the system keeps working and looks fine while nothing is protecting it.
4.5 AI identification and authentication
Identification is claiming an identity, and authentication is proving it. AI changes this area more than any other in the domain, because agents are a new kind of identity that acts on its own.
Human and non-human identities
Human identities belong to people: employees, contractors and customers. Non-human identities (NHIs) belong to everything else: workloads like containers and functions, services and APIs, service accounts, API keys, tools such as MCP servers, bots and AI agents. In most organizations non-human identities far outnumber human ones, and they're often managed far less carefully.
AI agents sit in between. They're non-human, but they act with something like human autonomy, often on behalf of a specific person. That means an agent needs its own identity, a link to the person it's acting for, and a named human owner who's accountable for it.
| Identity type | Example | How it usually authenticates | Accountable owner |
|---|---|---|---|
| Human user | An employee | SSO with MFA | The person |
| Workload | A container running inference | A short-lived identity issued by the platform | The platform team |
| Service | A backend API | Client credentials or mutual TLS | The application owner |
| Tool | An MCP server | Scoped OAuth tokens or mutual TLS | The tool's owner |
| AI agent | A procurement agent | Its own workload identity plus a delegated token from the user it serves | A named business owner |
Identity architectures for non-repudiation
Non-repudiation means someone can't credibly deny an action they took. It rests on unique identities, strong authentication, digital signatures and tamper-evident logs. For AI, every action an agent takes should be traceable to three things: the agent that acted, the person or process it acted for, and the authorization that allowed it.
Agents often work in a delegation chain: a user asks an agent, which hands part of the job to another agent, which calls a tool. Each hop needs to be recorded. OAuth's token exchange standard can carry an "acting on behalf of" claim, so the final token shows both the agent and the user behind it. Shared accounts and shared API keys break all of this; if every agent uses the same service account, the logs can't tell you which agent, or which user, did what.
Machine-to-machine authentication and API tokens
- The OAuth 2.0 client credentials flow lets a service authenticate as itself and receive an access token.
- Mutual TLS (mTLS) has both sides of a connection present certificates, so each proves its identity to the other.
- Workload identity has the platform issue short-lived, automatically rotated identities to workloads, replacing stored keys. SPIFFE is the open standard, and the major clouds offer their own versions.
- API keys are long-lived static strings. They're the weakest option: often over-scoped, rarely rotated and easy to leak through code, logs or prompts.
Good token hygiene means short lifetimes, the fewest possible scopes, an audience restriction so a token works only at the service it was issued for, and binding the token to the client that holds it so a stolen copy can't be replayed. A service should never pass a token it received straight through to another service; MCP's security guidance explicitly forbids this.
Federation, MFA and SSO
- Federated identity lets one identity system trust another, using standards like SAML and OpenID Connect. Employees sign in to AI tools with their corporate identity, and workloads can federate across clouds without stored keys.
- Single sign-on (SSO) lets one sign-in grant access to many applications. Requiring SSO for every approved AI tool centralizes control and offboarding, and the sign-in logs show you who's actually using what.
- Multi-factor authentication (MFA) requires two or more kinds of evidence: something you know, have or are. Administrators of AI platforms should use phishing-resistant MFA, such as passkeys or hardware security keys.
Agents can't complete MFA themselves. The pattern is for the person to authenticate with MFA and delegate a narrowly scoped token to the agent. For sensitive actions, the agent triggers step-up authentication, asking the person to confirm with MFA before it proceeds.
Secrets and key management for AI platforms
AI platforms collect secrets quickly: model provider keys, vector database credentials, tool tokens and signing keys. Keep them in a vault, prefer short-lived secrets generated on demand, rotate them, give each agent its own credentials, and revoke them when the agent is retired. Run secret scanning across code repositories, notebooks, logs and stored prompts, since all of them leak secrets.
The most important design rule: the model should never see a secret. If a credential sits in the prompt or context window, someone can talk the model into revealing it. Instead, the harness attaches the credential when it executes the tool call, and the model only ever sees the tool's result.
4.6 AI authorization and access controls
Authorization decides what an authenticated identity is allowed to do. For agents, it's the control that decides how much damage a manipulated agent can cause.
Least privilege and separation of duties for models and agents
The principle of least privilege (PoLP) gives each identity the minimum permissions for the minimum time. For agents, that means scoping permissions to each task instead of the agent's whole lifetime, starting read-only, and splitting different duties across different agents.
An agent acting for a user should never be able to do more than that user could. Its effective permissions should be the overlap of its own permissions and the user's. If an agent holds broad access of its own, any user who can talk to it can borrow that access, which is the confused deputy problem from 4.1.
Separation of duties (SoD) keeps one identity from completing a sensitive process alone. No single agent should be able to both create a new vendor and pay that vendor, and the agent that writes code shouldn't be the one that deploys it.
Authorization methods and just-in-time access
| Method | How it decides | Strength | Weakness |
|---|---|---|---|
| RBAC (role-based) | By the identity's role | Simple to understand and audit | Coarse; roles multiply over time |
| ABAC (attribute-based) | By attributes of the identity, the resource, the action and the context | Fine-grained, like letting an agent read only internal documents from the user's department | More complex to design and test |
| PBAC (policy-based) | By central policies written as code and evaluated at run time by a policy engine | Consistent, versioned and testable | Needs a policy engine and policy skills |
Watch the acronym: in Domain 3, PBAC also showed up meaning purpose-based access control. Read the question's context.
Just-in-time (JIT) access grants extra permission only when it's needed, for a short time, often after an approval, and then removes it automatically. For agents, JIT looks like this: the agent asks for permission to perform one specific action, a person approves, and the token expires minutes later. The goal is no standing privileges, permissions that sit unused but available to anyone who compromises the identity.
Execution boundaries and API scope limitations
An execution boundary limits where and how an agent's actions run: inside a sandbox, with outbound network access to an allow-list only, within certain file paths, in a staging environment instead of production, or only during set hours.
API scope limitation restricts which operations a token allows. A token that can read a calendar is safer than one with full mailbox access; a token for one code repository is safer than one for the whole organization. Constraints on parameters help too, like capping refunds an agent can issue at a set amount. Enforce these at the API gateway or policy engine, where the model can't argue with them.
Preventing excessive agency, over-delegation and lateral movement
Excessive agency is the OWASP name for an AI system that can take damaging actions because it has too much functionality, too many permissions or too much autonomy. A classic example is an email summarizing agent that also has permission to send email. One injected instruction in an incoming message, and it forwards the inbox to an attacker.

Over-delegation gives an agent more authority than its task needs, or lets agents pass their full authority down to sub-agents. Each hop in a delegation chain should narrow the authority it passes on, and chains should have a depth limit.
Lateral movement is an attacker using one compromised system to reach others. A compromised agent holding tokens for many systems is an ideal stepping stone, and in multi-agent systems an injected instruction can travel from one agent to the next.
The controls for all three come down to a short list: give agents the fewest tools and permissions possible, require human approval for high-impact actions, have the target system check authorization again instead of trusting the caller, segment agents from each other, use short-lived tokens, watch for agents behaving unusually, such as reaching systems they've never touched before, and keep a kill switch ready.
4.7 Identity lifecycle and auditing
AI identity inventories
You can't review or revoke an identity you don't know exists. An identity inventory lists every human and non-human identity that touches AI systems. For each agent or service identity, record its name, accountable owner, purpose, credentials, permissions, the systems it reaches, any delegation relationships, when it was created and when it was last used. Link each one to its entry in the AI use-case inventory from Domain 2, so a retired use case leads straight to the identities that need retiring with it.
Provisioning and deprovisioning
Human identities follow the familiar joiner, mover, leaver process. Non-human identities need the same discipline:
- Provisioning creates an agent identity only for an approved use case, through an automated workflow that assigns an owner and least-privilege permissions from the start.
- Transitions happen when an agent's purpose changes, its owner changes, or the model behind it is replaced. Each one triggers a fresh permission review. When an owner leaves the organization, the agent gets a new owner or gets disabled.
- Deprovisioning happens when a use case is retired or an agent is replaced: revoke its tokens, delete its keys, remove it from tool allow-lists, disable the account and archive its logs.
Two failures show up again and again. Orphaned identities are non-human identities whose owner left or whose purpose ended while their credentials stayed active; attackers love them because nobody's watching. And when an employee leaves, any tokens they delegated to agents must be revoked too, or an agent keeps acting on behalf of someone who no longer works there.
Periodic access reviews and ISPM
An access review, also called recertification, has owners confirm on a schedule that each identity still needs the access it has. For agents, compare the permissions granted with the permissions actually used, and remove the unused ones. Review higher-tier agents more often.
Identity Security Posture Management (ISPM) tools continuously check identity configurations across identity providers, clouds and SaaS applications. They flag excessive permissions, dormant accounts, missing MFA, old keys, risky delegations and non-human identities with no owner. ISPM does for identities what DSPM does for data. Its companion, identity threat detection and response (ITDR), looks for attacks on identities in progress.
Auditing AI resource use
- Token accounting tracks how many tokens each user, agent, application and use case consumes. It supports cost allocation and quota enforcement, and it's a security signal: a sudden spike can mean a stolen key, an extraction attempt, abuse or an agent stuck in a loop.
- Rate limit hits are worth auditing too. A burst of them from one identity is a lead to investigate, and the review tells you whether the limits still fit real use.
- Session time-outs end AI and agent sessions after a period of inactivity and cap their total length. Long-lived sessions hold context and tokens that an attacker can reuse, so they should end and require fresh authentication.
Because every request is tied to an identity (4.5), usage logs double as accountability records.
Practice questions
These are my own questions in the ISC2 style. Pick the BEST answer, and when two look right, put the safety of people first, then legal obligations, then the mission.
- An AI assistant summarizes incoming email and also has permission to send email. An attacker sends a message with hidden instructions telling the assistant to forward the inbox. Which design change BEST reduces this risk?
- A. Add a system prompt instruction that forbids forwarding
- B. Remove the send permission from the summarizing function
- C. Raise the content filter's sensitivity
- D. Log all outbound email
- An organization deploys an AI agent that can change production firewall rules. Which control is MOST important?
- A. Human approval before each change is applied
- B. A weekly review of the agent's logs
- C. A larger, more capable model
- D. A detailed system prompt describing approved changes
- Several AI agents share one service account to reach internal systems. What is the PRIMARY concern?
- A. Higher licensing costs
- B. Slower authentication
- C. Loss of accountability for which agent took which action
- D. Higher token consumption
- The service that filters a model's outputs becomes unavailable. What should the system do?
- A. Keep serving responses and log the outage
- B. Switch to a smaller model
- C. Retry the filter indefinitely
- D. Block or hold responses until filtering is restored
- An agent needs to call a payment API. Where should the API credentials be?
- A. In the harness, which attaches them when it executes the call, outside the model's context
- B. In the system prompt, encrypted
- C. In the agent's long-term memory
- D. In the user's prompt for each session
- A data science team runs notebook servers on cloud instances with public IP addresses and no authentication, for convenience. What should the security professional recommend FIRST?
- A. Install endpoint protection on the instances
- B. Put the notebooks behind authentication on a private network
- C. Train the data scientists on secure coding
- D. Turn on detailed logging
- A user with read-only access to HR records asks an AI assistant, which has broad access to the HR system, to change a colleague's salary. The assistant makes the change. Which principle was MOST directly violated?
- A. Separation of duties
- B. Secure defaults
- C. Data minimization
- D. Least privilege
- An employee who set up several AI agents leaves the organization. What should happen?
- A. Disable the employee's accounts and leave the agents running
- B. Delete the agents' logs
- C. Assign each agent a new owner or disable it, and revoke the tokens the employee delegated
- D. Let the agents run until the next annual access review
- Fire is detected in a data center full of dense AI racks. The suppression system uses an inert gas that lowers oxygen levels. What is the FIRST priority?
- A. Make sure everyone is evacuated before the gas discharges
- B. Shut down the GPU servers gracefully to protect model weights
- C. Copy the training data to another site
- D. Notify the cloud provider
- A team wants to be able to fully honor future requests to remove personal data from an AI system. Which architecture BEST supports that?
- A. Fine-tune the model on the personal data and use approximate unlearning when needed
- B. Keep personal data in a RAG index, with lineage, outside the model's weights
- C. Encrypt the model weights
- D. Train a larger model so each record has less influence
- An organization connects its AI assistant to third-party MCP servers. Which control BEST protects against a server that changes its tool descriptions after they were approved?
- A. Rate limiting tool calls
- B. A larger context window
- C. Allow-listing servers, pinning approved versions and checking tool definitions for changes
- D. Encrypting traffic to the server
- A team is choosing between a hosted model API and a self-hosted open model for an application that processes regulated data that must stay in the country. The team has little experience running GPU infrastructure. What should the security professional do FIRST?
- A. Recommend self-hosting, since it gives full control
- B. Recommend the hosted API, since it's faster to deploy
- C. Benchmark both models for accuracy
- D. Find out whether a hosted provider can guarantee in-country processing, and weigh that against the team's ability to run self-hosting securely
Show the answer key
- B. This is excessive agency: the function had a permission its job didn't need. Removing it eliminates the risk, while a prompt instruction (A) can be overridden by the same injection it's meant to stop.
- A. Firewall changes can cause outages and open the network, and they're hard to undo safely, so a person approves before they take effect. Weekly log review (B) finds the damage after it's done.
- C. Shared credentials destroy non-repudiation; the logs can't show which agent did what, or for whom.
- D. Guardrails fail closed. Serving unfiltered responses (A) is a guardrail failing open.
- A. The model should never see a secret, because anything in its context can be talked out of it. Encrypting the prompt (B) doesn't help once the model needs to read it.
- B. Exposed, unauthenticated notebooks are an open door, so close it first. The other steps help, but none of them closes it.
- D. The agent's effective permissions exceeded the user's, so it acted as a confused deputy. An agent's authority should never be more than that of the person it serves.
- C. Agents without an accountable owner become orphaned identities, and delegated tokens must end when the person who granted them leaves. Waiting for the annual review (D) leaves the gap open for months.
- A. Human life comes first, every time. Equipment and data come after people are safe.
- B. Deleting from a RAG index is complete and easy to verify, and lineage shows where the data went. Approximate unlearning (A) is hard to prove.
- C. Pinning and change detection catch a definition that changed after approval. Rate limits and encryption don't look at what the tool says it does.
- D. The requirement to keep data in the country and the team's skills both decide the right answer, so establish the facts before choosing. A and B each jump to a conclusion.
Sources
- ISC2 AI security certification outline
- NCSC and CISA, Guidelines for Secure AI System Development
- OWASP GenAI Security Project, including the Top 10 for LLM Applications and agentic AI guidance
- MITRE ATLAS
- Model Context Protocol specification, including security best practices
- Agent2Agent (A2A) protocol
- NIST SP 800-207, Zero Trust Architecture
- CIS Benchmarks
- SLSA, Supply-chain Levels for Software Artifacts
- SPIFFE workload identity standard
- RFC 8693, OAuth 2.0 Token Exchange
- OWASP AI Exchange