ISC2 AI Security Pilot Exam · Free study guide
Domain 3 Study Notes: AI Data Management, Privacy, and Model Security
How to use these notes
Domain 3 is about the two things an AI system can't work without: its data and its model. Half the domain covers data governance and privacy, and the other half covers protecting the model itself as a valuable, attackable asset. You don't need a security or privacy background; every term is defined where it first appears. I've followed the ISC2 outline section by section, and each section ends with how I expect it to show up on the exam.
One idea ties the domain together: a model is only as trustworthy as the data that shaped it. Poisoned, mislabeled or unlawfully collected data produces a model with the same problems baked in, and no amount of security around the finished model removes them. That's why so many questions here point back to provenance, integrity checks and controls early in the pipeline.
Everything here is my own work, built from public frameworks and guidance, especially the joint CISA, NSA and FBI guidance on AI data security from May 2025. If a reference book frames something differently, trust the book.
3.1 AI data governance, lineage and accountability
Data ownership roles
Every dataset that feeds an AI system needs named people behind it. These roles come up constantly:
| Role | What they do |
|---|---|
| Information owner (data owner) | A senior person accountable for the dataset. Decides its classification, who can access it and how it may be used, including whether it may train a model |
| Data steward | Manages quality, definitions, metadata and policy compliance day to day on the owner's behalf |
| Data custodian | The IT or security team that runs the technical controls: storage, backups, encryption and access |
| Data controller | A GDPR term for the organization that decides why and how personal data is processed. It carries the main legal responsibility |
| Data processor | A GDPR term for an organization that processes personal data on the controller's behalf, like an AI vendor running inference on your data |
| Data subject | The person the data is about |
The owner is accountable and the custodian implements; exam questions like to test that difference. The controller and processor line matters for AI vendors too. A vendor that uses your data for its own purposes, such as training its own models, may become a controller for that processing, which is one more reason contracts spell out what the vendor may do with your data.
Sovereignty and cross-border residency
Three terms get used interchangeably and shouldn't be:
- Data residency is where data is physically stored and processed. It's often a business or contractual choice.
- Data sovereignty means data is subject to the laws of the country where it sits, and sometimes to the laws of the provider's home country as well. A US cloud provider can be compelled by US legal process to produce data it stores in Europe.
- Data localization is a legal requirement that certain data stay inside a country.
AI makes residency harder to pin down. Inference may run in a different region from where data is stored, some providers route requests to whichever region has capacity, and fine-tuning, logs and caches all create new copies in new places. Under GDPR, personal data can leave the European Economic Area only with a recognized safeguard, such as an adequacy decision for the destination country, standard contractual clauses (SCCs) or binding corporate rules.
Classification, tagging and sensitive data impact assessments
Data classification assigns each dataset a sensitivity level, like public, internal, confidential or restricted. Tagging attaches that label as metadata, so tools can act on it automatically; a training pipeline can refuse restricted data, and a DLP tool can block it from leaving.
Two AI-specific rules are worth remembering. First, derived artifacts such as embeddings, fine-tuned models and outputs should inherit the highest classification of the data that went into them, unless someone analyzes them and decides otherwise. Second, aggregation can raise sensitivity; individually harmless records combined in one training set can reveal something none of them reveals alone.
A sensitive data impact assessment happens before sensitive data enters an AI pipeline. It asks what sensitive data is included, why it's needed, what happens if the model memorizes or exposes it, and which controls apply. It works like a DPIA, scoped to the data.
Provenance, lineage and sourcing
Provenance is where data came from: who created or collected it, when, and under what terms. Lineage is the path it took afterward through systems and transformations, all the way to the models it trained. Together they let you answer the question every incident eventually asks, which is how bad data got into the model and what else it touched. The CISA guidance recommends tracking provenance in a cryptographically signed, append-only record, so the history itself can't be altered.
Data also comes from two kinds of sources. Real data is collected from actual people and events. Synthetic data is generated by an algorithm or model to mimic the statistical patterns of real data. Synthetic data reduces privacy exposure and can fill gaps, but it can miss rare cases, carry the biases of whatever generated it, and leak real records if the generator memorized them. Label synthetic data as synthetic in your lineage records, so nobody mistakes it for the real thing later.
Chain of custody
Chain of custody is a documented, unbroken record of who handled data, when and what they did with it, from collection through deletion. The idea comes from forensics, where evidence is worthless if you can't show nobody tampered with it. For AI data the controls are hashes at each handoff, digital signatures, access logs, write-once storage and signed transfer records. When you suspect a dataset was poisoned, chain of custody is how you find where it happened.
The AI data lifecycle

Each stage carries its own risk. Collection is where unlawful or untrustworthy data gets in, validation is your best chance to catch bad labels and poisoned samples, and transformation is where lineage most often breaks. Retention and archiving bring the usual minimization and access questions, and deletion has to reach every copy, including the derived ones.
3.2 Data privacy and compliance controls
Privacy by design
Privacy by design means building privacy into a system from the start instead of bolting it on after launch. GDPR turns the idea into law as data protection by design and by default, which also requires default settings to be the most privacy-protective option. The outline names four principles:
- Consent, where consent is the legal basis, has to be freely given, specific, informed and unambiguous, and withdrawing it has to be as easy as giving it.
- Purpose limitation means data collected for one stated purpose isn't reused for an incompatible one.
- Lawful usage means every processing activity needs a legal basis. GDPR has six: consent, contract, legal obligation, vital interests, public task and legitimate interests.
- Data minimization means collecting and keeping only what the purpose needs.
In an AI pipeline, those principles look like stripping identifiers before training, a privacy review gate before any new dataset enters the pipeline, customer-data training turned off by default, and prompt logs that don't outlive their purpose. Ask whether the model needs names at all. Usually it doesn't.
Data protection techniques
| Technique | How it works | Reversible? | Typical use |
|---|---|---|---|
| Tokenization | Swaps a sensitive value for a random token and keeps the mapping in a secured vault | Yes, through the vault | Card numbers and IDs moving through pipelines; shrinks PCI scope |
| Masking | Hides part or all of a value. Static masking changes stored copies permanently; dynamic masking hides values at display time | Static no; dynamic yes, for authorized users | Test data, showing only the last four digits |
| Pseudonymization | Replaces identifiers with consistent stand-ins that can be re-linked using separately held information | Yes, with that information | Analysis that must link records; still personal data under GDPR |
| Anonymization | Removes any reasonable way to identify a person, permanently | No | Data that falls outside GDPR, if truly anonymous |
| Noise injection | Adds random variation to values or results | No | Blurring exact figures |
| Differential privacy | Adds calibrated noise so a result barely changes whether or not any one person's data is included | No | Statistics and model training |
Differential privacy is the only one of these with a mathematical guarantee. Its strength is set by a privacy budget, written as epsilon (ε): a smaller epsilon means stronger privacy and less accuracy.
True anonymization is harder than it looks. Researcher Latanya Sweeney showed that ZIP code, birth date and sex alone uniquely identify about 87% of Americans, and researchers re-identified people in Netflix's "anonymized" movie ratings by matching them against public IMDb reviews. When a question offers "remove the names" as anonymization, be suspicious.
Global privacy regulations
| Regulation | Who it covers | What matters for AI |
|---|---|---|
| GDPR | Anyone processing personal data of people in the EU, wherever the organization is based | A lawful basis for training and use, DPIAs, limits on solely automated decisions, the right to erasure. Fines reach €20 million or 4% of global annual turnover, whichever is higher |
| HIPAA | US healthcare providers, health plans and clearinghouses, plus their business associates | PHI needs a BAA before it goes to an AI vendor. De-identification follows either Safe Harbor (remove 18 listed identifiers) or Expert Determination. The minimum necessary standard limits what you share |
| PCI DSS | Anyone who stores, processes or transmits payment card data. It's an industry standard that the card brands enforce through contracts | Any system that touches card numbers enters PCI scope, so keep them out of prompts, logs and training data. Tokenize instead |
State laws such as California's CCPA add their own rules, but the three above are the ones the outline names.
AI-specific privacy challenges
- Unintentional bias. Privacy and fairness can pull against each other. Removing sensitive attributes protects privacy, and it also makes bias harder to measure, because you can't check outcomes by group without knowing the groups. Some organizations keep demographic data in a separate, tightly controlled store used only for fairness testing.
- Right to erasure. Deleting a person's influence from a trained model is still an unsolved problem at scale. The options are retraining without their data, training on separate shards so only one shard needs retraining, or filtering outputs as a stopgap.
- Memorization. Models can memorize training data and repeat it word for word. Researchers have extracted names, phone numbers and email addresses from a public language model this way.
- Membership inference. An attacker works out whether a specific person's record was in the training set. That alone can be a breach; knowing someone was in a cancer study's dataset reveals their diagnosis.
- Attribute inference. A model can infer sensitive traits, like health conditions, from ordinary data, creating sensitive data nobody collected.
3.3 Securing training data and datasets
Training data integrity
Data integrity means data is complete, accurate and unchanged except by authorized processes. For training data, an integrity failure turns into the model's behavior. The CISA guidance recommends a few layers of protection:
- record a cryptographic hash (a fingerprint such as SHA-256) when a dataset is approved, and verify it before every training run;
- have data producers digitally sign what they deliver, so you can confirm who it came from and that it wasn't changed;
- store approved datasets in write-once storage, and restrict who can modify them; and
- run validation checks for duplicates, outliers and unexpected shifts in the data's distribution.
Poisoning, backdoors and extraction
Data poisoning is an attack where someone inserts or alters training samples to change how the model behaves. It comes in a few forms:
- Untargeted poisoning degrades the model's overall accuracy, an attack on availability.
- Targeted poisoning makes the model misbehave only on inputs the attacker chose.
- Clean-label poisoning uses samples that look correctly labeled to a human reviewer, so manual review misses them.
A backdoor is poisoning with a hidden trigger. The model behaves normally until an input contains the trigger, like a specific pixel pattern or phrase, and then does what the attacker wants. Because it behaves normally otherwise, a backdoored model passes ordinary testing.
Poisoning at scale turned out to be cheap. Researchers showed in 2023 that many web-scale datasets list URLs instead of storing content, so an attacker can buy an expired domain the dataset points to and swap in poisoned content; they called this split-view poisoning, and estimated they could have poisoned 0.01% of one well-known image dataset for about $60. A 2025 study by the UK AI Security Institute and Anthropic found that around 250 malicious documents were enough to plant a backdoor in language models of every size they tested, so a bigger model and a bigger dataset didn't dilute the attack.
Extraction threats run the other way: instead of putting bad data in, the attacker gets training data out, either by querying the model until it repeats what it memorized (covered in 3.2) or by stealing the curated dataset from storage.
Defenses stack up across the pipeline. Use trusted sources with recorded provenance, verify content hashes when downloading URL-based datasets, deduplicate, run statistical outlier detection, test models for unexpected trigger behavior, and limit who can write to training data at all.
Label modification and malicious datasets
Label flipping changes the labels on training samples, say marking spam as legitimate, so the model learns the wrong lesson. The threat often comes from insiders or crowdsourced labelers. Controls include access control and audit trails on the labeling platform, separation of duties, several annotators per sample with agreement checks, and spot checks against a trusted gold-standard set.
Malicious datasets are published datasets that carry poisoned samples or even executable code. Some file formats and dataset loading scripts run code when you open them, so downloading a dataset can mean running a stranger's program. Scan datasets before use, load them in a sandbox, prefer formats that hold only data, and pin each dependency to a specific verified version.
Dataset tampering and versioning
Version datasets the way you version code. Tools like DVC and lakeFS keep each version immutable and identified by its hash, and every trained model should record the exact dataset version it learned from. That gives you four things: you can detect tampering when a hash doesn't match, reproduce any training run, roll back to a known-good version after finding poison, and list every model that trained on a bad version.
Recursive training and synthetic data trustworthiness
Recursive training happens when models learn from data that earlier models produced, including AI-generated content scraped from the web. A 2024 study in Nature showed that repeating this over generations causes model collapse: models gradually lose the rare cases at the edges of the data and drift toward bland, distorted output. Errors and poisoning compound too, since a flawed generator passes its flaws to every model trained on its output.
To trust synthetic data, validate it against a held-back set of real data, label every synthetic record in your lineage, keep real data in the training mix as an anchor, check the generator for memorized personal data and for bias, and record the generator's version as part of the data's provenance.
3.4 Cryptographic and data protection solutions
Data Security Posture Management
Data Security Posture Management (DSPM) tools scan cloud storage, databases and SaaS applications to find data, classify it, map who can reach it and flag risky exposure, like a public storage bucket, an over-permissioned account or customer data sitting somewhere it shouldn't. For AI, DSPM is how you find the training set someone copied into a personal bucket, the vector store full of HR documents, and the forgotten export a data scientist made six months ago. It answers two questions: where is our sensitive data, and who can get to it? Don't confuse it with CSPM, which checks the configuration of cloud infrastructure instead of the data inside it.
DLP and semantic filtering
Data Loss Prevention (DLP) detects and blocks sensitive data leaving through endpoints, email, the network or cloud apps. Classic DLP matches patterns, like the shape of a card number or Social Security number, or fingerprints of specific documents. For AI, DLP watches both directions: prompts going out to AI services, and responses coming back from models.
Patterns miss a lot with AI, because a model can paraphrase. A summary of a confidential acquisition plan contains no card numbers. Semantic filtering uses language models or classifiers to judge what text means instead of what it looks like, so it can catch a paraphrased secret on the way in or block a model from revealing restricted content on the way out. Semantic filters are models themselves, so they make mistakes and can be fooled; use them as one layer among several.
Encryption for each data state
| State | What it covers | How it's protected |
|---|---|---|
| At rest | Stored data: disks, databases, object storage, backups, vector stores and model weights | AES-256 encryption at the disk, database or field level |
| In transit | Data moving across networks, including between the app, the vector store and the model server | TLS 1.2 or later, and mutual TLS between services |
| In use | Data in memory while it's being processed | Confidential computing in TEEs (see Domain 2); homomorphic encryption and secure multiparty computation compute on protected data, at a large performance cost |
The CISA guidance recommends AES-256 and planning now for post-quantum cryptography, since NIST published its first post-quantum encryption standards in August 2024. Training data and model weights can stay valuable for years, which makes them a target for attackers who steal encrypted data today and wait for the means to decrypt it later.
Key management lifecycle
Encryption is only as good as the handling of its keys. Keys go through a lifecycle of generation, distribution, storage, use, rotation, revocation and destruction, and each stage needs controls.
- A Key Management Service (KMS) is a managed service that creates, stores and controls the use of keys, usually backed by hardware security modules (HSMs), with access policies and logs of every use.
- Envelope encryption encrypts data with a data key and then encrypts that data key with a master key held in the KMS. Rotating the master key doesn't require re-encrypting all the data.
- Bring Your Own Key (BYOK) means you generate keys yourself and import them into the cloud provider's KMS, which gives you control over creation and the ability to revoke them. Hold Your Own Key (HYOK) goes further and keeps keys in your own HSM outside the provider entirely, trading convenience for control.
- Secrets management covers API keys, tokens and passwords. AI applications are full of them: model API keys, vector database credentials and the credentials agents use to call tools. Keep them in a vault, never in code, notebooks or system prompts, and prefer short-lived credentials.
- Key rotation replaces keys on a schedule and immediately after a suspected compromise, which limits how much data any single key exposes.
Keep key administration separate from data administration, so no single person can both reach the data and control its keys.
Purpose-based and restricted data access
Role-based access control (RBAC) grants access by job role, and attribute-based access control (ABAC) decides using attributes of the user, the data and the situation. Purpose-based access control adds a declared purpose: data released for training the fraud model can be used for that and nothing else, which puts purpose limitation into the access system.
AI agents raise the stakes. An agent acting for a user should get that user's permissions narrowed to the task, never a broad service account. An agentic AI data catalog is a governed inventory listing which datasets agents may use, along with each dataset's owner, classification, allowed purposes and access policy, so an agent finds and requests data through policy.
The same rule applies to RAG. The retriever must enforce document permissions at query time and return only content the person asking is allowed to see. Otherwise the AI assistant becomes a way around every access control in the organization.
Media sanitization on high-density AI storage
NIST SP 800-88 defines three levels of sanitization:
- Clear overwrites data using normal read and write commands, which defeats simple recovery tools.
- Purge defeats laboratory recovery, using methods like cryptographic erase, block erase or degaussing for magnetic media.
- Destroy makes the media unusable through shredding, disintegrating, pulverizing or incinerating.
AI infrastructure runs on dense SSD and NVMe storage, and overwriting is unreliable there. SSDs spread writes across spare blocks the operating system can't address, so an overwrite can leave old data behind. Cryptographic erase works well on self-encrypting drives: destroying the drive's encryption key makes everything on it unreadable in seconds, provided you trust the drive's encryption. Micro-shredding cuts media into very small particles, because a single flash chip holds so much data that a fragment from an ordinary shredder can still contain an intact chip. Whatever the method, verify it and keep a certificate of sanitization. For data held by a cloud provider, encrypting with keys you control and destroying those keys is often the only sanitization you can perform yourself.
3.5 Model lifecycle and security governance
Model governance
Model governance applies the use-case governance from Domain 2 to the models themselves. Every model needs an owner who is accountable for it, a risk classification that decides how much control it gets, and an approval workflow that says who must sign off before the model reaches production, gets retrained or changes in a material way. Higher tiers need more approvers, and the people who built a model shouldn't be the only ones approving it.
Model documentation and approved registries
A model card is a short document that travels with a model. It records the intended uses and the uses it isn't suited for, a summary of the training data, evaluation results (including results broken out by group), known limitations, ethical considerations, the version and the owner. Google researchers proposed the format in 2019, and it's now standard practice.
An approved model registry is the central catalog of models and versions cleared for use, each with its metadata: lineage, training dataset version, evaluation results, approval record, hash and signature, and deployment status. The rule that makes a registry useful is simple: production systems load models only from the registry. That covers third-party and open models too, which go through the same review before they're registered.
Securing the deployment pipeline

The pipeline that carries a model from training to production is itself a target, since whoever controls it controls what reaches production. The outline names four stages to secure:
- Validation runs accuracy, bias and adversarial tests, and scans model files for embedded malicious code before anything else happens.
- Packaging bundles the model and its dependencies into a container with pinned versions, scans it for vulnerabilities and records it in the AIBOM.
- Serving runs the model on a server that requires authentication, uses TLS, runs with least privilege and sits on an isolated network segment. It verifies the model's signature before loading it.
- Credential management gives pipeline service accounts only the permissions they need, keeps secrets in a vault, and uses short-lived credentials so a stolen token expires quickly.
Input and output sanitization, updates and rollback
Input sanitization checks inputs for type, length and format and filters known attack patterns. For language models it can't be complete, because instructions hidden in natural language are hard to separate from legitimate text, and prompt injection has no full fix. That makes output handling just as important.
Output sanitization treats model output as untrusted input to whatever comes next. Encode it before displaying it on a web page so it can't run script, use parameterized queries before it touches a database, validate it before it triggers a command or tool call, and filter personal data and secrets out of it.
Updates go through the same pipeline and approvals as the original model. Roll them out in stages: a canary release sends a small share of traffic to the new version first, and a shadow deployment runs the new model alongside the old one and compares outputs without using the new ones.
Rollback means keeping the previous approved version ready, deciding in advance what triggers a rollback and who can pull it, and testing the procedure before you need it. Remember that a model's behavior also depends on its system prompt, its configuration and its RAG index, so those need versions and rollback paths as well.
Security lifecycle monitoring
- Model drift is performance degrading as the world changes. Data drift means the inputs look different from the training data; concept drift means the relationship between inputs and the right answer has changed.
- Bias tracking measures fairness metrics by group in production over time, because a model that was fair at launch can drift out of fairness.
- Anomaly detection watches for unusual query volumes, strange input patterns that suggest extraction or injection attempts, unexpected outputs, spikes in filter hits and spikes in cost.
Monitoring results feed the reassessment triggers from Domain 2.
Privacy-preserving audit trails
You need records of AI data events, like who accessed which dataset, what prompts went in and what decisions came out, to investigate incidents and prove accountability. Those records are sensitive data themselves, though, since prompts often contain personal or confidential information. A privacy-preserving audit trail keeps enough to investigate without becoming a new data store to protect:
- log metadata and hashes instead of full content where that's enough;
- pseudonymize user identifiers, and redact personal data from prompts before they're logged;
- encrypt logs, restrict who can read them and log that access too;
- set retention limits; and
- use append-only, tamper-evident storage so the logs can be trusted as evidence.
3.6 Model confidentiality, integrity and availability
The CIA triad you already know applies to the model as an asset in its own right.
Model confidentiality
A trained model's weights, the billions of numbers it learned, are the model. They're expensive to produce, and stolen weights let an attacker use the model for free, study it to design better attacks, or strip out its safety training. The architecture, training recipe, system prompts and fine-tuning data are intellectual property too.
Protect weights the way you'd protect your most sensitive data: encrypt them at rest, restrict and log every access, keep the number of copies small, apply insider-threat controls, and watch for large outbound transfers. A 2024 RAND report on securing model weights mapped dozens of distinct ways weights can be stolen and defined five security levels, from defending against opportunistic attackers up to defending against well-resourced nation states. The more capable and valuable the model, the higher the level it needs.
Model extraction prevention
Model extraction, also called model stealing, happens when an attacker sends a model many queries and uses the input and output pairs to train a copy that behaves the same way. The attacker never touches your servers; they use your API exactly as designed, just at volume and with a purpose. Defenses include:
- authentication on every model API, so queries tie back to an account;
- rate limits and quotas that make large-scale querying slow and expensive;
- monitoring for extraction patterns, such as high volumes of systematic queries that sweep across many kinds of input;
- returning less information, like a label instead of confidence scores, or rounded scores instead of exact ones; and
- terms of service that prohibit training competing models on your outputs, backed by watermarks that help prove it happened.
Model watermarking and fingerprinting
These two sound alike and work differently:
- Model watermarking deliberately embeds a hidden, verifiable signal in a model, either in its weights or as secret trigger inputs that produce a known output. The owner can later test a suspicious model for that signal and prove the model came from them.
- Model fingerprinting identifies a model by characteristics it already has, without changing it, such as how it responds to a set of crafted test queries or a hash of its weights. Fingerprints help you spot stolen copies and confirm that a vendor is serving the model it says it is.
A watermark is added on purpose, and a fingerprint is measured from what's already there. Neither is foolproof; fine-tuning a stolen model or distilling it into a new one can weaken both.
Model integrity
- Hashing records a model file's fingerprint when it's registered, and the serving system checks it again before loading.
- Model signing adds a digital signature that proves who produced the model and that it hasn't changed since. The OpenSSF model signing project, built on Sigstore, gives teams a standard way to do this.
- Version control gives every model version an immutable identifier traced to its data and code, so you always know exactly what's serving.
File format matters too. Some common model formats, including Python's pickle, can run arbitrary code when a model is loaded, so loading an untrusted model can compromise the server. Formats like safetensors store only the numbers.
Model availability
AI services are expensive to run, which gives attackers two ways to hurt you: take the service down, or run up the bill.
- Rate limiting caps how many requests a user or API key can make in a time window.
- Quota enforcement caps total usage, in tokens, requests or spend, per user, team or application over a period. Quotas are the main defense against denial of wallet, an attack that floods a pay-per-use AI service with costly requests to drive up the bill instead of taking it offline.
- Input limits and timeouts cap prompt size and processing time, which blunts sponge examples, inputs crafted to make a model burn as much compute as possible.
- Graceful degradation keeps the service useful when the model is overloaded or down, by falling back to a smaller model, cached answers or a non-AI path like a human queue. When safety is involved, the fallback must fail safe, moving to a known safe state instead of guessing.
Relying on one model provider is an availability risk too, so critical services may need a second provider ready to take over.
3.7 AI data and model supply chain risk
Domain 2 covered the contract side of AI suppliers. This section covers the technical side: how to judge the datasets and models you didn't build before you trust them.
Assessing third-party, open and public datasets
Before a dataset from outside the organization goes near a model, a supply chain risk assessment should answer these questions:
- Who created it, how was it collected, and is there documentation such as a dataset card?
- What license applies, and does it allow your intended use?
- If it holds personal data, was there a lawful basis or consent for this kind of use?
- Is it maintained, and does the publisher post hashes so you can verify your copy?
- Does it store content, or only links to content that could change after the fact?
- Could it contain copyrighted, harmful or illegal material?
That last question isn't hypothetical. A 2024 audit of popular AI datasets found that most listings on major hosting sites were missing license information, and many of the licenses that were listed were wrong. A team that trusted those labels could have trained a commercial model on data it had no right to use. Scale the depth of the assessment to the use; a dataset for an internal experiment needs less scrutiny than one training a customer-facing model.
Verifying provenance and authenticity
Provenance tells you where something claims to come from; authenticity verification proves the claim. For models and data from vendors and public hubs:
- download from the publisher's verified organization account instead of a look-alike;
- check signatures, and compare hashes against the values the publisher posted;
- ask vendors for an AIBOM and for evidence of how the model was built and tested; and
- pin each model to a specific verified version or hash, never just a name.
Pinning matters because names can be hijacked. Security researchers have shown that when a publisher's account on a model hub is deleted, an attacker can register the same name, so code that pulls a model by name downloads the attacker's version instead, and nobody notices. Typosquatting, publishing under a name one character off from a popular one, works the same way it does for software packages.
Risks of foundation and pre-trained models
A foundation model is a large model trained on broad data that other applications build on, and a pre-trained model is any model you start from instead of training from scratch. Building on one is sensible, and it means inheriting risks you can't fully see:
| Risk | What it looks like | Controls |
|---|---|---|
| Unknown training data | Copyrighted, personal or poisoned data you can't inspect | Vendor documentation, contract terms, your own testing |
| Hidden backdoors | Research in 2024 showed planted backdoor behavior can survive standard safety training | Source only from trusted publishers, red team before registering |
| Shared weaknesses | A jailbreak that works on the base model works on every product built on it | Your own guardrails, and monitoring advisories for the model |
| Malicious model files | Formats that run code when loaded | Scan files, load in a sandbox, prefer safe formats |
| Weakened safety after fine-tuning | Research has shown that fine-tuning, even on harmless data, can erode a model's safety behavior | Re-run safety and red team tests after every fine-tune |
| Inherited bias and hallucination | The base model's flaws show up in your outputs | Bias and accuracy testing in your own context |
| License limits | Acceptable use policies or commercial restrictions | Legal review at intake |
| Vendor dependency | Deprecations, silent updates, outages | Version pinning, notice terms, a fallback model |
All of this comes together at registry intake. A third-party model gets scanned, tested, red teamed, documented and approved before it's registered, the same as one you built.
Practice questions
As with Domain 2, these are my own questions in the ISC2 style. Pick the BEST answer, and when two look right, put the safety of people first, then legal obligations, then the mission.
- A data science team wants to train a customer-facing model on a popular open dataset from a public repository. What should happen FIRST?
- A. Assess the dataset's provenance, license and contents
- B. Begin training and monitor the model for anomalies
- C. Encrypt the dataset at rest
- D. Convert the dataset into synthetic data
- An organization learns that an unauthorized party altered a training dataset before the current production model was trained. The model supports clinical decisions. What should be done FIRST?
- A. Add output filters to catch incorrect recommendations
- B. Notify the team that supplied the dataset
- C. Stop or restrict the model's use in patient care
- D. Add more data to the current dataset and retrain
- Which technique gives a mathematical guarantee that limits what anyone can learn about an individual from a model's outputs?
- A. Tokenization
- B. Masking
- C. Pseudonymization
- D. Differential privacy
- An organization is returning leased SSD storage arrays that held AI training data, including regulated personal information. Which sanitization method is MOST appropriate?
- A. Degauss every drive
- B. Perform a verified, documented cryptographic erase of the self-encrypting drives
- C. Delete the files and reformat the drives
- D. Overwrite each drive with random data three times
- An employee receives an answer from the company's RAG-based assistant that includes details from a confidential executive memo they aren't authorized to read. What is the MOST likely root cause?
- A. The model hallucinated the details
- B. The retriever doesn't enforce document permissions for the person asking
- C. The memo wasn't encrypted at rest
- D. The model wasn't fine-tuned on company policy
- An attacker sends a high volume of queries that systematically sweep across the range of possible inputs to a public prediction API, which returns full confidence scores. What is the attacker MOST likely attempting?
- A. Model extraction
- B. Data poisoning
- C. Denial of wallet
- D. Prompt injection
- Which control BEST ensures that only approved models run in production?
- A. A model card for every model
- B. Continuous drift monitoring
- C. Loading only signed models from the approved registry, with signatures verified at load time
- D. Rate limiting on the serving endpoint
- A development team puts the API key for a commercial model service in an agent's system prompt so the agent can call the service. What is the BEST recommendation?
- A. Encrypt the system prompt
- B. Rotate the key every month
- C. Instruct the model never to reveal the key
- D. Move the key to a secrets vault and have the application retrieve short-lived credentials at runtime
- A hospital wants to share patient data with a research partner to train a model. The partner has no need to identify anyone. Which approach BEST reduces privacy risk?
- A. Pseudonymize identifiers and keep the re-linking key in-house
- B. De-identify the data using an accepted HIPAA method, limit it to needed fields, and prohibit re-identification by contract
- C. Encrypt the data in transit to the partner
- D. Remove names and send everything else
- A generative AI service that supports patient intake triage at a hospital becomes overloaded at peak times. Which design is MOST appropriate?
- A. Queue requests until capacity returns, however long that takes
- B. Disable rate limits so the service can accept more requests
- C. Fall back to a staffed triage process with clear escalation
- D. Assign every patient the most common triage category until the service recovers
- Who should decide whether a dataset of customer records may be used to train a new model?
- A. The information owner
- B. The data custodian
- C. The data scientist leading the project
- D. The cloud provider hosting the data
- A team loads a pre-trained model from a public hub by its name. The publisher's account is later deleted, and someone else registers the same name and uploads a different model. Which control would BEST have prevented the team from loading the new model?
- A. Fine-tuning the model on internal data
- B. Rate limiting downloads from the hub
- C. Encrypting the model at rest
- D. Pinning the model to a verified hash or signed version from the original publisher
Show the answer key
- A. Provenance, license and content checks come before anything touches the data. Monitoring after training (B) finds problems too late, and encryption (C) protects confidentiality and does nothing for trustworthiness.
- C. Patient safety comes first, so the immediate step is to stop or restrict the model in care. After that, roll back to a known-good dataset version and retrain. Output filters (A) leave the poison inside the model, and retraining on the tampered data (D) keeps it.
- D. Differential privacy is the only option with a provable bound. Pseudonymized data can be re-linked, and masking and tokenization protect specific values without limiting what a model learns.
- B. The drives are leaving your control and must be reusable, and overwriting is unreliable on SSDs. Degaussing (A) doesn't work on flash storage, and reformatting (C) leaves data recoverable.
- B. The assistant returned content the person couldn't open directly, which means retrieval ignored their permissions. That's an access control failure.
- A. Systematic, high-volume queries plus full confidence scores is the classic setup for training a copy. Defenses are rate limits, monitoring and returning less information.
- C. Signing plus a registry that production must load from is what enforces approval. Model cards (A) document, and they don't block anything.
- D. Secrets don't belong in prompts, because prompts leak. Telling the model to keep a secret (C) isn't a control, and rotation (B) only shortens how long a leaked key works.
- B. The partner doesn't need identities, so de-identify and minimize; contract terms cover what technology can't. Pseudonymized data (A) is still personal data, and removing names alone (D) leaves people re-identifiable.
- C. People's health is involved, so the fallback must be safe and staffed. A shared default category (D) is dangerous, and indefinite queuing (A) delays care.
- A. The information owner decides how a dataset may be used. The custodian implements controls, and the project lead is a requester.
- D. Pinning to a verified hash or signature means a swapped model won't match. The other options don't verify where the model came from.
Sources
- ISC2 AI security certification outline
- CISA, NSA, FBI and partners: AI Data Security, best practices for securing data used to train and operate AI systems (May 2025)
- GDPR full text
- HHS guidance on HIPAA de-identification
- PCI Security Standards Council
- NIST post-quantum cryptography standards
- NIST SP 800-88, Guidelines for Media Sanitization
- RAND, Securing AI Model Weights (2024)
- OpenSSF model signing project
- OWASP AI Exchange
- MITRE ATLAS