AI Security: Threats, Attacks, and Secure Deployment

AI security protects models, data, tools, and AI-enabled applications against intentional manipulation, unauthorized access, theft, and disruption. It covers threat modeling for learned systems, adversarial machine learning, prompt injection, model and data attacks, supply-chain integrity, model extraction, data poisoning, inference security, agent security, and secure deployment. It is not a second edition of AI safety, a general cybersecurity encyclopedia, or a substitute for an organization’s governance program.

The security problem changes when software interprets untrusted language, learns from data, and can call tools. An attacker may not need a memory corruption bug. They may manipulate a retrieved document, craft an input that changes a classifier’s decision, steal a model through queries, poison a feedback set, or persuade an agent to use a valid credential for an invalid purpose. Effective defense starts by modeling those paths explicitly.

A threat model for AI systems

Begin with an asset and trust-boundary map. Assets include model weights, prompts, system instructions, training and retrieval data, embeddings, evaluation sets, credentials, tool APIs, logs, user identities, and the decisions or actions produced by the system. Trust boundaries often sit between a user and an application, an application and a model provider, a model and retrieved content, and a model and an external tool.

Write threats as causal paths. “Prompt injection is risky” is too vague to guide a control. “An untrusted document is retrieved, its instructions are treated as authority, the agent calls a finance tool, and an external transfer is initiated” identifies the attacker-controlled input, confused deputy, privilege, and consequence. The same model can be low risk in a read-only summarizer and high risk in a tool-using workflow.

Identify attacker capability and proximity. Can the attacker submit arbitrary prompts, upload documents, influence labels, observe outputs, measure latency, create many accounts, or compromise a dependency? Can they reach the training pipeline or only the public inference endpoint? Threats should be ranked by exploitability, asset value, blast radius, and recoverability.

Security reviews should include application engineers, ML engineers, infrastructure operators, and domain owners. A security engineer may spot an SSRF path while a domain owner recognizes that a seemingly harmless write operation changes a regulated record. Review the complete trajectory, not only the model’s text response.

Security versus safety, ethics, and governance

Security protects confidentiality, integrity, availability, authenticity, and authorized use. Its questions include whether a model can be extracted, whether data can be exfiltrated, whether a deployment can be tampered with, and whether an agent can exceed its permissions. Safety focuses on harms from model behavior, malfunction, or misuse and asks whether residual hazards are acceptable. A model can be secure yet unsafe because it gives dangerous advice; it can be safe in its refusals yet insecure because its weights or customer data are exposed.

Ethics concerns what should be valued and whose interests or rights a system should respect. Ethical questions shape requirements, but security controls do not settle them. Governance, used here as a plain-text organizational concept, defines policies, roles, approvals, accountability, audit processes, and escalation. Governance decides who accepts residual risk; security supplies evidence about attack paths and control effectiveness.

These disciplines overlap at boundaries. Prompt injection can be a security exploit when it crosses an authorization boundary, a safety issue when it causes harmful behavior, and a governance issue when no owner has approved the tool capability. Keep the ownership and tests distinct even when an incident is shared. Replacing threat modeling with a safety statement, ethics review, or committee approval leaves technical attack paths unaddressed.

Prompt injection and instruction confusion

Prompt injection occurs when untrusted content influences a model to disregard or reinterpret instructions that should have higher authority. Direct injection comes from a user prompt. Indirect injection arrives through a web page, email, document, code comment, image, or retrieved record. The core failure is not that the model “believes text”; it is that natural-language content is being used as a weak authorization mechanism.

Do not treat system prompts as secrets or as a complete security boundary. Separate instructions from data in application code, label provenance, constrain output schemas, and place authorization decisions outside the model. A model may propose a tool call, but deterministic policy code must verify the user, resource, action, scope, and confirmation state before execution.

Reduce injection impact with least privilege and narrow tools. Give an agent a read-only search tool instead of unrestricted HTTP, an allowlisted document action instead of arbitrary file access, and a transaction preview instead of direct payment authority. Require confirmation for irreversible actions and make the confirmation describe the concrete target and parameters.

Test multi-step attacks. An indirect injection may first ask the agent to retrieve a secret, then encode it in a harmless-looking response, then use a second tool to transmit it. Single-turn refusal tests will miss this chain. Evaluate tool trajectories, cross-document contamination, markup and encoding tricks, and injections that arrive after a trusted instruction has already been established.

Logging must preserve enough context to investigate without creating a new leak. Store instruction and data provenance, tool decisions, policy outcomes, and correlation IDs. Redact secrets and sensitive content. An audit trail that records only “agent completed” cannot establish whether the user or an injected document caused the action.

Adversarial machine learning

Adversarial ML attacks deliberately exploit model behavior rather than conventional software defects. Evasion changes an input at inference time to alter a prediction. Poisoning changes training or feedback data so the learned decision boundary shifts. Backdoors make a model behave normally until a trigger appears. Membership inference and model inversion attempt to learn whether particular data was used or reconstruct information about it.

Threat models must match access. A black-box attacker may submit queries and observe labels, scores, or generated text. A gray-box attacker may know the architecture or training method. A white-box attacker may access weights or gradients. Controls that help in one setting can fail in another, so document assumptions rather than claiming adversarial robustness in the abstract.

Use robust data collection and validation. Track source, collector, transformation, annotator, and approval for high-impact training data. Detect sudden distribution changes, duplicate clusters, suspicious label patterns, and examples linked to a single untrusted contributor. Keep trusted holdouts and canary examples outside routine contributor access.

Robustness testing should include realistic perturbations for the modality: wording and formatting changes for language, compression and crop changes for images, noise and timing changes for audio, and sensor or environment variation for embodied systems. Avoid reporting one robustness score as a universal guarantee. State which attacks were tested and where the model remains vulnerable.

Defenses such as adversarial training, input normalization, confidence thresholds, and ensemble checks can improve resilience, but they add cost and may shift errors. Measure operational performance on benign cases as well as attack success. A defense that blocks legitimate users or creates a predictable blind spot needs a different control, not a stronger marketing claim.

Data poisoning and feedback integrity

Data poisoning attacks target the learning loop. An attacker may submit malicious examples, manipulate labels, exploit an automated feedback signal, compromise a data connector, or cause a retraining job to select a contaminated slice. The risk grows when the system continuously learns from user interactions without review.

Protect provenance at ingestion. Assign source identities, timestamps, collection context, and integrity checks to records. Separate raw evidence from approved training material. Changes to labels or inclusion criteria should produce a new version with an accountable reviewer. Do not silently overwrite the dataset that a deployed model used.

Use quorum or independent review for high-impact labels. Monitor contributor concentration, label disagreement, unusual language patterns, and sudden improvements that cannot be explained by a known intervention. A quality score should not be the only signal because a coordinated attacker can optimize for the validation metric.

Retraining gates should compare a candidate against a protected baseline and hidden holdouts. Require an explanation for large metric changes, inspect slices, and check for trigger behavior. Keep the last trusted model available and make rollback independent of the poisoned pipeline. If a data incident is suspected, suspend automated learning before trying to improve the new model.

Feedback is not automatically ground truth. A user clicking “helpful” may be rewarding speed, politeness, or a desired answer rather than correctness. Treat feedback as a security-sensitive signal with an explicit interpretation and abuse limits.

Model extraction, inversion, and intellectual property

Model extraction reconstructs a substitute model by querying a target and observing outputs. Rich probability scores, deterministic responses, generous quotas, and unrestricted batch access make extraction easier. Extraction can expose proprietary capability even when the weights never leave the serving environment.

Limit information and rate. Return only the output detail the product requires, apply per-identity and aggregate quotas, detect automated query patterns, and vary defenses carefully so they do not damage legitimate reliability. Track unusual prompt diversity, repeated boundary probing, high-volume low-value requests, and accounts that coordinate from many network locations.

Do not confuse rate limiting with a complete solution. Attackers can distribute queries, use adaptive sampling, or exploit an endpoint with a generous internal service identity. Tie quotas to authenticated principals and business purpose, and enforce limits at the gateway as well as inside the model service.

Model inversion and membership inference are privacy threats when outputs reveal training examples or sensitive attributes. Minimize sensitive training data, avoid unnecessary memorization, protect evaluation sets, and test whether distinctive records can be elicited. Differential privacy or other privacy-preserving methods may help for specific workloads, but assess their utility and implementation assumptions rather than applying a label without measurement.

Watermarking and fingerprinting can support attribution or detection, but they are not access control. Protect weights with encryption, signing, export restrictions, and audited administrative access. For hosted services, contract for retention, training use, model updates, and incident notification.

Supply-chain security for models and data

An AI release may include source code, containers, base images, model weights, tokenizers, adapters, prompt templates, evaluation suites, retrieval indexes, and tool definitions. Every component can be replaced, tampered with, or upgraded into an unsafe behavior. Maintain a software and model bill of materials with versions, digests, licenses, owners, and provenance.

Verify artifacts before execution and promotion. Fetch dependencies from approved registries, pin versions, scan packages and images, sign trusted model packages, and verify signatures in deployment workflows. Treat serialized model formats and custom loading code as executable supply-chain inputs, not passive data files.

Separate build and runtime identities. A training job should not be able to rewrite the production registry merely because it needs to publish a candidate. Deployment workers should receive only the artifact and configuration they require. Break-glass access should be time limited, logged, and reviewed.

Monitor upstream changes. A provider may update a hosted model, a tokenizer, a safety filter, or a data connector without changing your application code. Pin where possible, require notices for material changes, and re-run security and behavior tests before expanding traffic. Record the dependency state in the release evidence.

Open-source availability changes distribution, not risk. Public weights can be inspected and adapted, but they can also be modified, bundled with malicious loaders, or stripped of controls. Scan, verify, sandbox, and evaluate before use. Internal mirrors should preserve provenance rather than becoming untracked download caches.

Secure inference and deployment

Secure deployment begins at the API boundary. Authenticate callers, authorize model and data access, validate request size and schema, enforce quotas, and reject unsupported modalities before they reach expensive workers. Use separate endpoints or policy profiles for public, internal, and privileged workloads. Do not rely on the model to decide whether a caller may access a record.

Protect data in transit and at rest, including prompts, completions, embeddings, cached context, traces, and evaluation artifacts. Retain only what debugging and compliance require. Apply tenant-aware keys and access controls to caches and indexes. A secure model endpoint can still leak data through a shared retrieval cache or an overprivileged observability dashboard.

Isolate inference workers from unnecessary network access. Egress allowlists, private service endpoints, sandboxed code execution, and restricted metadata access reduce blast radius. If a model can generate code, keep execution separate from the model process and apply resource, filesystem, network, and time limits.

Validate outputs before they cross a trust boundary. Enforce typed schemas, escaping, destination allowlists, and business rules. A generated SQL statement should not be executed because it parses; it needs authorization, parameterization, and a constrained operation set. A generated email should not be sent to an arbitrary address merely because the model returned a valid string.

Health checks should test the security-relevant path: identity, policy, model, retrieval, and tool authorization. A green port check does not show that a deployment is using the intended policy bundle or that a deny rule still works.

Agent security and confused deputies

Agents combine a model with memory, retrieval, tools, planning, and repeated execution. Their risk comes from composition. A model that is acceptable for text generation becomes a security concern when it can read mail, browse the web, modify tickets, or spend money. Agent security therefore centers on authority, state, and action boundaries.

Give each agent a narrow identity and a short-lived credential. Bind permissions to user, task, resource, and purpose. Do not give an agent the same broad role as its human operator by default. Separate read and write tools, and require explicit elevation for sensitive actions.

Make plans inspectable and actions reversible. Dry-run modes, previews, approval queues, transaction limits, idempotency keys, and action receipts let users and operators see what happened. A confirmation should not be a generic “continue”; it should name the operation, target, amount, and downstream effect.

Protect memory from cross-user and cross-task contamination. Tag memories with tenant, user, source, sensitivity, and expiry. Do not let an untrusted conversation write durable instructions that influence a later privileged task. Retrieval should enforce access at query time and again before a tool uses the result.

Test long-horizon failure. Inject conflicting instructions into documents, interrupt a plan midway, revoke a credential, return malformed tool data, and simulate a downstream partial failure. The agent should stop or degrade safely, not improvise a broader privilege.

Inference security and observability

Inference security requires telemetry that can connect a request to an identity, policy decision, model version, retrieved sources, tool calls, and final outcome. Use correlation IDs across gateway, retrieval, model, policy, and downstream services. Record hashes or references where raw content is too sensitive.

Alert on security signals: repeated policy probing, unusual token or query volume, extraction-like sampling, cross-tenant access denials, tool authorization failures, unexpected egress, prompt patterns associated with injection, and sudden changes in output destinations. Combine signals rather than blocking on a single keyword that attackers can evade and legitimate users can trigger.

Incident responders need a freeze path. Be able to disable a tool, revoke an agent credential, route traffic to a restricted model, suspend retraining, quarantine a model version, or block a retrieval source without taking the whole product offline. Exercise these controls and record how long they take.

Preserve forensic evidence with retention and privacy controls. Security logs should support reconstruction of the attack while limiting access to sensitive prompts and documents. Access to AI logs is itself a privileged operation and should be audited.

Secure evaluation and red teaming

Security evaluation should be continuous and release-specific. Build suites for direct and indirect prompt injection, data exfiltration, privilege escalation, model extraction, malicious files, supply-chain tampering, unsafe tool calls, and cross-tenant isolation. Include benign near-misses so defenses are measured for usability as well as attack resistance.

Use both automated tests and human red teams. Automated suites provide repeatability and regression detection. Human attackers find composition failures, ambiguous authority, and workflows the test designer did not anticipate. Keep a sealed holdout set and rotate cases so the system is not trained to pass a public checklist.

Score complete trajectories. For an agent, the important result may be that it attempted a forbidden call even if a downstream firewall blocked it. For a retrieval system, measure whether a private document was selected or exposed, not only whether the final answer looked harmless. Record attack assumptions, success definitions, and residual risk.

Gate promotion on severity, not only on a single pass percentage. One critical cross-tenant leak may block a release despite hundreds of harmless test passes. Failed cases should create owners, remediation deadlines, and regression tests.

Incident response and recovery

Prepare playbooks for prompt injection, data leakage, poisoned data, extracted model behavior, compromised dependency, unauthorized tool action, and model tampering. Each playbook should identify the first containment step, evidence to preserve, people to notify, credentials to revoke, and safe fallback. Incident severity should reflect affected data, action irreversibility, scope, and exploitability.

Containment options include disabling a tool, rotating keys, blocking a tenant or source, pinning a prior model, restoring a trusted index, suspending feedback ingestion, and isolating a worker pool. Choose reversible controls first when they reduce exposure quickly, but do not preserve availability at the cost of continued exfiltration.

After containment, determine root cause across model, data, application, identity, and deployment layers. Add a minimal reproduction to the security suite, patch the authorization boundary, and verify that the fix survives paraphrase and multi-step variants. Do not “solve” an injection incident by adding a secret phrase to a system prompt.

Communicate precisely. State what was exposed or attempted, the time window, affected tenants or records, containment status, and remaining uncertainty. Avoid claiming that no data was accessed merely because logs show no successful response if the attacker could have used an unobserved side channel.

Security architecture patterns

These patterns are starting points, not universal templates. A read-only assistant may still require stronger controls if its documents contain regulated data. An agent may be low risk when tools are simulated and high risk when it can alter production. Recalculate the boundary whenever capability, data, or user population changes.

Secure deployment across the lifecycle

Security should begin before a model is selected. During design, document assets, trust boundaries, and abuse cases. During development, isolate credentials, scan dependencies, and test authorization independently of model output. During evaluation, attack the complete application and record residual risk. During release, verify artifact integrity, policy configuration, and rollback. During operation, monitor abuse and drift. During retirement, revoke access, delete or archive data according to policy, and remove stale endpoints.

Deployment pipelines should treat prompts, policy bundles, retrieval indexes, adapters, and tool definitions as release inputs. A weight-only approval is insufficient when a prompt change grants a new capability. Require versioned manifests and review the combined behavior.

Use progressive delivery for security-sensitive changes. Shadow tests can reveal unexpected tool proposals; canaries can limit blast radius; feature flags can disable a capability quickly. Ensure flags cannot be changed by the same compromised identity that operates the model.

Map dependencies and owners. If a provider changes a model, a vector index is rebuilt, or a connector changes permissions, identify which products need review. Security posture decays when the architecture diagram and ownership list stop matching the deployed system.

Security economics and practical prioritization

Prioritize controls by expected loss and reversibility, not by the novelty of the attack name. Strong authorization on a payment tool usually matters more than an elaborate prompt watermark. Tenant isolation and secret handling usually matter more than a perfect jailbreak classifier. Use threat scenarios to connect engineering work to consequence.

Track useful measures: time to revoke a tool, percentage of releases with signed artifacts, coverage of high-risk tools by trajectory tests, mean time to detect extraction patterns, number of unowned model endpoints, and age of unresolved critical findings. Metrics should trigger action and have denominators.

Security controls must be operable. A review queue that nobody staffs, a log nobody can search, or a kill switch nobody has exercised is a paper control. Budget testing, incident response, key rotation, and access reviews as part of running the AI product.

Common AI security mistakes

The most common mistake is treating natural language as authorization. Other failures include sending full prompts to unrestricted logs, granting agents broad user credentials, trusting retrieved documents as instructions, allowing model-generated URLs to control network requests, using mutable model tags in production, and training automatically on unreviewed feedback.

Teams also overfocus on jailbreak examples while ignoring ordinary data exposure, weak tenant isolation, vulnerable dependencies, and broken rollback. They may scan application code but not serialized model packages, container loaders, evaluation artifacts, or prompt templates. They may add a filter while leaving the tool API overprivileged.

Good security design is layered and boring: deterministic authorization, least privilege, provenance, signed artifacts, isolated execution, quotas, typed outputs, protected logs, adversarial tests, and a practiced response path. Models can assist with detection, but they should not be the sole judge of their own permission.

Connections to the wider AI security surface

Secure deployments often use model hosting controls for artifact protection, tenant isolation, gateway policy, and inference telemetry. AI infrastructure contributes network segmentation, compute isolation, key management, and supply-chain controls. Enterprise AI supplies the business context, owners, and risk tiers that determine which controls are required.

AI agents add tool, memory, and long-horizon risks. Large language models add prompt, context, extraction, and memorization considerations. AI safety evaluates harm and capability hazards; this page focuses on adversaries, boundaries, assets, and exploit paths.

A minimum viable security program

A small team can establish a credible baseline without waiting for a specialist organization. First, inventory models, data paths, tools, providers, and owners. Second, write threat scenarios for the highest-impact workflows. Third, remove broad credentials and put authorization in deterministic code. Fourth, protect logs, artifacts, indexes, and caches by tenant and sensitivity. Fifth, run injection, extraction, poisoning, and tool-abuse tests in the release pipeline. Sixth, exercise a freeze, revoke, and rollback procedure.

Then improve the loop. Add signed model and container artifacts, provenance-aware retraining, protected holdouts, anomaly detection, red-team coverage, and supplier review. Revisit the threat model whenever a system gains browsing, code execution, memory, external writes, new data, or a larger user population.

Document residual risk in concrete language: which attacks were tested, which were not, what controls reduce impact, and who owns the remaining exposure. Security is not the claim that an AI system cannot be attacked. It is the ability to make attacks harder, limit their authority, detect them, contain them, and recover without guessing.

Closing

AI security is application and platform security adapted to systems that learn, interpret untrusted content, and can act through tools. Threat-model assets and trust boundaries, protect data and artifacts, test adversarial behavior, constrain agents, secure inference, and keep deployment reversible. Prompt instructions can guide a model, but only deterministic authorization can grant permission.

Keep security distinct from safety, ethics, and governance while coordinating the evidence they need. A well-secured system can still produce harmful behavior; a well-reviewed policy can still leave an extraction endpoint exposed. The strongest programs connect these disciplines through explicit owners, versioned tests, auditable decisions, and incident practices that turn each failure into a new control and regression case.