AI safety is the engineering practice of identifying hazards from learned systems, evaluating dangerous capabilities and failure modes, and deploying mitigations that reduce harm under realistic misuse and malfunction. It is hazard analysis for models that generalize—not a synonym for corporate ethics statements, and not identical to application security. Lab safety publications from OpenAI and Anthropic inform practice, but product hazard ownership stays local. This guide owns hazard analysis, safety evaluations, and mitigations at product altitude. Organizational program design belongs in AI governance. Exploit classes and appsec controls belong in AI security (draft). Normative ethics debates belong in AI ethics (draft). Model mechanics that safety constrains appear across large language models, generative AI, AI agents, and reinforcement learning.
Treat safety as a lifecycle: anticipate hazards, measure them, mitigate, monitor, and revise claims when evidence changes.
Safety as hazard analysis for learned systems
Hazard analysis asks what can go wrong, how severe it is, how likely it is under foreseeable use and misuse, and what controls reduce risk to an acceptable level. For learned systems, hazards include incorrect high-stakes advice, unsafe tool actions, privacy leakage, biased harms, and enabling malicious users. Unlike fixed software, models change behavior with prompts, retrieved context, and fine-tunes—so hazards must be reassessed when those inputs change.
Start from use cases and irreversible actions. A chatbot that drafts email is a different hazard profile than an agent that sends email. Map assets, adversaries, and affected populations. Write hazards in causal language: “model outputs credentials from context → account takeover,” not vague “AI risk.”
Acceptability thresholds are product and legal decisions. Safety engineering makes residual risk visible; it does not invent a universal safe level. Document who accepted which residual risks.
Within artificial intelligence systems, safety sits beside reliability and security. Reliability is “works as intended under distribution”; safety is “does not cause unacceptable harm when it fails or is abused.”
Threat modeling sessions should include model owners, security engineers, and domain experts. Skip any one and you miss hazards: security alone may miss overconfident medical advice; domain experts alone may miss injection paths.
Maintain a living hazard register with owners, suite links, and last-reviewed dates. Registers that live only in slide decks do not block unsafe releases.
Misuse, malfunction, and systemic risks
Misuse: capable users steer the system toward harm (scams, weapons assistance, non-consensual imagery). Malfunction: the system fails for benign users (wrong medical tone, biased denials, unsafe robotics commands). Systemic risks: many deployments interact—market herding, information ecosystems, or correlated failures across vendors.
Taxonomies help coverage but must not become paperwork theater. Prefer living hazard registers tied to eval suites and owners. Update registers when new modalities ship—image generation and multimodal AI add channels attackers use to bypass text-only filters.
| Class | Example | Typical control family |
|---|---|---|
| Misuse | Jailbreak for scam copy | Filters, rate limits, account trust |
| Malfunction | Confident false advice | Grounding, abstention, human review |
| Privacy leakage | Training or context exfiltration | Data minimization, canaries, ACLs |
| Agency harm | Unsafe tool call | Permissions, sandbox, confirmations |
| Systemic | Correlated outages/abuse | Diversity, monitoring, coordinated disclosure |
Dual-use capabilities require graduated access: what is open, what is gated, what is refused. Decisions should cite eval evidence, not marketing optimism.
Malfunction often dominates real user harm volume even when misuse dominates headlines. Invest in abstention, grounding, and calibration—not only jailbreak theater. Dialog products in conversational AI inherit the same refusal and escalation needs. Machine learning evaluation hygiene still applies: slice metrics, distribution shift, and cost-sensitive thresholds.
Systemic risk planning includes dependency maps: which internal products share a base model, which vendors, and what happens if a provider emergency-patches a model out from under you.
Supply chain and third-party models
Most products compose foundation models, classifiers, embeddings, and tools from multiple vendors. Each dependency imports residual risk. Track model digests, filter versions, and subprocessors. Require security and safety questionnaires proportional to irreversibility.
When a vendor updates a model silently, your suites may go stale overnight. Prefer APIs that allow pinning and notification. Re-run critical suites on pin changes before expanding traffic.
Open-source components need license and malware scanning plus safety evals—the “open” label is not a safety certificate. Fine-tuning open weights can remove refusals; treat self-hosted stacks as higher residual misuse risk unless you rebuild mitigations.
Privacy as a safety hazard
Leakage of training data, retrieved documents, or user conversations is both a privacy incident and a safety hazard. Canary strings, minimization, encryption, and ACL-aware retrieval reduce risk. Log redaction prevents secondary leaks through support tooling.
Regulated data classes (health, financial, children’s data) need stricter defaults: shorter retention, harder access, and stronger abstention when unsure. Coordinate with privacy counsel; this page owns the hazard framing, not legal interpretation.
Evaluation suites for dangerous capabilities
Safety evals measure whether models exhibit behaviors you refuse to ship: assistance on violent crimes, biological risk questions at prohibited depth, cyber offense help, self-harm encouragement, child exploitation content, and domain-specific harms (financial manipulation, medical overclaim).
Suites need versions, scoring rubrics, and adversarial coverage—paraphrases, multi-turn inducement, multimodal payloads, and tool-using trajectories. Static question lists rot. Red teams update suites; CI blocks promotions on regressions—same discipline as fine-tuning gates.
Capability evals differ from policy compliance evals. A model may be capable of a dangerous skill yet refuse it; both facts matter. Over-refusal that blocks benign content is also a product hazard and should be measured, not ignored.
Do not fabricate benchmark leaderboards. Cite methods you actually run. When using third-party suites, record versions and known blind spots.
Multi-turn evals matter more than single prompts. Attackers escalate gradually, reframe refusals, and use roleplay. Suites that only test one-shot “how do I build a bomb” miss the real distribution of abuse.
Tool-using evals must include attempted privilege escalation: requests to dump secrets, disable logging, or email external parties. Score the whole trajectory, not only the final natural-language answer.
Children and vulnerable-user scenarios deserve dedicated cases with conservative thresholds. If your product is not designed for minors, enforce age gates in the application—model policy alone is insufficient.
Mitigations: filters, refusal, oversight, sandboxing
Layer mitigations: training-time preference shaping, inference-time filters, refusal policies, retrieval allowlists, tool permissioning, sandboxes, rate limits, and human oversight for high-impact actions. No single layer is enough. Prompt-only refusals fail under injection—see prompt engineering.
Filters classify inputs/outputs; they err. Tune thresholds with cost of false positives versus false negatives by domain. Oversight queues need SLAs; infinite “pending review” is an availability failure that pushes users to unsafe workarounds.
Sandboxing and least privilege are decisive for agents: the model proposing an action is not authorization. Enforce authorization in deterministic code paths.
Rollback and kill switches are mitigations. If a fine-tune weakens refusals, pin the previous adapter. Safety without rollback is theater.
Retrieval allowlists and document ACLs prevent “the model said so” from bypassing permissions—coordinate with RAG without turning this page into a retrieval encyclopedia. Safety owns the hazard of grounded-but-unauthorized answers.
Rate limits and anomaly detection slow automated abuse. Pair them with account reputation so researchers and power users are not collateral damage without appeal paths.
Human oversight should be targeted: review irreversible actions and high-severity classifier margins, not every token. Reviewer burnout creates rubber-stamping, which is worse than no review.
Alignment techniques at product altitude
Alignment here means steering model behavior toward specified policies and human preferences under evaluation—not solving philosophy. Techniques include supervised refusal examples, preference optimization, reinforcement learning from human or AI feedback (conceptual framing in RL and LLM guides), constitutional-style principle prompts, and tool policies.
Product altitude means: pick techniques you can evaluate and operate. Exotic research methods without suites and owners do not reduce shipped risk. Document intended policies as versioned artifacts prompts and trainers implement.
Alignment can trade off utility. Measure both. A model that refuses everything is “safe” only in a vacuous sense.
Preference data for safety must include hard benign cases near the decision boundary—otherwise models over-refuse and users route around the product. Annotator guidelines should separate “disallowed” from “uncomfortable but allowed.”
When policies change, retrain or reprompt and re-run suites. Silent policy edits without eval deltas recreate the fine-tuning safety regression problem.
Monitoring after deployment
Pre-launch suites miss novel attacks and distribution shift. Monitor: jailbreak success proxies, user reports, classifier trip rates, tool denials, hallucination complaints, and demographic slice disparities you are permitted to measure. Sample production traffic into review with privacy controls.
Incident response playbooks: severity scale, freeze generation or tools, notify stakeholders, patch filters, add eval cases, and communicate residual risk. Time-to-patch is a safety metric.
Feedback loops must not silently train on unredacted harmful content. Separating safety review corpora from training corpora prevents reinforcing the attacks you just handled.
Cross-product correlation: if multiple surfaces share a base model, a jailbreak on one may transfer. Share sealed findings across surfaces under need-to-know.
User reporting UX should be one click from harmful outputs, with categories that map to your hazard register. Unclassified “thumbs down” is weak safety signal.
Limits of safety claims
Safety claims should be scoped: model version, suite version, known residual risks, and out-of-scope uses. “Aligned” and “safe AGI” marketing language is not an engineering claim. Prefer “evaluated against suite X; residual risks Y; refused use cases Z.”
Open weights change residual risk: filters may be stripped. Closed APIs change residual risk: providers update models under you. Contract for notice and eval access where stakes require it.
Absence of evidence is not evidence of absence. Red-team coverage is always incomplete; say so.
Comparative claims (“safer than model B”) require identical suites and comparable sampling. Marketing charts without methods are not safety evidence.
Intersection with security and governance without merging them
Security focuses on confidentiality, integrity, availability, and exploitability of systems—prompt injection as an appsec issue, model theft, supply chain. Safety focuses on harm from model behavior and capability. They overlap on injection and abuse but differ in goals and tests.
Governance focuses on policies, roles, audits, and accountability across the organization. Safety engineering produces evidence those programs consume. Do not replace hazard evals with committee meetings, and do not replace committees with model cards alone.
Ethics asks what ought to be valued. Safety asks how to measure and mitigate hazards given stated policies. Keep the references clear when linking drafts later.
Shared rituals help: joint incident reviews, shared severity scales, and clear RACI charts. Shared ownership of every control helps no one.
Domain overlays
Healthcare, finance, children, elections, and critical infrastructure raise severity. Domain overlays add mandatory abstention, human-in-the-loop, and logging requirements. Reuse general suites but add domain cases. Do not assume a general chatbot suite covers biotech and clinical decision support.
Embodied and agentic systems raise physical and financial irreversibility. Confirmations, simulations, and spending caps are safety controls as much as UX features.
Education products need integrity controls against cheating assistance where institutional policies require them—and clarity for users about what is allowed. Ambiguity pushes risk onto students and teachers.
Worked sketches
Consumer LLM app: refusal training + output filters + injection tests + privacy redaction + kill switch; publish scoped model card.
Enterprise agent: tool allowlists, dry-run mode, dual control for payments, trajectory logging, weekly red-team on new tools.
Image product: prompt and output classifiers, provenance manifests, rate limits, CSAM pipelines, brand dual review—as detailed under image generation, owned here as hazard class coverage.
Organizational anti-patterns
Safety as a launch checkbox. No owner for the hazard register. Suites that never update. Filters without false-positive budgets. Agents with broad credentials. Fine-tunes without safety deltas. Claiming zero residual risk. Conflating ethics posters with eval evidence. Hiding incidents to protect marketing.
Building a minimal safety program for a product team
Even without a large trust-and-safety org, a product team can run a minimal viable safety loop: (1) write a one-page hazard register for the launch surface; (2) attach a versioned eval suite with clear fail criteria; (3) implement at least two mitigation layers for each top hazard; (4) assign an on-call owner for spikes in jailbreaks or harm reports; (5) schedule a monthly red-team hour that adds at least three new cases to the suite.
Scale the program with usage and irreversibility. A student demo and a payments agent should not share the same residual risk appetite. When irreversibility increases, add deterministic authorization and human approval before adding more model cleverness.
Vendor questionnaires should ask for suite methodology, update cadence, data retention, and abuse response times—not only for marketing PDFs. If a vendor cannot describe how refusals are tested, assume they are untested.
Document residual risk acceptances with dates and names. When staff turn over, the next owner needs to know why a control was weaker than ideal—not rediscover it via an incident.
Budget red-team time as a percentage of model change velocity. Teams that ship weekly adapters without weekly safety deltas accumulate invisible debt until a public failure collects it.
Finally, connect safety metrics to executive reporting without theater: show suite pass rates, severe incident counts with denominators, mean time to mitigate, and open residual risks accepted this quarter. Numbers that never reach decision-makers do not constrain launches.
When those metrics are visible, product managers start asking for safer designs earlier—before the model is chosen, not after the press cycle. That early constraint is the highest-leverage safety intervention most teams can afford.
Measurement quality and gaming
Safety metrics can be gamed: narrow suites, teaching to the test, or moving disallowed content into modalities you do not score. Adversarial diversity and sealed holdouts reduce gaming. Rotate a portion of red-team prompts out of the training-visible set.
Report rates with denominators: refusals per thousand sessions, severe incidents per million tool calls. Absolute counts without exposure hide risk as traffic grows.
Pair quantitative dashboards with qualitative review of severe cases. A single high-severity miss can outweigh a green average.
Research versus product safety—lab mandates differ; see research labs
Research explores frontiers of alignment and catastrophic risk. Product safety ships scoped controls for today’s systems. Both matter; conflating them produces either paralysis or reckless deployment. Product teams should track research for early warnings without waiting for settled theory before filtering scams and leaking secrets.
When research claims large capability jumps, re-run your dangerous-capability suites on the new model before enabling features—especially browsing, code execution, and long-horizon agents.
Boundary map
Governance owns programs and accountability. Security owns exploitability and appsec. Ethics owns normative debate. LLM/RL/agents/image guides own mechanisms that create hazards. This page owns hazard analysis, safety evals, mitigations, monitoring, and honest limits of claims for learned systems.
Closing
AI safety is disciplined hazard management for models that generalize: classify misuse and malfunction, evaluate dangerous capabilities, layer mitigations, monitor after deploy, and state residual risk without hype. Keep governance, security, and ethics adjacent—linked in intent, separate in ownership.
Ship only what you can evaluate and freeze. Everything else is an experiment that users experience as a product.
Make the boring artifacts real: a named hazard register, a versioned suite in CI, layered mitigations with owners, a kill switch that works, and incident reviews that add cases within a week. Those artifacts—not slogans—are what turn AI safety from a slide into an operable practice across LLMs, agents, generators, and multimodal stacks.