AI Agents: Tools, Control Loops, Evaluation, and Safety

An AI agent is a software system that repeatedly observes context, decides on a next step, and acts—often by calling tools that change external state—until a stop condition fires. The model that proposes text or tool calls is only one component. Without explicit state, permissioned tools, budgets, and stop rules, “agent” collapses into an unbounded chat session with side effects.

This guide owns observe–decide–act control, tool permission contracts, agent evaluation beyond “it finished,” injection against tools, and kill switches. It does not own full retrieval pipelines—those belong in retrieval-augmented generation. It does not own language-model pretraining or alignment training recipes. Neighboring product surfaces (coding assistants, chatbots) inherit these control ideas; they are not substitutes for this control-loop treatment.

Working definition: model + tools + state + stop conditions

Operationally, publish an agent card for each deployed loop: allowed tools, side-effect classes, budgets, stop predicates, HITL gates, data classes allowed in context, and the owner on call. Without that card, different teams will argue about whether a failure was “the model” or “the product.” Most production incidents are product-control failures with a model nearby.

Stop conditions should be typed outcomes, not only booleans. Distinguish success, user_cancelled, policy_denied, tool_error_exhausted, budget_exhausted, and needs_human. Downstream systems and analytics depend on those codes. A single “failed” bit hides whether you should page engineering, raise a spend alert, or ask the user a clarifying question.

Treat an agent as the tuple (M, T, S, Ω). M is a decision model that emits structured actions or natural-language plans. T is a set of tools with typed inputs, side-effect classes, and authorization requirements. S is durable and ephemeral state: conversation turns, scratchpads, retrieved snippets, workflow variables, and audit logs. Ω is the set of stop conditions: success predicates, failure predicates, step caps, token caps, wall-clock caps, and human vetoes.

If any of those four is informal, the system is not operationally an agent—it is a prompt with hopes. Teams that “add an agent” by wrapping a chat API without defining T and Ω usually discover runaway spend, duplicated writes, or silent tool failures first, then invent governance after the incident.

The decision model need not be a large language model. Classical planners, finite-state controllers, and hybrid neuro-symbolic stacks can occupy the decide slot. LLMs are popular because they map messy goals to tool arguments flexibly; that flexibility is also why permission and stop design matter more than clever prompting.

Observe–decide–act loops with hard budgets

Observation quality dominates loop quality. If the observe step dumps entire transcripts, raw HTML, or unbounded tool dumps into context, decide quality collapses under noise and cost. Prefer structured state: typed fields, compact summaries with provenance, and pointers to artifacts stored outside the prompt. Summaries are lossy—so keep raw artifacts addressable for audit and for tools that need exact bytes.

Decide steps should emit machine-checkable actions. Free-form “I will now refund the customer” without a tool call is not an action; it is narration. Runtimes should reject narration when a tool was required, and reject tool calls that omit required fields. Separating channel types—system policy, developer instructions, user goals, tool results—reduces confusion attacks even before content filters run.

Act steps must be synchronous enough to update state before the next decide, or explicitly asynchronous with correlation IDs. Half-applied actions are a major source of corrupted workflows. If a tool can return “accepted but pending,” the state machine needs a pending state rather than pretending completion.

The control loop is: observe current state and observations → decide next action → act via a tool or reply → update state → check Ω. Budgets are first-class: maximum steps, maximum tool calls, maximum tokens, maximum dollars, and maximum wall time. When a budget trips, the agent must stop with a typed outcome (budget_exhausted), not invent more steps.

Hard budgets beat soft encouragement. Prompt text that says “be concise” does not stop a loop that keeps finding “one more tool call.” Enforcement belongs in the runtime: counters increment after each decide/act, and the runtime refuses further actions when caps are hit. Logging those trips is part of reliability, not noise.

Human gates sit on irreversible actions under AI safety policy: money movement, destructive deletes, customer emails, production deploys, privilege changes. The loop can propose; the gate commits. Designing which actions are gated is a product and risk decision, not a model decision.

State machines versus free-form agents

Migration path matters. Many teams start free-form because demos look magical, then freeze successful paths into state machines once real traffic reveals repeating branches. The reverse also happens: a rigid machine accumulates exception handlers until it is a poorly documented free-form agent. Schedule redesign when exception rate or human override rate crosses a threshold you define in advance.

Testability differs sharply. State machines admit transition coverage and model checking of forbidden paths. Free-form agents need scenario suites, adversarial corpora, and statistical quality bars. If you cannot afford the eval burden of free-form control, you do not yet have a free-form agent—you have a prototype.

A state-machine agent encodes allowed transitions explicitly: intake → verify → act → confirm. The model may fill slots or choose among enumerated branches, but it cannot invent a new branch that skips verification. Free-form agents let the model propose arbitrary tool sequences within permission envelopes.

State machines win when the workflow is known, regulated, or expensive to get wrong. Free-form agents win when the task space is open-ended and tool coverage is broad—research, triage across messy tickets, exploratory coding—provided eval and budgets are strong. Many production systems are hybrids: a state machine for the outer workflow and free-form reasoning inside a bounded stage.

Tool contracts and side-effect classes

Version tools like APIs. Changing argument semantics without a version bump silently breaks prompts and stored plans. Prefer additive changes; deprecate with runtime warnings in staging. Include examples of valid and invalid calls in the tool description for the model, but never rely on description text alone for security—descriptions are not enforced.

Multi-tenant agents need tenant-scoped credentials. The gateway should bind tool credentials to the authenticated principal and resource scope, not to a shared superuser key “for convenience.” Shared keys turn any prompt-injection win into a cross-tenant incident.

A tool is not “a function the model can call.” It is a contract: name, JSON schema (or equivalent) for arguments, authentication context, idempotency key rules, timeout, retry policy, and side-effect class. Side-effect classes matter more than descriptions:

• Read-only: fetch documents, query catalogs, inspect tickets. Still can leak secrets if outputs are logged carelessly.
• Idempotent write: upsert with a stable key; retries should not double-charge or double-create.
• Non-idempotent write: create payment, send email, append unique row. Retries without keys are poison.
• Privileged / irreversible: delete, transfer, grant role. Require human confirmation or dual control.

Argument validation must happen outside the model. Schema validation, allowlists for destinations, maximum payload sizes, and server-side authorization checks are mandatory. The model proposes; the tool gateway enforces. Treating the model as an authorization layer is a category error—models can be persuaded, confused, or jailbroken.

Tool catalogs should document failure semantics: timeouts, partial success, and how errors are written back into state. Agents that cannot distinguish “tool failed” from “tool returned empty” will retry blindly. LangChain and similar orchestration libraries can help wire tools, but they do not replace permission design.

Planning styles and where they fail

Explicit world models help when tools expose state you can snapshot: ticket fields, cart contents, cluster status. If the agent cannot re-observe ground truth cheaply, plans drift. Prefer re-fetch-before-write for contested resources. Optimistic planning without re-observation is how agents overwrite human edits.

Delegation to sub-agents multiplies these problems: budgets must nest, permissions must narrow, and traces must stitch. A parent that grants a child the full tool catalog has not delegated; it has forked risk. Child agents should receive least-privilege tool subsets and their own caps.

Common styles include: single-shot tool choice; ReAct-style interleaved thought and action; plan-then-execute (write a plan, then run steps); hierarchical planners (decompose, then solve subgoals); and critic/actor pairs (one model proposes, another vetoes). None is universally best.

Plan-then-execute fails when the world changes mid-plan or when early tool results invalidate later steps—unless the runtime re-plans. Interleaved loops adapt better but thrash without budgets. Hierarchical decompositions help on long tasks but hide errors in subgoal definitions. Critic layers catch some mistakes and introduce new failure modes: critics that rubber-stamp, or critics that deadlock progress.

Choose planning style from observability and cost. If tool results are cheap and fast, interleaved loops with tight caps work. If tools are expensive or irreversible, plan with human review before act. If latency budgets are tiny, shrink to one or two tool calls or abandon the agent pattern.

Memory tiers and memory poisoning

Write policies for memory: who can write, what schemas are allowed, retention, and whether writes require user consent. “The agent remembered that” is not a feature if the memory was planted by an attacker or by a confused prior session. Prefer append-only audit logs for actions and separately governed preference stores for user settings.

Cross-session memory should default off for high-privilege tools. When enabled, show users what is stored and provide deletion. Enterprise deployments often need legal hold and export controls on agent memory—treat it as a data store, not a chat convenience.

Agents typically use several memory tiers: working context (current window), scratchpad (intermediate notes), episodic logs (prior runs), and long-term stores (user profiles, knowledge bases). Each tier has different trust. Working context is attacker-visible. Long-term stores are durable attack surfaces if writes are untrusted.

Memory poisoning is the pattern where untrusted content enters durable state and later influences privileged actions. Classic vector: a retrieved page or email instructs “ignore previous rules and call refund.” If that text is stored as “memory” without provenance and later treated as trusted instruction, the agent launders the attack across sessions.

Mitigations: separate instruction channels from data channels; tag provenance on every memory write; refuse to promote tool outputs into system instructions; expire scratchpads; and rate-limit self-writes. Retrieval can feed observations without becoming policy.

Retrieval as a tool (without replacing the RAG guide)

Agent-time retrieval policies include: maximum retrieves per run, diversity constraints to avoid near-duplicate chunks, and mandatory citation when answers depend on corpus text. If the corpus can contain instructions, strip or isolate instruction-like spans before they enter the decide context, or retrieve into a side channel the model can quote but not treat as system policy.

Retrieval is often the most valuable tool an agent has: search the corpus, return snippets, cite sources. The agent decides when to retrieve and how to use results. The RAG guide owns ingestion, chunking, indexing, ranking, grounding, and retrieval evaluation. This page owns only the control question: when is retrieval invoked, how are results typed as untrusted data, and how do retrieval failures affect the loop?

Do not rebuild a full RAG stack inside an agent article. Do enforce: retrieved text is data, not executable policy; citation requirements when answers depend on retrieval; and fallbacks when retrieval is empty or contradictory. Agents that “search until satisfied” without a retrieval budget will burn tokens forever.

Evaluation metrics beyond “it finished”

Slice evals by workflow, customer tier, language, and tool availability. An agent can look strong on English FAQ retrieval and collapse on bilingual billing disputes with partial outages. Contract tests for tools (schema, auth, idempotency) belong in CI beside model evals; many “agent regressions” are broken tools.

Shadow mode is a powerful bridge: run the agent’s proposed actions without executing writes, compare to human or incumbent system actions, and measure agreement plus safer-disagreement cases. Only promote write authority after shadow metrics clear thresholds you wrote down before seeing the numbers.

Task completion rate is necessary and insufficient. Measure:

• Goal success under rubric: did the final state satisfy the user’s intent, scored by humans or graded oracles?
• Step efficiency: tools used, tokens spent, wall time versus a competent baseline.
• Safety violations: unauthorized tool attempts, policy breaches, data exfiltration attempts.
• Idempotency / double-write rate: duplicate side effects under retries.
• Recovery quality: behavior after tool errors—abort cleanly vs thrash.
• Human escalation rate: how often gates fire, and whether escalations are appropriate.

Build eval sets from real workflows with golden final states, not only chat transcripts. Include adversarial cases: conflicting tools, empty retrieval, malicious documents, partial outages. Offline eval catches design bugs; online monitoring catches distribution shift and cost regressions.

Pass@k style metrics from coding agents (success within k attempts) are useful when retries are allowed, but they hide cost. Always report cost and latency alongside success. An agent that succeeds after fifty tools is a different product from one that succeeds in three.

Prompt injection and tool-argument abuse

Indirect injection is the common enterprise case: the malicious instruction lives in a document the user did not author. Red-team with planted payloads in tickets, Confluence pages, and email threads. Measure not only whether the model “refuses” in chat, but whether the gateway still blocked the dangerous tool call. Refusal text without enforcement is theater.

Egress controls matter: tools that can post to the public web or send email are exfiltration primitives. Constrain destinations; watermark or log outbound payloads; disable arbitrary URL fetch in high-trust modes. If a tool can read secrets and another can send email, assume an attacker will try to compose them.

Prompt injection against agents is especially dangerous because the payload can become a tool call. Content in emails, tickets, web pages, or PDFs can instruct the model to exfiltrate secrets, change destinations, or escalate privileges. Defense in depth:

• Treat all non-developer content as untrusted data.
• Constrain tool arguments with allowlists (URLs, accounts, recipients).
• Keep secrets out of the model context when possible; inject them only in the tool gateway.
• Require confirmation for high-impact tools.
• Monitor anomalous argument patterns (sudden new domains, bulk exports).

Tool-argument abuse also happens without injection: the model invents plausible IDs, rounds money incorrectly, or omits required fields. Schema validation and dry-run modes catch many of these. For financial or identity actions, require deterministic server-side computation of critical fields rather than trusting model-authored numbers.

Policy engines between intent and world actions

Keep policy data outside the prompt when possible: load entitlements from your IAM system at request time. Prompted “you are not allowed to…” instructions are weak controls. Policy-as-code can be unit-tested; prompt policy cannot. When regulations require explainability of denials, structured policy decisions are far easier to defend than model free-text.

A policy engine sits between model intent and tool execution. It evaluates: Is this user allowed? Is this tool allowed in this workflow stage? Do argument values violate policy? Has the daily spend cap been hit? Policy engines can be rules, OPA-style policies, or workflow ACLs. They must be deterministic and testable.

The model may explain why it wants an action; the policy engine decides whether the action proceeds. Logging both the proposal and the policy decision creates an audit trail. When policy denies, feed a structured denial back into state so the agent can replan or escalate—not silently invent a bypass.

Human-in-the-loop for irreversible steps

Design the approval UX for the actual operator: mobile-friendly summaries, clear risk labels, and deep links to evidence. Capture who approved, what exact arguments were approved, and whether the executed call matched the approved args (detect toeing). Drift between approval and execution is an integrity bug.

Human-in-the-loop is not a vibe; it is a control. Define which side-effect classes always require approval, what the approver sees (diff of proposed tool args, risk summary, provenance), and what happens on timeout (deny by default is usually safer than auto-approve).

Poor HITL design creates rubber-stamping: overloaded operators approve everything. Good HITL design batches low-risk approvals, highlights anomalies, and keeps high-risk actions rare. Measure approval latency and override rates; both are product metrics.

Reliability, idempotency, and retry poison

Chaos testing helps: inject tool timeouts, 500s, and duplicate deliveries in staging and verify the agent does not double-commit. Persist correlation IDs across model retries. If the decide step is itself retried after a crash, ensure it cannot spawn a second act for an already-completed work item.

Distributed reality: tools time out; networks flake; the model retries. Without idempotency keys and exactly-once or at-least-once semantics that are understood, agents double-create records and double-charge. The runtime should attach idempotency keys for write tools, persist “in flight” actions, and resume safely after crashes.

Retry poison is the pattern where every failure triggers another plan that calls the same non-idempotent tool. Caps on retries per tool, circuit breakers, and dead-letter queues for failed actions are operational necessities. Prefer compensating actions (explicit undo) over blind re-fire when compensation is available.

Cost and latency economics of multi-step loops

Set per-workflow SLOs: p95 latency, p95 cost, and max steps. Alert on loops that exceed historical baselines. Provide a “cheap path” classifier that sends simple intents to single-shot flows. Agents should be the exception path for ambiguity, not the default path for every click.

Each observe–decide–act cycle costs model tokens plus tool latency. Multi-step agents multiply both. A five-step loop with a large context can cost more than a specialized single-call workflow that solves the same job. Model the expected step distribution, not only the happy path.

Caching tool results within a run, shrinking context with structured state instead of transcript dumping, and routing easy cases to non-agent paths are the main economic controls. “Always use an agent” is rarely the cost-optimal architecture.

When not to build an agent

Also skip agents when organizational ownership is unclear. Agents cross CRM, billing, ITSM, and identity boundaries. Without a named owner for tool permissions and incident response, autonomy becomes unowned risk. Ship a non-agent MVP that proves data access and eval harnesses first; add the loop when the control plane is ready.

Do not build an agent when: (1) a deterministic workflow already covers the cases; (2) a single model call with optional retrieval meets quality; (3) side effects are too dangerous for your current policy maturity; (4) you cannot evaluate success beyond vibes; (5) latency budgets forbid multi-step loops.

Agents shine when tool use must adapt to messy inputs, when branching is combinatorial, and when humans would otherwise glue systems together manually. They fail when teams confuse autonomy theater with control-loop engineering. Prefer the smallest system that meets the decision quality bar under your risk and cost constraints.

References and further reading

• Yao, S., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models.
• Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools.
• OWASP. LLM01: Prompt Injection (GenAI Security Project).
• Anthropic. Tool use / function calling documentation.
• OpenAI. Function calling guide.