Reinforcement learning (RL) trains decision policies by interacting with an environment and collecting rewards over time. Unlike one-shot supervised prediction, an RL agent chooses actions that change future states, so credit for outcomes is delayed and entangled with exploration. This guide owns Markov decision process (MDP) framing, exploration versus exploitation, conceptual policy and value learning, reward design and hacking, and evaluation under sequential risk. It is not a catalog of tool-calling product agents—that lives in AI agents—and it is not a full large-language-model alignment stack—that lives in large language models. Classical prediction and evaluation discipline remain in machine learning; deep function approximators deepen in deep learning and neural networks.
Use RL when the problem is sequential control under feedback. Educational materials such as OpenAI Spinning Up popularized algorithms; production RL still fails on reward misspecification. Prefer supervised ranking or contextual bandits when actions do not reshape long horizons, or when you cannot safely explore.
Sequential decisions versus one-shot prediction
Supervised learning maps a fixed input to a label or score. The training distribution is usually treated as given. RL maps a history of states and actions to a policy that keeps acting. Each action can change the next observation, the available actions, and the eventual return. That closed loop is the reason RL needs different evaluation, different failure modes, and different data collection discipline.
One-shot classifiers can still sit inside RL systems—as reward models, state estimators, or action priors—but the learning problem for the policy is sequential. Confusing the two produces demos that look strong offline and collapse online: the offline dataset never saw the states the deployed policy visits.
Product “agents” that call tools under budgets are sequential in an operational sense, yet most of their reliability work is permissioning, grounding, and termination—not MDP solvers. Keep that boundary sharp: orchestration and tool safety belong with AI agents; reward, returns, and exploration belong here.
When the horizon is one and the action does not change tomorrow’s feature distribution in a material way, start with supervised learning or a contextual bandit. Escalate to full RL only when delayed credit and state coupling are first-class.
States, actions, rewards, and returns
A state summarizes what the agent needs to decide. An action is a controllable choice. A reward is a scalar signal emitted by the environment (or a learned proxy) after transitions. The return is a discounted or finite-horizon sum of rewards. Policies map states (or histories) to actions or action distributions. Value functions estimate expected return from a state or state–action pair under a policy.
Discounting encodes impatience and numerical stability; finite horizons encode episode length. Neither replaces careful reward design. Sparse rewards make learning slow; dense shaped rewards can invent shortcuts. The engineering artifact is the reward contract: what is measured, when it is emitted, and how it can be gamed.
Partial observability means the true Markov state is hidden. Agents then act on observations or belief states. Sensors lag, UI events omit hidden inventory, and users withhold intent. Treating raw observations as Markov states is a common silent failure.
| Object | Question it answers | Typical bug |
|---|---|---|
| State / observation | What do I know now? | Missing confounders; non-Markov UI |
| Action | What can I change? | Infeasible or irreversible actions unmarked |
| Reward | What is locally scored? | Proxy that diverges from intent |
| Return | What is long-run success? | Myopic optimization of step reward |
| Policy | How do I choose? | Overfit to simulator quirks |
MDPs and partial observability
An MDP assumes the next state distribution depends only on the current state and action. That Markov property is a modeling choice. When it fails, you need memory, belief tracking, or a POMDP framing. In practice, engineers expand the observation with histories, recurrent encoders, or engineered features until approximate Markov behavior is good enough for the horizon they care about.
Transition dynamics may be known (planning), learned (model-based RL), or left implicit (model-free RL). Known dynamics enable search and constraint checking; unknown dynamics force interaction or offline logs. Safety-critical control often keeps an explicit dynamics model even when the policy is learned, so violations can be checked before execution.
Episodic tasks reset; continuing tasks do not. Resetting hides long-run instability. Continuing control of industrial processes must evaluate stationary cost and constraint violation rates, not only episodic scores.
Within artificial intelligence, RL is one learning methodology beside supervised and unsupervised families. It does not subsume planning, search, or classical control—it often combines with them.
Exploration versus exploitation
Exploitation uses the current best estimate of valuable actions. Exploration tries uncertain or seemingly suboptimal actions to improve that estimate. Without exploration, policies freeze early. With naive exploration, systems damage inventory, annoy users, or violate safety envelopes.
Strategies range from ε-greedy and Boltzmann exploration to optimism under uncertainty, posterior sampling, and intrinsic motivation bonuses. In high-stakes domains, exploration is constrained: safe sets, shielded actions, human approval gates, or offline datasets that already contain diversity.
Simulators make exploration cheap and biased. Real systems make exploration expensive and politically visible. Budget exploration like you budget incidents: measure regret, user harm proxies, and recovery time—not only cumulative reward on a training curve.
Off-policy learning reuses past data; on-policy learning needs fresh rollouts under the current policy. Mixing them carelessly creates distribution shift that looks like “the optimizer failed.”
Value methods and policy methods (conceptual)
Value-based methods learn estimates of expected return and derive actions by maximizing those estimates (for example, Q-learning style updates with function approximation). Policy-based methods directly parameterize a policy and ascend estimated returns (REINFORCE-style ideas, actor–critic combinations). Actor–critic systems learn both a policy and a critic that reduces variance of policy updates.
At product altitude, the choice is about stability, sample efficiency, and action space shape. Discrete small action sets often start with value methods. High-dimensional continuous control often favors policy gradients with careful baselines. Hybrid methods dominate modern deep RL stacks, but the conceptual split still guides debugging: is the critic wrong, or is the policy update too aggressive?
Function approximation with deep networks introduces nonstationarity: the data distribution depends on the evolving policy. Replay buffers, target networks, trust-region or clipped updates, and normalization tricks exist to tame that loop. They are engineering controls, not proof of convergence.
Imitation learning and inverse RL sit nearby: they lean on expert trajectories when rewards are hard to write. Mention them when reward design is the bottleneck; do not treat them as a full substitute for sequential evaluation.
Model-based versus model-free trade-offs
Model-free RL updates the policy or value from experience without an explicit world model. It is flexible and often simpler to start, but sample-hungry. Model-based RL learns or uses dynamics to plan, imagine rollouts, or regularize the policy. It can be sample-efficient and more interpretable when the model is trustworthy—and catastrophic when the model is wrong in the regions the planner exploits.
Hybrid designs learn short-horizon models, use model-predictive control around a learned policy, or train policies inside learned simulators. The decision criterion is model risk versus interaction cost. If a wrong model can invent unsafe plans, keep hard constraints outside the learned model.
Offline RL learns from fixed logs without further environment interaction. It is attractive for recommendation and operations data, and fragile to extrapolation: policies may prefer actions never adequately covered. Conservatism, uncertainty penalties, and coverage diagnostics are part of the method, not optional polish.
Reward design and reward hacking in game AI and other interactive loops
Reward hacking occurs when the agent maximizes the measured reward while violating the stakeholder’s intent. Classic patterns include oscillating to farm bonuses, exploiting simulator bugs, hiding failures outside the metric window, or collapsing to degenerate but high-scoring behaviors.
Design rewards as specifications under adversarial reading. Prefer rewards tied to verifiable events. Separate training proxies from launch acceptance metrics. Add constraint costs for safety envelopes. Log trajectories that achieve suspiciously high reward and inspect them like security incidents.
Learned reward models (including preference models used in LLM post-training) inherit annotator bias, distribution shift, and their own hacking surface: policies may optimize stylistic cues the reward model likes. Classical RL reward hacking and preference-model exploitation are cousins; this guide owns the sequential framing, while LLM-specific preference stacks stay with large language models.
Shaping rewards to guide exploration is legitimate when the shaping potential is sound; careless shaping creates policies that chase the shaping signal forever. Document shaping terms and ablate them before production.
Simulation, offline RL, and data hunger
Simulators provide volume and resets. Domain randomization and system identification reduce sim-to-real gaps but never erase them. Treat simulator fidelity as a product claim with validation suites on hardware or live traffic shadows.
Real-world RL is data hungry because informative failures are rare and costly. Hierarchical policies, demonstrations, and strong priors reduce the burden. Batch RL on historical logs needs explicit coverage checks: if a region of state–action space is empty, do not trust optimistic values there.
Robotics and industrial control amplify these issues; dedicated embodied stacks will own hardware loops. Here, keep the general lesson: interaction data is a scarce resource with safety externalities, not a free gradient source.
Synthetic experience from generative models can help if distributional fidelity is measured. Unverified synthetic rollouts are another form of model risk.
Evaluation: policies, regret, and safety constraints
Offline metrics on logged policies are insufficient when the evaluation policy differs. Use off-policy evaluation carefully, with confidence intervals that widen under poor overlap. Online evaluation needs canaries, kill switches, and constraint monitors—not only average return.
Regret measures cumulative shortfall versus a reference. In products, translate regret into business and safety units: lost revenue, excess energy, near misses. Slice by context: rare states, peak load, new user cohorts.
Safety constraints are first-class: collision rates, dosage limits, spend caps, toxic content rates. A policy that wins on reward while breaching constraints has failed. Constrained RL and shielding formalize this; operationally, enforce hard blocks in the action interface regardless of the learned policy’s proposal.
Statistical testing for sequential interventions must account for nonstationarity and exploration noise. Peeking at live curves without a pre-registered stopping rule recreates classic A/B malpractice.
Where RLHF sits relative to classical RL
Reinforcement learning from human feedback (RLHF) and related preference optimization apply RL-style updates (or RL-inspired objectives) to language-model policies using a reward model trained from comparisons. Conceptually it is RL on a bandit-like or short-horizon MDP over prompts and completions, with a learned reward and strong KL regularization toward a reference model.
What RLHF is not: a general solution to long-horizon embodied control, nor a replacement for classical MDP engineering in robotics. What classical RL is not: a full account of tokenizer training, pretraining corpora, or inference ops for LLMs. Cross-link, do not merge. Preference data quality, reward-model drift, and over-optimization are shared themes with reward hacking above; implementation details belong in the LLM guide and, for weight adaptation recipes, in fine-tuning (draft until published).
Generative AI products may use RL-style loops for alignment or for non-language control; keep the learning-methodology ownership here and the modality map there.
When bandits or supervised ranking are enough
Contextual bandits handle repeated decisions where the immediate reward is observed and the action does not materially reshape long-term state—ad creative selection, simple recommendations with myopic feedback, UI variant choice. They still need exploration discipline, but they avoid full credit assignment over long horizons.
Supervised ranking with logged feedback often beats premature RL on marketplaces: train on counterfactual-aware objectives, then explore with bandits. Full RL becomes attractive when actions change inventory, user state, or environment dynamics that feed back into future features.
If you cannot define a reward more honest than a supervised label, stay supervised. RL does not create missing preference information; it amplifies whatever signal you feed it.
Failure modes in real-world control
Common failures include: reward hacking; simulator overfitting; unsafe exploration; non-Markov sensors; covariate shift after policy deployment; brittle hierarchical handoffs; ignored constraints; silent off-policy extrapolation; and organizational failure to staff on-call for policy behavior.
Mitigations stack: conservative action sets, shadow mode, staged exploration budgets, constraint monitors, trajectory audits, simulator validation, and rollback to last known-safe policy. Treat policy releases like binary releases with ownership and runbooks.
Compared with supervised model drift, RL drift can be self-reinforcing: the policy changes the data it later trains on. Break that loop with held-out evaluation environments and frozen calibration sets.
Choosing RL deliberately
Adopt RL when (1) decisions are sequential with delayed credit, (2) you can instrument rewards and constraints honestly, (3) you can explore safely or learn offline with adequate coverage, and (4) simpler bandit or supervised baselines are measurably insufficient. Otherwise, invest in better labels, better ranking features, or better classical control.
Staff the loop: reward owners, simulation owners, safety reviewers, and on-call for live policy anomalies. Unowned rewards become unpaid debt.
Write a one-page decision record before the first online experiment: horizon length, action irreversibility, exploration budget, constraint list, baseline comparator, and kill criteria. If the record is vague, the problem is not ready for RL.
Credit assignment and horizons
Long horizons dilute credit: an action early in an episode may help or hurt many steps later. Temporal-difference methods bootstrap value estimates to propagate credit; Monte Carlo methods wait for returns. Both struggle when rewards are sparse. Options, skills, and hierarchical policies compress horizons by learning temporally extended actions, at the cost of another design surface.
Product horizons are often mis-specified. Optimizing seven-day retention can harm thirty-day trust. Align the return window with the business liability window, and keep secondary monitors on longer lags. Discount factors are not a substitute for that alignment discussion.
Multi-agent settings add nonstationarity: other learners change the environment. Competitive and cooperative games need different evaluation protocols than single-agent control. Even “single user + platform” systems can be multi-agent when other algorithms react to your policy.
Action interfaces and irreversibility
The action interface is a safety control. Discrete menus, rate limits, confirmation for irreversible writes, and typed schemas reduce the blast radius of a bad policy. Continuous torque commands on hardware need different guards than ranking slots on a webpage.
Map every action to reversibility and human escalation. Irreversible actions should require higher confidence, secondary checks, or human approval—concepts that also appear in tool-using agent products, but here they bind the MDP action set itself.
Mask illegal actions in the policy head rather than hoping the reward will discourage them. Masking is clearer than soft penalties when constraints are hard.
Worked sketches
Warehouse routing: state includes locations and inventory; actions assign routes; reward blends throughput and lateness penalties; constraints forbid unsafe speeds. Start with model-predictive control plus learned cost shaping before full model-free RL.
Newsletter send-time: often a contextual bandit; escalate to RL only if send actions change long-run engagement state in ways myopic rewards miss—and only with unsubscribe and complaint constraints.
LLM helpfulness fine-tuning: preference model + KL-regularized policy updates; evaluate with held-out prompts and safety suites; details in LLM and fine-tuning guides, sequential reward risk framed here.
Energy setpoint control: continuing task with comfort and equipment constraints; prioritize model-based or classical control with learned residual policies; evaluate constraint violation rates hourly, not only energy saved.
Instrumentation and on-call
Log states, actions, rewards, constraints, and policy versions with enough fidelity to replay incidents. Redact sensitive fields. Dashboards should show reward, constraint breaches, exploration rate, and distribution shift indicators together. A green average reward with rising constraint alarms is a page, not a win.
On-call runbooks: freeze exploration, pin previous policy, widen shields, and capture trajectories for reward review. Practice the freeze path. An RL system without a freeze path is an unowned experiment.
Version the reward function like code. Silent reward edits are silent objective changes. Require review when proxies change, just as you would for a loss function change in supervised training.
Anti-patterns
Calling every tool-using chatbot “reinforcement learning.” Optimizing a proxy without adversarial review. Exploring in production without shields. Reporting training curves without constraint rates. Training only in simulation without hardware or shadow validation. Treating RLHF papers as a robotics stack. Replacing ranking teams with an unmonitored online learner. Using sparse binary success as the only reward in a long-horizon task without curriculum or shaping review. Trusting offline Q-values in unsupported state regions.
Boundary map for neighboring guides
Machine learning owns general prediction and evaluation hygiene. Deep learning and neural networks own function approximators used inside critics and policies. AI agents own tool orchestration and permissioned product loops. Large language models own pretraining, tokenization, and LLM post-training recipes where RLHF appears as one technique. Draft topics such as supervised learning, robotics AI, and fine-tuning deepen adjacent slices without receiving links until published.
If a section starts explaining transformer blocks or RAG chunking, you have left this guide’s ownership. If a section starts explaining OAuth for tools, you have entered AI agents. Stay on sequential reward, exploration, and policy evaluation.
Closing
Reinforcement learning is sequential decision-making under reward: states, actions, returns, exploration, and the constant risk that the measured objective is not the intended one. Keep tool-agent product architecture in AI agents, keep LLM post-training mechanics in large language models, and keep generic prediction discipline in machine learning. Design rewards as contracts, evaluate constraints as hard gates, and escalate from supervised and bandit baselines only with evidence.