Autonomous Mobility: Perception, Planning, Control, and Safety

Autonomous mobility is the engineering of vehicles that perceive a changing world, estimate their position, predict what nearby actors may do, plan a safe motion, and control the vehicle within a defined operational design domain. It is a closed-loop stack, not a single model and not a synonym for “self-driving.” A production system must also explain where it may operate, what happens when a sensor or compute path degrades, how safety evidence is assembled, and how software changes are validated.

This guide owns the autonomous-vehicle stack: sensing and fusion, localization, mapping, prediction, planning, control, ODD definition, simulation, validation, fallback, and safety cases. It depends on computer vision for visual perception methods, robotics AI for embodied systems and control context, edge AI for constrained onboard inference, reinforcement learning for selected policy-learning methods, AI safety for hazard reasoning, and AI chips for compute constraints. It does not replace those domains.

Define autonomy by an ODD

An operational design domain is the boundary of intended operation. It should name geography, road or route class, speed range, weather, lighting, traffic rules, map freshness, connectivity assumptions, vehicle condition, and permitted passenger or cargo context. “Works in cities” is not an ODD. “Mapped urban arterials within a service area, daylight or street-lit night, no flooding, below a defined precipitation rate, and speeds below a stated limit” is testable.

ODD boundaries are technical controls and product commitments. The vehicle needs a reliable way to estimate whether it remains inside them, including uncertainty. If localization confidence falls, a map is stale, or weather exceeds the validated envelope, the system should transition to a documented degraded mode rather than quietly operating beyond evidence.

ODD expansion should be incremental. Add one dimension at a time where possible, such as a new road class or weather condition, and identify interactions between dimensions. A model validated in daylight and rain separately may still fail at night in rain. Record exclusions as carefully as inclusions because exclusions define when fallback must engage.

Build a time-aligned sensor picture

Autonomous mobility begins with sensors, but sensor count is not sensor understanding. Cameras provide rich appearance and semantics; lidar contributes geometric structure; radar offers velocity and robustness through some weather; inertial and wheel measurements support motion estimation; ultrasonics can cover close-range zones. Each has failure modes, calibration requirements, latency, and blind spots.

Fusion must account for time and coordinate frames. A camera image captured 100 milliseconds ago and a lidar point cloud captured now do not describe the same world when a cyclist is moving across the path. Timestamp at acquisition, synchronize clocks, measure transport delay, and propagate uncertainty. Validate extrinsic calibration after maintenance, vibration, temperature changes, and sensor replacement.

Redundancy is not simply duplicating the same detector. Independent modalities and physically separated channels can fail differently. A second camera on the same dirty mount does not provide the same assurance as an independent range or velocity signal. Analyze common-cause failures such as power loss, shared software, blocked field of view, network congestion, and environmental conditions that defeat multiple sensors together.

Perception must produce uncertainty

Perception converts raw streams into tracked objects, lanes or route boundaries, free space, traffic controls, road surface, and scene context. The output should include confidence, age, provenance, and quality indicators rather than only a class label and bounding box. Downstream planning needs to know whether an object is confidently measured, extrapolated, occluded, or merely suspected.

Track management matters more than a single-frame score. Objects appear, disappear behind buses, split because of detector noise, and reappear with changed identity. Trackers should handle occlusion, merge and split events, uncertainty growth, and impossible motion. Evaluate rare but consequential cases such as a pedestrian partly hidden by a parked vehicle or a small object in glare.

Use slice-based evaluation. Aggregate precision and recall can hide failures by lighting, road geometry, object size, vulnerable-road-user type, sensor combination, region, and speed. Maintain challenge sets for construction zones, emergency vehicles, unusual cargo, fallen objects, and ambiguous right-of-way. Computer vision methods help structure these metrics, but mobility adds temporal and action consequences.

Localize against a changing world

Localization estimates the vehicle pose in a world frame. GNSS can be degraded by urban canyons, multipath, tunnels, or spoofing. Inertial integration drifts. Map matching can stabilize pose but inherits map errors. Visual and lidar odometry can bridge gaps but degrade with poor texture, repeated structures, dust, rain, or dynamic scenes.

Use multiple signals with explicit health checks. A filter should not hide disagreement by producing a smooth but wrong pose. Detect residual growth, jump discontinuities, impossible acceleration, and disagreement between independent estimators. When confidence drops, constrain speed and maneuver complexity or transition to a minimal-risk condition.

Maps are operational data, not immutable scenery. Roadworks, temporary lane closures, changed signals, parked vehicles, and new barriers can invalidate assumptions. Separate long-lived geometry from dynamic observations, version map releases, measure freshness, and define how the vehicle behaves when the map and sensors disagree.

Maps encode behavior as well as geometry

High-definition maps may contain lanes, boundaries, traffic controls, topology, elevation, preferred routes, and semantic attributes. Their value depends on accuracy and update latency. A precise map that is stale can be more dangerous than a coarser map whose uncertainty is honestly represented.

Map production needs provenance, review, change detection, and rollback. When a source proposes a new lane boundary, do not promote it directly into a safety-critical release. Compare against independent observations, inspect conflict slices, and stage the release in simulation and controlled operations.

Do not overfit the autonomy stack to mapped regularity. Perception should detect unexpected obstacles and road changes, while planning should remain conservative when a map element is absent, contradictory, or low confidence. The map is a prior, not permission to ignore the live scene.

Predict many futures, not one story

Prediction estimates how other road users may move. Drivers, cyclists, pedestrians, animals, and vehicles at intersections have different action spaces and social cues. A prediction module should represent multiple plausible futures with probabilities or uncertainty, not produce a single confident trajectory that hides ambiguity.

Behavior depends on interaction. A vehicle’s planned movement changes what a pedestrian or driver does next. Prediction and planning therefore form a coupled loop. A plausible forecast under the current plan may become wrong after the vehicle accelerates, signals, yields, or blocks a gap.

Evaluate calibration as well as displacement error. A system that is usually close but overconfident on rare crossing behavior can be less safe than one that is slightly less precise but signals uncertainty. Test occlusion, intent ambiguity, unusual yielding, and multi-agent interactions rather than only straight-lane trajectories.

Planning balances rules, comfort, and safety

Planning turns the route and scene into a maneuver or trajectory. A route planner chooses roads or lanes; a behavior planner chooses yield, stop, merge, follow, or overtake decisions; a motion planner generates a dynamically feasible path and speed profile. Keep these levels distinct enough to inspect their assumptions and failure states.

Hard constraints include collision avoidance, vehicle dynamics, legal boundaries, and stopping capability. Soft objectives include progress, comfort, energy, passenger preference, and social predictability. When constraints conflict, safety and minimal-risk behavior must dominate. A planner that maximizes progress by repeatedly accepting narrow margins is not robust autonomy.

Uncertainty should affect margins and behavior. Increase following distance when object state is uncertain, reduce speed when localization is weak, and avoid maneuvers that require precise predictions when several futures remain plausible. A graceful planner can choose to wait, pull over, or request assistance; it does not need a clever maneuver for every scene.

Control closes the loop under real dynamics

Control converts a target path into steering, throttle, and braking while accounting for vehicle dynamics, actuator limits, latency, tire or surface conditions, and passenger comfort. Controllers need measured feedback and a clear relationship to the planner’s assumptions. A perfect trajectory is irrelevant if the vehicle cannot track it within the available stopping distance.

Model predictive control, geometric tracking, and robust feedback methods can be combined with learned residuals, but learned corrections should remain bounded and monitored. Keep independent safety constraints around acceleration, jerk, steering rate, and braking. Verify behavior during actuator saturation, sensor dropout, low friction, and delayed commands.

Control health is part of autonomy health. Monitor tracking error, oscillation, command age, actuator response, and disagreement between commanded and measured motion. A planner may be healthy while a steering actuator or brake channel is not.

Make fallback behavior concrete

Fallback is not a vague promise that a human can intervene. Define what happens when a sensor fails, localization becomes uncertain, compute is overloaded, connectivity disappears, a planner cannot find a safe trajectory, or a control channel disagrees. The response may be a speed reduction, lane hold, controlled stop, pull-over, return-to-base, or remote assistance request depending on the ODD and vehicle.

Fallback transitions need detection thresholds, timing budgets, authority rules, and verified actuation. Ensure the system can reach the fallback state with the remaining energy, braking, steering, and compute resources. Test cascades: a blocked camera may raise perception uncertainty, which increases planner conservatism, which creates a traffic conflict, which requires a reliable stop.

Remote assistance should not be treated as instantaneous cognition. Communication can be delayed, unavailable, or ambiguous. The vehicle must remain safe while waiting, and the operator interface must expose scene history, uncertainty, map state, and the exact action authority being granted.

Use simulation as an evidence pipeline

Simulation is valuable because it allows dangerous, rare, and combinatorial scenarios to be repeated. It can test perception perturbations, sensor failures, route changes, interactions, and control limits before road testing. But simulation is not evidence merely because it produced a pass count.

Build scenarios from real incidents, near misses, disengagements, known blind spots, and ODD boundaries. Parameterize weather, lighting, actor intent, occlusion, latency, map freshness, and sensor noise. Use deterministic seeds when debugging and randomized distributions when measuring generalization. Keep scenario versions and expected safety properties with each run.

Validate simulation against the physical world. Compare distributions of object size, motion, illumination, sensor artifacts, traffic density, and control response. When simulation predicts improvement, confirm it on a held-out physical or replay set. Use robotics AI practices for embodied evaluation, but keep vehicle-specific route, traffic, and safety evidence explicit.

Replay is a bridge, not a shortcut. A log replay can test new planning or control logic against recorded scenes, but the new policy may take actions that would have changed the scene. Counterfactual evaluation needs a credible simulator, conservative assumptions, or controlled on-road validation.

Construct a safety case

A safety case is an organized argument supported by evidence that the system is acceptably safe for a defined context. It should connect hazards to mitigations, assumptions, verification activities, residual risk, and operational controls. “The model achieved 99% accuracy” is one measurement, not a safety case.

Start with hazards and misuse. Consider failure to detect, false detection, wrong localization, stale map, unsafe prediction, planner deadlock, actuator failure, cyber or communication loss, maintenance error, and operation outside the ODD. Assign responsibilities across onboard software, vehicle hardware, operator procedures, mapping, fleet operations, and service teams.

Evidence should be traceable to software and data versions. Link requirements to unit tests, simulation scenarios, closed-course tests, road tests, incident reviews, and monitoring results. Record exclusions and unresolved assumptions. A safety case must be updated when a model, sensor, map, route, vehicle component, or operational policy changes materially.

Validate with operational metrics

Measure more than miles without intervention. Track collision and near-collision rates, hard braking, minimum time-to-collision, disengagement causes, route completion, deadlock, comfort, localization loss, sensor health, fallback frequency, assistance requests, and recovery success. Normalize metrics by ODD slice and exposure so a fleet cannot hide a rare but severe failure in a large denominator.

Human review of incidents needs consistent taxonomy and protected evidence. Preserve synchronized sensor data, system decisions, predicted futures, planner candidates, control commands, and operator inputs subject to privacy and retention rules. Classify root cause rather than only the last visible symptom.

Release gates should include regression thresholds, critical scenario pass criteria, compute and latency budgets, and a rollback plan. Test staged rollout on a small fleet or constrained route. Software can pass offline tests while creating new interaction behavior in live traffic.

Respect onboard compute constraints

Autonomous mobility is an edge workload. Power, heat, vibration, weight, memory, bandwidth, and startup time constrain models and architectures. AI chips provide useful acceleration, but selecting hardware is only part of the design. Measure end-to-end latency, thermal throttling, memory pressure, scheduling interference, and behavior during partial accelerator failure.

Allocate budgets by safety relevance. A low-rate semantic model must not starve a fast obstacle path. Use graceful degradation, quantization or compression where validated, and watchdogs for missed deadlines. Cloud offload may support maps, fleet analytics, or non-critical assistance, but a vehicle should not depend on a round trip for an immediate collision-avoidance decision.

Edge AI provides deployment patterns for constrained inference and update management. Mobility adds strict timing, environmental exposure, and fallback obligations. Treat model update, hardware firmware, map release, and calibration as one change-control surface.

Manage learning and updates safely

Fleet data can reveal new edge cases, but collecting more data is not the same as improving safety. Curate incidents, near misses, rare ODD slices, and disagreement clusters. Protect personal information, blur or minimize unnecessary identities, and maintain lineage from an example to the model or rule that consumed it.

Use a staged lifecycle: collect, label, review, train, evaluate, simulate, closed-course test, limited deployment, monitor, and promote or roll back. Keep a known-good version available. When a release changes behavior in an interaction slice, require an explicit decision even if aggregate metrics improve.

Reinforcement learning can help with policy optimization in constrained simulation or offline settings, but exploration must not outrank operational safety. Reward design, distribution shift, and rare hazards need independent review.

Design for people and responsibility

Passengers, pedestrians, other drivers, remote operators, maintenance staff, and emergency responders all interact with the system. Human-machine interfaces should communicate mode, limitations, requested handoff, and current fallback state without implying more competence than the ODD supports.

Responsibility must be explicit. Product owners define service promises; safety engineers own hazard argumentation; perception, prediction, planning, and control teams own component evidence; fleet operations own route and maintenance constraints; operators own the authority and training assigned to them. Ambiguous ownership turns an incident into an argument instead of a correction.

Common failure modes in deployed fleets

Recurring failures include ODD creep (routes expand faster than evidence), map/version mismatch after construction, sensor occlusion treated as free space, overconfident prediction in rare interaction classes, planner oscillation at merge points, and silent thermal throttling that lengthens perception latency. Another failure is metric theater: celebrating miles without disengagements while under-sampling night rain, temporary traffic control, or emergency-vehicle interactions.

Operational remedies are procedural as much as algorithmic: freeze route expansion until scenario coverage catches up; require dual-source confirmation before clearing an occlusion; gate releases on latency budgets under heat soak; and assign a named owner for each high-severity disengagement class until it is closed with a linked fix.

Bound autonomy with evidence

Autonomous mobility matures when the stack is treated as a safety-critical feedback system rather than a collection of impressive demos. Specify the ODD, fuse time-aligned sensors, expose uncertainty, maintain trustworthy maps, predict multiple futures, plan conservatively, control within measured limits, define fallback, validate in simulation and reality, and keep a living safety case. The goal is not autonomy everywhere; it is dependable behavior within an honestly bounded domain.