AI infrastructure is the compute, memory, storage, networking, scheduling, and serving substrate that makes training and inference possible at useful cost and latency. Models and algorithms define what is learned; infrastructure defines whether a run finishes, whether tokens arrive within an SLO, and whether multi-tenant GPUs remain isolated. Peak FLOPs on a datasheet are rarely the binding constraint—bandwidth, memory capacity, collectives, and utilization usually are.
This guide owns the accelerator/memory/network stack, distributed training infrastructure, inference serving systems (batching, paging, KV cache), AI power and unit economics, and multi-tenant GPU isolation. It does not own model taxonomy and selection, chip microarchitecture encyclopedias, or generic cloud product catalogs. For representation learning and train/serve ML systems practice at the model layer, see deep learning. For tool-using control loops, see AI agents.
Training versus inference infrastructure priorities. Device and gateway placement is owned by edge AI
Capacity planning should maintain separate forecasts: training reservation calendars (weeks/months) versus inference peak-to-average ratios (minutes/hours). Using a single “GPU count” KPI hides the mismatch. Finance partners need both curves to avoid buying the wrong shape of capacity.
SLOs differ too. Training cares about job completion time and checkpoint integrity. Inference cares about availability and tail latency. Incident runbooks must not reuse generic CPU web-tier playbooks without accelerator-specific steps (fabric drains, MIG resets, NVLink errors).
Training optimizes for throughput of gradients over huge batches and long jobs: high bisection bandwidth, fast checkpoint storage, elasticity, and recovery from node failure. Inference optimizes for latency, concurrency, and cost per token under bursty traffic: efficient batching, KV cache management, right-sized instances, and rapid scale-out.
Shared clusters must not pretend these workloads are identical. A training job that saturates NVLink and a latency-sensitive inference service that needs stable tail latency conflict when colocated without isolation. Separate pools or hard partitions are common. When they share fabric, QoS and admission control become first-class.
Planning starts from workload shape: sequence lengths, model sizes, precision, target tokens/sec, and failure domains. Infrastructure without a workload model becomes a pile of GPUs waiting for tribal knowledge.
Accelerators and why bandwidth often beats peak FLOPs
Kernel autotuning and compiler stacks (XLA, Triton-class workflows, vendor libraries) change achieved performance as much as hardware generation. Pin versions in reproducible images. “Same GPU, different container” is a valid performance regression class.
Multi-instance GPU partitioning helps inference density for smaller models but can destroy bandwidth for large training jobs. Document which pools allow partitioning. Surprising MIG enablement is a classic footgun.
Accelerators (GPUs and specialized AI ASICs) deliver dense matrix throughput, but many layers are memory-bandwidth bound or communication bound. A kernel that cannot feed tensor cores from HBM will idle expensive ALUs. That is why roofline analysis—arithmetic intensity versus achieved bandwidth—matters more than brochure TFLOPs.
Precision formats (FP32, TF32, BF16, FP8, INT8) change both throughput and numerical behavior. Infrastructure teams must expose which formats are supported end-to-end: framework, kernels, interconnect reductions, and checkpoints. Mixed precision without loss scaling discipline becomes an outage generator.
Host-to-device transfer paths, PCIe generations, and CPU overhead still matter for data-heavy preprocessing and for small models where launch overhead dominates. Not every bottleneck lives on the accelerator die.
HBM and the memory wall
Fragmentation inside device memory allocators causes out-of-memory failures even when free bytes look sufficient. Serving stacks with paging reduce fragmentation; training frameworks need allocator arenas and careful caching. Capture OOM dumps with allocator traces, not only “CUDA OOM” screenshots.
Host RAM and NVMe are part of the memory hierarchy when offloading. Size them deliberately; thrashing to disk turns a slow job into a stuck job. Measure host memory bandwidth and PCIe saturation during offload experiments.
High Bandwidth Memory (HBM) capacity and bandwidth set hard limits on model size per device, batch size, and activation checkpointing strategy. The “memory wall” is the gap between compute growth and memory/bandwidth growth: more FLOPs help only if data arrives.
Techniques that buy effective memory—activation checkpointing, ZeRO-style optimizer sharding, tensor/pipeline parallelism, CPU offload, and quantization—are infrastructure-coupled algorithm choices. They move pressure onto interconnect, CPU RAM, or numerical risk. Treat them as capacity planning tools with measured overheads, not magic.
For inference, KV cache memory often dominates at long context. Paging KV to host memory or disk can extend context at latency cost. Capacity planning must include concurrent sessions × context × layers × precision, not just parameter bytes.
Interconnect fabrics and collective communication
Rail-optimized topologies and topology-aware scheduling can materially cut all-reduce time. If your scheduler is topology-blind, you will pay for high-end networking without receiving its benefits. Feed topology into placement scores.
Telemetry should distinguish compute time, communication time, and waiting time per step. Without that split, teams overbuy GPUs when they should fix fabric or data loading.
Multi-GPU and multi-node training rely on collectives: all-reduce, all-gather, reduce-scatter. Fabric choices—NVLink/NVSwitch domains, InfiniBand/RoCE, Ethernet—set latency and bandwidth for those collectives. Topology-aware placement matters: a “GPU job” scheduled across weak links becomes a congestion experiment.
NCCL-class libraries and their equivalents implement collectives; infrastructure must keep driver, firmware, and library versions coherent. Silent version skew produces mysterious hangs. Congestion control, adaptive routing, and isolation from storage traffic keep training jobs predictable.
Pipeline bubbles and stragglers are often network or I/O problems labeled as “PyTorch bugs.” Observability of collective times per rank is mandatory at scale.
Training storage and data pipeline bottlenecks
Local scratch versus shared parallel file systems versus object stores present different consistency and latency profiles. Checkpointing to object storage is elastic but needs multipart and retry discipline. Testing restore paths is as important as testing write paths.
Data lineage for training inputs intersects compliance. Infrastructure must support dataset version pins so runs are reproducible for audit—even when the physical bytes live on immutable object versions.
Training storage serves two masters: streaming training samples and checkpointing huge optimizer states. Throughput to GPUs can starve if decoding, shuffle, or object-store latency stalls the input pipeline. Overlap I/O with compute; measure GPU idle waiting on data.
Checkpoint frequency trades progress safety against I/O load. Asynchronous checkpointing and distributed checkpoint formats reduce stalls but need correctness testing. Disaster recovery objectives (RPO/RTO) for training runs belong in the design, not as an afterthought after a failed 30-day job.
Feature stores and lakehouses may feed training, but this page cares about the bandwidth and consistency of the path into accelerators—not the full ML platform catalog.
Cluster scheduling and utilization
Quota systems should encode accelerator type, memory class, and interconnect domain—not only “GPU count.” Two GPUs on different sides of a weak link are not equal to two GPUs in one NVLink domain for many training graphs.
Draining and maintenance need cordon semantics that finish or checkpoint jobs cleanly. Hard power pulls should be last resort. Practice game days.
Schedulers allocate accelerators, CPUs, memory, and sometimes topology. Bin-packing for utilization conflicts with packing for performance (keeping a job inside a high-bandwidth domain). Idle GPUs are expensive; fragmented allocations that force cross-switch traffic can be more expensive in wall time.
Queues need fair share, priority, and preemption policies that match organizational reality. Training research, continuous pretraining, and production inference should not silently steal from each other. Gang scheduling for multi-node jobs prevents partial allocations that deadlock.
Utilization metrics must distinguish “GPU busy” from “GPU productive.” A device at 100% SM utilization on a communication-wait pattern is not healthy. Track MFU (model FLOPs utilization) and goodput where possible.
Distributed training failures: stragglers and congestion
Partial failures—one NIC flaps while others live—produce especially nasty hangs. Health checks must include device-to-device bandwidth probes, not only ping. Automated isolation of bad nodes beats hoping NCCL timeouts are tuned perfectly.
Stragglers—slow ranks—cap synchronous training throughput. Causes include thermal throttling, noisy neighbors, degraded links, disk stalls, and uneven data shards. Detect with per-rank step time histograms. Mitigations include isolation, healthier fabrics, elastic strategies, and sometimes algorithmic asynchrony with care.
Congestion collapses collective bandwidth under incast or competing jobs. Separate storage and training fabrics when possible. Test failure injection: kill a node mid-run and verify resume from checkpoint within your RTO.
Distributed training algorithm families (data/tensor/pipeline parallelism) are covered at systems-learning altitude in deep learning; here the ownership is the cluster behavior that makes those strategies succeed or thrash.
Inference serving: batching, paging, and KV cache
Admission control protects tail latency: reject or queue when KV memory or batch slots are exhausted instead of accepting work that dooms everyone. Products often prefer clear 429/queue behavior over silent multi-second stalls.
Multi-model serving on one GPU needs memory planners that account for fragmentation and warm-up. Naive “load all adapters” strategies OOM in production after looking fine in staging with tiny traffic.
Inference servers batch requests to raise utilization without breaking latency SLOs. Continuous batching / iteration-level scheduling improves throughput for autoregressive decoding by admitting new sequences as others finish. Static batching is simpler and often worse under variable lengths.
KV cache stores attention keys/values per token per layer. It grows with context and concurrency. Paged attention-style memory managers reduce fragmentation. Prefix caching reuses KV for shared prompts (system prompts, RAG prefixes) when it is safe to do so across tenants—never across ACL boundaries.
Speculative decoding, quantization, and tensor parallelism are serving knobs with quality and complexity costs. Measure tokens/sec, time-to-first-token, and time-per-output-token at the percentiles your product promises—not only averages.
Multi-tenant networking and noisy neighbors
Rate-limit metadata and control-plane APIs too. Tenants can DoS shared control planes without touching GPU FLOPs. Observability tenants must not see other tenants’ prompts, weights, or job names.
Multi-tenant GPU clouds must prevent one tenant from starving another’s bandwidth, cache, or PCIe. Noisy neighbors appear as latency spikes and training slowdowns. Isolation tools include MIG-like partitioning, separate networks, cgroup limits, and admission control on oversubscribed fabrics.
Side channels and residual state in GPU memory are security concerns. Reliable teardown, memory scrubbing, and attestation belong in the tenancy story for sensitive weights and data.
Power, cooling, and facility constraints
Liquid cooling introduces leak detection and maintenance skills your org may lack. Factor training and vendor support into TCO. Air-cooled rooms with AI densification often hit thermal walls earlier than spreadsheets predict—validate with facility engineering, not only GPU TDP sums.
AI racks can draw tens of kilowatts. Power delivery, cooling (air vs liquid), and floor loading constrain how densely you can pack accelerators. Facility limits often bite before budget does. Plan for transient spikes and for maintenance windows that drain power domains safely.
Carbon and energy reporting increasingly matter to buyers. Track joules per training run and per million tokens as first-class metrics beside dollars.
Unit economics: dollars per token and dollars per training run
Include human on-call and page noise in operating cost if inference pages nightly. A cheaper GPU with unstable drivers can lose on total cost. Track cost of incidents beside cost of capacity.
For RAG-heavy products, retrieval and rerank infrastructure can rival generator cost. Attribute spend by stage so optimization targets the true dominant term—see RAG for retrieval-stage design, while this page accounts for the accelerators those stages may also consume.
Unit economics convert utilization into money. Dollars per million tokens should include accelerator time, memory capacity premiums, network egress, retrieval/tool costs if in-path, and idle reserved capacity. Dollars per training run include failed runs and checkpoint storage.
Reserved capacity lowers unit cost at the risk of idle waste; on-demand does the opposite. Spot/preemptible capacity fits fault-tolerant training with robust checkpointing; it is usually wrong for strict inference SLOs.
| Workload | Primary cost drivers | Common waste |
|---|---|---|
| Pretraining | Accelerator-hours, fabric, checkpoint I/O | Stragglers, idle from data stalls |
| Fine-tuning | Accelerator-hours, developer iteration | Overprovisioned GPUs for small jobs |
| Online inference | Concurrent capacity, KV memory, tail latency headroom | Low batch efficiency; oversized models |
Weight security and isolation boundaries
CI/CD for model artifacts should sign digests and verify at load time. Unsigned weights from “someone’s laptop” are an incident. Separate credentials for training data access versus inference-only roles.
Model weights are valuable assets. Protect them with encryption at rest, controlled decrypt in enclave or trusted boot paths where required, network policies that block exfiltration, and audit logs on export. Supply chain integrity for containers and CUDA stacks matters as much as application code.
Isolation boundaries separate tenants, environments (prod/stage), and privilege tiers (training data vs inference-only). Crossing boundaries for “temporary debugging” is a frequent incident root cause.
Edge versus datacenter placement
Edge fleets need signed OTA updates, rollback, and offline behavior when the mothership is unreachable. Datacenter mental models that assume always-on control planes fail outdoors. Budget for physical theft and tampering risks on edge appliances.
Edge placement reduces latency and can keep data local; datacenter placement maximizes model size and batching efficiency. Hybrid designs run small models or filters on-edge and escalate to centralized inference. Placement is constrained by power, connectivity, update mechanics, and physical security.
For vision fleets and camera topologies, see also computer vision deployment concerns; this page owns the shared accelerator and network economics of edge boxes versus datacenter pools.
Build versus buy: cloud, on-prem, and hybrid
Contractual accelerator availability matters: marketing “availability” is not capacity during scarcity. Negotiate and test burst capacity. Exit plans should include weight export and observability export—not only VM images.
Cloud buy accelerates access to scarce accelerators and managed serving. On-prem can win on steady utilization, data gravity, and specialized networking—if you can staff facilities and operations. Hybrid is common: burst training in cloud, steady inference on-prem, or the reverse.
Decision criteria: utilization forecast, data residency, capital vs operating expense, hiring plan for SRE/ML infra, and time-to-capacity. Spreadsheets that ignore engineering headcount lie.
Infrastructure observability for AI workloads
Trace inference requests across gateway → scheduler → worker → tokenizer → model → tools. Without distributed tracing, “slow chat” tickets bounce forever. Training jobs need step-level timelines exported to the same observability backend your SRE already trusts.
Observe accelerators (util, memory, throttling, ECC), collectives (time per op), storage (queue depth, throughput), network (RDMA retries, congestion), and serving (TTFT, TPOT, batch size, cache hit). Correlate with job IDs and model versions.
Alert on symptoms that predict brownouts: rising KV evictions, climbing all-reduce times, checkpoint lag, thermal throttle rates. Dashboards without action thresholds are decoration.
Change management for drivers, firmware, and collective libraries needs canary nodes. A cluster-wide upgrade that improves average throughput but tanks a critical inference pool is still a failure.
Infrastructure success looks boring: predictable step times, calm tail latency, recoverable failures, and cost curves that match forecasts. Heroic GPU counts without fabric, memory, and observability discipline produce expensive unpredictability. Build the substrate so model teams can iterate without inventing cluster archaeology each week.
As models and context lengths grow, revisit assumptions quarterly: KV cache sizing, fabric congestion points, checkpoint sizes, and power headroom. Yesterday’s architecture can be today’s bottleneck without any single component “failing.” Schedule capacity retrospectives after each major model generation bump.
When evaluating vendors, demand topology diagrams, noisy-neighbor test results, teardown/scrub guarantees, and exportable metrics. Brochure TFLOPs without those artifacts are not infrastructure evidence.
Reference architectures that actually get built
Three recurring shapes dominate practice. First, a dedicated training island: dense accelerators, high-bisection fabric, parallel file system or fast object checkpointing, and topology-aware scheduling—with inference forbidden except for eval jobs. Second, an inference island: heterogeneous instance sizes, autoscaling gateways, KV-aware schedulers, and strict tenancy—with training banned to protect latency. Third, a shared research pool with soft isolation where utilization matters more than hard SLOs; accept that research pools will be noisier.
Enterprises often add a fourth: a regulated enclave with encrypted storage, private networking, and attested nodes for sensitive weights and data. Enclaves cost utilization; they buy auditability. Do not pretend a shared research pool meets enclave requirements by renaming it.
Reference architectures should include failure domains: what happens when a top-of-rack switch dies, when a PDU maintenance window hits, when a cloud region loses a zone. If the document only shows the happy-path diagram, it is not an architecture—it is a marketing poster.
Capacity models you can defend in a budget review
Build bottoms-up models: tokens/day × cost/token for inference; successful accelerator-hours × $/hour for training; plus tax for idle headroom (often 20–40% for latency-critical inference). Tops-down “buy what the vendor recommends” models fail when product traffic is spiky or when training starts/stops with research cycles.
Sensitivity analysis matters. Show how cost moves with context length, batch efficiency, quantization, and cache hit rate. A 2× context increase can more than 2× KV memory and collapse concurrency. Executives need those levers labeled so product decisions (longer context) are visible as infrastructure decisions.
Include the cost of differentiation: custom interconnect, liquid cooling, or reserved scarce SKUs. Sometimes paying a premium for a managed scarce SKU beats missing a launch. Sometimes it is vanity. Force the comparison with dates and risks attached.
Security operations specific to AI clusters
Beyond weight encryption, watch for prompt and embedding stores that accumulate sensitive customer text on the same network as training data. Segment networks so inference logs cannot casually query training datasets. Apply DLP to egress from jump hosts that mount model buckets.
Supply chain attacks targeting CUDA libraries, Python wheels, and base images are high leverage. Pin digests, scan continuously, and verify signatures in admission controllers. A compromised base image on a GPU node is a cryptominer’s dream and an exfiltration platform.
Incident response needs GPU-specific forensics: capturing allocator state may be impossible after crash; preserve scheduler logs, NCCL debug dumps, and gateway traces early. Practice recovering a poisoned model registry entry—revoking digests and rotating credentials—before you need it.
Performance engineering loop
Adopt a continuous loop: profile → hypothesize bottleneck (compute, HBM, PCIe, collective, storage) → change one variable → measure MFU/goodput/tail latency → keep or revert. Randomly applying quantization, parallelism, or “turn on flash attention” without measurement creates folklore configs.
Keep golden workloads: a fixed training step benchmark and a fixed inference mix (short, long, tool-calling if applicable). Run them after every driver/firmware/framework upgrade. Golden workloads catch regressions that customer traffic will eventually find at 3 a.m.
Document the achieved roofline for your dominant kernels on your hardware. When a new model generation arrives, compare its arithmetic intensity to that roofline to predict whether you are about to become bandwidth-bound before you buy more FLOPs.
For teams also shipping retrieval-augmented products, remember that RAG rerankers and embedding services are first-class accelerator consumers in the unit-economics model—even when the “main” LLM is elsewhere. Attribute capacity to those services explicitly.
Checklist before scaling the next order of magnitude
Before multiplying accelerator count by ten, clear this checklist: topology-aware scheduling enabled and verified; collective telemetry dashboards live; checkpoint save/restore game-day passed; power and cooling headroom signed by facilities; tenancy scrub and noisy-neighbor tests recorded; inference admission control tested under KV exhaustion; cost model updated for the new concurrency and context targets; rollback plan for drivers and serving binaries rehearsed.
If any item is red, adding GPUs amplifies chaos. Many “we need more compute” escalations are really “we need better fabrics, cleaner data pipelines, or saner batching.” Revisit the roofline and the stage- lag charts before the purchase order.
Finally, align incentives: model teams rewarded only for quality may ignore unit economics; infra teams rewarded only for utilization may harm latency. Shared dashboards for quality, tail latency, and $/token keep the conversation honest. Infrastructure is a product consumed by model and application teams—give it SLIs, a roadmap, and a named owner.
When deep learning training recipes change—new parallelism strategies, new precision defaults—re-validate infrastructure assumptions the same week. Algorithm papers rarely list your Top-of-Rack as a dependency, but your wall-clock time does. Keep a short “infra impact” section in model launch templates so surprises surface early.
Treat accelerator scarcity as a product constraint visible to designers: prompt length defaults, retry policies, and feature flags that shed load gracefully. Infrastructure that cannot shed load becomes a pure scale problem; infrastructure that can shed load becomes an resilience advantage. Publish load-shed playbooks beside scale-up playbooks.
Keep a living inventory of SKUs, firmware, known-bad serial ranges, and cable plant. Hardware genealogy explains otherwise mysterious collective failures. When a rack misbehaves, genealogy shortens time-to-isolation more than another generic NCCL environment variable experiment.
Prefer boring excellence: versioned images, rehearsed restores, calm tails, and unit economics that survive contact with production. That is AI infrastructure done well—and it is a continuous operating practice, not a one-time cluster purchase.
References and further reading
- Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An insightful visual performance model for multicore architectures. CACM.
- Narayanan, D., et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM.
- Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention.
- NVIDIA. NCCL User Guide.
- Jouppi, N. P., et al. (related TPU/roofline systems literature) — use vendor architecture whitepapers for current HBM and interconnect figures.