AI Chips and Accelerators: Architectures, Memory, and Software Stacks

AI chips and accelerators are silicon and packaging choices purpose-built for dense linear algebra, memory bandwidth, and interconnect that deep learning training and inference demand. This guide owns accelerator architectures (GPU, TPU, NPU, ASIC), on-chip and package memory, train-versus-infer silicon economics, software stack portability, and utilization realities—not full cluster networking and scheduling, which belong with AI infrastructure. Model-hosting platforms and ai-cloud catalogs change quickly; here the focus is what the die and its software enable, and where chip choice ends so facility and fabric design begin.

Peak FLOPs on a slide rarely predict wall-clock time. Memory bandwidth, on-chip SRAM, sparsity support, numeric formats, kernel libraries, and how well your model maps to the datapath determine achieved throughput. Procurement without that mapping buys expensive space heaters.

Why CPUs lose on dense tensor math

General-purpose CPUs excel at branchy, latency-sensitive code with strong single-thread performance and large caches for diverse workloads. Dense neural network layers are different: enormous, regular matrix multiply-accumulate patterns with high arithmetic intensity if data arrives fast enough. CPUs have few massive SIMD lanes relative to dedicated accelerators and spend energy on speculation, complex cores, and cache hierarchies optimized for irregular access.

Tensor cores and matrix engines on GPUs, systolic arrays on TPUs, and MAC arrays on NPUs amortize control overhead across thousands of parallel ops. A CPU can run small models or preprocessing, but training multi-billion-parameter models or serving high-concurrency inference on CPU alone usually fails economically unless models are tiny or batch sizes are minimal.

That said, CPUs remain essential hosts: dataloader workers, feature preprocessing, orchestration, and small-model fallbacks. The question is placement, not religion. Edge devices often run hybrid SoCs with NPUs plus CPU clusters—see edge AI for deployment constraints; this page explains why the NPU exists beside the CPU.

Accelerator datapaths and sparsity

Accelerators implement a datapath from high-bandwidth memory (HBM or LPDDR) through on-chip buffers into array processors that execute fused multiply-add at high utilization. Effective designs keep arrays fed: double buffering, tiling, and fusion of ops (conv + bias + activation) reduce round trips to DRAM.

Sparsity—structured or unstructured zero patterns—can skip work if hardware and kernels exploit it. Structured sparsity (block or N:M patterns) maps cleanly to some generations; unstructured sparsity often needs metadata overhead that eats gains unless models are trained for it. Do not assume “50% zeros” equals “2× speed” without profiling on your exact chip and framework version.

Specialized formats (FP8, INT8, INT4, block floating) increase ops per watt when numerics remain stable for your task. Training may accept BF16/FP8 with loss scaling and careful accumulation; inference may quantize weights and activations with calibration or QAT. Format support is chip-specific and library-specific—portable PyTorch code is not portable performance.

Datapath depth also determines operator coverage. Some NPUs accelerate convolutions and matmuls well but fall back to CPU for exotic ops (custom activations, dynamic shapes), collapsing end-to-end throughput. Read operator coverage tables for your target graph, not only peak TFLOPs.

SRAM versus HBM: the memory hierarchy on die

Accelerators sit in a memory hierarchy: registers and SRAM scratchpads inside compute arrays, larger on-chip caches, then HBM or GDDR/LPDDR off-chip. Training large models is often HBM-capacity-bound: parameters, optimizer states, activations, and gradients must fit or be sharded across devices. Inference at long context is often KV-cache-bound in a similar way.

SRAM is fast and expensive in area; HBM is high bandwidth but limited capacity per stack. Architectures like Google’s TPU generations emphasize large systolic arrays fed by HBM with software-managed memory placement; GPUs balance SM counts, L2, and HBM channels. NPUs in phones prioritize watt-level LPDDR and on-chip SRAM for common vision and speech graphs.

Roofline analysis still applies: plot achieved FLOPs versus arithmetic intensity for your kernels. Low-intensity kernels (layer norms, reductions, small MLPs) are memory-bound; high-intensity large matmuls may compute-bound until they become communication-bound across chips.

Techniques that shrink memory footprint—activation checkpointing, optimizer sharding, weight quantization—move pressure between HBM, host RAM, and network. They are co-design levers with chip choice, not free lunch.

Interconnect within package and across GPUs

Single-die limits yield multi-die packages and multi-GPU domains. NVLink-class bridges, Infinity Fabric, and chiplet interconnects raise bandwidth between dies compared with PCIe alone. Within a node, fast links enable tensor parallelism shards that would stall on PCIe.

Across nodes, training scales with collective-friendly fabrics (InfiniBand, RoCE, specialized AI networks). The chip’s support for high-bandwidth links and GPUDirect-style paths affects whether a multi-GPU server behaves as one pool or a loose federation. This page stops at the server/package boundary; rack topology and congestion control live under AI infrastructure.

PCIe generations still matter for host↔device transfers, storage, and small models. Under-provisioned PCIe can starve accelerators during checkpoint load or embedding-heavy pipelines even when HBM bandwidth looks ample.

Software stacks and portability traps

Silicon without software is inert. Stacks include drivers, compiler/runtime (CUDA, ROCm, XLA, CoreML, vendor SDKs), kernel libraries (cuBLAS, cuDNN, custom fused kernels), and framework integrations (PyTorch, JAX, TensorFlow, ONNX Runtime). Performance portability is limited: a model fast on one generation may regress on another without re-tuning.

Lock-in appears through proprietary intrinsics, unsupported ops on alternate hardware, and optimized paths available only on one vendor’s library. ML platforms teams mitigate with abstraction layers, but someone still pays the profiling tax on each new SKU.

Container images pin driver, firmware, and library triples. Upgrades that help one workload can hurt another; golden benchmarks per dominant model should gate rollouts. For inference serving, the stack includes batching runtimes and KV-cache managers—not only raw matmuls.

Open ecosystems (OpenXLA, ONNX, Triton-class kernel authoring) reduce some friction but do not eliminate vendor-specific tuning. Budget engineering time for porting when diversifying suppliers.

Training silicon versus inference silicon

Training chips prioritize high HBM capacity, fast FP/BF/FP8 tensor throughput, bi-directional bandwidth for all-reduce, and reliability at sustained power. They tolerate higher cost and power per device because utilization is planned in long jobs. Multi-GPU servers with dense NVLink domains target large-model training and fine-tuning.

Inference silicon spans a wide range: datacenter GPUs with large memory for long-context large language models; L4-class accelerators optimized for efficient INT8/FP8 serving; ASICs (TPU inference pods, custom recommender chips) tuned for specific graphs; and tiny NPUs in handsets. Inference economics emphasize cost per token at your SLA, not peak training FLOPs.

Some chips marketed for both modes excel at one. Inference-heavy products may prefer lower power, strong INT8, and good batching; training-heavy buyers need optimizer-state headroom and fast interconnect. Using training SKUs for low-utilization inference can work but often wastes memory and power; using inference ASICs for large-scale training usually fails on memory and flexibility.

Batch size interacts with silicon choice: small batches on wide tensor cores underutilize arrays; dynamic shapes on inference ASICs may fall off fast paths. Match workload shape to silicon strengths with profiling, not SKU marketing names alone.

Utilization versus peak FLOPs

Utilization is the fraction of peak FLOPs achieved on real models—often single-digit to low-double-digit percentages at cluster level once data loading, communication, and idle time accumulate. On-device kernel utilization can be higher yet still miss SLA if serialization points exist.

MFU (model FLOPs utilization) and goodput (useful tokens or samples per second) are better procurement conversations than brochure peaks. A cheaper chip at 60% utilization beats a flagship at 15% for the same job—if software maturity is equal, which it rarely is.

Common utilization killers: host bottlenecks, suboptimal parallelization strategy, excessive precision, unfused ops, dynamic shapes forcing recompilation, small sequence lengths, and communication-bound tensor parallel splits. Fixing software often beats buying the next generation.

Power caps and thermal throttling reduce sustained FLOPs in dense racks. Facility cooling and power delivery are part of realized performance—another handoff point to AI infrastructure.

Procurement roadmap and supplier risk

Accelerator supply is cyclical and geopolitically sensitive. Enterprise roadmaps should include lead times, alternative SKUs, cloud burst capacity, and contractual exit ramps. Over-indexing on a single vendor simplifies software but concentrates risk; multi-vendor strategies raise porting cost—explicitly fund it.

Evaluate total cost: hardware, networking, power, cooling, software licenses, ml-platforms integration, and engineer time for tuning. A modest chip with excellent docs and stable drivers can beat a faster chip with immature stacks.

Watch deprecation schedules: older generations lose driver focus; inference fleets on deprecated SKUs become security and compliance debt. Plan refresh cycles with compatibility tests for your top models.

For regulated environments, document supply chain provenance and firmware update paths. Chips with opaque boot chains complicate attestation requirements.

Where chip choice ends and cluster design begins

Chip selection answers: what model sizes fit per device, what precision paths exist, what single-node parallelism is feasible, and what watts per unit throughput you can expect at tuned software. Cluster design answers: how many nodes, what network topology, how jobs are scheduled, how tenants are isolated, how checkpoints and serving scale—owned by AI infrastructure.

If multi-node training is required, network fabric and collective libraries dominate once per-chip HBM is exhausted. If single-node inference suffices, chip memory and serving runtime matter more than InfiniBand. Edge deployment shifts concern to LPDDR, thermal design power, and NPU toolchains—not HBM stacks.

Model-hosting services abstract some chip details but charge for the abstraction. Self-hosting keeps visibility into utilization and tuning at the cost of ops burden. Hybrid strategies—self-host steady state, burst to ai-cloud—still require knowing your per-chip unit economics to compare fairly.

GPU, TPU, NPU, and ASIC landscape at decision altitude

GPUs from vendors such as NVIDIA remain the default general-purpose AI accelerator: broad framework support, mature libraries, and wide operator coverage. TPUs from Google excel in XLA/JAX-centric training and inference pods when workloads map to their compilation model. NPUs appear in mobile SoCs and some datacenter inference cards, emphasizing efficiency on fixed graphs. Custom ASICs target narrow high-volume graphs—recommendation ranking, ads, video transcode ML—where flexibility tradeoffs are acceptable.

Choose GPUs when flexibility and ecosystem breadth dominate. Consider TPUs when you commit to Google’s stack and shapes compile cleanly. Consider NPUs at the edge or for strictly bounded graphs. Consider custom ASICs only at scale where engineering amortizes and graphs are stable for years.

Hugging Face and open-weight models increase portability of weights but not of performance; each chip still needs profiling. Community benchmarks are starting points—run your graph with your batching policy.

Economics: dollars per useful flop

Compare chips on dollars per sustained useful throughput under your software, not dollars per peak TFLOP. Include idle power for inference pools that scale to zero imperfectly. Include depreciation horizon—three-year plans differ from five-year facility amortization.

Inference ASICs can win on wedge workloads (stable CNNs, fixed-seq transformers) but struggle when product teams change architectures monthly. Flexibility has a price; quantify it as porting weeks times engineer cost.

Cloud vs owned silicon is a portfolio choice: owned wins at high steady utilization with staff to operate; cloud wins for spiky or experimental phases. Chips are the unit of owned capacity; understand them before signing long leases on the wrong SKU mix.

Interaction with machine learning practice

Chip constraints feed back into model design: parameter counts, activation checkpointing, quantization-aware training, and sequence length product decisions. Machine learning teams should involve hardware owners early when context windows or multimodal stacks jump an order of magnitude—otherwise models ship that cannot meet latency on the purchased fleet.

Training-data pipeline choices (resolution, sequence length, augmentation) change memory footprints and thus chip requirements. Silicon and data strategy are linked even when data scientists treat hardware as someone else’s problem.

Numeric precision paths from training to inference

Training typically flows FP32 master weights → BF16/FP16 forward/backward → FP32 accumulation for stability, with FP8 emerging on supported hardware for matmul-heavy regions. Inference may deploy INT8/INT4 weights with FP16 activations or full FP8 graphs when quality holds. Each transition requires calibration datasets representative of production traffic—not only validation splits from training.

Mixed precision is not automatic speedup: unsupported layers fall back to slower paths; loss scaling and gradient clipping need tuning when enabling FP8. Profile per-layer to find outliers that dominate latency despite small parameter counts (layer norm, softmax, embedding gathers).

Quantization interacts with chip generations: an INT8 kernel fast on one NPU may be absent on another, forcing FP16 fallback. Build a precision compatibility matrix during hardware eval, not after purchase.

Chiplet architectures and multi-die scaling

Moore’s law slowdown pushes chiplet designs: compute dies linked by high-speed interconnect on a package substrate. Benefits include yield ( smaller dies ), flexible SKU mixing ( compute + HBM stacks ), and incremental upgrades. Costs include complex thermal paths, NUMA-like locality effects, and software that must place shards aware of die boundaries.

Buyers should ask whether advertised “GPU memory” is uniform from all compute units or whether cross-die traffic penalizes certain parallelization strategies. Benchmark all-reduce and all-gather patterns that mirror your training framework defaults, not only single-device matmuls.

Future roadmaps increasingly tie superchip packaging to recommended parallelism recipes—ignore those pairing guides at your scheduling layer’s peril.

Edge NPUs versus datacenter accelerators

Edge NPUs optimize for milliwatt-to-watt budgets, fixed graph compilation, and deterministic latency for camera and mic pipelines. Datacenter accelerators optimize for bursty batching, large memory, and multi-tenant isolation. Porting a datacenter-trained transformer to a phone NPU without distillation or structural pruning usually fails on memory and op coverage.

Some enterprises deploy tiered silicon: tiny NPUs for wake and detect, edge GPUs in gateways for regional aggregation, datacenter pools for heavy reasoning. Chip procurement should follow that tiered architecture explicitly—one SKU rarely spans all tiers efficiently.

Thermal design power (TDP) and sustained versus burst clocks matter on edge; datacenter cards throttle under rack hotspots too, but monitoring stacks differ. Do not extrapolate edge benchmarks from datacenter cards without measurement.

Security, attestation, and confidential compute

Accelerators increasingly support confidential computing modes—encrypted memory, attested enclaves—for workloads that must protect weights and activations from host compromise. Uptake depends on performance overhead, key management maturity, and cloud provider support. Regulated buyers may require attestation before loading proprietary models.

Firmware and microcode updates on accelerators are part of the security surface. Patch cadence, rollback, and verification should match host OS discipline. Supply-chain attacks targeting driver stacks are high leverage because they touch every workload on the device.

Multi-tenant GPU clouds need memory scrubbing and isolation guarantees between jobs; chip features (MIG-like partitioning, IOMMU policies) interact with hypervisor choices—details belong in infrastructure runbooks, but chip capability gates what isolation is possible.

Benchmarking discipline beyond synthetic peaks

MLPerf and vendor microbenchmarks provide comparability but can diverge from your models. Build an internal benchmark suite: top three training steps, top three inference graphs, including dataloading and host preprocessing as measured in production containers.

Report distributions, not only means: p50 and p99 latency for inference, step-time variance for training. Chips that tighten variance reduce tail-SLA violations even if mean throughput looks similar.

Re-benchmark after every driver major version, framework upgrade, and model architecture change. Silicon performance is a three-way contract among die, software, and graph.

Practical evaluation checklist

Before standardizing on a chip generation: roofline dominant kernels for top models; end-to-end step time or tokens/sec at production batch; memory headroom for peak context; driver/firmware stability across a canary fleet; porting cost from incumbent stack; supply and deprecation outlook; power and cooling per rack; precision path coverage for intended quant formats; multi-die locality tests if applicable; and clear boundary with networking teams on when to leave single-node optimization for cluster work.

AI chips are not magic—they are specialized memory and math machines whose value appears only when software, models, and facility design align. Choose silicon with eyes open about utilization, lock-in, and where your organization’s real bottlenecks live.