Computer Vision: Tasks, Metrics, Architectures, and Deployment

Computer vision is the discipline of turning pixels and video into decisions: what is present, where it is, how it moves, what text it contains, and whether those estimates are good enough for a downstream action—rather than sampling new pixels as in image generation. Temporal stacks deepen under video AI. Those OCR and layout primitives also power document intelligence pipelines. It is not synonymous with deep learning, though modern systems usually use learned neural network backbones. Vision–language systems deepen under multimodal AI. Optics, lighting, labeling, metrics, and deployment topology including edge AI on-device vision are part of the system—not afterthoughts. Benchmark lineages such as COCO are associated with Microsoft research history; your production metrics still need site-specific slices.

This guide owns vision task taxonomy, metrics matched to decisions, camera and optics as model inputs, labeling practice, and edge versus cloud placement. For hierarchical representation learning, training systems, and transfer ladders, see deep learning. For unit-level network mechanics, see neural networks. Full generative image synthesis belongs elsewhere; document-only OCR products are a specialized neighbor, not this page’s center.

What computer vision predicts from pixels and video

Multiview and stereo setups add geometry but multiply calibration debt. If triangulation underpins the decision, invest in calibration monitoring; if appearance classification underpins it, invest in lighting control. Mixed systems should isolate which errors are photometric versus geometric when debugging.

Write the prediction contract explicitly: input resolution, allowed modalities (RGB, depth, event), output schema, and maximum latency. Ambiguous contracts produce demos that cannot be acceptance-tested. If two vendors claim “detection,” compare IoU thresholds, class lists, and whether scores are calibrated—otherwise you are comparing press releases.

Synthetic data can help rare geometries and dangerous scenarios, but only when the synthetic-to-real gap is measured on real holdouts. Accept synthetic for pretraining or augmentation when real tail data are scarce; never declare production readiness from synthetic test sets alone.

A vision system maps an image or clip x to a structured output y that a decision procedure consumes. y may be a class label, a set of boxes with scores, a dense mask, a track identity over time, a pose skeleton, a depth map, or OCR fields with bounding polygons. The form of y must match the decision. Choosing detection when you only need presence wastes label budget; choosing classification when you need localization creates silent spatial errors.

Video adds time: temporal consistency, identity continuity, and motion cues. Treating video as independent frames is a design choice with known failure modes (flicker, ID switches). Treating images as video of length one keeps APIs uniform but does not invent temporal reasoning.

Sensors matter. Rolling shutter, auto-exposure, white balance, compression, infrared versus RGB, and lens distortion change the distribution the model sees. A model trained on clean DSLR photos can fail on a warehouse camera with glare and H.264 artifacts even if the “task” is identical on paper.

Task families: classify, detect, segment, track, OCR, pose—including crop and livestock vision

Use this table as a starting decision aid, then rewrite the metric column for your cost model. The right task with the wrong metric still ships the wrong system.

Multi-task heads sharing a backbone can amortize compute, but task interference is real: optimizing mask quality can hurt box AP if losses are poorly weighted. Start with the primary decision task; add auxiliary heads when they improve the primary metric or provide needed outputs—not because multi-task sounds modern.

Task families differ by output geometry and supervision cost:

• Image classification: one or multi-label tags for the whole frame.
• Object detection: classed boxes (or rotated boxes) with scores.
• Semantic / instance / panoptic segmentation: per-pixel labels; instance separates objects; panoptic unifies stuff and things.
• Tracking: maintain identities across frames; often built on detection plus association.
• OCR / scene text: detect and read text; often a cascade into parsers.
• Pose / keypoints: anatomical or object landmarks.
• Depth / geometry: metric or relative structure for robotics and AR.

Pick the family from the decision and the annotation budget. Segmentation is powerful and expensive. Detection often hits the industrial sweet spot when “where” matters but pixel-perfect masks do not. Classification is enough for coarse routing. OCR cascades fail when detection is good but reading is wrong—or when reading is right but downstream parsers are brittle.

Metrics that match decisions (not vanity leaderboards)

For defect detection gating shipment, define a precision floor and maximize recall subject to that floor—or the reverse if false scrap is costlier. Put that rule in the acceptance test. For counting, prefer absolute and relative count error over mAP. For tracking gates, measure event-level false merges/splits. Metric mismatch is how projects “hit 0.9 mAP” and still fail the plant floor.

Leaderboard metrics are starting points, not gates. Defect detection may care about recall at a fixed precision; access control may care about false accepts; retail analytics may care about count error; medical triage may care about calibrated risk by subgroup. Publish the operating point: threshold, IoU criterion, and business cost of each error type.

Intersection over Union (IoU) thresholds define what counts as a correct localization. mAP averages across recall levels and classes; it can hide catastrophic failure on a critical rare class. For detection in production, report per-class AP, small/medium/large object slices, and confusion between confusable classes. For segmentation, IoU/Dice per class and boundary F-scores matter when edges drive the decision.

Calibration matters when scores trigger automation. A detector with good mAP but miscalibrated confidence will either flood humans or miss silently when thresholded. Temperature scaling and reliability diagrams are part of vision ops when scores are consumed as probabilities.

Classical vision versus learned perception

Calibration boards, structured light, and known camera intrinsics still unlock measurement tasks that pure appearance models approximate poorly. If you need millimeters, invest in geometry. If you need “is this SKU present on a messy shelf,” invest in diverse labeled appearance.

Classical pipelines used hand-designed features (edges, corners, SIFT-like descriptors) and geometric multi-view methods. They remain valuable for calibrated metrology, controlled lighting, and problems where physics dominates texture. Learned perception dominates open-world appearance variation: lighting changes, deformable objects, messy backgrounds.

Hybrids are common: classical undistortion and homographies feed a CNN/ViT; RANSAC cleans learned correspondences; Kalman filters smooth tracks. Do not romanticize either extreme. Use classical modules when constraints are geometric and measurable; use learning when appearance variability overwhelms handcrafted features.

CNN and ViT biases for visual structure

Input resolution is a first-class hyperparameter in vision. Doubling side length roughly quadruples spatial compute for dense CNNs and can explode attention memory for naive ViTs. Report accuracy–latency curves across resolutions rather than a single vanity resolution used only in papers.

Convolutional networks bake in locality and translation equivariance via weight sharing—strong priors for natural images. Vision transformers tokenize patches and mix globally with attention, shifting inductive burden onto data scale and regularization. Hybrids (convolutional stems, windowed attention) trade between the two.

Architecture choice belongs with the data regime and latency budget. For how these families fit broader representation learning and transfer, use the deep learning guide. Here the claim is narrower: the backbone’s bias interacts with object scale, clutter, and domain shift. Small objects vanish under aggressive downsampling; global attention can help long-range context but costs memory at high resolution.

Label ontology, agreement, and long-tail classes

Active learning loops should optimize decision risk, not only model uncertainty. Sampling the most uncertain pixels can waste budget on inherently ambiguous boundaries. Prefer samples that change thresholds, confuse critical pairs, or come from underperforming cameras.

The ontology is the product. Ambiguous class boundaries (“vehicle” vs “truck” vs “trailer”) create annotator disagreement that looks like model error. Write labeling guides with edge cases, and measure inter-annotator agreement before blaming the network. Multilabel scenes need clear policies for occlusion, truncation, and minimum visible area.

Long-tail classes dominate real deployments: rare defects, rare SKUs, rare animals, rare uniforms. Average metrics will look fine while the tail fails. Strategies include targeted collection, active learning, class-aware sampling, and sometimes collapsing rare classes into a monitored “other” with human review—not pretending the model is uniform.

Augmentation semantics that preserve labels

Test-time augmentation (TTA) can lift metrics at latency cost. If you use TTA in eval but not in serve, you will overstate production accuracy. Keep eval aligned with the serving path you will actually ship.

Augmentation must preserve label meaning. Horizontal flips break OCR and asymmetric classes (left vs right defects). Color jitter can destroy bruise or rash cues. Mosaic/mixup-style composites can create impossible scenes that hurt detectors if overused. Geometric warps must update boxes and masks consistently—silent label drift is a common training bug.

Augment for the deployment distribution: motion blur, compression, glare, rain, low light. Augmenting only with clean ImageNet-style crops trains a different camera than the one you install.

Train/serve image pipeline parity

Batch vs single-image normalization differences, GPU vs CPU decode libraries, and color-range full vs limited TV range are subtle parity killers. Lock decoder versions in deployment artifacts. Diff golden tensors in CI when anyone touches image I/O.

Training preprocessing and serving preprocessing must match: resize policy (letterbox vs stretch), color space, normalization, JPEG quality, EXIF orientation handling, and max dimension. Stretching to a square quietly changes object aspect ratios and harms detection. Letterboxing preserves aspect but changes effective scale statistics—record which you used.

EXIF orientation bugs flip “trained upright” into “served sideways.” Camera firmware updates change compression. Build golden image tests: known bytes in → known tensors out, in both train and serve paths. Many “model regressions” after a mobile app release are pipeline diffs.

Detection systems and small-object failure

Two-stage detectors propose regions then classify; single-stage detectors predict densely. Two-stage systems often win on difficult small objects at higher latency; single-stage systems win on throughput. Anchor-free heads reduce hyperparameter churn but do not remove the need for multi-scale features. Choose from measured latency–AP curves on your cameras, not from architecture fashion.

Open-vocabulary and grounding detectors expand class sets via language, which helps long-tail names but introduces prompt sensitivity and new evaluation needs. If your ontology is closed and safety-critical, a closed-set detector with known failure modes may be easier to certify than an open-vocabulary system that can invent labels.

Crowded scenes stress detectors: overlapping boxes, truncated objects, and heavy occlusion. Crowded-human benchmarks taught the field that “COCO-like” averages are insufficient. If your site is crowded, build a crowded validation slice and optimize for it explicitly.

Modern detectors combine a backbone, a neck for multi-scale features, and heads for classification and box regression (anchor-based or anchor-free). Multi-scale pyramids help, but very small objects still fail when they occupy few pixels after resize. Fixes are operational as much as architectural: increase input resolution (costly), zoom crops, better camera placement, or task redesign (presence sensors).

NMS and score thresholds shape production behavior. Aggressive NMS suppresses duplicate boxes but can delete overlapping valid objects. Soft-NMS and class-aware NMS variants trade differently. Tune thresholds on a validation set that mirrors camera distance and density—not on a pretty demo reel.

Segmentation quality and calibration

Human-in-the-loop correction of masks can be cheaper than chasing the last IoU point with model complexity—especially when errors are spatially obvious. Design UIs that edit polygons quickly; treat the model as a draft generator when stakes are high.

Segmentation errors concentrate on boundaries and thin structures. If the decision cares about area (spill size, wound area), boundary noise becomes metric noise. If the decision cares about topology (is the crack connected?), connected-component post-processing may matter more than pixel accuracy.

Class imbalance inside images (tiny defect on large background) biases toward background. Loss design (dice, focal variants, boundary losses) and sampling matter. Still report decision-level metrics, not only pixel accuracy that can be dominated by easy background.

Video tracking and identity switches (including player tracking contexts)

Camera handoff across overlapping views needs either shared calibration/re-ID or acceptance that identities reset at boundaries. Multi-camera tracking is a systems problem: clocks, homographies, and privacy boundaries included.

Tracking associates detections over time. Identity switches occur under occlusion, similar appearance, and camera gaps. Re-ID embeddings help but fail across clothing changes and lighting. For safety-critical counting, design for conservative identity breaks rather than silent merges that undercount or double-count events.

Evaluate with identity-aware metrics (e.g., IDF1-style measures) and event-level outcomes, not only frame mAP. A system can detect well and still fail the business KPI if tracks fragment.

OCR cascades into downstream systems

Language priors help recognition but can hallucinate plausible words. For identifiers (serials, license-like strings), constrain with check digits and allowed character sets. Never let an unconstrained language model “fix” a serial number without validation.

Scene text and document OCR are cascades: detect text regions → recognize strings → parse fields. End-to-end character error rate hides which stage failed. Log stage-wise errors. Downstream parsers should be tolerant to common confusions (O/0, I/1) when domain allows, and should fail closed when fields drive money or access.

This page covers OCR as a vision cascade with localization and reading metrics. Deep productization of document workflows is a neighboring specialty—do not treat this guide as a full document-intelligence manual.

Edge versus cloud and compression artifacts

Model update mechanics differ: edge fleets need signed artifacts, staged rollouts, and rollback when a site’s cameras disagree with the new weights. Cloud endpoints update faster but centralize outage risk. Choose the operational model you can staff.

Placement is a latency, bandwidth, privacy, and power decision. Edge inference reduces round-trips and can keep pixels on-prem; cloud inference simplifies model updates and heavy models. Hybrid designs run a small on-device gate and send crops or embeddings upstream.

Compression before inference is a distribution shift. Training on pristine PNG and serving on heavily compressed RTSP frames causes avoidable drops. Capture training data through the real encode path. Measure accuracy versus bitrate; sometimes a slightly larger on-edge model on cleaner local frames beats a huge cloud model on muddy streams.

Sensitive uses: people-in-frame and privacy

Demographic performance gaps are a vision-specific harm mode. Evaluate by relevant subgroups when people are scored; do not hide behind average accuracy. If you cannot measure subgroup performance, limit automation and keep humans in the loop.

People-in-frame systems inherit privacy, labor, and legal constraints. Minimize retention of raw video; prefer on-device redaction; document purpose limitation; and avoid secondary use of embeddings for unrelated surveillance. Accuracy claims do not erase consent and policy requirements. If you cannot state the lawful basis and retention policy, you are not ready to deploy.

Fleet failure modes across camera installs

Also monitor data volume and uptime: a silent camera producing black frames can look like a model that “predicts background.” Health checks on bitrates, blank-frame rates, and timestamp gaps belong beside accuracy dashboards.

A model that works on one camera often fails across a fleet: different mounts, lenses, firmware, times of day, and maintenance states (dirty domes). Monitor per-camera metrics and drift. Sudden drops often mean hardware or config change, not “AI decay.” Keep a golden set per site. Plan for camera replacement as a domain-shift event requiring validation, not a hot swap.

When camera swaps hurt accuracy, the root cause is usually distribution shift in intrinsics, mounting angle, compression, and color response—not a mysterious loss of “intelligence.” Treat hardware changes as model releases: validate on a site golden set before cutting over traffic.

Edge versus cloud placement should be written as an explicit decision record: latency SLO, privacy constraint, bandwidth cap, power budget, and update cadence. If two of those conflict, the architecture must say which wins. Ambiguous placement documents produce accidental cloud round-trips for frames that were never allowed to leave the site.

References and further reading

• Lin, T.-Y., et al. (2014). Microsoft COCO: Common Objects in Context.
• Ren, S., et al. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.
• He, K., et al. (2017). Mask R-CNN.
• Dosovitskiy, A., et al. (2020). An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale.
• Padilla, R., et al. (2020). A Survey on Performance Metrics for Object-Detection Algorithms.