Foundation Model

A foundation model is a large pretrained model that can be adapted to many downstream tasks through prompting, fine-tuning, or tooling.

One big pretrained model becomes a base layer that many applications build on.

Foundation models are trained on broad data at scale and adapted via prompting, fine-tuning, or tools. The term is used in research and policy discussions to describe general-purpose base models.

Policy and procurement increasingly regulate or review foundation models separately from narrow systems.

• Base for assistants and vertical apps
• Multimodal platforms
• Research benchmarks

• Amortizes pretraining cost across many tasks
• Enables rapid product experimentation

• Concentrated compute and data requirements
• Systemic risk considerations discussed by policymakers

• Foundation model ≠ finished application

Is every LLM a foundation model?

Many large pretrained LMs are described as foundation models; smaller task-specific models usually are not.

• NIST AI RMF

• Bommasani et al. — On the Opportunities and Risks of Foundation Models