Tokenization strategy representing rare words as compositions of frequent subword units like BPE pieces.
AI Glossary
Subword Tokenization
Coverage in development
Tokenization strategy representing rare words as compositions of frequent subword units like BPE pieces.
Definition
Plain English explanation
Subwords break rare words into known chunks.
Related glossary terms
Last reviewed
Sources
- NIST AI Risk Management Framework — risk vocabulary context for AI systems
- Brel Digital company and technology profiles — applied usage evidence where tagged
Correction request
If a technology assignment or hub description is inaccurate, submit a correction via the Corrections Policy.
All technologies → · AI Models → · APIs & SDKs → · Integrations → · Compliance → · Browse all companies → · Explore industries → · Compare →