Small Language Models (SLMs)

01. DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base.
02. MobileBERT - It’s a bidirectional transformer based on the BERT model, which is compressed and accelerated using several approaches.
03. Microsoft Phi-3 open models - A family of powerful, small language models (SLMs) with groundbreaking performance at low cost and low latency
04. FlauBERT - It’s a transformer model pretrained using a masked language modeling (MLM) objective (like BERT).
05. GPT-Neo is a small language model with 125 million parameters developed by EleutherAI
06. OpenAI GPT-2 model is a causal (unidirectional) transformer pretrained using language modeling on a very large corpus of ~40 GB of text data.