Llama: Open and efficient foundation language models¶
Authors: Hugo Touvron; Thibaut Lavril; Gautier Izacard; Xavier Martinet; Marie-Anne Lachaux; Timothée Lacroix; Baptiste Rozière; Naman Goyal; Eric Hambro; Faisal Azhar; Aurelien Rodriguez; Armand Joulin; Edouard Grave; Guillaume Lample
Year: 2023
Venue: arXiv preprint
type: other collection: sok-references status: auto
Links: arXiv:2302.13971
Summary¶
Auto-extracted summary (unreviewed)
LLaMA is a family of open foundation language models from 7B to 65B parameters trained on trillions of tokens drawn exclusively from publicly available data, optimized for the best performance at a given inference budget by training smaller models on more tokens. LLaMA-13B outperforms GPT-3 (175B) on most benchmarks despite being 10x smaller, and LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B; all models were released to the research community.
Abstract¶
We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | Open release of 7B-65B parameter foundation language models | llm | 0.85 |
| contribution | Training on exclusively public data to state-of-the-art quality | llm | 0.85 |
| contribution | Training smaller models longer to optimize inference cost | llm | 0.85 |
| contribution | Evaluation of bias and toxicity with responsible-AI benchmarks | llm | 0.85 |
| libraries | xformers | llm | 0.85 |
| models | LLaMA-7B | llm | 0.85 |
| models | LLaMA-13B | llm | 0.85 |
| models | LLaMA-65B | llm | 0.85 |
| models | GPT-3 (175B) | llm | 0.85 |
| models | Chinchilla-70B | llm | 0.85 |
| models | PaLM-540B | llm | 0.85 |
| tasks | language modeling | llm | 0.85 |
| tasks | few-shot benchmark evaluation | llm | 0.85 |
| tasks | bias and toxicity evaluation | llm | 0.85 |
| code url | https://github.com/facebookresearch/llama (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/facebookresearch/xformers (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/facebookresearch/metaseq/tree/main/projects/OPT/chronicles (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/facebookresearch/llama | llm | 0.85 |
| limitations | Models encode biases and toxicity, which the authors measure with responsible-AI benchmarks | llm | 0.85 |
| key result | benchmark performance: LLaMA-13B outperforms GPT-3 (175B) on most benchmarks (10x smaller model) | llm | 0.85 |
| key result | benchmark performance: LLaMA-65B competitive with Chinchilla-70B and PaLM-540B (largest model) | llm | 0.85 |
| key result | scaling observation: 7B model keeps improving beyond 1T training tokens (contrast with Chinchilla recommendation of 200B tokens for 10B model) | llm | 0.85 |
SoK reference [40]¶
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [Online]. Available: https://arxiv.org/abs/ 2302.13971
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: Leveled / somewhat HE 12 [p.9]
- library: PyTorch 1 [p.3]
- model: LLaMA 71 [p.1], GPT 31 [p.1], Transformer 1 [p.13], BERT 1 [p.11], RNN / LSTM 1 [p.11]
- dataset: HellaSwag 3 [p.4], ARC 3 [p.4]
- hardware: GPU 11 [p.1], NVIDIA A100 10 [p.4], NVIDIA V100 1 [p.5]
- technique: Fine-tuning 2 [p.6]
- task: Language modeling / generation 10 [p.3], Question answering 7 [p.5], Machine translation 4 [p.11], Medical / healthcare 4 [p.9], Speech recognition 2 [p.12]
- topic: Benchmark 31 [p.1], Large language models 13 [p.1], Open source 6 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/040_llama-open-and-efficient-foundation-language-models.pdf |
sok-references | 27 | 709.5 KB | 2e663675ae36 |
Database key: llama-open-and-efficient-foundation-language-models · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set llama-open-and-efficient-foundation-language-models COLUMN VALUE