Skip to content

Llama: Open and efficient foundation language models

Authors: Hugo Touvron; Thibaut Lavril; Gautier Izacard; Xavier Martinet; Marie-Anne Lachaux; Timothée Lacroix; Baptiste Rozière; Naman Goyal; Eric Hambro; Faisal Azhar; Aurelien Rodriguez; Armand Joulin; Edouard Grave; Guillaume Lample
Year: 2023
Venue: arXiv preprint

type: other collection: sok-references status: auto

Links: arXiv:2302.13971

Summary

Auto-extracted summary (unreviewed)

LLaMA is a family of open foundation language models from 7B to 65B parameters trained on trillions of tokens drawn exclusively from publicly available data, optimized for the best performance at a given inference budget by training smaller models on more tokens. LLaMA-13B outperforms GPT-3 (175B) on most benchmarks despite being 10x smaller, and LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B; all models were released to the research community.

Abstract

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution Open release of 7B-65B parameter foundation language models llm 0.85
contribution Training on exclusively public data to state-of-the-art quality llm 0.85
contribution Training smaller models longer to optimize inference cost llm 0.85
contribution Evaluation of bias and toxicity with responsible-AI benchmarks llm 0.85
libraries xformers llm 0.85
models LLaMA-7B llm 0.85
models LLaMA-13B llm 0.85
models LLaMA-65B llm 0.85
models GPT-3 (175B) llm 0.85
models Chinchilla-70B llm 0.85
models PaLM-540B llm 0.85
tasks language modeling llm 0.85
tasks few-shot benchmark evaluation llm 0.85
tasks bias and toxicity evaluation llm 0.85
code url https://github.com/facebookresearch/llama (URL found in the full text) regex 0.50
code url https://github.com/facebookresearch/xformers (URL found in the full text) regex 0.50
code url https://github.com/facebookresearch/metaseq/tree/main/projects/OPT/chronicles (URL found in the full text) regex 0.50
code url https://github.com/facebookresearch/llama llm 0.85
limitations Models encode biases and toxicity, which the authors measure with responsible-AI benchmarks llm 0.85
key result benchmark performance: LLaMA-13B outperforms GPT-3 (175B) on most benchmarks (10x smaller model) llm 0.85
key result benchmark performance: LLaMA-65B competitive with Chinchilla-70B and PaLM-540B (largest model) llm 0.85
key result scaling observation: 7B model keeps improving beyond 1T training tokens (contrast with Chinchilla recommendation of 200B tokens for 10B model) llm 0.85

SoK reference [40]

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023. [Online]. Available: https://arxiv.org/abs/ 2302.13971

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: Leveled / somewhat HE 12 [p.9]
  • library: PyTorch 1 [p.3]
  • model: LLaMA 71 [p.1], GPT 31 [p.1], Transformer 1 [p.13], BERT 1 [p.11], RNN / LSTM 1 [p.11]
  • dataset: HellaSwag 3 [p.4], ARC 3 [p.4]
  • hardware: GPU 11 [p.1], NVIDIA A100 10 [p.4], NVIDIA V100 1 [p.5]
  • technique: Fine-tuning 2 [p.6]
  • task: Language modeling / generation 10 [p.3], Question answering 7 [p.5], Machine translation 4 [p.11], Medical / healthcare 4 [p.9], Speech recognition 2 [p.12]
  • topic: Benchmark 31 [p.1], Large language models 13 [p.1], Open source 6 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/040_llama-open-and-efficient-foundation-language-models.pdf sok-references 27 709.5 KB 2e663675ae36

Database key: llama-open-and-efficient-foundation-language-models · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set llama-open-and-efficient-foundation-language-models COLUMN VALUE