Skip to content

fairseq: A fast, extensible toolkit for sequence modeling

Authors: Myle Ott; Sergey Edunov; Alexei Baevski; Angela Fan; Sam Gross; Nathan Ng; David Grangier; Michael Auli
Year: 2019
Venue: arXiv preprint

type: library collection: sok-references status: auto

Links: arXiv:1904.01038

Summary

Auto-extracted summary (unreviewed)

fairseq is Facebook AI Research's open-source PyTorch toolkit for sequence modeling (translation, summarization, language modeling and other text generation), built around five plug-in types and supporting distributed multi-GPU/multi-machine training with overlapped gradient synchronization, FP16 mixed precision with dynamic loss scaling, and optimized inference with incremental decoding, beam search, diverse beam search and top-k sampling. On WMT'14 English-German with a big Transformer on a V100, FP16 inference reaches 136.0 sentences/s versus 88.1 for FP32 (54% faster) with no accuracy loss.

Abstract

fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto
Source: arxiv

Extracted details

Field Value Source Confidence
contribution extensible plug-in architecture (models, criterions, tasks, optimizers, LR schedulers) llm 0.95
contribution efficient distributed training with gradient-sync overlap and gradient accumulation llm 0.95
contribution mixed-precision (FP16) training and inference with dynamic loss scaling llm 0.95
contribution optimized inference: incremental decoding, beam search, diverse beam search, top-k sampling llm 0.95
contribution state-of-the-art implementations and pretrained models for MT, summarization and LM llm 0.95
libraries PyTorch llm 0.95
libraries fairseq llm 0.95
libraries NCCL llm 0.95
models Transformer (big) llm 0.95
datasets WMT'14 English-German llm 0.95
tasks machine translation llm 0.95
tasks language modeling llm 0.95
tasks abstractive summarization llm 0.95
tasks story generation llm 0.95
tasks grammatical error correction llm 0.95
tasks multilingual sentence embeddings llm 0.95
tasks dialogue llm 0.95
hardware V100 GPU llm 0.95
code url https://github.com/pytorch/fairseq (URL found in the full text) regex 0.50
code url https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ (URL found in the full text) regex 0.50
code url https://github.com/pytorch/fairseq llm 0.95
key result translation speed: 88.1 sentences/s (FP32), 136.0 sentences/s (FP16) (big Transformer, WMT'14 En-De test set, V100 GPU) llm 0.95
key result FP16 decoding speed-up: 54% faster than FP32 with no loss in accuracy (inference) llm 0.95

SoK reference [70]

M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (Demonstrations), 2019, pp. 48–53.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • library: PyTorch 5 [p.1]
  • model: Transformer 9 [p.3], CNN 5 [p.4], RNN / LSTM 2 [p.3], Mixture of experts 2 [p.2]
  • dataset: WikiText 3 [p.4]
  • hardware: GPU 17 [p.1], Multi-GPU 3 [p.2], NVIDIA V100 2 [p.3]
  • technique: Convolution 8 [p.3], Softmax 3 [p.4]
  • task: Language modeling / generation 13 [p.1], Machine translation 13 [p.1]
  • topic: Open source 4 [p.1], Benchmark 3 [p.3]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/070_fairseq-a-fast-extensible-toolkit-for-sequence-modeling.pdf sok-references 6 245.9 KB e064f0dede87

Database key: fairseq-a-fast-extensible-toolkit-for-sequence-modeling · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set fairseq-a-fast-extensible-toolkit-for-sequence-modeling COLUMN VALUE