fairseq: A fast, extensible toolkit for sequence modeling¶
Authors: Myle Ott; Sergey Edunov; Alexei Baevski; Angela Fan; Sam Gross; Nathan Ng; David Grangier; Michael Auli
Year: 2019
Venue: arXiv preprint
type: library collection: sok-references status: auto
Links: arXiv:1904.01038
Summary¶
Auto-extracted summary (unreviewed)
fairseq is Facebook AI Research's open-source PyTorch toolkit for sequence modeling (translation, summarization, language modeling and other text generation), built around five plug-in types and supporting distributed multi-GPU/multi-machine training with overlapped gradient synchronization, FP16 mixed precision with dynamic loss scaling, and optimized inference with incremental decoding, beam search, diverse beam search and top-k sampling. On WMT'14 English-German with a big Transformer on a V100, FP16 inference reaches 136.0 sentences/s versus 88.1 for FP32 (54% faster) with no accuracy loss.
Abstract¶
fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | extensible plug-in architecture (models, criterions, tasks, optimizers, LR schedulers) | llm | 0.95 |
| contribution | efficient distributed training with gradient-sync overlap and gradient accumulation | llm | 0.95 |
| contribution | mixed-precision (FP16) training and inference with dynamic loss scaling | llm | 0.95 |
| contribution | optimized inference: incremental decoding, beam search, diverse beam search, top-k sampling | llm | 0.95 |
| contribution | state-of-the-art implementations and pretrained models for MT, summarization and LM | llm | 0.95 |
| libraries | PyTorch | llm | 0.95 |
| libraries | fairseq | llm | 0.95 |
| libraries | NCCL | llm | 0.95 |
| models | Transformer (big) | llm | 0.95 |
| datasets | WMT'14 English-German | llm | 0.95 |
| tasks | machine translation | llm | 0.95 |
| tasks | language modeling | llm | 0.95 |
| tasks | abstractive summarization | llm | 0.95 |
| tasks | story generation | llm | 0.95 |
| tasks | grammatical error correction | llm | 0.95 |
| tasks | multilingual sentence embeddings | llm | 0.95 |
| tasks | dialogue | llm | 0.95 |
| hardware | V100 GPU | llm | 0.95 |
| code url | https://github.com/pytorch/fairseq (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/moses-smt/mosesdecoder/blob/master/scripts/ (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/pytorch/fairseq | llm | 0.95 |
| key result | translation speed: 88.1 sentences/s (FP32), 136.0 sentences/s (FP16) (big Transformer, WMT'14 En-De test set, V100 GPU) | llm | 0.95 |
| key result | FP16 decoding speed-up: 54% faster than FP32 with no loss in accuracy (inference) | llm | 0.95 |
SoK reference [70]¶
M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” in Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics (Demonstrations), 2019, pp. 48–53.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- library: PyTorch 5 [p.1]
- model: Transformer 9 [p.3], CNN 5 [p.4], RNN / LSTM 2 [p.3], Mixture of experts 2 [p.2]
- dataset: WikiText 3 [p.4]
- hardware: GPU 17 [p.1], Multi-GPU 3 [p.2], NVIDIA V100 2 [p.3]
- technique: Convolution 8 [p.3], Softmax 3 [p.4]
- task: Language modeling / generation 13 [p.1], Machine translation 13 [p.1]
- topic: Open source 4 [p.1], Benchmark 3 [p.3]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/070_fairseq-a-fast-extensible-toolkit-for-sequence-modeling.pdf |
sok-references | 6 | 245.9 KB | e064f0dede87 |
Database key: fairseq-a-fast-extensible-toolkit-for-sequence-modeling · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set fairseq-a-fast-extensible-toolkit-for-sequence-modeling COLUMN VALUE