Improving language understanding by generative pretraining¶
Authors: A. Radford; K. Narasimhan; T. Salimans; I. Sutskever
Year: 2018
type: other collection: sok-references status: auto
Links: link
Summary¶
Auto-extracted summary (unreviewed)
This OpenAI paper introduces generative pre-training (GPT): a 12-layer decoder-only transformer (768-dimensional states, 12 attention heads) is pre-trained with a language-modeling objective on BooksCorpus and then discriminatively fine-tuned on each target task using task-aware input transformations that require minimal architecture changes. The single task-agnostic model improves the state of the art on 9 of 12 natural language understanding benchmarks, including absolute gains of 8.9% on Story Cloze, 5.7% on RACE and 1.5% on MultiNLI.
Abstract¶
Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).
Source: pdf
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | generative pre-training of a transformer language model followed by discriminative fine-tuning | llm | 0.90 |
| contribution | task-aware input transformations enabling transfer with minimal architecture changes | llm | 0.90 |
| contribution | state-of-the-art results on 9 of 12 NLU tasks with a single task-agnostic model | llm | 0.90 |
| libraries | ftfy | llm | 0.90 |
| libraries | spaCy | llm | 0.90 |
| models | GPT-1 (12-layer decoder-only transformer) | llm | 0.90 |
| datasets | BooksCorpus | llm | 0.90 |
| datasets | SNLI | llm | 0.90 |
| datasets | MultiNLI | llm | 0.90 |
| datasets | QNLI | llm | 0.90 |
| datasets | RTE | llm | 0.90 |
| datasets | SciTail | llm | 0.90 |
| datasets | RACE | llm | 0.90 |
| datasets | Story Cloze | llm | 0.90 |
| datasets | MRPC | llm | 0.90 |
| datasets | Quora Question Pairs | llm | 0.90 |
| datasets | STS-B | llm | 0.90 |
| datasets | SST-2 | llm | 0.90 |
| datasets | CoLA | llm | 0.90 |
| datasets | GLUE | llm | 0.90 |
| tasks | language modeling | llm | 0.90 |
| tasks | natural language inference | llm | 0.90 |
| tasks | question answering | llm | 0.90 |
| tasks | sentence similarity | llm | 0.90 |
| tasks | text classification | llm | 0.90 |
| limitations | underperforms the multi-task biLSTM baseline on the small RTE dataset (56% vs 61.7%) | llm | 0.90 |
| limitations | multi-task training not explored | llm | 0.90 |
| key result | absolute accuracy improvement on Story Cloze: +8.9% (commonsense reasoning) | llm | 0.90 |
| key result | absolute accuracy improvement on RACE: +5.7% (question answering) | llm | 0.90 |
| key result | absolute accuracy improvement on MultiNLI: +1.5% (textual entailment) | llm | 0.90 |
| key result | tasks with new state of the art: 9 of 12 (NLU benchmarks) | llm | 0.90 |
| key result | RTE accuracy: 56% (below the 61.7% of a multi-task biLSTM) (RTE, 2490 examples) | llm | 0.90 |
| key result | token-level perplexity: 18.4 (BooksCorpus language model) | llm | 0.90 |
SoK reference [39]¶
A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding by generative pretraining,” OpenAI, Tech. Rep., 2018, available at: https://cdn.openai.com/research-covers/language-unsupervised/ language understanding paper.pdf.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- model: Transformer 14 [p.2], RNN / LSTM 6 [p.2], CNN 2 [p.6], Autoencoder 2 [p.11]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 32 [p.5], SST-2 5 [p.6], GLUE 5 [p.2]
- technique: Softmax 4 [p.3], Fine-tuning 4 [p.5], Attention 3 [p.3], LayerNorm 2 [p.5]
- task: Question answering 14 [p.1], Language modeling / generation 8 [p.1], Machine translation 7 [p.1]
- topic: Benchmark 9 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/039_improving-language-understanding-by-generative-pre-training.pdf |
sok-references | 12 | 528.4 KB | eb81a65e0856 |
Database key: improving-language-understanding-by-generative-pretraining · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set improving-language-understanding-by-generative-pretraining COLUMN VALUE