BERT: Pre-training of deep bidirectional transformers for language understanding¶
Authors: J. Devlin; M.-W. Chang; K. Lee; K. Toutanova
Year: 2019
type: other collection: sok-references status: auto
Links: link
Summary¶
Auto-extracted summary (unreviewed)
BERT pre-trains a deep bidirectional Transformer encoder on unlabeled text with a masked-language-model objective (plus next-sentence prediction) so that the same model can be fine-tuned with a single added output layer for many NLP tasks. It sets new state-of-the-art results on eleven tasks, including GLUE 80.5, MultiNLI 86.7% accuracy, SQuAD v1.1 test F1 93.2 and SQuAD v2.0 test F1 83.1, with BERT-large outperforming BERT-base across all GLUE tasks.
Abstract¶
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).
Source: pdf
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | bidirectional pre-training via the masked language model objective | llm | 0.95 |
| contribution | single pre-trained architecture fine-tuned across sentence- and token-level tasks with minimal task-specific parameters | llm | 0.95 |
| contribution | state-of-the-art results on eleven NLP tasks | llm | 0.95 |
| contribution | release of BERT-base and BERT-large pre-trained models | llm | 0.95 |
| libraries | tensor2tensor | llm | 0.95 |
| models | BERT-base | llm | 0.95 |
| models | BERT-large | llm | 0.95 |
| datasets | GLUE | llm | 0.95 |
| datasets | MNLI | llm | 0.95 |
| datasets | QQP | llm | 0.95 |
| datasets | QNLI | llm | 0.95 |
| datasets | SST-2 | llm | 0.95 |
| datasets | CoLA | llm | 0.95 |
| datasets | STS-B | llm | 0.95 |
| datasets | MRPC | llm | 0.95 |
| datasets | RTE | llm | 0.95 |
| datasets | SQuAD v1.1 | llm | 0.95 |
| datasets | SQuAD v2.0 | llm | 0.95 |
| datasets | TriviaQA | llm | 0.95 |
| tasks | natural language understanding | llm | 0.95 |
| tasks | question answering | llm | 0.95 |
| tasks | natural language inference | llm | 0.95 |
| tasks | paraphrase and sentiment classification | llm | 0.95 |
| hardware | Cloud TPU | llm | 0.95 |
| code url | https://github.com/google-research/bert (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/tensorflow/tensor2tensor (URL found in the full text) | regex | 0.50 |
| limitations | fine-tuning of BERT-large was sometimes unstable on small datasets, requiring random restarts | llm | 0.95 |
| key result | GLUE score: 80.5 (BERT-large leaderboard); Table 1 average 79.6 (base) / 82.1 (large) (GLUE test set) | llm | 0.95 |
| key result | MultiNLI accuracy: 86.7% (BERT-large, MNLI test) | llm | 0.95 |
| key result | SQuAD v1.1 test F1: 93.2 (BERT-large with TriviaQA augmentation) | llm | 0.95 |
| key result | SQuAD v2.0 test F1: 83.1 (BERT-large) | llm | 0.95 |
| key result | fine-tuning time: about 30 minutes on a single Cloud TPU (SQuAD, Dev F1 91.0%) | llm | 0.95 |
SoK reference [38]¶
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423/
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- model: BERT 80 [p.1], GPT 28 [p.1], Transformer 22 [p.1], RNN / LSTM 13 [p.11], Autoencoder 2 [p.3]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 37 [p.3], GLUE 20 [p.1], SQuAD 19 [p.1], SST-2 4 [p.6], ImageNet 1 [p.10]
- hardware: TPU 7 [p.5], GPU 1 [p.5]
- technique: Fine-tuning 11 [p.2], Attention 10 [p.1], Softmax 4 [p.4]
- task: Question answering 6 [p.1], Language modeling / generation 6 [p.2], Named entity recognition 4 [p.1], Machine translation 3 [p.3]
- topic: Benchmark 8 [p.2], Open source 2 [p.2]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/038_bert-pre-training-of-deep-bidirectional-transformers-for-lan.pdf |
sok-references | 16 | 767.9 KB | 987545ffb087 |
Database key: bert-pre-training-of-deep-bidirectional-transformers-for-language · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set bert-pre-training-of-deep-bidirectional-transformers-for-language COLUMN VALUE