Skip to content

BERT: Pre-training of deep bidirectional transformers for language understanding

Authors: J. Devlin; M.-W. Chang; K. Lee; K. Toutanova
Year: 2019

type: other collection: sok-references status: auto

Links: link

Summary

Auto-extracted summary (unreviewed)

BERT pre-trains a deep bidirectional Transformer encoder on unlabeled text with a masked-language-model objective (plus next-sentence prediction) so that the same model can be fine-tuned with a single added output layer for many NLP tasks. It sets new state-of-the-art results on eleven tasks, including GLUE 80.5, MultiNLI 86.7% accuracy, SQuAD v1.1 test F1 93.2 and SQuAD v2.0 test F1 83.1, with BERT-large outperforming BERT-base across all GLUE tasks.

Abstract

We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).
Source: pdf

Extracted details

Field Value Source Confidence
contribution bidirectional pre-training via the masked language model objective llm 0.95
contribution single pre-trained architecture fine-tuned across sentence- and token-level tasks with minimal task-specific parameters llm 0.95
contribution state-of-the-art results on eleven NLP tasks llm 0.95
contribution release of BERT-base and BERT-large pre-trained models llm 0.95
libraries tensor2tensor llm 0.95
models BERT-base llm 0.95
models BERT-large llm 0.95
datasets GLUE llm 0.95
datasets MNLI llm 0.95
datasets QQP llm 0.95
datasets QNLI llm 0.95
datasets SST-2 llm 0.95
datasets CoLA llm 0.95
datasets STS-B llm 0.95
datasets MRPC llm 0.95
datasets RTE llm 0.95
datasets SQuAD v1.1 llm 0.95
datasets SQuAD v2.0 llm 0.95
datasets TriviaQA llm 0.95
tasks natural language understanding llm 0.95
tasks question answering llm 0.95
tasks natural language inference llm 0.95
tasks paraphrase and sentiment classification llm 0.95
hardware Cloud TPU llm 0.95
code url https://github.com/google-research/bert (URL found in the full text) regex 0.50
code url https://github.com/tensorflow/tensor2tensor (URL found in the full text) regex 0.50
limitations fine-tuning of BERT-large was sometimes unstable on small datasets, requiring random restarts llm 0.95
key result GLUE score: 80.5 (BERT-large leaderboard); Table 1 average 79.6 (base) / 82.1 (large) (GLUE test set) llm 0.95
key result MultiNLI accuracy: 86.7% (BERT-large, MNLI test) llm 0.95
key result SQuAD v1.1 test F1: 93.2 (BERT-large with TriviaQA augmentation) llm 0.95
key result SQuAD v2.0 test F1: 83.1 (BERT-large) llm 0.95
key result fine-tuning time: about 30 minutes on a single Cloud TPU (SQuAD, Dev F1 91.0%) llm 0.95

SoK reference [38]

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423/

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • model: BERT 80 [p.1], GPT 28 [p.1], Transformer 22 [p.1], RNN / LSTM 13 [p.11], Autoencoder 2 [p.3]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 37 [p.3], GLUE 20 [p.1], SQuAD 19 [p.1], SST-2 4 [p.6], ImageNet 1 [p.10]
  • hardware: TPU 7 [p.5], GPU 1 [p.5]
  • technique: Fine-tuning 11 [p.2], Attention 10 [p.1], Softmax 4 [p.4]
  • task: Question answering 6 [p.1], Language modeling / generation 6 [p.2], Named entity recognition 4 [p.1], Machine translation 3 [p.3]
  • topic: Benchmark 8 [p.2], Open source 2 [p.2]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/038_bert-pre-training-of-deep-bidirectional-transformers-for-lan.pdf sok-references 16 767.9 KB 987545ffb087

Database key: bert-pre-training-of-deep-bidirectional-transformers-for-language · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set bert-pre-training-of-deep-bidirectional-transformers-for-language COLUMN VALUE