Skip to content

GLUE: A multi-task benchmark and analysis platform for natural language understanding

Authors: A. Wang; A. Singh; J. Michael; F. Hill; O. Levy; S. Bowman
Year: 2018

type: benchmark collection: sok-references status: auto

Links: link

Summary

Auto-extracted summary (unreviewed)

GLUE is a benchmark of nine English sentence-understanding tasks (CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, RTE, WNLI) with an online leaderboard based partly on private test data and a 550-example hand-crafted diagnostic NLI set for probing linguistic phenomena. Baselines built on ELMo and other sentence encoders score fairly low (best multi-task average 69.0), and the authors conclude that solving GLUE is beyond current transfer-learning methods.

Extracted details

Field Value Source Confidence
contribution suite of nine NLU tasks spanning domains, data sizes and difficulty llm 0.90
contribution online evaluation platform and leaderboard with privately held test data llm 0.90
contribution expert-constructed diagnostic dataset for probing linguistic phenomena llm 0.90
contribution baseline evaluation of ELMo-based and sentence-representation models llm 0.90
models ELMo-based single-task and multi-task baselines llm 0.90
models CBoW llm 0.90
models Skip-Thought llm 0.90
models InferSent llm 0.90
models DisSent llm 0.90
models GenSen llm 0.90
datasets CoLA llm 0.90
datasets SST-2 llm 0.90
datasets MRPC llm 0.90
datasets QQP llm 0.90
datasets STS-B llm 0.90
datasets MNLI llm 0.90
datasets QNLI llm 0.90
datasets RTE llm 0.90
datasets WNLI llm 0.90
datasets GLUE diagnostic dataset llm 0.90
tasks natural language understanding llm 0.90
tasks sentiment classification llm 0.90
tasks paraphrase detection llm 0.90
tasks textual similarity llm 0.90
tasks natural language inference llm 0.90
tasks linguistic acceptability llm 0.90
code url https://gluebenchmark.com llm 0.90
limitations best baselines achieve fairly low absolute scores llm 0.90
limitations diagnostic analysis shows weak performance across most linguistic phenomena llm 0.90
key result GLUE average score: 69.0 (multi-task ELMo baseline) llm 0.90
key result GLUE average score: 64.8 (single-task ELMo baseline) llm 0.90
key result GLUE average score: 66.6 (GenSen sentence encoder) llm 0.90

SoK reference [41]

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, T. Linzen, G. Chrupała, and A. Alishahi, Eds. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 353–355. [Online]. Available: https://aclanthology.org/W18-5446/

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 28 [p.1], GLUE 13 [p.1], SST-2 3 [p.1], SQuAD 2 [p.2]
  • topic: Benchmark 12 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/041_glue-a-multi-task-benchmark-and-analysis-platform-for-natura.pdf sok-references 3 143.1 KB 48fb7275b439

Database key: glue-a-multi-task-benchmark-and-analysis-platform-for-natural-language · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set glue-a-multi-task-benchmark-and-analysis-platform-for-natural-language COLUMN VALUE