GLUE: A multi-task benchmark and analysis platform for natural language understanding¶
Authors: A. Wang; A. Singh; J. Michael; F. Hill; O. Levy; S. Bowman
Year: 2018
type: benchmark collection: sok-references status: auto
Links: link
Summary¶
Auto-extracted summary (unreviewed)
GLUE is a benchmark of nine English sentence-understanding tasks (CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, RTE, WNLI) with an online leaderboard based partly on private test data and a 550-example hand-crafted diagnostic NLI set for probing linguistic phenomena. Baselines built on ELMo and other sentence encoders score fairly low (best multi-task average 69.0), and the authors conclude that solving GLUE is beyond current transfer-learning methods.
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | suite of nine NLU tasks spanning domains, data sizes and difficulty | llm | 0.90 |
| contribution | online evaluation platform and leaderboard with privately held test data | llm | 0.90 |
| contribution | expert-constructed diagnostic dataset for probing linguistic phenomena | llm | 0.90 |
| contribution | baseline evaluation of ELMo-based and sentence-representation models | llm | 0.90 |
| models | ELMo-based single-task and multi-task baselines | llm | 0.90 |
| models | CBoW | llm | 0.90 |
| models | Skip-Thought | llm | 0.90 |
| models | InferSent | llm | 0.90 |
| models | DisSent | llm | 0.90 |
| models | GenSen | llm | 0.90 |
| datasets | CoLA | llm | 0.90 |
| datasets | SST-2 | llm | 0.90 |
| datasets | MRPC | llm | 0.90 |
| datasets | QQP | llm | 0.90 |
| datasets | STS-B | llm | 0.90 |
| datasets | MNLI | llm | 0.90 |
| datasets | QNLI | llm | 0.90 |
| datasets | RTE | llm | 0.90 |
| datasets | WNLI | llm | 0.90 |
| datasets | GLUE diagnostic dataset | llm | 0.90 |
| tasks | natural language understanding | llm | 0.90 |
| tasks | sentiment classification | llm | 0.90 |
| tasks | paraphrase detection | llm | 0.90 |
| tasks | textual similarity | llm | 0.90 |
| tasks | natural language inference | llm | 0.90 |
| tasks | linguistic acceptability | llm | 0.90 |
| code url | https://gluebenchmark.com | llm | 0.90 |
| limitations | best baselines achieve fairly low absolute scores | llm | 0.90 |
| limitations | diagnostic analysis shows weak performance across most linguistic phenomena | llm | 0.90 |
| key result | GLUE average score: 69.0 (multi-task ELMo baseline) | llm | 0.90 |
| key result | GLUE average score: 64.8 (single-task ELMo baseline) | llm | 0.90 |
| key result | GLUE average score: 66.6 (GenSen sentence encoder) | llm | 0.90 |
SoK reference [41]¶
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, T. Linzen, G. Chrupała, and A. Alishahi, Eds. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 353–355. [Online]. Available: https://aclanthology.org/W18-5446/
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 28 [p.1], GLUE 13 [p.1], SST-2 3 [p.1], SQuAD 2 [p.2]
- topic: Benchmark 12 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/041_glue-a-multi-task-benchmark-and-analysis-platform-for-natura.pdf |
sok-references | 3 | 143.1 KB | 48fb7275b439 |
Database key: glue-a-multi-task-benchmark-and-analysis-platform-for-natural-language · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set glue-a-multi-task-benchmark-and-analysis-platform-for-natural-language COLUMN VALUE