HellaSwag: Can a machine really finish your sentence?¶
Authors: R. Zellers; A. Holtzman; Y. Bisk; A. Farhadi; Y. Choi
Year: 2019
type: dataset collection: sok-references status: auto
Links: link
Summary¶
Auto-extracted summary (unreviewed)
HellaSwag is a commonsense natural-language-inference challenge dataset built with Adversarial Filtering, in which discriminators iteratively select machine-generated wrong endings that are obviously wrong to humans yet hard for models; examples are scaled in length and complexity to a 'Goldilocks' zone. Humans exceed 95% accuracy while every tested model stays below 50% (BERT-Large 47.3%, ESIM+ELMo 33.3%), and transfer from SWAG to HellaSwag is poor (34.6%). The authors argue for benchmarks that co-evolve adversarially with state-of-the-art models.
Abstract¶
Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as “A woman sits at a piano,” a machine must select the most likely followup: “She sets her fingers on the keys.” With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (°95% accuracy), state-of-the-art models struggle (†48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical ‘Goldilocks’ zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Source: pdf
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | HellaSwag dataset for physically situated commonsense reasoning | llm | 0.90 |
| contribution | Adversarial Filtering data collection with length/complexity scaling to a 'Goldilocks' zone | llm | 0.90 |
| contribution | evaluation showing a large human-model gap for pretrained models | llm | 0.90 |
| contribution | SWAG-to-HellaSwag transfer analysis and insights into pretrained models | llm | 0.90 |
| models | BERT-Large | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| models | OpenAI GPT | llm | 0.90 |
| models | ESIM+ELMo | llm | 0.90 |
| models | LSTM sentence encoder (GloVe/ELMo/BERT embeddings) | llm | 0.90 |
| models | FastText | llm | 0.90 |
| datasets | HellaSwag | llm | 0.90 |
| datasets | SWAG | llm | 0.90 |
| datasets | ActivityNet Captions | llm | 0.90 |
| datasets | WikiHow | llm | 0.90 |
| datasets | LSMDC | llm | 0.90 |
| tasks | commonsense natural language inference | llm | 0.90 |
| tasks | multiple-choice sentence completion | llm | 0.90 |
| hardware | Google Cloud TPU | llm | 0.90 |
| code url | https://rowanzellers.com/hellaswag | llm | 0.90 |
| key result | human accuracy: >95% (HellaSwag four-way multiple choice, majority vote of 5 workers) | llm | 0.90 |
| key result | BERT-Large accuracy: 47.3% (HellaSwag overall, best model) | llm | 0.90 |
| key result | ESIM+ELMo accuracy: 33.3% (vs 59% on SWAG) (HellaSwag overall) | llm | 0.90 |
| key result | BERT-Large accuracy on WikiHow subset: 45% (humans 96.5%) (HellaSwag WikiHow domain) | llm | 0.90 |
| key result | SWAG-trained BERT-Large on HellaSwag: 34.6% (transfer experiment, validation sets) | llm | 0.90 |
SoK reference [48]¶
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “HellaSwag: Can a machine really finish your sentence?” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. M`arquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 4791–4800. [Online]. Available: https://aclanthology.org/P19-1472/
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: Leveled / somewhat HE 3 [p.1]
- model: BERT 62 [p.1], GPT 14 [p.1], RNN / LSTM 9 [p.4], MLP 2 [p.4], CNN 1 [p.4], Linear regression 1 [p.8]
- dataset: HellaSwag 25 [p.1]
- hardware: GPU 6 [p.8], NVIDIA RTX 2 [p.8], TPU 2 [p.8]
- topic: Benchmark 9 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/048_hellaswag-can-a-machine-really-finish-your-sentence.pdf |
sok-references | 10 | 1.5 MB | ac3980adce5f |
Database key: hellaswag-can-a-machine-really-finish-your-sentence · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set hellaswag-can-a-machine-really-finish-your-sentence COLUMN VALUE