Skip to content

HellaSwag: Can a machine really finish your sentence?

Authors: R. Zellers; A. Holtzman; Y. Bisk; A. Farhadi; Y. Choi
Year: 2019

type: dataset collection: sok-references status: auto

Links: link

Summary

Auto-extracted summary (unreviewed)

HellaSwag is a commonsense natural-language-inference challenge dataset built with Adversarial Filtering, in which discriminators iteratively select machine-generated wrong endings that are obviously wrong to humans yet hard for models; examples are scaled in length and complexity to a 'Goldilocks' zone. Humans exceed 95% accuracy while every tested model stays below 50% (BERT-Large 47.3%, ESIM+ELMo 33.3%), and transfer from SWAG to HellaSwag is poor (34.6%). The authors argue for benchmarks that co-evolve adversarially with state-of-the-art models.

Abstract

Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as “A woman sits at a piano,” a machine must select the most likely followup: “She sets her fingers on the keys.” With the introduction of BERT (Devlin et al., 2018), near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even stateof-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (°95% accuracy), state-of-the-art models struggle (†48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical ‘Goldilocks’ zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.
Source: pdf

Extracted details

Field Value Source Confidence
contribution HellaSwag dataset for physically situated commonsense reasoning llm 0.90
contribution Adversarial Filtering data collection with length/complexity scaling to a 'Goldilocks' zone llm 0.90
contribution evaluation showing a large human-model gap for pretrained models llm 0.90
contribution SWAG-to-HellaSwag transfer analysis and insights into pretrained models llm 0.90
models BERT-Large llm 0.90
models BERT-base llm 0.90
models OpenAI GPT llm 0.90
models ESIM+ELMo llm 0.90
models LSTM sentence encoder (GloVe/ELMo/BERT embeddings) llm 0.90
models FastText llm 0.90
datasets HellaSwag llm 0.90
datasets SWAG llm 0.90
datasets ActivityNet Captions llm 0.90
datasets WikiHow llm 0.90
datasets LSMDC llm 0.90
tasks commonsense natural language inference llm 0.90
tasks multiple-choice sentence completion llm 0.90
hardware Google Cloud TPU llm 0.90
code url https://rowanzellers.com/hellaswag llm 0.90
key result human accuracy: >95% (HellaSwag four-way multiple choice, majority vote of 5 workers) llm 0.90
key result BERT-Large accuracy: 47.3% (HellaSwag overall, best model) llm 0.90
key result ESIM+ELMo accuracy: 33.3% (vs 59% on SWAG) (HellaSwag overall) llm 0.90
key result BERT-Large accuracy on WikiHow subset: 45% (humans 96.5%) (HellaSwag WikiHow domain) llm 0.90
key result SWAG-trained BERT-Large on HellaSwag: 34.6% (transfer experiment, validation sets) llm 0.90

SoK reference [48]

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “HellaSwag: Can a machine really finish your sentence?” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. M`arquez, Eds. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 4791–4800. [Online]. Available: https://aclanthology.org/P19-1472/

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: Leveled / somewhat HE 3 [p.1]
  • model: BERT 62 [p.1], GPT 14 [p.1], RNN / LSTM 9 [p.4], MLP 2 [p.4], CNN 1 [p.4], Linear regression 1 [p.8]
  • dataset: HellaSwag 25 [p.1]
  • hardware: GPU 6 [p.8], NVIDIA RTX 2 [p.8], TPU 2 [p.8]
  • topic: Benchmark 9 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/048_hellaswag-can-a-machine-really-finish-your-sentence.pdf sok-references 10 1.5 MB ac3980adce5f

Database key: hellaswag-can-a-machine-really-finish-your-sentence · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set hellaswag-can-a-machine-really-finish-your-sentence COLUMN VALUE