Skip to content

SQuAD: 100,000+ questions for machine comprehension of text

Authors: Pranav Rajpurkar; Jian Zhang; Konstantin Lopyrev; Percy Liang
Year: 2016
Venue: arXiv preprint

type: dataset collection: sok-references status: auto

Links: arXiv:1606.05250

Summary

Auto-extracted summary (unreviewed)

The paper introduces SQuAD, a reading-comprehension dataset of 107,785 crowdsourced question-answer pairs over 536 Wikipedia articles in which every answer is a text span of the passage, and analyzes the reasoning types required using dependency and constituency trees. A logistic-regression baseline reaches 51.0% F1 (40.4% exact match) on the test set, far below human performance of 86.8% F1 (77.0% exact match), establishing the dataset as a challenge benchmark.

Abstract

We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at https://stanford-qa.com
Source: arxiv

Extracted details

Field Value Source Confidence
contribution SQuAD dataset of 100,000+ span-based QA pairs on Wikipedia articles llm 0.90
contribution analysis of answer types and reasoning / syntactic divergence using dependency and constituency trees llm 0.90
contribution logistic regression baseline with feature ablations llm 0.90
contribution measurement of human performance on dev and test sets llm 0.90
models logistic regression llm 0.90
models sliding-window baseline llm 0.90
datasets SQuAD llm 0.90
tasks reading comprehension llm 0.90
tasks extractive question answering llm 0.90
code url https://worksheets.codalab.org/worksheets/0xd53d03a48ef64b329c16b9baf0f99b0c/ llm 0.90
key result dataset size: 107,785 QA pairs on 536 articles (SQuAD v1.0) llm 0.90
key result F1: 51.0% (logistic regression, test set) llm 0.90
key result exact match: 40.4% (logistic regression, test set) llm 0.90
key result F1: 86.8% (human performance, test set) llm 0.90
key result exact match: 77.0% (human performance, test set) llm 0.90
key result sentence selection accuracy: 79.3% (logistic regression selects the sentence containing the answer) llm 0.90

SoK reference [44]

P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras, Eds. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 2383–2392. [Online]. Available: https://aclanthology.org/ D16-1264/

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • model: Logistic regression 16 [p.1], CNN 3 [p.2], RNN / LSTM 1 [p.10]
  • dataset: SQuAD 21 [p.1], ImageNet 2 [p.1], Penn Treebank 2 [p.1]
  • task: Question answering 7 [p.2]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/044_squad-100-000-questions-for-machine-comprehension-of-text.pdf sok-references 10 446.1 KB 5169f9424ad6

Database key: squad-100-000-questions-for-machine-comprehension-of-text · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set squad-100-000-questions-for-machine-comprehension-of-text COLUMN VALUE