Think you have solved question answering? try ARC, the AI2 reasoning challenge¶
Authors: Peter Clark; Isaac Cowhey; Oren Etzioni; Tushar Khot; Ashish Sabharwal; Carissa Schoenick; Oyvind Tafjord
Year: 2018
Venue: arXiv preprint
type: dataset collection: sok-references status: auto
Links: arXiv:1803.05457
Summary¶
Auto-extracted summary (unreviewed)
Presents the AI2 Reasoning Challenge (ARC): 7,787 natural grade-school science questions split into a Challenge Set (2,590 questions that both a retrieval-based and a PMI word co-occurrence solver answer incorrectly) and an Easy Set (5,197), together with a 14M-sentence ARC Corpus and three neural baselines (DecompAttn, BiDAF, DGEM). None of the baselines, including models strong on SQuAD and SNLI, significantly outperform random guessing on the Challenge Set.
Abstract¶
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | ARC question set of 7,787 public-domain science exam questions | llm | 0.90 |
| contribution | Challenge/Easy partition defined by failure of retrieval and co-occurrence solvers | llm | 0.90 |
| contribution | ARC Corpus of 14M science-related sentences (1.4 GB) | llm | 0.90 |
| contribution | Baseline implementations (DecompAttn, BiDAF, DGEM) and a public leaderboard | llm | 0.90 |
| contribution | Taxonomy of knowledge and reasoning types appearing in Challenge questions | llm | 0.90 |
| models | DecompAttn | llm | 0.90 |
| models | BiDAF | llm | 0.90 |
| models | DGEM | llm | 0.90 |
| models | retrieval-based (IR) solver | llm | 0.90 |
| models | PMI word co-occurrence solver | llm | 0.90 |
| datasets | ARC | llm | 0.90 |
| datasets | ARC Corpus | llm | 0.90 |
| tasks | multiple-choice question answering | llm | 0.90 |
| tasks | science question answering | llm | 0.90 |
| tasks | reasoning | llm | 0.90 |
| code url | https://github.com/allenai/arc-solvers (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/allenai/arc-solvers | llm | 0.90 |
| limitations | Knowledge and reasoning type categorization is approximate and based on a subjective sample of 100 questions | llm | 0.90 |
| limitations | Only about 75% of sampled corpus documents were judged science-relevant | llm | 0.90 |
| key result | dataset size: 7,787 questions (Challenge 2,590; Easy 5,197) (ARC question set) | llm | 0.90 |
| key result | corpus size: 14M sentences (1.4 GB) (ARC Corpus) | llm | 0.90 |
| key result | baseline accuracy: not significantly above random (all tested baselines on the Challenge Set) | llm | 0.90 |
SoK reference [49]¶
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try ARC, the AI2 reasoning challenge,” arXiv:1803.05457v1, 2018.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- model: CNN 1 [p.2]
- dataset: ARC 62 [p.1], SQuAD 8 [p.1]
- task: Question answering 7 [p.1]
- topic: Open source 2 [p.7]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/049_think-you-have-solved-question-answering-try-arc-the-ai2-rea.pdf |
sok-references | 10 | 283.8 KB | b24b722c93f3 |
Database key: think-you-have-solved-question-answering-try-arc-the-ai2-reasoning · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set think-you-have-solved-question-answering-try-arc-the-ai2-reasoning COLUMN VALUE