Pointer sentinel mixture models¶
Authors: Stephen Merity; Caiming Xiong; James Bradbury; Richard Socher
Year: 2016
Venue: arXiv preprint
type: technique collection: sok-references status: auto
Links: arXiv:1609.07843
Summary¶
Auto-extracted summary (unreviewed)
Introduces the pointer sentinel mixture architecture for neural language models, which mixes a standard softmax vocabulary distribution with a pointer over recent context, letting the pointer component itself decide (via a sentinel) when to fall back to the softmax. A pointer sentinel-LSTM reaches 70.9 test perplexity on Penn Treebank, state of the art at the time, with less than a third of the parameters of large variational LSTMs. The paper also releases the WikiText-2 and WikiText-103 corpora for long-range language modeling. It is unrelated to cryptography or privacy-preserving computation.
Abstract¶
Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | Pointer sentinel mixture model for sequence prediction | llm | 0.90 |
| contribution | State-of-the-art Penn Treebank perplexity with far fewer parameters | llm | 0.90 |
| contribution | WikiText-2 and WikiText-103 language modeling datasets | llm | 0.90 |
| models | pointer sentinel-LSTM | llm | 0.90 |
| models | LSTM (2-layer, hidden 650) | llm | 0.90 |
| models | variational LSTM with zoneout | llm | 0.90 |
| datasets | Penn Treebank | llm | 0.90 |
| datasets | WikiText-2 | llm | 0.90 |
| datasets | WikiText-103 | llm | 0.90 |
| tasks | language modeling | llm | 0.90 |
| code url | https://github.com/yaringal/BayesianRNN (URL found in the full text) | regex | 0.50 |
| limitations | Training regenerates the window of RNN outputs after every gradient update, which slows training | llm | 0.90 |
| key result | test perplexity: 70.9 (Penn Treebank, pointer sentinel-LSTM (medium, H=650)) | llm | 0.90 |
| key result | parameter count: less than one third of large variational LSTM (Penn Treebank comparison) | llm | 0.90 |
SoK reference [46]¶
S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” 2016.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- model: RNN / LSTM 136 [p.1], MLP 1 [p.2]
- dataset: Penn Treebank 28 [p.1], WikiText 24 [p.1]
- hardware: GPU 3 [p.4]
- technique: Softmax 38 [p.1], Attention 2 [p.4]
- task: Language modeling / generation 21 [p.1], Machine translation 3 [p.5], Question answering 2 [p.5], Named entity recognition 2 [p.8], Finance 2 [p.12]
- topic: Benchmark 6 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/046_pointer-sentinel-mixture-models.pdf |
sok-references | 13 | 977.8 KB | 12e43c9ed8fa |
Database key: pointer-sentinel-mixture-models · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set pointer-sentinel-mixture-models COLUMN VALUE