Skip to content

Pointer sentinel mixture models

Authors: Stephen Merity; Caiming Xiong; James Bradbury; Richard Socher
Year: 2016
Venue: arXiv preprint

type: technique collection: sok-references status: auto

Links: arXiv:1609.07843

Summary

Auto-extracted summary (unreviewed)

Introduces the pointer sentinel mixture architecture for neural language models, which mixes a standard softmax vocabulary distribution with a pointer over recent context, letting the pointer component itself decide (via a sentinel) when to fall back to the softmax. A pointer sentinel-LSTM reaches 70.9 test perplexity on Penn Treebank, state of the art at the time, with less than a third of the parameters of large variational LSTMs. The paper also releases the WikiText-2 and WikiText-103 corpora for long-range language modeling. It is unrelated to cryptography or privacy-preserving computation.

Abstract

Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution Pointer sentinel mixture model for sequence prediction llm 0.90
contribution State-of-the-art Penn Treebank perplexity with far fewer parameters llm 0.90
contribution WikiText-2 and WikiText-103 language modeling datasets llm 0.90
models pointer sentinel-LSTM llm 0.90
models LSTM (2-layer, hidden 650) llm 0.90
models variational LSTM with zoneout llm 0.90
datasets Penn Treebank llm 0.90
datasets WikiText-2 llm 0.90
datasets WikiText-103 llm 0.90
tasks language modeling llm 0.90
code url https://github.com/yaringal/BayesianRNN (URL found in the full text) regex 0.50
limitations Training regenerates the window of RNN outputs after every gradient update, which slows training llm 0.90
key result test perplexity: 70.9 (Penn Treebank, pointer sentinel-LSTM (medium, H=650)) llm 0.90
key result parameter count: less than one third of large variational LSTM (Penn Treebank comparison) llm 0.90

SoK reference [46]

S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” 2016.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • model: RNN / LSTM 136 [p.1], MLP 1 [p.2]
  • dataset: Penn Treebank 28 [p.1], WikiText 24 [p.1]
  • hardware: GPU 3 [p.4]
  • technique: Softmax 38 [p.1], Attention 2 [p.4]
  • task: Language modeling / generation 21 [p.1], Machine translation 3 [p.5], Question answering 2 [p.5], Named entity recognition 2 [p.8], Finance 2 [p.12]
  • topic: Benchmark 6 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/046_pointer-sentinel-mixture-models.pdf sok-references 13 977.8 KB 12e43c9ed8fa

Database key: pointer-sentinel-mixture-models · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set pointer-sentinel-mixture-models COLUMN VALUE