Skip to content

Beyond memorization: Violating privacy via inference with large language models

Authors: Robin Staab; Mark Vero; Mislav Balunović; Martin Vechev
Year: 2024
Venue: arXiv preprint

type: attack collection: sok-references status: auto

Links: arXiv:2310.07298

Summary

Auto-extracted summary (unreviewed)

Studies whether pretrained LLMs can infer personal attributes (location, income, sex, age, occupation, etc.) from free text supplied at inference time, using a newly constructed dataset of real Reddit profiles (PersonalReddit). GPT-4 reaches up to 85% top-1 and 95.2% top-3 accuracy at roughly 100x lower cost and 240x less time than human labelers; the paper also explores privacy-invasive chatbots and shows that text anonymization and model alignment are ineffective mitigations.

Abstract

Current privacy research on large language models (LLMs) primarily focuses on the issue of extracting memorized training data. At the same time, models' inference capabilities have increased drastically. This raises the key question of whether current LLMs could violate individuals' privacy by inferring personal attributes from text given at inference time. In this work, we present the first comprehensive study on the capabilities of pretrained LLMs to infer personal attributes from text. We construct a dataset consisting of real Reddit profiles, and show that current LLMs can infer a wide range of personal attributes (e.g., location, income, sex), achieving up to $85\%$ top-1 and $95\%$ top-3 accuracy at a fraction of the cost ($100\times$) and time ($240\times$) required by humans. As people increasingly interact with LLM-powered chatbots across all aspects of life, we also explore the emerging threat of privacy-invasive chatbots trying to extract personal information through seemingly benign questions. Finally, we show that common mitigations, i.e., text anonymization and model alignment, are currently ineffective at protecting user privacy against LLM inference. Our findings highlight that current LLMs can infer personal data at a previously unattainable scale. In the absence of working defenses, we advocate for a broader discussion around LLM privacy implications beyond memorization, striving for a wider privacy protection.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution first comprehensive study of LLM inference of personal attributes from text llm 0.85
contribution PersonalReddit dataset built from real Reddit profiles with human labels llm 0.85
contribution evaluation of nine LLMs (GPT-4, GPT-3.5, Llama-2, PaLM 2, Claude) on attribute inference llm 0.85
contribution exploration of privacy-invasive chatbots that extract personal information via benign-looking questions llm 0.85
contribution shows anonymization and alignment do not protect against LLM inference llm 0.85
contribution release of code, prompts and synthetic chatlogs llm 0.85
models GPT-4 llm 0.85
models GPT-3.5 llm 0.85
models Llama-2-7B llm 0.85
models Llama-2-13B llm 0.85
models Llama-2-70B llm 0.85
models PaLM 2 Text llm 0.85
models PaLM 2 Chat llm 0.85
models Claude-2 llm 0.85
models Claude-Instant llm 0.85
models XGBoost llm 0.85
datasets PersonalReddit llm 0.85
datasets ACSIncome llm 0.85
tasks personal attribute inference from text llm 0.85
tasks author profiling llm 0.85
tasks LLM privacy attack llm 0.85
threat model adversary with LLM access inferring personal attributes from user-written text; privacy-invasive chatbot llm 0.85
code url https://github.com/eth-sri/llmprivacy (URL found in the full text) regex 0.50
code url https://github.com/google-research/lm-extraction-benchmark (URL found in the full text) regex 0.50
code url https://huggingface.co/docs/hub/models-the-hub.Daphne (URL found in the full text) regex 0.50
limitations models other than GPT-4 often failed to follow the output format, requiring GPT-4-based reformatting and manual parsing llm 0.85
limitations free-text answers required GPT-4 and human judgement to match against ground truth llm 0.85
llm relevance low llm 0.85
key result top-1 attribute inference accuracy: 85% (GPT-4 on PersonalReddit) llm 0.85
key result top-3 attribute inference accuracy: 95.2% (GPT-4 on PersonalReddit, labels with minimum certainty 3) llm 0.85
key result cost reduction vs human labelers: 100x (LLM inference vs human profilers) llm 0.85
key result time reduction vs human labelers: 240x (LLM inference vs human profilers) llm 0.85

SoK reference [4]

R. Staab, M. Vero, M. Balunovic, and M. Vechev, “Beyond memorization: Violating privacy via inference with large language models,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/ forum?id=kmn0BhQk7p

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • library: HuggingFace 1 [p.12]
  • model: GPT 50 [p.1], LLaMA 11 [p.2]
  • technique: Fine-tuning 6 [p.3]
  • task: Finance 11 [p.2]
  • topic: Large language models 108 [p.1], Open source 3 [p.3], Benchmark 2 [p.10]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/004_beyond-memorization-violating-privacy-via-inference-with-lar.pdf sok-references 47 3.3 MB 1acb05f14849

Database key: beyond-memorization-violating-privacy-via-inference-with-large-language · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set beyond-memorization-violating-privacy-via-inference-with-large-language COLUMN VALUE