Beyond memorization: Violating privacy via inference with large language models¶
Authors: Robin Staab; Mark Vero; Mislav Balunović; Martin Vechev
Year: 2024
Venue: arXiv preprint
type: attack collection: sok-references status: auto
Links: arXiv:2310.07298
Summary¶
Auto-extracted summary (unreviewed)
Studies whether pretrained LLMs can infer personal attributes (location, income, sex, age, occupation, etc.) from free text supplied at inference time, using a newly constructed dataset of real Reddit profiles (PersonalReddit). GPT-4 reaches up to 85% top-1 and 95.2% top-3 accuracy at roughly 100x lower cost and 240x less time than human labelers; the paper also explores privacy-invasive chatbots and shows that text anonymization and model alignment are ineffective mitigations.
Abstract¶
Current privacy research on large language models (LLMs) primarily focuses on the issue of extracting memorized training data. At the same time, models' inference capabilities have increased drastically. This raises the key question of whether current LLMs could violate individuals' privacy by inferring personal attributes from text given at inference time. In this work, we present the first comprehensive study on the capabilities of pretrained LLMs to infer personal attributes from text. We construct a dataset consisting of real Reddit profiles, and show that current LLMs can infer a wide range of personal attributes (e.g., location, income, sex), achieving up to $85\%$ top-1 and $95\%$ top-3 accuracy at a fraction of the cost ($100\times$) and time ($240\times$) required by humans. As people increasingly interact with LLM-powered chatbots across all aspects of life, we also explore the emerging threat of privacy-invasive chatbots trying to extract personal information through seemingly benign questions. Finally, we show that common mitigations, i.e., text anonymization and model alignment, are currently ineffective at protecting user privacy against LLM inference. Our findings highlight that current LLMs can infer personal data at a previously unattainable scale. In the absence of working defenses, we advocate for a broader discussion around LLM privacy implications beyond memorization, striving for a wider privacy protection.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | first comprehensive study of LLM inference of personal attributes from text | llm | 0.85 |
| contribution | PersonalReddit dataset built from real Reddit profiles with human labels | llm | 0.85 |
| contribution | evaluation of nine LLMs (GPT-4, GPT-3.5, Llama-2, PaLM 2, Claude) on attribute inference | llm | 0.85 |
| contribution | exploration of privacy-invasive chatbots that extract personal information via benign-looking questions | llm | 0.85 |
| contribution | shows anonymization and alignment do not protect against LLM inference | llm | 0.85 |
| contribution | release of code, prompts and synthetic chatlogs | llm | 0.85 |
| models | GPT-4 | llm | 0.85 |
| models | GPT-3.5 | llm | 0.85 |
| models | Llama-2-7B | llm | 0.85 |
| models | Llama-2-13B | llm | 0.85 |
| models | Llama-2-70B | llm | 0.85 |
| models | PaLM 2 Text | llm | 0.85 |
| models | PaLM 2 Chat | llm | 0.85 |
| models | Claude-2 | llm | 0.85 |
| models | Claude-Instant | llm | 0.85 |
| models | XGBoost | llm | 0.85 |
| datasets | PersonalReddit | llm | 0.85 |
| datasets | ACSIncome | llm | 0.85 |
| tasks | personal attribute inference from text | llm | 0.85 |
| tasks | author profiling | llm | 0.85 |
| tasks | LLM privacy attack | llm | 0.85 |
| threat model | adversary with LLM access inferring personal attributes from user-written text; privacy-invasive chatbot | llm | 0.85 |
| code url | https://github.com/eth-sri/llmprivacy (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/google-research/lm-extraction-benchmark (URL found in the full text) | regex | 0.50 |
| code url | https://huggingface.co/docs/hub/models-the-hub.Daphne (URL found in the full text) | regex | 0.50 |
| limitations | models other than GPT-4 often failed to follow the output format, requiring GPT-4-based reformatting and manual parsing | llm | 0.85 |
| limitations | free-text answers required GPT-4 and human judgement to match against ground truth | llm | 0.85 |
| llm relevance | low | llm | 0.85 |
| key result | top-1 attribute inference accuracy: 85% (GPT-4 on PersonalReddit) | llm | 0.85 |
| key result | top-3 attribute inference accuracy: 95.2% (GPT-4 on PersonalReddit, labels with minimum certainty 3) | llm | 0.85 |
| key result | cost reduction vs human labelers: 100x (LLM inference vs human profilers) | llm | 0.85 |
| key result | time reduction vs human labelers: 240x (LLM inference vs human profilers) | llm | 0.85 |
SoK reference [4]¶
R. Staab, M. Vero, M. Balunovic, and M. Vechev, “Beyond memorization: Violating privacy via inference with large language models,” in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/ forum?id=kmn0BhQk7p
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- library: HuggingFace 1 [p.12]
- model: GPT 50 [p.1], LLaMA 11 [p.2]
- technique: Fine-tuning 6 [p.3]
- task: Finance 11 [p.2]
- topic: Large language models 108 [p.1], Open source 3 [p.3], Benchmark 2 [p.10]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/004_beyond-memorization-violating-privacy-via-inference-with-lar.pdf |
sok-references | 47 | 3.3 MB | 1acb05f14849 |
Database key: beyond-memorization-violating-privacy-via-inference-with-large-language · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set beyond-memorization-violating-privacy-via-inference-with-large-language COLUMN VALUE