PUMA: Secure Inference of LLaMA-7B in Five Minutes¶
Authors: Ye Dong; Wen-jie Lu; Yancheng Zheng; Haoqi Wu; Derun Zhao; Jin Tan; Zhicong Huang; Cheng Hong; Tao Wei; Wenguang Chen
Year: 2023
Venue: arXiv 2023
type: system collection: catalogue status: auto
Links: DOI · arXiv:2307.12533
Summary¶
Auto-extracted summary (unreviewed)
PUMA is a three-party MPC framework (replicated secret sharing on SecretFlow-SPU) for secure Transformer inference that keeps both prompts and model weights private. It uses accurate polynomial approximations of GeLU and softmax plus faithful secure Embedding and LayerNorm protocols, so pretrained models run without fine-tuning: accuracy matches plaintext within 0.011 on GLUE tasks and perplexity within 0.02 on Wikitext-103, it is roughly 2x faster than MPCFormer, and LLaMA-7B generates one token in about 5 minutes.
Abstract¶
With ChatGPT as a representative, tons of companies have began to provide services based on large Transformers models. However, using such a service inevitably leak users' prompts to the model provider. Previous studies have studied secure inference for Transformer models using secure multiparty computation (MPC), where model parameters and clients' prompts are kept secret. Despite this, these frameworks are still limited in terms of model performance, efficiency, and deployment. To address these limitations, we propose framework PUMA to enable fast and secure Transformer model inference. Our framework designs high quality approximations for expensive functions such as GeLU and softmax, and significantly reduce the cost of secure inference while preserving the model performance. Additionally, we design secure Embedding and LayerNorm procedures that faithfully implement the desired functionality without undermining the Transformer architecture. PUMA is about $2\times$ faster than the state-of-the-art framework MPCFORMER(ICLR 2023) and has similar accuracy as plaintext models without fine-tuning (which the previous works failed to achieve). PUMA can even evaluate LLaMA-7B in around 5 minutes to generate 1 token. To our best knowledge, this is the first time that a model with such a parameter size is able to be evaluated under MPC. PUMA has been open-sourced in the Github repository of SecretFlow-SPU.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | end-to-end secure Transformer inference framework PUMA on replicated secret sharing (3PC) | llm | 0.90 |
| contribution | high-quality polynomial approximations for GeLU and softmax that preserve model accuracy without retraining | llm | 0.90 |
| contribution | secure Embedding and LayerNorm protocols faithful to the Transformer architecture | llm | 0.90 |
| contribution | first MPC evaluation of LLaMA-7B (one token in ~5 minutes); open-sourced in SecretFlow-SPU | llm | 0.90 |
| schemes | secret sharing | llm | 0.90 |
| schemes | 2-out-of-3 replicated secret sharing | llm | 0.90 |
| libraries | SecretFlow-SPU | llm | 0.90 |
| libraries | Flax | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| models | RoBERTa-base | llm | 0.90 |
| models | BERT-large | llm | 0.90 |
| models | GPT-2 base | llm | 0.90 |
| models | GPT-2 medium | llm | 0.90 |
| models | GPT-2 large | llm | 0.90 |
| models | LLaMA-7B | llm | 0.90 |
| datasets | CoLA | llm | 0.90 |
| datasets | RTE | llm | 0.90 |
| datasets | QNLI | llm | 0.90 |
| datasets | GLUE | llm | 0.90 |
| datasets | Wikitext-103 | llm | 0.90 |
| tasks | private transformer inference | llm | 0.90 |
| tasks | LLM inference | llm | 0.90 |
| tasks | text classification | llm | 0.90 |
| tasks | language modelling | llm | 0.90 |
| tasks | token generation | llm | 0.90 |
| hardware | 3x Alibaba Cloud ecs.g7.8xlarge (Intel Xeon Platinum 8369B, 32 vCPU, 128 GB RAM) | llm | 0.90 |
| hardware | 5 Gbps LAN, 1 ms RTT | llm | 0.90 |
| threat model | semi-honest, honest-majority three-party computation | llm | 0.90 |
| interaction | pure MPC | llm | 0.90 |
| code url | https://github.com/secretflow/spu/tree/main/examples/python/ml/flax_llama7b (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/secretflow/spu/tree/main/examples/python/ml/flax_llama7b | llm | 0.90 |
| limitations | inference cost is still quite high; combining with quantization and hardware acceleration left for future work | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | latency: ~5 minutes (LLaMA-7B generating 1 token under 3PC) | llm | 0.90 |
| key result | speed-up vs MPCFormer: ~2x (1.63x-2.41x) (BERT and GPT-2 models, LAN) | llm | 0.90 |
| key result | latency / communication: 33.9 s / 10.77 GB (BERT-base, one sentence of length 128) | llm | 0.90 |
| key result | latency / communication: 15.5 s / 3.77 GB (GPT-2 base, input length 32, generating 1 token) | llm | 0.90 |
| key result | accuracy gap vs plaintext: <= 0.011 (BERT-base/RoBERTa-base/BERT-large on CoLA, RTE, QNLI) | llm | 0.90 |
| key result | perplexity gap vs plaintext: <= 0.02 (GPT-2 base/medium/large on Wikitext-103) | llm | 0.90 |
System card: PUMA¶
| Group | 5. Interactive MPC and HE+MPC hybrids |
| Family | Replicated secret sharing 3PC |
| Arithmetic | Fixed point |
| Interaction | Interactive 3PC (honest majority) |
| Model scale | LLaMA-7B |
| Status | Active SOTA |
| Relation to the SoK | Model-preserving 3PC; first private LLaMA-7B inference. |
| Note | About five minutes per token for LLaMA-7B. Needs three non-colluding servers. |
| Code | https://github.com/secretflow/spu |
| Systems page | PUMA |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: MPC 35 [p.1], Secret sharing 16 [p.3], Differential privacy 2 [p.4]
- library: SecretFlow-SPU 7 [p.1], CrypTen 2 [p.3], HuggingFace 1 [p.12]
- model: GPT 41 [p.3], Transformer 35 [p.1], LLaMA 28 [p.1], RNN / LSTM 1 [p.11], Autoencoder 1 [p.10]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 17 [p.7], GLUE 3 [p.7]
- hardware: Intel Xeon 1 [p.6]
- technique: GELU 21 [p.1], Softmax 18 [p.1], LayerNorm 17 [p.1], Three-party computation 5 [p.2], Sigmoid 2 [p.4], Two-party computation 2 [p.10], Semi-honest model 2 [p.4], Malicious model 2 [p.10]
- task: Language modeling / generation 2 [p.10]
- topic: Private / secure inference 32 [p.1], Privacy-preserving machine learning 4 [p.11], Open source 4 [p.1], Large language models 3 [p.2], Benchmark 3 [p.7]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/5-068_puma-secure-inference-of-llama-7b-in-five-minutes.pdf |
catalogue | 13 | 259.3 KB | 84c82810a16b |
Database key: puma-secure-inference-of-llama-7b-in-five-minutes · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set puma-secure-inference-of-llama-7b-in-five-minutes COLUMN VALUE