Skip to content

PUMA: Secure Inference of LLaMA-7B in Five Minutes

Authors: Ye Dong; Wen-jie Lu; Yancheng Zheng; Haoqi Wu; Derun Zhao; Jin Tan; Zhicong Huang; Cheng Hong; Tao Wei; Wenguang Chen
Year: 2023
Venue: arXiv 2023

type: system collection: catalogue status: auto

Links: DOI · arXiv:2307.12533

Summary

Auto-extracted summary (unreviewed)

PUMA is a three-party MPC framework (replicated secret sharing on SecretFlow-SPU) for secure Transformer inference that keeps both prompts and model weights private. It uses accurate polynomial approximations of GeLU and softmax plus faithful secure Embedding and LayerNorm protocols, so pretrained models run without fine-tuning: accuracy matches plaintext within 0.011 on GLUE tasks and perplexity within 0.02 on Wikitext-103, it is roughly 2x faster than MPCFormer, and LLaMA-7B generates one token in about 5 minutes.

Abstract

With ChatGPT as a representative, tons of companies have began to provide services based on large Transformers models. However, using such a service inevitably leak users' prompts to the model provider. Previous studies have studied secure inference for Transformer models using secure multiparty computation (MPC), where model parameters and clients' prompts are kept secret. Despite this, these frameworks are still limited in terms of model performance, efficiency, and deployment. To address these limitations, we propose framework PUMA to enable fast and secure Transformer model inference. Our framework designs high quality approximations for expensive functions such as GeLU and softmax, and significantly reduce the cost of secure inference while preserving the model performance. Additionally, we design secure Embedding and LayerNorm procedures that faithfully implement the desired functionality without undermining the Transformer architecture. PUMA is about $2\times$ faster than the state-of-the-art framework MPCFORMER(ICLR 2023) and has similar accuracy as plaintext models without fine-tuning (which the previous works failed to achieve). PUMA can even evaluate LLaMA-7B in around 5 minutes to generate 1 token. To our best knowledge, this is the first time that a model with such a parameter size is able to be evaluated under MPC. PUMA has been open-sourced in the Github repository of SecretFlow-SPU.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution end-to-end secure Transformer inference framework PUMA on replicated secret sharing (3PC) llm 0.90
contribution high-quality polynomial approximations for GeLU and softmax that preserve model accuracy without retraining llm 0.90
contribution secure Embedding and LayerNorm protocols faithful to the Transformer architecture llm 0.90
contribution first MPC evaluation of LLaMA-7B (one token in ~5 minutes); open-sourced in SecretFlow-SPU llm 0.90
schemes secret sharing llm 0.90
schemes 2-out-of-3 replicated secret sharing llm 0.90
libraries SecretFlow-SPU llm 0.90
libraries Flax llm 0.90
models BERT-base llm 0.90
models RoBERTa-base llm 0.90
models BERT-large llm 0.90
models GPT-2 base llm 0.90
models GPT-2 medium llm 0.90
models GPT-2 large llm 0.90
models LLaMA-7B llm 0.90
datasets CoLA llm 0.90
datasets RTE llm 0.90
datasets QNLI llm 0.90
datasets GLUE llm 0.90
datasets Wikitext-103 llm 0.90
tasks private transformer inference llm 0.90
tasks LLM inference llm 0.90
tasks text classification llm 0.90
tasks language modelling llm 0.90
tasks token generation llm 0.90
hardware 3x Alibaba Cloud ecs.g7.8xlarge (Intel Xeon Platinum 8369B, 32 vCPU, 128 GB RAM) llm 0.90
hardware 5 Gbps LAN, 1 ms RTT llm 0.90
threat model semi-honest, honest-majority three-party computation llm 0.90
interaction pure MPC llm 0.90
code url https://github.com/secretflow/spu/tree/main/examples/python/ml/flax_llama7b (URL found in the full text) regex 0.50
code url https://github.com/secretflow/spu/tree/main/examples/python/ml/flax_llama7b llm 0.90
limitations inference cost is still quite high; combining with quantization and hardware acceleration left for future work llm 0.90
llm relevance high llm 0.90
key result latency: ~5 minutes (LLaMA-7B generating 1 token under 3PC) llm 0.90
key result speed-up vs MPCFormer: ~2x (1.63x-2.41x) (BERT and GPT-2 models, LAN) llm 0.90
key result latency / communication: 33.9 s / 10.77 GB (BERT-base, one sentence of length 128) llm 0.90
key result latency / communication: 15.5 s / 3.77 GB (GPT-2 base, input length 32, generating 1 token) llm 0.90
key result accuracy gap vs plaintext: <= 0.011 (BERT-base/RoBERTa-base/BERT-large on CoLA, RTE, QNLI) llm 0.90
key result perplexity gap vs plaintext: <= 0.02 (GPT-2 base/medium/large on Wikitext-103) llm 0.90

System card: PUMA

Group 5. Interactive MPC and HE+MPC hybrids
Family Replicated secret sharing 3PC
Arithmetic Fixed point
Interaction Interactive 3PC (honest majority)
Model scale LLaMA-7B
Status Active SOTA
Relation to the SoK Model-preserving 3PC; first private LLaMA-7B inference.
Note About five minutes per token for LLaMA-7B. Needs three non-colluding servers.
Code https://github.com/secretflow/spu
Systems page PUMA

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: MPC 35 [p.1], Secret sharing 16 [p.3], Differential privacy 2 [p.4]
  • library: SecretFlow-SPU 7 [p.1], CrypTen 2 [p.3], HuggingFace 1 [p.12]
  • model: GPT 41 [p.3], Transformer 35 [p.1], LLaMA 28 [p.1], RNN / LSTM 1 [p.11], Autoencoder 1 [p.10]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 17 [p.7], GLUE 3 [p.7]
  • hardware: Intel Xeon 1 [p.6]
  • technique: GELU 21 [p.1], Softmax 18 [p.1], LayerNorm 17 [p.1], Three-party computation 5 [p.2], Sigmoid 2 [p.4], Two-party computation 2 [p.10], Semi-honest model 2 [p.4], Malicious model 2 [p.10]
  • task: Language modeling / generation 2 [p.10]
  • topic: Private / secure inference 32 [p.1], Privacy-preserving machine learning 4 [p.11], Open source 4 [p.1], Large language models 3 [p.2], Benchmark 3 [p.7]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/5-068_puma-secure-inference-of-llama-7b-in-five-minutes.pdf catalogue 13 259.3 KB 84c82810a16b

Database key: puma-secure-inference-of-llama-7b-in-five-minutes · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set puma-secure-inference-of-llama-7b-in-five-minutes COLUMN VALUE