Skip to content

MPCFormer: Fast, Performant and Private Transformer Inference with MPC

Authors: Dacheng Li; Rulin Shao; Hongyi Wang; Han Guo; Eric P. Xing; Hao Zhang
Year: 2023
Venue: ICLR 2023

type: system collection: catalogue status: auto

Links: arXiv:2211.01452

Summary

Auto-extracted summary (unreviewed)

MPCFormer is a two-stage framework for private Transformer inference in MPC: it first replaces expensive non-linear functions such as GELU and Softmax with MPC-friendly approximations (e.g. quadratics), then uses knowledge distillation from the fine-tuned teacher to recover the accuracy lost by the approximations, which plain retraining fails to do. Run on an MPC engine such as CrypTen, it matches BERT-base on IMDb with a 5.3x speedup and retains 97% of BERT-base performance on GLUE with a 2.2x speedup, and remains effective for RoBERTa-base and BERT-large.

Abstract

Enabling private inference is crucial for many cloud inference services that are based on Transformer models. However, existing private inference solutions can increase the inference latency by more than 60x or significantly compromise the inference quality. In this paper, we design the framework MPCFORMER as a practical solution, using Secure Multi-Party Computation (MPC) and Knowledge Distillation (KD). Through extensive evaluations, we show that MPCFORMER significantly speeds up Transformer inference in MPC settings while achieving similar ML performance to the input model. On the IMDb dataset, it achieves similar performance to BERTBASE, while being 5.3x faster. On the GLUE benchmark, it achieves 97% performance of BERTBASE with a 2.2x speedup. MPCFORMER remains effective with different trained Transformer weights such as ROBERTABASE and larger models including BERTLarge. Code is available at https://github.com/MccRee177/MPCFormer.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution MPC-friendly approximations of GELU and Softmax for Transformers llm 0.80
contribution knowledge distillation (with intermediate-representation matching) to train the approximated student from the fine-tuned teacher llm 0.80
contribution two-stage framework: plug in approximations, then distill llm 0.80
contribution evaluation on BERT-base, RoBERTa-base and BERT-large showing speedups with near-equal accuracy llm 0.80
schemes MPC llm 0.80
libraries CrypTen llm 0.80
libraries HuggingFace Transformers llm 0.80
models BERT-base llm 0.80
models RoBERTa-base llm 0.80
models BERT-large llm 0.80
datasets IMDb llm 0.80
datasets GLUE llm 0.80
tasks text classification llm 0.80
tasks private transformer inference llm 0.80
threat model cloud inference service: client input and provider model weights kept private under MPC llm 0.80
interaction pure MPC llm 0.80
code url https://github.com/MccRee177/MPCFormer.1 (URL found in the full text) regex 0.50
code url https://github.com/MccRee177/MPCFormer llm 0.80
llm relevance high llm 0.80
key result speedup: 5.3x with similar performance to BERT-base (IMDb) llm 0.80
key result speedup and accuracy: 2.2x speedup at 97% of BERT-base performance (GLUE benchmark) llm 0.80
key result baseline MPC latency: about 60 s in MPC vs under 1 s in plaintext (vanilla BERT-base inference) llm 0.80

System card: MPCFormer

Group 5. Interactive MPC and HE+MPC hybrids
Family MPC (CrypTen) + distillation
Arithmetic Fixed point secret sharing
Interaction Interactive 2PC/3PC
Model scale BERT-Base/Large
Status Active
Relation to the SoK Replaces Softmax/GELU with quadratic forms and distills.
Note Model-modifying, like the SoK's N-MOD class but for MPC.
Code https://github.com/DachengLi1/MPCFormer
Systems page MPCFormer

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: MPC 86 [p.1], Secret sharing 22 [p.3], Paillier 2 [p.3], Oblivious transfer 2 [p.3]
  • library: CrypTen 7 [p.2], HuggingFace 2 [p.6]
  • model: Transformer 71 [p.1], CNN 5 [p.1], BERT 3 [p.15], GPT 2 [p.1]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 61 [p.7], GLUE 14 [p.1], IMDB 14 [p.1], SST-2 8 [p.8]
  • hardware: NVIDIA V100 2 [p.6], GPU 2 [p.6]
  • technique: Softmax 26 [p.2], GELU 20 [p.3], ReLU 9 [p.4], Knowledge distillation 6 [p.1], Matrix multiplication 5 [p.2], Two-party computation 3 [p.2], Semi-honest model 3 [p.6], Taylor expansion 2 [p.3]
  • topic: Benchmark 14 [p.2], Private / secure inference 6 [p.1], Homomorphic encryption 2 [p.3], Open source 2 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/5-067_mpcformer-fast-performant-and-private-transformer-inference.pdf catalogue 16 1.5 MB 0600cdb483eb

Database key: mpcformer-fast-performant-and-private-transformer-inference-with-mpc · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set mpcformer-fast-performant-and-private-transformer-inference-with-mpc COLUMN VALUE