MPCFormer: Fast, Performant and Private Transformer Inference with MPC¶
Authors: Dacheng Li; Rulin Shao; Hongyi Wang; Han Guo; Eric P. Xing; Hao Zhang
Year: 2023
Venue: ICLR 2023
type: system collection: catalogue status: auto
Links: arXiv:2211.01452
Summary¶
Auto-extracted summary (unreviewed)
MPCFormer is a two-stage framework for private Transformer inference in MPC: it first replaces expensive non-linear functions such as GELU and Softmax with MPC-friendly approximations (e.g. quadratics), then uses knowledge distillation from the fine-tuned teacher to recover the accuracy lost by the approximations, which plain retraining fails to do. Run on an MPC engine such as CrypTen, it matches BERT-base on IMDb with a 5.3x speedup and retains 97% of BERT-base performance on GLUE with a 2.2x speedup, and remains effective for RoBERTa-base and BERT-large.
Abstract¶
Enabling private inference is crucial for many cloud inference services that are based on Transformer models. However, existing private inference solutions can increase the inference latency by more than 60x or significantly compromise the inference quality. In this paper, we design the framework MPCFORMER as a practical solution, using Secure Multi-Party Computation (MPC) and Knowledge Distillation (KD). Through extensive evaluations, we show that MPCFORMER significantly speeds up Transformer inference in MPC settings while achieving similar ML performance to the input model. On the IMDb dataset, it achieves similar performance to BERTBASE, while being 5.3x faster. On the GLUE benchmark, it achieves 97% performance of BERTBASE with a 2.2x speedup. MPCFORMER remains effective with different trained Transformer weights such as ROBERTABASE and larger models including BERTLarge. Code is available at https://github.com/MccRee177/MPCFormer.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | MPC-friendly approximations of GELU and Softmax for Transformers | llm | 0.80 |
| contribution | knowledge distillation (with intermediate-representation matching) to train the approximated student from the fine-tuned teacher | llm | 0.80 |
| contribution | two-stage framework: plug in approximations, then distill | llm | 0.80 |
| contribution | evaluation on BERT-base, RoBERTa-base and BERT-large showing speedups with near-equal accuracy | llm | 0.80 |
| schemes | MPC | llm | 0.80 |
| libraries | CrypTen | llm | 0.80 |
| libraries | HuggingFace Transformers | llm | 0.80 |
| models | BERT-base | llm | 0.80 |
| models | RoBERTa-base | llm | 0.80 |
| models | BERT-large | llm | 0.80 |
| datasets | IMDb | llm | 0.80 |
| datasets | GLUE | llm | 0.80 |
| tasks | text classification | llm | 0.80 |
| tasks | private transformer inference | llm | 0.80 |
| threat model | cloud inference service: client input and provider model weights kept private under MPC | llm | 0.80 |
| interaction | pure MPC | llm | 0.80 |
| code url | https://github.com/MccRee177/MPCFormer.1 (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/MccRee177/MPCFormer | llm | 0.80 |
| llm relevance | high | llm | 0.80 |
| key result | speedup: 5.3x with similar performance to BERT-base (IMDb) | llm | 0.80 |
| key result | speedup and accuracy: 2.2x speedup at 97% of BERT-base performance (GLUE benchmark) | llm | 0.80 |
| key result | baseline MPC latency: about 60 s in MPC vs under 1 s in plaintext (vanilla BERT-base inference) | llm | 0.80 |
System card: MPCFormer¶
| Group | 5. Interactive MPC and HE+MPC hybrids |
| Family | MPC (CrypTen) + distillation |
| Arithmetic | Fixed point secret sharing |
| Interaction | Interactive 2PC/3PC |
| Model scale | BERT-Base/Large |
| Status | Active |
| Relation to the SoK | Replaces Softmax/GELU with quadratic forms and distills. |
| Note | Model-modifying, like the SoK's N-MOD class but for MPC. |
| Code | https://github.com/DachengLi1/MPCFormer |
| Systems page | MPCFormer |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: MPC 86 [p.1], Secret sharing 22 [p.3], Paillier 2 [p.3], Oblivious transfer 2 [p.3]
- library: CrypTen 7 [p.2], HuggingFace 2 [p.6]
- model: Transformer 71 [p.1], CNN 5 [p.1], BERT 3 [p.15], GPT 2 [p.1]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 61 [p.7], GLUE 14 [p.1], IMDB 14 [p.1], SST-2 8 [p.8]
- hardware: NVIDIA V100 2 [p.6], GPU 2 [p.6]
- technique: Softmax 26 [p.2], GELU 20 [p.3], ReLU 9 [p.4], Knowledge distillation 6 [p.1], Matrix multiplication 5 [p.2], Two-party computation 3 [p.2], Semi-honest model 3 [p.6], Taylor expansion 2 [p.3]
- topic: Benchmark 14 [p.2], Private / secure inference 6 [p.1], Homomorphic encryption 2 [p.3], Open source 2 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/5-067_mpcformer-fast-performant-and-private-transformer-inference.pdf |
catalogue | 16 | 1.5 MB | 0600cdb483eb |
Database key: mpcformer-fast-performant-and-private-transformer-inference-with-mpc · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set mpcformer-fast-performant-and-private-transformer-inference-with-mpc COLUMN VALUE