Skip to content

SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC

Authors: Jinglong Luo; Yehong Zhang; Zhuo Zhang; Jiaqi Zhang; Xin Mu; Hui Wang; Yue Yu; Zenglin Xu
Year: 2024
Venue: ACL Findings 2024

type: system collection: catalogue status: auto

Links: arXiv:2401.00793

Summary

Auto-extracted summary (unreviewed)

SecFormer is an SMPC-based privacy-preserving inference framework for Transformer models that combines model design with protocol design. It removes the costly exponential and maximum operations by redesigning Softmax (2Quad, with distillation) and builds efficient secret-sharing protocols for GeLU, LayerNorm and division using numerical methods. Implemented on CrypTen with three V100 servers, it improves average GLUE-task performance over MPCFormer by 3.4% (BERT-base) and 24.7% (BERT-large) and runs 3.57x/3.58x faster than PUMA, with only 0.9%/1.3% performance loss relative to PUMA.

Abstract

With the growing use of Transformer models hosted on cloud platforms to offer inference services, privacy concerns are escalating, especially concerning sensitive data like investment plans and bank account details. Secure Multi-Party Computing (SMPC) emerges as a promising solution to protect the privacy of inference data and model parameters. However, the application of SMPC in Privacy-Preserving Inference (PPI) for Transformer models often leads to considerable slowdowns or declines in performance. This is largely due to the multitude of nonlinear operations in the Transformer architecture, which are not well-suited to SMPC and difficult to circumvent or optimize effectively. To address this concern, we introduce a comprehensive PPI framework called SecFormer to achieve fast and accurate PPI for Transformer models. We successfully eliminate the high-cost exponential and maximum operations in PPI without sacrificing model performance and develop a suite of efficient SMPC protocols by employing suitable numerical computation methods to boost other complex nonlinear functions in PPI, including GeLU, LayerNorm, and a redesigned Softmax. Our extensive experiments reveal that SecFormer outperforms MPCFormer in performance, showing improvements of $3.4\%$ and $24.7\%$ for BERT${\text{BASE}}$ and BERT$$, respectively. In terms of efficiency, SecFormer is 3.57 and 3.58 times faster than PUMA for BERT$}{\text{BASE}}$ and BERT$$, demonstrating its effectiveness and speed.}
Source: arxiv

Extracted details

Field Value Source Confidence
contribution Comprehensive PPI framework combining Transformer model design and SMPC protocol design llm 0.90
contribution Elimination of exponential and maximum operations via a redesigned Softmax (2Quad) without sacrificing performance llm 0.90
contribution Efficient SMPC protocols for GeLU, LayerNorm and division based on suitable numerical computation methods llm 0.90
contribution Experiments showing better accuracy than MPCFormer and higher efficiency than PUMA on BERT-base and BERT-large llm 0.90
schemes secret sharing llm 0.90
libraries CrypTen llm 0.90
models BERT-base llm 0.90
models BERT-large llm 0.90
datasets QNLI llm 0.90
datasets CoLA llm 0.90
datasets STS-B llm 0.90
datasets MRPC llm 0.90
datasets RTE llm 0.90
tasks private Transformer inference llm 0.90
tasks natural language understanding (GLUE tasks) llm 0.90
hardware 3x Tesla V100 servers, 10 GB/s bandwidth llm 0.90
threat model semi-honest, secret-sharing based (CrypTen) among three servers llm 0.90
interaction pure MPC llm 0.90
code url https://github.com/features/copilot (URL found in the full text) regex 0.50
code url https://github.com/facebookresearch/CrypTen (URL found in the full text) regex 0.50
limitations Focuses on encoder-only Transformers such as BERT; not extended to decoder families like the GPT series llm 0.90
llm relevance high llm 0.90
key result average performance improvement over MPCFormer: +3.4% (78.8 vs 75.4) (BERT-base, 5 GLUE tasks) llm 0.90
key result average performance improvement over MPCFormer: +24.7% (80.8 vs 56.1) (BERT-large, 5 GLUE tasks) llm 0.90
key result inference speedup over PUMA: 3.57x (BERT-base, single-sample inference) llm 0.90
key result inference speedup over PUMA: 3.58x (BERT-large, single-sample inference) llm 0.90
key result performance degradation vs PUMA: 0.9% / 1.3% (BERT-base / BERT-large) llm 0.90
key result baseline runtime breakdown: 71 s per example, Softmax 54%, GeLU 23%, LayerNorm 10% (BERT-base, 12 layers, 512 tokens on SMPC library) llm 0.90

System card: SecFormer

Group 5. Interactive MPC and HE+MPC hybrids
Family Secret sharing (CrypTen)
Arithmetic Fixed point
Interaction Interactive 2PC/3PC
Model scale BERT-Base/Large
Status Active
Relation to the SoK Fourier-series GELU and Goldschmidt LayerNorm.
Note Balances distillation and model fidelity.
Code https://github.com/jinglong696/SecFormer
Systems page SecFormer

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: Secret sharing 11 [p.2], MPC 5 [p.2]
  • library: CrypTen 22 [p.6], HuggingFace 1 [p.15]
  • model: Transformer 34 [p.1], GPT 5 [p.9], BERT 3 [p.6], ViT 2 [p.12], LLaMA 1 [p.10], RNN / LSTM 1 [p.11]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 26 [p.6]
  • hardware: NVIDIA V100 1 [p.6]
  • technique: GELU 64 [p.1], Softmax 44 [p.1], LayerNorm 31 [p.1], Matrix multiplication 12 [p.3], Goldschmidt 8 [p.2], Fine-tuning 5 [p.4], Semi-honest model 5 [p.6], ReLU 4 [p.4], Newton iteration 2 [p.15], Two-party computation 2 [p.10]
  • topic: Private / secure inference 5 [p.1], Large language models 3 [p.9], Benchmark 2 [p.10], Open source 2 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/5-074_secformer-fast-and-accurate-privacy-preserving-inference-for.pdf catalogue 16 1.1 MB d037cc9846ee

Database key: secformer-fast-and-accurate-privacy-preserving-inference-for · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set secformer-fast-and-accurate-privacy-preserving-inference-for COLUMN VALUE