SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC¶
Authors: Jinglong Luo; Yehong Zhang; Zhuo Zhang; Jiaqi Zhang; Xin Mu; Hui Wang; Yue Yu; Zenglin Xu
Year: 2024
Venue: ACL Findings 2024
type: system collection: catalogue status: auto
Links: arXiv:2401.00793
Summary¶
Auto-extracted summary (unreviewed)
SecFormer is an SMPC-based privacy-preserving inference framework for Transformer models that combines model design with protocol design. It removes the costly exponential and maximum operations by redesigning Softmax (2Quad, with distillation) and builds efficient secret-sharing protocols for GeLU, LayerNorm and division using numerical methods. Implemented on CrypTen with three V100 servers, it improves average GLUE-task performance over MPCFormer by 3.4% (BERT-base) and 24.7% (BERT-large) and runs 3.57x/3.58x faster than PUMA, with only 0.9%/1.3% performance loss relative to PUMA.
Abstract¶
With the growing use of Transformer models hosted on cloud platforms to offer inference services, privacy concerns are escalating, especially concerning sensitive data like investment plans and bank account details. Secure Multi-Party Computing (SMPC) emerges as a promising solution to protect the privacy of inference data and model parameters. However, the application of SMPC in Privacy-Preserving Inference (PPI) for Transformer models often leads to considerable slowdowns or declines in performance. This is largely due to the multitude of nonlinear operations in the Transformer architecture, which are not well-suited to SMPC and difficult to circumvent or optimize effectively. To address this concern, we introduce a comprehensive PPI framework called SecFormer to achieve fast and accurate PPI for Transformer models. We successfully eliminate the high-cost exponential and maximum operations in PPI without sacrificing model performance and develop a suite of efficient SMPC protocols by employing suitable numerical computation methods to boost other complex nonlinear functions in PPI, including GeLU, LayerNorm, and a redesigned Softmax. Our extensive experiments reveal that SecFormer outperforms MPCFormer in performance, showing improvements of $3.4\%$ and $24.7\%$ for BERT${\text{BASE}}$ and BERT$$, respectively. In terms of efficiency, SecFormer is 3.57 and 3.58 times faster than PUMA for BERT$}{\text{BASE}}$ and BERT$$, demonstrating its effectiveness and speed.}
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | Comprehensive PPI framework combining Transformer model design and SMPC protocol design | llm | 0.90 |
| contribution | Elimination of exponential and maximum operations via a redesigned Softmax (2Quad) without sacrificing performance | llm | 0.90 |
| contribution | Efficient SMPC protocols for GeLU, LayerNorm and division based on suitable numerical computation methods | llm | 0.90 |
| contribution | Experiments showing better accuracy than MPCFormer and higher efficiency than PUMA on BERT-base and BERT-large | llm | 0.90 |
| schemes | secret sharing | llm | 0.90 |
| libraries | CrypTen | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| models | BERT-large | llm | 0.90 |
| datasets | QNLI | llm | 0.90 |
| datasets | CoLA | llm | 0.90 |
| datasets | STS-B | llm | 0.90 |
| datasets | MRPC | llm | 0.90 |
| datasets | RTE | llm | 0.90 |
| tasks | private Transformer inference | llm | 0.90 |
| tasks | natural language understanding (GLUE tasks) | llm | 0.90 |
| hardware | 3x Tesla V100 servers, 10 GB/s bandwidth | llm | 0.90 |
| threat model | semi-honest, secret-sharing based (CrypTen) among three servers | llm | 0.90 |
| interaction | pure MPC | llm | 0.90 |
| code url | https://github.com/features/copilot (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/facebookresearch/CrypTen (URL found in the full text) | regex | 0.50 |
| limitations | Focuses on encoder-only Transformers such as BERT; not extended to decoder families like the GPT series | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | average performance improvement over MPCFormer: +3.4% (78.8 vs 75.4) (BERT-base, 5 GLUE tasks) | llm | 0.90 |
| key result | average performance improvement over MPCFormer: +24.7% (80.8 vs 56.1) (BERT-large, 5 GLUE tasks) | llm | 0.90 |
| key result | inference speedup over PUMA: 3.57x (BERT-base, single-sample inference) | llm | 0.90 |
| key result | inference speedup over PUMA: 3.58x (BERT-large, single-sample inference) | llm | 0.90 |
| key result | performance degradation vs PUMA: 0.9% / 1.3% (BERT-base / BERT-large) | llm | 0.90 |
| key result | baseline runtime breakdown: 71 s per example, Softmax 54%, GeLU 23%, LayerNorm 10% (BERT-base, 12 layers, 512 tokens on SMPC library) | llm | 0.90 |
System card: SecFormer¶
| Group | 5. Interactive MPC and HE+MPC hybrids |
| Family | Secret sharing (CrypTen) |
| Arithmetic | Fixed point |
| Interaction | Interactive 2PC/3PC |
| Model scale | BERT-Base/Large |
| Status | Active |
| Relation to the SoK | Fourier-series GELU and Goldschmidt LayerNorm. |
| Note | Balances distillation and model fidelity. |
| Code | https://github.com/jinglong696/SecFormer |
| Systems page | SecFormer |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: Secret sharing 11 [p.2], MPC 5 [p.2]
- library: CrypTen 22 [p.6], HuggingFace 1 [p.15]
- model: Transformer 34 [p.1], GPT 5 [p.9], BERT 3 [p.6], ViT 2 [p.12], LLaMA 1 [p.10], RNN / LSTM 1 [p.11]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 26 [p.6]
- hardware: NVIDIA V100 1 [p.6]
- technique: GELU 64 [p.1], Softmax 44 [p.1], LayerNorm 31 [p.1], Matrix multiplication 12 [p.3], Goldschmidt 8 [p.2], Fine-tuning 5 [p.4], Semi-honest model 5 [p.6], ReLU 4 [p.4], Newton iteration 2 [p.15], Two-party computation 2 [p.10]
- topic: Private / secure inference 5 [p.1], Large language models 3 [p.9], Benchmark 2 [p.10], Open source 2 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/5-074_secformer-fast-and-accurate-privacy-preserving-inference-for.pdf |
catalogue | 16 | 1.1 MB | d037cc9846ee |
Database key: secformer-fast-and-accurate-privacy-preserving-inference-for · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set secformer-fast-and-accurate-privacy-preserving-inference-for COLUMN VALUE