Ditto: Quantization-aware Secure Inference of Transformers upon MPC¶
Authors: Haoqi Wu; Wenjing Fang; Yancheng Zheng; Junming Ma; Jin Tan; Yinggui Wang; Lei Wang
Year: 2024
Venue: ICML 2024
type: system collection: catalogue status: auto
Links: arXiv:2405.05525
Summary¶
Auto-extracted summary (unreviewed)
Ditto brings quantization into MPC-based secure Transformer inference. It applies an MPC-friendly quantization with quantization-aware distillation to keep model utility, and adds new MPC primitives for the type conversions quantization requires, implemented on SecretFlow-SPU. On BERT-base/large (GLUE) and GPT-2 base/medium (Wikitext-103) it is 3.14-4.40x faster than MPCFormer and 1.44-2.35x faster than PUMA with negligible utility degradation.
Abstract¶
Due to the rising privacy concerns on sensitive client data and trained models like Transformers, secure multi-party computation (MPC) techniques are employed to enable secure inference despite attendant overhead. Existing works attempt to reduce the overhead using more MPC-friendly non-linear function approximations. However, the integration of quantization widely used in plaintext inference into the MPC domain remains unclear. To bridge this gap, we propose the framework named Ditto to enable more efficient quantization-aware secure Transformer inference. Concretely, we first incorporate an MPC-friendly quantization into Transformer inference and employ a quantization-aware distillation procedure to maintain the model utility. Then, we propose novel MPC primitives to support the type conversions that are essential in quantization and implement the quantization-aware MPC execution of secure quantized inference. This approach significantly decreases both computation and communication overhead, leading to improvements in overall efficiency. We conduct extensive experiments on Bert and GPT2 models to evaluate the performance of Ditto. The results demonstrate that Ditto is about $3.14\sim 4.40\times$ faster than MPCFormer (ICLR 2023) and $1.44\sim 2.35\times$ faster than the state-of-the-art work PUMA with negligible utility degradation.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | MPC-friendly quantization scheme for Transformer inference | llm | 0.90 |
| contribution | quantization-aware distillation to maintain model utility | llm | 0.90 |
| contribution | novel MPC primitives for type conversion (dynamic bit-width switching) | llm | 0.90 |
| contribution | quantization-aware MPC execution implemented on SecretFlow-SPU | llm | 0.90 |
| schemes | secret sharing | llm | 0.90 |
| libraries | SecretFlow-SPU | llm | 0.90 |
| libraries | Hugging Face Transformers | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| models | BERT-large | llm | 0.90 |
| models | GPT-2 | llm | 0.90 |
| models | GPT-2-medium | llm | 0.90 |
| datasets | GLUE (RTE, CoLA, QQP, QNLI) | llm | 0.90 |
| datasets | Wikitext-103 | llm | 0.90 |
| tasks | secure Transformer inference | llm | 0.90 |
| tasks | text classification | llm | 0.90 |
| tasks | language modeling | llm | 0.90 |
| hardware | AMD Ryzen CPU (32 cores, 3.60GHz), 256GB RAM | llm | 0.90 |
| hardware | LAN 5Gbps / 0.4ms RTT | llm | 0.90 |
| hardware | WAN 400Mbps / 40ms RTT (simulated with tc) | llm | 0.90 |
| interaction | pure MPC | llm | 0.90 |
| code url | https://github.com/secretflow/spu.Wikipedia (URL found in the full text) | regex | 0.50 |
| code url | https://huggingface.co/Graphcore/gpt2-wikitext-103 (URL found in the full text) | regex | 0.50 |
| code url | https://huggingface.co/Graphcore/gpt2-medium-wikitext-103 (URL found in the full text) | regex | 0.50 |
| limitations | noticeable utility drop on CoLA attributed to its small size and unstable distillation | llm | 0.90 |
| limitations | 2ReLU Softmax approximation causes noticeable utility drops on BERT tasks | llm | 0.90 |
| limitations | Quad GELU approximation is less accurate than PUMA's polynomial approximation (trade-off for efficiency) | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | speed-up vs MPCFormer: 3.14-4.40x (BERT and GPT-2 models) | llm | 0.90 |
| key result | speed-up vs PUMA: 1.44-2.35x (BERT and GPT-2 models) | llm | 0.90 |
| key result | end-to-end runtime / communication: 30.58 s / 4.35 GB (BERT-base), 52.75 s / 10.68 GB (BERT-large), 20.3 s / 3.6 GB (GPT2-base), 33.56 s / 5.76 GB (GPT2-medium) (annotated points in Fig. 4, LAN, batch 1, seq len 128 (BERT) / 32 input + 1 generated token (GPT-2)) | llm | 0.90 |
| key result | utility (BERT-base, Quad approximation): RTE 67.82 / CoLA 54.52 / QQP 87.72 / QNLI 91.78 (baseline 68.59 / 57.06 / 87.96 / 91.62) | llm | 0.90 |
| key result | perplexity (GPT2-base, Quad): 13.78 (Wikitext-103, baseline 12.25) | llm | 0.90 |
System card: Ditto¶
| Group | 5. Interactive MPC and HE+MPC hybrids |
| Family | 3PC + quantization |
| Arithmetic | Mixed bit-width fixed point |
| Interaction | Interactive 3PC |
| Model scale | BERT, GPT-2 |
| Status | Active |
| Relation to the SoK | Quantization-aware MPC with type conversion. |
| Note | Speeds MPC by lowering bit-widths, a direction not yet explored in CKKS. |
| Code | https://github.com/secretflow/spu |
| Systems page | Ditto |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: MPC 99 [p.1], Secret sharing 4 [p.3]
- library: SecretFlow-SPU 8 [p.6], HuggingFace 2 [p.7], PyTorch 1 [p.5]
- model: Transformer 49 [p.1], GPT 42 [p.1], BERT 4 [p.10], ViT 1 [p.2], CNN 1 [p.2]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 38 [p.7], GLUE 4 [p.7]
- hardware: GPU 2 [p.12]
- technique: Quantization 148 [p.1], GELU 24 [p.1], Softmax 18 [p.2], Two-party computation 9 [p.2], Matrix multiplication 7 [p.1], LayerNorm 6 [p.5], Three-party computation 5 [p.2], Fine-tuning 4 [p.4], Semi-honest model 4 [p.2], Polynomial approximation 2 [p.7], Compiler 2 [p.6]
- topic: Private / secure inference 39 [p.1], Privacy-preserving machine learning 4 [p.11], Benchmark 4 [p.7], Large language models 3 [p.4], Open source 2 [p.2]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/5-075_ditto-quantization-aware-secure-inference-of-transformers-up.pdf |
catalogue | 19 | 1.2 MB | f2beaf658ea2 |
Database key: ditto-quantization-aware-secure-inference-of-transformers-upon-mpc · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set ditto-quantization-aware-secure-inference-of-transformers-upon-mpc COLUMN VALUE