Skip to content

Ditto: Quantization-aware Secure Inference of Transformers upon MPC

Authors: Haoqi Wu; Wenjing Fang; Yancheng Zheng; Junming Ma; Jin Tan; Yinggui Wang; Lei Wang
Year: 2024
Venue: ICML 2024

type: system collection: catalogue status: auto

Links: arXiv:2405.05525

Summary

Auto-extracted summary (unreviewed)

Ditto brings quantization into MPC-based secure Transformer inference. It applies an MPC-friendly quantization with quantization-aware distillation to keep model utility, and adds new MPC primitives for the type conversions quantization requires, implemented on SecretFlow-SPU. On BERT-base/large (GLUE) and GPT-2 base/medium (Wikitext-103) it is 3.14-4.40x faster than MPCFormer and 1.44-2.35x faster than PUMA with negligible utility degradation.

Abstract

Due to the rising privacy concerns on sensitive client data and trained models like Transformers, secure multi-party computation (MPC) techniques are employed to enable secure inference despite attendant overhead. Existing works attempt to reduce the overhead using more MPC-friendly non-linear function approximations. However, the integration of quantization widely used in plaintext inference into the MPC domain remains unclear. To bridge this gap, we propose the framework named Ditto to enable more efficient quantization-aware secure Transformer inference. Concretely, we first incorporate an MPC-friendly quantization into Transformer inference and employ a quantization-aware distillation procedure to maintain the model utility. Then, we propose novel MPC primitives to support the type conversions that are essential in quantization and implement the quantization-aware MPC execution of secure quantized inference. This approach significantly decreases both computation and communication overhead, leading to improvements in overall efficiency. We conduct extensive experiments on Bert and GPT2 models to evaluate the performance of Ditto. The results demonstrate that Ditto is about $3.14\sim 4.40\times$ faster than MPCFormer (ICLR 2023) and $1.44\sim 2.35\times$ faster than the state-of-the-art work PUMA with negligible utility degradation.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution MPC-friendly quantization scheme for Transformer inference llm 0.90
contribution quantization-aware distillation to maintain model utility llm 0.90
contribution novel MPC primitives for type conversion (dynamic bit-width switching) llm 0.90
contribution quantization-aware MPC execution implemented on SecretFlow-SPU llm 0.90
schemes secret sharing llm 0.90
libraries SecretFlow-SPU llm 0.90
libraries Hugging Face Transformers llm 0.90
models BERT-base llm 0.90
models BERT-large llm 0.90
models GPT-2 llm 0.90
models GPT-2-medium llm 0.90
datasets GLUE (RTE, CoLA, QQP, QNLI) llm 0.90
datasets Wikitext-103 llm 0.90
tasks secure Transformer inference llm 0.90
tasks text classification llm 0.90
tasks language modeling llm 0.90
hardware AMD Ryzen CPU (32 cores, 3.60GHz), 256GB RAM llm 0.90
hardware LAN 5Gbps / 0.4ms RTT llm 0.90
hardware WAN 400Mbps / 40ms RTT (simulated with tc) llm 0.90
interaction pure MPC llm 0.90
code url https://github.com/secretflow/spu.Wikipedia (URL found in the full text) regex 0.50
code url https://huggingface.co/Graphcore/gpt2-wikitext-103 (URL found in the full text) regex 0.50
code url https://huggingface.co/Graphcore/gpt2-medium-wikitext-103 (URL found in the full text) regex 0.50
limitations noticeable utility drop on CoLA attributed to its small size and unstable distillation llm 0.90
limitations 2ReLU Softmax approximation causes noticeable utility drops on BERT tasks llm 0.90
limitations Quad GELU approximation is less accurate than PUMA's polynomial approximation (trade-off for efficiency) llm 0.90
llm relevance high llm 0.90
key result speed-up vs MPCFormer: 3.14-4.40x (BERT and GPT-2 models) llm 0.90
key result speed-up vs PUMA: 1.44-2.35x (BERT and GPT-2 models) llm 0.90
key result end-to-end runtime / communication: 30.58 s / 4.35 GB (BERT-base), 52.75 s / 10.68 GB (BERT-large), 20.3 s / 3.6 GB (GPT2-base), 33.56 s / 5.76 GB (GPT2-medium) (annotated points in Fig. 4, LAN, batch 1, seq len 128 (BERT) / 32 input + 1 generated token (GPT-2)) llm 0.90
key result utility (BERT-base, Quad approximation): RTE 67.82 / CoLA 54.52 / QQP 87.72 / QNLI 91.78 (baseline 68.59 / 57.06 / 87.96 / 91.62) llm 0.90
key result perplexity (GPT2-base, Quad): 13.78 (Wikitext-103, baseline 12.25) llm 0.90

System card: Ditto

Group 5. Interactive MPC and HE+MPC hybrids
Family 3PC + quantization
Arithmetic Mixed bit-width fixed point
Interaction Interactive 3PC
Model scale BERT, GPT-2
Status Active
Relation to the SoK Quantization-aware MPC with type conversion.
Note Speeds MPC by lowering bit-widths, a direction not yet explored in CKKS.
Code https://github.com/secretflow/spu
Systems page Ditto

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: MPC 99 [p.1], Secret sharing 4 [p.3]
  • library: SecretFlow-SPU 8 [p.6], HuggingFace 2 [p.7], PyTorch 1 [p.5]
  • model: Transformer 49 [p.1], GPT 42 [p.1], BERT 4 [p.10], ViT 1 [p.2], CNN 1 [p.2]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 38 [p.7], GLUE 4 [p.7]
  • hardware: GPU 2 [p.12]
  • technique: Quantization 148 [p.1], GELU 24 [p.1], Softmax 18 [p.2], Two-party computation 9 [p.2], Matrix multiplication 7 [p.1], LayerNorm 6 [p.5], Three-party computation 5 [p.2], Fine-tuning 4 [p.4], Semi-honest model 4 [p.2], Polynomial approximation 2 [p.7], Compiler 2 [p.6]
  • topic: Private / secure inference 39 [p.1], Privacy-preserving machine learning 4 [p.11], Benchmark 4 [p.7], Large language models 3 [p.4], Open source 2 [p.2]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/5-075_ditto-quantization-aware-secure-inference-of-transformers-up.pdf catalogue 19 1.2 MB f2beaf658ea2

Database key: ditto-quantization-aware-secure-inference-of-transformers-upon-mpc · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set ditto-quantization-aware-secure-inference-of-transformers-upon-mpc COLUMN VALUE