Skip to content

THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption

Authors: Tianyu Chen; Hangbo Bao; Shaohan Huang; Li Dong; Binxing Jiao; Daxin Jiang; Haoyi Zhou; Jianxin Li; Furu Wei
Year: 2022
Venue: ACL Findings 2022

type: system collection: catalogue status: auto

Links: arXiv:2206.00216

Summary

Auto-extracted summary (unreviewed)

THE-X is an approximation workflow that makes pre-trained transformer models evaluable under homomorphic encryption by replacing GELU with ReLU, softmax with a learned estimation model, and LayerNorm with an approximation, then fine-tuning and converting the checkpoint with HE-transformer on a SEAL backend. With BERT-tiny on seven GLUE tasks the average utility loss is 1.48% (most of it from the LayerNorm approximation), and on CoNLL-2003 NER the F1 drop is 1.90.

Abstract

As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e.g., search history, medical record, bank account). Privacy-preserving inference of transformer models is on the demand of cloud service users. To protect privacy, it is an attractive choice to compute only with ciphertext in homomorphic encryption (HE). However, enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks, which are not supported by current HE tools yet. In this work, we introduce $\textit{THE-X}$, an approximation approach for transformers, which enables privacy-preserving inference of pre-trained models developed by popular frameworks. $\textit{THE-X}$ proposes a workflow to deal with complex computation in transformer networks, including all the non-polynomial functions like GELU, softmax, and LayerNorm. Experiments reveal our proposed $\textit{THE-X}$ can enable transformer inference on encrypted data for different downstream tasks, all with negligible performance drop but enjoying the theory-guaranteed privacy-preserving advantage.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution Workflow to approximate all non-polynomial transformer operations (GELU, softmax, LayerNorm) for HE inference llm 0.85
contribution Learned softmax estimation model that costs only 0.09% average accuracy llm 0.85
contribution Fine-tuning procedure with staged replacement (ReLU, ReLU-S, ReLU-S-L) before HE conversion llm 0.85
contribution HE inference of a fine-tuned BERT-tiny via HE-transformer with the SEAL backend llm 0.85
contribution Evaluation on GLUE sequence-level tasks and CoNLL-2003 token-level NER llm 0.85
schemes homomorphic encryption (HE-transformer with SEAL backend) llm 0.85
libraries HE-transformer llm 0.85
libraries SEAL llm 0.85
models BERT-tiny llm 0.85
datasets SST-2 llm 0.85
datasets MRPC llm 0.85
datasets STS-B llm 0.85
datasets QQP llm 0.85
datasets MNLI llm 0.85
datasets QNLI llm 0.85
datasets RTE llm 0.85
datasets CoNLL-2003 llm 0.85
tasks private transformer inference llm 0.85
tasks sentiment analysis llm 0.85
tasks paraphrase detection llm 0.85
tasks natural language inference llm 0.85
tasks semantic similarity llm 0.85
tasks named entity recognition llm 0.85
threat model client-server cloud inference; server computes only on ciphertext llm 0.85
interaction non-interactive FHE llm 0.85
limitations LayerNorm approximation is the main source of accuracy loss llm 0.85
limitations Evaluated only on BERT-tiny (2 layers, hidden size 128); CoLA not reported due to the model's limited capacity llm 0.85
limitations No HE latency numbers reported in the excerpt llm 0.85
llm relevance high llm 0.85
key result average GLUE performance drop: 1.48% (BERT-tiny, HE setting vs standard fine-tuning baseline) llm 0.85
key result SST-2 accuracy: 82.11 (baseline 82.45) (BERT-tiny under HE) llm 0.85
key result CoNLL-2003 F1: 81.67 (baseline 83.57, drop 1.90) (BERT-tiny under HE) llm 0.85
key result accuracy drop attributable to LayerNorm approximation: 1.08% (GLUE average) llm 0.85
key result accuracy drop attributable to softmax estimation: 0.09% (GLUE average) llm 0.85

System card: THE-X

Group 5. Interactive MPC and HE+MPC hybrids
Family HE (SEAL CKKS) + client-side non-linear
Arithmetic Approximate
Interaction Interactive (client computes non-linear steps)
Model scale BERT-Tiny/Base
Status Foundational
Relation to the SoK Early HE Transformer that offloads non-linear steps to the client.
Note Replaces Softmax/GELU and returns some intermediate values to the client, which leaks information.
Systems page THE-X

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: Differential privacy 4 [p.1], Partially homomorphic encryption 2 [p.2]
  • library: Microsoft SEAL 6 [p.3], TensorFlow 2 [p.5], PyTorch 1 [p.5]
  • model: BERT 10 [p.2], Transformer 5 [p.1], GPT 2 [p.2]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 14 [p.6], GLUE 5 [p.2], SST-2 1 [p.6]
  • technique: Softmax 31 [p.1], ReLU 25 [p.3], LayerNorm 19 [p.3], GELU 17 [p.1], Federated learning 7 [p.2], Fine-tuning 4 [p.1], Taylor expansion 2 [p.4]
  • assumption: Lattice-based 2 [p.2]
  • task: Named entity recognition 3 [p.2], Medical / healthcare 2 [p.2]
  • topic: Homomorphic encryption 39 [p.1], Private / secure inference 8 [p.1], Benchmark 6 [p.2]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/5-065_the-x-privacy-preserving-transformer-inference-with-homomorp.pdf catalogue 11 1.2 MB 6d783507185f

Database key: the-x-privacy-preserving-transformer-inference-with-homomorphic · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set the-x-privacy-preserving-transformer-inference-with-homomorphic COLUMN VALUE