Skip to content

Converting transformers to polynomial form for secure inference over homomorphic encryption

Authors: Itamar Zimerman; Moran Baruch; Nir Drucker; Gilad Ezov; Omri Soceanu; Lior Wolf
Year: 2024
Venue: arXiv

type: system collection: catalogue, sok-references status: auto

Links: arXiv:2311.08610

Summary

Auto-extracted summary (unreviewed)

The paper introduces what it calls the first polynomial transformer for HE: an HE-friendly architecture using scaled sigma-attention instead of softmax, BatchNorm instead of LayerNorm plus stabilizers, a range-minimization training phase, and polynomial replacements for the remaining non-polynomial operators, enabling secure CKKS inference. A polynomial BERT-like model on WikiText-103 reaches 19.89 perplexity versus 18.98 for the 6-layer baseline (18.91 vs 16.89 at 12 layers), while ViT on CIFAR-100 drops from 73.4% to 70.8% and Swin on Tiny-ImageNet from 59.4% to 58.9%, keeping the gap under 4%.

Abstract

Designing privacy-preserving deep learning models is a major challenge within the deep learning community. Homomorphic Encryption (HE) has emerged as one of the most promising approaches in this realm, enabling the decoupling of knowledge between the model owner and the data owner. Despite extensive research and application of this technology, primarily in convolutional neural networks, incorporating HE into transformer models has been challenging because of the difficulties in converting these models into a polynomial form. We break new ground by introducing the first polynomial transformer, providing the first demonstration of secure inference over HE with transformers. This includes a transformer architecture tailored for HE, alongside a novel method for converting operators to their polynomial equivalent. This innovation enables us to perform secure inference on LMs with WikiText-103. It also allows us to perform image classification with CIFAR-100 and Tiny-ImageNet. Our models yield results comparable to traditional methods, bridging the performance gap with transformers of similar scale and underscoring the viability of HE for state-of-the-art applications. Finally, we assess the stability of our models and conduct a series of ablations to quantify the contribution of each model component.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution HE-tailored transformer architecture with scaled sigma-attention and BatchNorm-based stabilization llm 0.85
contribution method for converting transformer operators to polynomial equivalents including range-minimization training llm 0.85
contribution first demonstration of secure transformer inference over HE on language modeling and image classification llm 0.85
contribution stability analysis and ablations quantifying each component's contribution llm 0.85
schemes CKKS llm 0.85
models BERT-like transformer (6 and 12 layers) llm 0.85
models ViT llm 0.85
models Swin Transformer llm 0.85
datasets WikiText-103 llm 0.85
datasets CIFAR-100 llm 0.85
datasets Tiny-ImageNet llm 0.85
datasets CIFAR-10 llm 0.85
tasks language modeling (next token prediction) llm 0.85
tasks image classification llm 0.85
tasks private transformer inference llm 0.85
threat model data owner encrypts input, model owner (untrusted environment) computes on ciphertexts llm 0.85
interaction non-interactive FHE llm 0.85
limitations at least 80% of the perplexity gap arises in the final polynomial-approximation stage, so more accurate polynomials are needed to close it llm 0.85
llm relevance high llm 0.85
key result perplexity: 19.89 polynomial vs 18.98 original (6-layer BERT-like transformer (53.3M params), WikiText-103) llm 0.85
key result perplexity: 18.91 polynomial vs 16.89 original (12-layer transformer (95.8M params), WikiText-103) llm 0.85
key result accuracy: 70.8% polynomial vs 73.4% original (ViT on CIFAR-100) llm 0.85
key result accuracy: 58.9% polynomial vs 59.4% original (Swin Transformer on Tiny-ImageNet) llm 0.85
key result accuracy gap: < 4% (polynomial vs original vision models) llm 0.85

System card: Polyformer

Group 0. CKKS LLM frameworks surveyed in the SoK
Family CKKS · Model-modifying (N-MOD)
Arithmetic Approximate real (CKKS)
Interaction non-interactive FHE
Model scale BERT-6L
Library HELayers
Hardware A100 GPU
Packing tile tensors
Latency 305 s (128 tokens next-word prediction)
Open source no
Status Niche
Relation to the SoK 100% polynomial transformer via scaled-sigma attention and range-minimization training.
Note Requires retraining; HELayers tile tensors.
Systems page Polyformer

SoK reference [82]

I. Zimerman, M. Baruch, N. Drucker, G. Ezov, O. Soceanu, and L. Wolf, “Converting transformers to polynomial form for secure inference over homomorphic encryption,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 2 [p.1]
  • library: HEaaN (library) 1 [p.10], PyTorch 1 [p.13]
  • model: ViT 19 [p.5], Transformer 12 [p.1], BERT 6 [p.3], CNN 6 [p.2], MLP 5 [p.6], ResNet 3 [p.2], AlexNet 1 [p.2]
  • dataset: CIFAR-100 8 [p.1], ImageNet 8 [p.1], CIFAR-10 6 [p.6], WikiText 1 [p.1]
  • hardware: GPU 4 [p.10], NVIDIA A100 2 [p.10], AMD EPYC 1 [p.10]
  • technique: LayerNorm 24 [p.2], Softmax 18 [p.1], GELU 18 [p.1], ReLU 16 [p.2], Attention 9 [p.2], Polynomial approximation 7 [p.3], Remez 6 [p.3], Convolution 6 [p.1], Bootstrapping 2 [p.2], Newton iteration 2 [p.3]
  • task: Image classification 3 [p.1], Language modeling / generation 2 [p.6]
  • topic: Homomorphic encryption 61 [p.1], Private / secure inference 14 [p.1], Benchmark 6 [p.6], Privacy-preserving machine learning 5 [p.1], Large language models 2 [p.5]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/0-018_converting-transformers-to-polynomial-form-for-secure-infere.pdf catalogue 13 5.1 MB 611fccd39a55
references/sok-references/082_converting-transformers-to-polynomial-form-for-secure-infere.pdf sok-references 13 5.1 MB 611fccd39a55 byte-identical to references/catalogue/0-018_converting-transformers-to-polynomial-form-for-secure-infere.pdf

Database key: converting-transformers-to-polynomial-form-for-secure-inference-over · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set converting-transformers-to-polynomial-form-for-secure-inference-over COLUMN VALUE