Converting transformers to polynomial form for secure inference over homomorphic encryption¶
Authors: Itamar Zimerman; Moran Baruch; Nir Drucker; Gilad Ezov; Omri Soceanu; Lior Wolf
Year: 2024
Venue: arXiv
type: system collection: catalogue, sok-references status: auto
Links: arXiv:2311.08610
Summary¶
Auto-extracted summary (unreviewed)
The paper introduces what it calls the first polynomial transformer for HE: an HE-friendly architecture using scaled sigma-attention instead of softmax, BatchNorm instead of LayerNorm plus stabilizers, a range-minimization training phase, and polynomial replacements for the remaining non-polynomial operators, enabling secure CKKS inference. A polynomial BERT-like model on WikiText-103 reaches 19.89 perplexity versus 18.98 for the 6-layer baseline (18.91 vs 16.89 at 12 layers), while ViT on CIFAR-100 drops from 73.4% to 70.8% and Swin on Tiny-ImageNet from 59.4% to 58.9%, keeping the gap under 4%.
Abstract¶
Designing privacy-preserving deep learning models is a major challenge within the deep learning community. Homomorphic Encryption (HE) has emerged as one of the most promising approaches in this realm, enabling the decoupling of knowledge between the model owner and the data owner. Despite extensive research and application of this technology, primarily in convolutional neural networks, incorporating HE into transformer models has been challenging because of the difficulties in converting these models into a polynomial form. We break new ground by introducing the first polynomial transformer, providing the first demonstration of secure inference over HE with transformers. This includes a transformer architecture tailored for HE, alongside a novel method for converting operators to their polynomial equivalent. This innovation enables us to perform secure inference on LMs with WikiText-103. It also allows us to perform image classification with CIFAR-100 and Tiny-ImageNet. Our models yield results comparable to traditional methods, bridging the performance gap with transformers of similar scale and underscoring the viability of HE for state-of-the-art applications. Finally, we assess the stability of our models and conduct a series of ablations to quantify the contribution of each model component.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | HE-tailored transformer architecture with scaled sigma-attention and BatchNorm-based stabilization | llm | 0.85 |
| contribution | method for converting transformer operators to polynomial equivalents including range-minimization training | llm | 0.85 |
| contribution | first demonstration of secure transformer inference over HE on language modeling and image classification | llm | 0.85 |
| contribution | stability analysis and ablations quantifying each component's contribution | llm | 0.85 |
| schemes | CKKS | llm | 0.85 |
| models | BERT-like transformer (6 and 12 layers) | llm | 0.85 |
| models | ViT | llm | 0.85 |
| models | Swin Transformer | llm | 0.85 |
| datasets | WikiText-103 | llm | 0.85 |
| datasets | CIFAR-100 | llm | 0.85 |
| datasets | Tiny-ImageNet | llm | 0.85 |
| datasets | CIFAR-10 | llm | 0.85 |
| tasks | language modeling (next token prediction) | llm | 0.85 |
| tasks | image classification | llm | 0.85 |
| tasks | private transformer inference | llm | 0.85 |
| threat model | data owner encrypts input, model owner (untrusted environment) computes on ciphertexts | llm | 0.85 |
| interaction | non-interactive FHE | llm | 0.85 |
| limitations | at least 80% of the perplexity gap arises in the final polynomial-approximation stage, so more accurate polynomials are needed to close it | llm | 0.85 |
| llm relevance | high | llm | 0.85 |
| key result | perplexity: 19.89 polynomial vs 18.98 original (6-layer BERT-like transformer (53.3M params), WikiText-103) | llm | 0.85 |
| key result | perplexity: 18.91 polynomial vs 16.89 original (12-layer transformer (95.8M params), WikiText-103) | llm | 0.85 |
| key result | accuracy: 70.8% polynomial vs 73.4% original (ViT on CIFAR-100) | llm | 0.85 |
| key result | accuracy: 58.9% polynomial vs 59.4% original (Swin Transformer on Tiny-ImageNet) | llm | 0.85 |
| key result | accuracy gap: < 4% (polynomial vs original vision models) | llm | 0.85 |
System card: Polyformer¶
| Group | 0. CKKS LLM frameworks surveyed in the SoK |
| Family | CKKS · Model-modifying (N-MOD) |
| Arithmetic | Approximate real (CKKS) |
| Interaction | non-interactive FHE |
| Model scale | BERT-6L |
| Library | HELayers |
| Hardware | A100 GPU |
| Packing | tile tensors |
| Latency | 305 s (128 tokens next-word prediction) |
| Open source | no |
| Status | Niche |
| Relation to the SoK | 100% polynomial transformer via scaled-sigma attention and range-minimization training. |
| Note | Requires retraining; HELayers tile tensors. |
| Systems page | Polyformer |
SoK reference [82]¶
I. Zimerman, M. Baruch, N. Drucker, G. Ezov, O. Soceanu, and L. Wolf, “Converting transformers to polynomial form for secure inference over homomorphic encryption,” in Proceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 2 [p.1]
- library: HEaaN (library) 1 [p.10], PyTorch 1 [p.13]
- model: ViT 19 [p.5], Transformer 12 [p.1], BERT 6 [p.3], CNN 6 [p.2], MLP 5 [p.6], ResNet 3 [p.2], AlexNet 1 [p.2]
- dataset: CIFAR-100 8 [p.1], ImageNet 8 [p.1], CIFAR-10 6 [p.6], WikiText 1 [p.1]
- hardware: GPU 4 [p.10], NVIDIA A100 2 [p.10], AMD EPYC 1 [p.10]
- technique: LayerNorm 24 [p.2], Softmax 18 [p.1], GELU 18 [p.1], ReLU 16 [p.2], Attention 9 [p.2], Polynomial approximation 7 [p.3], Remez 6 [p.3], Convolution 6 [p.1], Bootstrapping 2 [p.2], Newton iteration 2 [p.3]
- task: Image classification 3 [p.1], Language modeling / generation 2 [p.6]
- topic: Homomorphic encryption 61 [p.1], Private / secure inference 14 [p.1], Benchmark 6 [p.6], Privacy-preserving machine learning 5 [p.1], Large language models 2 [p.5]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/0-018_converting-transformers-to-polynomial-form-for-secure-infere.pdf |
catalogue | 13 | 5.1 MB | 611fccd39a55 |
|
references/sok-references/082_converting-transformers-to-polynomial-form-for-secure-infere.pdf |
sok-references | 13 | 5.1 MB | 611fccd39a55 |
byte-identical to references/catalogue/0-018_converting-transformers-to-polynomial-form-for-secure-infere.pdf |
Database key: converting-transformers-to-polynomial-form-for-secure-inference-over · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set converting-transformers-to-polynomial-form-for-secure-inference-over COLUMN VALUE