ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour¶
Authors: Xie, Tan, Boddeti, Lu
Year: 2026
Venue: IACR ePrint 2026/1817
type: system collection: catalogue status: auto
Links: ePrint 2026/1817
Summary¶
Auto-extracted summary (unreviewed)
ATLAS is a training-free framework that automatically chooses per-layer polynomial-approximation settings (softmax iterations, normalization and activation degrees) for CKKS Transformer inference by casting the choice as a multi-objective optimization over latency and accuracy. A two-stage NSGA-II search (layer-uniform configurations seeding a per-layer search) with data-subset and early-layer surrogate models completes in about one hour. Across BERT-base, ViT-Base and LLaMA-3 on two FHE backends it reduces multiplicative depth and end-to-end latency by about 35% versus an iterative-softmax baseline with negligible accuracy loss.
Abstract¶
Fully homomorphic encryption (FHE) lets a server run inference on encrypted data with strong privacy guarantees, but running a Transformer under FHE is expensive. Its non-linear operations, such as softmax, normalization, and activation, must be replaced with polynomial approximations that the CKKS scheme supports, and the depth of these approximations dominates inference cost. Existing FHE Transformers use hand-tuned approximation settings, such as iteration count and polynomial degree, applied uniformly across layers, models, and tasks. Hand-tuning is slow and error-prone. Even a single uniform setting has about 107 choices, and manual search cannot exploit layer-wise variation. AutoFHE, the only automated method with multi-objective search, targets ReLUonly CNNs and needs full fine-tuning per candidate, which is too costly for Transformers. Per-layer settings also push the search space to about 1085 for BERT and ViT and 10228 for LLaMA3, beyond both manual and fine-tuning-based search. We present ATLAS, a training-free framework that automates this search by treating each layer’s approximation setting as a multi-objective optimization over latency and accuracy. The problem is hard: the decision space is large (96 or 256 variables), each configuration takes 70 to 1,000 seconds to evaluate even in cleartext, and 85 to 90 percent of configurations are invalid. ATLAS handles this with a two-stage optimization strategy and a surrogate model, completing the search in about one hour. Compared to an iterative softmax baseline, ATLAS cuts multiplicative depth and end-to-end latency by about 35 percent with little accuracy loss, and works across encoderonly, decoder-only, and vision Transformers, complementing parallel work on packing and matrix multiplication.
Source: pdf
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | formulation of per-layer approximation configuration for FHE Transformers as a multi-objective optimization | llm | 0.90 |
| contribution | two-stage NSGA-II search: layer-uniform Pareto set seeding a per-layer (8L-variable) search | llm | 0.90 |
| contribution | data-subset and early-layer-proxy surrogates plus distributed evaluation to cut search cost | llm | 0.90 |
| contribution | about 35% reduction in multiplicative depth and latency with negligible accuracy loss in under one hour | llm | 0.90 |
| contribution | first automated per-layer approximation configuration for vision Transformers | llm | 0.90 |
| schemes | CKKS | llm | 0.90 |
| libraries | NEXUS-EndtoEnd (extended codebase) | llm | 0.90 |
| libraries | HuggingFace | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| models | ViT-Base | llm | 0.90 |
| models | LLaMA-3 | llm | 0.90 |
| datasets | SST-2 | llm | 0.90 |
| datasets | ImageNet | llm | 0.90 |
| tasks | private transformer inference | llm | 0.90 |
| tasks | text classification | llm | 0.90 |
| tasks | image classification | llm | 0.90 |
| security level | 128-bit | llm | 0.90 |
| threat model | honest-but-curious server, client-server | llm | 0.90 |
| interaction | non-interactive FHE | llm | 0.90 |
| code url | https://github.com/timzsu/NEXUS-End2End (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/jianhayes/ATLAS (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/jianhayes/ATLAS | llm | 0.90 |
| limitations | each configuration takes 70 to 1,000 s to evaluate even in cleartext | llm | 0.90 |
| limitations | 85 to 90 percent of configurations are invalid | llm | 0.90 |
| limitations | FHE implementation uses a basic row-packing scheme extended from NEXUS rather than optimized packing | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | multiplicative depth reduction: about 35% (vs iterative softmax baseline, BERT/ViT/LLaMA-3, two FHE backends) | llm | 0.90 |
| key result | end-to-end latency reduction: about 35% (vs iterative softmax baseline, negligible accuracy loss) | llm | 0.90 |
| key result | search time: about 1 hour (two-stage NSGA-II with surrogates) | llm | 0.90 |
| key result | data-subset surrogate: Spearman rho 0.9987, ~70x speed-up (BERT, 10 SST-2 sentences) | llm | 0.90 |
| key result | data-subset surrogate: Spearman rho 0.9977, ~7000x speed-up (ViT, 10 ImageNet images) | llm | 0.90 |
System card: ATLAS¶
| Group | 1. Newer CKKS / FHE LLM frameworks (2026, not in the SoK) |
| Family | CKKS approximation search |
| Arithmetic | Approximate real (CKKS) |
| Interaction | Non-interactive FHE |
| Model scale | Transformer (see paper) |
| Status | Active SOTA |
| Relation to the SoK | Automatically tunes per-layer polynomial degrees and iteration counts. |
| Note | Replaces hand-tuned approximation settings with layer-wise search. |
| Systems page | ATLAS |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 10 [p.1], Leveled / somewhat HE 2 [p.3]
- library: Phantom 8 [p.2], Orion 2 [p.4], HuggingFace 2 [p.8], PyTorch 1 [p.9]
- model: BERT 56 [p.1], Transformer 52 [p.1], ViT 30 [p.1], LLaMA 16 [p.1], MLP 16 [p.3], CNN 12 [p.1], GPT 1 [p.3]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 41 [p.8], SST-2 26 [p.8], ImageNet 10 [p.5], GLUE 5 [p.5], IMDB 2 [p.17]
- hardware: GPU 19 [p.8], CUDA 3 [p.9], NVIDIA RTX 2 [p.9], AMD EPYC 1 [p.9]
- technique: Softmax 54 [p.1], Bootstrapping 31 [p.2], Chebyshev 16 [p.3], GELU 13 [p.2], Polynomial approximation 9 [p.1], Fine-tuning 9 [p.1], LayerNorm 7 [p.2], ReLU 7 [p.1], Goldschmidt 6 [p.3], Rotations 6 [p.13], Matrix multiplication 6 [p.1], Minimax approximation 4 [p.2], Convolution 4 [p.13], Paterson-Stockmeyer 2 [p.18], SIMD packing 2 [p.3], RNS 2 [p.9], Non-interactive 2 [p.16]
- task: Medical / healthcare 5 [p.1]
- topic: Homomorphic encryption 117 [p.1], Large language models 7 [p.3], Private / secure inference 6 [p.2], Open source 5 [p.9], Benchmark 3 [p.5]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/1-126_atlas-automated-approximation-of-transformers-for-efficient.pdf |
catalogue | 19 | 1.6 MB | d547a7a6b4a9 |
Database key: atlas-automated-approximation-of-transformers-for-efficient-homomorphic · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set atlas-automated-approximation-of-transformers-for-efficient-homomorphic COLUMN VALUE