Skip to content

ATLAS: Automated Approximation of Transformers for Efficient Homomorphic Inference in One Hour

Authors: Xie, Tan, Boddeti, Lu
Year: 2026
Venue: IACR ePrint 2026/1817

type: system collection: catalogue status: auto

Links: ePrint 2026/1817

Summary

Auto-extracted summary (unreviewed)

ATLAS is a training-free framework that automatically chooses per-layer polynomial-approximation settings (softmax iterations, normalization and activation degrees) for CKKS Transformer inference by casting the choice as a multi-objective optimization over latency and accuracy. A two-stage NSGA-II search (layer-uniform configurations seeding a per-layer search) with data-subset and early-layer surrogate models completes in about one hour. Across BERT-base, ViT-Base and LLaMA-3 on two FHE backends it reduces multiplicative depth and end-to-end latency by about 35% versus an iterative-softmax baseline with negligible accuracy loss.

Abstract

Fully homomorphic encryption (FHE) lets a server run inference on encrypted data with strong privacy guarantees, but running a Transformer under FHE is expensive. Its non-linear operations, such as softmax, normalization, and activation, must be replaced with polynomial approximations that the CKKS scheme supports, and the depth of these approximations dominates inference cost. Existing FHE Transformers use hand-tuned approximation settings, such as iteration count and polynomial degree, applied uniformly across layers, models, and tasks. Hand-tuning is slow and error-prone. Even a single uniform setting has about 107 choices, and manual search cannot exploit layer-wise variation. AutoFHE, the only automated method with multi-objective search, targets ReLUonly CNNs and needs full fine-tuning per candidate, which is too costly for Transformers. Per-layer settings also push the search space to about 1085 for BERT and ViT and 10228 for LLaMA3, beyond both manual and fine-tuning-based search. We present ATLAS, a training-free framework that automates this search by treating each layer’s approximation setting as a multi-objective optimization over latency and accuracy. The problem is hard: the decision space is large (96 or 256 variables), each configuration takes 70 to 1,000 seconds to evaluate even in cleartext, and 85 to 90 percent of configurations are invalid. ATLAS handles this with a two-stage optimization strategy and a surrogate model, completing the search in about one hour. Compared to an iterative softmax baseline, ATLAS cuts multiplicative depth and end-to-end latency by about 35 percent with little accuracy loss, and works across encoderonly, decoder-only, and vision Transformers, complementing parallel work on packing and matrix multiplication.
Source: pdf

Extracted details

Field Value Source Confidence
contribution formulation of per-layer approximation configuration for FHE Transformers as a multi-objective optimization llm 0.90
contribution two-stage NSGA-II search: layer-uniform Pareto set seeding a per-layer (8L-variable) search llm 0.90
contribution data-subset and early-layer-proxy surrogates plus distributed evaluation to cut search cost llm 0.90
contribution about 35% reduction in multiplicative depth and latency with negligible accuracy loss in under one hour llm 0.90
contribution first automated per-layer approximation configuration for vision Transformers llm 0.90
schemes CKKS llm 0.90
libraries NEXUS-EndtoEnd (extended codebase) llm 0.90
libraries HuggingFace llm 0.90
models BERT-base llm 0.90
models ViT-Base llm 0.90
models LLaMA-3 llm 0.90
datasets SST-2 llm 0.90
datasets ImageNet llm 0.90
tasks private transformer inference llm 0.90
tasks text classification llm 0.90
tasks image classification llm 0.90
security level 128-bit llm 0.90
threat model honest-but-curious server, client-server llm 0.90
interaction non-interactive FHE llm 0.90
code url https://github.com/timzsu/NEXUS-End2End (URL found in the full text) regex 0.50
code url https://github.com/jianhayes/ATLAS (URL found in the full text) regex 0.50
code url https://github.com/jianhayes/ATLAS llm 0.90
limitations each configuration takes 70 to 1,000 s to evaluate even in cleartext llm 0.90
limitations 85 to 90 percent of configurations are invalid llm 0.90
limitations FHE implementation uses a basic row-packing scheme extended from NEXUS rather than optimized packing llm 0.90
llm relevance high llm 0.90
key result multiplicative depth reduction: about 35% (vs iterative softmax baseline, BERT/ViT/LLaMA-3, two FHE backends) llm 0.90
key result end-to-end latency reduction: about 35% (vs iterative softmax baseline, negligible accuracy loss) llm 0.90
key result search time: about 1 hour (two-stage NSGA-II with surrogates) llm 0.90
key result data-subset surrogate: Spearman rho 0.9987, ~70x speed-up (BERT, 10 SST-2 sentences) llm 0.90
key result data-subset surrogate: Spearman rho 0.9977, ~7000x speed-up (ViT, 10 ImageNet images) llm 0.90

System card: ATLAS

Group 1. Newer CKKS / FHE LLM frameworks (2026, not in the SoK)
Family CKKS approximation search
Arithmetic Approximate real (CKKS)
Interaction Non-interactive FHE
Model scale Transformer (see paper)
Status Active SOTA
Relation to the SoK Automatically tunes per-layer polynomial degrees and iteration counts.
Note Replaces hand-tuned approximation settings with layer-wise search.
Systems page ATLAS

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 10 [p.1], Leveled / somewhat HE 2 [p.3]
  • library: Phantom 8 [p.2], Orion 2 [p.4], HuggingFace 2 [p.8], PyTorch 1 [p.9]
  • model: BERT 56 [p.1], Transformer 52 [p.1], ViT 30 [p.1], LLaMA 16 [p.1], MLP 16 [p.3], CNN 12 [p.1], GPT 1 [p.3]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 41 [p.8], SST-2 26 [p.8], ImageNet 10 [p.5], GLUE 5 [p.5], IMDB 2 [p.17]
  • hardware: GPU 19 [p.8], CUDA 3 [p.9], NVIDIA RTX 2 [p.9], AMD EPYC 1 [p.9]
  • technique: Softmax 54 [p.1], Bootstrapping 31 [p.2], Chebyshev 16 [p.3], GELU 13 [p.2], Polynomial approximation 9 [p.1], Fine-tuning 9 [p.1], LayerNorm 7 [p.2], ReLU 7 [p.1], Goldschmidt 6 [p.3], Rotations 6 [p.13], Matrix multiplication 6 [p.1], Minimax approximation 4 [p.2], Convolution 4 [p.13], Paterson-Stockmeyer 2 [p.18], SIMD packing 2 [p.3], RNS 2 [p.9], Non-interactive 2 [p.16]
  • task: Medical / healthcare 5 [p.1]
  • topic: Homomorphic encryption 117 [p.1], Large language models 7 [p.3], Private / secure inference 6 [p.2], Open source 5 [p.9], Benchmark 3 [p.5]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/1-126_atlas-automated-approximation-of-transformers-for-efficient.pdf catalogue 19 1.6 MB d547a7a6b4a9

Database key: atlas-automated-approximation-of-transformers-for-efficient-homomorphic · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set atlas-automated-approximation-of-transformers-for-efficient-homomorphic COLUMN VALUE