Skip to content

ELLMo: Packing- and Depth-Aware Encrypted Transformer Inference

Authors: S. N. Guzelhan; L. Daksha; C. A. Domingo; G. Jonatan; J. Kim; J. L. Abellan; D. Kaeli; A. Joshi
Year: 2026
Venue: IACR ePrint 2026/198

type: system collection: catalogue, sok-references status: auto

Links: ePrint 2026/198 · Semantic Scholar

Summary

Auto-extracted summary (unreviewed)

ELLMo is a CKKS-based encrypted transformer inference design that introduces a shallow plaintext-ciphertext matrix multiplication (shifting rotations to plaintext, 28-50% fewer rotations than FIDESlib and THOR), fuses head-split/merge masking into the ciphertext-ciphertext multiply, and reduces nonlinear depth with a Statistical-max Softmax and a DelayNorm that avoid deep comparison trees and homomorphic divisions, cutting bootstrapping by up to 46%. Implemented on the GPU library FIDESlib (H200), it runs encrypted BERT-Tiny 1.4x faster than state-of-the-art baselines with 0-1.5% accuracy loss on SST-2, MRPC and RTE.

Abstract

Cloud-based Large Language Model (LLM) inference processes sensitive user inputs, yet current deployments offer limited confidentiality guarantees. Fully Homomorphic Encryption (FHE) can provide strong privacy, but it clashes with transformer architectures, where rigid ciphertext packing demands expensive rotations, and deep polynomial circuits for nonlinearities necessitate costly bootstrapping. Although recent work has reported promising speedups, maintaining model accuracy is a challenge. We address these issues with ELLMo, a packing- and depth-aware encrypted transformer design. ELLMo introduces a novel matrix multiplication algorithm to reduce ciphertext rotations. Further, head-split and merge steps are fused into this new algorithm at no additional cost. To reduce the depth of nonlinear layers, our contributions, Statistical-max Softmax and DelayNorm, help bypass deep comparison trees and homomorphic divisions to reduce bootstrapping by up to 46%. On encrypted BERT-Tiny, ELLMo achieves a 1.4× speedup over state-of-the-art baselines with 0%–1.5% accuracy loss across SST-2, MRPC, and RTE downstream tasks.
Source: pdf

Extracted details

Field Value Source Confidence
contribution Shallow PCMM algorithm reducing ciphertext rotations and key-switching llm 0.85
contribution Head-split and merge fused into CCMM at no extra cost llm 0.85
contribution Statistical-max Softmax (max approximated as mu + p*sigma) avoiding comparison trees llm 0.85
contribution DelayNorm delayed normalization avoiding homomorphic division in LayerNorm llm 0.85
contribution GPU implementation on FIDESlib with 1.4x speedup on BERT-Tiny llm 0.85
schemes CKKS llm 0.85
libraries FIDESlib llm 0.85
models BERT-Tiny llm 0.85
datasets SST-2 llm 0.85
datasets MRPC llm 0.85
datasets RTE llm 0.85
tasks private transformer inference llm 0.85
tasks GLUE text classification llm 0.85
hardware NVIDIA H200 GPU llm 0.85
security level 128-bit llm 0.85
threat model client-server: cloud evaluates the LLM on ciphertexts, only the user can decrypt llm 0.85
interaction non-interactive FHE llm 0.85
limitations Statistical-max Softmax degrades sharply when targeting more than ~95% distribution coverage (p > 2.5); accuracy depends on choosing p in the 80-95% coverage range llm 0.85
llm relevance high llm 0.85
key result speed-up: 1.4x (encrypted BERT-Tiny vs state-of-the-art baselines, FIDESlib on H200 GPU) llm 0.85
key result accuracy loss: 0%-1.5% (SST-2, MRPC, RTE) llm 0.85
key result bootstrapping reduction: up to 46% (via Statistical-max Softmax and DelayNorm) llm 0.85
key result rotation count reduction: 28%-50% (linear layers vs FIDESlib and THOR, BERT-Tiny and BERT-base configurations) llm 0.85

System card: ELLMO

Group 0. CKKS LLM frameworks surveyed in the SoK
Family CKKS · Model-modifying (N-MOD)
Arithmetic Approximate real (CKKS)
Interaction non-interactive FHE
Model scale BERT-Tiny
Library OpenFHE + FIDESlib
Hardware H200 GPU
Packing L-ALG
Latency 8 s unamortized
Open source partial
Status Active SOTA
Relation to the SoK Pre-computed plaintext transforms to minimize depth; delayed LayerNorm; statistical max estimation.
Note Fastest BERT-Tiny entry in the SoK; sequence length not reported.
Systems page ELLMO

SoK reference [77]

S. N. Guzelhan, L. Daksha, C. A. Domingo, G. Jonatan, J. Kim, J. L. Abellan, D. Kaeli, and A. Joshi, “Ellmo: Packing-and depthaware encrypted transformer inference,” Cryptology ePrint Archive, 2026.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 25 [p.1], BFV 3 [p.3]
  • library: FIDESlib 13 [p.3], Microsoft SEAL 3 [p.13], Orion 2 [p.13], Phantom 1 [p.16], HEonGPU 1 [p.16], Cheddar 1 [p.14], HEIR 1 [p.13], Fhelipe 1 [p.13], PyTorch 1 [p.13]
  • model: BERT 39 [p.1], Transformer 7 [p.1], Logistic regression 2 [p.8], GPT 1 [p.12], LLaMA 1 [p.12], CNN 1 [p.10]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 28 [p.1], SST-2 16 [p.1], GLUE 6 [p.2]
  • hardware: GPU 11 [p.3], Multi-GPU 2 [p.13], FPGA 1 [p.13]
  • technique: Rotations 38 [p.1], Softmax 34 [p.1], LayerNorm 27 [p.1], Bootstrapping 20 [p.1], Matrix multiplication 20 [p.1], Key switching 15 [p.2], Chebyshev 14 [p.4], GELU 13 [p.1], Polynomial approximation 6 [p.1], Rescaling 5 [p.3], RNS 5 [p.3], Compiler 5 [p.13], Attention 3 [p.2], Relinearization 2 [p.3], Newton iteration 2 [p.9], SIMD packing 2 [p.1], Convolution 2 [p.8], Two-party computation 2 [p.5]
  • task: Medical / healthcare 2 [p.1], Finance 2 [p.1]
  • topic: Homomorphic encryption 60 [p.1], Large language models 17 [p.1], Benchmark 5 [p.11], Private / secure inference 4 [p.1], Open source 4 [p.3]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/0-007_ellmo-packing-and-depth-aware-encrypted-transformer-inferenc.pdf catalogue 17 720.9 KB 40ca842a6358
references/sok-references/077_ellmo-packing-and-depth-aware-encrypted-transformer-inferenc.pdf sok-references 17 720.9 KB 40ca842a6358 byte-identical to references/catalogue/0-007_ellmo-packing-and-depth-aware-encrypted-transformer-inferenc.pdf

Database key: ellmo-packing-and-depth-aware-encrypted-transformer-inference · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set ellmo-packing-and-depth-aware-encrypted-transformer-inference COLUMN VALUE