MOAI: Module-optimizing architecture for non-interactive secure transformer inference¶
Authors: L. Zhang; X. Wang; J. J. Sim; Z. Huang; J. Zhong; H. WANG; P. Duan; K.-Y. Lam
Year: 2026
Venue: ICLR 2026; IACR ePrint 2025/991
type: system collection: catalogue, sok-references status: auto
Links: ePrint 2025/991
Summary¶
Auto-extracted summary (unreviewed)
MOAI is a CKKS-based, non-interactive framework for secure transformer inference that keeps consistent column/diagonal packing across all layers (removing format conversions), uses rotation-free Softmax and LayerNorm algorithms, and reduces rotations in matrix multiplications via column packing and interleaved batching. On BERT-base with 128 tokens it cuts end-to-end evaluation time by 52.8% versus THOR and by 55.7% versus Powerformer on an A100, reaching an amortized 2.36 minutes per input on a single H200 GPU (9.6 minutes on a 56-core CPU), and the method is also applied to LLaMA-3-8B.
Abstract¶
Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from: (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, removing 2448 HE rotations in BERT-base inference. (3) Column packing removes rotations in plaintext–ciphertext matrix multiplications and interleaved batching further reduces the rotations in ciphertext–ciphertext matrix multiplications. MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix multiplications of BERT-base. As a result, We achieve a 52.8% reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS’25). We then apply MOAI on the Powerformer’s framework and achieve a 55.7% reduction in evaluation time compared to Powerformer (Park et al., ACL’25), which approximates Softmax and LayerNorm with simpler functions in transformer and proposes HE-based non-interactive transformer inference. We report an amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI in LLaMA-3-8B. Our implementation is publicly available as open source. 1
Source: pdf
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | evaluation flow combining column and diagonal packing consistently across layers, eliminating format conversions | llm | 0.90 |
| contribution | rotation-free Softmax and LayerNorm algorithms (removing 2448 HE rotations in BERT-base) | llm | 0.90 |
| contribution | column packing and interleaved batching to reduce rotations in plaintext-ciphertext and ciphertext-ciphertext matrix multiplications | llm | 0.90 |
| contribution | 52.8% faster than THOR and 55.7% faster than Powerformer on BERT-base | llm | 0.90 |
| contribution | extension to decoder-only LLaMA-3-8B inference | llm | 0.90 |
| contribution | open-source CPU (SEAL) and GPU (Phantom) implementations | llm | 0.90 |
| schemes | CKKS | llm | 0.90 |
| libraries | SEAL | llm | 0.90 |
| libraries | FHE-MP-CNN | llm | 0.90 |
| libraries | Phantom | llm | 0.90 |
| libraries | OpenMP | llm | 0.90 |
| libraries | HuggingFace Transformers | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| models | LLaMA-3-8B | llm | 0.90 |
| datasets | SST-2 | llm | 0.90 |
| datasets | QNLI | llm | 0.90 |
| datasets | RTE | llm | 0.90 |
| tasks | text classification | llm | 0.90 |
| tasks | secure transformer inference | llm | 0.90 |
| tasks | LLM inference | llm | 0.90 |
| hardware | Intel Xeon Platinum 8480+ CPU (56 cores) | llm | 0.90 |
| hardware | NVIDIA H200 GPU | llm | 0.90 |
| hardware | NVIDIA A100 GPU | llm | 0.90 |
| security level | 128-bit | llm | 0.90 |
| threat model | client-server; untrusted cloud service provider computes on encrypted inputs and learns nothing about the user's input | llm | 0.90 |
| interaction | non-interactive FHE | llm | 0.90 |
| code url | https://github.com/dtc2025ag/MOAI.GPU (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/dtc2025ag/MOAI_GPU.3https (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/encryptorion-lab/phantom-fhe (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/dtc2025ag/MOAI | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | amortized latency: 2.36 min per input (BERT-base, 12 layers, 128 tokens, single H200 GPU, amortized over 256 inputs) | llm | 0.90 |
| key result | amortized latency: 9.6 min per input (BERT-base, 128 tokens, Xeon Platinum 8480+ CPU with 56 cores) | llm | 0.90 |
| key result | evaluation time reduction vs THOR: 52.8% (BERT-base, single A100 GPU, N=2^16, 2^15 slots) | llm | 0.90 |
| key result | evaluation time reduction vs Powerformer: 55.7% (BERT-base, single A100 GPU, Powerformer parameter setting) | llm | 0.90 |
| key result | microbenchmark speedup: up to 22x (Softmax) and 151x (LayerNorm) (versus prior works) | llm | 0.90 |
| key result | HE rotations: at least 1.7x fewer (all matrix multiplications of BERT-base versus state of the art) | llm | 0.90 |
System card: MOAI¶
| Group | 0. CKKS LLM frameworks surveyed in the SoK |
| Family | CKKS · Model-modifying (N-MOD) |
| Arithmetic | Approximate real (CKKS) |
| Interaction | non-interactive FHE |
| Model scale | BERT-Base, Llama-3-8B |
| Library | SEAL + Phantom |
| Hardware | H200 / A100 GPU |
| Packing | L-COL |
| Latency | 141 s BERT-Base amortized; 225 s Llama-3-8B decode (8 tokens) |
| Open source | yes |
| Status | Active SOTA |
| Relation to the SoK | Bootstrapping inside Softmax before division halves required levels (20 to 10). |
| Note | Depth-minimization placement; constant max values from plaintext profiling. |
| Systems page | MOAI |
SoK reference [14]¶
L. Zhang, X. Wang, J. J. Sim, Z. Huang, J. Zhong, H. WANG, P. Duan, and K.-Y. Lam, “MOAI: Module-optimizing architecture for non-interactive secure transformer inference,” in The Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=qJn4HtTzhH
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 21 [p.4], MPC 5 [p.2]
- library: Microsoft SEAL 3 [p.11], Phantom 2 [p.11], Orion 1 [p.15]
- model: BERT 27 [p.1], LLaMA 9 [p.1], Transformer 5 [p.1], CNN 1 [p.11]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 4 [p.11], SST-2 2 [p.11], GLUE 2 [p.13]
- hardware: GPU 21 [p.1], NVIDIA A100 5 [p.11], Intel Xeon 2 [p.11]
- technique: Softmax 71 [p.1], Rotations 65 [p.1], LayerNorm 53 [p.1], Matrix multiplication 32 [p.1], Bootstrapping 23 [p.4], GELU 22 [p.4], Goldschmidt 13 [p.8], SIMD packing 11 [p.4], Attention 9 [p.4], Non-interactive 8 [p.1], Fine-tuning 4 [p.2], Minimax approximation 3 [p.9], Remez 3 [p.16], Relinearization 2 [p.7], Knowledge distillation 2 [p.2]
- topic: Homomorphic encryption 91 [p.1], Open source 11 [p.11], Private / secure inference 5 [p.2], Large language models 4 [p.1], Benchmark 4 [p.4]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/0-014_moai-module-optimizing-architecture-for-non-interactive-secu.pdf |
catalogue | 32 | 2.7 MB | 8c694fb9249c |
|
references/sok-references/014_moai-module-optimizing-architecture-for-non-interactive-secu.pdf |
sok-references | 32 | 2.7 MB | 8c694fb9249c |
byte-identical to references/catalogue/0-014_moai-module-optimizing-architecture-for-non-interactive-secu.pdf |
Database key: moai-module-optimizing-architecture-for-non-interactive-secure · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set moai-module-optimizing-architecture-for-non-interactive-secure COLUMN VALUE