Skip to content

MOAI: Module-optimizing architecture for non-interactive secure transformer inference

Authors: L. Zhang; X. Wang; J. J. Sim; Z. Huang; J. Zhong; H. WANG; P. Duan; K.-Y. Lam
Year: 2026
Venue: ICLR 2026; IACR ePrint 2025/991

type: system collection: catalogue, sok-references status: auto

Links: ePrint 2025/991

Summary

Auto-extracted summary (unreviewed)

MOAI is a CKKS-based, non-interactive framework for secure transformer inference that keeps consistent column/diagonal packing across all layers (removing format conversions), uses rotation-free Softmax and LayerNorm algorithms, and reduces rotations in matrix multiplications via column packing and interleaved batching. On BERT-base with 128 tokens it cuts end-to-end evaluation time by 52.8% versus THOR and by 55.7% versus Powerformer on an A100, reaching an amortized 2.36 minutes per input on a single H200 GPU (9.6 minutes on a 56-core CPU), and the method is also applied to LLaMA-3-8B.

Abstract

Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from: (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, removing 2448 HE rotations in BERT-base inference. (3) Column packing removes rotations in plaintext–ciphertext matrix multiplications and interleaved batching further reduces the rotations in ciphertext–ciphertext matrix multiplications. MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix multiplications of BERT-base. As a result, We achieve a 52.8% reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS’25). We then apply MOAI on the Powerformer’s framework and achieve a 55.7% reduction in evaluation time compared to Powerformer (Park et al., ACL’25), which approximates Softmax and LayerNorm with simpler functions in transformer and proposes HE-based non-interactive transformer inference. We report an amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI in LLaMA-3-8B. Our implementation is publicly available as open source. 1
Source: pdf

Extracted details

Field Value Source Confidence
contribution evaluation flow combining column and diagonal packing consistently across layers, eliminating format conversions llm 0.90
contribution rotation-free Softmax and LayerNorm algorithms (removing 2448 HE rotations in BERT-base) llm 0.90
contribution column packing and interleaved batching to reduce rotations in plaintext-ciphertext and ciphertext-ciphertext matrix multiplications llm 0.90
contribution 52.8% faster than THOR and 55.7% faster than Powerformer on BERT-base llm 0.90
contribution extension to decoder-only LLaMA-3-8B inference llm 0.90
contribution open-source CPU (SEAL) and GPU (Phantom) implementations llm 0.90
schemes CKKS llm 0.90
libraries SEAL llm 0.90
libraries FHE-MP-CNN llm 0.90
libraries Phantom llm 0.90
libraries OpenMP llm 0.90
libraries HuggingFace Transformers llm 0.90
models BERT-base llm 0.90
models LLaMA-3-8B llm 0.90
datasets SST-2 llm 0.90
datasets QNLI llm 0.90
datasets RTE llm 0.90
tasks text classification llm 0.90
tasks secure transformer inference llm 0.90
tasks LLM inference llm 0.90
hardware Intel Xeon Platinum 8480+ CPU (56 cores) llm 0.90
hardware NVIDIA H200 GPU llm 0.90
hardware NVIDIA A100 GPU llm 0.90
security level 128-bit llm 0.90
threat model client-server; untrusted cloud service provider computes on encrypted inputs and learns nothing about the user's input llm 0.90
interaction non-interactive FHE llm 0.90
code url https://github.com/dtc2025ag/MOAI.GPU (URL found in the full text) regex 0.50
code url https://github.com/dtc2025ag/MOAI_GPU.3https (URL found in the full text) regex 0.50
code url https://github.com/encryptorion-lab/phantom-fhe (URL found in the full text) regex 0.50
code url https://github.com/dtc2025ag/MOAI llm 0.90
llm relevance high llm 0.90
key result amortized latency: 2.36 min per input (BERT-base, 12 layers, 128 tokens, single H200 GPU, amortized over 256 inputs) llm 0.90
key result amortized latency: 9.6 min per input (BERT-base, 128 tokens, Xeon Platinum 8480+ CPU with 56 cores) llm 0.90
key result evaluation time reduction vs THOR: 52.8% (BERT-base, single A100 GPU, N=2^16, 2^15 slots) llm 0.90
key result evaluation time reduction vs Powerformer: 55.7% (BERT-base, single A100 GPU, Powerformer parameter setting) llm 0.90
key result microbenchmark speedup: up to 22x (Softmax) and 151x (LayerNorm) (versus prior works) llm 0.90
key result HE rotations: at least 1.7x fewer (all matrix multiplications of BERT-base versus state of the art) llm 0.90

System card: MOAI

Group 0. CKKS LLM frameworks surveyed in the SoK
Family CKKS · Model-modifying (N-MOD)
Arithmetic Approximate real (CKKS)
Interaction non-interactive FHE
Model scale BERT-Base, Llama-3-8B
Library SEAL + Phantom
Hardware H200 / A100 GPU
Packing L-COL
Latency 141 s BERT-Base amortized; 225 s Llama-3-8B decode (8 tokens)
Open source yes
Status Active SOTA
Relation to the SoK Bootstrapping inside Softmax before division halves required levels (20 to 10).
Note Depth-minimization placement; constant max values from plaintext profiling.
Systems page MOAI

SoK reference [14]

L. Zhang, X. Wang, J. J. Sim, Z. Huang, J. Zhong, H. WANG, P. Duan, and K.-Y. Lam, “MOAI: Module-optimizing architecture for non-interactive secure transformer inference,” in The Fourteenth International Conference on Learning Representations, 2026. [Online]. Available: https://openreview.net/forum?id=qJn4HtTzhH

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 21 [p.4], MPC 5 [p.2]
  • library: Microsoft SEAL 3 [p.11], Phantom 2 [p.11], Orion 1 [p.15]
  • model: BERT 27 [p.1], LLaMA 9 [p.1], Transformer 5 [p.1], CNN 1 [p.11]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 4 [p.11], SST-2 2 [p.11], GLUE 2 [p.13]
  • hardware: GPU 21 [p.1], NVIDIA A100 5 [p.11], Intel Xeon 2 [p.11]
  • technique: Softmax 71 [p.1], Rotations 65 [p.1], LayerNorm 53 [p.1], Matrix multiplication 32 [p.1], Bootstrapping 23 [p.4], GELU 22 [p.4], Goldschmidt 13 [p.8], SIMD packing 11 [p.4], Attention 9 [p.4], Non-interactive 8 [p.1], Fine-tuning 4 [p.2], Minimax approximation 3 [p.9], Remez 3 [p.16], Relinearization 2 [p.7], Knowledge distillation 2 [p.2]
  • topic: Homomorphic encryption 91 [p.1], Open source 11 [p.11], Private / secure inference 5 [p.2], Large language models 4 [p.1], Benchmark 4 [p.4]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/0-014_moai-module-optimizing-architecture-for-non-interactive-secu.pdf catalogue 32 2.7 MB 8c694fb9249c
references/sok-references/014_moai-module-optimizing-architecture-for-non-interactive-secu.pdf sok-references 32 2.7 MB 8c694fb9249c byte-identical to references/catalogue/0-014_moai-module-optimizing-architecture-for-non-interactive-secu.pdf

Database key: moai-module-optimizing-architecture-for-non-interactive-secure · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set moai-module-optimizing-architecture-for-non-interactive-secure COLUMN VALUE