Skip to content

THOR: Secure transformer inference with homomorphic encryption

Authors: J. Moon; D. Yoo; X. Jiang; M. Kim
Year: 2025
Venue: IACR ePrint / CCS 2025

type: system collection: catalogue, sok-references status: auto

Links: ePrint 2024/1881

Summary

Auto-extracted summary (unreviewed)

THOR is a non-interactive secure transformer inference framework built on RNS-CKKS. It introduces matrix multiplication algorithms based on diagonal-major encoding and compact packing, extended to plaintext-ciphertext MM with parallel submatrix computation and ciphertext-ciphertext MM with a baby-step giant-step strategy, plus adaptive iterative approximations for softmax, LayerNorm, GELU and Tanh. Using the Liberate.FHE library on a single A100 GPU, THOR runs BERT-base with 128 tokens end to end in 602 s (about 10 minutes) with GLUE accuracy comparable to plaintext, and its PC-MM and CC-MM are up to 5.3x faster than BOLT and 9.7x faster than Powerformer.

Abstract

As large language models are increasingly deployed in cloud environments, privacy concerns have become a significant issue. To address this challenge, we present THOR, a non-interactive framework for secure transformer inference using homomorphic encryption. We first propose efficient matrix multiplication algorithms based on diagonal-major encoding and compact ciphertext packing. We extend these basic algorithms to support plaintext-ciphertext matrix multiplication (PC-MM) using parallel submatrix computation and ciphertext-ciphertext multiplication (CC-MM) with a baby-step giant-step strategy. We also design efficient evaluation strategies for non-linear functions such as softmax, LayerNorm, GELU, and Tanh, by integrating advanced approximation techniques with adaptive iterative methods. Our matrix multiplication algorithms outperform state-of-the-art methods, achieving up to 5.3× speedup in PC-MM for R768×768 ×R768×128 over BOLT (Pang et al., IEEE S&P 2024) and 9.7× in CC-MM for 12 × (R64×128 × R128×128) over Powerformer (Park et al., Preprint). THOR enables secure inference on the BERTbase model with 128 tokens in 10 minutes on a single GPU, while maintaining comparable accuracy on GLUE tasks.
Source: pdf

Extracted details

Field Value Source Confidence
contribution Homomorphic matrix multiplication based on diagonal-major encoding and compact ciphertext packing llm 0.90
contribution PC-MM via parallel submatrix computation and CC-MM via baby-step giant-step llm 0.90
contribution Efficient softmax with only 3 square-and-normalize iterations (3 bootstrappings vs 8) llm 0.90
contribution Approximation strategies for LayerNorm, GELU and Tanh llm 0.90
contribution End-to-end BERT-base inference on a single GPU with open-source code llm 0.90
schemes CKKS llm 0.90
libraries Liberate.FHE llm 0.90
models BERT-base llm 0.90
datasets MRPC llm 0.90
datasets RTE llm 0.90
datasets SST-2 llm 0.90
tasks private transformer inference llm 0.90
tasks text classification (GLUE) llm 0.90
hardware NVIDIA A100 GPU llm 0.90
security level 128-bit llm 0.90
threat model client-server; non-interactive outsourced inference llm 0.90
interaction non-interactive FHE llm 0.90
code url https://github.com/crypto-starlab/THOR.BERT (URL found in the full text) regex 0.50
code url https://github.com/google-research/bert (URL found in the full text) regex 0.50
code url https://github.com/Desilo/liberate-fhe (URL found in the full text) regex 0.50
code url https://github.com/crypto-starlab/THOR llm 0.90
limitations Bootstrapping accounts for more than half of the end-to-end runtime llm 0.90
limitations Evaluated only on encoder-only BERT-base; extension to decoder models such as GPT left as future work llm 0.90
llm relevance high llm 0.90
key result end-to-end latency: 602.26 s (about 10 minutes) (BERT-base, 128 tokens, single A100 GPU, N = 2^16) llm 0.90
key result bootstrapping time: 337.86 s of 602.26 s (BERT-base end-to-end breakdown) llm 0.90
key result PC-MM speed-up: 5.3x over BOLT, 11.2x over Powerformer (R768x768 x R768x128) llm 0.90
key result CC-MM speed-up: 9.7x over Powerformer, 36x over BOLT (12 x (R64x128 x R128x128)) llm 0.90
key result softmax speed-up: 2.64x over Cho et al. (14.5 ms amortized vs 38.2 ms) (256 softmax operations of dimension 128) llm 0.90
key result accuracy: comparable to plaintext (GLUE MRPC, RTE, SST-2) llm 0.90

System card: THOR

Group 0. CKKS LLM frameworks surveyed in the SoK
Family CKKS · Model-preserving (N-PRE)
Arithmetic Approximate real (CKKS)
Interaction non-interactive FHE
Model scale BERT-Base
Library Liberate.FHE
Hardware A100 GPU
Packing L-COL
Latency 602 s / 128-token sequence
Open source yes
Status Active SOTA
Relation to the SoK Model-preserving CKKS evaluation of BERT-Base using normalize-and-square Softmax, Goldschmidt division and aSOR.
Note First model-preserving BERT-Base under CKKS; only 5-6 orders of magnitude slower than FP16 GPU cleartext.
Code https://github.com/Desilo/thor
Systems page THOR

SoK reference [16]

J. Moon, D. Yoo, X. Jiang, and M. Kim, “THOR: Secure transformer inference with homomorphic encryption,” in ACM CCS 2025: 32nd Conference on Computer and Communications Security, C.-Y. Huang, J.-C. Chen, S.-P. Shieh, D. Lie, and V. Cortier, Eds. Taipei, Taiwan: ACM Press, Oct. 13–17, 2025, pp. 3765–3779.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: MPC 7 [p.1], CKKS 4 [p.3]
  • library: HElib 2 [p.13], Liberate.FHE 1 [p.13]
  • model: Transformer 15 [p.1], BERT 14 [p.1], RNN / LSTM 3 [p.1], GPT 2 [p.1], LLaMA 2 [p.1]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 4 [p.12], SST-2 2 [p.12], GLUE 2 [p.1]
  • hardware: GPU 4 [p.1], NVIDIA A100 1 [p.12]
  • technique: Matrix multiplication 51 [p.1], Rotations 40 [p.2], Softmax 34 [p.1], Bootstrapping 18 [p.3], GELU 15 [p.1], Key switching 14 [p.2], LayerNorm 11 [p.1], Attention 11 [p.1], SIMD packing 9 [p.1], Relinearization 4 [p.3], Goldschmidt 4 [p.2], Polynomial approximation 3 [p.9], Non-interactive 3 [p.1], RNS 2 [p.11], Fine-tuning 2 [p.10]
  • topic: Homomorphic encryption 28 [p.1], Private / secure inference 7 [p.1], Open source 3 [p.12], Large language models 2 [p.1], Benchmark 2 [p.12]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/0-001_thor-secure-transformer-inference-with-homomorphic-encryptio.pdf catalogue 15 2.0 MB ddb57004b56e
references/sok-references/016_thor-secure-transformer-inference-with-homomorphic-encryptio.pdf sok-references 15 2.0 MB ddb57004b56e byte-identical to references/catalogue/0-001_thor-secure-transformer-inference-with-homomorphic-encryptio.pdf

Database key: thor-secure-transformer-inference-with-homomorphic-encryption · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set thor-secure-transformer-inference-with-homomorphic-encryption COLUMN VALUE