THOR: Secure transformer inference with homomorphic encryption¶
Authors: J. Moon; D. Yoo; X. Jiang; M. Kim
Year: 2025
Venue: IACR ePrint / CCS 2025
type: system collection: catalogue, sok-references status: auto
Links: ePrint 2024/1881
Summary¶
Auto-extracted summary (unreviewed)
THOR is a non-interactive secure transformer inference framework built on RNS-CKKS. It introduces matrix multiplication algorithms based on diagonal-major encoding and compact packing, extended to plaintext-ciphertext MM with parallel submatrix computation and ciphertext-ciphertext MM with a baby-step giant-step strategy, plus adaptive iterative approximations for softmax, LayerNorm, GELU and Tanh. Using the Liberate.FHE library on a single A100 GPU, THOR runs BERT-base with 128 tokens end to end in 602 s (about 10 minutes) with GLUE accuracy comparable to plaintext, and its PC-MM and CC-MM are up to 5.3x faster than BOLT and 9.7x faster than Powerformer.
Abstract¶
As large language models are increasingly deployed in cloud environments, privacy concerns have become a significant issue. To address this challenge, we present THOR, a non-interactive framework for secure transformer inference using homomorphic encryption. We first propose efficient matrix multiplication algorithms based on diagonal-major encoding and compact ciphertext packing. We extend these basic algorithms to support plaintext-ciphertext matrix multiplication (PC-MM) using parallel submatrix computation and ciphertext-ciphertext multiplication (CC-MM) with a baby-step giant-step strategy. We also design efficient evaluation strategies for non-linear functions such as softmax, LayerNorm, GELU, and Tanh, by integrating advanced approximation techniques with adaptive iterative methods. Our matrix multiplication algorithms outperform state-of-the-art methods, achieving up to 5.3× speedup in PC-MM for R768×768 ×R768×128 over BOLT (Pang et al., IEEE S&P 2024) and 9.7× in CC-MM for 12 × (R64×128 × R128×128) over Powerformer (Park et al., Preprint). THOR enables secure inference on the BERTbase model with 128 tokens in 10 minutes on a single GPU, while maintaining comparable accuracy on GLUE tasks.
Source: pdf
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | Homomorphic matrix multiplication based on diagonal-major encoding and compact ciphertext packing | llm | 0.90 |
| contribution | PC-MM via parallel submatrix computation and CC-MM via baby-step giant-step | llm | 0.90 |
| contribution | Efficient softmax with only 3 square-and-normalize iterations (3 bootstrappings vs 8) | llm | 0.90 |
| contribution | Approximation strategies for LayerNorm, GELU and Tanh | llm | 0.90 |
| contribution | End-to-end BERT-base inference on a single GPU with open-source code | llm | 0.90 |
| schemes | CKKS | llm | 0.90 |
| libraries | Liberate.FHE | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| datasets | MRPC | llm | 0.90 |
| datasets | RTE | llm | 0.90 |
| datasets | SST-2 | llm | 0.90 |
| tasks | private transformer inference | llm | 0.90 |
| tasks | text classification (GLUE) | llm | 0.90 |
| hardware | NVIDIA A100 GPU | llm | 0.90 |
| security level | 128-bit | llm | 0.90 |
| threat model | client-server; non-interactive outsourced inference | llm | 0.90 |
| interaction | non-interactive FHE | llm | 0.90 |
| code url | https://github.com/crypto-starlab/THOR.BERT (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/google-research/bert (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/Desilo/liberate-fhe (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/crypto-starlab/THOR | llm | 0.90 |
| limitations | Bootstrapping accounts for more than half of the end-to-end runtime | llm | 0.90 |
| limitations | Evaluated only on encoder-only BERT-base; extension to decoder models such as GPT left as future work | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | end-to-end latency: 602.26 s (about 10 minutes) (BERT-base, 128 tokens, single A100 GPU, N = 2^16) | llm | 0.90 |
| key result | bootstrapping time: 337.86 s of 602.26 s (BERT-base end-to-end breakdown) | llm | 0.90 |
| key result | PC-MM speed-up: 5.3x over BOLT, 11.2x over Powerformer (R768x768 x R768x128) | llm | 0.90 |
| key result | CC-MM speed-up: 9.7x over Powerformer, 36x over BOLT (12 x (R64x128 x R128x128)) | llm | 0.90 |
| key result | softmax speed-up: 2.64x over Cho et al. (14.5 ms amortized vs 38.2 ms) (256 softmax operations of dimension 128) | llm | 0.90 |
| key result | accuracy: comparable to plaintext (GLUE MRPC, RTE, SST-2) | llm | 0.90 |
System card: THOR¶
| Group | 0. CKKS LLM frameworks surveyed in the SoK |
| Family | CKKS · Model-preserving (N-PRE) |
| Arithmetic | Approximate real (CKKS) |
| Interaction | non-interactive FHE |
| Model scale | BERT-Base |
| Library | Liberate.FHE |
| Hardware | A100 GPU |
| Packing | L-COL |
| Latency | 602 s / 128-token sequence |
| Open source | yes |
| Status | Active SOTA |
| Relation to the SoK | Model-preserving CKKS evaluation of BERT-Base using normalize-and-square Softmax, Goldschmidt division and aSOR. |
| Note | First model-preserving BERT-Base under CKKS; only 5-6 orders of magnitude slower than FP16 GPU cleartext. |
| Code | https://github.com/Desilo/thor |
| Systems page | THOR |
SoK reference [16]¶
J. Moon, D. Yoo, X. Jiang, and M. Kim, “THOR: Secure transformer inference with homomorphic encryption,” in ACM CCS 2025: 32nd Conference on Computer and Communications Security, C.-Y. Huang, J.-C. Chen, S.-P. Shieh, D. Lie, and V. Cortier, Eds. Taipei, Taiwan: ACM Press, Oct. 13–17, 2025, pp. 3765–3779.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: MPC 7 [p.1], CKKS 4 [p.3]
- library: HElib 2 [p.13], Liberate.FHE 1 [p.13]
- model: Transformer 15 [p.1], BERT 14 [p.1], RNN / LSTM 3 [p.1], GPT 2 [p.1], LLaMA 2 [p.1]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 4 [p.12], SST-2 2 [p.12], GLUE 2 [p.1]
- hardware: GPU 4 [p.1], NVIDIA A100 1 [p.12]
- technique: Matrix multiplication 51 [p.1], Rotations 40 [p.2], Softmax 34 [p.1], Bootstrapping 18 [p.3], GELU 15 [p.1], Key switching 14 [p.2], LayerNorm 11 [p.1], Attention 11 [p.1], SIMD packing 9 [p.1], Relinearization 4 [p.3], Goldschmidt 4 [p.2], Polynomial approximation 3 [p.9], Non-interactive 3 [p.1], RNS 2 [p.11], Fine-tuning 2 [p.10]
- topic: Homomorphic encryption 28 [p.1], Private / secure inference 7 [p.1], Open source 3 [p.12], Large language models 2 [p.1], Benchmark 2 [p.12]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/0-001_thor-secure-transformer-inference-with-homomorphic-encryptio.pdf |
catalogue | 15 | 2.0 MB | ddb57004b56e |
|
references/sok-references/016_thor-secure-transformer-inference-with-homomorphic-encryptio.pdf |
sok-references | 15 | 2.0 MB | ddb57004b56e |
byte-identical to references/catalogue/0-001_thor-secure-transformer-inference-with-homomorphic-encryptio.pdf |
Database key: thor-secure-transformer-inference-with-homomorphic-encryption · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set thor-secure-transformer-inference-with-homomorphic-encryption COLUMN VALUE