Powerformer: Efficient and high-accuracy privacy-preserving language model with homomorphic encryption¶
Authors: D. Park; E. Lee; J.-W. Lee
Year: 2025
Venue: ACL 2025 (long)
type: system collection: catalogue, sok-references status: auto
Links: link
Summary¶
Auto-extracted summary (unreviewed)
Powerformer is an HE-based private language model that distills BERT-base so that softmax and layer normalization are replaced by HE-friendly power (BPMax) and linear (Batch LN) functions, approximates GELU and tanh with a pseudo-sign composite polynomial method, and uses a homomorphic matrix multiplication tailored to transformers. On the GPU Liberate.FHE library with 128-bit security it cuts BERT-base inference time by 45% relative to THOR (10.43 min to 5.74 min on one A100) with no accuracy loss on RTE, MRPC and SST-2.
Abstract¶
We propose Powerformer, an efficient homomorphic encryption (HE)-based privacypreserving language model (PPLM) designed to reduce computational overhead while maintaining model performance. Powerformer incorporates three key techniques to optimize encrypted computations: 1) A novel distillation technique that replaces softmax and layer normalization with computationally efficient power and linear functions, ensuring no performance degradation while enabling seamless encrypted computation. 2) A pseudo-sign composite approximation method that accurately approximates GELU and tanh functions with minimal computational overhead. 3) A homomorphic matrix multiplication algorithm specifically optimized for Transformer models, enhancing efficiency in encrypted environments. By integrating these techniques, Powerformer based on the BERT-base model achieves a 45% reduction in computation time compared to the state-of-the-art HE-based PPLM without any loss in accuracy.
Source: pdf
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | Distillation replacing softmax and layer norm with power/linear functions (BPMax, Batch LN) without accuracy loss | llm | 0.90 |
| contribution | Pseudo-sign composite approximation for GELU and tanh | llm | 0.90 |
| contribution | Homomorphic matrix multiplication optimized for transformer layers | llm | 0.90 |
| contribution | 45% faster encrypted BERT-base inference than THOR | llm | 0.90 |
| schemes | CKKS | llm | 0.90 |
| libraries | Liberate.FHE (GPU) | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| datasets | GLUE (RTE, MRPC, SST-2) | llm | 0.90 |
| datasets | SuperGLUE BoolQ | llm | 0.90 |
| tasks | private language model inference | llm | 0.90 |
| tasks | text classification | llm | 0.90 |
| hardware | NVIDIA A100 GPU (single) | llm | 0.90 |
| security level | 128-bit | llm | 0.90 |
| threat model | semi-honest client and server | llm | 0.90 |
| interaction | non-interactive FHE | llm | 0.90 |
| code url | https://github.com/thrudgelmir/Powerformer.11090 (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/thrudgelmir/Powerformer | llm | 0.90 |
| limitations | Assumes a semi-honest security model; a malicious setting would require MPC-based designs | llm | 0.90 |
| limitations | Slight accuracy decrease (0.77%) on longer sequences (BoolQ, length 256) attributed to the normalization parameter l | llm | 0.90 |
| limitations | Parameter l must be fine-tuned to avoid feature explosion without hurting accuracy | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | encrypted inference latency: 5.74 min (BERT-base, sequence length 128, single A100 GPU) | llm | 0.90 |
| key result | speed-up: 45% reduction in computation time vs THOR (10.43 min) (same BERT-base setting) | llm | 0.90 |
| key result | average accuracy (plaintext, HE-friendly model): 83.11% vs 82.86% baseline (RTE/MRPC/SST-2, p=5, c=5, 1-step distillation with extra loss) | llm | 0.90 |
| key result | accuracy drop on longer context: 0.77% (BoolQ, sequence length 256) | llm | 0.90 |
System card: Powerformer¶
| Group | 0. CKKS LLM frameworks surveyed in the SoK |
| Family | CKKS · Model-modifying (N-MOD) |
| Arithmetic | Approximate real (CKKS) |
| Interaction | non-interactive FHE |
| Model scale | BERT-Base |
| Library | Liberate.FHE |
| Hardware | A100 GPU |
| Packing | L-COL |
| Latency | 345 s unamortized (128 tokens) |
| Open source | yes |
| Status | Active SOTA |
| Relation to the SoK | Batch rectifier power max replaces Softmax; 90% reduction in Softmax cost. |
| Note | Requires retraining/distillation. |
| Systems page | Powerformer |
SoK reference [71]¶
D. Park, E. Lee, and J.-W. Lee, “Powerformer: Efficient and highaccuracy privacy-preserving language model with homomorphic encryption,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 11 090–11 111.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 5 [p.2], MPC 4 [p.2]
- library: Liberate.FHE 1 [p.7]
- model: BERT 19 [p.1], Transformer 10 [p.1]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 11 [p.7], SST-2 4 [p.7], GLUE 3 [p.7]
- hardware: GPU 3 [p.1], NVIDIA A100 1 [p.7]
- technique: Minimax approximation 28 [p.4], Matrix multiplication 27 [p.1], Softmax 25 [p.1], Rotations 19 [p.6], GELU 11 [p.1], Remez 6 [p.4], LayerNorm 5 [p.1], Relinearization 4 [p.7], ReLU 4 [p.2], RNS 4 [p.2], Fine-tuning 4 [p.1], Semi-honest model 4 [p.10], Bootstrapping 3 [p.7], Chebyshev 2 [p.14], Attention 2 [p.9]
- topic: Homomorphic encryption 47 [p.1], Large language models 2 [p.11], Benchmark 2 [p.8]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/0-013_powerformer-efficient-and-high-accuracy-privacy-preserving-l.pdf |
catalogue | 22 | 874.0 KB | 58812c43b9e2 |
|
references/sok-references/071_powerformer-efficient-and-high-accuracy-privacy-preserving-l.pdf |
sok-references | 22 | 874.0 KB | 58812c43b9e2 |
byte-identical to references/catalogue/0-013_powerformer-efficient-and-high-accuracy-privacy-preserving-l.pdf |
Database key: powerformer-efficient-and-high-accuracy-privacy-preserving-language · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set powerformer-efficient-and-high-accuracy-privacy-preserving-language COLUMN VALUE