Skip to content

Powerformer: Efficient and high-accuracy privacy-preserving language model with homomorphic encryption

Authors: D. Park; E. Lee; J.-W. Lee
Year: 2025
Venue: ACL 2025 (long)

type: system collection: catalogue, sok-references status: auto

Links: link

Summary

Auto-extracted summary (unreviewed)

Powerformer is an HE-based private language model that distills BERT-base so that softmax and layer normalization are replaced by HE-friendly power (BPMax) and linear (Batch LN) functions, approximates GELU and tanh with a pseudo-sign composite polynomial method, and uses a homomorphic matrix multiplication tailored to transformers. On the GPU Liberate.FHE library with 128-bit security it cuts BERT-base inference time by 45% relative to THOR (10.43 min to 5.74 min on one A100) with no accuracy loss on RTE, MRPC and SST-2.

Abstract

We propose Powerformer, an efficient homomorphic encryption (HE)-based privacypreserving language model (PPLM) designed to reduce computational overhead while maintaining model performance. Powerformer incorporates three key techniques to optimize encrypted computations: 1) A novel distillation technique that replaces softmax and layer normalization with computationally efficient power and linear functions, ensuring no performance degradation while enabling seamless encrypted computation. 2) A pseudo-sign composite approximation method that accurately approximates GELU and tanh functions with minimal computational overhead. 3) A homomorphic matrix multiplication algorithm specifically optimized for Transformer models, enhancing efficiency in encrypted environments. By integrating these techniques, Powerformer based on the BERT-base model achieves a 45% reduction in computation time compared to the state-of-the-art HE-based PPLM without any loss in accuracy.
Source: pdf

Extracted details

Field Value Source Confidence
contribution Distillation replacing softmax and layer norm with power/linear functions (BPMax, Batch LN) without accuracy loss llm 0.90
contribution Pseudo-sign composite approximation for GELU and tanh llm 0.90
contribution Homomorphic matrix multiplication optimized for transformer layers llm 0.90
contribution 45% faster encrypted BERT-base inference than THOR llm 0.90
schemes CKKS llm 0.90
libraries Liberate.FHE (GPU) llm 0.90
models BERT-base llm 0.90
datasets GLUE (RTE, MRPC, SST-2) llm 0.90
datasets SuperGLUE BoolQ llm 0.90
tasks private language model inference llm 0.90
tasks text classification llm 0.90
hardware NVIDIA A100 GPU (single) llm 0.90
security level 128-bit llm 0.90
threat model semi-honest client and server llm 0.90
interaction non-interactive FHE llm 0.90
code url https://github.com/thrudgelmir/Powerformer.11090 (URL found in the full text) regex 0.50
code url https://github.com/thrudgelmir/Powerformer llm 0.90
limitations Assumes a semi-honest security model; a malicious setting would require MPC-based designs llm 0.90
limitations Slight accuracy decrease (0.77%) on longer sequences (BoolQ, length 256) attributed to the normalization parameter l llm 0.90
limitations Parameter l must be fine-tuned to avoid feature explosion without hurting accuracy llm 0.90
llm relevance high llm 0.90
key result encrypted inference latency: 5.74 min (BERT-base, sequence length 128, single A100 GPU) llm 0.90
key result speed-up: 45% reduction in computation time vs THOR (10.43 min) (same BERT-base setting) llm 0.90
key result average accuracy (plaintext, HE-friendly model): 83.11% vs 82.86% baseline (RTE/MRPC/SST-2, p=5, c=5, 1-step distillation with extra loss) llm 0.90
key result accuracy drop on longer context: 0.77% (BoolQ, sequence length 256) llm 0.90

System card: Powerformer

Group 0. CKKS LLM frameworks surveyed in the SoK
Family CKKS · Model-modifying (N-MOD)
Arithmetic Approximate real (CKKS)
Interaction non-interactive FHE
Model scale BERT-Base
Library Liberate.FHE
Hardware A100 GPU
Packing L-COL
Latency 345 s unamortized (128 tokens)
Open source yes
Status Active SOTA
Relation to the SoK Batch rectifier power max replaces Softmax; 90% reduction in Softmax cost.
Note Requires retraining/distillation.
Systems page Powerformer

SoK reference [71]

D. Park, E. Lee, and J.-W. Lee, “Powerformer: Efficient and highaccuracy privacy-preserving language model with homomorphic encryption,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025, pp. 11 090–11 111.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 5 [p.2], MPC 4 [p.2]
  • library: Liberate.FHE 1 [p.7]
  • model: BERT 19 [p.1], Transformer 10 [p.1]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 11 [p.7], SST-2 4 [p.7], GLUE 3 [p.7]
  • hardware: GPU 3 [p.1], NVIDIA A100 1 [p.7]
  • technique: Minimax approximation 28 [p.4], Matrix multiplication 27 [p.1], Softmax 25 [p.1], Rotations 19 [p.6], GELU 11 [p.1], Remez 6 [p.4], LayerNorm 5 [p.1], Relinearization 4 [p.7], ReLU 4 [p.2], RNS 4 [p.2], Fine-tuning 4 [p.1], Semi-honest model 4 [p.10], Bootstrapping 3 [p.7], Chebyshev 2 [p.14], Attention 2 [p.9]
  • topic: Homomorphic encryption 47 [p.1], Large language models 2 [p.11], Benchmark 2 [p.8]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/0-013_powerformer-efficient-and-high-accuracy-privacy-preserving-l.pdf catalogue 22 874.0 KB 58812c43b9e2
references/sok-references/071_powerformer-efficient-and-high-accuracy-privacy-preserving-l.pdf sok-references 22 874.0 KB 58812c43b9e2 byte-identical to references/catalogue/0-013_powerformer-efficient-and-high-accuracy-privacy-preserving-l.pdf

Database key: powerformer-efficient-and-high-accuracy-privacy-preserving-language · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set powerformer-efficient-and-high-accuracy-privacy-preserving-language COLUMN VALUE