Skip to content

CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryption

Authors: Hanjun Park; Byeongseo Min; Jiheon Woo; Min-Wook Jeong; Jongho Shin; Yongwoo Lee; Young-Sik Kim; Yongjune Kim
Year: 2026
Venue: arXiv 2602.01621

type: system collection: catalogue status: auto

Links: arXiv:2602.01621

Summary

Auto-extracted summary (unreviewed)

CGF-softmax reformulates the softmax denominator through the cumulant generating function so that encrypted evaluation under CKKS needs neither homomorphic division nor explicit maximum subtraction, reducing multiplicative depth to 7-10 and avoiding bootstrapping. In strictly non-interactive HE inference on Vision Transformers and LLaMA-3.2-1B it keeps accuracy within 1% of plaintext, and a 256x256 softmax runs in 3.25 s versus 107.65 s for the domain-scaling baseline of Cho et al.

Abstract

Homomorphic encryption (HE) is a prominent framework for privacy-preserving machine learning, enabling inference directly on encrypted data. However, evaluating softmax, a core component of transformer architectures, remains particularly challenging in HE due to its multivariate structure, the large dynamic range induced by exponential functions, and the costly division operation. In this paper, we propose CGF-softmax, which reformulates the softmax denominator through the cumulant generating function (CGF). By eliminating both homomorphic division and explicit maximum subtraction, this reformulation substantially reduces multiplicative depth while preserving key properties of softmax. Extensive experiments on Vision Transformers and large language models show that CGF-softmax provides an efficient and accurate approximation of softmax in encrypted inference. In particular, it achieves inference accuracy close to that of high-depth exact methods, while requiring substantially lower computational cost through reduced multiplicative depth.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution CGF-based reformulation of softmax eliminating homomorphic division and max subtraction llm 0.85
contribution theoretical analysis including an asymptotic approximation-error bound llm 0.85
contribution depth and operation-count comparison against prior HE softmax approximations llm 0.85
contribution encrypted-inference accuracy evaluation on Vision Transformers and LLaMA-3.2-1B llm 0.85
schemes CKKS llm 0.85
libraries desilofhe llm 0.85
models Vision Transformer (ViT) llm 0.85
models LLaMA-3.2-1B llm 0.85
datasets ImageNet-1k llm 0.85
tasks private transformer inference llm 0.85
tasks image classification llm 0.85
tasks language modeling llm 0.85
hardware dual AMD EPYC 7763 CPUs (128 cores), 1 TB RAM llm 0.85
threat model strictly non-interactive outsourced inference, server computes on encrypted input llm 0.85
interaction non-interactive FHE llm 0.85
code url https://huggingface.co/meta-llama/Llama-3.2-1B (URL found in the full text) regex 0.50
code url https://github.com/rwightman/pytorch-image-models (URL found in the full text) regex 0.50
limitations for small models (e.g. BERT) on simple tasks (e.g. SST-2) existing low-depth softmax replacements such as Power-softmax and BPMax may already be sufficient llm 0.85
llm relevance high llm 0.85
key result multiplicative depth of softmax: 7 to 10 (including LLaMA-3.2-1B and ImageNet-1k with 1000 classes) llm 0.85
key result softmax latency: 3.25 s vs 107.65 s for Cho et al. (256x256 input, interval [-128, 0], 8 CPU threads, no bootstrapping needed) llm 0.85
key result accuracy degradation: within 1% of plaintext baseline (LLaMA-3.2-1B and Vision Transformers) llm 0.85
key result evaluation noise: 3.05e-10 (multiplicative depth 7) llm 0.85

System card: CGF-Softmax

Group 1. Newer CKKS / FHE LLM frameworks (2026, not in the SoK)
Family CKKS non-linear evaluation
Arithmetic Approximate real (CKKS)
Interaction Non-interactive FHE
Model scale ViT and LLMs
Status Active SOTA
Relation to the SoK Softmax without homomorphic division or max subtraction.
Note Lower depth than normalize-and-square, but it is an approximation of Softmax rather than exact model preservation.
Systems page CGF-Softmax

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 5 [p.3], MPC 5 [p.1]
  • library: HuggingFace 2 [p.8], PyTorch 1 [p.12]
  • model: ViT 20 [p.1], LLaMA 19 [p.2], Transformer 9 [p.1], BERT 7 [p.4], GPT 3 [p.7], LoRA 3 [p.11]
  • dataset: SST-2 12 [p.2], ImageNet 9 [p.2], GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 5 [p.7], GLUE 1 [p.12], HellaSwag 1 [p.7], ARC 1 [p.7]
  • hardware: AMD EPYC 1 [p.7], GPU 1 [p.10]
  • technique: Softmax 255 [p.1], Fine-tuning 16 [p.7], Polynomial approximation 15 [p.2], Bootstrapping 8 [p.2], Chebyshev 6 [p.3], Goldschmidt 6 [p.3], Non-interactive 5 [p.1], Attention 3 [p.2], Knowledge distillation 3 [p.17], Rescaling 2 [p.15], ReLU 2 [p.4], Rotations 2 [p.3]
  • task: Finance 6 [p.2]
  • topic: Homomorphic encryption 52 [p.1], Large language models 10 [p.1], Benchmark 10 [p.2], Privacy-preserving machine learning 6 [p.1], Private / secure inference 3 [p.10]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/1-128_cgf-softmax-a-cumulant-based-softmax-reformulation-for-effic.pdf catalogue 19 970.2 KB da08c8d06c68

Database key: cgf-softmax-a-cumulant-based-softmax-reformulation-for-efficient · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set cgf-softmax-a-cumulant-based-softmax-reformulation-for-efficient COLUMN VALUE