CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryption¶
Authors: Hanjun Park; Byeongseo Min; Jiheon Woo; Min-Wook Jeong; Jongho Shin; Yongwoo Lee; Young-Sik Kim; Yongjune Kim
Year: 2026
Venue: arXiv 2602.01621
type: system collection: catalogue status: auto
Links: arXiv:2602.01621
Summary¶
Auto-extracted summary (unreviewed)
CGF-softmax reformulates the softmax denominator through the cumulant generating function so that encrypted evaluation under CKKS needs neither homomorphic division nor explicit maximum subtraction, reducing multiplicative depth to 7-10 and avoiding bootstrapping. In strictly non-interactive HE inference on Vision Transformers and LLaMA-3.2-1B it keeps accuracy within 1% of plaintext, and a 256x256 softmax runs in 3.25 s versus 107.65 s for the domain-scaling baseline of Cho et al.
Abstract¶
Homomorphic encryption (HE) is a prominent framework for privacy-preserving machine learning, enabling inference directly on encrypted data. However, evaluating softmax, a core component of transformer architectures, remains particularly challenging in HE due to its multivariate structure, the large dynamic range induced by exponential functions, and the costly division operation. In this paper, we propose CGF-softmax, which reformulates the softmax denominator through the cumulant generating function (CGF). By eliminating both homomorphic division and explicit maximum subtraction, this reformulation substantially reduces multiplicative depth while preserving key properties of softmax. Extensive experiments on Vision Transformers and large language models show that CGF-softmax provides an efficient and accurate approximation of softmax in encrypted inference. In particular, it achieves inference accuracy close to that of high-depth exact methods, while requiring substantially lower computational cost through reduced multiplicative depth.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | CGF-based reformulation of softmax eliminating homomorphic division and max subtraction | llm | 0.85 |
| contribution | theoretical analysis including an asymptotic approximation-error bound | llm | 0.85 |
| contribution | depth and operation-count comparison against prior HE softmax approximations | llm | 0.85 |
| contribution | encrypted-inference accuracy evaluation on Vision Transformers and LLaMA-3.2-1B | llm | 0.85 |
| schemes | CKKS | llm | 0.85 |
| libraries | desilofhe | llm | 0.85 |
| models | Vision Transformer (ViT) | llm | 0.85 |
| models | LLaMA-3.2-1B | llm | 0.85 |
| datasets | ImageNet-1k | llm | 0.85 |
| tasks | private transformer inference | llm | 0.85 |
| tasks | image classification | llm | 0.85 |
| tasks | language modeling | llm | 0.85 |
| hardware | dual AMD EPYC 7763 CPUs (128 cores), 1 TB RAM | llm | 0.85 |
| threat model | strictly non-interactive outsourced inference, server computes on encrypted input | llm | 0.85 |
| interaction | non-interactive FHE | llm | 0.85 |
| code url | https://huggingface.co/meta-llama/Llama-3.2-1B (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/rwightman/pytorch-image-models (URL found in the full text) | regex | 0.50 |
| limitations | for small models (e.g. BERT) on simple tasks (e.g. SST-2) existing low-depth softmax replacements such as Power-softmax and BPMax may already be sufficient | llm | 0.85 |
| llm relevance | high | llm | 0.85 |
| key result | multiplicative depth of softmax: 7 to 10 (including LLaMA-3.2-1B and ImageNet-1k with 1000 classes) | llm | 0.85 |
| key result | softmax latency: 3.25 s vs 107.65 s for Cho et al. (256x256 input, interval [-128, 0], 8 CPU threads, no bootstrapping needed) | llm | 0.85 |
| key result | accuracy degradation: within 1% of plaintext baseline (LLaMA-3.2-1B and Vision Transformers) | llm | 0.85 |
| key result | evaluation noise: 3.05e-10 (multiplicative depth 7) | llm | 0.85 |
System card: CGF-Softmax¶
| Group | 1. Newer CKKS / FHE LLM frameworks (2026, not in the SoK) |
| Family | CKKS non-linear evaluation |
| Arithmetic | Approximate real (CKKS) |
| Interaction | Non-interactive FHE |
| Model scale | ViT and LLMs |
| Status | Active SOTA |
| Relation to the SoK | Softmax without homomorphic division or max subtraction. |
| Note | Lower depth than normalize-and-square, but it is an approximation of Softmax rather than exact model preservation. |
| Systems page | CGF-Softmax |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 5 [p.3], MPC 5 [p.1]
- library: HuggingFace 2 [p.8], PyTorch 1 [p.12]
- model: ViT 20 [p.1], LLaMA 19 [p.2], Transformer 9 [p.1], BERT 7 [p.4], GPT 3 [p.7], LoRA 3 [p.11]
- dataset: SST-2 12 [p.2], ImageNet 9 [p.2], GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 5 [p.7], GLUE 1 [p.12], HellaSwag 1 [p.7], ARC 1 [p.7]
- hardware: AMD EPYC 1 [p.7], GPU 1 [p.10]
- technique: Softmax 255 [p.1], Fine-tuning 16 [p.7], Polynomial approximation 15 [p.2], Bootstrapping 8 [p.2], Chebyshev 6 [p.3], Goldschmidt 6 [p.3], Non-interactive 5 [p.1], Attention 3 [p.2], Knowledge distillation 3 [p.17], Rescaling 2 [p.15], ReLU 2 [p.4], Rotations 2 [p.3]
- task: Finance 6 [p.2]
- topic: Homomorphic encryption 52 [p.1], Large language models 10 [p.1], Benchmark 10 [p.2], Privacy-preserving machine learning 6 [p.1], Private / secure inference 3 [p.10]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/1-128_cgf-softmax-a-cumulant-based-softmax-reformulation-for-effic.pdf |
catalogue | 19 | 970.2 KB | da08c8d06c68 |
Database key: cgf-softmax-a-cumulant-based-softmax-reformulation-for-efficient · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set cgf-softmax-a-cumulant-based-softmax-reformulation-for-efficient COLUMN VALUE