Scaling up FHE-based Privacy-Preserving ML: Higher Throughput, Longer Inputs for LLama-3-8B (v1: Scaling up Privacy-Preserving ML: A CKKS Implementation of Llama-2-7B)¶
Authors: Jaiyoung Park; Sejin Park; Jai Hyun Park; Jung Ho Ahn; Jung Hee Cheon; Guillaume Hanrot; Jung Woo Kim; Minje Park; Damien Stehlé
Year: 2026
Venue: arXiv 2601.18511
type: system collection: catalogue, sok-references status: auto
Links: arXiv:2601.18511
Summary¶
Auto-extracted summary (unreviewed)
This paper scales CKKS-based non-interactive private inference of Llama-3-8B in two directions: faster processing of 128 fully encrypted tokens, and support for prompts of thousands of tokens when only the final part is sensitive. It mitigates activation outliers with ML techniques (token prepending, orthogonal rotations), introduces a slim polynomial evaluation for sparsely packed ciphertexts to speed up homomorphic SoftMax, and designs a bootstrapping-frugal plaintext-ciphertext attention (PC-attention) with a new PCMM algorithm for heterogeneous public/private prompts. The resulting GPU system, Sylph, runs 128-token prefill in 20 s and generation at 18 s/token on 8 RTX PRO 6000 GPUs (vs 295 s for Cerium on 8 H100s), and handles a 4096-token input with the last 128 encrypted in 64 s prefill and 22 s/token.
Abstract¶
As large language models (LLMs) become ubiquitous, privacy concerns pertaining to inference keep growing. Fully homomorphic encryption (FHE) has emerged as a primary cryptographic solution for non-interactive confidential LLM inference. However, existing solutions scale poorly with input token length, focusing on small models or input sizes. They also suffer from large outlier values, which strongly impact the evaluation of non-linear layers, leading to heavy polynomial approximation costs. We scale up FHE-based LLM inference in two directions. First, we accelerate FHE-based inference for 128 encrypted tokens. We adopt ML techniques (token prepending and orthogonal rotations) to mitigate outlier impacts on the FHE evaluation of non-linear layers. Separately, we devise a novel polynomial evaluation method for sparsely-packed ciphertexts to speed up our homomorphic SoftMax implementation. We combine these with recent fast homomorphic linear algebra techniques, achieving significantly improved efficiency. Second, we expand the prompt size up to thousands of tokens for contexts where only the final part of the input is sensitive and encrypted. Processing this requires handling standard plaintext-plaintext and ciphertext-ciphertext components, alongside a wide homomorphic computation for a novel plaintext-ciphertext component. To address this, we devise a dedicated homomorphic linear algebra algorithm, building a shallow homomorphic attention circuit that minimizes bootstrapping costs. Based on these ingredients, we present a CKKS-based end-to-end implementation of Llama-3-8B private inference. On 8 NVIDIA RTX PRO 6000 GPUs, 128 encrypted tokens take 20s for summarization and 18s/token for generation (vastly outperforming the SOTA 295s on costlier H100 GPUs). For a heterogeneous 4096-token input (last 128 encrypted), it takes 64s for summarization and 22s/token for generation.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | Outlier mitigation for FHE via token prepending and adaptive orthogonal rotations, enabling low-degree polynomial approximations | llm | 0.85 |
| contribution | Slim polynomial evaluation algorithm for sparsely packed ciphertexts, applied to homomorphic SoftMax | llm | 0.85 |
| contribution | Heterogeneous public/private prompt framework with a novel plaintext-ciphertext attention (PC-attention) stage | llm | 0.85 |
| contribution | Bootstrapping-frugal shallow PC-attention circuit with a new plaintext-ciphertext matrix multiplication (PCMM) algorithm | llm | 0.85 |
| contribution | Sylph: GPU-accelerated end-to-end CKKS implementation of Llama-3-8B private inference | llm | 0.85 |
| schemes | CKKS | llm | 0.85 |
| models | LLaMA-3-8B | llm | 0.85 |
| tasks | private LLM inference (prefill/summarization and token generation) | llm | 0.85 |
| hardware | NVIDIA RTX PRO 6000 GPUs (1-8) | llm | 0.85 |
| hardware | NVIDIA B200 GPUs (1-4) | llm | 0.85 |
| security level | 128-bit | llm | 0.85 |
| threat model | client-server outsourced FHE inference (no non-collusion assumption needed) | llm | 0.85 |
| interaction | non-interactive FHE | llm | 0.85 |
| limitations | Work focuses on the prefill stage; decoding-stage gains are more modest | llm | 0.85 |
| limitations | Long-context support assumes only the last part of the prompt is sensitive and encrypted | llm | 0.85 |
| llm relevance | high | llm | 0.85 |
| key result | prefill latency (summarization): 20 s (Llama-3-8B, 128 encrypted tokens, 8x RTX PRO 6000) | llm | 0.85 |
| key result | generation latency: 18 s/token (Llama-3-8B, 128 encrypted tokens, 8x RTX PRO 6000) | llm | 0.85 |
| key result | prior SOTA prefill latency (Cerium): 295 s (128 encrypted tokens, 8x H100) | llm | 0.85 |
| key result | prefill / generation latency, heterogeneous prompt: 64 s / 22 s per token (3968 public + 128 private tokens, 8x RTX PRO 6000) | llm | 0.85 |
| key result | prefill / decode latency on B200: 26 s / 12 s (128 encrypted tokens, 4x B200) | llm | 0.85 |
| key result | MOAI prefill latency (baseline): 5,963 s (128 encrypted tokens, 1x B200) | llm | 0.85 |
System card: SplitPrompt¶
| Group | 0. CKKS LLM frameworks surveyed in the SoK |
| Family | CKKS · Model-preserving, hybrid public-private prompt split (N-PRE†) |
| Arithmetic | Approximate real (CKKS) |
| Interaction | non-interactive FHE (prompt partitioning) |
| Model scale | Llama-2-7B, Llama-3-8B |
| Library | HEaaN2 |
| Hardware | 8x RTX 4090 GPUs |
| Packing | L-ALG |
| Latency | 33-34 s per output token; 85 s summarization with 4096 tokens (128 encrypted) |
| Open source | no |
| Status | Active SOTA |
| Relation to the SoK | Relaxed threat model: only sensitive tail of the prompt is encrypted, rest processed in the clear. |
| Note | Normalize-and-square Softmax; Chebyshev SiLU; shows practicality of hybrid prompt partitioning. |
| Systems page | SplitPrompt |
SoK reference [78]¶
J. Park, S. Park, J. H. Park, J. H. Ahn, J. H. Cheon, G. Hanrot, J. W. Kim, M. Park, and D. Stehl´e, “Scaling up privacypreserving ml: A ckks implementation of llama-2-7b,” arXiv preprint arXiv:2601.18511, 2026.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 30 [p.1], MPC 3 [p.1]
- library: HElib 1 [p.14]
- model: LLaMA 31 [p.1], Transformer 13 [p.2], GPT 2 [p.14], BERT 1 [p.1]
- hardware: GPU 51 [p.1], NVIDIA H100 3 [p.1], Multi-GPU 3 [p.12], NVIDIA A100 2 [p.2], NVIDIA RTX 1 [p.13], Intel Xeon 1 [p.13], AMD EPYC 1 [p.13]
- technique: Softmax 46 [p.1], Bootstrapping 40 [p.1], Rotations 19 [p.1], Key switching 10 [p.5], Matrix multiplication 10 [p.3], KV cache 10 [p.6], Polynomial approximation 5 [p.1], Rescaling 3 [p.16], Two-party computation 3 [p.14], Paterson-Stockmeyer 2 [p.9], NTT 2 [p.13], Quantization 2 [p.14], Non-interactive 2 [p.1]
- assumption: RLWE 4 [p.4], Lattice-based 3 [p.15], Module-LWE 2 [p.8]
- task: Language modeling / generation 11 [p.2]
- topic: Homomorphic encryption 64 [p.1], Large language models 37 [p.1], Private / secure inference 6 [p.2], Benchmark 2 [p.2]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/0-005_scaling-up-fhe-based-privacy-preserving-ml-higher-throughput.pdf |
catalogue | 17 | 2.2 MB | 11fa21050f82 |
|
references/sok-references/078_scaling-up-privacypreserving-ml-a-ckks-implementation-of-lla.pdf |
sok-references | 17 | 2.2 MB | 11fa21050f82 |
byte-identical to references/catalogue/0-005_scaling-up-fhe-based-privacy-preserving-ml-higher-throughput.pdf |
Database key: scaling-up-fhe-based-privacy-preserving-ml-higher-throughput-longer · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set scaling-up-fhe-based-privacy-preserving-ml-higher-throughput-longer COLUMN VALUE