Skip to content

Scaling up FHE-based Privacy-Preserving ML: Higher Throughput, Longer Inputs for LLama-3-8B (v1: Scaling up Privacy-Preserving ML: A CKKS Implementation of Llama-2-7B)

Authors: Jaiyoung Park; Sejin Park; Jai Hyun Park; Jung Ho Ahn; Jung Hee Cheon; Guillaume Hanrot; Jung Woo Kim; Minje Park; Damien Stehlé
Year: 2026
Venue: arXiv 2601.18511

type: system collection: catalogue, sok-references status: auto

Links: arXiv:2601.18511

Summary

Auto-extracted summary (unreviewed)

This paper scales CKKS-based non-interactive private inference of Llama-3-8B in two directions: faster processing of 128 fully encrypted tokens, and support for prompts of thousands of tokens when only the final part is sensitive. It mitigates activation outliers with ML techniques (token prepending, orthogonal rotations), introduces a slim polynomial evaluation for sparsely packed ciphertexts to speed up homomorphic SoftMax, and designs a bootstrapping-frugal plaintext-ciphertext attention (PC-attention) with a new PCMM algorithm for heterogeneous public/private prompts. The resulting GPU system, Sylph, runs 128-token prefill in 20 s and generation at 18 s/token on 8 RTX PRO 6000 GPUs (vs 295 s for Cerium on 8 H100s), and handles a 4096-token input with the last 128 encrypted in 64 s prefill and 22 s/token.

Abstract

As large language models (LLMs) become ubiquitous, privacy concerns pertaining to inference keep growing. Fully homomorphic encryption (FHE) has emerged as a primary cryptographic solution for non-interactive confidential LLM inference. However, existing solutions scale poorly with input token length, focusing on small models or input sizes. They also suffer from large outlier values, which strongly impact the evaluation of non-linear layers, leading to heavy polynomial approximation costs. We scale up FHE-based LLM inference in two directions. First, we accelerate FHE-based inference for 128 encrypted tokens. We adopt ML techniques (token prepending and orthogonal rotations) to mitigate outlier impacts on the FHE evaluation of non-linear layers. Separately, we devise a novel polynomial evaluation method for sparsely-packed ciphertexts to speed up our homomorphic SoftMax implementation. We combine these with recent fast homomorphic linear algebra techniques, achieving significantly improved efficiency. Second, we expand the prompt size up to thousands of tokens for contexts where only the final part of the input is sensitive and encrypted. Processing this requires handling standard plaintext-plaintext and ciphertext-ciphertext components, alongside a wide homomorphic computation for a novel plaintext-ciphertext component. To address this, we devise a dedicated homomorphic linear algebra algorithm, building a shallow homomorphic attention circuit that minimizes bootstrapping costs. Based on these ingredients, we present a CKKS-based end-to-end implementation of Llama-3-8B private inference. On 8 NVIDIA RTX PRO 6000 GPUs, 128 encrypted tokens take 20s for summarization and 18s/token for generation (vastly outperforming the SOTA 295s on costlier H100 GPUs). For a heterogeneous 4096-token input (last 128 encrypted), it takes 64s for summarization and 22s/token for generation.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution Outlier mitigation for FHE via token prepending and adaptive orthogonal rotations, enabling low-degree polynomial approximations llm 0.85
contribution Slim polynomial evaluation algorithm for sparsely packed ciphertexts, applied to homomorphic SoftMax llm 0.85
contribution Heterogeneous public/private prompt framework with a novel plaintext-ciphertext attention (PC-attention) stage llm 0.85
contribution Bootstrapping-frugal shallow PC-attention circuit with a new plaintext-ciphertext matrix multiplication (PCMM) algorithm llm 0.85
contribution Sylph: GPU-accelerated end-to-end CKKS implementation of Llama-3-8B private inference llm 0.85
schemes CKKS llm 0.85
models LLaMA-3-8B llm 0.85
tasks private LLM inference (prefill/summarization and token generation) llm 0.85
hardware NVIDIA RTX PRO 6000 GPUs (1-8) llm 0.85
hardware NVIDIA B200 GPUs (1-4) llm 0.85
security level 128-bit llm 0.85
threat model client-server outsourced FHE inference (no non-collusion assumption needed) llm 0.85
interaction non-interactive FHE llm 0.85
limitations Work focuses on the prefill stage; decoding-stage gains are more modest llm 0.85
limitations Long-context support assumes only the last part of the prompt is sensitive and encrypted llm 0.85
llm relevance high llm 0.85
key result prefill latency (summarization): 20 s (Llama-3-8B, 128 encrypted tokens, 8x RTX PRO 6000) llm 0.85
key result generation latency: 18 s/token (Llama-3-8B, 128 encrypted tokens, 8x RTX PRO 6000) llm 0.85
key result prior SOTA prefill latency (Cerium): 295 s (128 encrypted tokens, 8x H100) llm 0.85
key result prefill / generation latency, heterogeneous prompt: 64 s / 22 s per token (3968 public + 128 private tokens, 8x RTX PRO 6000) llm 0.85
key result prefill / decode latency on B200: 26 s / 12 s (128 encrypted tokens, 4x B200) llm 0.85
key result MOAI prefill latency (baseline): 5,963 s (128 encrypted tokens, 1x B200) llm 0.85

System card: SplitPrompt

Group 0. CKKS LLM frameworks surveyed in the SoK
Family CKKS · Model-preserving, hybrid public-private prompt split (N-PRE†)
Arithmetic Approximate real (CKKS)
Interaction non-interactive FHE (prompt partitioning)
Model scale Llama-2-7B, Llama-3-8B
Library HEaaN2
Hardware 8x RTX 4090 GPUs
Packing L-ALG
Latency 33-34 s per output token; 85 s summarization with 4096 tokens (128 encrypted)
Open source no
Status Active SOTA
Relation to the SoK Relaxed threat model: only sensitive tail of the prompt is encrypted, rest processed in the clear.
Note Normalize-and-square Softmax; Chebyshev SiLU; shows practicality of hybrid prompt partitioning.
Systems page SplitPrompt

SoK reference [78]

J. Park, S. Park, J. H. Park, J. H. Ahn, J. H. Cheon, G. Hanrot, J. W. Kim, M. Park, and D. Stehl´e, “Scaling up privacypreserving ml: A ckks implementation of llama-2-7b,” arXiv preprint arXiv:2601.18511, 2026.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 30 [p.1], MPC 3 [p.1]
  • library: HElib 1 [p.14]
  • model: LLaMA 31 [p.1], Transformer 13 [p.2], GPT 2 [p.14], BERT 1 [p.1]
  • hardware: GPU 51 [p.1], NVIDIA H100 3 [p.1], Multi-GPU 3 [p.12], NVIDIA A100 2 [p.2], NVIDIA RTX 1 [p.13], Intel Xeon 1 [p.13], AMD EPYC 1 [p.13]
  • technique: Softmax 46 [p.1], Bootstrapping 40 [p.1], Rotations 19 [p.1], Key switching 10 [p.5], Matrix multiplication 10 [p.3], KV cache 10 [p.6], Polynomial approximation 5 [p.1], Rescaling 3 [p.16], Two-party computation 3 [p.14], Paterson-Stockmeyer 2 [p.9], NTT 2 [p.13], Quantization 2 [p.14], Non-interactive 2 [p.1]
  • assumption: RLWE 4 [p.4], Lattice-based 3 [p.15], Module-LWE 2 [p.8]
  • task: Language modeling / generation 11 [p.2]
  • topic: Homomorphic encryption 64 [p.1], Large language models 37 [p.1], Private / secure inference 6 [p.2], Benchmark 2 [p.2]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/0-005_scaling-up-fhe-based-privacy-preserving-ml-higher-throughput.pdf catalogue 17 2.2 MB 11fa21050f82
references/sok-references/078_scaling-up-privacypreserving-ml-a-ckks-implementation-of-lla.pdf sok-references 17 2.2 MB 11fa21050f82 byte-identical to references/catalogue/0-005_scaling-up-fhe-based-privacy-preserving-ml-higher-throughput.pdf

Database key: scaling-up-fhe-based-privacy-preserving-ml-higher-throughput-longer · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set scaling-up-fhe-based-privacy-preserving-ml-higher-throughput-longer COLUMN VALUE