An Open-Source End-to-End FHE Implementation for Privacy-Preserving Llama 3 8B Inference¶
Authors: Yuhang Fan; Yusi Chen; Kanyu Ye; Zhuoran Ji
Year: 2026
Venue: arXiv 2609.12378
type: system collection: catalogue status: auto
Links: arXiv:2609.12378
Summary¶
Auto-extracted summary (unreviewed)
Odin is a non-interactive CKKS inference system for Llama-3 that co-designs ciphertext packing with model execution: a feature-major cross-layer layout unifies residual connections and layer interfaces, transient intra-operator layouts serve linear projections and attention, and QK^T scores feed Softmax directly while PV consumes the probabilities without repacking. Nonlinear ops (RMSNorm, Softmax, SiLU) use minimax polynomials with calibrated input ranges and joint error allocation. On Llama-3-8B with a 128-token input, all 32 layers run in 366.4 s on one H100 80 GB with 58.9 GiB peak memory, a 4.51x speedup over THOR (1651.9 s).
Abstract¶
Cloud LLM services typically require users to send prompts to a model provider, creating a privacy risk. Fully homomorphic encryption (FHE) lets a server perform inference without decrypting the input, but representing data as ciphertexts adds storage and computational overhead. In CKKS-based LLM inference, the packing scheme maps logical tensors to ciphertexts and slots. It therefore determines the ciphertext count and the homomorphic cost of linear layers, and it constrains how data pass between linear layers, attention, and nonlinear computation. As models and sequences grow, inefficient layouts accumulate encoding, compute, and layout-conversion overhead. We present Odin, an FHE inference system that co-designs ciphertext packing and model execution for Llama. Starting from a THOR-style baseline whose bottleneck is weight encoding, Odin uses a feature-major cross-layer layout to unify residual connections and layer interfaces, and builds transient intra-operator layouts for linear projections and attention. This reduces redundant plaintext encoding of weights in wide projections. Within attention, QK^T produces scores that Softmax can consume directly, and PV consumes the resulting probabilities, avoiding intermediate repacking. For nonlinear ops, we use minimax polynomial approximation with input-range control and joint error allocation guided by model quality, reducing polynomial degree and multiplicative depth. To our knowledge, Odin is the first open-source end-to-end GPU CKKS implementation of Llama-3. With Llama-3-8B weights and a 128-token input, Odin evaluates all 32 Transformer layers on a single NVIDIA H100 80 GB GPU. Server-side end-to-end FHE evaluation takes 366.4 s and 58.9 GiB peak device memory. Under the same model, input, CKKS parameters, and hardware, THOR takes 1651.9 s, a 4.51x speedup.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | feature-major cross-layer ciphertext layout unifying residual and inter-layer interfaces | llm | 0.90 |
| contribution | transient intra-operator layouts for linear projections and attention that reuse encoded weight diagonals | llm | 0.90 |
| contribution | attention dataflow where Softmax consumes QK^T output directly and PV consumes Softmax output without repacking | llm | 0.90 |
| contribution | minimax polynomial approximation with input-range calibration and model-quality-guided joint error allocation | llm | 0.90 |
| contribution | first open-source end-to-end GPU CKKS implementation of Llama-3 (model conversion pipeline, GPU execution, configs) | llm | 0.90 |
| schemes | CKKS | llm | 0.90 |
| libraries | EasyFHE | llm | 0.90 |
| models | LLaMA-3-8B | llm | 0.90 |
| datasets | WikiText-2 | llm | 0.90 |
| datasets | C4 | llm | 0.90 |
| tasks | private LLM inference | llm | 0.90 |
| tasks | language modeling | llm | 0.90 |
| hardware | NVIDIA H100 80 GB GPU | llm | 0.90 |
| threat model | client-server, single untrusted server (non-interactive, no non-collusion assumption) | llm | 0.90 |
| interaction | non-interactive FHE | llm | 0.90 |
| limitations | safety expansion factor (1.2) on approximation intervals raises polynomial degree for a given error target | llm | 0.90 |
| limitations | per-layer optimal error budgets are intractable; operators of the same type share one budget across layers | llm | 0.90 |
| limitations | future work needed to extend the cross-layer interface to more models and input scales and to co-optimize packing, nonlinear precision and bootstrapping schedules | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | end-to-end server-side FHE latency: 366.4 s (Llama-3-8B, 128-token input, 32 layers, 1x H100 80 GB) | llm | 0.90 |
| key result | peak device memory: 58.9 GiB (Llama-3-8B, 128-token input, 1x H100 80 GB) | llm | 0.90 |
| key result | speedup over THOR: 4.51x (1651.9 s -> 366.4 s) (same model, input, CKKS parameters and hardware) | llm | 0.90 |
| key result | single-layer plaintext-ciphertext projection time: 41.37 s -> 5.32 s (Llama-3-8B, one layer, H100) | llm | 0.90 |
| key result | intermediate layer execution time: 60.87 s -> 11.12 s (Llama-3-8B, one intermediate layer, H100) | llm | 0.90 |
System card: Odin¶
| Group | 1. Newer CKKS / FHE LLM frameworks (2026, not in the SoK) |
| Family | CKKS framework |
| Arithmetic | Approximate real (CKKS) |
| Interaction | Non-interactive FHE |
| Model scale | Llama-3-8B |
| Status | Active SOTA |
| Relation to the SoK | Open-source end-to-end CKKS Llama-3-8B with co-designed packing. |
| Note | Feature-major cross-layer layout and transient per-operator layouts avoid repacking between attention and projections. |
| Systems page | Odin |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 32 [p.1], MPC 5 [p.1], Secret sharing 2 [p.1]
- library: PyTorch 1 [p.10]
- model: LLaMA 52 [p.1], Transformer 19 [p.1], BERT 6 [p.2], MLP 1 [p.12]
- dataset: WikiText 4 [p.4]
- hardware: GPU 10 [p.1], NVIDIA H100 6 [p.1], NVIDIA A100 1 [p.12], NVIDIA RTX 1 [p.12]
- technique: Softmax 72 [p.1], Bootstrapping 19 [p.3], Rotations 19 [p.1], Remez 9 [p.3], SIMD packing 7 [p.1], Minimax approximation 6 [p.1], Matrix multiplication 6 [p.2], Rescaling 5 [p.3], Non-interactive 5 [p.5], Polynomial approximation 3 [p.2], GELU 2 [p.2], LayerNorm 2 [p.2], Two-party computation 2 [p.2]
- assumption: RLWE 4 [p.3], Module-LWE 2 [p.5]
- topic: Homomorphic encryption 56 [p.1], Large language models 14 [p.1], Open source 6 [p.1], Private / secure inference 5 [p.2]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/1-120_an-open-source-end-to-end-fhe-implementation-for-privacy-pre.pdf |
catalogue | 13 | 155.5 KB | c65b9aba7215 |
Database key: an-open-source-end-to-end-fhe-implementation-for-privacy-preserving · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set an-open-source-end-to-end-fhe-implementation-for-privacy-preserving COLUMN VALUE