Cachemir: Fully Homomorphic Encrypted Inference of Generative Large Language Model with KV Cache¶
Authors: Ye Yu; Yifan Zhou; Yi Chen; Pedro Soto; Wenjie Xiong; Meng Li
Year: 2026
Venue: arXiv 2602.11470
type: system collection: catalogue, sok-references status: auto
Links: arXiv:2602.11470
Summary¶
Auto-extracted summary (unreviewed)
Cachemir is a CKKS-based FHE framework for autoregressive LLM decoding that makes the KV cache usable under encryption through KV-cache-compatible packing, an interleaved replicated packing scheme for the vector-matrix multiplications of linear layers, and a KV-cache-aware bootstrapping placement with DAG simplification. On CPU it is 48.83x faster than MOAI and 67.16x faster than THOR, and on an A100 GPU it generates one output token for Llama-3-8B in under 100 seconds.
Abstract¶
Generative large language models (LLMs) have revolutionized multiple domains. Modern LLMs predominantly rely on an autoregressive decoding strategy, which generates output tokens sequentially and employs a key-value cache (KV cache) to avoid redundant computation. However, the widespread deployment of LLMs has raised serious privacy concerns, as users are feeding all types of data into the model, motivating the development of secure inference frameworks based on fully homomorphic encryption (FHE). A major limitation of existing FHE-based frameworks is their inability to effectively integrate the KV cache, resulting in prohibitively high latency for autoregressive decoding. In this paper, we propose Cachemir, a KV Cache Accelerated Homomorphic Encrypted LLM Inference Regime to overcome this limitation. Cachemir comprises three key technical contributions: 1) a set of novel HE packing algorithms specifically designed to leverage the computational advantages of the KV cache; 2) an interleaved replicated packing algorithm to efficiently compute the vector-matrix multiplications that result from using the KV cache in Transformer linear layers; and 3) an augmented bootstrapping placement strategy that accounts for the KV cache to minimize bootstrapping cost. We demonstrate that Cachemir achieves $48.83\times$ and $67.16\times$ speedup over MOAI (ICML'25) and THOR (CCS'25) respectively on CPU and consumes less than 100 seconds on GPU to generate an output token for Llama-3-8B.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | HE packing algorithms that exploit the KV cache during encrypted decoding | llm | 0.90 |
| contribution | interleaved replicated packing for efficient encrypted vector-matrix multiplication | llm | 0.90 |
| contribution | augmented bootstrapping placement with DAG simplification that accounts for the KV cache | llm | 0.90 |
| contribution | CPU (Lattigo) and GPU (customized Phantom) implementation and end-to-end evaluation on GPT-2, TinyLlama-1.1B and Llama-3-8B | llm | 0.90 |
| schemes | CKKS | llm | 0.90 |
| libraries | Lattigo | llm | 0.90 |
| libraries | Phantom | llm | 0.90 |
| models | GPT-2 base | llm | 0.90 |
| models | TinyLlama-1.1B | llm | 0.90 |
| models | Llama-3-8B | llm | 0.90 |
| datasets | MRPC | llm | 0.90 |
| datasets | WikiText-2 | llm | 0.90 |
| tasks | private LLM inference | llm | 0.90 |
| tasks | autoregressive text generation | llm | 0.90 |
| hardware | Intel Xeon Platinum 8558P CPU (192 threads) | llm | 0.90 |
| hardware | NVIDIA A100-80GB GPU | llm | 0.90 |
| security level | 128-bit | llm | 0.90 |
| threat model | client-server, single untrusted server (no non-colluding third party) | llm | 0.90 |
| interaction | non-interactive FHE | llm | 0.90 |
| code url | https://github.com/tuneinsight/lattigo.EPFL-LDS (URL found in the full text) | regex | 0.50 |
| code url | https://anonymous.4open.science/r/Cachemir-B254 | llm | 0.90 |
| limitations | baseline latencies are estimated from theoretical operation counts and measured per-operation latency rather than measured end-to-end | llm | 0.90 |
| limitations | models were not fine-tuned on downstream tasks | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | speed-up over MOAI: 48.83x (CPU) | llm | 0.90 |
| key result | speed-up over THOR: 67.16x (CPU) | llm | 0.90 |
| key result | latency per output token: < 100 s (Llama-3-8B on A100 GPU) | llm | 0.90 |
| key result | self-attention decoding step speed-up: 14x to 149x over NEXUS/THOR/MOAI (various models and sequence lengths) | llm | 0.90 |
| key result | VMM latency reduction: 10x to 21x (7x to 19x fewer rotations) (hidden dims 768/2048/4096) | llm | 0.90 |
| key result | bootstrapping placement speed-up: 1.98x (234.24 ms vs 463.31 ms) (vs Orion placement algorithm) | llm | 0.90 |
System card: CACHEMIR¶
| Group | 0. CKKS LLM frameworks surveyed in the SoK |
| Family | CKKS · Model-preserving (N-PRE) |
| Arithmetic | Approximate real (CKKS) |
| Interaction | non-interactive FHE |
| Model scale | Llama-3-8B |
| Library | Lattigo + Phantom |
| Hardware | A100 GPU |
| Packing | L-CYC |
| Latency | 97 s time-per-output-token |
| Open source | no |
| Status | Active SOTA |
| Relation to the SoK | Graph-based bootstrapping placement (extends Orion) and cyclic encodings for multi-head attention. |
| Note | Model-preserving Llama-3-8B decoding with KV cache; about 4 orders of magnitude slower than cleartext. |
| Systems page | CACHEMIR |
SoK reference [74]¶
Y. Yu, Y. Zhou, Y. Chen, P. Soto, W. Xiong, and M. Li, “Cachemir: Fully homomorphic encrypted inference of generative large language model with kv cache,” arXiv preprint arXiv:2602.11470, 2026.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 11 [p.3], MPC 11 [p.1]
- library: Orion 16 [p.2], Phantom 3 [p.10], Lattigo 2 [p.10], Fhelipe 2 [p.12], HElib 1 [p.13], HELayers 1 [p.13]
- model: LLaMA 27 [p.1], Transformer 25 [p.1], GPT 7 [p.1], BERT 3 [p.2], CNN 2 [p.9], ViT 1 [p.14], RNN / LSTM 1 [p.14], GNN 1 [p.14]
- dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 2 [p.10]
- hardware: GPU 9 [p.1], NVIDIA A100 1 [p.10], Intel Xeon 1 [p.10]
- technique: KV cache 71 [p.1], Rotations 63 [p.2], Bootstrapping 59 [p.1], Softmax 29 [p.1], Matrix multiplication 12 [p.2], Attention 10 [p.2], RNS 8 [p.3], SIMD packing 5 [p.3], Convolution 5 [p.2], Pruning 5 [p.3], LayerNorm 4 [p.2], Two-party computation 4 [p.13], Goldschmidt 3 [p.9], Fine-tuning 3 [p.10], GELU 2 [p.2], Non-interactive 2 [p.14]
- task: Language modeling / generation 43 [p.1], Medical / healthcare 2 [p.1], Finance 2 [p.1]
- topic: Large language models 44 [p.1], Homomorphic encryption 41 [p.1], Private / secure inference 17 [p.1], Open source 4 [p.13], Benchmark 2 [p.10]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/0-003_cachemir-fully-homomorphic-encrypted-inference-of-generative.pdf |
catalogue | 16 | 1.8 MB | 67db77545b62 |
|
references/sok-references/074_cachemir-fully-homomorphic-encrypted-inference-of-generative.pdf |
sok-references | 16 | 1.8 MB | 67db77545b62 |
byte-identical to references/catalogue/0-003_cachemir-fully-homomorphic-encrypted-inference-of-generative.pdf |
Database key: cachemir-fully-homomorphic-encrypted-inference-of-generative-large · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set cachemir-fully-homomorphic-encrypted-inference-of-generative-large COLUMN VALUE