Skip to content

Cachemir: Fully Homomorphic Encrypted Inference of Generative Large Language Model with KV Cache

Authors: Ye Yu; Yifan Zhou; Yi Chen; Pedro Soto; Wenjie Xiong; Meng Li
Year: 2026
Venue: arXiv 2602.11470

type: system collection: catalogue, sok-references status: auto

Links: arXiv:2602.11470

Summary

Auto-extracted summary (unreviewed)

Cachemir is a CKKS-based FHE framework for autoregressive LLM decoding that makes the KV cache usable under encryption through KV-cache-compatible packing, an interleaved replicated packing scheme for the vector-matrix multiplications of linear layers, and a KV-cache-aware bootstrapping placement with DAG simplification. On CPU it is 48.83x faster than MOAI and 67.16x faster than THOR, and on an A100 GPU it generates one output token for Llama-3-8B in under 100 seconds.

Abstract

Generative large language models (LLMs) have revolutionized multiple domains. Modern LLMs predominantly rely on an autoregressive decoding strategy, which generates output tokens sequentially and employs a key-value cache (KV cache) to avoid redundant computation. However, the widespread deployment of LLMs has raised serious privacy concerns, as users are feeding all types of data into the model, motivating the development of secure inference frameworks based on fully homomorphic encryption (FHE). A major limitation of existing FHE-based frameworks is their inability to effectively integrate the KV cache, resulting in prohibitively high latency for autoregressive decoding. In this paper, we propose Cachemir, a KV Cache Accelerated Homomorphic Encrypted LLM Inference Regime to overcome this limitation. Cachemir comprises three key technical contributions: 1) a set of novel HE packing algorithms specifically designed to leverage the computational advantages of the KV cache; 2) an interleaved replicated packing algorithm to efficiently compute the vector-matrix multiplications that result from using the KV cache in Transformer linear layers; and 3) an augmented bootstrapping placement strategy that accounts for the KV cache to minimize bootstrapping cost. We demonstrate that Cachemir achieves $48.83\times$ and $67.16\times$ speedup over MOAI (ICML'25) and THOR (CCS'25) respectively on CPU and consumes less than 100 seconds on GPU to generate an output token for Llama-3-8B.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution HE packing algorithms that exploit the KV cache during encrypted decoding llm 0.90
contribution interleaved replicated packing for efficient encrypted vector-matrix multiplication llm 0.90
contribution augmented bootstrapping placement with DAG simplification that accounts for the KV cache llm 0.90
contribution CPU (Lattigo) and GPU (customized Phantom) implementation and end-to-end evaluation on GPT-2, TinyLlama-1.1B and Llama-3-8B llm 0.90
schemes CKKS llm 0.90
libraries Lattigo llm 0.90
libraries Phantom llm 0.90
models GPT-2 base llm 0.90
models TinyLlama-1.1B llm 0.90
models Llama-3-8B llm 0.90
datasets MRPC llm 0.90
datasets WikiText-2 llm 0.90
tasks private LLM inference llm 0.90
tasks autoregressive text generation llm 0.90
hardware Intel Xeon Platinum 8558P CPU (192 threads) llm 0.90
hardware NVIDIA A100-80GB GPU llm 0.90
security level 128-bit llm 0.90
threat model client-server, single untrusted server (no non-colluding third party) llm 0.90
interaction non-interactive FHE llm 0.90
code url https://github.com/tuneinsight/lattigo.EPFL-LDS (URL found in the full text) regex 0.50
code url https://anonymous.4open.science/r/Cachemir-B254 llm 0.90
limitations baseline latencies are estimated from theoretical operation counts and measured per-operation latency rather than measured end-to-end llm 0.90
limitations models were not fine-tuned on downstream tasks llm 0.90
llm relevance high llm 0.90
key result speed-up over MOAI: 48.83x (CPU) llm 0.90
key result speed-up over THOR: 67.16x (CPU) llm 0.90
key result latency per output token: < 100 s (Llama-3-8B on A100 GPU) llm 0.90
key result self-attention decoding step speed-up: 14x to 149x over NEXUS/THOR/MOAI (various models and sequence lengths) llm 0.90
key result VMM latency reduction: 10x to 21x (7x to 19x fewer rotations) (hidden dims 768/2048/4096) llm 0.90
key result bootstrapping placement speed-up: 1.98x (234.24 ms vs 463.31 ms) (vs Orion placement algorithm) llm 0.90

System card: CACHEMIR

Group 0. CKKS LLM frameworks surveyed in the SoK
Family CKKS · Model-preserving (N-PRE)
Arithmetic Approximate real (CKKS)
Interaction non-interactive FHE
Model scale Llama-3-8B
Library Lattigo + Phantom
Hardware A100 GPU
Packing L-CYC
Latency 97 s time-per-output-token
Open source no
Status Active SOTA
Relation to the SoK Graph-based bootstrapping placement (extends Orion) and cyclic encodings for multi-head attention.
Note Model-preserving Llama-3-8B decoding with KV cache; about 4 orders of magnitude slower than cleartext.
Systems page CACHEMIR

SoK reference [74]

Y. Yu, Y. Zhou, Y. Chen, P. Soto, W. Xiong, and M. Li, “Cachemir: Fully homomorphic encrypted inference of generative large language model with kv cache,” arXiv preprint arXiv:2602.11470, 2026.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 11 [p.3], MPC 11 [p.1]
  • library: Orion 16 [p.2], Phantom 3 [p.10], Lattigo 2 [p.10], Fhelipe 2 [p.12], HElib 1 [p.13], HELayers 1 [p.13]
  • model: LLaMA 27 [p.1], Transformer 25 [p.1], GPT 7 [p.1], BERT 3 [p.2], CNN 2 [p.9], ViT 1 [p.14], RNN / LSTM 1 [p.14], GNN 1 [p.14]
  • dataset: GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 2 [p.10]
  • hardware: GPU 9 [p.1], NVIDIA A100 1 [p.10], Intel Xeon 1 [p.10]
  • technique: KV cache 71 [p.1], Rotations 63 [p.2], Bootstrapping 59 [p.1], Softmax 29 [p.1], Matrix multiplication 12 [p.2], Attention 10 [p.2], RNS 8 [p.3], SIMD packing 5 [p.3], Convolution 5 [p.2], Pruning 5 [p.3], LayerNorm 4 [p.2], Two-party computation 4 [p.13], Goldschmidt 3 [p.9], Fine-tuning 3 [p.10], GELU 2 [p.2], Non-interactive 2 [p.14]
  • task: Language modeling / generation 43 [p.1], Medical / healthcare 2 [p.1], Finance 2 [p.1]
  • topic: Large language models 44 [p.1], Homomorphic encryption 41 [p.1], Private / secure inference 17 [p.1], Open source 4 [p.13], Benchmark 2 [p.10]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/0-003_cachemir-fully-homomorphic-encrypted-inference-of-generative.pdf catalogue 16 1.8 MB 67db77545b62
references/sok-references/074_cachemir-fully-homomorphic-encrypted-inference-of-generative.pdf sok-references 16 1.8 MB 67db77545b62 byte-identical to references/catalogue/0-003_cachemir-fully-homomorphic-encrypted-inference-of-generative.pdf

Database key: cachemir-fully-homomorphic-encrypted-inference-of-generative-large · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set cachemir-fully-homomorphic-encrypted-inference-of-generative-large COLUMN VALUE