Skip to content

A scalable multi-GPU framework for encrypted large-model inference

Authors: Siddharth Jayashankar; Joshua Kim; Michael B. Sullivan; Wenting Zheng; Dimitrios Skarlatos
Year: 2025
Venue: arXiv 2512.11269

type: system collection: catalogue, sok-references status: auto

Links: DOI · arXiv:2512.11269

Summary

Auto-extracted summary (unreviewed)

Cerium is a multi-GPU framework for CKKS-based encrypted inference on large models that pairs a domain-specific language with an optimizing compiler and a runtime: it generates fused GPU kernels automatically, compresses plaintext weights with sparse polynomial representations, manages terabyte-scale memory footprints between host and GPUs, and parallelises computation across devices with communication-aware partitioning. On NVIDIA GPUs it beats expert-written GPU libraries by up to 2.25x on small models, matches the CraterLake FHE ASIC, performs CKKS bootstrapping in 7.5 ms, and is the first system to run end-to-end encrypted BERT-Base inference (8 s) and Llama3-8B prefill (43 s in the arXiv abstract, 134 s in the paper body).

Abstract

Encrypted AI using fully homomorphic encryption (FHE) enables inference directly over encrypted queries, providing strong privacy guarantees. However, its computational and memory overheads have limited practical deployment. Custom FHE accelerators improve performance, but rely on advanced manufacturing technologies that limit their accessibility. GPUs offer a more widely available alternative, yet achieving ASIC-class performance on GPUs is challenging. Large models such as LLMs compound these challenges by requiring optimized kernels, terabyte-scale memory management, and efficient execution across multiple devices. We present Cerium, a multi-GPU framework for large-scale FHE inference. Cerium combines a domain-specific language, an optimizing compiler, and a runtime system to automatically generate optimized GPU kernels, compress plaintext data, manage terabyte-scale memory footprints, and optimize multi-GPU communication. Implemented on NVIDIA GPUs, Cerium outperforms expert-written GPU libraries by up to 2.25x and achieves performance competitive with state-of-the-art FHE ASICs, matching the CraterLake ASIC. It is the first GPU system to execute CKKS bootstrapping in under 10 milliseconds, achieving 7.5 milliseconds, and is the first to demonstrate encrypted inference for BERT-Base and Llama3-8B prefill in 8 seconds and 43 seconds, respectively. The Cerium framework is open-source and available at https://github.com/CMU-CAOS/Cerium
Source: arxiv

Extracted details

Field Value Source Confidence
contribution multi-GPU FHE inference framework with DSL, optimizing compiler and runtime llm 0.85
contribution new IR constructs and compiler passes that automatically generate fused GPU kernels llm 0.85
contribution sparse polynomial representations and memory-efficient data layouts to compress plaintext data llm 0.85
contribution terabyte-scale memory management across host and GPUs llm 0.85
contribution communication-aware multi-GPU parallelization llm 0.85
contribution first GPU CKKS bootstrapping under 10 ms and first encrypted BERT-Base and Llama3-8B inference llm 0.85
schemes CKKS llm 0.85
libraries Cheddar (baseline GPU library) llm 0.85
models BERT-base llm 0.85
models Llama3-8B llm 0.85
models ResNet-20 llm 0.85
tasks encrypted LLM inference (prefill) llm 0.85
tasks encrypted CNN inference llm 0.85
tasks CKKS bootstrapping llm 0.85
hardware NVIDIA GPUs (multi-GPU) llm 0.85
security level 128-bit (N = 64K, max ciphertext modulus 1782 bits) llm 0.85
threat model client-server; encrypted queries to a cloud-hosted model llm 0.85
interaction non-interactive FHE llm 0.85
code url https://github.com/Desilo/liberate-fhe (URL found in the full text) regex 0.50
code url https://github.com/CMU-CAOS/Cerium llm 0.85
llm relevance high llm 0.85
key result speed-up vs expert-written GPU libraries: up to 2.25x (small models, NVIDIA GPUs) llm 0.85
key result CKKS bootstrapping latency: 7.5 ms (first GPU system under 10 ms) llm 0.85
key result BERT-Base encrypted inference latency: 8 s (multi-GPU, 128-bit security) llm 0.85
key result Llama3-8B encrypted prefill latency: 43 s (arXiv abstract) / 134 s (paper body) (multi-GPU, 128-bit security) llm 0.85
key result ASIC comparison: matches CraterLake FHE ASIC (performance competitive with state-of-the-art FHE ASICs) llm 0.85

System card: CERIUM

Group 0. CKKS LLM frameworks surveyed in the SoK
Family CKKS · Model-preserving (N-PRE)
Arithmetic Approximate real (CKKS)
Interaction non-interactive FHE
Model scale Llama-3-8B, BERT-Base
Library Custom multi-GPU CKKS compiler
Hardware 1-8x A100/H100/B200 GPUs
Packing compressed sparse plaintext encoding
Latency 134 s prefill (128 tokens, 8x B200); 66 s BERT-Base on A100
Open source planned
Status Active SOTA
Relation to the SoK First end-to-end multi-GPU CKKS compiler with kernel fusion; ~14 KB ciphertext memory per model parameter.
Note Best-known runtimes for model-preserving Llama-3-8B; bootstrapping under 10 ms on multi-GPU cluster.
Systems page CERIUM

SoK reference [18]

S. Jayashankar, J. Kim, M. B. Sullivan, W. Zheng, and D. Skarlatos, “A scalable multi-gpu framework for encrypted large-model inference,” arXiv preprint arXiv:2512.11269, 2025.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 11 [p.2], MPC 3 [p.11]
  • library: Cheddar 8 [p.1], HElib 1 [p.12], Phantom 1 [p.13], Liberate.FHE 1 [p.12], Orion 1 [p.12], HEIR 1 [p.12]
  • model: BERT 31 [p.1], ResNet 18 [p.1], MobileNet 2 [p.3], CNN 2 [p.1], LoRA 1 [p.9]
  • dataset: GLUE 2 [p.9], CIFAR-10 1 [p.9], GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 1 [p.9]
  • hardware: GPU 154 [p.1], ASIC 29 [p.1], Multi-GPU 21 [p.1], NVIDIA A100 11 [p.9], NVIDIA H100 8 [p.6], CUDA 6 [p.4], TPU 1 [p.11]
  • technique: Compiler 42 [p.1], NTT 20 [p.3], Bootstrapping 19 [p.1], RNS 18 [p.2], Softmax 8 [p.1], Matrix multiplication 5 [p.3], GELU 4 [p.1], Convolution 4 [p.2], Key switching 3 [p.8], Rescaling 3 [p.4], ReLU 3 [p.2], Rotations 3 [p.2]
  • topic: Homomorphic encryption 123 [p.1], Large language models 15 [p.1], Benchmark 14 [p.9], Private / secure inference 6 [p.1], Open source 2 [p.2]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/0-004_a-scalable-multi-gpu-framework-for-encrypted-large-model-inf.pdf catalogue 13 653.0 KB 2979e81a8ab4
references/sok-references/018_a-scalable-multi-gpu-framework-for-encrypted-large-model-inf.pdf sok-references 13 653.0 KB 2979e81a8ab4 byte-identical to references/catalogue/0-004_a-scalable-multi-gpu-framework-for-encrypted-large-model-inf.pdf

Database key: a-scalable-multi-gpu-framework-for-encrypted-large-model-inference · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set a-scalable-multi-gpu-framework-for-encrypted-large-model-inference COLUMN VALUE