A scalable multi-GPU framework for encrypted large-model inference¶
Authors: Siddharth Jayashankar; Joshua Kim; Michael B. Sullivan; Wenting Zheng; Dimitrios Skarlatos
Year: 2025
Venue: arXiv 2512.11269
type: system collection: catalogue, sok-references status: auto
Links: DOI · arXiv:2512.11269
Summary¶
Auto-extracted summary (unreviewed)
Cerium is a multi-GPU framework for CKKS-based encrypted inference on large models that pairs a domain-specific language with an optimizing compiler and a runtime: it generates fused GPU kernels automatically, compresses plaintext weights with sparse polynomial representations, manages terabyte-scale memory footprints between host and GPUs, and parallelises computation across devices with communication-aware partitioning. On NVIDIA GPUs it beats expert-written GPU libraries by up to 2.25x on small models, matches the CraterLake FHE ASIC, performs CKKS bootstrapping in 7.5 ms, and is the first system to run end-to-end encrypted BERT-Base inference (8 s) and Llama3-8B prefill (43 s in the arXiv abstract, 134 s in the paper body).
Abstract¶
Encrypted AI using fully homomorphic encryption (FHE) enables inference directly over encrypted queries, providing strong privacy guarantees. However, its computational and memory overheads have limited practical deployment. Custom FHE accelerators improve performance, but rely on advanced manufacturing technologies that limit their accessibility. GPUs offer a more widely available alternative, yet achieving ASIC-class performance on GPUs is challenging. Large models such as LLMs compound these challenges by requiring optimized kernels, terabyte-scale memory management, and efficient execution across multiple devices. We present Cerium, a multi-GPU framework for large-scale FHE inference. Cerium combines a domain-specific language, an optimizing compiler, and a runtime system to automatically generate optimized GPU kernels, compress plaintext data, manage terabyte-scale memory footprints, and optimize multi-GPU communication. Implemented on NVIDIA GPUs, Cerium outperforms expert-written GPU libraries by up to 2.25x and achieves performance competitive with state-of-the-art FHE ASICs, matching the CraterLake ASIC. It is the first GPU system to execute CKKS bootstrapping in under 10 milliseconds, achieving 7.5 milliseconds, and is the first to demonstrate encrypted inference for BERT-Base and Llama3-8B prefill in 8 seconds and 43 seconds, respectively. The Cerium framework is open-source and available at https://github.com/CMU-CAOS/Cerium
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | multi-GPU FHE inference framework with DSL, optimizing compiler and runtime | llm | 0.85 |
| contribution | new IR constructs and compiler passes that automatically generate fused GPU kernels | llm | 0.85 |
| contribution | sparse polynomial representations and memory-efficient data layouts to compress plaintext data | llm | 0.85 |
| contribution | terabyte-scale memory management across host and GPUs | llm | 0.85 |
| contribution | communication-aware multi-GPU parallelization | llm | 0.85 |
| contribution | first GPU CKKS bootstrapping under 10 ms and first encrypted BERT-Base and Llama3-8B inference | llm | 0.85 |
| schemes | CKKS | llm | 0.85 |
| libraries | Cheddar (baseline GPU library) | llm | 0.85 |
| models | BERT-base | llm | 0.85 |
| models | Llama3-8B | llm | 0.85 |
| models | ResNet-20 | llm | 0.85 |
| tasks | encrypted LLM inference (prefill) | llm | 0.85 |
| tasks | encrypted CNN inference | llm | 0.85 |
| tasks | CKKS bootstrapping | llm | 0.85 |
| hardware | NVIDIA GPUs (multi-GPU) | llm | 0.85 |
| security level | 128-bit (N = 64K, max ciphertext modulus 1782 bits) | llm | 0.85 |
| threat model | client-server; encrypted queries to a cloud-hosted model | llm | 0.85 |
| interaction | non-interactive FHE | llm | 0.85 |
| code url | https://github.com/Desilo/liberate-fhe (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/CMU-CAOS/Cerium | llm | 0.85 |
| llm relevance | high | llm | 0.85 |
| key result | speed-up vs expert-written GPU libraries: up to 2.25x (small models, NVIDIA GPUs) | llm | 0.85 |
| key result | CKKS bootstrapping latency: 7.5 ms (first GPU system under 10 ms) | llm | 0.85 |
| key result | BERT-Base encrypted inference latency: 8 s (multi-GPU, 128-bit security) | llm | 0.85 |
| key result | Llama3-8B encrypted prefill latency: 43 s (arXiv abstract) / 134 s (paper body) (multi-GPU, 128-bit security) | llm | 0.85 |
| key result | ASIC comparison: matches CraterLake FHE ASIC (performance competitive with state-of-the-art FHE ASICs) | llm | 0.85 |
System card: CERIUM¶
| Group | 0. CKKS LLM frameworks surveyed in the SoK |
| Family | CKKS · Model-preserving (N-PRE) |
| Arithmetic | Approximate real (CKKS) |
| Interaction | non-interactive FHE |
| Model scale | Llama-3-8B, BERT-Base |
| Library | Custom multi-GPU CKKS compiler |
| Hardware | 1-8x A100/H100/B200 GPUs |
| Packing | compressed sparse plaintext encoding |
| Latency | 134 s prefill (128 tokens, 8x B200); 66 s BERT-Base on A100 |
| Open source | planned |
| Status | Active SOTA |
| Relation to the SoK | First end-to-end multi-GPU CKKS compiler with kernel fusion; ~14 KB ciphertext memory per model parameter. |
| Note | Best-known runtimes for model-preserving Llama-3-8B; bootstrapping under 10 ms on multi-GPU cluster. |
| Systems page | CERIUM |
SoK reference [18]¶
S. Jayashankar, J. Kim, M. B. Sullivan, W. Zheng, and D. Skarlatos, “A scalable multi-gpu framework for encrypted large-model inference,” arXiv preprint arXiv:2512.11269, 2025.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 11 [p.2], MPC 3 [p.11]
- library: Cheddar 8 [p.1], HElib 1 [p.12], Phantom 1 [p.13], Liberate.FHE 1 [p.12], Orion 1 [p.12], HEIR 1 [p.12]
- model: BERT 31 [p.1], ResNet 18 [p.1], MobileNet 2 [p.3], CNN 2 [p.1], LoRA 1 [p.9]
- dataset: GLUE 2 [p.9], CIFAR-10 1 [p.9], GLUE tasks (QNLI/RTE/MRPC/CoLA/QQP/MNLI/STS-B) 1 [p.9]
- hardware: GPU 154 [p.1], ASIC 29 [p.1], Multi-GPU 21 [p.1], NVIDIA A100 11 [p.9], NVIDIA H100 8 [p.6], CUDA 6 [p.4], TPU 1 [p.11]
- technique: Compiler 42 [p.1], NTT 20 [p.3], Bootstrapping 19 [p.1], RNS 18 [p.2], Softmax 8 [p.1], Matrix multiplication 5 [p.3], GELU 4 [p.1], Convolution 4 [p.2], Key switching 3 [p.8], Rescaling 3 [p.4], ReLU 3 [p.2], Rotations 3 [p.2]
- topic: Homomorphic encryption 123 [p.1], Large language models 15 [p.1], Benchmark 14 [p.9], Private / secure inference 6 [p.1], Open source 2 [p.2]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/0-004_a-scalable-multi-gpu-framework-for-encrypted-large-model-inf.pdf |
catalogue | 13 | 653.0 KB | 2979e81a8ab4 |
|
references/sok-references/018_a-scalable-multi-gpu-framework-for-encrypted-large-model-inf.pdf |
sok-references | 13 | 653.0 KB | 2979e81a8ab4 |
byte-identical to references/catalogue/0-004_a-scalable-multi-gpu-framework-for-encrypted-large-model-inf.pdf |
Database key: a-scalable-multi-gpu-framework-for-encrypted-large-model-inference · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set a-scalable-multi-gpu-framework-for-encrypted-large-model-inference COLUMN VALUE