Skip to content

Terrazzo: Memory-Aware GPU Framework for Private Transformer Inference

Authors: Shokri, Tsoutsos
Year: 2026
Venue: IACR ePrint 2026/1870

type: system collection: catalogue, schemes status: auto

Links: ePrint 2026/1870

Summary

Auto-extracted summary (unreviewed)

Terrazzo is a GPU framework for the GL homomorphic encryption scheme, which supports native matrix multiplication but whose large ciphertexts and keys previously made GPU deployment impractical; it bootstraps into a compact Y-decoded representation, tiles work over independently schedulable ciphertext slices using an L2/occupancy model, confines grafting-based modulus management to application levels around a sprout-free bootstrap, and keeps model weights at cleartext size with an expansion-free plaintext pipeline. For unmodified BERT-base (128 tokens) it achieves 7.04 s amortized per input on a 32 GB RTX 5090 at 256-input occupancy and 29.98 s on an A100, a 2.20-20.09x speed-up over prior single-A100 systems, with per-slice bootstrap latency of 11.39 ms on the RTX 5090.

Abstract

Fully homomorphic encryption (FHE) allows a server to run inference directly on encrypted data, making it a promising foundation for private transformer inference. Its dominant scheme, CKKS, has no native matrix multiplication, so the encrypted matrix multiplications at the heart of transformers dominate inference cost. The recent GL scheme supports matrix multiplication natively, but its ciphertexts, plaintexts, and evaluation keys are so large that a direct GPU implementation would require terabytes of VRAM for even small language models. We present Terrazzo, a GPU framework for the GL scheme, co-designed across cryptography, algorithms, and kernels to fit private inference on a commodity GPU. Terrazzo bootstraps into a compact Y-decoded representation, so nonlinear layers never hold full ciphertexts live; tiles all work over GL’s independently schedulable ciphertext slices, sizing each tile per level against an L2 and occupancy model; confines grafting-based modulus management to application levels around a sprout-free bootstrap; and keeps every model weight at its cleartext footprint with an expansion-free plaintext pipeline. On BERT-base at full 256-input occupancy, Terrazzo achieves a 7.04 s amortized time per input on a consumer 32 GB RTX 5090. Its A100 time is 29.98 s amortized, a 2.20–20.09× speedup over prior single-A100 systems. Per-slice bootstrap speedup is 1.12–2.03× on the RTX 5090 and 1.08–1.36× on the A100.
Source: pdf

Extracted details

Field Value Source Confidence
contribution first GPU framework for the GL scheme with native homomorphic matrix multiplication llm 0.90
contribution bootstrapping into a compact Y-decoded representation so nonlinear layers never hold full ciphertexts llm 0.90
contribution per-level tiling of all kernels over GL ciphertext slices sized by an L2 and occupancy model llm 0.90
contribution grafting-based modulus management confined to application levels around a sprout-free bootstrap llm 0.90
contribution expansion-free plaintext pipeline keeping weights at cleartext footprint llm 0.90
contribution end-to-end encrypted BERT-base inference on a consumer GPU llm 0.90
schemes GL scheme llm 0.90
schemes CKKS llm 0.90
libraries CUDA llm 0.90
models BERT-base llm 0.90
tasks private transformer inference llm 0.90
tasks homomorphic bootstrapping on GPU llm 0.90
hardware NVIDIA GeForce RTX 5090 (32 GB) llm 0.90
hardware NVIDIA A100 PCIe (80 GB) llm 0.90
hardware AMD Ryzen Threadripper 7970X (32 cores), 157 GB RAM llm 0.90
security level 128-bit llm 0.90
threat model client-server with a single untrusted server; non-interactive, only the client can decrypt llm 0.90
interaction non-interactive FHE llm 0.90
limitations evaluated only on encoder-only BERT-base; decoder-only models and multi-GPU scaling left for future work llm 0.90
limitations reported latency is amortized over a 256-input batch while baselines report single-input latency llm 0.90
llm relevance high llm 0.90
key result BERT-base amortized latency per input: 7.04 s (RTX 5090, 128 tokens, 256-input batch) llm 0.90
key result BERT-base amortized latency per input: 29.98 s (A100 80GB, 128 tokens, 256-input batch) llm 0.90
key result speed-up over prior single-A100 systems: 2.20x-20.09x (vs Cerium (66.0 s) to THOR (602.4 s), single-input baselines) llm 0.90
key result bootstrap latency per ring-2^16 slice: 11.39 ms (RTX 5090), 29.41 ms (A100) (1.12-2.03x over Cheddar/Theodosian on RTX 5090, 1.08-1.36x on A100) llm 0.90
key result CCMM + relin + rescale: 0.975 ms per 128x128 matrix (RTX 5090, 21 Q limbs) llm 0.90
key result PCMM + rescale: 0.271 ms per 128x128 matrix (RTX 5090, 21 Q limbs) llm 0.90

System card: Terrazzo (GL scheme on GPU)

Group 1. Newer CKKS / FHE LLM frameworks (2026, not in the SoK)
Family Gentry-Lee matrix FHE
Arithmetic Approximate real matrices
Interaction Non-interactive FHE
Model scale Small language models
Status Active SOTA (emerging)
Relation to the SoK First private Transformer inference on the GL matrix scheme, the SoK's flagged successor to CKKS.
Note Co-designs cryptography, algorithms and kernels so GL keys and ciphertexts fit on a commodity GPU.
Systems page Terrazzo (GL scheme on GPU)

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: CKKS 43 [p.1], Matrix-native FHE 11 [p.1], MPC 11 [p.1], Function secret sharing 2 [p.18]
  • library: Cheddar 13 [p.1], Phantom 1 [p.21]
  • model: BERT 24 [p.1], CNN 2 [p.16], Transformer 1 [p.1], LLaMA 1 [p.15]
  • dataset: SST-2 1 [p.15]
  • hardware: GPU 52 [p.1], NVIDIA A100 40 [p.1], NVIDIA RTX 23 [p.1], CUDA 5 [p.8], Multi-GPU 4 [p.17], ASIC 1 [p.16]
  • technique: Bootstrapping 35 [p.1], Rotations 28 [p.1], RNS 23 [p.2], NTT 20 [p.3], Rescaling 17 [p.3], Softmax 17 [p.5], LayerNorm 15 [p.7], Matrix multiplication 14 [p.1], GELU 9 [p.7], Key switching 5 [p.3], Polynomial approximation 4 [p.5], Non-interactive 4 [p.1], Relinearization 3 [p.4], Convolution 3 [p.8], Chebyshev 2 [p.5], Two-party computation 2 [p.20], Malicious model 2 [p.19]
  • assumption: RLWE 12 [p.2]
  • task: Language modeling / generation 3 [p.3], Medical / healthcare 3 [p.1], Finance 2 [p.1]
  • topic: Homomorphic encryption 39 [p.1], Private / secure inference 9 [p.1], Large language models 5 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/1-123_terrazzo-memory-aware-gpu-framework-for-private-transformer.pdf catalogue 22 712.1 KB 455ef00c2f51
references/schemes/s26-4_eprint-2026-1870.pdf schemes 22 712.1 KB 455ef00c2f51 byte-identical to references/catalogue/1-123_terrazzo-memory-aware-gpu-framework-for-private-transformer.pdf

Database key: terrazzo-memory-aware-gpu-framework-for-private-transformer-inference · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set terrazzo-memory-aware-gpu-framework-for-private-transformer-inference COLUMN VALUE