Terrazzo: Memory-Aware GPU Framework for Private Transformer Inference¶
Authors: Shokri, Tsoutsos
Year: 2026
Venue: IACR ePrint 2026/1870
type: system collection: catalogue, schemes status: auto
Links: ePrint 2026/1870
Summary¶
Auto-extracted summary (unreviewed)
Terrazzo is a GPU framework for the GL homomorphic encryption scheme, which supports native matrix multiplication but whose large ciphertexts and keys previously made GPU deployment impractical; it bootstraps into a compact Y-decoded representation, tiles work over independently schedulable ciphertext slices using an L2/occupancy model, confines grafting-based modulus management to application levels around a sprout-free bootstrap, and keeps model weights at cleartext size with an expansion-free plaintext pipeline. For unmodified BERT-base (128 tokens) it achieves 7.04 s amortized per input on a 32 GB RTX 5090 at 256-input occupancy and 29.98 s on an A100, a 2.20-20.09x speed-up over prior single-A100 systems, with per-slice bootstrap latency of 11.39 ms on the RTX 5090.
Abstract¶
Fully homomorphic encryption (FHE) allows a server to run inference directly on encrypted data, making it a promising foundation for private transformer inference. Its dominant scheme, CKKS, has no native matrix multiplication, so the encrypted matrix multiplications at the heart of transformers dominate inference cost. The recent GL scheme supports matrix multiplication natively, but its ciphertexts, plaintexts, and evaluation keys are so large that a direct GPU implementation would require terabytes of VRAM for even small language models. We present Terrazzo, a GPU framework for the GL scheme, co-designed across cryptography, algorithms, and kernels to fit private inference on a commodity GPU. Terrazzo bootstraps into a compact Y-decoded representation, so nonlinear layers never hold full ciphertexts live; tiles all work over GL’s independently schedulable ciphertext slices, sizing each tile per level against an L2 and occupancy model; confines grafting-based modulus management to application levels around a sprout-free bootstrap; and keeps every model weight at its cleartext footprint with an expansion-free plaintext pipeline. On BERT-base at full 256-input occupancy, Terrazzo achieves a 7.04 s amortized time per input on a consumer 32 GB RTX 5090. Its A100 time is 29.98 s amortized, a 2.20–20.09× speedup over prior single-A100 systems. Per-slice bootstrap speedup is 1.12–2.03× on the RTX 5090 and 1.08–1.36× on the A100.
Source: pdf
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | first GPU framework for the GL scheme with native homomorphic matrix multiplication | llm | 0.90 |
| contribution | bootstrapping into a compact Y-decoded representation so nonlinear layers never hold full ciphertexts | llm | 0.90 |
| contribution | per-level tiling of all kernels over GL ciphertext slices sized by an L2 and occupancy model | llm | 0.90 |
| contribution | grafting-based modulus management confined to application levels around a sprout-free bootstrap | llm | 0.90 |
| contribution | expansion-free plaintext pipeline keeping weights at cleartext footprint | llm | 0.90 |
| contribution | end-to-end encrypted BERT-base inference on a consumer GPU | llm | 0.90 |
| schemes | GL scheme | llm | 0.90 |
| schemes | CKKS | llm | 0.90 |
| libraries | CUDA | llm | 0.90 |
| models | BERT-base | llm | 0.90 |
| tasks | private transformer inference | llm | 0.90 |
| tasks | homomorphic bootstrapping on GPU | llm | 0.90 |
| hardware | NVIDIA GeForce RTX 5090 (32 GB) | llm | 0.90 |
| hardware | NVIDIA A100 PCIe (80 GB) | llm | 0.90 |
| hardware | AMD Ryzen Threadripper 7970X (32 cores), 157 GB RAM | llm | 0.90 |
| security level | 128-bit | llm | 0.90 |
| threat model | client-server with a single untrusted server; non-interactive, only the client can decrypt | llm | 0.90 |
| interaction | non-interactive FHE | llm | 0.90 |
| limitations | evaluated only on encoder-only BERT-base; decoder-only models and multi-GPU scaling left for future work | llm | 0.90 |
| limitations | reported latency is amortized over a 256-input batch while baselines report single-input latency | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | BERT-base amortized latency per input: 7.04 s (RTX 5090, 128 tokens, 256-input batch) | llm | 0.90 |
| key result | BERT-base amortized latency per input: 29.98 s (A100 80GB, 128 tokens, 256-input batch) | llm | 0.90 |
| key result | speed-up over prior single-A100 systems: 2.20x-20.09x (vs Cerium (66.0 s) to THOR (602.4 s), single-input baselines) | llm | 0.90 |
| key result | bootstrap latency per ring-2^16 slice: 11.39 ms (RTX 5090), 29.41 ms (A100) (1.12-2.03x over Cheddar/Theodosian on RTX 5090, 1.08-1.36x on A100) | llm | 0.90 |
| key result | CCMM + relin + rescale: 0.975 ms per 128x128 matrix (RTX 5090, 21 Q limbs) | llm | 0.90 |
| key result | PCMM + rescale: 0.271 ms per 128x128 matrix (RTX 5090, 21 Q limbs) | llm | 0.90 |
System card: Terrazzo (GL scheme on GPU)¶
| Group | 1. Newer CKKS / FHE LLM frameworks (2026, not in the SoK) |
| Family | Gentry-Lee matrix FHE |
| Arithmetic | Approximate real matrices |
| Interaction | Non-interactive FHE |
| Model scale | Small language models |
| Status | Active SOTA (emerging) |
| Relation to the SoK | First private Transformer inference on the GL matrix scheme, the SoK's flagged successor to CKKS. |
| Note | Co-designs cryptography, algorithms and kernels so GL keys and ciphertexts fit on a commodity GPU. |
| Systems page | Terrazzo (GL scheme on GPU) |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: CKKS 43 [p.1], Matrix-native FHE 11 [p.1], MPC 11 [p.1], Function secret sharing 2 [p.18]
- library: Cheddar 13 [p.1], Phantom 1 [p.21]
- model: BERT 24 [p.1], CNN 2 [p.16], Transformer 1 [p.1], LLaMA 1 [p.15]
- dataset: SST-2 1 [p.15]
- hardware: GPU 52 [p.1], NVIDIA A100 40 [p.1], NVIDIA RTX 23 [p.1], CUDA 5 [p.8], Multi-GPU 4 [p.17], ASIC 1 [p.16]
- technique: Bootstrapping 35 [p.1], Rotations 28 [p.1], RNS 23 [p.2], NTT 20 [p.3], Rescaling 17 [p.3], Softmax 17 [p.5], LayerNorm 15 [p.7], Matrix multiplication 14 [p.1], GELU 9 [p.7], Key switching 5 [p.3], Polynomial approximation 4 [p.5], Non-interactive 4 [p.1], Relinearization 3 [p.4], Convolution 3 [p.8], Chebyshev 2 [p.5], Two-party computation 2 [p.20], Malicious model 2 [p.19]
- assumption: RLWE 12 [p.2]
- task: Language modeling / generation 3 [p.3], Medical / healthcare 3 [p.1], Finance 2 [p.1]
- topic: Homomorphic encryption 39 [p.1], Private / secure inference 9 [p.1], Large language models 5 [p.1]
Related papers¶
- this paper follow up of Fully Homomorphic Encryption for Matrix Arithmetic — follow-up listed for scheme "Gentry–Lee (GL) matrix FHE"
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/1-123_terrazzo-memory-aware-gpu-framework-for-private-transformer.pdf |
catalogue | 22 | 712.1 KB | 455ef00c2f51 |
|
references/schemes/s26-4_eprint-2026-1870.pdf |
schemes | 22 | 712.1 KB | 455ef00c2f51 |
byte-identical to references/catalogue/1-123_terrazzo-memory-aware-gpu-framework-for-private-transformer.pdf |
Database key: terrazzo-memory-aware-gpu-framework-for-private-transformer-inference · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set terrazzo-memory-aware-gpu-framework-for-private-transformer-inference COLUMN VALUE