Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study¶
Authors: Jianwei Zhu; Hang Yin; Peng Deng; Aline Almeida; Shunfan Zhou
Year: 2024
Venue: arXiv 2024
type: system collection: catalogue status: auto
Links: arXiv:2409.03992
Summary¶
Auto-extracted summary (unreviewed)
This report benchmarks the overhead of running LLM inference in TEE (confidential computing) mode on NVIDIA H100 and H200 GPUs paired with AMD SEV-SNP and Intel TDX CPUs, using vLLM with Llama-3.1-8B, Phi-3-14B and 4-bit Llama-3.1-70B on ShareGPT prompts. GPU-internal compute is essentially unaffected; the overhead comes from encrypted CPU-GPU PCIe transfers, stays below about 7-9% in throughput for typical queries, and approaches zero for larger models and longer sequences, although time-to-first-token can rise by 18-26% for small models.
Abstract¶
This report evaluates the performance impact of enabling Trusted Execution Environments (TEE) on NVIDIA Hopper GPUs for large language model (LLM) inference tasks. We benchmark the overhead introduced by TEE mode across various LLMs and token lengths, with a particular focus on the bottleneck caused by CPU-GPU data transfers via PCIe. Our results indicate that while there is minimal computational overhead within the GPU, the overall performance penalty is primarily attributable to data transfer. For the majority of typical LLM queries, the overhead remains below 7%, with larger models and longer sequences experiencing nearly zero overhead.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | measurement of TEE-on vs TEE-off throughput and latency for LLM inference on Hopper GPUs | llm | 0.90 |
| contribution | identification of CPU-GPU data transfer (PCIe bounce buffers) as the dominant overhead source | llm | 0.90 |
| contribution | characterization of how overhead shrinks with model size and sequence length | llm | 0.90 |
| contribution | comparison of H100 and H200 behavior under TEE mode | llm | 0.90 |
| schemes | TEE | llm | 0.90 |
| libraries | vLLM | llm | 0.90 |
| libraries | CUDA | llm | 0.90 |
| libraries | bitsandbytes | llm | 0.90 |
| models | Llama-3.1-8B-Instruct | llm | 0.90 |
| models | Phi-3-14B-128k-Instruct | llm | 0.90 |
| models | Llama-3.1-70B-Instruct (4-bit quantized) | llm | 0.90 |
| datasets | ShareGPT | llm | 0.90 |
| tasks | LLM inference | llm | 0.90 |
| tasks | confidential computing | llm | 0.90 |
| hardware | NVIDIA H100 NVL GPU | llm | 0.90 |
| hardware | NVIDIA H200 NVL GPU | llm | 0.90 |
| hardware | AMD EPYC 9V84 CPU (SEV-SNP) | llm | 0.90 |
| hardware | Intel Xeon Platinum 8558 CPU (TDX) | llm | 0.90 |
| threat model | hardware TEE isolating GPU computation from the OS and host owner, with encrypted CPU-GPU transfers and remote attestation | llm | 0.90 |
| interaction | TEE | llm | 0.90 |
| code url | https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered (URL found in the full text) | regex | 0.50 |
| limitations | some measured overheads are negative, which the authors attribute to precision loss in measurement | llm | 0.90 |
| llm relevance | high | llm | 0.90 |
| key result | throughput overhead (TPS): 6.85% (H100), 8.84% (H200) (Llama-3.1-8B, batch size 1) | llm | 0.90 |
| key result | throughput overhead (TPS): -0.13% (H100), 2.29% (H200) (Llama-3.1-70B 4-bit) | llm | 0.90 |
| key result | average overhead: < 9% (below 7% for typical queries) (all models and sequence lengths) | llm | 0.90 |
| key result | time-to-first-token overhead: 19.03% (H100), 20.95% (H200) (Llama-3.1-8B) | llm | 0.90 |
| key result | QPS overhead: 3.22% (H100), 7.55% (H200) (Llama-3.1-8B, dynamic batching) | llm | 0.90 |
System card: Confidential computing on H100: benchmark¶
| Group | 6. Trusted hardware, statistical and integrity-only alternatives |
| Family | GPU TEE evaluation |
| Arithmetic | Native floating point |
| Interaction | Trusted hardware |
| Model scale | Llama-3-70B |
| Status | Active |
| Relation to the SoK | Quantifies TEE overhead for LLM inference. |
| Note | Reports under 7% overhead for most LLM queries, versus about 10^4x for CKKS. |
| Systems page | Confidential computing on H100: benchmark |
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- scheme: TEE / confidential computing 62 [p.1]
- model: LLaMA 25 [p.3]
- hardware: GPU 30 [p.1], NVIDIA H100 18 [p.1], AMD EPYC 1 [p.3], CUDA 1 [p.3]
- task: Language modeling / generation 5 [p.2]
- topic: Large language models 11 [p.1], Benchmark 3 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/catalogue/6-082_confidential-computing-on-nvidia-hopper-gpus-a-performance-b.pdf |
catalogue | 7 | 2.1 MB | 3326e83a2837 |
Database key: confidential-computing-on-nvidia-hopper-gpus-a-performance-benchmark · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set confidential-computing-on-nvidia-hopper-gpus-a-performance-benchmark COLUMN VALUE