Skip to content

Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study

Authors: Jianwei Zhu; Hang Yin; Peng Deng; Aline Almeida; Shunfan Zhou
Year: 2024
Venue: arXiv 2024

type: system collection: catalogue status: auto

Links: arXiv:2409.03992

Summary

Auto-extracted summary (unreviewed)

This report benchmarks the overhead of running LLM inference in TEE (confidential computing) mode on NVIDIA H100 and H200 GPUs paired with AMD SEV-SNP and Intel TDX CPUs, using vLLM with Llama-3.1-8B, Phi-3-14B and 4-bit Llama-3.1-70B on ShareGPT prompts. GPU-internal compute is essentially unaffected; the overhead comes from encrypted CPU-GPU PCIe transfers, stays below about 7-9% in throughput for typical queries, and approaches zero for larger models and longer sequences, although time-to-first-token can rise by 18-26% for small models.

Abstract

This report evaluates the performance impact of enabling Trusted Execution Environments (TEE) on NVIDIA Hopper GPUs for large language model (LLM) inference tasks. We benchmark the overhead introduced by TEE mode across various LLMs and token lengths, with a particular focus on the bottleneck caused by CPU-GPU data transfers via PCIe. Our results indicate that while there is minimal computational overhead within the GPU, the overall performance penalty is primarily attributable to data transfer. For the majority of typical LLM queries, the overhead remains below 7%, with larger models and longer sequences experiencing nearly zero overhead.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution measurement of TEE-on vs TEE-off throughput and latency for LLM inference on Hopper GPUs llm 0.90
contribution identification of CPU-GPU data transfer (PCIe bounce buffers) as the dominant overhead source llm 0.90
contribution characterization of how overhead shrinks with model size and sequence length llm 0.90
contribution comparison of H100 and H200 behavior under TEE mode llm 0.90
schemes TEE llm 0.90
libraries vLLM llm 0.90
libraries CUDA llm 0.90
libraries bitsandbytes llm 0.90
models Llama-3.1-8B-Instruct llm 0.90
models Phi-3-14B-128k-Instruct llm 0.90
models Llama-3.1-70B-Instruct (4-bit quantized) llm 0.90
datasets ShareGPT llm 0.90
tasks LLM inference llm 0.90
tasks confidential computing llm 0.90
hardware NVIDIA H100 NVL GPU llm 0.90
hardware NVIDIA H200 NVL GPU llm 0.90
hardware AMD EPYC 9V84 CPU (SEV-SNP) llm 0.90
hardware Intel Xeon Platinum 8558 CPU (TDX) llm 0.90
threat model hardware TEE isolating GPU computation from the OS and host owner, with encrypted CPU-GPU transfers and remote attestation llm 0.90
interaction TEE llm 0.90
code url https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered (URL found in the full text) regex 0.50
limitations some measured overheads are negative, which the authors attribute to precision loss in measurement llm 0.90
llm relevance high llm 0.90
key result throughput overhead (TPS): 6.85% (H100), 8.84% (H200) (Llama-3.1-8B, batch size 1) llm 0.90
key result throughput overhead (TPS): -0.13% (H100), 2.29% (H200) (Llama-3.1-70B 4-bit) llm 0.90
key result average overhead: < 9% (below 7% for typical queries) (all models and sequence lengths) llm 0.90
key result time-to-first-token overhead: 19.03% (H100), 20.95% (H200) (Llama-3.1-8B) llm 0.90
key result QPS overhead: 3.22% (H100), 7.55% (H200) (Llama-3.1-8B, dynamic batching) llm 0.90

System card: Confidential computing on H100: benchmark

Group 6. Trusted hardware, statistical and integrity-only alternatives
Family GPU TEE evaluation
Arithmetic Native floating point
Interaction Trusted hardware
Model scale Llama-3-70B
Status Active
Relation to the SoK Quantifies TEE overhead for LLM inference.
Note Reports under 7% overhead for most LLM queries, versus about 10^4x for CKKS.
Systems page Confidential computing on H100: benchmark

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • scheme: TEE / confidential computing 62 [p.1]
  • model: LLaMA 25 [p.3]
  • hardware: GPU 30 [p.1], NVIDIA H100 18 [p.1], AMD EPYC 1 [p.3], CUDA 1 [p.3]
  • task: Language modeling / generation 5 [p.2]
  • topic: Large language models 11 [p.1], Benchmark 3 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/catalogue/6-082_confidential-computing-on-nvidia-hopper-gpus-a-performance-b.pdf catalogue 7 2.1 MB 3326e83a2837

Database key: confidential-computing-on-nvidia-hopper-gpus-a-performance-benchmark · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set confidential-computing-on-nvidia-hopper-gpus-a-performance-benchmark COLUMN VALUE