Cost-efficient llm training with lifetimeaware tensor offloading via gpudirect storage¶
Authors: Ziqi Yuan; Haoyang Zhang; Yirui Eric Zhou; Apoorve Mohan; I-Hsin Chung; Seetharami Seelam; Jian Huang
Year: 2025
Venue: arXiv preprint
type: system collection: sok-references status: auto
Links: arXiv:2506.06472
Summary¶
Auto-extracted summary (unreviewed)
Teraio is a tensor-offloading framework that expands GPU memory for multi-GPU LLM training with low-cost PCIe SSDs. It profiles the first few training iterations to estimate each tensor's lifetime, generates a fine-grained offload/prefetch plan compiled into the PyTorch program, and executes it with GPUDirect Storage to bypass the CPU. On Llama3-8B/70B and Granite-code-base-8B it trains 1.47x faster on average than ZeRO-Offload and ZeRO-Infinity and reaches 80.7% of the throughput of an ideal unlimited-memory GPU.
Abstract¶
We present the design and implementation of a new lifetime-aware tensor offloading framework for GPU memory expansion using low-cost PCIe-based solid-state drives (SSDs). Our framework, TERAIO, is developed explicitly for large language model (LLM) training with multiple GPUs and multiple SSDs. Its design is driven by our observation that the active tensors take only a small fraction (1.7% on average) of allocated GPU memory in each LLM training iteration, the inactive tensors are usually large and will not be used for a long period of time, creating ample opportunities for offloading/prefetching tensors to/from slow SSDs without stalling the GPU training process. TERAIO accurately estimates the lifetime (active period of time in GPU memory) of each tensor with the profiling of the first few iterations in the training process. With the tensor lifetime analysis, TERAIO will generate an optimized tensor offloading/prefetching plan and integrate it into the compiled LLM program via PyTorch. TERAIO has a runtime tensor migration engine to execute the offloading/prefetching plan via GPUDirect storage, which allows direct tensor migration between GPUs and SSDs for alleviating the CPU bottleneck and maximizing the SSD bandwidth utilization. In comparison with state-of-the-art studies such as ZeRO-Offload and ZeRO-Infinity, we show that TERAIO improves the training performance of various LLMs by 1.47x on average, and achieves 80.7% of the ideal performance assuming unlimited GPU memory.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | lifetime-aware tensor offloading framework for LLM training with multiple GPUs and SSDs | llm | 0.90 |
| contribution | tensor lifetime profiler based on the first few training iterations | llm | 0.90 |
| contribution | optimized offloading/prefetching plan integrated into the compiled PyTorch program | llm | 0.90 |
| contribution | runtime tensor migration engine using GPUDirect Storage for direct GPU-SSD transfers | llm | 0.90 |
| contribution | observation that active tensors occupy only 1.7% of allocated GPU memory per iteration | llm | 0.90 |
| libraries | PyTorch | llm | 0.90 |
| libraries | TorchTitan | llm | 0.90 |
| models | LLaMA-3-8B | llm | 0.90 |
| models | LLaMA-3-70B | llm | 0.90 |
| models | Granite-code-base-8B | llm | 0.90 |
| datasets | C4 | llm | 0.90 |
| tasks | LLM training | llm | 0.90 |
| tasks | GPU memory expansion / tensor offloading | llm | 0.90 |
| hardware | 2x NVIDIA H100 NVL (94GB HBM each) | llm | 0.90 |
| hardware | 2x AMD EPYC 9334 CPU | llm | 0.90 |
| hardware | 1.5TB DDR5 | llm | 0.90 |
| hardware | 8x Samsung 990 PRO 2TB PCIe Gen5 SSDs (two RAID-0 arrays) | llm | 0.90 |
| code url | https://huggingface.co/google-t5/t5-11b (URL found in the full text) | regex | 0.50 |
| code url | https://huggingface.co/ibm-granite (URL found in the full text) | regex | 0.50 |
| limitations | experiments limited to 8 PCIe SSDs due to available slots | llm | 0.90 |
| limitations | evaluation uses full-precision training only to keep comparison with ZeRO fair | llm | 0.90 |
| key result | training throughput speed-up: 1.47x on average (vs ZeRO-Offload and ZeRO-Infinity, Llama3-8B/70B and Granite-code-base-8B) | llm | 0.90 |
| key result | fraction of ideal throughput: 80.7% (vs ideal system with unlimited GPU memory) | llm | 0.90 |
| key result | training cost reduction: up to 5.41x (vs training using only GPU memory) | llm | 0.90 |
| key result | cost-efficiency improvement: 1.45x on average (vs existing offloading frameworks) | llm | 0.90 |
| key result | active tensor share of allocated GPU memory: 1.7% on average (per LLM training iteration) | llm | 0.90 |
SoK reference [94]¶
Z. Yuan, H. Zhang, Y. E. Zhou, A. Mohan, I. Chung, S. Seelam, J. Huang et al., “Cost-efficient llm training with lifetimeaware tensor offloading via gpudirect storage,” arXiv preprint arXiv:2506.06472, 2025.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- library: PyTorch 17 [p.1]
- model: LLaMA 7 [p.3], GPT 6 [p.3], BERT 1 [p.4], Diffusion model 1 [p.11]
- dataset: Iris 1 [p.11]
- hardware: GPU 137 [p.1], NVIDIA H100 10 [p.3], Multi-GPU 3 [p.3], AMD EPYC 1 [p.7], CUDA 1 [p.5]
- topic: Large language models 54 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/094_cost-efficient-llm-training-with-lifetimeaware-tensor-offloa.pdf |
sok-references | 12 | 1.7 MB | c2f083235e14 |
Database key: cost-efficient-llm-training-with-lifetimeaware-tensor-offloading-via · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set cost-efficient-llm-training-with-lifetimeaware-tensor-offloading-via COLUMN VALUE