Skip to content

Cost-efficient llm training with lifetimeaware tensor offloading via gpudirect storage

Authors: Ziqi Yuan; Haoyang Zhang; Yirui Eric Zhou; Apoorve Mohan; I-Hsin Chung; Seetharami Seelam; Jian Huang
Year: 2025
Venue: arXiv preprint

type: system collection: sok-references status: auto

Links: arXiv:2506.06472

Summary

Auto-extracted summary (unreviewed)

Teraio is a tensor-offloading framework that expands GPU memory for multi-GPU LLM training with low-cost PCIe SSDs. It profiles the first few training iterations to estimate each tensor's lifetime, generates a fine-grained offload/prefetch plan compiled into the PyTorch program, and executes it with GPUDirect Storage to bypass the CPU. On Llama3-8B/70B and Granite-code-base-8B it trains 1.47x faster on average than ZeRO-Offload and ZeRO-Infinity and reaches 80.7% of the throughput of an ideal unlimited-memory GPU.

Abstract

We present the design and implementation of a new lifetime-aware tensor offloading framework for GPU memory expansion using low-cost PCIe-based solid-state drives (SSDs). Our framework, TERAIO, is developed explicitly for large language model (LLM) training with multiple GPUs and multiple SSDs. Its design is driven by our observation that the active tensors take only a small fraction (1.7% on average) of allocated GPU memory in each LLM training iteration, the inactive tensors are usually large and will not be used for a long period of time, creating ample opportunities for offloading/prefetching tensors to/from slow SSDs without stalling the GPU training process. TERAIO accurately estimates the lifetime (active period of time in GPU memory) of each tensor with the profiling of the first few iterations in the training process. With the tensor lifetime analysis, TERAIO will generate an optimized tensor offloading/prefetching plan and integrate it into the compiled LLM program via PyTorch. TERAIO has a runtime tensor migration engine to execute the offloading/prefetching plan via GPUDirect storage, which allows direct tensor migration between GPUs and SSDs for alleviating the CPU bottleneck and maximizing the SSD bandwidth utilization. In comparison with state-of-the-art studies such as ZeRO-Offload and ZeRO-Infinity, we show that TERAIO improves the training performance of various LLMs by 1.47x on average, and achieves 80.7% of the ideal performance assuming unlimited GPU memory.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution lifetime-aware tensor offloading framework for LLM training with multiple GPUs and SSDs llm 0.90
contribution tensor lifetime profiler based on the first few training iterations llm 0.90
contribution optimized offloading/prefetching plan integrated into the compiled PyTorch program llm 0.90
contribution runtime tensor migration engine using GPUDirect Storage for direct GPU-SSD transfers llm 0.90
contribution observation that active tensors occupy only 1.7% of allocated GPU memory per iteration llm 0.90
libraries PyTorch llm 0.90
libraries TorchTitan llm 0.90
models LLaMA-3-8B llm 0.90
models LLaMA-3-70B llm 0.90
models Granite-code-base-8B llm 0.90
datasets C4 llm 0.90
tasks LLM training llm 0.90
tasks GPU memory expansion / tensor offloading llm 0.90
hardware 2x NVIDIA H100 NVL (94GB HBM each) llm 0.90
hardware 2x AMD EPYC 9334 CPU llm 0.90
hardware 1.5TB DDR5 llm 0.90
hardware 8x Samsung 990 PRO 2TB PCIe Gen5 SSDs (two RAID-0 arrays) llm 0.90
code url https://huggingface.co/google-t5/t5-11b (URL found in the full text) regex 0.50
code url https://huggingface.co/ibm-granite (URL found in the full text) regex 0.50
limitations experiments limited to 8 PCIe SSDs due to available slots llm 0.90
limitations evaluation uses full-precision training only to keep comparison with ZeRO fair llm 0.90
key result training throughput speed-up: 1.47x on average (vs ZeRO-Offload and ZeRO-Infinity, Llama3-8B/70B and Granite-code-base-8B) llm 0.90
key result fraction of ideal throughput: 80.7% (vs ideal system with unlimited GPU memory) llm 0.90
key result training cost reduction: up to 5.41x (vs training using only GPU memory) llm 0.90
key result cost-efficiency improvement: 1.45x on average (vs existing offloading frameworks) llm 0.90
key result active tensor share of allocated GPU memory: 1.7% on average (per LLM training iteration) llm 0.90

SoK reference [94]

Z. Yuan, H. Zhang, Y. E. Zhou, A. Mohan, I. Chung, S. Seelam, J. Huang et al., “Cost-efficient llm training with lifetimeaware tensor offloading via gpudirect storage,” arXiv preprint arXiv:2506.06472, 2025.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • library: PyTorch 17 [p.1]
  • model: LLaMA 7 [p.3], GPT 6 [p.3], BERT 1 [p.4], Diffusion model 1 [p.11]
  • dataset: Iris 1 [p.11]
  • hardware: GPU 137 [p.1], NVIDIA H100 10 [p.3], Multi-GPU 3 [p.3], AMD EPYC 1 [p.7], CUDA 1 [p.5]
  • topic: Large language models 54 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/094_cost-efficient-llm-training-with-lifetimeaware-tensor-offloa.pdf sok-references 12 1.7 MB c2f083235e14

Database key: cost-efficient-llm-training-with-lifetimeaware-tensor-offloading-via · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set cost-efficient-llm-training-with-lifetimeaware-tensor-offloading-via COLUMN VALUE