Large language model inference acceleration: A comprehensive hardware perspective¶
Authors: Jinhao Li; Jiaming Xu; Shan Huang; Yonghua Chen; Wen Li; Jun Liu; Yaoxiu Lian; Jiayi Pan; Li Ding; Hao Zhou; Yu Wang; Guohao Dai
Year: 2025
Venue: arXiv preprint
type: survey collection: sok-references status: auto
Links: arXiv:2410.04466
Summary¶
Auto-extracted summary (unreviewed)
This survey covers efficient generative LLM inference across hardware platforms (CPU, GPU, FPGA, ASIC and PIM/NDP), first reviewing mainstream generative LLM architectures and the inference process, then summarizing platform-specific optimization methods and reported inference results. It quantitatively compares power, absolute throughput (tokens/s) and energy efficiency (tokens/J) at batch sizes 1 and 8 across platforms and methods, and identifies multimodality, inference-time compute and energy efficiency as the trends shaping edge AI.
Abstract¶
Large Language Models (LLMs) have demonstrated remarkable capabilities across various fields, from natural language understanding to text generation. Compared to non-generative LLMs like BERT and DeBERTa, generative LLMs like GPT series and Llama series are currently the main focus due to their superior algorithmic performance. The advancements in generative LLMs are closely intertwined with the development of hardware capabilities. Various hardware platforms exhibit distinct hardware characteristics, which can help improve LLM inference performance. Therefore, this paper comprehensively surveys efficient generative LLM inference on different hardware platforms. First, we provide an overview of the algorithm architecture of mainstream generative LLMs and delve into the inference process. Then, we summarize different optimization methods for different platforms such as CPU, GPU, FPGA, ASIC, and PIM/NDP, and provide inference results for generative LLMs. Furthermore, we perform a qualitative and quantitative comparison of inference performance with batch sizes 1 and 8 on different hardware platforms by considering hardware power consumption, absolute inference speed (tokens/s), and energy efficiency (tokens/J). We compare the performance of the same optimization methods across different hardware platforms, the performance across different hardware platforms, and the performance of different methods on the same hardware platform. This provides a systematic and comprehensive summary of existing inference acceleration work by integrating software optimization methods and hardware platforms. We point out that three trends (multimodality, inference-time compute, and higher inference energy efficiency) are promising to redefine the capabilities of edge artificial intelligence systems. Our project is available at https://dai.sjtu.edu.cn/project.html.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | Overview of generative LLM architectures and the inference process | llm | 0.85 |
| contribution | Survey of inference optimization methods on CPU, GPU, FPGA, ASIC and PIM/NDP platforms | llm | 0.85 |
| contribution | Qualitative and quantitative cross-platform comparison of tokens/s, power and tokens/J at batch sizes 1 and 8 | llm | 0.85 |
| contribution | Identification of trends: multimodality, inference-time compute, higher inference energy efficiency | llm | 0.85 |
| libraries | llama.cpp | llm | 0.85 |
| models | LLaMA-2-7B | llm | 0.85 |
| models | LLaMA-2-70B | llm | 0.85 |
| models | GPT-3-7B | llm | 0.85 |
| models | GPT-6.7B | llm | 0.85 |
| models | OPT-30B | llm | 0.85 |
| models | GPT-3-175B | llm | 0.85 |
| tasks | generative LLM inference | llm | 0.85 |
| tasks | edge AI inference | llm | 0.85 |
| hardware | CPU | llm | 0.85 |
| hardware | GPU | llm | 0.85 |
| hardware | FPGA | llm | 0.85 |
| hardware | ASIC | llm | 0.85 |
| hardware | PIM/NDP | llm | 0.85 |
| hardware | NVIDIA A100 GPU | llm | 0.85 |
| hardware | NVIDIA H100 GPU | llm | 0.85 |
| hardware | Xilinx U280 FPGA | llm | 0.85 |
| hardware | AMD MI100 GPU | llm | 0.85 |
| hardware | Snapdragon 8 Gen 3 | llm | 0.85 |
| code url | https://github.com/Kimho666/LLM_Hardware_Survey (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/ggerganov/llama.cpp (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/NVIDIA/TensorRT-LLM (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/Kimho666/LLM_Hardware_Survey | llm | 0.85 |
| key result | throughput (survey-reported): 390-450 tokens/s (GLITCHES heterogeneous system, A100/V100S GPU + 7x U280 FPGAs, Llama2-7B) | llm | 0.85 |
| key result | throughput (survey-reported): ~3k tokens/s at ~76 W (NeuPIMs PIM-NPU, batch size 8, GPT3-7B) | llm | 0.85 |
| key result | throughput (survey-reported): 167 / 220 / 900 tokens/s (AiMX-xPU (1x H100 + 3 AiMX), OPT-30B, batch 1 / 8 / 32) | llm | 0.85 |
| key result | throughput (survey-reported): 13.75 tokens/s (Hermes NDP-DIMM + GPU, LLaMA2-70B on consumer-grade hardware) | llm | 0.85 |
SoK reference [98]¶
J. Li, J. Xu, S. Huang, Y. Chen, W. Li, J. Liu, Y. Lian, J. Pan, L. Ding, H. Zhou, Y. Wang, and G. Dai, “Large language model inference acceleration: A comprehensive hardware perspective,” 2025. [Online]. Available: https://arxiv.org/abs/2410.04466
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- library: HuggingFace 4 [p.16], PyTorch 1 [p.15]
- model: GPT 51 [p.1], LLaMA 33 [p.1], Transformer 26 [p.2], BERT 6 [p.1], Mixture of experts 6 [p.5], RNN / LSTM 5 [p.5], LoRA 2 [p.13], k-means 1 [p.12]
- hardware: GPU 205 [p.1], FPGA 97 [p.1], NVIDIA A100 74 [p.3], ASIC 51 [p.1], NVIDIA RTX 19 [p.9], NVIDIA V100 15 [p.3], NVIDIA H100 9 [p.3], Intel Xeon 9 [p.8], TPU 8 [p.5], Apple silicon 7 [p.8], NVIDIA T4 3 [p.9], Raspberry Pi 3 [p.8], AMD EPYC 3 [p.16], ARM 3 [p.3], CUDA 3 [p.2], NVIDIA Jetson 2 [p.37], AVX-512 2 [p.8]
- technique: Quantization 233 [p.3], Pruning 25 [p.14], Matrix multiplication 23 [p.4], Lookup table 21 [p.7], Attention 17 [p.4], Softmax 13 [p.4], KV cache 12 [p.4], Fine-tuning 4 [p.21], SIMD packing 3 [p.3], Rotations 3 [p.11], Convolution 3 [p.5], ReLU 2 [p.15]
- task: Language modeling / generation 122 [p.1], IoT / edge 6 [p.2]
- topic: Large language models 451 [p.1], Open source 11 [p.1], Benchmark 2 [p.21]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/098_large-language-model-inference-acceleration-a-comprehensive.pdf |
sok-references | 54 | 2.9 MB | 4b333f462621 |
Database key: large-language-model-inference-acceleration-a-comprehensive-hardware · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set large-language-model-inference-acceleration-a-comprehensive-hardware COLUMN VALUE