Skip to content

ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs

Authors: Yujia Zhai; Chengquan Jiang; Leyuan Wang; Xiaoying Jia; Shang Zhang; Zizhong Chen; Xin Liu; Yibo Zhu
Year: 2023
Venue: arXiv preprint

type: library collection: sok-references status: auto

Links: arXiv:2210.03052

Summary

Auto-extracted summary (unreviewed)

ByteTransformer is a plaintext GPU inference library for BERT-like transformers that removes the cost of zero-padding on variable-length inputs through a padding-free algorithm and architecture-aware fused multi-head attention kernels. On an NVIDIA A100 the fused MHA is 6.13x faster than PyTorch, and end-to-end BERT forward passes beat PyTorch JIT, TensorFlow XLA, TurboTransformer, DeepSpeed-Inference and FasterTransformer by 87%, 131%, 138%, 74% and 55%. It contains no cryptography or privacy mechanism.

Abstract

Transformers have become keystone models in natural language processing over the past decade. They have achieved great popularity in deep learning applications, but the increasing sizes of the parameter spaces required by transformer models generate a commensurate need to accelerate performance. Natural language processing problems are also routinely faced with variable-length sequences, as word counts commonly vary among sentences. Existing deep learning frameworks pad variable-length sequences to a maximal length, which adds significant memory and computational overhead. In this paper, we present ByteTransformer, a high-performance transformer boosted for variable-length inputs. We propose a padding-free algorithm that liberates the entire transformer from redundant computations on zero padded tokens. In addition to algorithmic-level optimization, we provide architecture-aware optimizations for transformer functional modules, especially the performance-critical algorithm Multi-Head Attention (MHA). Experimental results on an NVIDIA A100 GPU with variable-length sequence inputs validate that our fused MHA outperforms PyTorch by 6.13x. The end-to-end performance of ByteTransformer for a forward BERT transformer surpasses state-of-the-art transformer frameworks, such as PyTorch JIT, TensorFlow XLA, Tencent TurboTransformer, Microsoft DeepSpeed-Inference and NVIDIA FasterTransformer, by 87\%, 131\%, 138\%, 74\% and 55\%, respectively. We also demonstrate the general applicability of our optimization methods to other BERT-like models, including ALBERT, DistilBERT, and DeBERTa.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution padding-free algorithm that skips computation on zero-padded tokens across the whole transformer llm 0.85
contribution architecture-aware fused multi-head attention kernels llm 0.85
contribution optimizations of other transformer functional modules llm 0.85
contribution open-source high-performance transformer library llm 0.85
contribution demonstrated applicability to ALBERT, DistilBERT and DeBERTa llm 0.85
libraries CUTLASS llm 0.85
libraries PyTorch llm 0.85
libraries TensorFlow XLA llm 0.85
libraries TurboTransformer llm 0.85
libraries DeepSpeed-Inference llm 0.85
libraries FasterTransformer llm 0.85
models BERT llm 0.85
models ALBERT llm 0.85
models DistilBERT llm 0.85
models DeBERTa llm 0.85
tasks transformer inference llm 0.85
tasks NLP llm 0.85
hardware NVIDIA A100 GPU llm 0.85
code url https://github.com/bytedance/ByteTransformer.quent (URL found in the full text) regex 0.50
code url https://github.com/NVIDIA/FasterTransformer (URL found in the full text) regex 0.50
code url https://github.com/NVIDIA/TensorRT/tree/main/plugin/bertQKVToContextPluginl (URL found in the full text) regex 0.50
code url https://github.com/bytedance/ByteTransformer llm 0.85
limitations authors state they are still working toward making ByteTransformer completely open-source llm 0.85
key result speed-up of fused MHA: 6.13x over PyTorch (variable-length inputs, A100 GPU) llm 0.85
key result end-to-end BERT speed-up: 87% over PyTorch JIT, 131% over TensorFlow XLA, 138% over TurboTransformer, 74% over DeepSpeed-Inference, 55% over FasterTransformer (forward BERT transformer, A100 GPU) llm 0.85

SoK reference [96]

Y. Zhai, C. Jiang, L. Wang, X. Jia, S. Zhang, Z. Chen, X. Liu, and Y. Zhu, “ ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs ,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Los Alamitos, CA, USA: IEEE Computer Society, May 2023, pp. 344–355. [Online]. Available: https://doi.ieeecomputersociety.org/ 10.1109/IPDPS54959.2023.00042

Listed on the reference page of the site.

SoK reference [97]

——, “ByteTransformer: A high-performance transformer boosted for variable-length inputs,” https://github.com/bytedance/ ByteTransformer, 2022, accessed: 2026-04-24.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • library: PyTorch 25 [p.1], TensorFlow 13 [p.1]
  • model: BERT 59 [p.1], Transformer 8 [p.1], GPT 3 [p.1]
  • hardware: GPU 21 [p.1], NVIDIA A100 7 [p.1], CUDA 5 [p.5], Intel Xeon 1 [p.8]
  • technique: LayerNorm 32 [p.2], Softmax 26 [p.3], GELU 10 [p.4], Attention 9 [p.2]
  • task: Language modeling / generation 2 [p.1], Machine translation 2 [p.1]
  • topic: Benchmark 12 [p.2], Open source 8 [p.1]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/096_bytetransformer-a-high-performance-transformer-boosted-for-v.pdf sok-references 12 1.4 MB 26ebcc3d8e12
references/sok-references/097_bytetransformer-a-high-performance-transformer-boosted-for-v.pdf sok-references 12 1.4 MB 26ebcc3d8e12 byte-identical to references/sok-references/096_bytetransformer-a-high-performance-transformer-boosted-for-v.pdf

Database key: bytetransformer-a-high-performance-transformer-boosted-for-variable · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set bytetransformer-a-high-performance-transformer-boosted-for-variable COLUMN VALUE