ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs¶
Authors: Yujia Zhai; Chengquan Jiang; Leyuan Wang; Xiaoying Jia; Shang Zhang; Zizhong Chen; Xin Liu; Yibo Zhu
Year: 2023
Venue: arXiv preprint
type: library collection: sok-references status: auto
Links: arXiv:2210.03052
Summary¶
Auto-extracted summary (unreviewed)
ByteTransformer is a plaintext GPU inference library for BERT-like transformers that removes the cost of zero-padding on variable-length inputs through a padding-free algorithm and architecture-aware fused multi-head attention kernels. On an NVIDIA A100 the fused MHA is 6.13x faster than PyTorch, and end-to-end BERT forward passes beat PyTorch JIT, TensorFlow XLA, TurboTransformer, DeepSpeed-Inference and FasterTransformer by 87%, 131%, 138%, 74% and 55%. It contains no cryptography or privacy mechanism.
Abstract¶
Transformers have become keystone models in natural language processing over the past decade. They have achieved great popularity in deep learning applications, but the increasing sizes of the parameter spaces required by transformer models generate a commensurate need to accelerate performance. Natural language processing problems are also routinely faced with variable-length sequences, as word counts commonly vary among sentences. Existing deep learning frameworks pad variable-length sequences to a maximal length, which adds significant memory and computational overhead. In this paper, we present ByteTransformer, a high-performance transformer boosted for variable-length inputs. We propose a padding-free algorithm that liberates the entire transformer from redundant computations on zero padded tokens. In addition to algorithmic-level optimization, we provide architecture-aware optimizations for transformer functional modules, especially the performance-critical algorithm Multi-Head Attention (MHA). Experimental results on an NVIDIA A100 GPU with variable-length sequence inputs validate that our fused MHA outperforms PyTorch by 6.13x. The end-to-end performance of ByteTransformer for a forward BERT transformer surpasses state-of-the-art transformer frameworks, such as PyTorch JIT, TensorFlow XLA, Tencent TurboTransformer, Microsoft DeepSpeed-Inference and NVIDIA FasterTransformer, by 87\%, 131\%, 138\%, 74\% and 55\%, respectively. We also demonstrate the general applicability of our optimization methods to other BERT-like models, including ALBERT, DistilBERT, and DeBERTa.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | padding-free algorithm that skips computation on zero-padded tokens across the whole transformer | llm | 0.85 |
| contribution | architecture-aware fused multi-head attention kernels | llm | 0.85 |
| contribution | optimizations of other transformer functional modules | llm | 0.85 |
| contribution | open-source high-performance transformer library | llm | 0.85 |
| contribution | demonstrated applicability to ALBERT, DistilBERT and DeBERTa | llm | 0.85 |
| libraries | CUTLASS | llm | 0.85 |
| libraries | PyTorch | llm | 0.85 |
| libraries | TensorFlow XLA | llm | 0.85 |
| libraries | TurboTransformer | llm | 0.85 |
| libraries | DeepSpeed-Inference | llm | 0.85 |
| libraries | FasterTransformer | llm | 0.85 |
| models | BERT | llm | 0.85 |
| models | ALBERT | llm | 0.85 |
| models | DistilBERT | llm | 0.85 |
| models | DeBERTa | llm | 0.85 |
| tasks | transformer inference | llm | 0.85 |
| tasks | NLP | llm | 0.85 |
| hardware | NVIDIA A100 GPU | llm | 0.85 |
| code url | https://github.com/bytedance/ByteTransformer.quent (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/NVIDIA/FasterTransformer (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/NVIDIA/TensorRT/tree/main/plugin/bertQKVToContextPluginl (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/bytedance/ByteTransformer | llm | 0.85 |
| limitations | authors state they are still working toward making ByteTransformer completely open-source | llm | 0.85 |
| key result | speed-up of fused MHA: 6.13x over PyTorch (variable-length inputs, A100 GPU) | llm | 0.85 |
| key result | end-to-end BERT speed-up: 87% over PyTorch JIT, 131% over TensorFlow XLA, 138% over TurboTransformer, 74% over DeepSpeed-Inference, 55% over FasterTransformer (forward BERT transformer, A100 GPU) | llm | 0.85 |
SoK reference [96]¶
Y. Zhai, C. Jiang, L. Wang, X. Jia, S. Zhang, Z. Chen, X. Liu, and Y. Zhu, “ ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs ,” in 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS). Los Alamitos, CA, USA: IEEE Computer Society, May 2023, pp. 344–355. [Online]. Available: https://doi.ieeecomputersociety.org/ 10.1109/IPDPS54959.2023.00042
Listed on the reference page of the site.
SoK reference [97]¶
——, “ByteTransformer: A high-performance transformer boosted for variable-length inputs,” https://github.com/bytedance/ ByteTransformer, 2022, accessed: 2026-04-24.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- library: PyTorch 25 [p.1], TensorFlow 13 [p.1]
- model: BERT 59 [p.1], Transformer 8 [p.1], GPT 3 [p.1]
- hardware: GPU 21 [p.1], NVIDIA A100 7 [p.1], CUDA 5 [p.5], Intel Xeon 1 [p.8]
- technique: LayerNorm 32 [p.2], Softmax 26 [p.3], GELU 10 [p.4], Attention 9 [p.2]
- task: Language modeling / generation 2 [p.1], Machine translation 2 [p.1]
- topic: Benchmark 12 [p.2], Open source 8 [p.1]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/096_bytetransformer-a-high-performance-transformer-boosted-for-v.pdf |
sok-references | 12 | 1.4 MB | 26ebcc3d8e12 |
|
references/sok-references/097_bytetransformer-a-high-performance-transformer-boosted-for-v.pdf |
sok-references | 12 | 1.4 MB | 26ebcc3d8e12 |
byte-identical to references/sok-references/096_bytetransformer-a-high-performance-transformer-boosted-for-v.pdf |
Database key: bytetransformer-a-high-performance-transformer-boosted-for-variable · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set bytetransformer-a-high-performance-transformer-boosted-for-variable COLUMN VALUE