Bitnet: Scaling 1-bit transformers for large language models¶
Authors: Hongyu Wang; Shuming Ma; Li Dong; Shaohan Huang; Huaijie Wang; Lingxiao Ma; Fan Yang; Ruiping Wang; Yi Wu; Furu Wei
Year: 2023
Venue: arXiv preprint
type: technique collection: sok-references status: auto
Links: arXiv:2310.11453
Summary¶
Auto-extracted summary (unreviewed)
Introduces BitNet, a Transformer architecture for LLMs whose BitLinear layer replaces nn.Linear so that binary (1-bit) weights with 8-bit activations are trained from scratch. On language modeling, BitNet reaches perplexity and zero/few-shot accuracy close to FP16 Transformers while greatly cutting memory and energy, clearly beats post-training quantization methods at low bit widths, and follows a scaling law similar to full-precision models.
Abstract¶
The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scaling law akin to full-precision Transformers, suggesting its potential for effective scaling to even larger language models while maintaining efficiency and performance benefits.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | BitLinear drop-in replacement for nn.Linear enabling 1-bit weight training from scratch | llm | 0.85 |
| contribution | scalable and stable 1-bit Transformer architecture (W1A8) | llm | 0.85 |
| contribution | scaling-law analysis showing behaviour akin to full-precision Transformers | llm | 0.85 |
| contribution | training-stability analysis allowing larger learning rates | llm | 0.85 |
| contribution | comparison against post-training quantization (Absmax, SmoothQuant, GPTQ, QuIP) | llm | 0.85 |
| models | BitNet (1.3B, 2.7B, 6.7B) | llm | 0.85 |
| models | FP16 Transformer | llm | 0.85 |
| datasets | Hellaswag | llm | 0.85 |
| datasets | Winogrande | llm | 0.85 |
| datasets | Winograd | llm | 0.85 |
| datasets | Storycloze | llm | 0.85 |
| tasks | language modeling | llm | 0.85 |
| tasks | zero-shot and few-shot downstream evaluation | llm | 0.85 |
| tasks | LLM quantization | llm | 0.85 |
| limitations | future work needed to scale BitNet in model size and training steps | llm | 0.85 |
| limitations | only evaluated on the Transformer architecture; other architectures (e.g. RetNet) left for future work | llm | 0.85 |
| llm relevance | low | llm | 0.85 |
| key result | perplexity: 17.07 (BitNet W1A8) vs 15.19 (FP16 Transformer) (6.7B models, validation set) | llm | 0.85 |
| key result | average zero-shot accuracy: 55.9 (BitNet) vs 57.8 (FP16) (6.7B models, Winogrande/Winograd/Storycloze/Hellaswag) | llm | 0.85 |
| key result | average zero-shot accuracy vs best PTQ at low bits: 55.9 (BitNet, 1-bit) vs 49.0 (QuIP, 2-bit) (6.7B models) | llm | 0.85 |
| key result | inference energy cost reduction vs FP16 Transformer: 8.8x to 38.8x (model sizes from 100M to 30B (Figure 1)) | llm | 0.85 |
SoK reference [76]¶
H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei, “Bitnet: Scaling 1-bit transformers for large language models,” arXiv preprint arXiv:2310.11453, 2023.
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- library: PyTorch 1 [p.2]
- model: Transformer 49 [p.1], BERT 3 [p.2], GPT 2 [p.10], LLaMA 2 [p.12], CNN 2 [p.2]
- dataset: HellaSwag 1 [p.12]
- technique: Quantization 53 [p.1], Matrix multiplication 10 [p.3], LayerNorm 4 [p.3], Convolution 2 [p.2], Fine-tuning 2 [p.2]
- task: Language modeling / generation 5 [p.1], Machine translation 4 [p.2]
- topic: Large language models 25 [p.1], Benchmark 2 [p.2]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/076_bitnet-scaling-1-bit-transformers-for-large-language-models.pdf |
sok-references | 14 | 572.7 KB | 55424deeba35 |
Database key: bitnet-scaling-1-bit-transformers-for-large-language-models · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set bitnet-scaling-1-bit-transformers-for-large-language-models COLUMN VALUE