Skip to content

Bitnet: Scaling 1-bit transformers for large language models

Authors: Hongyu Wang; Shuming Ma; Li Dong; Shaohan Huang; Huaijie Wang; Lingxiao Ma; Fan Yang; Ruiping Wang; Yi Wu; Furu Wei
Year: 2023
Venue: arXiv preprint

type: technique collection: sok-references status: auto

Links: arXiv:2310.11453

Summary

Auto-extracted summary (unreviewed)

Introduces BitNet, a Transformer architecture for LLMs whose BitLinear layer replaces nn.Linear so that binary (1-bit) weights with 8-bit activations are trained from scratch. On language modeling, BitNet reaches perplexity and zero/few-shot accuracy close to FP16 Transformers while greatly cutting memory and energy, clearly beats post-training quantization methods at low bit widths, and follows a scaling law similar to full-precision models.

Abstract

The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scaling law akin to full-precision Transformers, suggesting its potential for effective scaling to even larger language models while maintaining efficiency and performance benefits.
Source: arxiv

Extracted details

Field Value Source Confidence
contribution BitLinear drop-in replacement for nn.Linear enabling 1-bit weight training from scratch llm 0.85
contribution scalable and stable 1-bit Transformer architecture (W1A8) llm 0.85
contribution scaling-law analysis showing behaviour akin to full-precision Transformers llm 0.85
contribution training-stability analysis allowing larger learning rates llm 0.85
contribution comparison against post-training quantization (Absmax, SmoothQuant, GPTQ, QuIP) llm 0.85
models BitNet (1.3B, 2.7B, 6.7B) llm 0.85
models FP16 Transformer llm 0.85
datasets Hellaswag llm 0.85
datasets Winogrande llm 0.85
datasets Winograd llm 0.85
datasets Storycloze llm 0.85
tasks language modeling llm 0.85
tasks zero-shot and few-shot downstream evaluation llm 0.85
tasks LLM quantization llm 0.85
limitations future work needed to scale BitNet in model size and training steps llm 0.85
limitations only evaluated on the Transformer architecture; other architectures (e.g. RetNet) left for future work llm 0.85
llm relevance low llm 0.85
key result perplexity: 17.07 (BitNet W1A8) vs 15.19 (FP16 Transformer) (6.7B models, validation set) llm 0.85
key result average zero-shot accuracy: 55.9 (BitNet) vs 57.8 (FP16) (6.7B models, Winogrande/Winograd/Storycloze/Hellaswag) llm 0.85
key result average zero-shot accuracy vs best PTQ at low bits: 55.9 (BitNet, 1-bit) vs 49.0 (QuIP, 2-bit) (6.7B models) llm 0.85
key result inference energy cost reduction vs FP16 Transformer: 8.8x to 38.8x (model sizes from 100M to 30B (Figure 1)) llm 0.85

SoK reference [76]

H. Wang, S. Ma, L. Dong, S. Huang, H. Wang, L. Ma, F. Yang, R. Wang, Y. Wu, and F. Wei, “Bitnet: Scaling 1-bit transformers for large language models,” arXiv preprint arXiv:2310.11453, 2023.

Listed on the reference page of the site.

Keyword mentions

Counts of vocabulary terms in the full text (first page of occurrence in brackets).

  • library: PyTorch 1 [p.2]
  • model: Transformer 49 [p.1], BERT 3 [p.2], GPT 2 [p.10], LLaMA 2 [p.12], CNN 2 [p.2]
  • dataset: HellaSwag 1 [p.12]
  • technique: Quantization 53 [p.1], Matrix multiplication 10 [p.3], LayerNorm 4 [p.3], Convolution 2 [p.2], Fine-tuning 2 [p.2]
  • task: Language modeling / generation 5 [p.1], Machine translation 4 [p.2]
  • topic: Large language models 25 [p.1], Benchmark 2 [p.2]

Files

Path (under papers/) Collection Pages Size SHA-256 Note
references/sok-references/076_bitnet-scaling-1-bit-transformers-for-large-language-models.pdf sok-references 14 572.7 KB 55424deeba35

Database key: bitnet-scaling-1-bit-transformers-for-large-language-models · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set bitnet-scaling-1-bit-transformers-for-large-language-models COLUMN VALUE