Theory, analysis, and best practices for sigmoid selfattention¶
Authors: Jason Ramapuram; Federico Danieli; Eeshan Dhekane; Floris Weers; Dan Busbridge; Pierre Ablin; Tatiana Likhomanenko; Jagrit Digani; Zijin Gu; Amitis Shidani; Russ Webb
Year: 2025
Venue: arXiv preprint
type: technique collection: sok-references status: auto
Links: arXiv:2409.04431
Summary¶
Auto-extracted summary (unreviewed)
Revisits element-wise sigmoid attention as a drop-in replacement for row-wise softmax attention in transformers. The authors prove that sigmoid-attention transformers are universal function approximators with improved regularity (tighter Jacobian bound), identify large initial attention norms as the main training obstacle and mitigate it with ALiBi, a bias initialization b = -ln(n), LayerScale or hybrid normalization, and release FlashSigmoid, a FlashAttention2-style kernel giving a 17% inference kernel speed-up on H100 GPUs. Across vision, speech and language models up to 7B parameters, normalized sigmoid attention matches softmax attention.
Abstract¶
Attention is a key part of the transformer architecture. It is a sequence-to-sequence mapping that transforms each sequence element into a weighted sum of values. The weights are typically obtained as the softmax of dot products between keys and queries. Recent work has explored alternatives to softmax attention in transformers, such as ReLU and sigmoid activations. In this work, we revisit sigmoid attention and conduct an in-depth theoretical and empirical analysis. Theoretically, we prove that transformers with sigmoid attention are universal function approximators and benefit from improved regularity compared to softmax attention. Through detailed empirical analysis, we identify stabilization of large initial attention norms during the early stages of training as a crucial factor for the successful training of models with sigmoid attention, outperforming prior attempts. We also introduce FLASHSIGMOID, a hardware-aware and memory-efficient implementation of sigmoid attention yielding a 17% inference kernel speed-up over FLASHATTENTION2 on H100 GPUs. Experiments across language, vision, and speech show that properly normalized sigmoid attention matches the strong performance of softmax attention on a wide range of domains and scales, which previous attempts at sigmoid attention were unable to fully achieve. Our work unifies prior art and establishes best practices for sigmoid attention as a drop-in softmax replacement in transformers.
Source: arxiv
Extracted details¶
| Field | Value | Source | Confidence |
|---|---|---|---|
| contribution | Proof that sigmoid attention transformers are universal sequence-to-sequence approximators | llm | 0.85 |
| contribution | Regularity analysis with worst-case Jacobian bound for sigmoid attention | llm | 0.85 |
| contribution | FlashSigmoid: hardware-aware, memory-efficient sigmoid attention kernel extending FlashAttention2 | llm | 0.85 |
| contribution | Best practices for stable training: ALiBi or negative bias initialization, LayerScale, hybrid norm | llm | 0.85 |
| contribution | Empirical validation across supervised and self-supervised vision, ASR and autoregressive LM | llm | 0.85 |
| libraries | FlashAttention2 (extended) | llm | 0.85 |
| libraries | axlearn | llm | 0.85 |
| models | ViT-B/16 | llm | 0.85 |
| models | ViT-L/16 (MAE) | llm | 0.85 |
| models | 255M ASR Transformer | llm | 0.85 |
| models | 1B LLM | llm | 0.85 |
| models | 7B LLM | llm | 0.85 |
| models | single-layer autoregressive transformer (ablations) | llm | 0.85 |
| datasets | C4 (realnews split) | llm | 0.85 |
| datasets | TED-LIUM v3 | llm | 0.85 |
| tasks | supervised image classification | llm | 0.85 |
| tasks | self-supervised image representation learning (SimCLR, BYOL, MAE) | llm | 0.85 |
| tasks | automatic speech recognition | llm | 0.85 |
| tasks | autoregressive language modeling | llm | 0.85 |
| hardware | H100 GPU | llm | 0.85 |
| code url | https://github.com/apple/ml-sigmoid-attention.Preprint (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/apple/axlearn (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/togethercomputer/RedPajama-Data.Ekin (URL found in the full text) | regex | 0.50 |
| code url | https://github.com/apple/ml-sigmoid-attention | llm | 0.85 |
| limitations | LM and ASR are sensitive to the initial attention norm and require ALiBi, bias initialization or an extra hybrid-norm layer | llm | 0.85 |
| limitations | Vision tasks (except MAE) rely on LayerScale to match softmax attention | llm | 0.85 |
| llm relevance | low | llm | 0.85 |
| key result | inference kernel speed-up: 17% (FlashSigmoid vs FlashAttention2 on H100 GPUs) | llm | 0.85 |
| key result | real-world inference speed-up: up to 8% (FlashSigmoid vs FlashAttention2) | llm | 0.85 |
| key result | task performance: matches softmax attention (vision, ASR and 1B/7B LLM training losses) | llm | 0.85 |
SoK reference [84]¶
J. Ramapuram, F. Danieli, E. G. Dhekane, F. Weers, D. Busbridge, P. Ablin, T. Likhomanenko, J. Digani, Z. Gu, A. Shidani, and R. Webb, “Theory, analysis, and best practices for sigmoid selfattention,” in The Thirteenth International Conference on Learning Representations, 2025. [Online]. Available: https://openreview.net/ forum?id=Zhdhg6n2OG
Listed on the reference page of the site.
Keyword mentions¶
Counts of vocabulary terms in the full text (first page of occurrence in brackets).
- library: PyTorch 4 [p.9]
- model: ViT 23 [p.6], Transformer 16 [p.3], MLP 10 [p.23], RNN / LSTM 3 [p.14], LLaMA 2 [p.17], Autoencoder 2 [p.5], GPT 1 [p.16]
- dataset: ImageNet 7 [p.8], ARC 5 [p.10], LAMBADA 2 [p.10]
- hardware: GPU 80 [p.1], NVIDIA A100 22 [p.5], NVIDIA H100 22 [p.1], CUDA 1 [p.56]
- technique: Sigmoid 479 [p.1], Softmax 195 [p.1], Attention 56 [p.1], LayerNorm 34 [p.6], ReLU 10 [p.1], Fine-tuning 5 [p.9], Convolution 4 [p.14], Quantization 4 [p.22]
- task: Language modeling / generation 10 [p.5], Speech recognition 9 [p.5], Image classification 2 [p.9]
- topic: Benchmark 13 [p.5], Open source 7 [p.1], Large language models 6 [p.4]
Files¶
| Path (under papers/) | Collection | Pages | Size | SHA-256 | Note |
|---|---|---|---|---|---|
references/sok-references/084_theory-analysis-and-best-practices-for-sigmoid-self-attentio.pdf |
sok-references | 56 | 6.3 MB | 81fe5a2eabda |
Database key: theory-analysis-and-best-practices-for-sigmoid-selfattention · created 2026-11-05 · updated 2026-11-05 · edit with python3 tools/paperdb.py set theory-analysis-and-best-practices-for-sigmoid-selfattention COLUMN VALUE