Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract / Description of output

Sigmoid output layers are widely used in multi-label classification (MLC) tasks, in which multiple labels can be assigned to any input. In many practical MLC tasks, the number of possible labels is in the thousands, often exceeding the number of input features and resulting in a low-rank output layer. In multi-class classification, it is known that such a lowrank output layer is a bottleneck that can result in unargmaxable classes: classes which cannot be predicted for any input. In this paper, we show that for MLC tasks, the analogous sigmoid bottleneck results in exponentially many unargmaxable label combinations. We explain how to detect these unargmaxable outputs and demonstrate their presence in three widely used MLC datasets. We then show that they can be prevented in practice by introducing a Discrete Fourier Transform (DFT) output layer, which guarantees that all sparse label combinations with up to k active labels are argmaxable. Our DFT layer trains faster and is more parameter efficient, matching the F1@k score of a sigmoid layer while using up to 50% fewer trainable parameters. Our code is publicly available at https://github.com/andreasgrv/sigmoid-bottleneck.
Original languageEnglish
Title of host publicationProceedings of the 38th Annual AAAI Conference on Artificial Intelligence
Subtitle of host publicationAAAI Technical Track on Machine Learning II
PublisherAAAI Press
Number of pages9
ISBN (Electronic)9781577358879
Publication statusPublished - 24 Mar 2024
EventThe 38th Annual AAAI Conference on Artificial Intelligence - Vancouver, Canada
Duration: 20 Feb 202427 Feb 2024
Conference number: 38

Publication series

NameProceedings of the AAAI Conference on Artificial Intelligence
PublisherAAAI Press
ISSN (Print)2159-5399
ISSN (Electronic)2374-3468


ConferenceThe 38th Annual AAAI Conference on Artificial Intelligence
Abbreviated titleAAAI 2024
Internet address


Dive into the research topics of 'Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification'. Together they form a unique fingerprint.

Cite this