Affiliations: [a] Dipartimento di Ingegneria dell’Informazione, Elettronica e Telecomunicazioni (DIET), Sapienza University of Rome, Rome, Italy
| [b] Department of Computer, Control, and Management Engineering Antonio Ruberti (DIAG), Rome, Italy
| [c]
Geomatics Research Group, KU Leuven, Gent, Belgium
| [d]
School of Informatics, University of Edinburgh, Edinburgh, UK
Correspondence:
[*]
Corresponding author: Simone Scardapane, DIET Department, Sapienza University of Rome, Via Eudossiana 18, 00184, Rome, Italy. E-mails: [email protected], [email protected].
Abstract: This article summarizes principles and ideas from the emerging area of applying conditional computation methods to the design of neural networks. In particular, we focus on neural networks that can dynamically activate or de-activate parts of their computational graph conditionally on their input. Examples include the dynamic selection of, e.g., input tokens, layers (or sets of layers), and sub-modules inside each layer (e.g., channels in a convolutional filter). We first provide a general formalism to describe these techniques in an uniform way. Then, we introduce three notable implementations of these principles: mixture-of-experts (MoEs) networks, token selection mechanisms, and early-exit neural networks. The paper aims to provide a tutorial-like introduction to this growing field. To this end, we analyze the benefits of these modular designs in terms of efficiency, explainability, and transfer learning, with a focus on emerging applicative areas ranging from automated scientific discovery to semantic communication.
