CONTRACTIVITY OF NEURAL ODES: AN EIGENVALUE OPTIMIZATION PROBLEM

We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ordinary differential equations (ODEs). When looking at contractivity properties of a one-layer weight-tied neural ODE (Formula presented) (with (Formula presented), A is a given n × n matrix, (Formula presented) denotes an activation function and for a vector (Formula presented) has to be interpreted entry-wise), we are led to study the logarithmic norm of a set of products of type DA, where D is a diagonal matrix such that (Formula presented). Specifically, given a real number c (usually c = 0), the problem consists in finding the largest positive interval I ⊆ [0, ∞) such that the logarithmic norm μ(DA) ≤ c for all diagonal matrices D with Dii ∈ I. We propose a two-level nested methodology: an inner level where, for a given I, we compute an optimizer D*(I) by a gradient system approach, and an outer level where we tune I so that the value c is reached by μ(D*(I)A). We extend the proposed two-level approach to the general multilayer, and possibly time-dependent, case (Formula presented) and we propose several numerical examples to illustrate its behaviour, including its stabilizing performance on a one-layer neural ODE applied to the classification of the MNIST handwritten digits dataset.