Research Seminar in Mathematics - Fast and efficient neural networks’ training via low-rank gradient flows

Speaker

Francesco Tudisco, GSSI Gran Sasso Science Institute, L’Aquila, Italy.

Abstract

Neural networks have achieved tremendous success in a large variety of applications. However, their memory footprint and computational demand can render them impractical in application settings with limited hardware or energy resources. At the same time, overparametrization seems to be necessary in order to overcome the highly nonconvex nature of the optimization problem. An optimal trade-off is then to be found in order to reduce networks’ dimensions while maintaining high performance.

Popular approaches in the literature are based on pruning techniques that look for “winning tickets”, smaller subnetworks achieving approximately the initial performance. However, these techniques are not able to reduce the memory footprint of the training phase and can be unstable with respect to the input weights. In this talk we will present a training algorithm that looks for “low-rank lottery tickets” by interpreting the training phase as a continuous ODE and by integrating it within the manifold of low-rank matrices. The low-rank subnetworks and their ranks are determined and adapted during the training phase, allowing the overall time and memory resources required by both training and inference phases to be reduced significantly. We will illustrate the efficiency of this approach on a variety of fully connected and convolutional networks.

The talk is based on:
S Schotthöfer, E Zangrando, J Kusch, G Ceruti, F Tudisco
Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
https://arxiv.org/pdf/2205.13571.pdf
(to appear on NeurIPS 2022)

Welcome!