Modern spectral theory meets deep learning: nonconvex learning with limited resources

Optimizing non-convex objective functions is central to machine learning (ML), such as modern deep learning and classical latent variable model learning. Spectral methods that go beyond matrices tackle the problem of non-convexity in learning latent variable models and provide provable performance guarantees using tensor decompositions. In contrast, the popular gradient-based optimization methods for deep neural networks often lack convergence guarantees. Our objective is to develop modern spectral theory - generalized tensor decompositions - that advances existing spectral methods to be adaptable for learning deep neural networks under limited resources. The talk focuses on non-convex learning with limited resources, a central challenge in modern ML settings as federated learning and personalized ML on constrained devices gain increasing popularity. An overview of our group’s research progress on performance guaranteed spectral methods for deep neural networks will be presented. Using tensor representation theory, we extend traditional neural nets to more general tensorial neural networks that are interpretable and compact. We demonstrate tensorial neural networks' advantages over traditional neural networks by theoretically proving their improved generalization and expressive power with fewer parameters. Using the developed new class of latent variable model learning, we establish guaranteed learning methods for deep ResNets. Finally, we obtain certifiably robust deep networks using latent variables and architecture design in the spectral domain.