PhD Proposal: Move Faster or Go Smaller: Efficient Models and Learning Paradigms for Resource-Constrained Systems

As deep learning algorithms continue to grow more ambitious in task complexity, so do the sizes of their underlying architectures and training complexities. The ability to host and quickly compute powerful models on more modest devices, such as mobile devices, sensors, smartwatches, etc. is a necessary research question to consider as Internet of Things (IoT) technologies continue to expand in technology and demand. Motivated by these challenges, we consider two approaches to improving efficiency: (1) reducing model size and (2) minimizing the amount of computation and time spent training. Emerging frameworks such as federated learning (FL), graph neural networks (GNNs), and tensorial neural networks (TNNs) impose additional privacy and resource constraints which require novel variations upon classical methods and entirely new strategies altogether.The first approach (1) involves dimensionality reduction of taxing model structures. We first propose a novel Count Sketch parametrization of fully-connected and convolutional weights which is desirable for small clients operating in an FL ecosystem. We then apply Count Sketch compression to GNNs to sketch nonlinear activations and node embeddings, resulting in a message passing protocol which achieves sublinear training time and memory complexities with respect to graph size. Returning to FL and inspired by recent literature in adaptive sparsification/pruning via locality-sensitive hashing (LSH), we develop a new hashing family, the Periodic Gaussian Hash (PGHash), which enables federated clients to dynamically prune massive, fully-connected weights via private, on-device LSH.The second approach (2) seeks to reduce the computational and time complexity of various learning paradigms through an ad hoc mixture of momentum, automated machine learning (AutoML), and asynchronous methods. We first accelerate the principal component analysis (PCA) underlying unsupervised spectral clustering via our novel variation of the classical power method, DMPower, which incorporates a semi-adaptive momentum term. Shifting our focus to TNNs, we develop an automated framework and open-source library, AutoTNN, for compressing and minimizing floating point operations through convolutional TNNs. Looping back to FL, we propose SWIFT, a novel approach to peer-to-peer (decentralized) federated learning which allows clients to train in an asynchronous, free-flowing manner.

Examining Committee

Chair:

Dr. Furong Huang

Department Representative:

Dr. Nirupam Roy

Members:

Dr. Tom Goldstein