Communication-Efficient Distributed Machine Learning

This proposal outlines a research program for collaborative machine learning (ML) in distributed environments where overwhelming quantities of data need to be exchanged during the course of a computation. In such scenarios, communication becomes a bottleneck for fast computation. Our objective is to develop a systematic methodology and analytic framework for communication-ecient distributed ML. The ML tasks considered here include classication, sequential hypothesis testing, linear inverse problems, matrix and tensor completion, deep neural network training, reinforcement learning, and generative adversarial networks. Two frameworks will be developed, the rst based on interactive successive renement codes and the second based on adapting existing numerical methods. We will investigate the optimality ofour algorithms through a combination of information theoretic and geometric analyses.The formulation of ML problems as a distributed function computation problemsis novel and brings a sharper focus to the communication aspects of implementing machine learning algorithms in a distributed manner. Our problem formulations bring together the elds of information theory, machine learning, and optimization. This research will develop frameworks for systematically constructing interactive as well as non-interactive communication ecient algorithms for machine learning. Novel methods for estimating stopping times will be developed. Progress will deepen our understanding of the relation between geometric structures that underlie ML algorithms and the amount of communication needed in a distributed setting.