Collaborative Research: Foundations of Deep Learning: Theory, Robustness, and the Brain

A truly comprehensive theory of machine learning has the potential of informing science and engineering in the same profound way Maxwell's equations did. It was the development of that theory by Maxwell that truly unleashed the potential of electricity, leading to radio, radars, computers, and the Internet. In an analogy, deep learning (DL) has found over the past decade many applications, so far without a comprehensive theory. An eventual theory of learning that explains why and how deep networks work and what their limitations are may thus enable the development of even more powerful learning approaches – especially if the goal of reconnecting DL to brain research bears fruit. In the long term, the ability to develop and build better intelligent machines will be essential to any technology-based economy. After all, even in its current – still highly imperfect –state, DL is impacting or about to impact just about every aspect of our society and life. The investigators also plan to complement their theoretical research with the educational goal of training a diverse population of young researchers from mathematics, computer science, statistics, electrical engineering, and computational neuroscience in the field of machine learning and of its theoretical underpinnings.

The investigators propose to join forces in a multi-pronged and collaborative assault on the profound mysteries of DL, informed by the sum of their experience, expertise, ideas, and insight. The research goals are threefold: to develop a sound foundational/mathematical understanding of DL; in doing so to advance the foundational understanding of learning more generally; and to advance the practice of DL by addressing its above-mentioned weaknesses. Of six foundational thrusts, the first two focus on the standard decomposition of the prediction error in approximation and sample (or estimation) error. Their goal is to extend classical results in approximation theory and theory of learnability to DL. These two are then supported by a research project that is specific to deep learning: analysis of the dynamics of gradient descent in training a network. The fourth theme is about robustness against adversaries and shifts, a powerful test for theories which is also important for practical deployment of learning systems. The fifth thrust is about developing the theory of control through DL, as well as exploring dynamical systems aspects of deep reinforcement learning. The final topic connects research on DL to its origins - and possibly its future: networks of neurons in the brain. The proposed research also promises to advance the foundations of learning theory. Success in this project will result in sharper mathematical techniques for machine learning and comprehensive foundations of machine learning robustness, broadly construed. It will also ultimately enable development of learning algorithms that transcend deep learning and guide the way towards creating more intelligent machines, and shed new light on our own intelligence.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.