| Title | : | Towards Next-Generation ML/AI: Robustness, Optimization, Privacy |
| Speaker | : | Dr. Krishna Pillutla (Google Research) |
| Details | : | Tue, 17 Jan, 2023 10:00 AM @ CS25 |
| Abstract: | : | Two trends have taken hold in machine learning and artificial intelligence: a move to massive, general-purpose, pre-trained models as well as a move to small, on-device models trained on distributed data. Both these disparate settings face some common challenges: a need for (a) robustness to deployment conditions that differ from training, (b) faster optimization, and (c) protection of data privacy. As a result of the former trend, large language models have displayed emergent capabilities they have not been trained for. Recent models such as GPT-3 have attained the ability to generate remarkably human-like long-form text. I will describe Mauve, a measure to quantify the goodness of this emergent capability. It measures the gap between the distribution of generated text and that of human-written text. Experimentally, Mauve correlates the strongest with human evaluations of the generated text and can quantify a number of its qualitative properties. The move to massively distributed on-device federated learning of models opens up new challenges due to the natural diversity of the underlying user data and the need to protect its privacy. I will discuss how to reframe the learning problem to make the model robust to natural distribution shifts arising from deployment on diverse users who do not conform to the population trends. I will describe a distributed optimization algorithm and show how to implement it with end-to-end differential privacy. To conclude, I will discuss my ongoing efforts and future plans to work toward the next generation of ML/AI by combining the best of both worlds with applications to differentially private language models and text generation to decentralized learning. Bio: Krishna Pillutla is a visiting researcher (postdoc) at Google Research, USA in the Federated Learning team. He obtained his Ph.D. at the University of Washington where he was advised by Zaid Harchaoui and Sham Kakade. Before that, he received his M.S. from Carnegie Mellon University and B.Tech. from IIT Bombay where he was advised by Nina Balcan and J. Saketha Nath respectively. Krishna's research has been recognized by a NeurIPS outstanding paper award (2021) and a JP Morgan Ph.D. fellowship (2019-20). |
