Date: Wednesday, November 15, 2023

Time: 9:30-10:30 am EST

Location: Coda C1215 Midtown or this zoom link

James Seale Smith

Machine Learning PhD Candidate

School of Interactive Computing
Georgia Institute of Technology

Committee

1. Dr. Zsolt Kira (Advisor), School of Interactive Computing, Georgia Tech
2. Dr. James Hays, School of Interactive Computing, Georgia Tech
3. Dr. Larry Heck, School of Electrical and Computer Engineering and the School of Interactive Computing, Georgia Tech
4. Dr. Judy Hoffman, School of Interactive Computing, Georgia Tech
5. Dr. Leonid Karlinsky, MIT-IBM Watson AI Lab

Abstract

Machine learning models suffer from a phenomenon known as catastrophic forgetting when learning novel concepts from continuously shifting training data. Typical solutions for this continual learning problem require extensive replay of previously seen data, which increases memory costs and may violate data privacy. To address these challenges, we first explore replacing this replay data with alternatives: (i) unlabeled data “from the wild” and (ii) synthetic data generated via model inversion. Our work using this alternative replay data boasts strong performance on replay-free continual learning for image classification. Next, we consider an alternative solution to entirely replace replay data: pre-training. Specifically, we leverage strongly pre-trained models and continuously edit them with prompts and low-rank adapters for both (i) image classification and (ii) natural-language visual reasoning. Finally, we extend the idea of continual learning using pre-trained models to the proposed setting of continual customization of text-to-image diffusion models.  We hope that our work on enabling models to learn from evolving data distributions and adapt to new tasks will help unlock the full potential of machine learning in addressing emerging real-world challenges.