Date: Wednesday, November 15, 2023

Time: 2-3pm EST

Location: Coda C0915 Atlantic; Zoom Link

Karan Samel

Machine Learning Ph.D. Student

School of Interactive Computing
Georgia Institute of Technology

Committee

Dr. Irfan Essa (Advisor) — School of Interactive Computing, Georgia Institute of Technology

Dr. Alan Ritter — School of Interactive Computing, Georgia Institute of Technology

Dr. Thomas Ploetz — School of Interactive Computing, Georgia Institute of Technology

Abstract

Deep learning models have shown state-of-the-art performance on classification tasks by optimizing latent representations end-to-end on large amounts of training data. For domains with limited data, it is crucial to incorporate prior factual knowledge and step-by-step functional logic into models as inductive biases to improve model performance and data efficiency. 

The first part of the proposal demonstrates how knowledge graphs and taxonomies are used as additional data sources to improve downstream task performance, where gains in E-commerce applications are shown. The second part illustrates how to leverage functional logic to improve the computation and data efficiencies of complex machine learning tasks, such as visual question answering or agent policy learning. In the third part, the proposed work combines knowledge and step-by-step instructions as external inputs within large foundational models commonly used today. The particular use case focuses on multi-modal foundational models for video summarization, where external step-by-step instructions and knowledge requirements are proposed to improve summarization performance.