Prof. Heng Ji
University of Illinois Urbana-Champaign
USA
Heng Ji is a professor of Computer Science at University of Illinois Urbana-Champaign. She is an Amazon Scholar and Director of Amazon-Illinois Center AICE. Her research interests focus on multimedia multilingual knowledge extraction and AI for Science. The awards she received include "AI's 10 to Watch", NSF CAREER, “Young Scientist” by World Laureates Association Forum, "Young Scientist" and Global Future Council member by World Economic Forum, “Women Leaders of Conversational AI” by Project Voice, ACL2020 and NAACL2021 Best Demo Paper Awards. She is the coordinator for NIST TAC-KBP task 2010-2022, Program Committee Co-Chair of many conferences including NAACL-HLT2018 and AACL-IJCNLP2022.
Lecture abstract: Enabling Large Neural Models to Self-Learn Symbolic Knowledge Recent large neural models have shown impressive performance on various data modalities, including natural language, vision, programming language and molecules. However, they still have surprising deficiency (near-random performance) in acquiring certain types of knowledge such as structured knowledge and action knowledge. In this talk I propose a two-way knowledge acquisition framework to make symbolic and neural learning approaches mutually enhance each other. In the first stage, we will elicit and acquire explicit symbolic knowledge from large neural models. In the second stage, we will leverage the acquired symbolic knowledge along with external knowledge to augment and enhance these large neural models. Three recent case studies will be presented to demonstrate this framework: (1) The first task is to induce event schemas (stereotypical structures of events and their connections) from large language models by incremental prompting and verification [Li et al., ACL2023], and apply the induced schemas to enhance event extraction and event prediction. (2) In the second task, we noticed that current large video-language models rely on object recognition abilities as a shortcut for action understanding. We utilize a Knowledge Patcher network to elicit new action knowledge from the current models by designing specialized probing tasks and loss functions, and a Knowledge Fuser component to integrate the Patcher into frozen video-language models. (3) In the third task, we use large-scale molecule-language models to discover molecule subgraph structures ("building blocks") which contribute to blood brain barrier permeability in the kinase inhibitor family, and propose several candidate kinase inhibitor variants with improved ability to pass the blood brain barrier to accelerate better drug discovery. Then we can encode such graph pattern knowledge using lightweight adapter modules, bottleneck feed-forward networks that are inserted into different locations of backbone large molecule-language models.