Prof. Reut Tsarfaty

Prof. Reut Tsarfaty

Associate Professor

Department of Computer Science, Bar-Ilan University

Israel Israel

Reut Tsarfaty is Associate Professor at Bar-llan University, leading the Open Natural Language Processing research lab (The ONLP Lab). Her research focuses on natural language processing broadly interpreted to cover morphological, syntactic and semantic phenomena, extended for the analysis of typologically different languages. She is an international expert on models for morphologically rich languages (MRLs) in general and on Hebrew NLP in particular. Reut's research is funded by an ERC-Starting-Grant #677352, an ISF grant #1739/26, a grant by MOST (Israel's ministery of science and eduction) and Faculty Research Awards (FRA) from Google.

Abstract Summary: On the linguistics and epistemology of LLMs Large language models (LLMs) have shown sweeping success in many NLP tasks, boasting two kinds of capabilities: (i) "understanding" language (ii) "knowing" stuff. The former has to do with the linguistic capabilities of the model, the way it transfers forms into meanings, and the latter has to do with its epistemic state, the evidence that the model has for facts which it has observed during pre-training. In this talk, we focus on the well-known Machine Reading Comprehension task, and investigate the interplay between the linguistic and epistemic capabilities in LLMs. In human comprehension, human readers seamlessly integrate what they already know with what the utterance is trying to tell them. Here we show a series of experiments with contemporary LLMs that cast doubt on their ability to effectively integrate the linguistic aspect of the utterance with the epistemic state that the model possesses, with respect to the utterance content. Specifically we show how negation, counterfactuals and modal contexts undermine the ability to correctly answer questions due to interfering background knowledge. These experiments demonstrate certain epistemic biases that the model possesses, that may override or even contradict what we would conceive of as human-like understanding. This work sheds new light on the issues of consistency, coherence, and general usability of current LLMs as effective human-like comprehenders and responders.

SPEAKER