Reasoning in Language Models
TL;DR
The work done in this thesis indicates that LLMs do exhibit properties and behaviors one would expect of a system capable of reasoning, but always with caveats: models solve tasks requiring reasoning only some of the time, and the relations between concepts and propositions appear to be reflected in their internal representations, but imperfectly.
Abstract
Whether language models can reason is an important factor in their potential as a technology, both in terms of how well they can tackle novel problems and how well they can justify their decisions or actions by giving reasons for them. The question of whether they can is also contentious. Some argue that LLMs already show signs of general intelligence; others argue that their apparent reasoning rests predominantly on having seen similar problems during training. This thesis approaches the question from two angles. First, we develop methods that can be used to identify to what degree aspects of reasoning occur in LLMs. Second, we explore how LLM reasoning can be used to help detect and characterize toxic language. To look for reasoning in language models we use different strategies. First, we expect reasoning to be reflected in the model’s behavior, specifically its abilities. Thus, we evaluate LLMs on a newly constructed benchmark whose correct answers, in a zero-shot setting, are only reachable by ruling out interpretations that are logically incompatible with the premises provided. Second, we expect the capacity to reason to be reflected in the model’s internals. We want to identify if hidden states accurately reflect various reasoning-relevant aspects of the inputs. To gauge this, we develop and apply new interpretability methods and approaches. One approach makes controlled substitutions in model inputs to identify if certain properties are used. Two others use probes to find out if concepts and propositions are represented in a way that respects the (logical) relations between them. To enable further experimentation in this direction, we also improve an existing probing method. The second part of the thesis turns to toxic language, which communicates hateful, derogatory, or offensive ideas. We argue for a toxic reasoning approach, which focuses on characterizing the implications of the content. We propose a schema of relevant traits, which describes the toxicity that is expressed, and is compatible with existing taxonomies. We also construct the multilingual ToxiREX dataset of contextual Reddit comments annotated with this schema, and use it to evaluate language models for toxic reasoning. The work done in this thesis indicates that LLMs do exhibit properties and behaviors one would expect of a system capable of reasoning, but always with caveats: models solve tasks requiring reasoning only some of the time, and the relations between concepts and propositions appear to be reflected in their internal representations, but imperfectly. These results reflect the incomplete status of this line of research. For toxic language, previous approaches do not robustly solve the problem. For a problem that is constantly evolving, the toxic reasoning approach provides a promising direction, and the schema and dataset provide the infrastructure needed to further demonstrate its benefits.