Executive Summary This work is a literature review on the task of closed-domain event extraction, a task within natural language processing (NLP) where the goal is to detect the presence of an event (an occurrence of an action or state change) and event-related information (known as arguments) and to return the event structure. In our review, we focus on recent state-of-the-art (SOTA) approaches which can be applied to unstructured English-language text. After the introduction, we begin by providing an overview of the terminology of closed-domain event extraction, including a breakdown of the constituent sub-tasks. Next, we discuss the dataset requirements for this task, and note the core datasets used in the literature. We begin our review of the literature by providing a typology of event extraction approaches based on the common differences between extant approaches. We provide a table summarising the main results in the literature and discuss standards of evaluation in the literature. We found that all reviewed SOTA approaches utilise a PLM, with the choice of model often dependent on whether a classification or generative approach is taken-no single approach is dominant. BERT and BART are popular choices of PLM in the literature. We argue that evaluation standards have been inconsistent, resulting in results which are difficult to compare. We then describe some of the details of the most performant models in our table. Finally, we discuss some of the main themes we have identified in our review. In particular, we outline some clear issues in the extant literature. We argue that there is a need for a new, open source dataset to act as the primary benchmark to facilitate more academic research, with high inter-annotator agreement (IAA) ensuring that good performance on the benchmark is meaningful. We also call for standardisation around pre-processing and evaluation of results to ensure the comparability of results. We note that low-resource performance has typically been under-explored. In our recommendations for future work, we argue that there is clear potential to assess the performance of modern, causal large language models (LLMs) such as the Llama or GPT model families. We argue that it there is clear potential to explore both in-context learning and fine-tuned approaches with these models. We also argue that these models have clear potential applications in the generation of synthetic data. We conclude by summarising our main points. 1
Joanna Cameron Knight, Phil Swatton, Alex Hickey et al.· Alan Turing Institute Resear...· 0 citations
Executive Summary This work approaches the problem of selectively forgetting knowledge from a large language model (LLM) for the purposes of safety, copyright, security, or otherwise. Also known as machine unlearning, this entails training a model to forget certain elements of the dataset on which it was trained. Unlearning methods must be evaluated both in terms of the extent to which the information has successfully been forgotten, and the performance of the unlearned model on the remaining (retained) data. We build on the work of TOFU (Task of Fictitious Unlearning) [21], which provides a dataset and benchmark for evaluating unlearning techniques. We create a new, TOFU-inspired question–answer dataset for the task of machine unlearning. The new dataset includes 10,500 question–answer pairs relating to over 1,000 distinct, synthetic entities of several types. Each question–answer pair is tagged with the entities it refers to, with the graph representation of our dataset containing over 2,600 edges between different entities. We perform two experiments with our dataset. The first experiment aims to capture whether the difficulty of forgetting a concept from a LLM depends on its granularity. For example, is unlearning more likely to be successful if forgetting a single book, rather than the book's author (as an author is connected to multiple books)? We find that granularity does not have a tangible effect on model performance in our dataset. There may be a small effect from granularity on the difficulty of forgetting, but this is not statistically significant across our results. Our second experiment explores the knock-on effect of forgetting a relationship between two entities. For example, if unlearning has been run on a model to forget only who wrote a book, but not the book or author themselves, is the model worse at responding to other questions about that book or author? We find that the model performance is lower on questions that contain the entities pertained in the relationship, than on those that do not. 1
Jack Dymond, Phil Swatton, Jack Roberts et al.· Alan Turing Institute Resear...· 0 citations
Executive Summary This work approaches the problem of selectively forgetting knowledge from a large language model (LLM) for the purposes of safety, copyright, security, or otherwise. Also known as machine unlearning, this entails training a model to forget certain elements of the dataset on which it was trained. Unlearning methods must be evaluated both in terms of the extent to which the information has successfully been forgotten, and the performance of the unlearned model on the remaining (retained) data. We build on the work of TOFU (Task of Fictitious Unlearning) [21], which provides a dataset and benchmark for evaluating unlearning techniques. We create a new, TOFU-inspired question–answer dataset for the task of machine unlearning. The new dataset includes 10,500 question–answer pairs relating to over 1,000 distinct, synthetic entities of several types. Each question–answer pair is tagged with the entities it refers to, with the graph representation of our dataset containing over 2,600 edges between different entities. We perform two experiments with our dataset. The first experiment aims to capture whether the difficulty of forgetting a concept from a LLM depends on its granularity. For example, is unlearning more likely to be successful if forgetting a single book, rather than the book's author (as an author is connected to multiple books)? We find that granularity does not have a tangible effect on model performance in our dataset. There may be a small effect from granularity on the difficulty of forgetting, but this is not statistically significant across our results. Our second experiment explores the knock-on effect of forgetting a relationship between two entities. For example, if unlearning has been run on a model to forget only who wrote a book, but not the book or author themselves, is the model worse at responding to other questions about that book or author? We find that the model performance is lower on questions that contain the entities pertained in the relationship, than on those that do not. 1
Jack Dymond, Phil Swatton, Jack Roberts et al.· Alan Turing Institute Resear...· 0 citations
