This work proposes a novel LAL method for classification that exploits symmetry and independence properties of the active learning problem with an Attentive Conditional Neural Process model and gives the model the ability to adapt to non-standard objectives.
We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we prop...
Assaf Ben-Kish, Akarsh Kumar, James R. Glass et al.· 0 citations
Continual learning requires a model to retain knowledge of old tasks while sequentially learning new tasks, but standard neural networks typically suffer from catastrophic forgetting in this setting. To address this challenge, a new method is proposed based on a learnable knowledge transfer network. Specifically, a tra...
Han Ju· International Conference on...· 0 citations
A central challenge in continual learning is to acquire new knowledge without forgetting what the model has already learned. This challenge appears in language model training when training data comes from various domain-, user-, or task-specific distributions that are encountered unevenly over time. In such settings, s...
Anton Baumann, Jonas Hübotter, Zeynep Akata et al.· 0 citations
Combining predictions from different models can improve performance at machine learning tasks, but the training of the individual models and the rule used to combine them are typically chosen separately, and by ad hoc means. Recent advances in distributional optimisation (i.e. where the optimisation occurs over the set...
Cong-Ye Wang, Yan-Kai Lin, Zhe-Yang Shen et al.· 0 citations
Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and reinforcement learning (RL) to this end. Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is...
The dominance of Neural Networks (NNs) in RL is partially due to their incremental learning capability, which naturally suits the online, non-stationary nature of self-play training. However, gradient-boosted trees like LightGBM are widely recognised as the state of the art for tabular data in supervised learning, ofte...
David Milec, Spyridon Samothrakis, Michael Fairbank et al.· IEEE Transactions on Games· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
