We have 3 of 66 papers
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
The results show that early abstract training can reliably shape the capabilities language models later acquire, and in small language models, such a warm-up improves specific capabilities that are not reflected in language-modeling perplexity.
This paper proposes conditioned initialization, a principled scheme that initializes attention weights to improve the spectral properties of the attention layer and shows that conditioned initialization can potentially reduce the condition number of the attention Jacobian, leading to more stable optimization.
The results show that standard transformers are rarely a local optimum in the space of architectures, and suggests that there may be room for improved architectures that better support multiple capabilities simultaneously, such as fluency and robust reasoning.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
