FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models
FactorEngram, a factorized n-gram memory with basis-level contextual gating that improves language modeling and downstream task performance and retrieves sparsity-regularized coefficients over a dictionary of basis vectors shared across patterns, so related patterns can reuse common components.