Higher-Dimensional Rotary Position Embedding
HDR-RoPE is proposed, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace and significantly enhances channel coupling and rotational degrees of freedom while maintai...