Skip to content
Book Open access

Diagnosing and Correcting Geometric Bias in Diversity-Driven Code Instruction Data Selection

Nov 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · pp. 6219-6230 · 0 citations · 4 references

Abstract

Diversity-driven data selection offers a scalable way to reduce code instruction-tuning costs, yet it relies on a fragile assumption: geometric spread in representation space should correspond to training-useful semantic coverage. In code data selection, our diagnosis reveals a geometric bias behind this assumption: dense embedding geometry can be strongly influenced by rare surface forms, verbose constraints, and implementation artifacts, causing diversity objectives to over-explore locally sparse or globally peripheral samples whose training utility is not guaranteed. To correct this bias, we propose Semantically Anchored Geometric Coverage (SAGC), a framework for reshaping representation geometry using LLM-generated macro-intent labels as semantic anchors for dense micro-context representations. Rather than replacing standard diversity objectives, SAGC reshapes the geometry exposed to them, using macro-intent anchors to constrain dense-space exploration toward high-level algorithmic structure without requiring downstream validation signals, gradient feedback, or evaluation results. Using only 20k samples selected from a 446k pool, SAGC reaches 0.4006 on LiveCodeBench, approaching full-data training at 0.4027. Further diagnostics and out-of-benchmark evaluations show that semantic anchoring provides a structured correction to this geometric bias.

Read PDF

Similar papers

Open access Sep 2026

Quality over Quantity: Diversity-Aware Data Selection for Efficient Verilog Code Generation

VeriSelector is proposed, the first data selection framework for Verilog code generation that jointly optimizes quality and diversity, and outperforms full-dataset training and state-of-the-art baselines.

Yi-Heng Shen, Wei Zheng, Xiao Wei et al. · 0 citations
Book Open access 2027

Selecting Memory Layouts for Model-Based Microarchitectural Performance Evaluation

Virtual memory research increasingly relies on empirical performance models that use simulated address translation metrics (x)---such as TLB miss rate and page-walk cycles---to predict benchmark runtimes (y) for new hardware designs. The accuracy of these models depends on the training data: a set of (x,y) pairs collec...

Mohammad Agbarya, Idan Yaniv, Dan Tsafrir · 0 citations

Hardware-Aware Features for CUTLASS Kernel Selection

A hardware-aware representation for CUTLASS kernel selection is introduced that augments candidate configurations with statically computable estimates of induced hardware behavior, showing that explicitly representing candidate-induced hardware behavior provides a useful inductive bias for learned kernel selection.

Shriram Chandran, Dominic Rinderer, Yakup Budanaz et al. · 0 citations
Preprint Oct 2026

UltraData-Code: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data

Code generation has become a core capability of large language models (LLMs), and code data is central to the pre-training process that develops it. As obtaining public code of sufficient quality becomes more difficult, further progress cannot rely on corpus expansion alone. Tiered data management instead applies incre...

Cheng-Ying Tu, Heng-Yu Zhao, Shuai-Kang Xue et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Follow the Gradient: Tracing Safety Degradation Back to Training Data

Fine-tuning Large Language Models (LLMs) on instruction data can quietly erode their safety, even when no training example looks harmful. Because this safety-degrading data looks benign, standard content filters struggle to catch it. We tackle this problem by developing SimpleLoRIF, a scalable gradient embedding method...

S. Reddy, Amro Abbas, Anshuman Suri et al. · 0 citations
Book Open access Oct 2026

OrthCoder: Balancing Security and Utility in Secure Code Generation via Orthogonal Adaptation

Large Language Models (LLMs) have significantly advanced automated code generation, yet they frequently produce code with critical security vulnerabilities. While recent alignment techniques (e.g., supervised fine-tuning and preference optimization) have attempted to mitigate this, existing methods often suffer from th...

Meng Tian, Youliang Yuan, Zhiqing Zhong et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.