Context-Aware Sequential Ranking for Dynamic Test Case Prioritization in Continuous Integration

Abstract

Regression testing in continuous integration (CI) helps preserve delivered software value by exposing regressions early, but limited feedback windows make test order consequential. This study evaluates a leakage-safe dynamic test case prioritization framework that reconstructs job–build–commit provenance, combines strictly prior-build history with change–test similarity, applies class-conditional conformal calibration, and evaluates an operator-level deep Q-network (DQN). The primary held-out evaluation contains 273 failure-bearing jobs and 45,774 job–test rows from four Java projects. Pretrained history–semantic fusion achieved an equal-project macro failing-test-entity APFD surrogate (FTE-APFD surrogate) of 0.8840, compared with 0.8689 for history-only and 0.7023 for pretrained semantic-only ranking; the fusion–history difference did not survive Holm correction. A train-only TF–IDF control reached 0.7502 for semantic-only ranking and 0.8871 when fused with history in a seed-averaged diagnostic, so the benchmark does not establish unique superiority of the pretrained representation. A protocol-aligned RETECS reimplementation achieved 0.8296 ± 0.0111 FTE-APFD surrogate across five seeds. Adaptive conformal calibration reduced the candidate fraction from 0.3171 to 0.2598 while retaining 0.9058 failing-test coverage. The candidate-informed gated DQN remained below strong deterministic fusion rankings at 10%, 25%, and 50% budgets; five-seed, short-budget, hyperparameter, pre-execution-budget, and project-exclusion sensitivities did not establish a consistent DQN advantage. An additional three-project operational analysis containing 444 passing test jobs showed high failing-row coverage but strongly project-dependent candidate burden. Overall, the evidence favors strong leakage-safe deterministic rankings with explicit uncertainty control, while the evaluated reinforcement-learning design remains a local negative result rather than a general conclusion about reinforcement learning.