Reliability-aware ensemble recommendation for open-network commerce: a dual-metric framework for jointly evaluating ranking accuracy and serving correctness

Abstract

Recommendation quality in open-network commerce is conventionally measured through offline ranking metrics that implicitly assume reliable upstream retrieval, stable item identities, and consistent catalog availability. In open networks such as India's Open Network for Digital Commerce (ONDC), these assumptions are routinely violated: catalogs are contributed by heterogeneous, independently operated providers, transactions follow asynchronous Beckn-style protocol lifecycles, and identifiers drift across service boundaries. This paper presents LocalMarket, a reliability-aware recommendation platform (Next.js frontend, FastAPI backend, ensemble ranking layer) that implements ONDC-pattern transaction flows in a controlled, repository-reproducible environment. We propose and validate a dual-metric evaluation framework that decouples model effectiveness—measured through Mean Average Precision (MAP@10), Recall@10 and Normalized Discounted Cumulative Gain (NDCG@10)—from serving correctness, measured through a novel Route Resolution Success Rate (RRSR) and Wishlist Consistency Rate (WCR). Using a controlled within-system ablation across Model-only, Reliability-only, and Combined configurations on fixed traffic traces, we utilize a convex-combination ensemble containing neural collaborative filtering (NCF) (utilizing only GMF embeddings for retrieval latency), graph-based, session-aware, and content-based components primarily as a vehicle for testing this framework. While this ensemble configuration improves NDCG@10 by 70.7% over a Popularity-based baseline, our primary finding validates that four targeted reliability interventions raise RRSR from 60.0 to 99.8% and WCR from 0.0 to 100.0%. Critically, the combined configuration preserves both gains simultaneously, indicating that model-level and serving-level improvements are additive rather than substitutive. The results support a systems-level thesis: in open-network commerce recommenders, ranking accuracy and serving correctness are independent, jointly necessary axes of quality, and reporting only one materially overstates the value delivered to end users. We further contribute a reusable integration failure-mode taxonomy, a constrained ensemble-weight selection protocol, and a reproducible experiment-replay procedure intended to generalize to other open-network commerce deployments.