LLM-Assisted Reviewer Assignment via Auditable Expertise Matching

Abstract

Accurate reviewer assignment is essential for scalable peer review, yet practical systems must balance semantic match quality with transparency, auditability, and manageable reviewer workloads. We study an end-to-end, LLM-assisted reviewer–paper matching pipeline in which large language models are used primarily for affinity construction by extracting and normalizing high-salience topical key phrases from submissions and reviewer publication summaries. We implement an agentic ensemble of heterogeneous producer models to generate candidate keyword sets, followed by a stronger judge agent that selects the best candidate for each input and representation, yielding compact, machine-parseable keyword profiles. Using a dataset of 663 submissions from 85 computer-science venues and 524 reviewer profiles, we evaluate three representations (titles, abstract summaries, and full abstracts) and compare lexical, neural, and graph-based affinity estimators. These include TF–IDF and Jaccard scoring (and their mean), Sentence-BERT similarity, keyword-aware retrieval with two-stage bi-encoder/cross-encoder re-ranking, and graph-based matchers over knowledge graphs and document-local graphs using Node2Vec embeddings with Jaccard-style fusion. We report MRR@3, Precision@3, and MAP@3. In addition, we simulate conference-level reviewer assignment to evaluate workload realism and inequality, summarizing reviewer loads both per-conference (macro) and globally (pooled), and quantifying inequality with the Gini coefficient. Results highlight consistent trade-offs across representations and matchers: two-stage re-ranking improves early-rank performance; keyword-aware Sentence-BERT increases top-3 concentration; and KG edge overlap is competitive on full abstracts, while some graph variants substantially concentrate reviewer workloads.