2026· Serbian Journal of Electrical Engineering· Vol 23, pp. 227-256· 0 citations· 10 references
TL;DR
BTU-Chatbot demonstrates that an Agentic RAG pipeline can deliver source-grounded, citation-attributed answers to university-specific queries at low operating cost, although further improvements in generation faithfulness are needed before full deployment.
Abstract
University regulations and announcements at Bursa Technical University (BTU) are dispersed across PDF files, web pages, and notice boards, causing information retrieval inefficiencies for students and administrative staff. This study presents the development and quantitative evaluation of BTU-Chatbot, an Agentic Retrieval-Augmented Generation (RAG) powered conversational assistant that consolidates fragmented institutional information into a single citation-aware dialogue interface. Ninety-seven PDF documents were processed using PyPDFLoader and regular-expression-based preprocessing, then embedded into 1536-dimensional vectors using OpenAI’s text-embedding-ada-v2 model and stored in a 29 MB ChromaDB collection. The vector-based retrieval layer selects the three most relevant passages per query using cosine similarity search. The upper layer implements a LangChain-orchestrated multi-agent ReAct loop, in which the retrieve tool accesses the vector database while the Google_search_univ tool performs domain-restricted searches limited to the “*.btu.edu.tr” domain. GPT-4o-mini, with a 128k context window, serves as the generative backbone. System reliability was measured using the RAGAS metric suite. The best performing run achieved Context Recall = 0.97, Context Precision = 0.99, and F1- RP (the F1 score of Context Recall and Precision) = 0.954, demonstrating nearperfect retrieval accuracy. The average cost per query was 6.6×10⁻⁵ USD, with 7.6 s of latency for typical 124-token exchanges. BTU-Chatbot demonstrates that an Agentic RAG pipeline can deliver source-grounded, citation-attributed answers to university-specific queries at low operating cost, although further improvements in generation faithfulness are needed before full deployment.
Conversational recommendation for e-commerce is increasingly mediated by large language models (LLMs), yet many real-world deployments operate under a stricter requirement: recommendations must be drawn only from a merchant's fixed catalog, without web search or unsupported product claims. In this setting, the main challenge is reliability under hard constraints: the system must satisfy user requirements, remain grounded in available inventory, and preserve preferences across multiple conversational turns. We present MACS (Multi-Agent Commerce System), a hybrid multi-agent framework for reliable conversational recommendation in fixed-catalog settings. MACS uses LLMs for language-facing tasks such as interpreting user requests, eliciting preferences, and generating responses, while correctness-critical operations, including product retrieval, hard-constraint filtering, brand exclusion, and progressive relaxation, are executed deterministically by the merchant agent. A session-persistent preference layer tracks constraints across turns, enabling consistent handling of budget overwrites and exclusion reversals. On a 140-query single-turn benchmark, MACS achieves the highest pass rate (87.1%) and perfect brand compliance (1.000). On a 10-scenario multi-turn benchmark, MACS achieves the strongest macro Pass@5 (72% vs. 56% GPT+Catalog / 52% Gemini+Catalog) with zero constraint drift. The advantage is sharpest on exclusion reversal (100% vs. 20% / 0%) and constraint accumulation (100% vs. 60% / 40%). Mean judged response quality is similar across systems (0.751 vs. 0.736). These results suggest that hybrid architectures combining deterministic constraint enforcement with session-persistent preference tracking provide stronger reliability-oriented performance than catalog-bound prompt-only baselines in the fixed-catalog merchant setting.
Juli Huang, Hanna Clay, Sajjad Beygi et al.· 0 citations
This work presents SimpleWikiSearch, whose corpus construction, retrieval stack, tool contract, and evaluation protocol are explicit and runnable and provides a domain-specific agent harness and a controlled offline environment for reproducible agentic-search evaluation.
Scientific user facilities accumulate decades of operational knowledge that no single search index covers: electronic logbooks, technical documents, internal wikis, operations chat messages, maintenance records, and live control-system data. We present APS-RAG, Advanced Photon Source Retrieval Augmented Generation, a deployed platform that makes the institutional knowledge at the Advanced Photon Source (APS) accessible to staff through natural-language queries, along with an operations-grounded evaluation. The retrieval engine fuses dense, sparse, and knowledge-graph (KG) channels with query-type-adaptive reciprocal-rank fusion, adds a corrective agentic loop, and runs a native-tool ReAct executor over a Model Context Protocol (MCP) tooling layer. We construct APS-Bench, a 50-question, question-answering (QA) dataset with auditable gold answers. Every retrieval-augmented variant numerically improves strict vital-nugget recall over a naive BM25 baseline (63.8%), with the full corrective Agentic GraphRAG scoring (70.3%). The cross-encoder reranker contributes significantly to answer quality: removing it and allowing the LLM to score relevance drastically reduces strict vital recall by 32.8%. The graph channel and corrective loop contribute positively as expected, but the performance gains are marginal. Additionally, we also compare the performance of open-source and closed-source LLMs in final answer synthesis. We release the APS-Bench construction methodology, the six-layer evaluation harness, and the underlying codebase, along with the'/aps-rag'retrieval agent skill framework, to support reproduction and adoption at other facilities. Together, the deployed platform and its operations-grounded evaluation present a promising workflow for trustworthy, statistically grounded AI assistance in facility operations, transferable to other large scientific instruments.
Rajat Sainju, Dariusz Jarosz, Hairong Shang et al.· 0 citations
Readers of online news often lack the time and domain expertise required to verify unfamiliar claims and sources. Professional fact-checkers address this gap through lateral reading, an iterative workflow of asking investigative questions, searching for external evidence, and synthesizing findings with attribution. We present an iterative multi-agent Retrieval-Augmented Generation (RAG) system that operationalizes this workflow for the TREC 2025 DRAGUN Track. Given a news article, specialized agents (1) generate investigative queries, (2) retrieve and filter evidence from the MS MARCO V2.1 Segmented Corpus using a three-stage retriever (BM25+RM3, cross-encoder reranking, and LLM-based selection), and (3) apply an information-sufficiency evaluator that decides whether additional searching is required before writing. The final report generator produces a 250-word trustworthiness report grounded in retrieved segments, guided by automatically generated critical investigative questions. On the official DRAGUN rubric-based evaluation with 30 news articles, our system using GPT-4.1 ranked first on report generation quality, achieving the highest mean supportive score (0.230) with low contradiction (0.013).
Dake Zhang, Mark D. Smucker· Annual International ACM SIG...· 1 citation
Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.
Chunming Wu, Dafei Qiu, Congde Yuan et al.· 0 citations
This work introduces SearchOS, a system-level multi-agent framework that turns fragile, implicit search progress into explicit, persistent, and shared state, and introduces a Search Tool Middleware Harness that intercepts model and tool interactions to record grounded evidence and react to stalls or budget exhaustion.
Yuyao Zhang, Junjie Gao, Zhengxian Wu et al.· 0 citations