Vector Database Management System for Similarity Search on Commodity Hardware

Abstract

Vector database management systems support semantic search, recommendation, and retrieval-augmented generation, but many available deployments assume cloud infrastructure, specialist accelerators, or server-scale memory. This creates a practical problem for students, researchers, and small development teams who need reproducible vector search on ordinary workstations with limited memory and no GPU support. This study developed and evaluated a lightweight vector database management system for similarity search on commodity hardware. The system integrated HDF5-based persistent vector storage, FAISS IndexIVFPQ approximate nearest neighbour indexing, a modular query-processing pipeline, and correctness validation against exhaustive FAISS IndexFlatL2 ground truth. Evaluation used pre-trained GloVe embeddings with 50, 100, 200, and 300 dimensions, covering dataset sizes between 10,000 and 200,000 vectors. The evaluation reports retrieval accuracy, index memory footprint, query latency, and correctness validation. Across all datasets, the system achieved an average recall@10 of 0.8978, with all configurations exceeding the 0.85 retrieval target and recall@1 reaching 1.0000. IndexIVFPQ memory footprint remained between 0.37 MB and 30.50 MB, giving 5.2x to 7.5x compression relative to IndexFlatL2. Query latency ranged from 0.102 ms to 14.558 ms, with three of the four configurations remaining less than 10 ms. Correctness validation on the 100d and 300d benchmark configurations produced recall@10 values greater than 0.90 against exact search. The results indicate that an integrated HDF5 and FAISS IndexIVFPQ architecture provides a reproducible local vector search workflow on commodity hardware, although larger and higher-dimensional workloads require further tuning of index construction and query latency.