PUFFER: Incremental Fuzzy Deduplication for Continuously Evolving Corpora
This work introduces PUFFER (Provenance-aware Updatable Fuzzy Filtering for Evolving Repositories), a MinHash-LSH fuzzy-deduplication pipeline built around two design choices, which was 11x faster than LSHBloom and 35x faster than Milvus-LSH.
Xiao Yang, Erik Edward Aldape, Beren Millidge
· 0 citations