A Scalable Clustering-Based Approach for Efficient Mining of Large Datasets

Abstract

Clustering is an unsupervised machine learning technique used to identify and group similar data points within large datasets without relying on predefined class labels or target outcomes. It plays an important role in data mining by facilitating the discovery of meaningful groups, hidden patterns, and relevant data distributions. Traditional clustering methods generally assume that clusters are spherical and have relatively similar sizes, which can reduce their effectiveness when dealing with irregularly shaped clusters or datasets containing outliers. To address the challenges associated with clustering large databases, several approaches employ random sampling and partitioning techniques. In such methods, a large dataset is first divided into smaller partitions, and each partition is clustered independently. The resulting partial clusters are subsequently combined and clustered again to generate the final set of clusters. This approach reduces the computational burden associated with processing the entire dataset simultaneously. With the rapid growth of data volumes, several parallel clustering techniques based on the Map Reduce framework have also been proposed to improve scalability. However, applying the conventional K-Means algorithm to large datasets in a parallel environment may require the repeated reading and processing of data during each iteration. Consequently, this iterative process can significantly increase input/output operations and network communication overhead. To overcome these limitations, a series-based K-Means clustering approach, referred to as CBK Means, can be employed to reduce the amount of data processed while maintaining or improving clustering quality. The method uses representative data verification to effectively reduce the dataset size before or during the clustering process. Experimental results indicate that CBK Means provides improved efficiency and scalability compared with the conventional K-Means algorithm, while also achieving better clustering accuracy.