Premium
A distributed parallel algorithm for inferring hierarchical groups from large‐scale text corpuses
Author(s) -
Seshadri Karthick,
S. Mercy Shalinie,
Manohar Sidharth
Publication year - 2017
Publication title -
concurrency and computation: practice and experience
Language(s) - English
Resource type - Journals
SCImago Journal Rank - 0.309
H-Index - 67
eISSN - 1532-0634
pISSN - 1532-0626
DOI - 10.1002/cpe.4404
Subject(s) - computer science , modularity (biology) , cluster analysis , key (lock) , exploit , scale (ratio) , memory footprint , algorithm , heuristic , hierarchical clustering , workstation , scheme (mathematics) , data mining , theoretical computer science , artificial intelligence , mathematics , mathematical analysis , genetics , physics , computer security , quantum mechanics , biology , operating system
Summary We propose a distributed parallel algorithm for inferring the hierarchical groups present in a large‐scale text corpus. The algorithm is designed to deal with corpuses that typically do not fit into the main memory of a workstation computer. The key contribution of this paper lies in its proposal and verification of a parallel distributed algorithm that exploits the advantages of two complementary techniques based on (i) localized modularity optimization and (ii) spectral clustering. Based on our experimental observations, these are complementary in the sense that the former excels at finding coarse groups in a large‐scale network, while the latter demands a heavy memory footprint but is effective in inferring tightly knit fine‐grained groups. Empirical evaluation of the distributed implementation scheme shows that the algorithm exhibits a significant speed‐up when compared to existing algorithms like Louvain and, at the same time, produces better quality clusters than either Louvain or spectral clustering algorithms in terms of the F‐score and Rand index.