Open AccessAutomatic Threshold Selections by exploration and exploitation of optimization algorithm in Record Deduplication
Author(s) -
K. Deepa,
C. Vivek,
S. Palanivel Rajan
Publication year - 2016
Publication title -
journal of advances in chemistry
Language(s) - English
Resource type - Journals
ISSN - 2321-807X
DOI - 10.24297/jac.v12i11.820
Subject(s) - data deduplication , computer science , fitness function , algorithm , swarm intelligence , particle swarm optimization , genetic algorithm , data mining , similarity (geometry) , function (biology) , process (computing) , artificial intelligence , machine learning , database , evolutionary biology , image (mathematics) , biology , operating system
A deduplication process uses similarity function to identify the two entries are duplicate or not by setting the threshold. This threshold setting is an important issue to achieve more accuracy and it relies more on human intervention. Swarm Intelligence algorithm such as PSO and ABC have been used for automatic detection of threshold to find the duplicate records. Though the algorithms performed well there is still an insufficiency regarding the solution search equation, which is used to generate new candidate solutions based on the information of previous solutions. The proposed work addressed two problems: first to find the optimal equation using Genetic Algorithm(GA) and next it adopts an modified Artificial Bee Colony (ABC) to get the optimal threshold to detect the duplicate records more accurately and also it reduces human intervention. CORA dataset is considered to analyze the proposed algorithm.