An Effective Knowledge-Based Pre-processing System with Emojis and Emoticons Handling on Twitter and Google+

Social networks, nowadays, are an imperative source for users to express their opinions freely on diverse topics including various brands and services. Presence of millions of users results in the generation of gigantic volume of data. The generated data are actually a treasure trove for business organizations to understand public sentiments towards their brand and services. This data is dynamic and highly unstructured due to varying writing patterns such as use of slangs, misspelled words, emojis and so on. The unstructured data makes pre-processing an underlying and challenging step in sentiment analysis. Therefore, authors have thoroughly explored a series of pre-processing steps on two social networks and observed that the sequence order of pre-processing steps plays an important role in improving overall pre-processing results. Hence, an improved ordered sequence of pre-processing steps has been proposed. It has also been observed that the presence of emojis in the text act as a pivot in determining users’ sentiments. Therefore, a detailed handling of emojis has also been included in the proposed pre-processing steps. New dictionaries have been compiled to provide a language to the emotional contents carried by emojis and emoticons. Few existing dictionaries have also been extended to make them more comprehensive for lookup task. Additional pre-procesing steps for handling multiword usernames and hashtags have also been incorporated in the proposed work. Further, experiments have been carried out to compare the proposed system with the existing ones. Results show that the proposed system outsmart the existing approaches mainly due to implementation of pre-processing steps in an ordered sequence and handling of emojis.