Large Scale Microblogging Intentions Analysis with Pattern Based Approach

In recent years, social networks have become very popular. Twitter, a micro-blogging service, is estimated to have about 200 million registered users and these users create approximately 65 million tweets a day. Twitter constitutes a powerful medium today that people use to express their thoughts and intentions. The challenge is that each tweet is limited in 140 characters, and is hence very short. It may contain slang and misspelled words. Thus, it is difficult to apply traditional NLP techniques which are designed for working with formal languages, into Twitter domain. Another challenge is that the total volume of tweets is extremely high, and it takes a long time to process. In this paper, we describe a large-scale distributed system for intentions analysis process based on lexico semantic patterns using Hadoop Distributed File System (HDFS) and MapReduce functions. We conduct a case study of user intentions in the commercial field. The proposed method has stably performed data gathering and data loading. Besides, it has maintained stable load balancing of memory and CPU resources during data processing by the HDFS system. The proposed MapReduce functions have effectively performed intentions analysis in the experiments. Finally, obtained results show the importance and effectiveness of intentions detection using semantic patterns