Text Similarity Measurement of Semantic Cognition Based on Word Vector Distance Decentralization With Clustering Analysis | Zendy

Shenghan Zhou | Zendy; Xingxing Xu | Zendy; Yinglai Liu | Zendy; Runfeng Chang | Zendy; Yiyong Xiao | Zendy

Open Access

Text Similarity Measurement of Semantic Cognition Based on Word Vector Distance Decentralization With Clustering Analysis

Author(s) -

Shenghan Zhou,

Xingxing Xu,

Yinglai Liu,

Runfeng Chang,

Yiyong Xiao

Publication year - 2019

Publication title -

ieee access

Language(s) - English

Resource type - Journals

SCImago Journal Rank - 0.587

H-Index - 127

ISSN - 2169-3536

DOI - 10.1109/access.2019.2932334

Subject(s) - computer science , cluster analysis , artificial intelligence , semantic similarity , similarity (geometry) , natural language processing , sentence , word (group theory) , similarity measure , support vector machine , information retrieval , data mining , image (mathematics) , philosophy , linguistics

Text similarity measurement, which is a basic task in natural language processing, is widely used in text information mining, news classification and clustering, artificial intelligence, and other fields. This paper proposes a text similarity measure method named word vector distance decentralization (WVDD) which can deal with complex semantic relations, including sentence components, word order and weights for Chinese language. Then, the clustering analysis is performed for the obtained similarity results. A K-means algorithm based on Spark architecture for parallel computing is adopted to accelerate clustering speed here. In experimental verification, the test sets are significant number of customer comments posted on the Jingdong website, which is a comprehensive online shopping mall. F-measure is used to evaluate the accuracy of the results obtained by the proposed method. The superiority of the proposed method is verified and compared with the sentence vector model (Doc2vec) and bag-of-words model. The proposed method can be applied to analyze network language, such as customers’ comments online and web chat data.

The content you want is available to Zendy users.

Already have an account? Click here to sign in.

Having issues? You can contact us here

Accelerating Research