English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Tools annual

Global: Zendy Free Plan

Global: Zendy Tools monthly

Global: Zendy Plus monthly

This article improves the algorithm of term weighting in automated text classification. The traditional TFIDF algorithm is a common method that is used to measure term weighting in text classification.However, the algorithm does not take the distribution of terms in inter-class. In order to solve the problem, variance which describes the distribution of terms in inter-class and intra-class is used to revise TFIDF algorithm. This article mainly researched about the construction of LFHW term sets and new approaches to term weighting, These new approaches are also applied to the hierarchical classification system.Compared with traditional TFIDF algorithm ,the results of simulation experiment have demonstrated that the improved TFIDF algorithm can get better classification results. Introduction Text classification refers to determine a category for the text of each set of documents according to the predefined topic categories, Human knowledge has great growth, artificial knowledge classification has been difficult to adapt, so the technology of text classification is paid attention by the academic and industrial circles. With a simple and accurate method for the document representation into a form that the computer can process is based for the text classification, the classic text representation method is the vector space model.feature weighting is the core algorithm for Vector space model, feature weighting calculation plays a crucial role on text categorization effect. Based on the analysis of traditional feature extraction algorithm TFIDF, variance which describes the distribution of feature terms in inter-class and intra-class is used to describe the distributions of the feature terms inter-class and intra-class, finally the application of VSM algorithm is used for automatic text classification, the experimental results show that the improved TFIDF algorithm has better performance in text classification. The traditional TFIDF The traditional TFIDF algorithm method is proposed by Gerald Salton and McGill for document feature representation in the light of vector space information retrieval paradigm. In this method, the words appear in a document is called the feature words (Term), each term has a corresponding weighting, the weighting of words manifest important degree in document recognition, term weighting and term appear frequency in the document is proportional, with the frequency of term in all the documents is inversely proportional. TFIDF is: TF ×IDF. The TF (Term Frequency) is called the word frequency, refers to the frequency of terms in the given document, if exclusion of forbidden and individual high-frequency words, the more frequency of terms in the given document , the stronger ability of terms for the document characterization; IDF (Inverse Document Frequency) is called the inverse document frequency, reflects the appear frequently of a term in a document set by document statistics, The smaller the number that document contain terms in document set, the more representative terms [1]. IDF(ti)=log(N(C)/N(ti,C)) (1) Where N ( C ) for all the documents, N (ti, C ) for the documents that contain the terms.TFIDF algorithm mainly has 2 features as the following: Proceedings of the 2nd International Conference On Systems Engineering and Modeling (ICSEM-13) Published by Atlantis Press, Paris, France. © the authors, 2013 0765 Inverse document frequency is purpose that the smaller the number that document contain terms is, the more representative terms is. N(C) for the total number of documents in the training set, without taking increase the number of documents in the same class into consideration. If the term ti appears frequently in the document of the same class, the term can be a good representative of this class, this term should be given a higher weighting. In formula (1), when the number N(ti, C) of document that contains term ti in class Ci increases, but IDF decreases. Therefore, such a kind of representative terms that appeared in the same class in while less appearance in other, the weighting in TFIDF algorithm not only has been strengthened, but weaken. For the two different terms t1 ,t2 in the same class .the frequency of t1 in each document is more average than t2, so t1 is more representative than t2 in the ability of characterization of classes. If t2 appears great quantities in one or two document of the class, whereas appears rarely in the other document of the class, do not eliminate that the one or two document is the special case in the class, this term is not representative, the weights to be lower, but TFIDF algorithm can not distinguish for which. Improvement feature weighting algorithm Based on the analysis of the section second, in addition to frequency, weighting is closely related to the following two kinds of terms: (1) The less the classes of terms distribution is and the more number the texts of the same class has, the stronger ability of terms distinguishing is, and also the bigger weighting. (2) The more average the terms in each document of the class is, the more representative the terms has, the bigger weights would be given.According to this idea proposes, put forward improvement of feature weighting algorithm TFIDF-De-Di: WeightTFIDF(tij)=TF(tij)×IDF(tj)×De×(1-Di) (2) Where, TF(tij) for word frequency, IDF(tj) for inverse document frequency, De for dispersion coefficient of inter-class, Di for dispersion coefficient of intra-class [2]. This article will look term ti as a random variable, the value of ti in inter-class will be representation by the frequency of ti in inter-class. By the definition of the variance, the distribution variance D(ti) of ti in inter-class reflects the degree of dispersion of ti in inter-class, the less D(ti) is, the more uniform the distribution of ti in inter-class is. That is if the term ti more evenly distributed in each class, D(ti) is smaller, the contribution to the classification of term ti is smaller, if, the term is uniformly distributed in inter-class, then D(ti) is 0, no contribution to classification. Using this feature of variance, this paper uses D(ti) to modified TFIDF formula, just can make up for the TFIDF that did not consider the distribution of term in inter-class. From the above analysis, the value of ti in inter-class will be representation by the frequency of ti in inter-class, but the distribution probability of term ti in inter-class is very difficult to calculate, it is more difficult to calculate the variance, so this use the average variance of term ti approximation instead of D(ti). Set up a total of N class, TF(ti) representing the occurrence frequency of the term in class Ci, ) ( i t TF represents the average frequency of term in each class, the calculation formula is:  = = n i i i t TF n t TF 1 ) ( 1 ) ( . If square of average variance of t1 in inter-class for the De, The formula of square of average variance of t1 is

Improved term selection algorithm based on variance in text categorization