Field of Invention
The present invention relates to a technical field of information retrieval, and more particularly to a clustering method for multilingual documents.
Description of Related Arts
When accessing to the internet, users often search concerned information on a search engine. Information retrieval systems which are similar to the search engine usually filter and search bulk data, and processing time is required to be fast enough for providing the users a timely response, in such a manner that waiting of the users is avoided.
The clustering technique in the information retrieval system guarantees a searching time fast enough for providing the users sufficient information. Clustering, which refers to categorizing information in the information retrieval system, is an effective improvement strategy in the information retrieval system and capable of providing more complete information for users. Applying clustering technique in information retrieval enables the users to quickly locate contents they interested in during processes of information retrieval. Compared with information retrieval systems without applying the clustering technique, information retrieval systems allying the clustering technique has an effect of reducing waiting time of the users and has characteristics of clearer classification.
Accordingly, in order to solve technical problems mentioned above, the present invention provides a clustering method for multilingual documents which is capable of fusing the multilingual documents.
Technical solutions for solving the technical problems mentioned above are as follows. A clustering method for multilingual documents, comprising following steps of:
step 1: establishing a similar words bank comprising multilingual words;
step 2: extracting eight eigenvalues;
step 3: calculating a similarity of any two documents i and j according to the eight eigenvalues;
step 4: selecting accumulation points from a set of the documents to establish a cluster;
step 5: adding residual documents which are not selected in the set to the cluster; and
step 6: disposing the cluster in a circular ring structure.
Preferably, wherein in the step 1, multilingual words having identical or similar meanings are recorded in each line of the similar words bank, and whether the multilingual words are verbs or nouns is marked.
Preferably, in the step 2, the eight eigenvalues comprise: an eigenvalue of citation relationships (f1), an eigenvalue of identical references (f2), an eigenvalue of identical strings (f3), an eigenvalue of similar strings (f4), an eigenvalue of identical nouns (f5), an eigenvalue of similar nouns (f6), an eigenvalue of identical verbs (f7), and an eigenvalue of similar verbs (f8);
wherein the eight eigenvalues are not limited to a particular language, and the multilingual documents are fused in classification of the clusters;
wherein citation documents refer to references listed in a document;
the identical strings refer to strings formed by a section of identical words;
the similar strings refer to strings having a section of identical words or formed by a section of similar words recorded in the similar words bank;
the identical nouns refer to absolutely identical nouns;
the similar nouns refer to nouns recorded in a same line of the similar words bank;
the identical verbs refer to absolutely identical verbs; and
the similar verbs refer to verbs recorded in a same line of the similar words bank;
wherein for a document i, an eigenvector thereof is F(i),
F(i)=(f1(i),f2(i),f3(i),f4(i),f5(i),f6(i),f7(i),f8(i).
Preferably, in the step 3, importance of the eight eigenvalue is f1>2>f3>f4>f5>f6>f7>f8;
wherein the step 3 specifically comprises a step of calculating products of eigenvalues of any two documents i and j, wherein the step of calculating the products comprises:
calculating a product of citation documents f1(i)fi(j), wherein W is defined as a weight of one document in i and j cited by the other document in i and j;
bool represents that whether a citation relationship exists, wherein a value of bool is 0 or 1, the value 0 represents that the citation relationship does not exist, and the value 1 represents that the citation relationship exists; wherein a calculating expression is:
f
1(i)f1(j)=bool×W;
calculating a product of the identical references f2(i)f2(j), wherein d is defined as a weighting factor of division and d≧1;
Refs represents a number of the references;
Max{Refs(i),Refs(j)} represents a maximum of the number of the references selected from i and j;
CommonRefs(i,j) represents a number of identical references in the two documents of i and j, and a calculating expression is:
calculating a product of the identical strings f3(i)f3(j), wherein CommonStrs(i,j) is defined as identical strings in the two documents i and j; Length represents a length of the strings, and thus Length(CommonStrs(i,j)) represents a total length of the identical strings, Max{Length(i),Length(j)} represents a maximum of a total length of the two documents i and j; and a calculating expression is:
calculating a product of the similar strings f4(i)f4(j), wherein SimilarStrs(i,j) is defined as similar strings in the two documents i and j, and a calculating expression is:
calculating a product of the identical nouns f5(i)f5(i), CommonNouns(i,j) is defined as identical nouns in the two documents i and j; Nouns represents a total number of nouns in the documents, and thus Max{Nouns(i), Nouns(j)} represents a maximum of the total number of the nouns in the two documents i and j, and a calculating expression is:
calculating a product of the similar nouns f6(i)f6(j), wherein SimilarNouns(i,j) is defined as nouns having similar meanings in the two documents i and j, and a calculating expression is:
calculating a product of the identical verbs, wherein CommonVerbs(i,j) is defined as identical verbs in the two documents i and j, Verbs represents a total number of verbs in the documents, and thus Max{Verbs(i),Verbs(j)} represents a maximum of the total number of the nouns in the two documents i and j, and a calculating expression is:
and
calculating a product of the similar verbs f8(i)f8(j), SimilarVerbs(i,j) is defined as verbs having similar meanings in the two documents i and j, and a calculating expression is:
based on calculations of products of the eigenvalues, a similarity of the two documents i and j is defined as:
Preferably, in the step 4, on an initial condition, two most dissimilar documents, i.e., with a minimum Proximity(i,j), are selected for serving as two initial accumulation points p1 and p2, p1 and p2 are added to an accumulation point set denoted as Points; residual accumulation points are selected according to a following maximum and minimum formula:
wherein in the formula, pr, r=1, 2, . . . , m represents documents selected as the accumulation points, then an (m+1)th accumulation point is selected from documents which haven't been selected as the accumulation points and added to the set Points, a threshold value Th is set for the formula mentioned above; when a stopping accumulation point selected satisfies
the accumulation points are stopped selecting; in addition, the stopping accumulation point is not added to the set Points.
Preferably, in the step 5, N represents a total number of documents participating in clustering, M represents a total number of accumulation points selected;
in the beginning, M documents serve as accumulation points of the clustering, residual N−M documents are added in the M clusters;
Cluster(pr), r=1, 2, . . . , M represents a set of each cluster;
in the beginning, each set only has one documents serving as the accumulation points;
for a document i not participating in the clusters, a most similar cluster is calculated according to a following expression:
in the expression mentioned above, a similarity of between a document i not added in the clusters and all documents in the set Cluster(pr) of each cluster, an average is taken for serving as a similarity of the document i and the clusters; a maximum of all the clusters is taken for serving as a most similar cluster to the document i;
the residual N−M documents are added to the set of the clusters, each time a document iq having a maximum similarity is added to the set of the clusters, and the Cluster(pq) is updated, and finally all the documents are added to the set of the clusters.
Preferably, in the step 6, M clusters are disposed in the circular ring structure, in such a manner that clusters having more similar characteristics are distributed closer, and clusters having more dissimilar characteristics are distributed farther; wherein in an initial condition, two clusters are randomly selected to be added to the circular ring structure, and residual M−2 clusters are added to the circular ring structure in sequence according to a following formula:
when each cluster pr is added to the circular ring structure, a suitable position is sought according to the formula mentioned above and a new ring for disposing the cluster pr is added between two most similar clusters ps and pt;
wherein in the circular ring structure, the closer is a cluster to the cluster pr, the more similar is the cluster to the cluster pr; and otherwise the farther is a cluster to the cluster pr, the more dissimilar is the cluster to the cluster pr.
The clustering method of the present invention is capable of fusing multilingual documents, and linking multilingual words by the similar words bank. Based on the similar words bank and other information, the eigenvalues are extracted and the accumulation points are selected for classifying. According to the similarity, the documents are added to the clusters, and according to the similarity the clusters are added to the circular ring structure for arrangement. The present invention is capable of helping users to quickly look up a series of documents in relative classification by key words. Compared with a condition that the clustering mechanism is not provided, the present invention is capable of responding in a faster speed, avoiding troubles of manually looking up of the users and reducing waiting time of the users. The method of the present invention is capable of providing clear classification for the documents, providing more accurate and complete information, in such a manner that the users are capable of fully understanding progress of subjects that the documents belongs to in the classification.
Further description of the present invention is illustrated combining with accompanying drawings.
Referring to
Firstly, a similar words bank is established, wherein multilingual words having identical or similar meanings are recorded in each line of the similar words bank, and whether the words are verbs or nouns is marked. N documents participating in clustering serves as an input.
Based on the similar words bank, contents and citations of the documents, extract eight eigenvalues of citation relationship (f1), identical references (f2), identical strings (f3), similar strings (f4), identical nouns (f5), similar nouns (f6), identical verbs (f7) and similar verbs (f8) to form an eigenvector F(i),
F(i)=(f1(i),f2(i),f3(i),f4(i),f5(i),f6(i),f7(i),f8(i)).
Calculate a product of references
f
1(i)f1(j)=bool×W;
calculate a product of the identical references
calculate a product of the identical strings
calculate a product of the similar strings
calculate a product of the identical nouns
calculate a product of the similar nouns
calculate a product of the identical verbs
calculate a product of the similar verbs
Based on calculation of product of the eigenvalues, similarity of any two documents i and j is calculated, Proximity(i,j)==Σq=18fq(i)fq(j). Thus, the N documents in total form an N×N similarity matrix.
Based on the N×N similarity matrix, accumulation points are selected from a set of the documents. On an initial condition, two most dissimilar documents, i.e., with a minimum Proximity(i,j), are selected for serving as two initial accumulation points p1 and p2. Add p1 and p2 to an accumulation point set denoted as Points. Residual accumulation points are selected according to a following maximum and minimum
Add the residual accumulation points to the accumulation point set Points in sequence. Until a stopping accumulation point greater than a threshold value Th is selected, i.e.,
stop selecting accumulation points, wherein the stopping accumulation point is not added to the set Points.
In the formula, pr, r=1, 2, . . . , m represents documents selected as the accumulation points. Then an (m+1)th accumulation point is selected from documents which haven't been selected as the accumulation points and added to the set Points. A threshold value Th is set for the formula mentioned above. When a stopping accumulation point selected satisfies
stop selecting the accumulation points. In addition, the stopping accumulation point is not added to the set Points. Thus, M accumulation points, i.e., M clusters, are selected.
Add residual N−M documents to the M clusters denoted as Cluster(pr), r=1, 2, . . . , M. In the beginning, each set only has one documents selected as the accumulation points. For a document i not added to the clusters, calculate a most similar cluster according to a formula
The residual N−M documents are added in sequence to a set of the cluster. Select a document iq with a greatest similarity is in each time to add to the set of the cluster and update the Cluster(pq) till all documents are added to the set of the cluster.
Dispose the M clusters in a structure of a circular ring. At the beginning, two of the clusters are randomly selected and disposed in the circular ring. M−2 clusters are left. Randomly select one cluster is randomly from the M−2 clusters, and find an appropriate position for the cluster selected in the circular ring according to a formula
A new ring pr is added between the clusters ps and pt which are most similar.
In the whole process, a final output comprises the M clusters and is disposed in the structure of the circular ring. Each cluster comprises similar documents without restriction on languages. The closer is a distance between clusters in the structure of the circular ring, the more similar are the clusters; and otherwise the farther is the distance therebetween, the more dissimilar are the clusters.
This is a U.S. National Stage under 35 U.S.C 371 of the International Application PCT/CN2013/083524, filed Sep. 16, 2013, which claims priority under 35 U.S.C. 119(a-d) to CN 201310416693.8, filed Sep. 12, 2013.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/CN13/83524 | 9/16/2013 | WO | 00 |