Patent Application
20110055228
The present invention relates to a cooccurrence dictionary creating system, a scoring system, a cooccurrence dictionary creating method, a scoring method, and a program thereof.
This application claims priority to Japanese Patent Application No. 2008-094980 filed on Apr. 1, 2008, and Japanese Patent Application No. 2008-124254 filed on May 12, 2008, the entire contents of which are hereby incorporated by reference.
In recent years, various types of documents have been converted into electronic documents. It is important to effectively use various pieces of information written in the electronic documents. In order to effectively use the electronic documents, natural language processing techniques have attracted attention.
When the documents are processed semantically with natural language processing, cooccurrence word information is used in many cases.
For example, since it is considered that similar cooccurrence words are semantically similar to each other, the semantic similarity of two words is calculated to be higher when the cooccurrence words are more similar to each other. In addition, in hiragana-to-kanji conversion, a candidate from among conversion candidates is determined to be likely when it often cooccurs with a previously established word.
Examples of the conventional cooccurrence dictionary creating system are disclosed in Patent Document 1.
The cooccurrence dictionary creating system of Patent Document 1 includes a document analyzing section which analyzes a given document group, a word extracting section which extracts words existing in the given document group and causing a storage unit to store the extracted words, a word-chain extracting section which extracts word chains existing in the given document group and causing the storage unit to store the extracted word chains, a number-of-cooccurrences detecting section which detects the number of cooccurrences between each word and each word chain and causing the storage unit to store the number of cooccurrences, a concept information quantifying section which detects a cooccurrence degree in accordance with the number of cooccurrences, and for quantifying word concept information based on the detected cooccurrence degree and causing the storage unit to store the quantified concept information, and a concept information database creating section which makes a database of the word concept information which is obtained by the concept information quantifying section.
The above-mentioned “word chain” is a chain of n (n is two or more) words which are continuous in a document.
According to Patent Document 1, first of all, each sentence in a document group is subjected to a morpheme analysis. Then, all words or word chains (chains constituted by two or more words) are extracted from the result of the morpheme analysis and are stored in the storage unit. Subsequently, the number-of-cooccurrences detecting section extracts, from among the extracted independent words (nouns, pronouns, verbs, adjectives, adverbs) or word chains, cooccurring independent words or word chains and counts the number of appearances. The number-of-cooccurrences detecting section sends the counting result to the concept information quantifying section. Here, the number of appearances is counted when words or word chains cooccur in a predetermined document range. The “predetermined document range” is either a document, a paragraph, or a sentence. Then, the concept information quantifying section calculates the cooccurrence degree for each of the extracted words or word chains with each of the words or word chains based on the counting result by the number-of-cooccurrences detecting section. Here, the cooccurrence degree is a value obtained by dividing the number of cooccurrences by the number of appearances of one word, which constitutes the cooccurrence information, and normalizing the result.
The first problem of the conventional technique is that it is difficult to generate a high quality cooccurrence dictionary. This is because while the cooccurrence dictionary creating system disclosed in Patent Document 1 collects all the cooccurrences within a predetermined range such as a document, a paragraph, or a sentence, the collected cooccurrences include the ones without any semantic relationships in practice. Considering a case of obtaining cooccurrence information from a sentence “Curry Wa Karai Ga, Fukujinzuke Wa Shoppai. (these Japanese words mean ‘the curry is spicy, and the pickles are salty’)”, for example, “Curry, Karai (this Japanese word means ‘spicy’)”, “Curry, Fukujinzuke (this Japanese word means ‘pickles’)”, “Fukujinzuke (this Japanese word means ‘pickles’), Shoppai (this Japanese word means ‘salty’)”, “Curry, Shoppai (this Japanese word means ‘salty’)”, “Fukujinzuke (this Japanese word means ‘pickles’), Karai (this Japanese word means ‘spicy’)”, and the like are obtained as the cooccurrences according to the technique of Patent Document 1. Here, three type cooccurrences of “Curry, Karai (this Japanese word means ‘spicy’)”, “Curry, Fukujinzuke (this Japanese word means ‘pickles’)”, and “Fukujinzuke (this Japanese word means ‘pickles’), Shoppai (this Japanese word means ‘salty’)” are semantically appropriate. However, although “Curry, Shoppai (this Japanese word means ‘salty’)”, “Fukujinzuke (this Japanese word means ‘pickles’), Karai (this Japanese word means ‘spicy’)” are grammatically appropriate, these expressions are not usually used. As described above, the cooccurrence dictionary creating system disclosed in Patent Document 1 collects a great number of cooccurrences with low semantic relationships. This tendency appears more markedly when the range for obtaining the cooccurrences becomes wider from a sentence to a paragraph to a document.
The second problem of the conventional technique is that a large storage region is required for storing cooccurrence information, and that the storage capacity of the cooccurrence dictionary becomes large. This is because in the cooccurrence dictionary creating system disclosed in Patent Document 1, as the number of words in a document group or the number n of the word chains for the expressions (referred to as a complex expression) constituted by complex words increases, the number of the types of the word chains increases. In order to store the cooccurrence degrees of the complex expressions, it is necessary to provide a region for storing a number of numerical values, which would be a square number of the types of the word chains, in the worst case. For example, it is assumed that 1000 words are used in the document group and n is 3. Then, the number of the types of the complex expressions becomes approximately one billion (=1000×1000×1000) in the worst case. That is, the cooccurrence dictionary creating system disclosed in Patent Document 1, which is configured to store all the cooccurrence degrees, requires a region for storing a numerical value that is the square number of one billion in its cooccurrence dictionary.
Patent Document 1: Japanese Unexamined Patent Publication, First Publication No. 2006-215850
Non-Patent Document 1: Akiko Aizawa, “Kyoki Ni Motozuku Ruijisei Syakudo (Cooccurrence-based Similarity Criteria)”, Operation research as a management science research, vol. 52, No. 11, PP. 706-718, 2007
Non-Patent Document 2: T. Hofmann, “Probabilistic Latent Semantic indexing”, Proc. of SIGIR '99, pp. 50-57, 1999
Non-Patent Document 3: M. A. Hearst, Text Tiling: Segmenting Text into Multiparagraph Subtopic Passages, Computational Linguistics, Vol. 23, No. 1, pp. 33-64, 1997
The present invention was made in view of the above problem, and the object thereof is to provide a cooccurrence dictionary creating system, a scoring system, a cooccurrence dictionary creating method, a scoring method, and a program thereof, which are capable of producing a cooccurrence dictionary while taking semantic relationships into consideration.
In addition, the object of the present invention is to provide a cooccurrence dictionary creating system, a scoring system, a cooccurrence dictionary creating method, a scoring method, and a program thereof, which are capable of producing a cooccurrence dictionary with a small storage region corresponding to complex expressions by extracting only the complex expressions with meaning.
A cooccurrence dictionary creating system according to an aspect of the present invention includes a language analyzing section which subjects a text to a morpheme analysis, a clause specification, and a modification relationship analysis between clauses, a cooccurrence relationship collecting section which collects cooccurrences of nouns in each clause of the text, modification relationships of nouns and declinable words, and modification relationships between declinable words as cooccurrence relationships, a cooccurrence score calculating section which calculates a cooccurrence score of the cooccurrence relationship based on a frequency of the collected cooccurrence relationship, and a cooccurrence dictionary storage section which stores a cooccurrence dictionary in which a correspondence between the calculated cooccurrence score and the cooccurrence relationship is described.
According to the present invention, a unit constituting a cooccurrence relationship is defined as a clause. Accordingly, it is not necessary to distinguish a noun clause from a noun, and a declinable word clause from a declinable word, respectively. For this reason, the description will be made while omitting the expression “clause” in some cases. However, a word and not a clause will be meant only when specified with the expression “word”.
According to the present invention, it is possible to produce a cooccurrence dictionary while taking semantic relationships into consideration. This is because cooccurrence relationships relating to declinable words are limited only to modification relationships.
In addition, according to the present invention, a cooccurrence collection unit is set to be a clause. Therefore, it is possible to extract only the complex expressions with meaning. As a result, it is possible to produce a cooccurrence dictionary with a small storage region corresponding to the complex expressions.
The detailed description will be made of a first embodiment for implementing the present invention with reference to accompanying drawings.
According to the first embodiment of the present invention, the configuration includes a corpus input section 1 which inputs a text as a collection source of cooccurrence relationships, a storage unit 2 which stores the text and a created cooccurrence dictionary, a data processing unit 3 operated by a program control, and a cooccurrence dictionary display section 4 which displays the content of the created cooccurrence dictionary.
Each of the corpus input section 1, the storage unit 2, the data processing unit 3, and the cooccurrence dictionary display section 4 will be described.
The corpus input section 1 causes a corpus storage section 20 to store the text data as a collection source of the cooccurrence relationships. The corpus is constituted by a “text” which represents a text main body and an “ID” which represents an identifier for the respective pieces of data. The “ID” may be designated in advance, or may be automatically given. For example, IDs of sequential integer numbers may be given to the respective pieces of data in the input order thereof.
The storage unit 2 includes a corpus storage section 20 and a cooccurrence dictionary storage section 21.
The corpus storage section 20 stores the text date input by the corpus input section 1.
The cooccurrence dictionary storage section 21 stores a cooccurrence dictionary created by a cooccurrence dictionary creating section 30.
The data processing unit 3 includes the cooccurrence dictionary creating section 30 and a cooccurrence dictionary output section 31.
The cooccurrence dictionary creating section 30 includes a language analyzing section 300, a cooccurrence relationship collecting section 301, and a cooccurrence score calculating section 302.
The language analyzing section 300 reads the text data from the corpus storage section 20, performs a morpheme analysis, a clause specification, and a modification relationship analysis between clauses for each piece of the text data. The language analyzing section 300 outputs the analysis results to the cooccurrence relationship collecting section 301.
The cooccurrence relationship collecting section 301 collects nouns, declinable words, cooccurrence relationships between nouns, cooccurrence relationships in each of which a noun is in a modification relationship with a declinable word, and cooccurrence relationships in each of which a declinable word is in a modification relationship with another declinable word from the analysis results of the language analyzing section 300. In addition, the cooccurrence relationship collecting section 301 obtains the frequencies of the nouns, declinable words, and the respective cooccurrence relationships. The cooccurrence relationship collecting section 301 outputs the collected cooccurrence relationships and the obtained frequencies of the cooccurrence relationships to the cooccurrence score calculating section 302. Here, the cooccurrence relationships between nouns are collected only when each of the nouns cooccurs in a predetermined document range. The predetermined document range is either a document, a paragraph, or a sentence.
The cooccurrence score calculating section 302 receives the respective nouns, the declinable words, the cooccurrence relationships, and the frequencies thereof, and then calculates a cooccurrence score for the respective cooccurrence relationships. Thereafter, the cooccurrence score calculating section 302 causes the cooccurrence dictionary storage section 21 to store the respective cooccurrence relationships and the calculated cooccurrence scores. The cooccurrence score indicates the degree in which two words are used at a same time, and is calculated so as to be higher when two words are used at a same time more frequently. It is possible to use an arbitrary cooccurrence intensity calculating method for the cooccurrence scores. In addition, logarithms of the frequencies may be taken as the cooccurrence score so as not to allow a cooccurrence relationship with a high frequency to be too advantageous.
In addition, it is considered that two words in a relationship, in which the two words unevenly cooccur, have a deep semantic relationship. Accordingly, it is applicable to use as the cooccurrence score a value obtained by dividing a frequency of a cooccurrence relationship by a frequency of one of the two words in the cooccurrence relationship, or by the sum of the frequencies of both of the two words.
In addition, the semantic relationship is stronger for words which are semantically easy to be used at the same time, and weaker for words which are rarely used at the same time.
It is also applicable to use a Dice coefficient, self-mutual information, a Jaccard coefficient, and the like which are the criteria for the cooccurrence intensities in Non-Patent Document 1. For example, the Dice coefficient disclosed in Non-Patent Document 1 is calculated so as to satisfy a formula “Dice coefficient=2×f12/(f1+f2)” when it is assumed that f12 represents a frequency of a cooccurrence relationship and that f1 and f2 represent the frequencies of two words constituting the cooccurrence relationship.
As disclosed in Non-Patent Document 2, it is also applicable to use a method of estimating a cooccurrence ratio of two arbitrary words from a group of cooccurrence relationships. In Non-Patent Document 2, P(w_i|w_j) is calculated by estimating P(w_i|z_k), P(w_j|z_k) and P(z_k) while assuming that a cooccurrence ratio of two words w_i and w_j (0≦i, j≦n, i≠j) satisfies “P(w_i|w_j)=ΣP(w_i|z_k)P(w_j|z_k)P(z_k)” from the group of the cooccurrence relationships. Here, n represents the number of the types of the words constituting the cooccurrence relationship. k in z_k represents an index. Σ represents an operator taking the sum of all k. z_k represents a cluster in which cooccurrence words with similar distributions are collected. The number of k is designated by a user. P(z_k) represents an appearance ratio of the respective clusters. P(w_i|z_k) represents a creation ratio of w_i when a cluster z_k appears. P(w_j|z_k) represents a creation ratio of w_j when a cluster z_k appears. In Non-Patent Document 2, the more similar the distributions of the cooccurrence words are, the higher P(w_i|z_k) and P(w_j|z_k) become at the same time. Accordingly, the cooccurrence ratio is calculated so as to be higher for two words with higher ratios in which they are created from the same cluster. That is, according to Non-Patent Document 2, the cooccurrence ratio is appropriately calculated from the similarity of the distributions of the cooccurrence words for each word. For this reason, it is possible to calculate the cooccurrence score for the cooccurrence relationship which may naturally cooccur but does not cooccur in the document by accident.
The cooccurrence dictionary output section 31 reads the cooccurrence relationships described in the cooccurrence dictionary and the cooccurrence scores thereof from the cooccurrence dictionary storage section 21, and outputs the cooccurrence relationships and the cooccurrence scores to the cooccurrence dictionary display section 4. The cooccurrence dictionary output section 31 may output the cooccurrence relationships after sorting them in order from the lower cooccurrence score or in order from the higher cooccurrence score. The cooccurrence dictionary output section 31 may designate at least one word to output only the cooccurrence relationship including the input word. In addition, the cooccurrence dictionary output section 31 may output only the cooccurrence relationships with cooccurrence scores of not less than a predetermined level, cooccurrence relationships with cooccurrence scores of not more than a predetermined level, or cooccurrence relationships with cooccurrence scores of not less than a predetermined level and not more than a predetermined level.
The cooccurrence dictionary display section 4 displays the cooccurrence relationships output from the cooccurrence dictionary output section 31 along with the cooccurrence scores thereof.
In this embodiment, the cooccurrence collection unit for the cooccurrence dictionary creating section 30 is set as a clause which is a minimum unit of a meaning in a sentence. The cooccurrence dictionary creating section 30 limits the cooccurrence of a noun and a declinable word and the cooccurrence between declinable words only to the modification relationships. Accordingly, it is possible to reduce the collection amount of the cooccurrence relationships without any semantic relationships, and to thereby produce a cooccurrence dictionary with a high quality and a low capacity.
Next, the overall operations of this embodiment will be described in detail with reference to
First, the corpus input section 1 causes the corpus storage section 20 to store the text data as a collection source of the cooccurrence relationships (step S1 in
Then, the language analyzing section 300 reads the text data from the corpus storage section 20, and performs a morpheme analysis, a clause specification, and a modification relationship analysis between clauses (step S2 in
The language analyzing section 300 performs a morpheme analysis (step S101). This processing of step S101 is also referred to as a morpheme analysis.
Then, the language analyzing section 300 organizes the results of the morpheme analysis to each clause unit, and specifies whether each of the clauses is a noun clause or a declinable word clause (step S102). This processing of step S102 is also referred to as a clause specification. Here, it is determined whether each of the clauses is a noun clause or a declinable word clause by the type of the word class of the independent word which is found first after searching a morpheme from the end of a clause. If a noun is found first, it is determined to be a noun clause. If a declinable word is found first, it is determined to be a declinable word clause.
At last, modification relationships between clauses are analyzed (step S103). This processing of step S103 is also referred to as a modification relationship analysis.
In the drawing illustrating the processing result of step S103 (the diagram in the bottom part in
Returning to the explanation of
In addition, the declinable words are changed to root forms from the results of the morpheme analysis. For example, “Myoni” is changed to “Myoda”. After such processing, cooccurrences between nouns, modification relationships between nouns and declinable words, and modification relationships between declinable words are collected, and the frequencies are counted.
Moreover, frequencies of just the nouns and just the declinable words are also recorded when these frequencies are necessary for the calculation of the cooccurrence scores. Here, directions are not fixed for the cooccurrence relationships in the embodiments of the present invention. That is, a cooccurrence relationship constituted by the same words is counted as one type by determining the order relationships of the two words with a size of value of a character code.
Returning to the explanation of
Returning to the explanation of
Next, the description will be made of the effects of this embodiment.
In this embodiment, the language analyzing section 300 performs the morpheme analysis, the clause specification, and the modification relationship analysis between clauses. Then, the cooccurrence relationship collecting section 301 collects the respective pieces of data regarding the cooccurrences of noun clauses, modification relationships between noun clauses and declinable word clauses, and modification relationships between declinable word clauses. Thereafter, the cooccurrence score calculating section 302 calculates the cooccurrence scores of the cooccurrence relationships based on the frequencies of the collected cooccurrence relationships. As a result, the cooccurrence relationships relating to the declinable words are limited only to the modification relationships. Accordingly, it is possible to create the cooccurrence dictionary from the cooccurrence relationships with strong semantic relationships.
For example, when cooccurrence words are simply collected from a sentence “Curry Wa Karai Ga, Fukujinzuke Wa Shoppai. (these Japanese words mean ‘the curry is spicy, and the pickles are salty’)”, cooccurrence relationships with weak semantic relationships such as “Curry, Shoppai (this Japanese word means ‘salty’)”, “Fukujinzuke (this Japanese word means ‘pickles’), Karai (this Japanese word means ‘spicy’)”, and the like are also collected. On the other hand, when the cooccurrences of nouns and declinable words are limited only to the modification relationships, only ones with strong semantic relationships such as “Curry, Karai (this Japanese word means ‘spicy’)”, “Fukujinzuke (this Japanese word means ‘pickles’), Shoppai (this Japanese word means ‘salty’)” are collected. Here, the cooccurrences between nouns have semantic relationships in many cases even when the nouns are not in the modification relationships. Accordingly, the cooccurrences between nouns are not limited only to the modification relationships.
According to this embodiment, since the cooccurrence collection unit is set to a clause, there are no cooccurrence relationships with weak semantic relationships. As a result, it is possible to create the cooccurrence dictionary with a small storage region. A clause means “the one which is obtained by splitting a sentence into pieces which are as small as possible within a range in which the meaning thereof are understandable” which is similar to a general definition thereof. When the collection unit is set to a clause, the complex expressions, each of which does not constitute a unit of meaning, can be excluded. Accordingly, it is possible to reduce the size of storage capacity of the cooccurrence dictionary by the corresponding amount. In addition, when the cooccurrences are collected in units of meaning, the cooccurrence relationships which do not reflect the meaning of the sentence are not collected. Accordingly, it is possible to reduce the size of storage region, and create the cooccurrence dictionary with a high quality.
For example, when “Kensaku Engine Wa Kosoku Ni Keyword Wo Fukumu Bunsho Wo Sagasu Koto Ga Dekiru (these Japanese words mean ‘search engines make it possible to search documents including keywords at a high speed’)” is subjected to morpheme analysis, “Kensaku/Engine/Wa/Kosoku/Ni/Keyword/Wo/Fukumu/Bunsho/Wo/Sagasu/Koto/Ga/Dekiru” is obtained. Here, the descriptions of the word classes are omitted.
On the other hand, the clauses in the sentence are as follows: “Kensaku Engine Wa/Kosoku Ni/Keyword Wo/Fukumu/Busnho Wo/Sagasu/Koto Ga/Dekiru”. When word chains are taken as a basic unit, complex expressions without any meaning such as “Wa Kosoku”, “Fukumu Busho” and the like are also collected.
In addition, when words are taken as collection units, cooccurrences with weak semantic relationships such as “Engine, Bunsho” and “Engine, Keywords” are collected. On the other hand, when clauses are taken as collection units, cooccurrence relationships, which appropriately reflect the meaning of the sentence, such as “Kensaku Engine, Bunsho”, “Kensaku Engine, Keyword” and the like can be collected.
Next, the detailed description will be made of a second embodiment of the present invention with reference to accompanying drawings.
The second embodiment of the present invention is different from the first embodiment (
The language analyzing section 300 reads the text data from the corpus storage section 20, and performs a morpheme analysis, a clause specification, and a modification relationship analysis between clauses for each pieces of the text data. Thereafter, the language analyzing section 300 outputs the analysis results to the topic splitting section 500.
The topic splitting section 500 detects changing points of topics in the respective text data from the analysis result of the language analyzing section 300. Then, the topic splitting section 500 splits the original analysis result at the respective changing points, and outputs the split analysis result to the cooccurrence relationship collecting section 301. Since the cooccurrence relationships between nouns in different topics have weak semantic relationships, the topic splitting section 500 splits the cooccurrence relationships for each topic and outputs the split results to the cooccurrence relationship collecting section 301 at a later stage. As a result, it is possible to collect the cooccurrence relationships with stronger semantic relationships.
There is an example of a text “Kino Nikkei Heikin Ga Boraku Shi Te Ita Ga, Kaigai Toshika No Eikyo De Arou Ka. Nanka, Hara Ga Hette Kita. Konbini Itte Ko Yo. (these Japanese words mean ‘the Nikkei average plunged yesterday possibly because of the influence of overseas investors. I feel like I am getting hungry. Let's go to the convenience store.’)” In this text, the topic is changed at the point of “Nanka, Hara Ga Hette Kita. (these Japanese words mean ‘I feel like I am getting hungry.’)” Accordingly, it is understood that the words “Nikkei Heikin (these Japanese words mean ‘Nikkei average’), Konbini (this Japanese word means ‘convenience store’)” cooccur by accident. On the other hand, when the two words cooccur in the same topic, for example, in a sentence like “The business climate for the convenience store industry is in a good condition, and Nikkei average is rising”, two words cooccur not by accident but with a relationship. That is, it is possible to reduce the accidental cooccurrence relationships by collecting the cooccurrence relationships from the same topics. For this reason, it is possible to create the cooccurrence dictionary with a higher quality.
The topic splitting section 500 can use any arbitrary method which can split the topics based on the results of the morpheme analysis, the clause specification, and the modification relationship analysis. For example, when n or more type of nouns, which are used in the sentences before a point, do not occur in the following sentences after the point, the topic splitting section 500 may split the topics. This is based on an assumption that words representing the same contents should be used if the same topic continues. In the above-mentioned text, there is no same noun between the sentence “Kino Nikkei Heikin Ga Boraku Shi Te Ita Ga, Kaigai Toshika No Eikyo De Arou Ka. (these Japanese words mean ‘the Nikkei average plunged yesterday possibly because of the influence of overseas investors.’)” and the sentence “Nanka, Hara Ga Hette Kita. (these Japanese words mean ‘I feel like I am getting hungry.’)
Accordingly, it is possible to think that there has been a change of topic. In addition, the topic splitting section 500 may split the topics when expressions indicating the changes of the topics appear. “Hanashi Wa Kawaru Ga (these Japanese words mean ‘let us move on to another topic’), “Tokorode (this Japanese word means ‘by the way’)”, “Totsuzen De Aruga (these Japanese words mean ‘it is sudden, but’)” and the like can be exemplified as the expressions indicating the change of topic. Moreover, the topic splitting section 500 may split the topics when there is no conjunction at the beginnings of the sentences. This is because it is considered that two sequential sentences have a relationship when there is a conjunction between them, and on the other hand, that each of the two sequential sentences belongs to a different topic when there is no conjunction between them. Furthermore, it is possible to use the technique in Non-Patent Document 3 for the topic splitting section 500. According to Non-Patent Document 3, word columns are regarded as pseudo-paragraphs, and overlapping words in two adjacent pseudo-paragraphs are measured. Then, a position where there is less overlapping is regarded as a topic changing point, and the topics are split at the topic changing point.
The cooccurrence relationship collecting section 301 has the same functions as those of the cooccurrence relationship collecting section 301 in the first embodiment except that it collects the cooccurrence relationships for each analysis result split at the topic changing points.
The other configurations are the same as those in the first embodiments.
Next, overall operations of this embodiment will be described in detail with reference to
Since steps S11 and S12 in
The topic splitting section 500 receives the analysis results by the language analyzing section 300, and detects the topic changing points in the text. Then the topic splitting section 500 splits the analysis results based on the detected changing points (step S13 in
In this example, the topic splitting section 500 splits the topics when two types or more nouns do not overlap in two sequential sentences. For example, the following description will be made while exemplifying a text “1) Saikin Toshi Ni Kyomi Ga Dete Kita Tame Nikkei Heikin Wo Check Suru Yoni Natta. (these Japanese words mean ‘I have started to check the Nikkei average since I recently became interested in investment.’) 2) Kino Nikkei Heikin Ga Boraku Shi Te Ita Ga, Kaigai Toshika No Eikyo De Arou Ka. (these Japanese words mean ‘the Nikkei average plunged yesterday possibly because of the influence of overseas investors.’) 3) Nanka, Hara Ga Hette Kita. (these Japanese words mean ‘I feel like I am getting hungry.’) 4) Konbini Itte Ko Yo. (these Japanese words mean ‘let's go to the convenience store.’)” In addition, the numbers 1) to 4) are given to the sentences, respectively, for the explanation, and these numbers are not actually written in the text.
The topic splitting section 500 counts the number of types of overlapping nouns in adjacent two sentences, and splits the topics between the sentences where two or more types of nouns do not overlap. The nouns in the respective sentences can be extracted from the output of the language analyzing section 300. As a result, three types of nouns “Toshi, Nikkei, Heikin” overlap in the sentences 1) and 2). In addition, there are no overlapping nouns in the sentences 2) and 3). Moreover, there are no overlapping nouns in the sentences 3) and 4). Accordingly, the topic splitting section 500 splits the text into three parts of “Saikin Toshi Ni Kyomi Ga Dete Kita Tame Nikkei Heikin Wo Check Suru Yoni Natta. Kino Nikkei Heikin Ga Boraku Shi Te Ita Ga, Kaigai Toshika No Eikyo De Arou Ka. (these Japanese words mean ‘I have started to check the Nikkei average since I recently became interested in investment. The Nikkei average plunged yesterday possibly because of the influence of overseas investors.’)”, “Nanka, Hara Ga Hette Kita. (these Japanese words mean ‘I feel like I am getting hungry.’)”, and “Konbini Itte Ko Yo. (these Japanese words mean ‘let's go to the convenience store.’)”
Since steps S14 to S16 in
Next, the effects of this embodiment will be described.
This embodiment has the following effects in addition to the effects of the first embodiment. That is, it is possible to collect the cooccurrences between nouns only from the same topics by providing the topic splitting section 500. Accordingly, it is possible to create the cooccurrence dictionary while limiting the target only to the cooccurrence relationships with stronger semantic relationships. In addition, the cooccurrences of the nouns and the declinable words and the cooccurrences between the declinable words are naturally limited to the modification relationships of the nouns and the declinable words and the modification relationships between declinable words in the sentences. For this reason, it is possible to collect the cooccurrence relationships with stronger semantic relationships regardless of whether the topics are split.
Next, the detailed description will be made of a third embodiment of the present invention with reference to accompanying drawings.
The third embodiment of the present invention is different from the first embodiment (
The storage unit 9 is different from the storage unit 2 in that the storage unit 9 further includes a text data storage section 22 and a storage section 23 for text data with typicality score in addition to the corpus storage section 20 and the cooccurrence dictionary storage section 21.
The data processing unit 7 is different from the data processing unit 3 in that the data processing unit 7 includes a cooccurrence dictionary creating section 70, a typicality scoring section 71, and a text data selecting section 72 instead of the cooccurrence dictionary creating section 30 and the cooccurrence dictionary output section 31.
The cooccurrence dictionary creating section 70 creates the cooccurrence dictionary based on the text, which is stored in the corpus storage section 20 by the corpus input section 1, as a collection source of the cooccurrence relationships, and causes the cooccurrence dictionary storage section 21 to store the created cooccurrence dictionary. The cooccurrence dictionary creating section 70 has the same configuration as that of the cooccurrence dictionary creating section 30 or the same configuration as that of the cooccurrence dictionary creating section 50 in the second embodiment.
The text data input section 6 causes the text data storage section 22 to store the text data as a target to which the typicality is given by the cooccurrence dictionary. The text data includes a “text” representing the text main body, an “ID” representing an identifier of the respective pieces of the data, and an “initial score” which is set typicality score designated in advance.
The “IDs” may be designated in advance, or IDs of sequential integer numbers may be automatically given in the input order thereof. In addition, the “text” may be a document, or may be a relationship constituted by a plurality of words extracted by some method.
The larger the value of the “initial score” is, the higher the evaluation is. In addition, when there is no need to give the “initial scores”, or when the “initial scores” are not given, a same value such as 0, 1, or the like is used for all the text. In addition, the text data input section 6 may be configured to automatically input the output from the other natural language processing systems, such as the hiragana-to-kanji conversion candidates, the information search results, information extraction results, and the like, and the “initial score” may be the score of the respective systems. For example, as the “initial scores”, the scores of the hiragana-to-kanji conversion candidates, the reliability of the information extraction results, which is given by the information extracting unit, the degrees of fitness of the search engines, the inverse numbers of the orders, or the like can be considered.
The typicality scoring section 71 reads the text data stored in the text data storage section 22 and the cooccurrence dictionary data stored in the cooccurrence dictionary storage section 21. Then, the typicality scoring section 71 extracts the cooccurrence relationships from the respective pieces of the text data, and calculates the typicality scores for the respective pieces of the text data from the initial scores and the cooccurrence scores of the cooccurrence relationships of the respective pieces of the text data. Thereafter, the typicality scoring section 71 causes the storage section 23 for text data with typicality score to store the respective texts and the typicality scores thereof.
Here, the calculation of the typicality scores is performed such that the typicality scores are higher when the respective cooccurrence scores and the initial scores are higher. For example, it can be considered that the sum, the product, or the combination of the sum and the product of the respective cooccurrence scores and the initial scores are employed as the typicality score.
The text data selecting section 72 reads the text and the typicality scores thereof from the storage section 23 for text data with typicality score. Then, the text data selecting section 72 selects the text data based on the size relationships of the typicality scores or on values, and outputs the data to the text data display section 8.
The text data display section 8 displays the text data, which has been selected by the text data selecting section 72 based on the typicality of the content, along with the typicality score thereof.
Next, the overall operations of this embodiment will be described in detail with reference to
In this embodiment, the cooccurrence dictionary storage section 21 has a function of producing the cooccurrence dictionary and a function of giving typicality scores to the text which is the target, to which the typicality is given, using the produced cooccurrence dictionary. The operation of producing the cooccurrence dictionary is the same as the operation of producing the cooccurrence dictionary in the first or second embodiment. Accordingly, the operations after producing the cooccurrence dictionary will be described hereinafter.
First, the text data input section 6 causes the text data storage section 22 to store the text data to which the typicality is given by the cooccurrence dictionary (step S21 in
Next, the typicality scoring section 71 reads the text data from the text data storage section 22. Then, the typicality scoring section 71 extracts the cooccurrence relationships from the respective pieces of the text data (step S22 in
The typicality scoring section 71 regards some combinations of words as cooccurrence relationships in the case of the records constituted by a plurality of words and when the texts are not sentences, as shown in
For example, the “attributions” are viewpoints of the evaluations for the “objects” in
Returning to the explanation of
Next, the typicality scoring section 71 calculates the typicality scores for the typicality of the respective pieces of the text data obtained in step S22 based on the cooccurrence relationship of the respective pieces of the text data extracted in step S22, the initial score of the respective text data read in step S22, and the cooccurrence score of the respective cooccurrence relationships obtained in step S23 (step S24 in
The description will be made of the operation while exemplifying the calculation of the typicality score for the text of ID=1 in
As for the typicality scores of the texts of IDs=1 to 4 in
The typicality scoring section 71 calculates the typicality scores from the data, which is stored in the text data storage section 22, shown in
Returning to the explanation of
At last, the text data display section 8 displays the text selected by the text data selecting section 72 (step S26 in
Next, the effects of this embodiment will be described.
According to this embodiment, it is possible to calculate the degrees of the semantic typicality of the contents of the text data. This is because the cooccurrence dictionary is used which is created in the first or second embodiment by limiting its target only to the cooccurrence relationships with high semantic relationships.
In this embodiment, when the text as a target to which the typicality is given is a sentence, the typicality scoring section 71 subjects the text to morpheme analysis, the clause specification, and the modification relationship analysis between the clauses. Then, the typicality scoring section 71 collects the cooccurrences of nouns, modification relationships of nouns and declinable words, and modification relationships between declinable words in the text in a unit of clause, as the cooccurrence relationships. Thereafter, the typicality scoring section 71 obtains the cooccurrence scores corresponding to the collected cooccurrence relationships from the cooccurrence dictionary, and calculates the degrees of the typicality of the contents of the text. Accordingly, it is possible to more precisely calculate the degrees of the semantic typicality of the contents of the text.
In addition, the cooccurrence relationships collected from the text as a target to which the typicality is given may not be limited to the ones relating to cooccurrences of nouns, modification relationships of nouns and declinable words, and modification relationships between declinable words. In such a case, it is possible to achieve precision to some degree since the cooccurrence dictionary created by targeting only the cooccurrence relationships with strong semantic relationships is used.
Moreover, according to this embodiment, when the text as a target to which the degree of the typicality is given is a record constituted by a plurality of words, the typicality scoring section 71 collects the combinations of words which have meaning when combined together as the cooccurrence relationships from among all the combinations of words. Then, the typicality scoring section 71 obtains the cooccurrence scores corresponding to the collected cooccurrence relationships from the cooccurrence dictionary, and calculates the degrees of the typicality of the contents of the text. For this reason, it is possible to more precisely calculate the degrees of the semantic typicality of the contents of the text.
In addition, it may not be limited to the combinations of words which have meaning when combined together. In such a case, it is possible to achieve precision to some degree since the cooccurrence dictionary created by targeting only the cooccurrence relationships with strong semantic relationships is used.
Next, the detailed description will be made of a fourth embodiment of the present invention with reference to accompanying drawings.
The fourth embodiment of the present invention is different from the third embodiment (
The storage unit 10 is different from the storage unit 9 in that the storage unit 10 does not include the corpus storage section 20.
The data processing unit 11 is different from the data processing unit 7 in that the data processing unit 11 does not include the cooccurrence dictionary creating section 70.
This embodiment is different from the third embodiment in that the cooccurrence dictionary produced using the cooccurrence dictionary creating section 30 of the first embodiment or the cooccurrence dictionary creating section 50 of the second embodiment is stored in advance in the cooccurrence dictionary storage section 21.
Next, the overall operations of this embodiment will be described. In this embodiment, since the cooccurrence dictionary is stored in advance in the cooccurrence dictionary storage section 21, the operation of producing the cooccurrence dictionary is not needed. The other operations, that is, the operation of the typicality scoring section 71 giving the typicality to the text data using the cooccurrence dictionary stored in the cooccurrence dictionary storage section 21, the operation of the text data selecting section 72 selecting the text to be displayed on the text data display section 8 based on the typicality scores of the respective texts, and the like are the same as those in the third embodiment. For this reason, the description thereof will be omitted.
Next, the effects of this embodiment will be described.
According to this embodiment, it is possible to achieve the same effects as those of the third embodiment and calculate the degrees of the semantic typicality of the contents of the text data at a high speed. This is because there is no need to take time to create the cooccurrence dictionary by using the cooccurrence dictionary produced in advance.
As described above, the embodiments of the present invention were described. However, the present invention is not limited to each of the above embodiments, and various types of additions and modifications are available. It is needless to say that the functions of the present invention can be implemented by hardware. In addition, the functions can be implemented by a computer and a program. The program can be provided by being recorded in a computer readable recording medium such as a magnetic disc, a semiconductor memory, or the like. The program is read by a computer at the time of starting the computer. The read program controls the operations of the computer. Accordingly the program causes the computer to function as the respective function parts of the data processing unit in the respective embodiments, and to execute the processing steps as described above.
The present invention can be applied to a system and the like for producing a cooccurrence dictionary to be used for semantic analysis of natural languages such as modification relationship analysis, document proofreading, hiragana-to-kanji conversion, evaluation of semantic consistency of an information extraction result, evaluation of a degree of semantic typicality of a text, and the like.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2008-094980 | Apr 2008 | JP | national |
| 2008-124254 | May 2008 | JP | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/JP2009/056804 | 4/1/2009 | WO | 00 | 9/13/2010 |
