Patent Application
20100049499
The present invention relates to a document analyzing apparatus and a method thereof. More specifically, the present invention relates to a novel document analyzing apparatus and its method capable of extracting or detecting a unique term (keyword) according to a chronological order from a linguistic material which increases in time series, such as news, web news, web logs, a newspaper, a magazine, an interview record, a deposition, a questionnaire, a novel, etc.
The world of disaster management is an academic field being in need of cooperation with a number of academic fields, and is a practical field being in need of cooperation between practionners and researchers. This means that it is difficult to be well versed in an entire world surrounding the disaster management.
Not only understanding of the information in relation to such a disaster management is hampered by lack of knowledge for the respective fields, but also because the information are collected, saved and summarized by a technique on a discipline basis, data and research products having formats each of which conforms to search of the respective disciplines are often hard to use and hard to understand. In the world of the disaster management, this makes it difficult to make a communication between researchers who are different in disciplines, and between practionners and researchers of the disaster management.
From this background, in the world of the disaster management, with the goal of making easy exchanges of information between the practionners and the researchers, prompting a cross-disciplinary study and spreading a research product to a practical area, a need f is heightened for constructing the basis of the research support and the practical support capable of searching data and information, and a research product in relation to the disaster management of a self field to be used by researchers and practionners in other fields without any constraints due to the kind of the medium no matter when or where by using a user-friendly interface.
An inventor, et al. had tried to develop an inclusive database (Cross Media Database, hereinafter referred to as “XMDB”.) including a search/display function for sharing or exchanging information between disaster management researchers and disaster management practitioners (Nonpatent Document 1: Nozomu Yositomi, Go Urakawa, Ayumu Simoda, Hironori Kawakata, Haruo Hayasi, “Construction of cross media database for sharing disaster management information” Journal of Institute of Social Safety Science, No. 6, pp. 315-322, 2004).
The data and information to be accumulated in the XMDB are not restricted to the data and information in relation to natural phenomena, such as an observation result of shakes by a strong-motion seismograph and rainfalls around the nation observed by the Meteorological Agency. For promoting the development of research and spreading the research products and the past teaching to the practical field, data and information in relation to the disaster as a social phenomenon, such as records of experiences, records of addressing the disaster (style and memo), disaster reports, published materials, newspaper articles, web-news articles become the objects of making a database.
In the world of the disaster management, activities for social-scientific study relating to disasters have long been developed (Nonpatent Document 2: Hiroyuki Kameda “Study of integrated disaster management counter measure against urban disasters in the light of the South Hyogo earthquake in 1995” urgent projects of the Ministry of Education, Culture, Sports, Science and Technology, 37 pp. 1995).
As a study of disasters, in addition to a natural-scientific study applying a mechanics covering a disaster as a natural phenomena, a study considering phases as a society including victims of a disaster who experience the disaster, workers for addressing a disaster, persons outside a disaster area, and a social phenomenon for dealing problem of the reconstruction from a disaster has often been tackled with the occurrence of the Great Hanshin Awaji Earthquake in 1995 and the 9.11 terrorist attacks in 2001 as a turning point. The study treating with the social phenomenon needs to make a database of records of the condition of the disaster as well as the framework of the natural science.
In the natural disaster science, various analyses are performed based on observation results of the shakes of the strong-motion seismograph and observation results of the movements of clouds by a weather satellite, to thereby deepen the understanding the generation process of a hazard of nature such as the earthquake and heavy rain, or to allow a study of the improvement of resistance of the structure by using these results as inputs and external forces of a simulation.
In the filed dealing with the social phenomenon, similar to the approach of the natural disaster science aimed at the understanding of the natural phenomena and improvement of the resilience of the structure, it is required to prepare things for compiling data and materials to a database to thereby extract and systematize teachings and knowledge, and implement an effective response to disasters. Furthermore, various records in relation to the past responses to the disasters in addition to the study are located as important intelligence information that practionners go through.
However, the records of the social phenomenon under the disasters in relation to the social phenomenon cause following problems due to their data format as linguistic materials (text materials) when being accumulated in the XMDB and being performed with information retrieval.
The first problem is that at a time of accumulation to the database, for applying keywords representing contents of respective records, a large number of human resources and specialized knowledge are required. The XMDB mounts a function of information retrieval based on the time, space, theme, and therefore, as data to be accumulated, three kinds of meta data, such as chronological information like created date and time of data, position information induced in the data, and a keyword representative of the content of the data are required to be applied to a record.
Applying such meta data is placed as an important procedure in the scene of the intelligence as well, and becomes an indispensable procedure for managing intelligence information, or analyzing a trend (Nonpatent Document 3: Tutomu Matumura “operational intelligence—tactic information theory for decision” Nihon Keizai Shimbun, Inc., 220 pp. 2006).
For the task of applying the keywords representative of the contents of the data, human resources having inclusive understandings as to the disaster management field are required. However, there is not such a person in reality, and reading one by one large amounts of data generated from the various source of the information and then applying keywords by a person taking the occurrence of the disaster this opportunity is substantially impossible, and in addition thereto, arbitrariness (subjective sensation) by the person is necessarily interposed.
The second problem is with which keyword the information retrieval has to be performed. One who has inclusive understandings about the world of the disaster management or is familiar with the individual cases of the disasters would easily imagine keywords required for information retrieval based on the existing knowledge. However, it is natural that it is difficult for practionners who do not have a specialized knowledge to imagine an appropriate search keyword, and researchers themselves also only have knowledge about the theme biased to the respective research fields, and are not familiar with all the cases of the disaster.
On the other hand, a method of extracting keywords from the document data is proposed in a Patent Document 1 (Japanese Patent Application Laid-Open No. 2004-5711 [G06F 17/30]), etc.
The keyword extracting device and its method in the Patent Document 1 is aimed at a fixedly-determined amount of documents, and thus cannot effectively deal with a text data cluster having a characteristic of having an order in time series, or increasing the information amount in time series such as news, for example.
Therefore, it is a primary object of the present invention to provide novel document analyzing apparatus and a method thereof.
Another object of the present invention is to provide a document analyzing apparatus and a method thereof capable of detecting appropriate unique terms (keywords) and appropriate ubiquitous terms from a linguistic material which increases in time series.
The present invention employs following features in order to solve the above-described problems. It should be noted that reference numerals and the supplements inside the parentheses show one example of a corresponding relationship with the embodiments described later for easy understanding of the present invention, and do not limit the present invention.
A first invention is a document analyzing apparatus analyzing a linguistic material which increases in time series, comprises: a text corpus producer for producing a text corpus including text data of unit documents having a chronological order, and in which unit documents later in the chronological order are larger in number than unit documents earlier in the chronological order; a morpheme analyzer for adding parts-of-speech information to morphemes making up of the text data included in the corpus text; an unnecessary morpheme remover for removing an unnecessary morpheme from the text data on the basis of the parts-of-speech information; a calculator for calculating, with respect to a morpheme which is not removed by the unnecessary morpheme remover, a chronological incremental TFIDF for each morpheme to obtain an actual measurement of the chronological incremental TFIDF; and a residual analyzer for evaluating a residual value for each morpheme by performing a residual analysis between the actual measurement calculated by the calculator and an estimate value of a cumulative total value of the chronological incremental TFIDF estimated in a previous corpus.
In the first invention, a document analyzing apparatus is typically constituted of a computer. The text corpus producer (S3: a reference numeral illustratively showing a corresponding part in embodiments, and this holds true the following.) makes a current corpus including unit documents being larger in number than those of a corpus earlier in chronological order when a preset time elapses. In a case of the web news successively increasing with time, for example, as a set time (set time is arbitrary) elapses, by using the text data of the web news, a corpus text is produced, but as a linguistic material, there are not only documents successively increasing but also documents having a merely chronological order. In the latter case, a corpus producer may not sequentially produce a corpus text with the course of time, but may prepare or produce a plurality of corpuses being successive in chronological order at once.
The morpheme analyzer (S5), in a case of the text data having a language system in which segmentation to morphemes is not performed like Japanese language, by utilizing a morpheme analyzing tool, such as Chasen (http://chasen.naist.jp/hiki/ChaSen/), for example, the text data of the unit document included in the corpus is segmented to morphemes, to each of which parts-of-speech information is added. However, in a case of the language system in which morphemes in the text have already been segmented like English language, for example, a task of segmenting to morphemes is not required and therefore, in the morpheme analyzer, tagging processing is performed, for example, to add words-of-speech information to respective morphemes making up of the text.
An unnecessary morpheme remover (S7) removes a morpheme having a kind of parts-of-speech that is set in advance as an unnecessary morpheme on the basis of the above-described parts-of-speech information added to each of the morphemes. That is, at a time of the morphological analysis, it is selected whether or not the morpheme is adopted as a candidate of a unique term and /or a ubiquitous term on the basis of the parts-of-speech information added to each of the morphemes. Here, the kind of the parts-of-speech which makes a morpheme unnecessary can be arbitrarily set.
A calculator (S11) calculates a TF (Term Frequency), that is, a frequency of appearance (total number) of a keyword candidate in the unit document with respect to each of the morphemes remained in the corpus, and moreover calculates an IDF (Inversed Document Frequency) taking a parameter of the time into account, that is, an originality value that is a value indicating that the morpheme does not appear in other documents, to thereby calculate a chronological incremental TFIDF (Term Frequency Inversed Document Frequency) of that morpheme in the corpus as “TF”דIDF”.
A residual analyzer (S17) performs a residual analysis between an estimate value of the cumulative total value of the chronological incremental TFIDF of the relevant morpheme estimated in a corpus earlier in the chronological order and the actual measurement of the cumulative total value calculated by the calculator, to thereby evaluate a residual value (positive, negative) of that morpheme.
According to the first invention, even if the linguistic material is a type of increasing in time series, the corpus producer produces a text corpus including unit documents in which unit documents later in the chronological order are larger in number than unit documents earlier in the chronological order, and a regression curve that renders the cumulative total value of the chronological incremental TFIDF as a response and the cumulative total value of the TF as an explanatory variable is produced on the basis of the corpuses, and therefore, a flow of the processing in which assuming that indexes of the cumulative total value of the chronological incremental TFIDF of the current corpus are distributed on the regression curve produced in the previous corpus, and the estimate value of the cumulative total value of the chronological incremental TFIDF of the current corpus taking the cumulative total value of the TF of the current corpus as an input is obtained, allows the linguistic material to be surely analyzed.
A second invention is according to the first invention, and further comprises a regression curve producer for producing a regression curve in each corpus between a cumulative total value of a chronological incremental TFIDF prior to the corpus and a cumulative total value of a TF prior to the corpus, wherein the residual analyzer performs a residual analysis between a regression curve produced by the regression curve producer in a previous corpus and an actual measurement of the chronological incremental TFIDF of each morpheme calculated by the calculator in a current corpus.
In the second invention, the regression curve producer calculates a constant by taking a cumulative total value(ΣTF) of the TF being an explanatory variable as X, and taking the cumulative total value (Σ chronological incremental TFIDF) of a chronological incremental TFIDF being a dependent variable as Y to thereby produce a regression curve. Here, the calculation of such regression curve is to be made in advance in the corpus earlier in chronological order. According to the second invention, in the corpus earlier in chronological order, a regression curve for estimating or anticipating the cumulative total value of the chronological incremental TFIDF in the corpus later in chronological order is prepared, capable of performing the residual analysis in the later corpus quickly.
A third invention is according to the first or second invention, further comprises a unique term selector for selecting a morpheme for which a positive residual value can be obtained as a result of the residual analysis by the residual analyzer as a unique term in the corpus.
In the third invention, a unique term selector (S21, S21A, S21B) selects a morpheme having a positive residual value (larger value) as a unique term. According to the third invention, only the residual value is selected as a parameter, and therefore, it is possible to select a unique term being objective. The unique term functions as a keyword indicating the characteristic of the corpus.
A fourth invention is according to the third invention, and the unique term selector includes a filterer for performing filtering processing.
In the fourth invention, in a case that a user selectively sets a filtering as an option, a computer (14) executes a filtering 1 for removing a term (morpheme) about which the number of documents the term appears is once during Δt (1) and/or a filtering 2 for removing a morpheme with a high frequency of appearance from the relationship between the number of documents the term appears and the frequency of appearance of the term (morpheme) (2), for example. This makes it possible to remove a morpheme representing an extremely high discriminating value.
A fifth invention is according to the third or fourth invention, further comprises a unique term outputter for visually outputting the unique term selected by the unique term selector.
In the fifth invention, the computer (14) visually displays (outputs) in graph form the unique term selected by the unique term selectors as shown in
A sixth invention is according to any one of the first to fifth inventions, and further comprises a ubiquitous term selector for selecting a morpheme for which a negative residual value can be obtained as a result of the residual analysis by the residual analyzer as a ubiquitous term of the corpus.
In the sixth invention, the ubiquitous term selector (S21) selects a morpheme having a negative residual value (larger value) as a ubiquitous term. According to the sixth invention, only the residual value is selected as a parameter, and therefore, it is possible to select a ubiquitous term being objective. The ubiquitous term functions as an index for grouping other corpuses as well as this corpus.
A seventh invention is according to the sixth invention, and further comprises a ubiquitous term outputter for visually outputting the ubiquitous term selected by the ubiquitous term selector.
In the seventh invention, the computer (14) visually displays (outputs) the ubiquitous term selected by the ubiquitous term selector as shown in
An eighth invention is according to the fifth invention, and further comprises a document outputter for visually outputting, with respect to at least one of the unique terms output by the unique term outputter, a unit document including the unique term.
In the eighth invention, on the basis of a discriminating value (DVti) list of the morpheme (ti) produced in each time point, for example, a sum of the discriminating values with respect to unique terms (top ten words with a high discriminating value) is evaluated for each unit document included in the current corpus. At least one unit document (document) is selected as a “noticeable article” being higher in the sum of the discriminating values (RV), for example, and the selected unit document is read from the text data table (20), for example, to display at least a headline thereof together with the unique term. According to the eighth invention, at least the headline of the unit document (article) including the term (morpheme) higher in the sum of the discriminating values is displayed along with the content as necessary. This makes it possible to complement the information of a context of the morpheme lost in the analysis, and this makes it easy to understand and interpret the morpheme representing a high peculiarity.
A ninth invention is a document analyzing program for analyzing a linguistic material which increases in time series, and causes a computer to function as a corpus text producing means for producing a corpus text including text data of unit documents having a chronological order, and in which unit documents later in the chronological order are larger in number than unit documents earlier in the chronological order; a morpheme analyzing means for adding parts-of-speech information to morphemes making up of the text data included in the corpus text; an unnecessary morpheme removing means for removing an unnecessary morpheme from the text data on the basis of the parts-of-speech information; a calculating means for calculating, with respect to the morphemes which are not removed by the unnecessary morpheme removing means, a chronological incremental TFIDF for each morpheme and each unit document to obtain an actual measurement of the chronological incremental TFIDF; and a residual analyzing means for evaluating a residual value for each morpheme by performing a residual analysis between the actual measurement calculated by the calculating means and an estimate value of the cumulative total value of the chronological incremental TFIDF estimated in the previous corpus.
A tenth invention is a document analyzing method for analyzing a linguistic material which increases in time series, including steps of: a text corpus producing step for producing a text corpus including text data of unit documents having a chronological order and in which unit documents later in the chronological order are larger in number than unit documents earlier in the chronological order; a morpheme analyzing step for adding parts-of-speech information to morphemes making up of the text data included in the text corpus; an unnecessary morpheme removing step for removing an unnecessary morpheme from the text data on the basis of the parts-of-speech information; a calculating step for calculating, with respect to the morphemes which are not removed by the unnecessary morpheme removing step, a chronological incremental TFIDF for each morpheme to obtain an actual measurement of the chronological incremental TFIDF; and
a residual analyzing step for evaluating a residual value for each morpheme by performing a residual analysis between the actual measurement calculated by the calculating step and an estimate value of the cumulative total value of the chronological incremental TFIDF estimated in the previous corpus.
The ninth invention and the tenth invention are basically similar to the first invention.
According to the present invention, in accordance with the increase of the linguistic material, a corpus in which the number of unit documents is increased in chronological order is produced, and therefore, even the linguistic material, which increases in time series, can be surely analyzed or construed, so that a unique term, a ubiquitous term and etc. can be extracted therefrom.
The above described objects and other objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.
A document analyzing apparatus 10 of one embodiment according to this invention shown in
The text database 16 successively stores text data of web news in time series, acquired by the computer 14 over the network 12, and the computer 14 sequentially analyzes or construes the text data of the web news to thereby extract unique terms (keywords) which change in time series.
As an example of the unit document, in a case of the web news, articles within a predetermined time period, articles within one day, one article, one paragraph, one sentence, and etc. are cited. When a newspaper is taken as an example, one newspaper, one article, one paragraph, one sentence, and etc. are cited. In a case of a literary work (novel) or the like, there are one work, one chapter, one paragraph, one sentence, and etc.
Besides, in a case that a weblog on the web is an object to be analyzed, diary of one day may be taken as a unit document, and one inquiry, a complaint, etc. to a call center may be taken as a unit document. An arbitrary unit is defined as a “unit document” with respect to the linguistic material to thereby produce the database 20.
As shown in
When the text data table 20 is produced by the computer 14, the text data table can be produced from web news acquired over the network 12, for example, by utilizing an application installed on the computer 14, such as DBMS (Data Base Management System).
Additionally, data including text data 24 (
In the embodiment described later, some pieces of web news are tried to be used as a linguistic material body increasing in time series from which a keyword (unique term) is to be detected. However, as other linguistic materials of such a kind, data including an arbitrary time-dependency, such as a newspaper, a magazine, a weblog, an interview record, a deposition, a questionnaire, a novel, etc. can be assumed.
The analysis database 18 stores in advance all dictionaries and grammatical rules necessary for the keyword detection in this embodiment, such as a parts-of-speech dictionary for a morpheme analysis to be described later, etc., and accumulates results of the analysis. Here, this analysis database 18 may be made up of the internal memory of the computer 14 as well as the above-described text database 16.
The computer 14 extracts or detects a keyword according to a keyword extracting program as shown in
Referring to
Then, when an arbitrary set time is set by the user, the set time is stored in an appropriate memory area (register) of the computer 14, so that the computer 14 can determine whether or not the time set in the step S1 elapses by comparing the internal clock data with the set time set to the register.
If “YES” is determined in the step S1, the computer 14 next executes corpus producing processing in a step S3 to read the text data of a unit document increased during the set time (Δt) from the text data table 20 shown in
The corpus Ct shown in
Here, the “corpus” is defined as a set of written language for a language analysis, or a set of audio linguistic material, and specifically indicates ones constructed by an electronic text, and generally indicates collected ones of electronic and original text clusters. However, in this embodiment, by interpreting the aforementioned definition broadly, morpheme clusters each having information of a chronological incremental TFIDF and a TF (both are described later) with respect to the original text is called a corpus for convenience. Accordingly, it is to be understood that the text corpus, here, means a linguistic material body including text data of at least one record, that is, at least one unit document.
Succeedingly, in a step S5, the text data 24 (
It should be noted that in this embodiment, free morphological analysis software which is called “Chasen” (http://chasen.naist.jp/hiki/ChaSen/), as one example, is introduced to the computer 14 so as to be used.
Additionally, if the document is Japanese language, in this embodiment, a tool like the aforementioned “Chasen” is used such that the document is first segmented into morphemes to be extracted, and the parts-of-speech is applied to each of the extracted morphemes. However, in the language system such as English language, for example, since segmentation has already been done, morpheme extracting processing is not required, but processing of specifying the parts-of-speech is required, and therefore, tagging (discriminating the parts-of-speech) processing is performed in the step S5.
Furthermore, the morpheme (cluster) and parts-of-speech information analyzed in the step S5 are accumulated in the text database 16.
In a succeeding step S7, the computer 14 executes unnecessary morpheme removing processing in order to remove morphemes with the kind of the parts-of-speech which is set as an unnecessary term on the basis of the above-described parts-of-speech information.
That is, at a time of the morphological analysis, it is determined whether or not the morpheme should be adopted as a keyword candidate on the basis of the “parts-of-speech information” applied to each morpheme. The kind of the parts-of-speech of the morpheme (candidate of a unique term (keyword)/ubiquitous term) set as an unnecessary term is different depending on the parts-of-speech system to be output by the morpheme analyzing system and the intention of the analysis by the user. The kind of the parts-of-speech selected as an unnecessary morpheme can be decided by the user as necessary. In the experiment actually analyzed by the inventor, et al., morphemes in the result of the analysis by means of the “Chasen” which are not independent and do not take a form of suffix other than a noun, a verb, an adverb, and an adjective are rendered as unnecessary morphemes. Here, an unnecessary term removing rule about what kinds of parts-of-speech of the morpheme are to be an unnecessary term may be set in advance in the analysis database 18.
After execution of the step S7, one or more necessary morphemes remain in the corpus accumulated in the text database 16, for example. Accordingly, the processing from steps S9 to S19 is performed on each of the morpheme which are not removed and remain in the corpus. Thus, the computer 14 designates the morpheme to be processed according to the order selected by an appropriate rule in the step S9.
In the next step S11, the computer 14 evaluates the chronological incremental TFIDF with respect to the morpheme designated in the step S9. Here, the “TF” is Term Frequency, that is, a frequency (total number) (frequency of appearance) of the keyword candidate in the unit document, the “IDF” taking a parameter of the time into consideration represents an Inversed Document Frequency (the number of inversed appearing documents), that is, originality representing not appears in other corpuses. Accordingly, the “chronological incremental TFIDF” is “TF”דIDF”, may be called a Term Frequency Inversed Document Frequency, and sometimes be represented as TF*IDF, but here, it is represented as a chronological incremental TFIDF. The chronological incremental TFIDF indicates an appearance rate of the morpheme, and this is a kind of weighing index.
Even if the number of articles is successively changed as shown in
On the contrary thereto, one record dealt in the system of this embodiment has the chronological information or the order information 26 (
Accordingly, in the document analyzing apparatus 10 in this embodiment, in a case that a TFIDF with respect to a certain article dj is to be evaluated, the TFIDF is successively calculated by utilizing not the total number N of unit documents based on all the articles finally collected and the DF based thereon, but the Nj (the total number of articles before the article dj is transmitted) by taking the time based on the number of articles which has already been transmitted before the article dj into account, and DF (ti, dj) (the number documents in which the morpheme ti appears before the article dj is transmitted). In the document analyzing apparatus 10 of this embodiment, a corpus is set such that the number of unit documents included therein is increased in chronological order as shown in
More specifically, the general TFIDF is calculated in a following equation (1), and the chronological incremental TFIDF defined here is calculated in a following equation (2).
TFIDF(ti, dj)=TF(ti, dj)*IDF(ti)
IDF(ti)=log10(N/DF(ti)) (1)
chronological incremental TFIDF (ti, dj)=TF(ti, dj)*IDF(ti, dj)
IDF(ti, dj)=log10(Nj/DF(ti, dj)) (2)
The ti is, here, a morpheme having i as an identifier (ID). That is, this is a keyword candidate being an object or target for which the TFIDF (ti, dj) is to be calculated.
The dj represents the j-th unit document. That is, this is a document including a keyword candidate being an object or targe for which the TFIDF (ti, dj) and the chronological incremental TFIDF (ti, dj) are to be calculated. Here, the unit of the document can be arbitrarily set, such as a chapter, an article, a sentence, etc., and an article of the web news is taken as a document unit in this embodiment.
The TFIDF (ti, dj) and the chronological incremental TFIDF (ti, dj) are values calculated for each morpheme ti in the j-th unit document.
The TF (ti, dj) is a value calculated for each morpheme of the j-th unit document, and is the number of appearances of the morphemes ti in the unit document dj (total number).
The DF (ti, dj) is the number of unit documents that the morpheme ti appears in the first to j-th unit documents.
It should be noted that the aforementioned Nj is the number of unit documents appearing while the unit document dj occurs, and if an ID of the numerals is applied in due order to the unit documents from one (1), the value of N is actually the same value as
It is assumed that morphemes t1, t2, t3, . . . appearing in respective articles (unit documents) d1, d2, d3, . . . change as shown in
In this manner, the chronological incremental TFIDF is calculated in the step S11, and then, in a succeeding step S13, the computer 14 calculates a Σ chronological incremental TFIDF being a cumulative total value of the chronological incremental TFIDF and a Σ TF being a cumulative total value of the TF as actual measurements prior to that corpus Ct. Here, since the chronological incremental TFIDF (ti, dj) is as shown in
In a succeeding step S15, the computer 14 evaluates a constant a and a constant b by assigning the ΣTF being the cumulative total value of the TF (ti, dj) evaluated as for the corpus Ct to X, and the Σ chronological incremental TFIDF being the cumulative total value of the chronological incremental TFIDF (ti, dj) to Y of the following equation (2) to thereby produce a regression curve shown in
Y=aXb (2)
Then, the computer 14 evaluates a difference (residual value) between the Σ chronological incremental TFIDF being the cumulative total value of the chronological incremental TFIDF (ti, dj) in the corpus Ct at time j calculated in the preceding step S13 and the estimate value by the regression curve Y=aXb evaluated in the step S15 with respect to the previous corpus Ct−Δt in the step S17 (
Referring to
In the step S17, a residual analysis is performed between an estimate value or a anticipated value of the Σ chronological incremental TFIDF and an actual measurement for each morpheme, to thereby successively store the feature value, that is, the residual value for each morpheme, like adding it to the text data table 20 (
In a step S19, when it is determined that the residual analysis is ended with respect to the last morpheme, the computer 14 selects unique terms (keywords) and general words or ubiquitous terms according to the feature value (residual value) stored in the database 16 as described above in a next step S21. For example, morphemes that each of the positive residual value is an upper predetermined number ranking are selected as unique terms, that is, keywords representative of the corpus. On the contrary thereto, morphemes that each of the negative residual value is a lower predetermined number ranking are selected as general words or ubiquitous terms. The general term corresponds to the keyword representative of the entire constructed text database (linguistic material). Accordingly, if the general term is used, text data (linguistic material) with the same theme can be effectively found.
Succeedingly, the computer 14 displays the unique terms and the ubiquitous terms which are selected in the step S21 on the display not shown in a final step S23.
In the display example in
As a display example, a display of a tabular form shown in
Here, of course, another arbitrary display form can be contrived, and the display is not restricted to the display examples in
In the experiment actually made by the inventor, et al., some pieces of web news issued as to the Niigata-ken Chuetsu Earthquake (occurred at 17:56, Oct. 23, 2004. Magnitude 6.8) in 2004 were used. The reason why the Niigata-ken Chuetsu Earthquake disaster is taken as a target is that it is considered this is a relatively large-scale disaster occurred in this country after the popularization of the Internet, and this makes it possible to collect and analyze a large number of news articles.
The news articles in relation to the Niigata-ken Chuetsu Earthquake disaster delivered on the news contents of the typical portal site after Oct. 23 2004 were collected to thereby produce a database by taking a transmission date and time, a releasing newspaper office, a title (headline), a body of article as fields. A work of collecting all the articles within 24 hours from the update on the portal site is performed. The collecting period is about 6 months ranging from the occurrence of the disaster to Apr. 30 2005. The number of collected pieces of web news is 2623. On the day when the earthquake occurs, the first news articles were updated at 6:59 p.m., and 42 pieces were transmitted during that day. The day when the number of articles is the most was the next day of 24th to the occurrence of the earthquake and 179 pieces.
The text data of the web news in relation to the aforementioned Niigata-ken Chuetsu Earthquake disaster collected during the 6 months were registered as text data table 20 shown in
Thereafter, for the purpose of specifying the keyword candidate (morpheme), a morphological analysis is executed in accordance with the step S5 to study units of the term to be adopted as a keyword, and according to the step S7, units which are not proper to the keyword were removed from the units of the term decided in the step S5.
Japanese language can be segmented into units, such as a paragraph, a sentence, a segment, a term, a letter or character, etc., and the unit generally used as a keyword is a term. However, for the study of Japanese language, there is no strict definition for a term. For example, in a case of the “Niigata-ken Chuetsu Earthquake”, this can be considered as a term as it is, but this can be divided, such as (1) “Niigata/ken/Chuetsu/Earthquake”, (2) “Niigata ken/Chuetsu/Earthquake”, (3) “Niigata ken Chuetsu/Earthquake”. Since there are plurality of patterns in accordance with ideas and viewpoints, this consideration with respect to such a compound term makes it difficult to objectively specify words.
Hence, in this embodiment, it is decided to cut out words which can be extracted as a keyword by the morphological analysis generally being used.
It should be noted that the experiment dealt with Japanese language, and thus the morphemes or words are almost of Japanese language.
One example of the result of the morphological analysis is shown: “Niigata/Ken/Chuetsu/Jishin/wa/jyumin/no/raifu rain/ni/mo/zindai/na/higai/wo/oyoboshi (oyobosu)/ta/.” The analysis result in the aforementioned example (1) is output, and with respect to the morpheme taking an inflected form of a term, a basic form is also output like “oyoboshi (oyobosu)”. The morphological analysis attains accuracy of 96-98% or more at the current technical level.
The unit of the morpheme is, here, adopted as a unit of a keyword. In the unit of the morpheme, a compound term such as the “Niigata-ken Chuetsu Earthquake” cannot be gotten. However, there is no appropriate concept or definition as to a term at the present stage, and there is no analytic method for cutting a term out of the language data. The unit of the morpheme allows analysis with high accuracy, and therefore, in this research, the unit of the morpheme is made as a candidate of keyword.
As a result of attempting a morphological analysis on all the articles of the web news, 15211 kinds of morphemes (morphemes of 623765 in total) can be obtained.
Succeedingly, removal of unnecessary words is performed. In the morpheme cluster obtained by the morphological analysis, some are not fit for keywords. The words which are not fit for the keywords here indicate morphemes which do not have a meaning in themselves, like a postpositional term, such as “ga”, “wo”. Generally, such terms are called an unnecessary term (unnecessary morpheme). It is impossible to gain the meaning and the content from the unnecessary term itself.
By noting the parts-of-speech of each morpheme obtained by the morphological analysis, the removal of morphemes which are not fit for the keyword is studied from the difficulty belonging to such unnecessary terms. The parts-of-speech regarded as an unnecessary term are determined on the basis of the parts-of-speech information adopted by the morpheme analyzing system used in this embodiment.
The postpositional term (“ga”, “wo”), an auxiliary verb (“reru”, “rareru”), a conjunction (“shikashi”), and a symbol (“punctuation marks”) are the parts-of-speech having a grammatical function, but have no meaning in themselves and are not suitable for a keyword. Furthermore, the parts-of-speech which make sense by being connected to other morphemes cannot make sense by one morpheme, and thus are not suitable for a keyword. This corresponds to a morpheme which takes a non-independent form and a suffix form (“koto”, “shimau”, “rashii”), a conjunctive noun (“tai”, “ken”), a prefix (“o”, “yaku”), and a prenoun adjectival (“kono”, “sono”) out of the noun, verb, and adjective. Besides, a pronoun (“sore”, “watashi”) which indicates other words and thus cannot have a meaning of itself, and a filler (“eeto”, “unto”) for taking a rest are not suitable for a keyword as well. Furthermore, since an interjection (“ohayou”, “iie”) such as greetings, supportive responses are mainly used during a conversation, it is considered that this is less related to a disaster event.
When the aforementioned parts-of-speech is removed, morphemes which do not take a non-independent form and a suffix form out of the noun, verb, adjective and an adverb are adopted as candidates for keyword.
As a result of removing the unnecessary words on the basis of the parts-of-speech information, 15211 kinds of morphemes evaluated in the morphological analysis (step S5) are decreased to 14109 kinds (521240 morphemes in total). Out of the 14109 kinds, 1122 kinds of the morphemes (72 article) appeared from 1 to 10 hours after the occurrence of the earthquake, 3581 kinds of the morphemes (481 articles) appeared from 10 to 100 hours, 5691 kinds of the morphemes (1230 articles) appeared from 100 to 1,000 hours, and 2716 kinds of the morphemes (840 articles) appeared from 1000 to 4529 hours.
Next, according to the aforementioned equation (1), by weighing each of the extracted keyword candidates extracted from the news articles, the keyword was evaluated such that how characteristic the keyword is, or how important the keyword is as a keyword representative of the change within a certain time period.
If information on the index indicating the degree of characteristics is added to the keyword at a certain time point, a characteristic keyword can be specified on the basis of the evaluation result of the index. Thus, in this embodiment, by executing the step S11, applying an index indicating the degree of characteristics to a keyword is considered.
If a certain matter is mainly transmitted on the web news at a certain time point, a term representing the meaning of the matter may frequently appear. However, out of the keywords frequently appearing, two types of keywords can be assumed, one is keywords which are frequently used for constructing documents in any news articles, and the other is keywords which are frequently used in a part of the news articles. The keyword which characteristically represents news articles indicates the latter.
There is the aforementioned TFIDF as an index of applying a high or heavy weight to the latter keyword. As described above, when the TF (ti, dj) indicates the number of keywords ti appearing in the article dj, and the DF (ti) indicates the number of documents in which the keyword ti appears, and the IDF (ti) is an inverse number of the ratio of the number of documents in which the keyword ti appears to the total document number. That is, in this embodiment, a low or light weight is applied to a morpheme which seems to appear in any articles, and a high or heavy weight is applied to a morpheme which seldom appears in other articles. The chronological incremental TFIDF taking a product between the IDF and the TF is an index for representing how frequently the keyword appears in the article, and how rarely the keyword appears in other articles, and it can be said the that this is an index for evaluating the degree of characteristics of the keyword.
Then, in a case of evaluating a chronological incremental TFIDF with respect to a certain article dj in this embodiment, not the N and DF based on the total articles of 2623 finally collected, but the Nj (the total number of the articles before the article dj is transmitted) considering a time based on the number of articles which has been transmitted before the article dj is issued and the DF (ti, dj) (the number of documents in which the morpheme ti appears before the article dj is transmitted) are used to successively calculate a TFIDF at a time point when the article dj is transmitted. This is called a chronological incremental TFIDF.
As an example of a linguistic material body which increases in the course of time, materials in relation to a risk and/or disaster are enumerated. The linguistic material in the risk management field increases in number with time from the occurrence of the risk or disaster. A normal TFIDF takes constant N and DF, and does not respond to the weighting with respect to the morpheme extracted from the linguistic material increased in time series. In this embodiment, the total document number and the number of documents in which an arbitrary morpheme appears are regarded as parameters changing based on the chronological information to thereby use the TFIDF with modification. Additionally, if the TFIDF is thus evaluated, in a case that the TFIDF of a morpheme first appearing at a time when the article dj is issued is evaluated, the DF becomes 1, and the IDF is evaluated to be high, and a high weight is consequently applied to the morpheme which first appears. As described above, the index considering the concept of the time is called the chronological incremental TFIDF.
Here, it is difficult to evaluate whether or not the keyword is characteristic by only the value of the chronological incremental TFIDF. As a pattern in which the value of the chronological incremental TFIDF at a certain time point is highly evaluated, there are a case that even if the value of the TF is low, since the IDF is high (DF is low), the chronological incremental TFIDF is evaluated to be a high value, and a case that even if the IDF is low (DF is high), sine the TF takes a significantly large value, the chronological incremental TFIDF is calculated to be a high value. The fact the TF is significantly large is that it is highly possible that the term is, due to the high generality of the term, a term which has to be used many times for describing the articles. It is thus impossible to simply evaluate whether the keyword is characteristic by the value of the chronological incremental TFIDF.
The fact that the information at a certain time point is characteristic can be grasped from the comparison between a set of keywords which had been talked at a previous time point and a set of keywords which has been talked at a certain point. If there is a difference between them, this seems to mean that there is a great difference in quality before and after an arbitrary time point. That is, by comparing the corpus at a certain point and a corpus after an arbitrary time elapses from the certain point, it is considered that it is possible to grasp a change of the quality of the information, and specify the keyword which brings about the change.
Here, in this embodiment, as described above, by performing a residual analysis (step S17), the characteristics of the corpuses at a certain point and a next time point were compared with each other.
The functional relationship shown in
Here, the
As shown in
That is, the residual between the cumulative total value of the chronological incremental TFIDF at the certain time t and the estimate value based on the relational expression constructed by the corpus at the time t−Δt indicates the changes of the corpus itself during the time Δt, and only the morpheme with a large residual is considered to be a keyword representative of the content of the linguistic material occurring during the time Δt.
Thus, in this embodiment, as an index for evaluating a feature amount of a keyword indicating the change in the quality of the information content at the time t, a difference (residual) is adopted between the estimate value of the cumulative total value of the chronological incremental TFIDF by the relational expression based on the TF constituted of corpuses at an arbitrary time period t−Δt and the cumulative total value of the chronological incremental TFIDF, and an actual measurement of the cumulative total value of the chronological incremental TFIDF at the time t. The keyword taking a markedly high residual is here called a characteristic term or a unique term (residual value: positive), and the keyword taking a markedly low residual is called a general term or a ubiquitous term (residual value: negative).
According to a process shown in the flowchart in
In this manner, in the document analyzing apparatus 10 shown in
1) A database of text data (some pieces of web news in this case) increasing in time series is constructed.
2) Each text is segmented into morphemes to which parts-of-speech information is added.
3) On the basis of the parts-of-speech information, nouns, verbs, adverbs, adjectives except for the non-independent form or the suffix form thereof are extracted.
4) The TF and the chronological incremental TFIDF based on the chronological information with respect to morphemes for each document (web-news article, here) are evaluated.
5) In order to extract keywords representative of characteristic texts from the time t−Δt to the time t, a relational expression between the cumulative total value of the TF and the cumulative total value of the chronological incremental TFIDF in the corpus until the t−Δt is evaluated, and the difference between the estimate value and the actual measurement of the cumulative total value of the chronological incremental TFIDF at the time t is evaluated based thereon. This residual value is regarded as a feature amount of each of the keywords which appears during the time Δt.
6) The keywords in arbitrary upper ranks from the largest residual value are selected, and with respect to the articles in which the keywords are detected, the keywords are taken as meta data of the linguistic material.
The system in this embodiment is intended to be applied to pieces of web news taking up the Niigata-ken Chuetsu Earthquake disaster in 2004.
According to the model of the course of the disaster which has already been implemented by carefully taking an ethnography from a microscopic viewpoint as to the actions of the victims directly after the occurrence of the disaster of the Great Hanshin Awaji Earthquake, it is said that with respect to the course of the disaster, a condition is changed in quality according to a power of 10, such as 10 hours, 100 hours, 1000 hours. The period from 1-10 hours is said to be a disorientation period or a period of disaster during which it is impossible to grasp what happens due to the drastic changes in the environment by the disaster, and the next period from 10-100 hours is a formation period of a society of a disaster area during which activities of saving life, an establishment of shelters, and the like are performed. The period from 100-1000 hours is a period during which the society of the disaster area is maintained, a flow of the society is restored, and the life of the victims of the disaster is stabilized. The period from 1000 hours onward corresponds to a period returning to the reality during which a reconstruction of a social stock is performed.
With reference to the model of the course of the disaster, a keyword detection was tried by setting the Δt to be used in the keyword detection to 1 hour, 3 hours, 8 hours, 8 hours, 24 hours, 24 hours, and 24 hours in respective seven phases, such as 1-10 hours, 10-100 hours, 100-500 hours, 500-1000 hours, 1000-2000 hours, 2000-3000 hours, and 3000-4500 hours.
In order to more observe what kinds of keywords detected in
By noting the keywords in associated with these activities in
Next, with reference to
Lastly, a similar analysis is intended as to the activities of reconstructing the social stock.
From the above-described consideration with respect to the keywords about which the residuals are positive, the keywords assumed in the theory of the course of the disaster on the basis of the result of the ethnography search in the disaster area of the Great Hanshin-Awaji earthquake occurring in 1995 and the linguistic analysis relating to the news articles taken in the WTC terrorist attack in 2001 are characteristically detected for each time phase, and in the analysis result utilizing the web news of the Niigata-ken Chuetsu Earthquake disaster in 2004, a conformity to the model of the course of the disaster in which a disaster process changes in quality by taking the time of a power of 10 as a milestone was confirmed.
Furthermore, each of the sets of keywords shown in
Some keywords which are not shown in
In accordance with this, the feature amount of the keyword like “volunteer” in relation to the activity for supporting a snow-removing work becomes high again. In a case of the Niigata-ken Chuetsu Earthquake, as the “dam”, the “drain”, the “snow removal”, and the “snow clearing” are detected, it became evident that an influence of a secondary disaster by a natural hazard except for the earthquake, such as an influence of the landslide disaster due to a rainfall occurring after the main quake and a risk of breaking a building due to a heavy snow are taken characteristically.
Although inappropriate words such as the “same city”, the “current time”, and the “possible” which are not fit for the keyword are partly detected, since the keywords representative of each phase from the occurrence of the disaster to the reconstruction are detected as in the aforementioned study based on
If visualization (monitor display) is performed by utilizing the feature amounts of the keywords as shown in
Additionally, in the aforementioned embodiment, the text corpus is produced for every set time (S1, S3). However, the text data increasing in time series is accumulated in the text database 16, and a text block, that is, a corpus may be demarcated every lapse of an arbitrary duration Δt.
As described above, the analysis technique of this invention is, as to the appearing distribution words, of comparing the corpus Ct at an arbitrary time point and the corpus Ct−Δt extended back by the Δt from that time point, and extracting a unique term whose appearing characteristic is significantly different between the t−Δt and the t as a unique term. Thus, if a term different from the words of the corpus increasing in time series appears during the Δt, a discriminating value for measuring the peculiarity indicates a high value.
In the analysis technique (algorithm) in this invention, if the discriminating value indicates a high value, two patterns below can be assumed. One is a case that a document (article) which is highly associated with this art at the time point t and includes a lot of words being highly associated with this art is added to the corpus, and the other is a case that a document which is not so highly associated with this art in that point t, and includes words being lowly associated with this art is added to the corpus.
For example, with respect to the web news corpus in relation to the Niigata-Chuetsu Oki (offshore) earthquake in 2007 analyzed by the inventor, et al., in a set of the feature articles, in the news reporting the result of the elimination matches of All-Japan Senior High School Baseball Championship Tournament, the results of the past games of the high schools in Kashiwazaki City being a main disaster area were placed, and therefore, these were added to the corpus. In these articles, the results of the past games played in that day of all the high schools in the Niigata Prefecture are also placed other than the results of the past games of the high schools in Kashiwazaki City. In the results of the past games, a lot of descriptions, such as “×× of two-base hit, ×× of three-base hits” are included, and the morphemes of “two-base hit” and “three-base hit” indicate significantly high discriminating values.
In the latter case, a high discriminating value may be applied to a term being less associated with this art of the corpus increasing in time series, so that the possibility of sometimes causing the user to erroneously understand the news cannot be denied.
Here, in another embodiment of this invention shown in
In addition, the present invention is for performing an analysis of a unique term (keyword) by using a morpheme as a unit and visualizing it. A defect of the analysis by taking a morpheme as a unit is that the information on the context that each morpheme (unique term) essentially has is lost, and this makes it difficult to understand and interpret what the term with a high peculiarity represents. Thus, in this embodiment below, a technique of complementing the information on the context by displaying an article to be noted, and supporting the understanding and interpretation of the analysis result is proposed.
In
Here, in this embodiment, before starting the operation in
Next, after execution of the processing until the step S17, the computer 14 stores, in the memory of the computer 14, the frequency of appearance TF (Δt, ti) of the term (morpheme) during the time period Δt and the number of documents (articles) in which the term (morpheme) appears DF (Δt, ti) within the time period Δt in the format in
Here, these frequency of appearance TF (Δt, ti) and the number of documents in which the term appears DF (Δt, ti) are not used if the user does not select the filtering as an option. In this case, “YES” is determined in a step S20A, and unique terms and ubiquitous terms (general term) are selected in a step S21 in a manner the same as the step S21 in
When the filtering option is set, “NO” is determined in step S20A, and therefore, in a succeeding step S20B, the computer 14 determines whether or not the filtering flag is “1” with reference to a flag area of the memory (not shown). The fact that “YES” is determined in the step S20B means that the filtering 1 is selected as an option, and the fact that “NO” is determined means that the filtering 2 is selected as an option.
If the filtering 1 is selected as an option, the computer 14 selects unique terms and ubiquitous terms by the filter 1 in a next step S21A.
More specifically, with reference to the data of the number of documents in which the term appears DF (Δt, ti) in each time period Δt stored in the step S18 in the memory in
If the filtering 2 is selected as an option, the computer 14 selects unique terms and ubiquitous terms by the filter 2 in a next step S21B.
More specifically, the number of documents in which the term appears DF (Δt, ti) and the frequency of appearance TF (Δt, ti) which are stored in the step S18 are read, and a regression curve (
Here,
The graph display in the step S23 in a case that the filtering 2 is selected is as shown in
After the analysis result is visually displayed in the step S23, the computer 14 determines whether or not the noticeable article displaying flag is “1” with reference to the memory in a step S25. If “NO”, the process is directly ended, but if “YES”, a displaying step of the noticeable articles on the monitor 15B is executed in a step S27.
More specifically, when a residual value is evaluated in the preceding step S17, a list of the discriminating value DVti of the term ti is produced at each time point, and therefore, a sum of the discriminating values (RV=ΣDVti) is evaluated for each document as to the unique term (the top ten words with a high discriminating value) included in the document in the time Δt. Then, the top three documents being high in the sum RV of the discriminating value are selected as “noticeable articles”. With respect to the selected “noticeable articles”, unique terms (top 10) included in at least the headline and the content are displayed as shown in the Table 2.
Which document the morpheme ti listed up in the aforementioned discriminating value list is included in can be specified by referring to the text data table 20 shown in
In the table 2, with respect to the two articles including two words “active” and “earthquake resistant” each having the sum of the discriminating values RV “19.0” and the one article including one term “telephone” having the sum of the discriminating values RV “12.7”, at least the headline, preferably including the content, is displayed. This makes it possible to complement the information of the context of the morphemes lost by the analysis, and thus avoid difficulty of understanding and interpreting what the term showing a high peculiarity represents.
Here, in the above-described embodiment, with respect to the top three morphemes being high in the sum of the discriminating values RV, the “articles” including them, that is, the unit documents are displayed, but the number of morphemes about which the article is displayed is arbitrary. With respect to only the top morpheme, the article (headline) including this may be displayed, and with respect to the top ten morphemes, the articles and the headlines may be displayed.
Additionally, in order to visually output the selected unique terms and general words, these are displayed on the monitor in this embodiment, but in place of the display or in addition to the display, a printout by a printer, for example, may be possible.
In
Although the present invention has been described and illustrated in detail, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation, the spirit and scope of the present invention being limited only by the terms of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2006-315238 | Nov 2006 | JP | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/JP2007/073257 | 11/22/2007 | WO | 00 | 5/20/2009 |
