Patent Grant
7042434
The present invention relates to a system for analysing textual data. In particular, the analysis results in revealing information hidden in textual data. Such a system can be used in a method for developing consumer products, or for better targeting the marketing consumer products, when the textual data that is to be analysed is obtained from consumers.
Developing and marketing new (consumer) goods, and in particular goods in the fast moving consumer goods area, involves in many cases at some stage obtaining information from consumers as to what sort of products they want, and/or obtaining information from consumers to effectively target marketing and sales, and/or research during the product development process, and/or monitoring the consumer appraisal once the product is/has been evaluated before or after being launched. Gathering such information can be done in many different ways, e.g. by interviewing individuals or groups of consumers, feedback after purchase, spontaneous email (e.g. from consumers to care-lines of product websites), questionnaires call for free-text answers, et cetera. Usually such information is obtained in a textual format or can be translated into that.
Whether the information appears directly expressed or implicit in the text, it may be desired to perform some analysis or information extraction and/or interpretation to obtain this information. Existing examples of information extraction from factual documents (e.g. reports, scientific prose, news feeds, legal texts, etc.) are based on the recognition of entities and events. In order to fill pre-defined templates, the linguistic elements which represent the entities need recognising, the syntactic-structures in which the entities are embedded need disentangling, and the nature of the semantic links between entities need interpreting.
Although in principle similar information extraction techniques could be attempted with unstructured subjective textual data, the infinite multiplicity of the modes of expression used by individuals on subjects such as personal issues, opinions, beliefs and habits does not make these methods wholly appropriate and does not give satisfactory results. Examples of subjective or unstructured text are literature, free-text questionnaire answers, interviews and loosely-directed monologues, focus-groups interactions, spontaneous communications, etc. The information which is to be revealed are not events or entities, but rather qualifiers, concepts, opinions, etc. as well as the characteristics of a specific linguistic expression.
There are also techniques known for analysing textual data which can be grouped under the header linguistic analysis techniques. Examples of such techniques are extraction of linguistic units or linguistic features from textual chains, analysis of lexical semantic patterns, analysis of collocations and coherent groups, analysis of co-occurrences and conceptual chains, analysis of affect, and others. Such techniques are discussed in:
While such information extraction or linguistic analysis techniques may provide valuable information, there is more information which is hidden in textual data which is not revealed by these methods.
A wide variety of statistical methods are available for extracting information from numerical data. Such statistical methods can be anything from simple counting to more advanced statistical techniques, e.g. dimension reduction, clustering, hypothesis testing, model fitting, correlation and others. Various such statistical techniques are described in:
While such statistical methods may provide information from numerical data, they are not directly applicable to textual data.
New profiling and segmentations of the consumer population are being sought, which are no longer based on demographics but on new drivers like life style, educational attainment, attitudes to the environment, health issues, social preferences, etc.
Hence, there was a need for a tool or system that can be used to analyse unstructured, subjective or personal text.
It has now been found that the above may be achieved (at least in part) by a system for revealing information by subjecting textual data to:
a) a module for entering textual data
b) a module (A) for linguistic analysis for analysing said data,
c) a module (B) for statistical analysis of the results of module (A).
Preferably the system of the invention is an electronic system for example a computer system
In this, the statistical analysis is understood not to relate just to simple counting and calculating averages and percentages. It will be clear to the person of average skill what statistical analysis is and what is to be understood as linguistic analysis. There are also examples of linguistic analysis tools that have some basic statistical analysis incorporated therein, and some statistical tools which have a minimum of linguistic analysis embedded in them. For example, word frequency counts which define a word as a string of characters comprised between two successive spaces (system with minimal linguistic analysis); for example, customised text editors which manage manually-defined labels indexing the semantic contents of portions of documents and return findings on the relative importance of certain themes across a number of texts (system with minimal statistical analysis). For the purpose of this invention, these will still be called linguistic tools, and statistical tools, respectively, i.e. independent of the presence of a bit of the other technique in them.
RU 2107950 XP 002181209 (Epodoc) discloses the use of acoustic and linguistic analysis and statistic criteria for decision making. This document does not indicate the advantages of first applying a linguistic analysis and subsequently applying a statistical analysis on the results thereof.
It has now been found that statistical analyses may be performed on data that is obtained from a preceding linguistic analysis of textual data, and that such a combination of techniques yields information from unstructured, subjective or personal texts not available so far.
In order to practice this invention it is preferred if such techniques are present as modules in a system, especially because there is a wide range of both linguistic and statistical techniques to choose from. Of course, such a system will contain some form of module to enter the textual data to be analysed. A system according to the invention may also comprise a module to present the output, but it may also be possible that the module that provides the statistical analysis provides the output itself.
In the system according to the invention, the linguistic analysis is preferably achieved by subjecting the textual data to one or more of:
a) extraction of linguistic units or linguistic features from textual chains
b) removal of stop-words [list re-ordered]
c) analysis of lexical semantic patterns
d) analysis of collocations and coherent groups
e) analysis of co-occurrences and conceptual chains
f) analysis of lexical complexity, co-reference
g) analysis of affect
h) tagging units in the textual chain yields: Part-Of-Speech tags, semantic descriptors, other indexing tags, syntactic tagging, functional tagging
i) analysis of syntactic complexity
If one or more of the techniques c) to i) is chosen, it is preferably applied after a preliminary extraction of linguistic units or linguistic features from textual chains (a) and/or the removal of stop words (b). The extraction of linguistic units or linguistic features is preferably done in such as way that it yields one or more of:
A1 letters
A2 morphemes
A3 words (e.g. simple, compound), lemmas & word classes (POS)
In the system according to the invention the statistical analysis is preferably achieved by subjecting the results of module (A) to one or more of:
These statistical techniques are known as such.
It has been found that it may be preferred to analyse texts by a combination of techniques according to the invention in which the linguistic analysis is the extraction of linguistic units or linguistic features and the statistical analysis is dimension reduction. Alternatively, it may be preferred to combine the extraction of linguistic units or linguistic features (as linguistic analysis) with clustering as statistical analysis. Another preferred combination is wherein the linguistic analysis is the analysis of co-occurrences and collocations and the statistical analysis in hypothesis testing.
The system according to the present invention is in particular suitable for textual data which are captured from one or more of:
Although the origin of the text may be anything as set out above (and even other), the physical format of such textual data to be analysed is one or more of:
The system according to the invention may have any suitable physical form. Given the amount of information to be processed the module for entering textual data preferably comprises means for getting the textual data into an electronically readable format and has optionally further one or more of the following functionalities: concatenating or splitting files by respondent or by question, creating tagged or lemmatised sentences, all manner of string/line/tagged sequences searching and replacement, multi-criteria retrieval. The information may be stored using electronic storage means, e.g. in the form of databases, computer readable data carriers or other.
The system according to the present invention may further contain a module for linguistic post processing after module (B). In that case, such linguistic post-processing comprises preferably one or more of
The output, either with or without linguistic post processing, may be generated in any suitable form. Examples are graphs, tables, lists, computer-screen projections, networks, webs, trees, dynamic lay out graphs, hard copies thereof.
As set out herein before, the invention provides a system for analysing textual data. Although analysing single sets of textual data can be desired for some purposes, it may provide further insight in information if two or more different sets of textual data are analysed using a system according to the present invention and the two or more outputs may subsequently be compared. Likewise, it may be preferred to compare the revealed information obtained using a system according to the invention with existing numerical or textual data. For example, when using the system according to the invention for improving the development of consumer products, such existing numerical or textual data may comprise or relate to one or more of physical and/or chemical properties of products and/or packaging, physical measurements, demographic information, quantitative questionnaire results, sensory panel results, consumer behaviour, time/date information.
Although the system according to the present invention may be used for analysing existing textual data, the system can also be adapted such that the textual data to be analysed are captured and analysed in real time. For specific purposes, it may also be preferred to feed back the obtained output using the system according to the invention to the individuals who produced the original textual data.
Example 1
Table 1 shows the snow reports taken from the BBC online website (http://www.bbc.co.uk/weather/sports/features/skiing_text.shtml) on the 6th March 2001. The description for Scotland was excluded from the analysis as both the presentation and information contained within the report were judged to be very different to those used for the other countries.
These data from table 1 were separated into linguistic units of words (groups of characters separated by white spaces or punctuation) and stop words were removed using a standard list (which in this case was taken from the webpage: http://www-fog.bio.unipd.it/waishelp/stoplist.html).
These were formed into a words-country frequency matrix showing the number of times each word was used in describing the conditions present within each country. A subset of this matrix (as an example, whole matrix is too big to display) is shown in Table 2.
The dimension reduction technique, correspondence analysis (as described in Greenacre (1984) Theory and Applications Of Correspondence Analysis, London: Academic Press) was then applied to this frequency matrix and the scores for the first two dimensions extracted for both the countries and words.
These results can also be related to other sources of data. For example, the snow reports also contains measures of the depth of snow. If we take an average of these values for each country as a measure of the average depth of snow present in resorts in that country then we can relate this data to the output from the textual analysis.
The results from the correspondence analysis may alternatively be related to other measures such as temperature or altitude. If a series of reports were collected over time, time series modelling techniques may be used. Alternatively, given other relevant information, model building approaches to develop predictive models may be used, for example to be able to model visitor satisfaction or resort income with the resort as a function of the language used in the snow condition reports.
Example 2
An alternative embodiment is illustrated below using the same data set as in Example 1 (table 1). The data is subjected to the same linguistic analysis as in the previous embodiment. Distances between countries were then calculated using Jaccard coefficients, i.e. for each pair of countries the fraction of the words used in the description of conditions for either country which were used in the description of the conditions in both companies were calculated. This gave a similarity matrix shown in table 3: each pair of countries has a measure of similarity between 0 and 1, where a similarity of 0 would mean that the reports from the two countries have no words in common and a similarity of 1 would mean that the sets of words used in the reports for the two countries were identical. This was then subjected to a cluster analysis using Ward's method (SAS/STAT User's Guide, 1990). Choosing a four cluster solution gave us clusters of countries {America, Canada, Germany, Switzerland}, {Andorra, Austria, France, Italy}, {Norway} and {Bulgaria}.
These clusters identify countries experiencing similar snow conditions reflected in the use of similar language in their snow reports. An idea of the conditions represented by each of these clusters can be obtained by observing the words used in the descriptions of the conditions of the countries in the cluster which are used infrequently elsewhere. For example, the words “lift”, “great” and “open” are used in all of the reports in the countries {America, Canada, Germany, Switzerland} cluster but only once or twice elsewhere, the conditions in the countries in this cluster are characterised by good skiing conditions. The words “rain” and “wet” are used in all of the reports in the {Andorra, Austria, France, Italy} cluster but not in any other reports, characterising the conditions in these countries as being spoiled by rainy conditions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 01-53027 | Aug 2001 | KR | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4659183 | Suzawa | Apr 1987 | A |
| 5907222 | Lengyel et al. | May 1999 | A |
| 6388388 | Weindorf et al. | May 2002 | B1 |
| 6636190 | Hirakata et al. | Oct 2003 | B1 |
| Number | Date | Country |
|---|---|---|
| 10148808 | Jun 1998 | JP |
| 10-203715 | Jul 1998 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20030043106 A1 | Mar 2003 | US |
