Patent Application
20080133508
The present invention relates to a search system and, in particular to a search engine which can model a large data set, such as web pages on the Internet, and efficiently respond to natural language queries.
Internet search engines are continually being developed to provide succinct and useful responses to queries. The task is made extremely difficult by the wide variety of data that forms the pages of the web sites accessible on the Internet using the hypertext transfer protocol (http). The search engines allow free text queries of the web pages, and generally the engines use the words of the queries for direct comparison with words held in data sets that represent the web pages. The data sets may maintain simply keywords or meta data representative of the web pages, or a more extensive list of the words of a page, as for AltaVista™. Although the engines may omit common words, such as “the”, “a” and “of”, the direct word comparison with the words held in the representative data sets tends to give rise to a large and unmanageable number of responses, or “hits”, to a query. The accuracy and quality of the search results generated are problematic. The search engines also take no account of the actual meaning of the words in relation to the query, or the grammatical structure of the query. It therefore is desired to provide a search system or engine which alleviates these problems or at least provides a useful alternative.
In accordance with the present invention there is provided a search system, including:
query means for processing a query to assign respective weights to terms of said query and to generate a query vector including said weights; and
index means responsive to said query vector to output at least one index to data in response to said query.
The present invention also provides a search engine stored on a computer readable storage medium, including:
a query analyser for processing a query to assign respective weights to terms of said query and to generate a query vector including said weights; and
an index network responsive to said query vector to output at least one index to data in response to said query.
Preferred embodiments of the present invention are hereinafter described, by way of example only, with reference to the accompanying drawings, wherein:
a to 4e are diagrams of nodes generated during training of an index network.
A search system, as shown in
The NLQ analyser 4, as described in detail below, is able to process natural language text queries to extract only the terms of the query which are definitive of the query and most important for obtaining a satisfactory response. The NLQ analyser 4, as described in detail below, analyses the grammatical structure of a query and assigns specific weights to terms which are considered important to the query. The terms, being words or language elements, which are retained are also determined using the terms and synonyms maintained by the dictionary and thesaurus 10. For example if the query is “I want information on red cars” the NLQ analyser 4 deletes the terms “I want information on” and retains the terms which define the subject matter of the query being “red cars”: The NLQ analyser 4 recognises the term “red” as an adjective and the term “cars” as the plural of the noun “car” and will assign a greater weight to the substantive noun rather than the adjective. The query vector has fields or parameters which represent every respective word held in the dictionary 10 of terms for the data 14. Once the weights are assigned for a term that weight is placed in the appropriate field in the query vector. For example, the term “red” may be assigned a weight of 0.5, whereas the term “car” may be assigned a weight of 1.0. Using this weight and vector structure, the NLQ analyser 4 can distinguish queries on the basis of syntactic structure and pass the information to the SGNN 6 in the query vector. For example, the query “glass of wine” can be distinguished between “wine glass” by assigning different weights to the terms “wine” and “glass”.
The SGNN 6, as described in detail below, has a neural network structure where each of the nodes has a vector of the same format as the query vector but which is assigned weights that represents respective text of the data 14 or the child nodes of succeeding levels. The nodes which have vectors representing a respective text at a respective location also retain the index for that location as a data item. The query vector is processed by the SGNN 6 by beginning at the upper node of the network 6 and obtaining inner products between the query vector and a node vector of a node. The nodes of the network 6 are traversed only along paths where the inner products exceed a predetermined level until the highest inner products are obtained. The nodes corresponding to the highest inner products and their indices are returned to the clusterer 8 in order to formulate the response and clear content categories.
A search system or engine for data on the Internet 20 has the architecture shown in
The operational module 24 is essentially an on-line module of the search system and includes the following software components:
The NLQ analyser 4 receives a short passage of text (typically a sentence or two) as the query from the interface 34 and generates a structured vector suitable to pass to the SGNN searcher 36 as a query. For retrieval from a text database in English, the main task of the NLQ analyser 4 is to make use of the structure and linguistic patterns of the English text to identify terms, such as keywords/key-phrases, and assign linguistic weights to them to reflect their importance to the query. The actual weights are often application dependent, but generally those types of terms which normally carry the most important information (e.g. a noun phrase in a query as a simple example) are assigned the highest weights, while less important terms are assigned lower weights.
The NLQ analyser 4 is based on the assumption that when a person reads a sentence, he/she reads it a section at a time. For example, the sentence “I want to buy a car in Clayton” can be read:
[I want to buy] [a car] [in Clayton]
From the viewpoint of speech recognition, these sections correspond in some way to prosodic patterns. From the viewpoint of linguistic analysis, these sections represent the local linguistic structures of the sentence. Several observations about the sections are:
Once low level sections are recognised, they can be grouped together to form higher level sections. For example, the sections [I want to buy] and [a car] can be grouped together to form a bigger section [I want to buy a car]. Such a process can be repeated and eventually all the sections can be put together to represent the original sentence.
To execute the above process, the NLQ analyser 4 addresses the following issues:
The above issues are further compounded by the fact that the NLQ analyser 4 has to deal with a large vocabulary in the absence of semantic knowledge.
To address the issues the NLQ analyser 4 executes a new set of techniques in the field of natural language processing, which are described below.
A data structure, referred to herein as a “key-centred phrase structure frame” is adopted to recognise sentence sections. An example of such a frame is given below for a noun phrase (NP):
NP−>det adj*noun (1)
The frame for a noun phrase section includes a determinant (det), an adjective (adj) and a noun. The category in the frame preceded by an asterisk, “noun” in this example, is the key category that will match a query term in the section. The categories are determined by accessing a dictionary 30, as described below. For a normal parser, categories in the right hand side (RHS) of a grammar rule are matched from left to right. With a key-centred phrase structure frame, however, the key category is used as an anchor to attach the frame to the word that matches this category in a sentence. Also, each element in the RHS of the frame acts as a word holder, once matched, it will hold the word that matches it plus the additional information about the word such as its morphological information. Some of the elements in the RHS of a frame may be labelled as “repeatable” or “omissible”. The main advantage of this type of knowledge representation for identifying sections is to allow the NLQ analyser 4 to focus on the key information in the sentence that is possibly useful for the retrieval process.
Once a frame is attached to a word in the original sentence of a query, the key category of the frame is aligned with the word. A tolerant bidirectional pattern matching mechanism is then applied to capture the local structures that involve that word and collect information necessary for later stages.
The pattern matching process starts from the word(s) adjacent to the word to which the frame is attached. It collects the words towards the two ends of the sentence, trying to match them with the rest of the categories in the frame. This is referred to herein as “frame instantiation”. The pattern matching is tolerant to structural errors in the sense that if a word in the sentence does not match the category under consideration, an attempt will be made to match the word with next category. This frame instantiation process also records additional information used by later processes, which includes:
After a new frame has been instantiated, it is compared with the existing instantiated frames to prune any redundant frames. For example, a noun phrase frame is considered to be redundant if it shares the same key word with another noun phrase but the covered-length and/or preference is less than the latter. This step can sometimes significantly reduce the number of frames to be considered in later stages therefore making the search system more efficient.
Usually, the grammar rules of a NLQ parser can be applied in any order. However, the NLQ analyser 4 adopts a partially ordered frame invocation scheme that allows the application of the frames in a flexible yet controllable manner. This is based on the understanding that the frames can be classified into different classes according to their roles. For example, a frame whose RHS contains only terminal categories such as the NP given above is used to recognise the lowest level sections. Once these lowest level sections have been recognised, another type of frame may be used to extend these sections, for example, using the rule
PP−>prepNP (2)
to extend a NP into a PP (preposition phrase) in a “bottom-up” manner. A frame whose RHS consists merely of non-terminals belongs to yet another group, and is used in a top-down analysis to enforce expectations for the overall structure of the query. After defining these groups, they are applied in a particular order to gradually expand the sections from lowest level to highest level. However, with the frames in the same group there exists no order, they can be applied in any order. This frame invocation scheme provides control over how the frames are applied yet still preserves a great deal of flexibility.
As mentioned above, due to the multiple senses many English words have, the pattern matching process often produces multiple instantiated frames for the local structures of the same term, some of which are pruned immediately after the matching process, but some still remain. Different ways of linking these frames lead to different interpretations of the sentence structure. To facilitate the selection of the best interpretation, a data structure referred herein as a “frame relation graph” is used. The frame relation graph is an acyclic, directed graph which includes nodes, weights and links defining the relationships between the nodes. The nodes in the graph are the frames that have been instantiated by the terms or words in the sentence. As each instantiated frame represents a section in the sentence, terms “node”, “frame” and “section” can be used interchangeably below in describing the NLQ analyser 4. The nodes in a frame relation graph can be separated into the following three classes:
When linked by the arrows, a start node, one or more intermediate nodes, and an end node form a path consisting of a sequence of frames that cover some lowest level sections. For example, in
The first and second rules together ensure that only non-overlapped frames can appear in the same path, and only two nodes that represent two adjacent instantiated frames (i.e. two instantiated frames attached to the sentence adjacently) can be linked by an arrow in the graph.
Considering a very simple example of the frame relation graph, suppose a sentence contains the words w1, . . . , w10 in that order, and eight frames with the following coverage:
Here the second frame is designed to match a phrase such as “a large bottle of red wine”. The frames f4 and f5 given above correspond to this situation. They represent competing frames of which only one will be selected eventually.
Given the different paths in a frame relation graph, a strategy referred to herein as “parallel parsing and best-fit selection” is executed by the NLQ analyser 4 to decide the path that best represents the overall structure of the sentence. The parallel parsing process parses each of the paths in the graph. The parsing method executed is similar to the frame instantiation described above, and employs both “bottom-up” and “top-down” frames, with the following differences:
There are three possibilities for the result of the parallel parsing process:
In the first case above, the path that gives the complete parse is selected to provide the output of the NLQ analyser 4 for a query. In the second case, the output path is selected based on the following values:
In the third case, the original sentence is structurally ungrammatical according to the grammar rules of the NLQ analyser 4. The following additional value is then determined to select the best-fit path as the output:
In the second and third cases, if there is still more than one path selected after considering the values of the given parameters, which is very rare, a path is arbitrarily selected as the output when no other knowledge (e.g. contextual or domain knowledge) is available to perform further comparison.
The NLQ analyser 4 produces for any given query a nodal tree structure. The root of the tree represents the complete sentence. The terminal nodes in the tree represent the frames appearing in a selected path of the frame relation graph of the query. The intermediate nodes show how these frames are linked together to cover the complete sentence. The linguistic categories of these frames and the words contained in them, such as NP, VP (verb phrases), prep, verb, adj and noun, are used by the NLQ analyser 4 to generate weights for the terms of the sections. The thesauri 32, as described below, is used to retrieve synonyms of the terms. The query vector containing the terms, their synonyms, and their weights is then generated by the NLQ analyser 4 and passed to the SGNN searcher 36. The query vector can have the following format:
As discussed above, the NLQ analyser 4 uses key-centred phrase structure frames to capture the local structures of a user query around some potentially useful words/phrases, and then applies parallel parsing and best-fit selection to the frame relation graph to select the best interpretation of the overall structure of the query. The NLQ analyser 4 offers robustness at different levels of query analysis in an extremely simple yet effective way and with minimum syntactic knowledge. This makes the NLQ analyser 4 ideal for NLQ analysis in which both grammatical queries and ungrammatical keyword-based queries are common, and also attractive for rapid development of NLQ interfaces for large scale data sets of electronic information systems, such as the Internet. The NLQ analyser 4 is also designed in such a way that it is able to accommodate higher level knowledge such as semantic or contextual knowledge, if it is available, so that it can be applied to the applications in which an integrated NLQ analysis is required using syntactic, semantic, contextual, or even pragmatic knowledge together. With its close association with prosodic pattern recognition, the NLQ analyser 4 can also be applied to spoken language understanding tasks as well.
The dictionaries 30 include about 25,000 entries covering more than 20 parts of speech. Most of the entries contain only the root form of a word such as “car”, “operate”, etc. A morphological analyser is used to deal with various inflected forms of a word such as “cars” for “car” and “operated” for “operate”, which expands the number of words recognisable by the NLQ analyser 4 to over 42,000. This is adequate for ordinary size general domains. For speciality domains, e.g. medicine, the dictionaries can be enhanced by domain specific terms, most of which will be nouns.
The thesauri 32 contain three manually generated thesaurus files: “thes.noun”, “thes.verb” and “thes.adj”. The number of entries in each of the thesauri may be for example:
The word in an entry to which a set of synonyms is attached is called a headword. In the file thes.noun, all headwords are nouns. Similarly, in the files thes.verb and thes.adj, all headwords are verbs and adjectives respectively. A headword with multiple senses is listed in multiple thesaurus files. For example, the word “rent” which can be used as both “noun” and “verb” is listed in both thes.noun and thes.verb. In thes.noun, the word “rent” is associated with the synonyms which have similar meanings when it is used as a noun, for example, “payment, rental”; in thes.verb it is associated with the synonyms which have similar meanings when it is used as a verb, for instance, “hire, lease”.
Most of the headwords of the thesaurus files are carefully selected from the Webster's Thesaurus published by Book Essentials Publications, Larchmont, New York. This relatively small size thesaurus is used because it contains commonly used words and their common synonyms. However, the book does not give parts of speech of the headwords selected. Therefore, after selection, the words are checked against the Oxford Advanced Learner's dictionary to find their parts of speech and separate them into the above three different files.
The synonyms in each entry are selected by consulting both the Webster's Thesaurus mentioned above and The Macquarie Thesaurus published by The Macquarie Library Pty. Ltd. Not all the synonyms in those two thesauri for a given headwords are selected because a too rich thesaurus can cause over-expansion problems in the NLQ analyser 4 rather than assist. The average number of synonyms for an entry is about four words, with the maximum being five.
For each entry, the synonyms for the headword are ordered according to their semantic similarities to the headword. This process is somewhat subjective but is quite effective when these files are used. The ordering of the synonyms facilitates the selection of different number of synonyms for different applications.
The set of thesaurus files provides a good basis for developing application specific or domain dependent thesauri, which can be derived by editing these files during testing of the system.
A self-generating neural network (SGNN) is a hierarchical clustering network based on both concept formation, as described in K. McKusick and P. Langley, “Constrains on Tree Structure in Concept Formation”, In Proc. IJCAI'91, Vol. 2, Sydney, August 1991, pages 810-816, and self-organizing neural networks, as described in T. Kohonen, “Self-Organization and Associative Memory”, Spring-Verlag, 1989. The SGNN is a hybrid of conventional AI and neural networks, and exhibits the advantages of both. The SGNN:
A tree structure SGNN, i.e. a self-generating neural tree (SGNT) is described below.
In describing the SGNT generation procedure, the following definitions apply:
W
j
=<w
j1
, . . . , w
jn>,
∀j,d(nk,ei)≦d(nj,ei),
where d(nj, ei) is the distance between neuron nj and instance ei.
Any distance measure can be used. For example a modified Euclidean distance measure as follows:
where ρk is the weight for the k-th attribute.
The SGNN 6 executes the following algorithm, described in pseudo-C code language to generate the index network of the SGNN 6. The algorithm is a hierarchical clustering algorithm.
Input:
Output:
Method:
where the routines are defined as follows:
according to the updating rule below.
where wjk,i is the k-th weight of nj after we have seen the first i examples which are covered by nj and ai,k is the value of the k-th attribute of i-th example covered by nj.
The above algorithm can be described using the following simple example, as shown in
It can be seen from the above, the SGNN provides efficient hierarchical indexing and classification. The data items are at the bottom of the hierarchy. All the intermediate nodes are categories or classes. The classification procedure is top down. In the above example, if a data item 1.8 is to be classified, the algorithm calculates the distances from it to the neurons level by level. In the first two level (1.5, 2.5 and 3.5), the closest neuron (winner) to 1.8 is 1.5. The calculation and comparison will continue for the sub-tree rooted by 1.5. So, the distances from 1.8 to 1, 1.5 and 2 are calculated and the winner to is found to be 2. This means 1.8 belongs to the category represented by 2.
For the SGNN 6 the training examples are extracted from web pages comprising text. The SGNN 6 adopts a SGNN learning procedure. Each keyword is assigned its own internal identification (id) value, which can be the location or index in a vector. The feature extractor 12, described below, assign weights to keywords or phrases to reflect their importance to the corresponding web pages/documents. Each final training example can be represented by
Ic;Ik,Wk{,Ik,Wk}
where Ic is the web page id, Ik is keyword/phrase id, Wk is the weight of the keyword, and the brackets simply represent “zero or more”. A training example therefore consists of a web page id followed by one or more keyword-weight pairs.
During the query answering mode of the system, the keyword-weight pairs generated by the NLQ analyser 4 have a similar form. The only difference is there is no Ic.
Each neuron or node in the SGNN 6 has the following form:
Ic,Wc{,Ic,Wc};Ik,W{,Ik,Wk};Pc;Ps;[D]
where Wc is the weight for web page Ic, and Pc and Ps are pointers to the first child neuron and the next sibling neuron of the current neuron. “[D] ” stands for zero or one data item, which is an index or data item to be returned by a search of the index network. A neuron or node consists of at least one web page-weight pair followed by at least one keyword-weight pair followed by pointers to its child and sibling neurons. If the neuron is a leaf node in the SGNN, then a data item can be attached to it's end. This data item can be an index or URL to a web page, document, or advertisement depending upon the type of search engine required. The pointers are used to form a tree of the neurons that form the SGNN index network. The weight of a web page reflects the possibility that the web page is one of the answers to the current user query. If a neuron is the final winner of a search of the index network, the Ic's in the neuron are sorted according to their weights and passed to the user in that order as the answer to the query. To simplify the representation, only one Ic and Wc pair may be used.
During the training phase, the inner product of the weight vectors of the training example and a particular neuron is calculated as a similarity measure or distances, as follows:
D=ΣW
ke
·W
kn
During the query answering mode the same inner product is calculated for the query vector and the weight vectors of the neurons to provide a similarity measure for the vectors. The weights of both vectors are normalised to a range from 0 to 1. Therefore the greater the inner product, the more similar the neuron is to the query. These values of the inner products are also used to order the results according to their similarity to the query, such that the output of the SGNN 6 is an ordered list of results:
where Weight1>Weight2> . . . >Weightn and are derived directly from the inner products for each neuron above a predetermined threshold level.
The feature extractor 12 is used both during generation or training of the SGNN and during the query or search process.
During the training phase, the extractor 12 extracts key terms, i.e. words and phrases (referred to as features) from each document/web page (hereinafter referred to as document) and assigns a weight to each of these features. The weight is dependent on how well that feature describes the document and brings out what is unique about the document. Thus, the input to the feature extractor is the set of documents, where each document is represented as a sequence of characters, with delimiters that break this sequence into words. The output during the training phase is a set of <feature, weight> pairs for each document, and this is used by the SGNN to generate the training examples.
During the search phase, the feature extractor 12 is used in much the same way to extract a topic name for each group of documents, where the grouping is generated by the clusterer 8, as described below. The input is a sequence of characters for each group of documents, and the output for each group is the unique feature with the highest weight for that group.
The methodology for extracting features is similar to that described in J. D. Cohen, “Highlights: Language and Domain Independent Automatic Indexing Terms for Abstracting”, Journal of the American Society for Information Science, 46(3): 162-174, 1995, for generating highlights or abstracts for documents that are retrieved from a search engine.
In the extractor 12, first the distribution of “ngrams” (a sequence of characters of length n) in the document space is determined. For example, the sequence of characters “to build” will give rise to the following 5 grams “to bu”, “o bui”, “buil” and “build”. The distribution of the ngrams over the document space is computed by counting the occurrence of ngrams in the documents. Next, each ngram is assigned a score per document that indicates how novel or unique it is for that document. Words and phrases in each document are then scored on the basis of the scores of the ngrams that make up a particular word or phrase. Thus, features that are most descriptive for a document have a higher weight.
Instead of using the feature extractor to only choose the features for documents, as described in J. D. Cohen, “Highlights: Language and Domain Independent Automatic Indexing Terms for Abstracting”, Journal of the American Society for Information Science, 46(3): 162-174, 1995, the search system uses both the features and their scores during the training phase to generate the index network 6. Hence the index network 6 is based both on the correct features for each document and a weight for each feature that reflects the feature's importance in describing the document. This reduces both the noise that is input to the SGNN 6 as well as the computation cost of generating the index network.
The clusterer 8 is used during the search phase to group together related documents and reduce the information overload when a large number of results are returned by the search engine.
The clusterer 8 uses a method for grouping together related documents that follows a standard model for document clustering described in G. Salton (ed.), “The SMART Retrieval System—Experiments in Automatic Document Processing”, Prentice-Hall, New Jersey, 1971. This model has three main features. First, each document is represented by a vector of word frequencies, where commonly occurring words have been excluded using a stoplist or heuristic feature selection techniques. Second, a distance measure is defined as a function of these vectors, so that the similarity or distance between any pair of documents in the vector space can be quantified. Finally, the clusterer 8 uses this distance measure to group related documents into clusters.
The clustering algorithm used by the clusterer 8 is a standard single pass algorithm that partitions the document set into disjoint sets or partitions as described E. Rasmussen, “Clustering Algorithms”, Information Retrieval (W. B. Frakes and R. Baeza-Yates ed.), Prentice-Hall, New Jersey, 1992. This algorithm proceeds as follows: the first document D1 is used to initialise the first cluster C1. For each of the remaining documents, Dk is assigned to the nearest cluster Cj or a new cluster if none is sufficiently close. In order to compare documents to clusters, each cluster is represented by its centroid where the centroid represents the average of the word frequency vectors of all the documents in the cluster. A new cluster is started when none of the existing clusters are sufficiently close, where the sufficiency is determined by a specified distance threshold T.
Traditional clustering algorithms described in the E. Rasmussen article mentioned above, typically determine this threshold T a priori for a particular domain or determine the required number of clusters based on the application and then choose the threshold T that generates the required number of clusters. However, this imposes artificial constraints on the search for structure in the document set, and the clusterer 8 departs from standard techniques in that the threshold T is automatically selected to reflect the natural structure in the document set.
The automatic determination of the threshold T is done as follows. First, many different partitions of the document set are generated by varying the threshold T. Large threshold values result in a small number of general clusters while small threshold values produce a large number of more specific clusters. Next, each partition is assigned a value that indicates the quality of the partition. This value takes into account cohesion, i.e. the closeness of the documents within the same cluster as well as the isolation of different clusters. This value is the sum of the inter-cluster distances and the intra-cluster distances. The inter-cluster distance is the distance of each document from its cluster centroid and the intra-cluster distance is the distance of each cluster centroid from the global centroid (the average of the word frequency vectors of all the documents in the document set). When there is one document per cluster or when all documents are grouped into a cluster, this value takes on the maximum value, which is the sum of the distances of the documents from the global centroid. The best partition is when this value is minimised and a few compact clusters are obtained. By this process, the distance threshold T that generates clusters that reflect the natural structure of the document set is determined. Once the clusters are generated, the feature extractor 12 is used to choose a unique topic name based on the documents that make up the clusters.
Many modifications will be apparent to those skilled in the art without departing from the scope of the present invention as herein described with reference to the accompanying drawings.
| Number | Date | Country | Kind |
|---|---|---|---|
| PQ 1381/99 | Jul 1999 | AU | national |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10030331 | May 2002 | US |
| Child | 11938758 | US |
