Patent Application
20050256865
The described technology relates generally to searching of databases and particularly to searching of web databases.
The World Wide Web (“web”) provides a vast amount of information that is accessible via web pages. Web pages can contain either static content or dynamic content. Static content refers generally to information that may stay the same across many accesses of a web page. Dynamic content refers generally to information that is stored in a web database and is added to a web page in response to a search request. Dynamic content represents what has been referred to as the deep web or hidden web.
Many search engine services allow users to search for static content of the web. After a user submits a search request or query that includes search terms, the search engine service identifies web pages that may be related to those search terms. These web pages are the search result. To quickly identify related web pages, the search engine services may maintain a mapping of keywords to web pages. This mapping may be generated by “crawling” the web to identify the keywords of each web page. To crawl the web, a search engine service may use a list of root web pages to identify all web pages that are accessible through those root web pages. The keywords of any particular web page can be identified using various well-known information retrieval techniques, such as identifying the words of a headline, the words supplied in the metadata of the web page, the words that are highlighted, and so on.
These search engine services, however, do not in general provide for searching of dynamic content, which is also considered noncrawlable content. One problem with searching of dynamic content is that the content of the web databases cannot be effectively retrieved and indexed for several reasons. One reason is that the content of multiple web databases may be too large to retrieve and index. Another reason is that the schema of web databases are hidden behind the search interface, that is, only the attributes of the search web page (and result web page) are exposed to a user. Another problem with searching of dynamic content is that the generated index would need to support both unstructured and structured queries. An unstructured query is a list of search terms that are generally used when searching for documents. For example, an unstructured query may be “Harry Potter Rowling.” A structured query is a list of attributes and attribute values that are generally used when searching a database. For example, a structured query may be “title=Harry Potter and author=Rowling.”
Considerable research has been conducted into developing a “metasearcher” that provides searching across multiple web databases. When the metasearcher receives a query, it selects the web databases that most likely contain relevant content, referred to as “source selection.” The metasearcher then translates the query into a suitable format for each of the identified web databases, referred to as “query translation.” For example, the metasearcher would need to understand how to map the attributes of the metasearcher's queries into site attributes of each selected web database. For example, the metasearcher may use an attribute named “format” to refer to the medium (e.g., paperback or hardback) of a book, whereas a web database may use an attribute named “type” to refer to the same data. Query translation needs to map the format attribute of the metasearcher to the type attribute of the web database. The metasearcher sends the translated queries to the selected web databases, referred to as “dispatching.” When the metasearcher receives the results of the searches, it integrates them into an overall result, referred to as “result integration.”
It would be desirable to have a technique for efficiently generating indexes for web databases that would allow for effective searching using both unstructured and structured queries.
A method and system for indexing and searching web databases within a domain is provided. In one embodiment, a search system generates an index for web databases by generatively sampling the databases and uses that index to identify and formulate queries for searching the databases. The generated index is referred to as a domain-attribute index and contains a domain-level index and, for each database, a site-level index. A site-level index for a database maps site attributes to distinct attribute values within the database. The domain-level index for a domain maps attribute values to database and site attribute pairs that contain those attribute values. To generate a site-level index for a database within a certain domain, the search system starts out with an initial set of the sample data for that domain. The search system selects a site attribute of the database that corresponds to a domain attribute using the domain-to-site mapping. The search system selects an attribute value from the sample data for that domain attribute. The search system then generates and submits a sampling query to the database to search for entries that match the selected attribute value in the selected site attribute. Upon receiving the entries as the query result, the search system generates the site-level index for that database, which maps site attributes to distinct attribute values contained within those site attributes. The search system may generate the domain-level index from the various site-level indexes.
In one embodiment, the search system allows users to specify unstructured queries or structured queries when searching multiple databases. The search system maps an unstructured query to a series of structured queries by first identifying whether the terms of the unstructured query represent an attribute or an attribute value based on the domain-level index. After identifying the set of attribute terms and attribute value terms, the search system generates a query for each combination of attribute term and attribute value term. The search system can use the attribute and attribute value pairs of a structured query directly without having to perform a mapping that is similar to that used for unstructured queries. The search system uses the domain-level index to identify to which databases the queries should be submitted.
A method and system for indexing and searching web databases is provided. In one embodiment, a search system generates an index for web databases by generatively sampling the databases and uses that index to identify and formulate queries for searching the databases. The generated index is referred to as a domain-attribute index and contains a domain-level index and, for each database, a site-level index. A domain is a collection of databases directed to a particular topic (e.g., books or cars). Each domain has a domain schema (also referred to as a “global schema”) that defines a set of domain attributes that are generally found in the schemas of the databases of the domain. For example, when the domain relates to books, the domain schema may include an author attribute and a title attribute since most book databases would have those attributes. A site-level index for a database maps site attributes to distinct attribute values within the database. For example, if the database contains an entry with an author attribute that has the attribute value of Tolkien, then the site-level index maps the author attribute to Tolkien. The search system generates the site-level indexes by sampling the databases as described below. The domain-level index maps attribute values to database and site attribute pairs that contain those attribute values. For example, a domain-level index would include an entry for the attribute value of Tolkien that is mapped to a database that contains an entry for a Tolkien book and to the author attribute of that database. The search system may generate the domain-level index from the site-level indexes. The domain-level index is a merged and inverted form of the site-level indexes. To process a query, the search system uses the domain-attribute index to select which databases should be searched and uses a domain-to-site attribute mapping to translate the query into a query that is appropriate for each selected database. The search system then submits the translated queries to the selected databases. The search system then integrates the results of the queries using the domain-to-site attribute mapping to map the results from site attributes to the domain attributes.
To generate a site-level index for a database within a certain domain, the search system starts out with an initial set of the sample data for that domain. The sample data provides sample entries for databases within the domain. The entries provide attribute values for the domain attributes of the domain schema. For example, sample data within a domain of books may contain one entry with the title attribute set to “Harry Potter and the Order of the Phoenix” and with the author attribute set to Rowling and another entry with the title attribute set to “The Fellowship of the Ring” and with the author attribute set to Tolkien. To sample a database, the search system selects a site attribute of the database that corresponds to a domain attribute using the domain-to-site mapping. The search system selects an attribute value from the sample data for that domain attribute. The search system then generates and submits a sampling query to the database to search for entries that match the selected attribute value in the selected site attribute. For example, if the selected site attribute is author and the selected attribute value is Tolkien, then the query will search for entries for books written by Tolkien. Upon receiving the entries as the query result, the search system generates the site-level index for that database, which maps site attributes to distinct attribute values contained within those site attributes. For example, the result may include an entry for Tolkien's book “Silmarillion” published by Del Rey, an entry for Tolkien's book “Roverandom” published by Houghton Mifflin, and an entry for Tolkien's book “The Hobbit” published by Houghton Mifflin. If so, then the site-level index will map the title site attribute to the words Silmarillion, Roverandom, and Hobbit and the publisher site attribute to the words of Del, Rey, Houghton, and Mifflin. The search system may also track the number of times that each attribute value occurs within the sample results. For example, an occurrence count of two would be associated with the attribute value of Houghton because it occurs in two entries of this sample result. To increase the coverage of a database, that is, the number of entries that are represented in the site-level index, the search system uses the data of the sample results to formulate new sampling queries for that database. For example, the search system may formulate a sampling query with the site publisher attribute set to Houghton, which would retrieve all entries for books published by Houghton Mifflin. The search system would then update the site-level index so that it “covered” all Houghton Mifflin books of the database thus increasing the coverage of the database. The search system may use that sampling result to generate further sampling queries. The search system, thus, generatively expands the coverage of a database based on previous sampling results of the sampling searches.
In one embodiment, the search system generates the domain-level index from the various site-level indexes. The search system adds an entry to the domain-level index for each distinct attribute value among the site-level indexes. For example, if five databases have entries for books by Tolkien then the domain-level index would have an entry for Tolkien with at least five sub-entries that each identify one of the five databases and the author site attribute of that database. When one of the databases contains an entry for the book “J. R. R. Tolkien: Architect of Middle Earth: A Biography,” the entry of the domain-level index for Tolkien may also contain a sub-entry for that database that identifies the title site attribute. The domain-level index may also contain frequency information in each sub-entry that indicates frequency of that attribute value within each database. The search system may use the frequency information when selecting databases for submitting queries. In one embodiment, the search system may divide or segregate the domain-level index based on domain attributes. For example, the domain-level index may have a sub-index for the title domain attribute and another sub-index for the author domain attribute. Each of the sub-indexes would include an entry for each distinct attribute value for that attribute within the databases. The use of sub-indexes facilitates translating structured queries by allowing the search system to first select the appropriate sub-index for the attribute of the structured query and then the entry that contains the attribute value (which contains sub-entries that all correspond to the attribute), rather than first selecting the attribute value and then having to search for sub-entries that correspond to the attribute.
In one embodiment, the search system allows users to specify unstructured queries or structured queries when searching multiple databases. The search system maps an unstructured query to a series of structured queries by first identifying whether the terms of the unstructured query represent an attribute or an attribute value based on the domain-level index. For example, an unstructured query may be “title author Tolkien” when the user wants to search for books written about the author Tolkien. The search system identifies that the term title and the term author are both domain attributes because they are names of attributes within the domain schema. The search system may also identify that the terms title, author, and Tolkien are attribute values because they are attribute values within a database. The terms title and author may be attribute values because one of the databases has a entry for the book “1997 Harvard Business School Core Collection: An Author, Title, and Subject Guide.” After identifying the set of attribute terms and attribute value terms, the search system generates a query for each combination of attribute term and attribute value term. For example, the search system would generate queries for the title attribute with attribute values of title, author, and Tolkien and would generate queries for the author attribute with attribute values of title, author, and Tolkien. The search system can use the attribute and attribute value pairs of a structured query directly without having to perform a mapping that is similar to that used for unstructured queries. With either unstructured queries or structured queries, the search system would translate the domain attributes to the site attributes of the selected databases. The search system then submits these queries to the databases selected based on the domain-attribute index because, for example, they are known to have entries corresponding to the attribute and attribute value of a query. The search system may also identify the order in which the queries are to be submitted based on information of the domain-attribute index. For example, the search system may elect to order the databases based on whether the domain-attribute index indicates they are likely to return a lot of entries (e.g., the frequency number indicates that a large number of entries will be returned).
Because the sampling of a database may place a heavy load on the database, the search system may use various techniques to ensure greater coverage of the database with less sampling. The search system may use various sampling criteria that may include random sampling, attribute-adaptive sampling, and value-adaptive sampling. When using random sampling, the search system randomly selects a site attribute and randomly selects an attribute value for the selected attribute that has not yet been selected. When using attribute-adaptive sampling, the search system selects a site attribute that has the smallest number of distinct attribute values among all the attributes of the databases. The search system then randomly selects an unused attribute value for that site attribute. When using value-adaptive sampling, the search system selects the site attribute that has the smallest number of distinct attribute values (like attribute-adaptive sampling), but selects an attribute value for that attribute that has not yet been selected and that has the largest number of occurrences among all attribute values that have not yet been selected. An attribute with the smallest number of distinct values within the domain-attribute index may also have a small number of distinct values in the sampled database. That is, on average a value from this attribute may have a higher chance than values from other attributes to match to a large portion of the sampled database entries. Similarly, a value with more occurrences in the domain-attribute index may also appear more frequently in the sampled database. That is, sampling the database with a value with more occurrences may retrieve a larger portion of the database entries.
In one embodiment, the search system applies a sampling stop criterion to terminate the sampling of a sampled database. Although exhaustive sampling of the database may be possible, it places a heavy burden on the database, and the database's web site may lock out users who access the database too often. The search system may use an absolute sampling stop criterion that is based on the maximum number of queries or a maximum time. Alternatively, the search system may use a relative sampling stop criterion based on the marginal value of the last few sampling queries as indicated by the amount of new content that was added to the site-level index based on the query results. The search system may use a combination of absolute and relative sampling stop criteria. For example, the search system may stop the sampling after 1000 queries or when the marginal value is below a threshold, whichever occurs first.
The computing device on which the search system is implemented may include a central processing unit, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), and storage devices (e.g., disk drives). The memory and storage devices are computer-readable media that may contain instructions that implement the search system. In addition, the data structures and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communications link. Various communications links may be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection. One skilled in the art will appreciate that the search system can be used with databases other than web databases. For example, the databases may be databases of different organizations (e.g., companies or governmental entities) that can be searched via a metasearcher.
The search system may be implemented in various operating environments that include personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The search system may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. One skilled in the art will appreciate that the illustrated data structures are just sample types of data structures that can be used to organize the data.
One skilled in the art will appreciate that although specific embodiments of the search system have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention. Accordingly, the invention is not limited except by the appended claims.
