Patent Application
20070078850
Not applicable.
Not applicable.
Many types of commercial goods are now available via the World Wide Web. Some conventional web sites allow a user to browse products from a single company or distributor. Other conventional sites can allow a browser to view products from one or a few predetermined sites or commercial locations.
What is needed is a system and method for allowing a user to view sale and product information from a variety of product web sites in a single location. The system and method should allow a user to view offers for sale of any type of desired product. Additionally, the system and method should provide a user with detailed information about available products in response to a product request.
In an embodiment, the invention provides a system and method for extracting detailed product information for products that are available from an internet website and delivering the product information in response to a product request. The product information provided to the users can be based on information provided by a retailer, or the information can be obtained by searching web sites and extracting the product information. Products matching a query can then be provided in a gallery view to allow for easy comparison by a user.
I. Overview
In an embodiment, the invention includes a system and method for providing detailed commercial offer information to a user in response to a request for a product, service, or other type of commercial offer. For example, when a product request is received from a user, the user is provided with detailed information about product availability from a variety of sellers. The detailed information can include information from retailers who have agreed to provide product information. The detailed information can also include information obtained by crawling publicly available web sites and extracting product information from the crawled web sites. The detailed information can include the name of the product, a picture of the product, the price of the product, a description of the product, and/or other information specifying a product for sale.
II. Identifying Commercial Offer Pages
In an embodiment of the invention, the method begins by identifying potential pages that contain a commercial offer. For convenience, the method will be described with reference to “product pages”, or pages where the commercial offer is an offer for sale of a product. However, the description that follows applies generally to any type of goods or services that can be offered by a merchant or other commercial entity.
As a preliminary step, a web crawler can be used to pre-search publicly available web documents. During a pre-search, a group of searchable documents is crawled and searched to catalog the type and content of each document. A pre-search can occur at any convenient interval, such as once a day or once a week. The group of searchable documents can represent any convenient grouping. In an embodiment, documents from web locations in a specific country can be pre-searched. In another embodiment, documents from a known commercial site can be pre-searched to obtain information about available products listed on the site. In still another embodiment, all searchable documents available via the Internet can be pre-searched to identify and classify product pages. In such an embodiment, the pre-search for product information can take place as part of a pre-search for a conventional search engine.
For each document in the group of searchable documents, the document can be classified as a product or non-product page. A product page is a document containing information about one or more products. Product pages can include documents describing a product for sale, documents containing a special offer for a particular product, documents describing accessories for a product, and other types of documents describing information related to a product.
Product pages can be identified by any convenient method. In an embodiment, a document can be classified by searching the document for product characteristics, such as a price for a product, a product description, or an image of a product. Alternatively, a product page can be identified based on the presence of a link that indicates an item is for sale, such as a link labeled “buy now” or “add to shopping cart.”
In an embodiment, product pages can be identified and/or classified by first breaking down a large number of available documents into smaller groups or “chunks”. The smaller groups of documents can each contain one or more documents. The documents in a small document group can be a related group of documents, such as a documents that are organized under a common parent document on a web site, such as documents organized under “microsoft.com.” In another embodiment, one or more web sites may have a similar format or structure that can be specifically targeted for product page identification and extraction. For example, “amazon.com” is a parent site for a number of web pages having a similar format that also contain product listing. A web site (or sites) having a format or structure that can be targeted for product identification and extraction can be referred to as a “head site.”
III. Extracting Commercial Offer Records
After breaking down the available documents into chunks, the documents in each chunk are analyzed to identify product pages. In an embodiment, the analysis begins with the first document in the document group. For a group of documents that are related to one another, the first document can be the parent document or some other document logically related to the remaining documents in the grouping. HTML and meta information is then extracted from the document. The HTML and meta information can then be analyzed to classify the document, for example, as a product or non-product page. In an embodiment, the HTML and meta-information data is analyzed to identify any indications of a price, such as a price identifier or a phrase/snippet of words indicating a price or product for sale. The price identifier or pricing phrase can be in the text of the document or in a hyperlink in the document to a separate document or web location. In another embodiment, the document can be classified as a product or non-product document based on the presence of words, phrases, or other document features that are commonly found on product pages. In such an embodiment, a search engine can be trained to identify product pages. A test group of documents can be reviewed by humans to develop a training set of documents. The parameters of a search engine can then be tuned based on the product versus non-product judgments from the training documents. In still another embodiment, the parameters of the search engine can be tuned to separately classify a subset of product documents, such as product documents containing special offers or product documents describing accessories for a product.
If a document is classified as a product page, product information elements corresponding to one or more products available on the product page is extracted. The extracted information for a product can include the product name, model, manufacturer, price, any special offers, ratings and/or reviews of the product, or an image of the product. Extracted product information that is related to a single product can be referred to as a product record.
Preferably, product information elements are extracted automatically by an entity extractor. Some information elements can be extracted by identifying common keywords associated with a certain category, such as known brand names. Other information elements can be identified for extraction by training the entity extractor. First, a known set of training documents are reviewed by humans to identify various types of product data. The training documents are then used to optimize parameters in the entity extractor so that various information elements (brand, price, image, rating, etc.) are extracted correctly.
In a preferred embodiment, multiple sets of parameters for an entity extractor are available to allow for different extractor optimizations. In such an embodiment, one or more parameter sets can be developed that are targeted for use on a group of documents organized under a specific parent document, such as the head site for an individual retailer that has a large and/or desirable collection of products offered on the web site. The targeted parameter sets can be optimized based on the particular format used by the individual retailer. Using the targeted parameter sets allows for improved extraction from commercial sites that are known to have large and/or desirable product collections. In an embodiment, the parameter set used by the entity extractor is selected each time a new chunk of documents is analyzed. If parent document corresponding to a particular parameter set is contained in the chunk, product information for all product pages in the chunk can be extracted using the targeted parameter set. Otherwise, a default parameter set can be used. In another embodiment, the documents within a chunk may not all share the same parent document. In such an embodiment, a new extractor parameter set can be selected as needed based on the correspondence, if any, of each document in the chunk with a targeted parameter set. The extraction parameter set to use for a particular document can be selected by analyzing one or more characteristics of the document (or parent document), such as searching the document for a keyword or by analyzing the URL (universal resource locator) for the document.
The above procedures can be repeated to produce a product record for each product contained on an identified product page. The resulting product records can then be converted into any convenient data format, such as XML. This allows the product records to be used by a search engine that is targeted to providing commercially available products. After converting the product records into XML format, the product records can be stored in a database. Alternatively, the data contained in the product records can be incorporated or overlaid as meta-data into an existing web document index to allow for searching of the product records.
In an embodiment, commercial data extracted from a document can be used to form product records having one or more of the following categories: 1) The name of the commercial offer; 2) A description of the product or service that comprises the commercial offer; 3) The merchant offering the product or service; 4) At least one price for the product or service; 5) One or more special pricing offers currently available for the product or service; 6) A URL for an image related to the commercial offer; 7) A classification or categorization of the product or service based on the offering Merchant's taxonomy scheme (for example, an ornamental lamp could be classified by a merchant as being in the category/subcategory “Home furnishings/Home decor”); 8) The manufacturer of a product (publisher if the product is a book); 9) The model number or universal product code of the product; 10) The type of document where the commercial offer was found, such as an offer listing document, an offer details document, or a document containing mixed types of information; and 11) Locale (geographical) information regarding the document containing the commercial offer.
After extracting product records from a document, the product records can be converted into a format that can be easily searched using an available search engine. This allows a commercial offer to be “ranked” in response to a commercial offer query in a manner similar to how a web document is ranked by a search engine in response to a search query. In an embodiment, metadata from the product records can be overlaid on to an existing web document index to allow for commercial searching. In such an embodiment, the metadata could represent keywords, the web document index could be an inverted index for searching, and the product records for a single document could represent the “document” associated with the metadata keyword. In another embodiment, the product records can be converted into an HTML format to allow searching by a conventional web search engine. In such an embodiment, converting the product records can include using the data in the product records to populate corresponding fields in an HTML format document. For example, the name of the product, service or other commercial offering can be used to populate the title field of an HTML document. A description for the commercial offering can be used as the body text of the HTML document. The conversion can also allow population of other fields not directly related to a product record. For example, a product record quality can be determined for a commercial offering, possibly based on the number or type of product records available after extraction. This product record quality can be used to populate a page quality field in the HTML document.
In an embodiment, after converting the product records for a product into an HTML document, the document can be pre-searched to form a convenient data structure for searching, such as an inverted index of keywords. Preferably, the index or other search data structure can be adapted for commercial offer search, such as by including known merchants and products as searchable words or phrases.
By converting the product records and information generated from the product records into a searchable format, such as an HTML format, the ranking algorithm of a search engine can be used to rank the available commercial offers corresponding to commercial offer query. The rankings can be used, for example, to determine the order of display for commercial offers corresponding to a product query and/or whether a commercial offer should be displayed at all. The commercial offer rankings can also be further improved by modifying how the search engine is used. For example.
In addition to extracting product records, the pre-search can also be used to construct an inverted index of words and/or word phrases. The inverted index can be used to correlate product records with words or phrases found in the product records. This allows product records related to a search term to be quickly retrieved in response to a user product search request. Alternatively, other data structures can also be constructed to assist in organizing the product data for improving response time to user requests.
In an embodiment, the product records found during a pre-search can be further processed and classified prior to being stored in a database. In such an embodiment, the product description and other information elements in the product record are categorized in a detailed way to allow for comparisons between products. For example, based on keywords or other information extracted by the entity extractor, the product can be classified in a product category, such electronics, automotive, etc. Depending on the extracted information, the product may also be able to be placed in a narrower subcategory, such as a DVD player or a multi-disc DVD player. The additional processing can also be used to create a uniform format for information elements extracted by the entity extractor. For example, the extracted information elements can be analyzed and used to fill in a template of available features for an item. This allows comparison of available features for two or more items of a similar type.
In an embodiment where product information is categorized, the categorized information can be searched using a structured query request. In a structured query request, the product information can be searched using a query that asks for one or more keywords in a specific category. For example, structured queries can be submitted to request information about automobiles of a particular brand or DVD changers that can store more than a specified number of discs. In an embodiment, a user can submit a structured query by specifying both a query category and a keyword associated with the query category within the query. In another embodiment, a user interface can be provided to facilitate submission of a structured query. For example, a drop-down menu can be provided containing a list of potential query categories. A user can then select a query category from the list and specify a keyword to be found in the selected category. In still another embodiment, similar products (or commercial offers) could be clustered and annotated with hash values. In such an embodiment, the a structured query request could be used to identify similar items based on distances between hash calculations stored per record for the items.
In still another embodiment, the product records extracted from the documents found by crawling web sites can be combined with other product records provided by an information stream received from a seller or retailer. In such an embodiment, one or more sellers can provide an information stream containing information elements about products available for sale. These provided information elements can be converted into product records and aggregated with the other product records.
IV. Display of Results
After analyzing the results of the pre-search, the resulting product records can be used to form responses to user product requests. In an embodiment, a user can submit a product request as a keyword search request to the commercial product search engine. For example, a user could submit a search request for a particular brand of electric guitar by using “<brand>electric guitar” as keywords. The product search engine would then return offers to sell products matching the search.
In another embodiment, rather than simply providing a listing of web sites, the product search engine provides the user with a gallery that displays various information elements from the product records. For example, the initial gallery can include the price of each product, a product picture, and a link to the commercial web site offering the product. Other information elements can also be presented, such as a comparison of product features. The displayed results can also be refined by organizing the results based on various criteria, such as store name, product price, or whether the product is being offered by a confirmed merchant or a non-confirmed merchant.
V. General Operating Environment
The search engine 70 may include a web crawler 81 for traversing the web sites 30, 40, and 50 and an index 83 for indexing the traversed web sites. The search engine 70 may also include a keyword search component 85 for searching the index 83 for results in response to a search query from the user computer 10. In an embodiment, keyword search component 85 can include a structured query component for matching a product record with a search query based on both a query category and an associated keyword. A document separator 87 can be included to separate out desired HTML and meta information from documents found by the web crawler. The search engine 70 may also include a page classifier 88 for classifying pages as product or non-product pages. Additionally, search engine 70 can include an entity extractor 89 to extract information elements about a product from a product page, such as brand name, price, product reviews, and images of the product. The extracted information can be stored in a database or index structure (not shown), possibly after further processing. Alternatively, entity extractor 89 can include a display component for displaying information elements extracted from one or more product records in a gallery.
The invention is described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
With reference to
Computer 110 typically includes a variety of computer readable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation,
The computer 110 may also include other removable/nonremovable, volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
The computer 110 in the present invention will operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in
When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
Although many other internal components of the computer 110 are not shown, those of ordinary skill in the art will appreciate that such components and the interconnection are well known. Accordingly, additional details concerning the internal construction of the computer 110 need not be disclosed in connection with the present invention.
VI. Exemplary Embodiments
As documents containing product and other commercial offers are identified, index builder 630 parses the documents and extracts any commercial offer information. Preferably, the documents can be classified according to the type of information in the document. The information in the documents can also be converted into a searchable document format. Additionally, the documents can be partitioned and categorized. For example, the documents can be indexed using a keyword or other type of index. Content related to a single offer can also be stored in a single logical location to allow for easy retrieval of related product information. Any links to related pages can also be noted for a given commercial offer. After building the index, the information extracted and/or generated by index builder 630 can be stored in one or more index nodes 640.
The principles and modes of operation of this invention have been described above with reference to various exemplary and preferred embodiments. As understood by those of skill in the art, the overall invention, as defined by the claims, encompasses other preferred embodiments not specifically enumerated herein.
