A portion of the disclosure of this patent document contains material, which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.
Embodiments of the invention disclosed herein relates generally to providing a framework for aggregating information of web pages from a web site. More specifically, embodiments of the present invention are directed towards systems, methods and computer program products for aggregating a plurality of features across a given website domain and providing an aggregate feature set for the domain.
The aggregation of information on the Internet is an increasingly important task. Given the size and breadth of the Internet, however, aggregation is an equally difficult task. Indeed, as the Internet expands exponentially, the problem of aggregating data efficiently and effectively becomes increasingly difficult.
Aggregation of data on the Internet has numerous advantages. For example, aggregation of data across a subset of a website helps describe those pages with little analyzable content (e.g., images, Flash, etc.). Aggregation also allows for predictions to be made on unseen pages on the basis of the analysis of seen pages, thus allowing services to retrieve the most relevant and newest content available. For example, a search engine employing aggregation may be able to retrieve unseen pages in response to a user query, which an aspect of search engines unable to be addressed with current indexing techniques. Finally, aggregation has the advantages of working with multiple units of data, as opposed to desegregation, which works at the level of an individual unit. By using aggregation one can make determinations and predictions about a given content item given the context of the content item.
Thus, there is a need in the art for systems, methods and computer program products that provide a framework for aggregating information of web pages from a web site. Embodiments of the presently described invention address the advantages previously described and apply a customizable framework for exploiting the features of webpages in the context of information retrieval on the Internet.
The present invention is directed towards systems, methods and computer program products for providing a framework for aggregating information of webpages from a website. The method of the present invention comprises selecting a website and extracting one or more features from web pages associated with the website. In one embodiment, the method may further comprise constructing a sitemap associated with a given website.
The method aggregates the extracted features for the web pages. In a first embodiment, the feature set may comprise one or more numerical features, such as the number of in-links or an adult score. In a second embodiment, the feature set may comprise one or more categorical features, such as a classification or keyword match. In a third embodiment, the feature set comprises one or more rule-based features, such as a URL matching algorithm, which may analyze substrings of URLs. The method may store the aggregated features, which may be on a per web site basis.
The system of the present invention comprises one or more client devices coupled to a network and a content server coupled to the network, the content server operative to transmit and received data from the client devices. In one embodiment, the content server is operative to construct a sitemap associated with a given website. In a second embodiment, the content server is operative to reduce the noise of the aggregated features.
The system may comprise a crawler operative to selecting a website and a feature store operative to store a feature set comprising one or more features. In a first embodiment, the feature set may comprise one or more numerical features, such as the number of in-links or adult score. In a second embodiment, the feature set may comprise one or more categorical features, such as a classification or keyword match. In a third embodiment, the feature set comprises one or more rule-based features, such as a URL matching algorithm, which may analyze substrings of the URL.
The system may also comprise an aggregation module operative to load the feature set, extract the plurality of features from a plurality of webpages associated with the website and aggregate the extracted features for the plurality of web pages, as well as a content store operative to store the aggregated features.
The invention is illustrated in the figures of the accompanying drawings which are meant to be exemplary and not limiting, in which like references are intended to refer to like or corresponding parts, and in which:
In the following description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.
Crawler 112 is operative to analyze one or more web pages located on a given content provider 106. In one embodiment, crawler 112 may partition a crawling process to analyze a set of given webpages associated with a given site. For example, a given website may be identified as “www.example.com”. The crawler may use a regular expression such as “*.example.com” to crawl all pages for the website and any alternative subdomains such as “www2.example.com”, “sports.example.com”, “shopping.example.com”, etc. Alternatively, crawler 112 may intelligently divide a given website into subject-specific sites. That is, “sports.example.com” may be classified as a first website and “shopping.example.com” may be classified as a second website, although both are located within the same “example.com” domain.
Crawler 112 is operative to store information regarding websites 118 within a content store 120 at the content provider 108. In one embodiment, information regarding websites 118 may comprise records in a relational database, or any alternative indexing database schema known in the art, e.g., an object-oriented database, a hybrid object-relational database, a flat file data store such as a CSV table, etc. Alternatively, or in conjunction with the foregoing, crawler 112 may further be operative to receive additional data regarding the websites and web pages including, but not limited, to information known in the art retrieved by search engine crawlers. Content store 120 may enable content provider 108 to update information related to individual webpages, entire websites or a combination of both, such as logical groupings of webpages within a single website.
Crawler 112 is further coupled to an aggregation module 114 at the content provider 108 and operative to generate aggregate features for a given website. Aggregation module 114 is couple to a feature store 116, operative to store one or more features utilized to aggregate information across a plurality of webpages. In one embodiment, aggregation module 114 may receive one or more web pages corresponding to a logical grouping of a website's pages. Aggregation module 114 may load the aggregated features from feature store 116, analyzes the plurality of webpages and generate a feature vector for the given plurality of pages. In one embodiment, a feature vector may comprise a vector operative to store a plurality of feature values in a structure such as an associative array. For example, a value stored within a feature vector may comprise a key/value pair such as “inlinks=10” indicating the number of inlinks for a given webpage is 10. The aggregation module 114 may store the given feature vector within content store 120, e.g., in association with the web pages. In one embodiment, content server 110 may further be operative to perform administrative tasks, such as cleaning up noise in the returned aggregate features or building a sitemap for a given website.
The method 200 may construct a site map for the selected website, step 204. In one embodiment, a site map is built dynamically on the basis of link analysis of a currently selected website. For example, an automated program may crawl a given web site to determine a structure for the web site. Alternatively, a site map may be retrieved from the selected website and the site map may conform to a predetermined site map standard.
The method 200 continues by loading a feature set for a given website, step 206. In one embodiment, a feature set may comprise a list of features against which to analyze a web page. For example, a feature set may include numerical features such as the number of in-links, an adult score or any number of features described numerically. Alternatively, or in conjunction with the foregoing, a feature set may comprise a list of categorical features, such as a page classification, keywords, etc. Still further, a feature set may comprise a list of rules applicable to a given webpage. For example, a rule may comprise analyzing a URL to determine if various substrings within the URL match a given rule. Types of features are described more fully with respect to
After loading a feature set, the method 200 may extract and aggregate page-level features, step 208. In one embodiment, the method 200 aggregates one or more features across one or more web pages associated with a given website. In accordance with this embodiment, a feature vector may be returned that describes the selected website. Extracting and aggregating page-level features are discussed in detail with respect to
After extracting and aggregating page-level features, the method 200 may reduce noise within the aggregated features, step 210. In one embodiment, a Bayesian network with the same structure as generated in step 204 may be used to remove noise from the aggregated features. In this embodiment, features are encoded as prior probabilities and conditional independence imposed by the hierarchal structure of the site map may be used to compute the posterior probabilities on the tree. Additionally, the method 200 may derive conditional probability semantics between parents and children within the site map according to alternative models, such as UNITS or Darwin taxonomy. UNITS is described in greater detail in commonly owned U.S. Pat. No. 7,051,023 entitled “SYSTEMS AND METHODS FOR GENERATING CONCEPT UNITS FROM SEARCH QUERIES,” and filed on Nov. 17, 2003 under attorney docket number 600189-384, the disclosure of which is hereby incorporated by reference in its entirety.
The method 200 may store the generated feature set, step 212. In one embodiment, the method 200 stores a generated feature set in a database along with information identifying the selected website, e.g., on a per website basis. In an alternative embodiment, the method 200 stores the generated feature set in an ancillary database and associates the feature set with the selected website via an indexing mechanism.
The method 3100 continues by loading a feature list, step 3104. In one embodiment, a feature list may comprise one or more numerical features describing a web page. For example, a feature list may comprise numerical features such as the number of page in-links, the adult score, etc. As illustrated, a numerical feature may be a simple count of a particular aspect of a web page, such as the number of in-links, or maybe a more sophisticated value determined by a more sophisticated algorithm (e.g., an adult score calculated by a ancillary classifier).
The method 3100 calculates a feature vector for the selected subpage, the feature vector comprising values retrieved from the feature list, step 3106. In one embodiment, a generated feature vector may comprise an N-dimensional vector for storing numerical data related to a given web page. In one embodiment, the fields of the generated feature vector may be populated on the basis of the features present within the feature list, which may be the feature list retrieved in step 3104. After calculating a feature vector, the method 300 may store the feature vector associated with a given subpage, step 3108. In one embodiment, a feature vector is stored in a database, or similar data store, along with information regarding the selected subpage, such as a database maintaining an index of one or more web pages. The method 3100 may determine if any received subpages are remaining, step 3110, repeating steps 3102, 3104, 3106, 3108 and 3110 for the remaining subpages.
The remaining steps (3112, 3114, 3116, 3118) are directed towards aggregating the collected features for a given website. In the illustrated embodiment, subpages analyzed in steps 3102, 3104, 3106, 3108 and 3110 are aggregated to determine an aggregate feature set for a given website. Alternative embodiments exist, however, wherein a subset of a given website may be utilized to aggregate features, as described previously.
The method 3100 continues with the selection of a given feature from a list of available features, step 3112. In one embodiment, the selected features may correspond to the feature list retrieved in step 3104. In an alternative embodiment, a feature list comprising a subset of the loaded feature list may be utilized. The method 3100 iterates through the selected features (step 3118) and calculates the mean, standard deviation and the probability of the numerical features, steps 3114 and 3116, respectively.
Upon selecting a given subpage, the method 3200 inspects the subpage to identify categorical features, step 3206. In one embodiment, identifying a categorical feature comprises a single algorithm, but may also comprise a plurality of algorithms. For example, identifying categorical features may comprise using a classifier to assign a classification label to a given subpage. In a second example, identifying a categorical feature may comprise identifying a plurality of keywords for a given subpage by analyzing the text of a given subpage. After identifying a plurality of categorical features, the method 3200 ma update a categorical feature vector associated with the determined categorical features, step 3208. In one embodiment, updating a categorical feature vector comprises storing the vector in a relational database along with the associated subpage. The method performed in steps 3204, 3206 and 3208 may repeat for any remaining subpages, step 3210.
After collecting one or more categorical features, the method 3200 selects a given feature, step 3212, and calculates the probability of the select feature, step 3214. In one embodiment, the calculated probability corresponds to the probability of the occurrence of the selected feature against subpages analyzed by the method in steps 3204, 3206 and 3208.
For a given URL, one or more applicable rules may be determined, step 3306. In one embodiment, a rule associated with a URL may comprise information describing the URL. For example, a rule may indicate that a first substring within the URL is a constant across a subsite or that a second substring within the URL is case-insensitive. For example, a subsite containing the URLs “http://www.example.com/sports/baseball/stats?playerid=143&sid=32fsgirg240” and “http://www.example.com/sports/baseball/team?teamid=13&sid=32fsgirg240” may have a first rule indicating the substring “&sid=32fsgirg240” remains constant for that subsite and is associated with a session ID. Alternatively, a subsite containing the URLs “http://www.example.com/sports/baseball/search?q=david+ortiz” and “http://www.example.com/sports/baseball/search?q=Boston+red+sox” may generate a second rule indicating that characters after the string “q=” are case-insensitive and correspond to a query.
After determining one or more rules, the method 3300 updates a support vector associated with the determined rules, step 3308. In one embodiment, updating a support vector may comprise incrementing the number of times a given rule matches a selected URL, thus indicating “support” for a rule as being proportional to the number of URLs matching the rule.
In software implementations, computer software (e.g., programs or other instructions) and/or data is stored on a machine readable medium as part of a computer program product, and is loaded into a computer system or other device or machine via a removable storage drive, hard drive, or communications interface. Computer programs (also called computer control logic or computer readable program code) are stored in a main and/or secondary memory, and executed by one or more processors (controllers, or the like) to cause the one or more processors to perform the functions of the invention as described herein. In this document, the terms “machine readable medium,” “computer program medium” and “computer usable medium” are used to generally refer to media such as a random access memory (RAM); a read only memory (ROM); a removable storage unit (e.g., a magnetic or optical disc, flash memory device, or the like); a hard disk; electronic, electromagnetic, optical, acoustical, or other form of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.); or the like.
Notably, the figures and examples above are not meant to limit the scope of the present invention to a single embodiment, as other embodiments are possible by way of interchange of some or all of the described or illustrated elements. Moreover, where certain elements of the present invention can be partially or fully implemented using known components, only those portions of such known components that are necessary for an understanding of the present invention are described, and detailed descriptions of other portions of such known components are omitted so as not to obscure the invention. In the present specification, an embodiment showing a singular component should not necessarily be limited to other embodiments including a plurality of the same component, and vice-versa, unless explicitly stated otherwise herein. Moreover, applicants do not intend for any term in the specification or claims to be ascribed an uncommon or special meaning unless explicitly set forth as such. Further, the present invention encompasses present and future known equivalents to the known components referred to herein by way of illustration.
The foregoing description of the specific embodiments so fully reveals the general nature of the invention that others can, by applying knowledge within the skill of the relevant art(s) (including the contents of the documents cited and incorporated by reference herein), readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present invention. Such adaptations and modifications are therefore intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled artisan in light of the teachings and guidance presented herein, in combination with the knowledge of one skilled in the relevant art(s).
While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example, and not limitation. It would be apparent to one skilled in the relevant art(s) that various changes in form and detail could be made therein without departing from the spirit and scope of the invention. Thus, the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.