Patent Application
20120047121
Web search engines operate by storing information about web pages in a search engine index and employing the indexed information to return search results to user search queries. The web page indexing process includes employing web crawlers to retrieve web pages and index information about the crawled web pages. Determining which web pages to crawl and how often to crawl the web pages is a complex problem that can have a significant impact on a search engine. If a search engine crawls web pages too frequently, the search engine places a significant load on the content servers serving the web pages in terms of bandwidth and CPU resources. As a result, webmasters may not welcome the search engine's crawlers, especially if competitor search engines are crawling the web sites more intelligently resulting in fewer crawls. Inefficient crawling also increases the operating costs of the search engine. On the other hand, if a search engine does not crawl web pages frequently enough, the search engine will be slow to capture changes to web pages. As a result, the search engine's search results will suffer in relevance and freshness. If the search engine also does not effectively discover new web pages, the relevance and freshness of the search engine's search results will be further degraded, which in turn will result in a poorer search experience for end users.
For each web page, the crawl frequency (determining when the content will be crawled or re-crawled) is often determined primarily by importance computed for the web page. As the crawl frequency of a web page is based on its importance, important web pages are scheduled to be crawled often (e.g., daily), while less important web pages are scheduled to be crawled less often (e.g., monthly). In some cases, additional information gathered by the crawler may be taken into account to learn if the content is changing often and employed to determine crawl frequency.
Managing a relevant search engine index requires not only crawling new relevant web pages to index new content, but also re-crawling existing web pages to take into account content changes, such as, for instance, in-page content changes, or link changes (the most drastic of these changes being new pages and old, empty or deleted pages). As such, the crawler is really performing two tasks at once: (1) detecting page changes—new/updated content and links, or not; and (2) gathering new/updated content/links.
Unfortunately, the current architecture of typical search engines provides a catch-22. To figure out if a web page has any changes which need to be captured, the search engine has to first crawl the web page to look for changes. This means that for pages that are very important, even if they don't change frequently, the search engine needs to constantly re-crawl them. Unfortunately, in that situation, the search engine has wasted the crawl bandwidth on both the search engine servers and the websites' servers.
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
Embodiments of the present invention relate to a client application installed on end user computers that generates metadata from the content of web pages visited by end users and provides the metadata to a search engine. When an end user accesses a web page, the web page is downloaded and displayed to the end user. The client application operates by accessing the content of the web page and generating a content signature of the web page from the web page content. The client application then provides the content signature of the web page to a search engine. The search engine may employ content signatures provided by client applications on end user computers in a number of manners. For instance, the search engine may employ content signatures to discover web pages being visiting by end users but not currently indexed by the search engine. The search engine may employ such information to determine to index the newly discovered web pages. Additionally, the search engine may analyze the content signatures to identify changes to the content of web pages. This information may help the search engine in calculating the optimum crawl frequency of web pages based on this analysis.
The present invention is described in detail below with reference to the attached drawing figures, wherein:
The subject matter of the present invention is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms “step” and/or “block” may be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.
As noted above, embodiments of the present invention provide a client application that is installed on end users' computers and operates to provide metadata of web pages visited by end users to a search engine. The metadata allows, among other things, the search engine to identify new web pages and changes to web page content.
When an end user visits a web page, the end user's computer downloads the web page and presents the web page to the user. The client application operates in the background by accessing the content of the web page to create a content signature of the web page. As used herein, the term “content signature” refers to a representation of a web page generated from the content of the page. For instance, the content signature may be a hash value generated by the client application applying a hash function to the web page content. The client application provides the generated content signature to a search engine.
In practice, the client application is preferably installed on a number of end user computers. Whenever the end users visit web pages, the client applications generate content signatures and provide the content signatures to a search engine along with the associated URLs. As such, the search engine may benefit by receiving information regarding web pages visited by a large number of end users and determine if the content has updated based on, for instance, the last received content signature.
The search engine may employ the content signatures for a variety of purposes. For instance, the search engine may employ the content signatures to discover web pages being visited by end users that are not currently indexed. The search engine may then determine to crawl and index the newly discovered web pages. Additionally, the search engine may analyze the content signatures to identify changes to the content of web pages and employ that information in determining the crawl frequency for different web pages. In some instances, the content signatures may simply allow the search engine to determine that the content of web pages has changed. In other instances, the content signatures may allow the search engine to determine the extent of change to the web page content and/or the location of the changed web page content. Such additional information may further facilitate providing intelligent crawling of web pages by the search engine.
In essence, the client application allows the search engine to offload the task of crawling web pages to discover content changes to users who are downloading the web pages anyways. As such, the webmasters don't see any additional load while the search engine may conserve crawling resources by crawling web pages more intelligently based on the content signature information provided by the client applications. Additionally, the resources of web servers are not consumed by inefficient crawling.
Accordingly, in one aspect, an embodiment of the present invention is directed to one or more computer storage media storing a client application comprising computer-useable instructions that, when used by an end user's computing device, cause the end user's computing device to perform a method. The method includes accessing content of a web page downloaded to the end user's computing device. The method also includes generating a content signature from the content of the web page. The method further includes communicating the content signature from the end user's computing device to a search engine server.
In another embodiment, an aspect of the invention is directed to one or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform a method. The method includes receiving, at a search engine server from an end user's computing device, a content signature of a web page, the content signature comprising a representation of the web page generated from the content of the web page by a client application on the end user's computing device. The method also includes storing the content signature of the web page.
A further embodiment of the invention is directed to one or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform a method. The method includes receiving, at a search engine server, content signatures of a plurality of web pages, the content signatures having been generated at a plurality of end user computing devices from content of web pages downloaded and displayed by the end user computing devices. The method also includes analyzing the content signatures to identify content changes on web pages. The method further includes controlling crawling of web pages based on analysis of the content signatures.
Having briefly described an overview of embodiments of the present invention, an exemplary operating environment in which embodiments of the present invention may be implemented is described below in order to provide a general context for various aspects of the present invention. Referring initially to
The invention may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules including routines, programs, objects, components, data structures, etc., refer to code that perform particular tasks or implement particular abstract data types. The invention may be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general-purpose computers, more specialty computing devices, etc. The invention may also be practiced in distributed computing environments where tasks are performed by remote-processing devices that are linked through a communications network.
With reference to
Computing device 100 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 100 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 100. Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.
Memory 112 includes computer-storage media in the form of volatile and/or nonvolatile memory. The memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical-disc drives, etc. Computing device 100 includes one or more processors that read data from various entities such as memory 112 or I/O components 120. Presentation component(s) 116 present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc.
I/O ports 118 allow computing device 100 to be logically coupled to other devices including I/O components 120, some of which may be built in. Illustrative components include a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc.
Referring now to
Among other components not shown, the system 200 includes a user device 202, a content server 204, and a search engine server 206. Each of the components shown in
The user device 202 may be any type of computing device owned and/or operated by an end user that can access network 208. For instance, the user device 202 may be a desktop computer, a laptop computer, a tablet computer, a mobile device, or any other device having network access. Generally, an end user may employ the user device 202 to, among other things, access electronic documents maintained by content servers, such as the content server 204. For instance, the end user may employ a web browser 210 on the user device 202 to access and view electronic documents from the content server 204. Although a web browser is shown in
The content server 204 may be any type of server device capable of hosting electronic documents and serving the electronic documents to computing devices such as the user device 202. For instance, the content server 204 may be a web server maintaining web pages, such as web page 214.
The search engine server 206 may be any type of server device capable of providing search services to user devices, such as the user device 202. Among other things, the search engine server 206 includes a crawler 216 that is employed to crawl web pages maintained by content servers, such as the web page 214 on content server 204, and index data from the web pages in a search engine index 220. When users submit search queries to the search engine 206, the search engine 206 searches the search engine index 220 to provide search results in response to the search queries.
In accordance with embodiments of the present invention, the user device 202 includes a client application 212 that operates to generate content signatures from the content of web pages accessed by the user device 202 and provide the content signatures to the search engine server 206. The client application 212 may be a component of the web browser 210, a browser extension (e.g., a search toolbar), or other application installed on the user device 202. In some embodiments, a search system provider who provides the search services via the search engine server 206 provides the client application to the end user for installation on the user device 202.
When a user employs the web browser 210 (or other component) to access a web page from a content server, such as web page 214 on content server 204, the web browser 210 downloads the web page and presents the content of the web page via a graphical user interface provided by the web browser 210. The client application 212 accesses content from the web page and generates a content signature from the content. As such, the content signature is a representation of the web page generated from the content of the web page. The content of the web page accessed by the client application and used by the client application 212 to generate the content signature may be any portion of the content that may be displayed to the end user via the web browser 210. For instance, the content of the web page may include text, images, sounds, videos, etc. In some embodiments, the content signature may be generated based on the text only. In other embodiments, other portions of the presentable content of the web page may be employed to generate the content signature. In further embodiments, multiple content signatures may be generated for a web page using different portions of the content (e.g., one content signature for text, another for images, etc.).
A variety of different algorithms may be employed to generate the content signature of a web page. In some embodiments, the content signature is a hash value that is generated by applying a hash function to the web page. Any of a variety of different hash functions may be employed by the client application 212 to generate content signatures within the scope of embodiments of the present invention.
After the client application 212 generates a content signature of a web page from the web page content, the client application 212 provides the content signature to the search engine server 206. In some embodiments, the client application 212 may provide the content signature for a web page accessed by the user device 210 upon downloading the web page and generating the content signature. In other embodiments, the client application 212 may batch content signatures generated from various web pages accessed by the user device 212 and periodically provide the batched content signatures to the search engine server 206.
The client application 212 may provide additional metadata when providing a content signature for a web page to the search engine server 206. In some embodiments, the client application 212 provides an indication of the URL of the web page and a timestamp indicating, for instance, a time that the web page was accessed and/or the content signature was created. Other metadata may also be provided to the search engine service 206, such as, for instance, a location of the end user.
Although only a single user device is shown in
Content signatures provided to the search engine server 206 by client applications on user devices, such as the client application 212 on user device 202, may be stored in web page metadata storage 218. Other metadata provided by the client application 212 may also be stored in the web page metadata storage 218.
The search engine server 206 may employ the content signatures provided by user devices and stored in the web page metadata storage 218 for a variety of purposes. For instance, in some embodiments, the search engine server 206 may analyze the content signatures to discover web page being visited by end users but not currently indexed in the search engine index 220. Based on the discovery of such web pages, the search engine 206 may determine to crawl and index the web pages.
In some embodiments of the present invention, the search engine server 206 may employ the content signatures to determine changes in web page content and control crawling of web pages for indexing purposes. Initially, the search engine server 206 may analyze the content signatures of web pages to identify changes in the content of web pages. In some instances, the algorithm employed by client applications on user devices may only support identifying whether the content of a web page content has changed. For instance, the search engine server 206 may access two or more content signatures for a given web page (e.g., corresponding with a particular URL) and compare the content signatures. If the content signatures are the same, the web page content has not changed. However, if the content signatures have changed, the web page content has changed. This information may be employed by the search engine server 206 to control crawling of web pages.
In one embodiment, each web page is classified into a category based on its change history evidenced by its content signatures and crawled based on its category. For instance, the crawler may include the following rules:
In some embodiments, the algorithm employed by client applications may generate content signatures that allow for a determination of the extent to which a web page has changed and/or the location of changes with the web page. For instance, a hash function may be employed in which the difference in hash values generated from the content of a web page is statistically related to the extent of change in the web page. For example, if 15 percent of the web page content has changed, probabilistically 15 percent of bits in the hash values will be different. If the content signatures provide such additional information, the search engine server 206 may employ the additional information to even more intelligently determine the crawl frequency of web pages. For instance, significant content changes may warrant crawling a web page while insignificant content changes may not.
Turning now to
A client application accesses content of the web page, as shown at block 304. As indicated previously, the client application may be a component of a web browser on the user device, a browser extension, or other application installed on the user device. After accessing the web page content, the client application generates a content signature from the web page content, as shown at block 306. In some embodiments, the content signature is a hash value generated by the client application by applying a hash function to the web page content.
As shown at block 308, the client application communicates the content signature to a search engine server. In some embodiments, the client application may communicate other metadata in addition to the content signature. For instance, in one embodiment, the client application provides an indication of the web page's URL with the content signature. As another example, the client application may provide a timestamp representing a time that the web page was downloaded or the content signature was created, for instance. Other types of metadata, such as the location of the end user, may also be provided to the search engine server in embodiments of the present invention.
Referring next to
The search engine server may store received content signatures, as shown at block 404. As such, the search engine server may collect content signatures for various web pages from a number of different end user computers. The search engine server may employ the content signatures for a number of purposes. As such, the search engine server analyzes the content signatures, as shown at block 406.
In one embodiment, the search engine server may examine the content signatures to identify web pages visited by end users but not currently indexed in the search engine index. As shown in the method 500 of
In another embodiment depicted by the method 600 shown in
As can be understood, embodiments of the present invention provide for generating content signatures on end user computers and providing the content signatures to a search engine server, which may analyze the content signatures to, for instance, discover new web pages and control the crawling frequency of web pages.
The present invention has been described in relation to particular embodiments, which are intended in all respects to be illustrative rather than restrictive. Alternative embodiments will become apparent to those of ordinary skill in the art to which the present invention pertains without departing from its scope.
From the foregoing, it will be seen that this invention is one well adapted to attain all the ends and objects set forth above, together with other advantages which are obvious and inherent to the system and method. It will be understood that certain features and subcombinations are of utility and may be employed without reference to other features and subcombinations. This is contemplated by and is within the scope of the claims.
