Patent Application
20080044016
The present invention may involve novel methods, apparatus, message formats, and/or data structures for determining whether or not documents are similar. The following description is presented to enable one skilled in the art to make and use the invention, and is provided in the context of particular applications and their requirements. Thus, the following description of embodiments consistent with the present invention provides illustration and description, but is not intended to be exhaustive or to limit the present invention to the precise form disclosed. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles set forth below may be applied to other embodiments and applications. For example, although a series of acts may be described with reference to a flow diagram, the order of acts may differ in other implementations when the performance of one act is not dependent on the completion of another act. Further, non-dependent acts may be performed in parallel. No element, act or instruction used in the description should be construed as critical or essential to the present invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used. In the following, “information” may refer to the actual information, or a pointer to, or a location of, such information. Thus, the present invention is not intended to be limited to the embodiments shown and the inventor regards her invention to include any patentable subject matter described.
In the following, environments in which the present invention may be employed are introduced in § 4.1. Then, exemplary embodiments consistent with the present invention are described in § 4.2. Finally, some conclusions about the present invention are set forth in § 4.3.
The following exemplary embodiments are presented to illustrate examples of utility of embodiments consistent with the present invention and to illustrate examples of contexts in which such embodiments may operate. However, the present invention can be used in other environments and its use is not intended to be limited to the exemplary environment 100 and search facility 200 introduced below with reference to
The information access facility 110 may include a browsing operation 112 which may include a navigation operation 114 and a user interface operation 116. The browsing operation 112 may access the network 160 via input/output interface operations 118. For example, in the context of a personal computer, the browsing operation 112 may be a browser (such as “Firefox” from Mozilla, “Internet Explorer” from Microsoft Corporation of Redmond, Wash., “Opera” from Opera Software, “Netscape” from Time Warner, Inc.) and the input/output interface operations may include a modem or network interface card (or NIC) and networking software. Other examples of possible information access facilities 110 include untethered devices, such as personal digital assistants and mobile telephones for example, set-top boxes, kiosks, etc.
Each of the content providers 180 may include stored resources (also referred to as content) 136, a resource retrieval operation 184 that accesses and provides content in response to a request, and input/output interface operation(s) 182. These operations of the content providers 180 may be performed by computers, such as personal computers or servers for example. Accordingly, the stored resources 186 may be embodied as data stored on some type of storage medium such as a magnetic disk(s), an optical disk(s), etc. In this particular environment 100, the term “document” may be interpreted to include addressable content, such as a Web page for example.
The search facility 130 may perform crawling, indexing/sorting, and query processing functions. These functions may be performed by the same entity or separate entities. Further, these functions may be performed at the same location or at different locations. In any event, at a crawling facility 150, a crawling operation 152 gets content from various sources accessible via the network 160, and stores such content, or a form of such content, as indicated by 154. Then, at an automated indexing/sorting facility 140, an automated indexing/sorting operation 142 may access the stored content 154 and may generate a content index (e.g., an inverted index, to be described below) and content ratings (e.g., PageRanks, to be described below) 140. Finally, a query processing operation 134 accepts queries and returns query results based on the content index (and the content ratings) 140. The crawling, indexing/sorting and query processing functions may be performed by one or more computers.
Although embodiments consistent with the present invention may be used with a number of different types of search engines, it might be used with an advanced search facility, such as the one presently available from Google, Inc. of Mountain View, Calif.
The advanced search facility 200 illustrated in
The advanced search facility uses the link structure of the Web, as well as other techniques, to improve search results. (See, e.g., U.S. Pat. No. 6,285,999 and the article S. Brin and L. Page, “The Anatomy of a Large-Scale Hypertextual Search Engine,” Seventh International World Wide Web Conference, Brisbane, Australia, both incorporated herein by reference.)
Referring back to
The crawling part 150′ may be distributed across a number of machines. A single URLserver (not shown) serves lists of uniform resource locations (“URLs”) 206 to a number of crawlers. Based on this list of URLs 206, the crawling operation 202 crawls the network 160′ and gets Web pages 208. A pre-indexing operation 210 may then generate page rankings 212, as well as a repository 214 from these Web pages 208. The page rankings 212 may include a number of URL fingerprint (i.e., a unique value), PageRank value (as introduced above) pairs. The repository 214 may include URL, content type and compressed page triples.
Regarding the indexing/sorting part 140′, the indexing/sorting operations 220 may generate an inverted index 226. The indexing/sorting operations 220 may also generate page ranks 228 from the citation rankings 212. The page ranks 228 may include document ID, PageRank value pairs.
Regarding the query processing part 134′, the searching operations 230 may be run by a Web server and may use a lexicon 232, together with the inverted index 226 and the PageRanks 228, to generate query results in response to a query. The query results may be based on a combination of (i) information derived from PageRanks 228 and (ii) information derived from how closely a particular document matches the terms contained in the query (also referred to as the information retrieval (or “IR”) component). Having described exemplary environments in which the present invention may be used, exemplary embodiments consistent with the present invention are now described in § 4.2 below.
Exemplary methods consistent with the present invention are described in § 4.2.1 below. Then, exemplary apparatus consistent with the present invention are described in § 4.2.2 below. Finally, refinements alternative and extensions of such embodiments are described in § 4.2.3 below.
§ 4.2.1 Exemplary Methods
Referring back to block 310, in some embodiments consistent with the present invention, the documents might be Web pages. In other embodiments consistent with the present invention, the documents might be a set of token sequence bit strings derived from source documents, such as Web pages for example.
Referring back to block 320, in some embodiments consistent with the present invention, the first document similarity technique might be order dependent, and/or frequency independent (e.g., with respect to document words, document n-grams, document tokens, etc.).
Still referring to block 320, in some embodiments consistent with the present invention, the first document similarity technique might include (i) fingerprinting every sub-sequence of k tokens to generate (n−k+1) shingles, (ii) fingerprinting each shingle with m different fingerprinting functions fi for 1≦i≦m to generate (n−k+1) values for each of the m fingerprinting functions fi, (iii) determining, for each i, the smallest value to create an m-dimensional vector of minvalues, (iv) reducing the m-dimensional vector of minvalues to an m′-dimensional vector of supershingles by fingerprinting non-overlapping sequences of minvalues, and (v) concluding that two documents are near-duplicates if and only if their supershingle vectors agree in at least two supershingles (or, alternatively, if the two documents agree in at least one megashingle). The first document similarity technique might be any of the techniques described in the Broder paper introduced in § 1.2.3 above or described in the Fetterly papers (D. Fetterly, M. Manasse, and M. Najork, “On the Evolution of Clusters of Near-Duplicate Web Pages,” 1st Latin American Web Congress, pp. 37-45 (November 2003); and D. Fetterly, M. Manasse, and M. Najork, “Detecting Phrase-Level Duplication on the World Wide Web,” 28th Annual International ACM SIGIR Conference (August 2005), both incorporated herein by reference). In such embodiments, the parameter m might be set to 84, the parameter l might be set to 14, the parameter m′ might be set to 6 and the parameter k might be set to any value from 5 to 10 (e.g., 8). Some embodiments consistent with the present invention might use the following parameter values: m=84; =14; m′=6; and k=some value from 5 to 10. Some embodiments consistent with the present invention might omit the wrapping of the shingling “window” from the end of the document, in which case (n−k+1) shingles are generated. If, on the other hand, the shingling window can wrap around from the end of the document to its beginning, n shingles are generated.
Referring back to block 330, in some embodiments consistent with the present invention, the second document similarity technique might be order independent, and/or frequency dependent (e.g., with respect to document words, document n-grams, document tokens, etc.).
Still referring to block 330, in some embodiments consistent with the present invention, the set of documents might be a set of token sequence bit strings, and the second document similarity technique might include (i) projecting each token into b-dimensional space by randomly choosing a predetermined number b of entries from {−1, 1}, (ii) for each document, creating a b-dimensional vector by adding the projections of all the tokens in its token sequence, and creating a final vector for the document by setting every positive entry in the b-dimensional vector to 1 and every non-positive entry to 0, and (iii) determining a similarity between two documents based on a number of bits that agree in corresponding projections of the two documents. The second document similarity technique might be any of the techniques described in the Charikar paper introduced in § 1.2.3 above. In such embodiments, the parameter b might be set such that a bit string of 48 bytes is stored per document (e.g., b might be set to 384). Naturally, some embodiments consistent with the present invention might select a value for the parameter b to use less (or more) space.
In some embodiments consistent with the present invention, the second technique might operate on overlapping sequences of tokens (i.e., shingles) instead of on individual tokens.
Still referring to block 330 of
As indicated by loop 420-460, a number of acts might be performed for each pair of near duplicate documents in the initial set. Specifically, a similarity value might be determined using the second document similarity technique. (Block 430) Whether the determined similarity value is less than a threshold might then be determined. (Decision block 440) If it is determined that the determined similarity value is less than the threshold, then the current pair of near duplicate documents might be removed from the initial set to generate an updated set (Block 450) before the method 400 continues to block 460. If, on the other hand, it is determined that the determined similarity value is not less than the threshold, then the method 400 might directly proceed to block 460.
Referring to block 460, once all of the pairs of near duplicate documents in the initial set have been processed, the final set of near duplicate documents might be set to the most recent updated set of near duplicate documents (or to the initial set of near duplicate documents in the event that the determined similarity value was never less than the threshold) (Block 470) before the method 400 is left (Node 480).
Referring back to block 410, in some embodiments consistent with the present invention, the set of documents might be Web pages. Alternatively, the set of documents might be a set of token sequence bit strings.
Referring back to block 430, in some embodiments consistent with the present invention, the second document similarity technique might include (i) projecting each of a number of token sequence bit strings into b-dimensional space by randomly choosing a predetermined number b of entries from {−1,1}, (ii) for each document, creating a b-dimensional vector by adding the projections of all the tokens in its token sequence, and creating a final vector for the document by setting every positive entry in the b-dimensional vector to 1 and every non-positive entry to 0, and (iii) determining a similarity between two documents based on a number of bits that agree in corresponding projections of the two documents. Referring back to block 440, in such an embodiment, the predetermined number b might be 384 and the threshold might be set to 372. In some embodiments consistent with the present invention, the threshold might be set to approximately 97% (or at least 96%) of the predetermined number b. The predetermined number b might be a lower value. For example, setting b to values as low as 192 has provided good results. Indeed, the present inventor believes that setting b to values of 100 or even slightly less might provide adequate results. In these lower settings of b, the threshold might be set to about 97% of b, or at least 96% of b.
§ 4.2.2 Exemplary Apparatus
The processor(s) 510 may execute machine-executable instructions (e.g., C or C++ running on the Solaris operating system available from Sun Microsystems Inc. of Palo Alto, Calif. or the Linux operating system widely available from a number of vendors such as Red Hat, Inc. of Durham, N.C.) to effect one or more aspects of the present invention. At least a portion of the machine executable instructions may be stored (temporarily or more permanently) on the storage device(s) 520 and/or may be received from an external source via an input interface unit 530.
Some aspects of exemplary embodiments consistent with the present invention may be performed in the general context of computer-executable instructions, such as program modules, being executed by a personal computer. However, methods may be performed by (and data structures may be stored on) other apparatus. Program modules may include routines, programs, objects, components, data structures, etc. that perform an operation(s) or implement particular abstract data types. Moreover, those skilled in the art will appreciate that at least some aspects of the present invention may be practiced with other configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network computers, minicomputers, set-top boxes, mainframe computers, and the like. At least some aspects of the present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked through a communications network. In a distributed computing environment, program modules may be located in local and/or remote memory storage devices.
In one embodiment consistent with the present invention, the machine 500 may be one or more conventional personal computers or servers. In this case, the processing unit(s) 510 may be one or more microprocessors, the bus 540 may include a system bus that couples various system components including a system memory to the processing unit(s). The system bus 540 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The storage devices 520 may include system memory, such as read only memory (ROM) and/or random access memory (RAM). A basic input/output system (BIOS), containing basic routines that help to transfer information between elements within the personal computer, such as during start-up, may be stored in ROM. The storage device(s) 520 may also include a hard disk drive for reading from and writing to a hard disk, a magnetic disk drive for reading from or writing to a (e.g., removable) magnetic disk, and an optical disk drive for reading from or writing to a removable (magneto-) optical disk such as a compact disk or other (magneto-) optical media. The hard disk drive, magnetic disk drive, and (magneto-) optical disk drive may be coupled with the system bus 540 by a hard disk drive interface, a magnetic disk drive interface, and an (magneto-) optical drive interface, respectively. The drives and their associated storage media may provide nonvolatile storage of machine-readable instructions, data structures, program modules and other data for the personal computer. Although the exemplary environment described herein employs a hard disk, a removable magnetic disk and a removable optical disk, those skilled in the art will appreciate that other types of storage media (with appropriate interface devices), may be used instead of, or in addition to, the storage devices introduced above.
A user may enter commands and information into the personal computer through input devices 532, such as a keyboard and pointing device (e.g., a mouse) for example. Other input devices such as a microphone, a joystick, a game pad, a satellite dish, a scanner, or the like, may also (or alternatively) be included. These and other input devices are often connected to the processing unit(s) 510 through a serial port interface 530 coupled to the system bus 540. Input devices may be connected by other interfaces 530, such as a parallel port, a game port or a universal serial bus (USB). However, in the context of a search facility 130, no input devices, other than those needed to accept queries, and possibly those for system administration and maintenance, are needed.
The output device(s) 534 may include a monitor or other type of display device, which may also be connected to the system bus 540 via an interface 530, such as a video adapter for example. In addition to (or instead of) the monitor, the personal computer may include other (peripheral) output devices (not shown), such as speakers and printers for example. Again, in the context of a search facility 130, no output devices, other than those needed to communicate query results, and possibly those for system administration and maintenance, are needed.
The computer may operate in a networked environment which defines logical and/or physical connections to one or more remote computers, such as a remote computer. The remote computer may be another personal computer, a server, a router, a network computer, a peer device or other common network node, and may include many or all of the elements described above relative to the personal computer. The logical and/or physical connections may include a local area network (LAN) and a wide area network (WAN). An intranet and the Internet may be used instead of, or in addition to, such networks.
When used in a LAN, the personal computer may be connected to the LAN through a network interface adapter (or “NIC”) 530. When used in a WAN, such as the Internet, the personal computer may include a modem or other means for establishing communications over the wide area network. In a networked environment, at least some of the program modules depicted relative to the personal computer may be stored in the remote memory storage device. The network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
Referring once again to
§ 4.2.3 Refinements, Alternatives and Extensions
Although embodiments consistent with the present invention were described as processing Web page documents, embodiments consistent with the present invention can operate on various other types of documents (e.g., snippets extracted from other documents, text documents, spreadsheets, database records, media streams, emails, email SPAM, bit sequences, genetic sequences, nucleotide sequences, representations of chemical structures, representations of molecular structures, characteristics of physical objects, etc.). Thus, embodiments consistent with the present invention have various applications. For example, embodiments consistent with the present invention might be used for spam detection, since spam is often replicated many times, perhaps with subtle differences. As another example, embodiments consistent with the present invention might be use to detect redundant snippets of news stories.
Some near-duplicate document detection algorithms perform poorly on pairs of Web pages from the same Website. The present inventor believes that this is mostly due to boilerplate text. In some alternative embodiments consistent with the present invention, boilerplate might be detected, and then removed or ignored in near-duplicate document analysis. Alternatively, or in addition, an algorithm used to analyze Web pages on the same Website to find near-duplicate documents might be different (and potentially slower) than another algorithm used to analyze pairs of Web pages on different Websites.
Referring back to block 320 of
Still referring to
In some exemplary embodiments described above, a Charikar-based technique was run after a Broder-Fetterly-based technique. This is because a Charikar-based technique can be tuned to a finer degree (e.g., 372 bits of 384 bit vectors match) than a Broder-Fetterly-based technique (e.g., 2 of 6 matching supershingles). Thus, in such embodiments, the second technique can be tuned to a finer degree than the first technique. However, in alternative embodiments consistent with the present invention, other considerations might be used in determining which technique to run first. For example, in some embodiments consistent with the present invention, the second technique might take longer to run (and/or require more storage) than the first technique.
Recall that documents may be processed to generate tokens. Some embodiments consistent with the present invention might apply special processing to URLs and/or images. For example, in some embodiments consistent with the present invention, every URL contained in the text of the page might be broken at slashes and dots, and treated like a sequence of individual terms. In some embodiments consistent with the present invention, in order to distinguish pages with different images, the URL in an IMG-tag might be considered to be a term in the page. More specifically, if the URL points to a different host, the whole URL might be considered to be a term. If, on the other hand, it points to the host of the page itself, only the filename of the URL might be used as term. Thus if a page and its images on the same host are mirrored on a different host, the URLs of the IMG-tags might generate the same tokens in the original and mirrored version. URLs can be processed using other, alternative techniques. Indeed, some embodiments consistent with the present invention might ignore URLs, or simply treat URLs as a term.
Although exemplary embodiments described above might use Rabin's fingerprinting technique to generate tokens, tokens can be generated using other fingerprinting techniques (e.g., fingerprinting techniques referenced in the Hoad and Zobel paper).
Some embodiments consistent with the present invention might increase recall by, in addition to determining a “final” set of near-duplicate documents as described above, determining a second final set of near-duplicate documents using a Charikar-based technique (preferably with a high threshold, such as 97% of b, or at least 96% of b). A union of the final set and second final set of near duplicate documents is taken to obtain a “high recall” set of near duplicate documents.
As can be appreciated from the foregoing, improved near-duplicate detection techniques are disclosed. These near-duplicate detection techniques performed well, particularly when analyzing Web pages from the same Website. These techniques did so without sacrificing much in the number of returned correct pairs.
