THE FIGURES
FIG. 1 is a flowchart of a prior art technique.
FIG. 2A is a flowchart in accordance with an exemplary embodiment of the present invention.
FIG. 2B is a flowchart in accordance with a further embodiment of the present invention.
FIG. 2C is a flowchart in accordance with an exemplary embodiment of the present invention.
FIG. 3 is a flowchart of the deep-scanning step in accordance with an exemplary embodiment of the present invention.
FIG. 4 is a flowchart of the reverse-scanning step in accordance with an exemplary embodiment of the present invention.
FIG. 5A is a flowchart of the performing step in accordance with an exemplary embodiment of the present invention.
FIG. 5B is a flowchart of the performing step in accordance with an exemplary embodiment of the present invention.
FIG. 5C is a flowchart of the performing step in accordance with an exemplary embodiment of the present invention.
FIG. 5D is a flowchart of the performing step in accordance with an exemplary embodiment of the present invention.
FIG. 6 is a flowchart of the computing step in accordance with an exemplary embodiment of the present invention.
DETAILED DESCRIPTION OF THE INVENTION
The present invention provides a method, system, and service of analyzing electronic documents in an intranet, where the intranet includes a plurality of web sites. In an exemplary embodiment, the method, system, and service include (1) crawling HTML content and text content in a set of the sites, (2) deep-scanning non-HTML content and non-text content in the set of sites, (3) reverse-scanning the set of sites, (4) performing a semantic analysis of the crawled content and the deep-scanned content, (5) correlating the results of the semantic analysis with the results of the reverse-scanning, and (6) comparing user navigation patterns and content from the members of the set of sites. In a further embodiment, the method, system, and service further include combining the results of the performing, the results of the correlating, and the results of the comparing.
Referring to FIG. 2A, in an exemplary embodiment, the present invention includes a step 212 of crawling HTML content and text content in a set of the sites, a step 214 of deep-scanning non-HTML content and non-text content in the set of sites, a step 216 of reverse-scanning the set of sites, a step 218 of performing a semantic analysis of the crawled content and the deep-scanned content, a step 220 of correlating the results of the semantic analysis with the results of the reverse-scanning, and a step 222 of comparing user navigation patterns and content from the members of the set of sites. Referring next to FIG. 2B, in a further embodiment, the present invention further includes a step 230 of combining the results of the performing, the results of the correlating, and the results of the comparing.
In an exemplary embodiment, the method, system, and service include (1) crawling HTML content and text content in a set of the sites, (2) deep-scanning non-HTML content and non-text content in the set of sites, (3) reverse-scanning the set of sites, (4) performing a semantic analysis of the crawled content and the deep-scanned content, (5) correlating the results of the semantic analysis with the results of the reverse-scanning, (6) comparing user navigation patterns and content from the members of the set of sites, and (7) combining the results of the performing, the results of the correlating, and the results of the comparing.
Referring to FIG. 2C, in an exemplary embodiment, the present invention includes a step 242 of crawling HTML content and text content in a set of the sites, a step 244 of deep-scanning non-HTML content and non-text content in the set of sites, a step 246 of reverse-scanning the set of sites, a step 248 of performing a semantic analysis of the crawled content and the deep-scanned content, a step 250 of correlating the results of the semantic analysis with the results of the reverse-scanning, a step 252 of comparing user navigation patterns and content from the members of the set of sites, and step 254 of combining the results of the performing, the results of the correlating, and the results of the comparing.
Crawlin
In an exemplary embodiment, crawling step 212 includes fetching documents from web sites in the intranet. Crawling step 212 includes crawling all HTML and text pages from the intranet (i.e. enterprise) web-sites, while honoring standard internet crawling practices (e.g. politeness, robots.txt, meta-tags such as no-index, no-follow). As each crawled document can contain links to other documents that may possibly be on different sites, crawling step 212 crawls only the set of sites targeted for analysis.
Crawling step 212 further includes (a) writing each crawled document to a file-based storage system and (b) annotating each crawled document. In an exemplary embodiment, the annotating includes de-tagging the document, generating hash and shingle values based on document content, determining domains, and/or determining sub-domains.
Once the annotating is complete, crawling step 212 generates the following 4-tuple entry for each annotated document:
- Document-id Hash-value Size Domain/Sitename.
The tuple is used in performing step 218 (e.g., in duplicate analysis). Additionally, crawling step 212 writes the URLs crawled for each of the targeted sites to a central list for use in reverse-scanning step 216.
Deep-Scanning
Referring to FIG. 3, in an exemplary embodiment, if the text content for a member of the set of sites is greater than one megabyte, deep-scanning step 214 includes a step 310 of crawling the first megabyte of the text content of the member and a step 312 of using the first megabyte of the text content of the member for hash generation. In an exemplary embodiment, the deep-scanning fetches all static non-HTML content and non-text content (e.g., .pdf files, images, .exe files, Microsoft PowerPoint files) from the sites. For each URL fetched by deep-scanning, a tuple similar to the one generating in crawling step 212 is generated for duplicate analysis. In addition, the URL is written to a central list for reverse-scanning step 216.
Reverse-Scanning
Referring to FIG. 4, in an exemplary embodiment, reverse-scanning step 216 includes a step 410 of clustering the members of the set of sites into sitename-based sets, a step 412 of, for each of the sitename-based sets, generating a list of Internet Protocol (IP) addresses with open http ports in the same subnet as the sitename-based set, and a step 414 of, for each of the sitename-based sets, scanning all IP addresses for the members of the sitename-based set. In an exemplary embodiment, reverse-scanning step 216 (a) takes a set of URLs and another set of IP addresses and (b) scans the set of IP addresses using the URLs to determine content overlap with the given URL set. Thus, for each URL-IP pair thus formed, a content hash is generated and compared against the content hash from the original URL.
In an exemplary embodiment, reverse-scanning step 216 fetches content from the sites in the same subnet as the targeted sites in order to perform an “intra-server” duplicate analysis. In an exemplary embodiment, reverse-scanning step 216 (a) checks for servers with open http ports in the same subnet as the targeted sites and (b) then makes GET calls to these http sites for the HTML pages that were found on the targeted servers (by replacing the sitename in the original URL with the sitename of the http site).
Performing
Referring to FIG. 5A, in an exemplary embodiment, performing step 218 includes a step 510 of performing exact duplicate detection on the text content of each member of the set of sites. In an exemplary embodiment, performing step 510 includes generating hash-values for the text content of each member of the set of sites and comparing these hash-values against each other. Referring to FIG. 5B, in an exemplary embodiment, performing step 218 includes a step 512 of performing near-duplicate detection on the text content of each member of the set of sites. In an exemplary embodiment, performing step 512 includes employing the near-duplicate detection technique described at the following URL:
- http://www.informatik.uni-trier.de/˜ley/db/conf/cpm/cpm2000.html#Broder00
Referring to FIG. 5C, in an exemplary embodiment, performing step 218 includes a step 514 of performing main topics detection on the text content of each member of the set of sites. In an exemplary embodiment, performing step 514 includes employing term identification techniques, classification techniques, and tfidf (term-frequency inverse document frequency) techniques. Referring to FIG. 5D, in an exemplary embodiment, performing step 218 includes a step 516 of aggregating topics in the text content of each member of the set of sites by organizational structure.
In an exemplary embodiment, performing step 218 (a) merges the set of tuples generated during crawling step 212 and deep-scanning step 214 and (b) compares the content hashes for each page on a site against pages from all other sites.
Correlating
In an exemplary embodiment, correlating step 220 includes correlating user navigation patterns from standard website analysis metrics and specific content from domains and reverse scanned site.
Combining Referring to FIG. 6 in an exemplary embodiment, combining step 230 includes a step 610 of listing the combined results in descending order of duplicate content. In an exemplary embodiment, listing step 610 generates an overlap report that contains information about the amount of overlap on each of the sites scanned. Additionally, the data from reverse-scanning of each site is added to the information of each site. The overlap report thus produced lists all sites in descending order of content that is overlapped.
In an exemplary embodiment, listing step 610 generates the overlap report for each site, where the overlap reports depicts the amount of duplicate content on that site, and the list of sites with which content is duplicated.
Conclusion
Having fully described a preferred embodiment of the invention and various alternatives, those skilled in the art will recognize, given the teachings herein, that numerous alternatives and equivalents exist which do not depart from the invention. It is therefore intended that the invention not be limited by the foregoing description, but only by the appended claims.
Patent Application
20070282799
