Patent Application
20080168049
A parallel corpus is a collection of documents where the content of the documents is provided in multiple separate languages. Examples of such parallel corpora include European Parliament Proceedings, which are written in eleven European languages, and biblical text, which has been written in a number of languages. Parallel corpora are valuable resources for training machine translation systems, cross-language information retrieval systems and other data driven natural language processing systems.
Documents that can be used to form parallel corpora can also be found in multi-lingual websites on the Internet. Such sites typically provide the same content in different languages on different parallel pages of the site. Thus, one page may provide the content in English while another page provides the same content in Chinese. For bilingual websites, the two parallel pages are referred to as a parallel pair.
Given the size of the Internet, an automatic system is needed to identify websites that may contain parallel pages and to identify the specific pages that form parallel pairs.
The discussion above is merely provided for general background information and is not intended to be used as an aid in determining the scope of the claimed subject matter.
Network pages are identified based on whether the pages include anchor text and/or image alternative text that indicate that the network pages contain links to pages that are translations of each other. A plurality of pages and a plurality of respective uniform resource locators are downloaded from a server associated with the domain name of the identified network pages. The uniform resource locators are used to identify a set of candidate parallel page pairs and a set of features are created for each candidate parallel page pair. The sets of features are used to identify parallel page pairs, wherein the pages in a parallel page pair are translations of each other.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the background.
Embodiments described herein identify pages on a network that are translations of each other. These pages are referred to as parallel pairs. The embodiments involve identifying candidate sites that may contain parallel pairs, identifying candidate parallel pairs, and verifying that the candidate parallel pairs are translations of each other.
In step 100, a candidate site identifier 200 of a parallel page identifier 202 searches for websites with specific text to identify candidate network sites that may contain parallel pages. Under one embodiment, candidate site identifier 200 submits a search to a search engine 204 that includes an index 206. Based on the search criteria in the search request, search engine 204 examines index 206 to identify web pages, such as pages in domain site pages 218, 220 and 22, that include the text provided by candidate site identifier 200. Search engine 204 then returns the uniform resource locator (URL) for each of the web pages that it finds.
Under one embodiment, the text that candidate site identifier 200 searches for includes a list of predefined strings that include some type of reference to a language.
Under some embodiments, to avoid identifying incorrect web pages, the search is limited to anchor text and image alternative text. Anchor text is text found between an open anchor tag, <a>, and close anchor tag, </a>, in a Hyper-Text Markup Language (HTML) document. Such anchor tags are used to identify links to network pages. Within the open anchor tag, the link to another network page is defined by setting an “href” attribute equal to the uniform resource locator (URL) of the linked network page. The text or image to be displayed on the current page to represent the link is placed between the open anchor tag and close anchor tag. For example: <a href=“http://www.xxxx.com/aa/bb/eng/cc/content_e.html”> English Version </a>
In this anchor tag structure, “English Version” would be displayed on the current page and when a user clicked on this phrase, the network browser would request and display the web page at the URL: http://www.xxxx.com/aa/bb/eng/cc/conetent_e.html.
In HTML it is also possible to include an image between the open anchor tag and the close anchor tag to allow an image to represent the link such that if the user clicks on the image, the web page defined by the URL in the open anchor tag will be requested. To identify the image, an image tag, <img>, is inserted between the open and close anchor tags. Within the image tag, a source attribute, “src”, provides the network path to the file that contains the image, and an alternative text attribute, “alt”, provides text that is to be displayed on the page if the image can not be located or can not be rendered. For example:
Thus, under some embodiments, both the anchor text and the image alternative text is searched to determine if it contains the list of predefined strings that are associated with candidate pairs such as the strings found in
Based on the search of anchor text and image alternative text, search engine 204 returns a list 210 of uniform resource locators for network pages that include the search strings, such as those in
At step 102, candidate site identifier 200 uses URL list 210 to download all of the pages associated with the domain name of each URL in URL list 210. The domain name is the portion of the URL after the prefix http:// and before the next forward slash “/”. For example, in the URL examples above, the domain name is www.xxxx.com. Typically, the network pages for a domain name are stored on one or more servers for the domain. To download the pages for a domain name, any of a large number of known tools such as “wget”, which is available at http://www.gnu.org/software/wget may be used. These tools request the pages from the domains such as domain site pages 218, 220 and 222. These downloaded pages are then stored as local downloaded pages 224 that have a directory hierarchy based on the hierarchies in the domains. In addition to downloading the pages, the URL for each page is also downloaded and stored.
At step 104, a candidate pairs identifier 226 in parallel page identifier 202 uses the URLs of the downloaded pages 224 to identify candidate pairs 228, which represent pages that may be translations of each other.
In the method of
At step 400, a base pattern is selected. For example, under one embodiment, the base patterns consists of “e”, “en”, “eng”, “engl” and “English.” At step 402, the URLs in downloaded pages 224 are searched to identify URLs that contain the base pattern. At step 404, if a URL is found that contains the base pattern, an alternative pattern is selected at step 406. An alternative pattern is a character or sequence of characters that indicates a document in another language. For example, in an embodiment for Chinese, the alternative pattern list would include “c”, “ch”, “chi” and “Chinese.”
After selecting one of the alternative patterns from the alternative pattern list, the URL that contains the base pattern is modified at step 408 to form a modified URL by replacing the base pattern with the selected alternative pattern. At step 410, the URLs associated with the same domain name as the URL that contained the base pattern at step 402 are searched to determine if any of the URLs are within an edit distance threshold of the modified URL. The edit distance may be calculated in any of a number of known manners including adding one to the edit distance for each insertion, deletion, or movement of a character that is needed to transform the URL under consideration into the modified URL. Under many embodiments, the edit distance threshold is greater than one such that a candidate pair may be identified even though there are differences between the modified URL and one of the downloaded URLs.
At step 412, the process determines if at least one URL is within the edit distance threshold of the modified URL. If none of the URLs are within the edit distance threshold of the modified URL, the process continues at step 414 where a determination is made as to whether there are additional alternative patterns that need to be considered. If there are more alternative patterns, the process continues at step 406 by selecting the next alternative pattern and steps 408, 410 and 412 are repeated for the new alternative pattern. If there are no more alternative patterns, the process returns to step 402 to continue to search for URLs that contain the base pattern.
If there is at least one URL that is within the edit distance threshold of the modified URL at step 412, the best matching URL is selected at step 416. Under one embodiment, the best matching URL is the URL with the smallest edit distance. At step 418, the best matching URL determined at step 416, and the URL with the base pattern are removed from further consideration and their pages are placed as candidate pairs in candidate pairs 228. The process then returns to step 414 to determine if there are additional alternative patterns that should be searched.
The search for URLs that contain the base pattern continues until no URLs are found at step 404. The process then determines if there are more base patterns in the base patterns list at step 420. If there are more base patterns, the next base pattern is selected by returning to step 400 and the steps described above are performed for the newly selected base pattern. When there are no more base patterns in the base pattern list at step 420, the process of
After the candidate pairs have been identified at step 104 of
The first feature is a file length ratio, which is the number of bytes in one page of the candidate pair divided by the number of bytes in the other page of the candidate pair. A second feature is the difference between the HTML structures of the two pages in the candidate pair. To determine the difference in the HTML structures, a linear sequence of HTML tags is extracted from each page of the candidate pair and the case of the tags is normalized to either all uppercase or all lowercase. In addition, attributes such as “meta”, “font” and “scripts” are removed from the tags. The linear sequences of HTML tags are then compared to one another to identify tags that are found in one but not the other page. An example of such a tool is sdif, which is available at http://linexcommand.org/man_pages/sdif1.html. For example, if two pages have the following linear sequences of HTML tags:
Then sdif would produce the following results:
Under one embodiment, the difference score is determined as the ratio of the number of unaligned lines divided by the total number of aligned lines and the total number of unaligned lines. For example, in the example above, the number of unaligned lines is four and the total number of aligned and unaligned lines is twelve resulting in a structural difference score of 4/12=⅓. In general, lower difference scores are associated with more similar pages.
The third feature extracted from the two pages is a measure of the similarity of the non-HTML content on the page. To determine this similarity, the HTML tags are removed from the pages and the remaining text is applied to a translation alignment tool that aligns sentences in the two pages based on a bilingual dictionary and/or a statistical translation model. Under one embodiment, the Champollion Tool Kit, which is available at http://champollion.sourceforge.net/, is used to perform the alignment. The score for the content similarity is then determined as the ratio of the number of aligned sentences over the number of aligned and unaligned sentences.
Once the features for the candidate pairs have been determine, the process of
At step 112, the process determines if there are candidate pairs to be classified. If there are no candidate pairs to be classified, all of the candidate pairs that were identified in step 104 were used to form the training candidate pairs. As such, the process returns to step 100 to locate new candidate pairs that can be classified.
If candidate pairs are available to be classified at step 112 or if training was not needed at step 108, a candidate pair from candidate pairs 228 is selected for classification by parallel page verification unit 238 at step 114. At step 116, the features for the selected candidate pair are applied to a k-nearest neighbor classifier 236. K-nearest neighbor classifier 236 uses training candidate pairs with manual classification 232 to identify the training candidate pairs that have the closest feature vectors to the feature vector of the selected candidate pair. Each feature vector consists of the values of the features determined in step 106. The distance between feature vectors can be determined either as a Euclidean distance or as an angular distance. The k training candidate pairs that have the nearest feature vectors to the feature vector of the selected candidate pair are identified by k-nearest neighbor classifier 236.
The classifications of the k-nearest neighbor training candidate pairs are then examined to determine which classification is most common among the k-nearest neighbors. For example, if a majority of the k-nearest neighbor training candidate pairs was classified as containing parallel pages, k-nearest neighbor classifier 236 would classify the selected candidate pair as containing parallel pages. If a majority of the k-nearest neighbor training candidate pairs were classified as not containing parallel pages, the selected candidate pair would be classified by as not containing parallel pages.
Under one embodiment, a tenfold cross-validation experiment was conducted to identify the optimal value for k. Under some embodiments, k=15 was set for three-dimensional feature vectors and k=7 was set for two-dimensional feature vectors.
If k-nearest neighbor classifier 236 classifies a candidate pair as a parallel page, parallel page verification unit 238 stores the candidate pair as parallel pages 240 at step 118.
At step 120, the process of
Embodiments are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with various embodiments include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, telephony systems, distributed computing environments that include any of the above systems or devices, and the like.
Embodiments may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Some embodiments are designed to be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules are located in both local and remote computer storage media including memory storage devices.
With reference to
Computer 510 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 510 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 510. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer readable media.
The system memory 530 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 531 and random access memory (RAM) 532. A basic input/output system 533 (BIOS), containing the basic routines that help to transfer information between elements within computer 510, such as during start-up, is typically stored in ROM 531. RAM 532 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 520. By way of example, and not limitation,
The computer 510 may also include other removable/non-removable volatile/nonvolatile computer storage media. By way of example only,
The drives and their associated computer storage media discussed above and illustrated in
A user may enter commands and information into the computer 510 through input devices such as a keyboard 562, a microphone 563, and a pointing device 561, such as a mouse, trackball or touch pad. These and other input devices are often connected to the processing unit 520 through a user input interface 560 that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 591 or other type of display device is also connected to the system bus 521 via an interface, such as a video interface 590.
The computer 510 is operated in a networked environment using logical connections to one or more remote computers, such as a remote computer 580. The remote computer 580 may be a personal computer, a hand-held device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 510. The logical connections depicted in
When used in a LAN networking environment, the computer 510 is connected to the LAN 571 through a network interface or adapter 570. When used in a WAN networking environment, the computer 510 typically includes a modem 572 or other means for establishing communications over the WAN 573, such as the Internet. The modem 572, which may be internal or external, may be connected to the system bus 521 via the user input interface 560, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 510, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation,
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
