Patent Grant
7711682
The present invention relates to a method and system for searching Web information, and in particular to a method and system for searching hypertext based multilingual Web information.
Along with the popularization of the Internet, the amount of the content information on the Internet increases dramatically and therefore a method for searching the content to be queried on the Internet by using keywords is also used widely. Especially non-English users represent the fastest growing group of new Internet users, who require obtaining information not only from information sources expressed in their native languages, but also from a large collection of multilingual documents. On the other hand, technologies for the Internet application globalization bring unified methodology to build multilingual Web sites to serve the visitors from the world.
Since most users prefer to search the Web in their mother tongues or it is difficult for them to express the keywords in other languages, for example, it is difficult for many users in non-English countries to express the keywords in English, which is most commonly used by the Internet content. Therefore, these users can only find limited or relative localized information based on current content match approach. To solve the problem, translation based approaches are proposed. These methods use a translation engine, which translates user queries to different languages and then submits to different search engines. The drawbacks of these solutions are obvious: first, machine translation is not as accurate as human translation, and some terms are difficult to translate to target languages that could be understood by search engines; secondly, translation based solution is difficult to be scaled with low cost and effectively, since the all queries must be first caught and translated before being submitted. Huge amount of queries will bring heavy load to the translation engine.
A general aspect of the invention is to provide solutions to the problem of language translation in the searching of multilingual Web information. The present invention uses a hyperlink approach to solve the language translation problems. The present invention helps global Internet users to find authority Web pages by the query in his/her native language (as described below).
A particular aspect of the invention is to provide methods of searching hypertext based multilingual Web information, searching on a network for keywords to be queried. An example of a method includes the following steps: a receiving step for receiving keywords input by a user; a native language hypertext searching step for searching on the network, according to the keywords to be queried, for all hypertexts whose representing language is the same as a language representing the keywords and which matches the keywords to be queried; a hyperlink extracting step for extracting hyperlinks related to an arbitrary language from all the searched hypertexts; a hyperlink ranking step for ranking the extracted hyperlinks according to the correlativity of the hyperlinks with the keywords to be queried; and an output step for returning to the user ranked search result.
Another aspect of the invention is to provide a system to search hypertext based multilingual Web information, searching on a network for keywords to be queried. An example of a system includes: receiving means for receiving keywords input by a user; native language hypertext searching means for searching on the network, according to the keywords to be queried, for all hypertexts whose representing language is the same as a language representing the keywords and which matches the keywords to be queried; hyperlink extracting means for extracting hyperlinks related to an arbitrary language from all the searched hypertexts; hyperlink ranking means for ranking the extracted hyperlinks according to the correlativity of the hyperlinks with the keywords to be queried; and an output means for returning to the user ranked search result.
The above aspect and features of the present invention will become more apparent from the following description in conjunction with the drawings, in which:
The present invention provides solutions solving the problems of language translation in the searching of multilingual Web information. It uses a hyperlink approach to solve these language translation problems. The present invention will help global Internet users to find authority Web pages by the query in his/her native language, as described below.
In order to implement the solutions, the present invention provides methods, systems and apparatus for searching hypertext based multilingual Web information, when searching on a network for keywords to be queried. A particular embodiment includes the following steps: a receiving step for receiving keywords input by a user; a native language hypertext searching step for searching on the network, according to the keywords to be queried, for all hypertexts whose representing language is the same as a language representing the keywords and which matches the keywords to be queried; a hyperlink extracting step for extracting hyperlinks related to an arbitrary language from all the searched hypertexts; a hyperlink ranking step for ranking the extracted hyperlinks according to the correlativity of the hyperlinks with the keywords to be queried; and an output step for returning to the user ranked search result.
The present invention also provides a system of searching hypertext based multilingual Web information for searching on a network for keywords to be queried. An example system includes: receiving means for receiving keywords input by a user; native language hypertext searching means for searching on the network, according to the keywords to be queried, for all hypertexts whose representing language is the same as a language representing the keywords and which matches the keywords to be queried; hyperlink extracting means for extracting hyperlinks related to an arbitrary language from all the searched hypertexts; hyperlink ranking means for ranking the extracted hyperlinks according to the correlativity of the hyperlinks with the keywords to be queried; and an output means for returning to the user ranked search result.
By using the hypertext based multilingual Web information searching method and system, an accurate cross language searching can be provided without extra machine translation effort. Moreover, conducting multilingual Web information searching by using hypertexts is more accurate and objective than machine translation, even than human translation. An advantage of the present invention is that it can establish the relationship between the keywords expressed in a user's mother tongue (native language) and target documents without translation by identifying a Web hyperlink structure and collecting HTML hyperlink texts linked to Web pages (abbreviated as “hypertext”) from multilingual Web pages.
From a study of the Web the following facts are noted. First, although most high quality Web contents are composed in English language, there are translated versions scattered around the Web, and these globalized contents are either tailored or simply a hyperlink to their original version locations. Second, many Web surfers wish to find a start point to solve their particular problems, and these problems could be programming tips, official FAQs (frequently asked questions) or how to begin a new search topic. For such search requirements, users want the authority pages of the solutions, regardless of whether these pages are in their native language. The authority pages and a central Web page show a relation of strengthening each other: a good central Web page represents that a Web page points to many good authority pages; and a good authority page represents that a Web page is pointed by many good central Web pages.
Web document includes navigation links by placing hyperlink in the page. Usually the page author will assign the hyperlink a description text to tell the reader what the hyperlink is about, and the description is called hypertext or anchor text. A Web page may be linked by thousands of pages, or even millions of for popular sites. Each hypertext pointed to a same URL (Web page) may be varied due to different page composers and their preferred languages. For example, the homepage of Carnegie Mellon University is www (followed by) cmu.edu, CMU. However, in a Chinese Web page, etc. including Chinese characters, the hyperlink to the university homepage will be “ (Carnage Mellon University)”, and so on. For another example, the homepage of Kyoto University is www (followed by) Kyoto-u.acjp, while in the Chinese and Japanese Web pages, etc. including Chinese characters, the hyperlink text of the university will be “
(Kyoto University)”. Here, “
(Kyoto University)” is both Japanese hypertext and Chinese hypertext. It can be understood that, along with the development of international exchange, hypertexts of different languages can be provided on a same Web page or Web site to facilitate readers' queries. For example, a Chinese hypertext can be provided on a Japanese Web site or an English Web site, and a hypertext in another language, such as an English hypertext, can also be provided on a Chinese Web page. After a reader conducts a hypertext query based on his familiar language, a further selection is conducted as to whether to read the original content linked to by the hypertext.
The present invention only defines the representing language of the hypertext to be searched is identical to the representing language of keywords. Those skilled in the art can understand that defining the type of the Web page language to be searched is not necessary for the present invention.
Therefore, there are multilingual hypertexts for a same URL (Web page), and these hypertexts in different languages are optimized “translation” because the hypertexts are not word to word translation but human interpretation that tells the visitors what the page is about. Moreover, hypertext is more accurate and objective than machine, or even human translation.
That is, the basic idea of hypertext based multilingual Web search is to retrieve related URLs according to the hypertexts and their target URLs. Useful Internet resources can be distinguished by using “concept” translation and expanding queries through iterative searching regardless of the language used by each Web page. In addition, by using a statistical approach, hypertext based searching further removes the hyperlinks such as “click it”, which has little value.
For example, supposing a Chinese Internet user tries to locate the homepage of “Reader's Digest” magazine, he/she will input “ (Reader's Digest)” (keyword) expressed in Chinese, since many Chinese Web pages include hyperlinks to the Web site of the magazine of “Reader's Digest” and most of the hypertexts corresponding to the hyperlinks include “Reader's Digest” expressed in Chinese (
(Reader's Digest)), by matching the hypertexts with the keyword and analyzing the hyperlink distribution, the URL www (followed by) rd.com of the magazine of “Reader's Digest” can be retrieved.
Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
Below, the process that the searching system of the present invention provides a searching service will be described.
Before providing the searching service, Web crawler 1 downloads from the Internet 2 Web pages as many as possible, and stores them in a Web repository 31. Then the data index module 3 extracts from the Web pages all the hyperlink texts and corresponding hyperlink addresses as well as Web page titles, and provides indices for fast access. The data index module 3 completes primitive processing of the data, makes the above index data form a URL DB (database) and stores them in the Web repository 31. The Web repository 31 includes a URL DB and a URL registry. The URL registry is used for accessing quickly the data entry in the URL DB, that is to say, once a certain URL is specified, the data corresponding to the URL can be found directly in the URL DB according to the URL registry, without searching from the first entry to the last entry in the URL DB. In
Via the searching interface 6, a user submits a search service request, i.e., submits the keyword to be queried, then the searching interface 6 transfers the keyword to be queried to the analysis evaluation module 5. The analysis evaluation module 5 dynamically matches the keyword to be queried based on the existing index data. The analysis evaluation module 5 comprises a hypertext IR (information retrieving) 51 and a hyperlink analysis 52. The hypertext IR 51 computes the similarity S of each hypertext with respect to the keyword to be queried, while the hyperlink analysis 52 computes the rank of each Web page (URL), and then the analysis evaluation module 5 conducts comprehensive evaluation based on the computing results of the hypertext IR 51 and the hyperlink analysis 52, and ranks the results of comprehensive evaluation. Finally, the analysis evaluation module 5 returns the ranking result to the user through the searching interface 6, and then the user can find the URL meeting the requirements most closely.
Referring to (Shang Hai Jiao Tong Da Xue)”. After receiving the keyword in the language A (e.g., Chinese), such as “
(Shang Hai Jiao Tong Da Xue)”, at step 102, all the hypertexts expressed in the language A are searched and are matched with the keyword such as “
(Shang Hai Jiao Tong Da Xue)”. Preferably, the search can be implemented quickly by using the index data in the URL DB downloaded and established by the data index module 3 from the Internet. As described above, the text index 32 shown in
(Shang Hai Jiao Tong Da Xue)” include “
(Jiao Da)”, “
(Shang Hai Jiao Da)”, “
(Jiao Tong Da Xue)”, “
(Shang Hai Jiao Tong Da Xue)”, “
(Shang Jiao Da)” and “
(Shang Hai Jiao Da)”.
At step 103, a URL is extracted from all the matched hypertexts expressed in the language A, and the extracted URL relates not only to pointing to the Web page expressed in language A, but also to pointing to the Web page expressed in the language other than the language A. Therefore, by extracting the URL to which the hypertext corresponds, the expansion to multilingual information searching is implemented. Preferably, by using the “URL index” in the URL DB described below in
At step 104, a similarity S of the hypertext with respect to the keyword to be queried is computed. Each of hypertext di and keyword Q can be represented by a vector. The cosine value of the angle between the two vectors represents their similarity S, and VSM (vector space model) expressed by the following equation is introduced to compute the text similarity S.
where di represents the i'th hypertext in the Web page d, dij represents the j'th dimension of the i'th hypertext, and qj represents the j'th dimension of the keyword Q. The j'th dimension means that, each of a hypertext and a keyword can be divided into a multi-dimensional vector, for example, a hypertext “IBM software group” is a three dimensional vector, while a keyword “IBM software development tool” is a four dimensional vector. For the Chinese language, each Chinese character corresponds to one dimension, while for the English language, each word corresponds to one dimension. If the hypertext di conforms to the keyword Q completely, the value of S is 1; if no word in the hypertext di appears in the keyword Q, then the value of S is 0.
Meanwhile, at step 105, the rank of a URL (Web page) is computed. Popular Web pages can be found through Web hyperlink tracing, and there are more hyperlinks pointing to high quality Web pages than average. The rank of a URL (Web page) can be obtained by the following recursive and/or iterative computation.
where, PR(v) refers to the rank of the Web page v, which is determined by the PR values of all the Web pages including the hyperlink pointing to it. For example, the Web page v is linked by 10 Web pages, then the ranks of these 10 Web pages determine the rank of the Web page v; outlink(u) refers to the number of all the hyperlinks in the Web page u (which necessarily include the hyperlink to v); n is the number of all Web pages; and ε is an adjusting parameter representing the influence degree of the hyperlink linked to the Web page v in the page of equation (2) on the computing result. The greater the value of ε is, the more a single hyperlink linked to the Web page contributes to the PR value of v. The value of ε is typically between 0-1, and it is preferable to be 0.1-0.2 which is an optimum value according to the experimental evaluation.
Next, at step 106, by combining the hyperlink (hypertext) similarity S and URL (Web page) rank PR(v), a comprehensive evaluation can be conducted to obtain the following result.
R(d)=PR(d){1+δ·S(d,Q)} (3)
where, R(d) is the value of a comprehensive evaluation result combining the equations 1 and 2, which computes dynamically the rank of each Web page with respect to a certain keyword according to the fixed rank value PR(d) of each Web page d and the keyword Q. Where, S represents the similarity of hypertext di included in the Web page d with respect to the keyword Q; the parameter δ is used to adjust the weight of the equation 1 and the equation 2 in the equation 3; and δ represents whether the influence of the hyperlink analysis in equation 3 on the result of the equation is greater or the influence of the result of text correlation analysis on the result of the equation is greater, the value of which is between 0-1. The greater δ is, the greater the influence of text correlation on the result of the equation is, and δ is preferably to be 0.4.
Thereby, ranking is conducted according to the comprehensive evaluation result of equation 3, for example, ranking is conducted according to the size of the value of R(d), where the value of R(d) of the Web address most correlated with the keyword to be queried is the greatest. Then at step 107, the ranked result is returned to the user in the form of a hypertext list, for example. In the hypertext list, the known HTML representing forms are usually used: each hypertext corresponds to its URL address, which can link to the Web page pointed to by it.
Optionally, at a hyperlink ranking step, the rank of a hyperlink can be determined according to the number that the hyperlink is pointed to. The hyperlink that is pointed to most including the information matched most with the keyword to be queried, and correlates most with the keyword to be queried, so the hyperlink is ranked first.
(Shang Hai Jiao Tong Da Xue)”, and its PR value is 3.455E-7. All the hyperlink texts linking to the Web page are shown in the “anchor text” field, i.e., include “
(Jiao Da)”, “
(Shang Hai Jiao Da)”, “
(Jiao Tong Da Xue)”, “
(Shang Hai Jiao Tong Da Xue)”, “
(Shang Jiao Da)” and “
(Shang Hai Jiao Da)”, while the index values in the URL DB, of the Web pages including the hyperlinks of the Web page, corresponding to the above hyperlink texts respectively, are shown in the “URL index” field, i.e. are correspondingly “5894”, “23658”, “584521”, “296”, “99” and “3257”, respectively.
Referring to (Shang Hai Jiao Tong Da Xue)”. As described above, preferably, by using the index data downloaded from the Internet and established by a data indexing module 3, the searching means 502 can make the search faster. The hyperlink extracting means 503 extracts URL from all the hypertexts matched with the keyword, which can be implemented through the URL DB shown in
While the best modes for carrying out the invention has been described above, the present invention is not limited to the above embodiment, and can have various changes and modifications without departing from the scope of the present invention.
Variations described for the present invention can be realized in any combination desirable for each particular application. Thus particular limitations, and/or embodiment enhancements described herein, which may have particular advantages to a particular application need not be used for all applications. Also, not all limitations need be implemented in methods, systems and/or apparatus including one or more concepts of the present invention. The methods of the present invention include signal methods, wherein at least one signal is received and/or transmitted in performing one or more steps. The term signal includes transmissions on the Internet, etc.
The present invention can be realized in hardware, software, or a combination of hardware and software. A visualization tool according to the present invention can be realized in a centralized fashion in one computer system, or in a distributed fashion where different elements are spread across several interconnected computer systems. Any kind of computer system—or other apparatus adapted for carrying out the methods and/or functions described herein—is suitable. A typical combination of hardware and software could be a general purpose computer system with a computer program that, when being loaded and executed, controls the computer system such that it carries out the methods described herein. The present invention can also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which—when loaded in a computer system—is able to carry out these methods.
Computer program means or computer program in the present context include any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after conversion to another language, code or notation, and/or reproduction in a different material form.
Thus the invention includes an article of manufacture which comprises a computer usable medium having computer readable program code means embodied therein for causing a function described above. The computer readable program code means in the article of manufacture comprises computer readable program code means for causing a computer to effect the steps of a method of this invention. Similarly, the present invention may be implemented as a computer program product comprising a computer usable medium having computer readable program code means embodied therein for causing a function described above. The computer readable program code means in the computer program product comprising computer readable program code means for causing a computer to affect one or more functions of this invention. Furthermore, the present invention may be implemented as a program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for causing one or more functions of this invention.
It is noted that the foregoing has outlined some of the more pertinent objects and embodiments of the present invention. This invention may be used for many applications. Thus, although the description is made for particular arrangements and methods, the intent and concept of the invention is suitable and applicable to other arrangements and applications. It will be clear to those skilled in the art that modifications to the disclosed embodiments can be effected without departing from the spirit and scope of the invention. The described embodiments ought to be construed to be merely illustrative of some of the more prominent features and applications of the invention. Other beneficial results can be realized by applying the disclosed invention in a different manner or modifying the invention in ways known to those familiar with the art.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2004 1 0055624 | Jul 2004 | CN | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5999929 | Goodman | Dec 1999 | A |
| 6112203 | Bharat et al. | Aug 2000 | A |
| 6285999 | Page | Sep 2001 | B1 |
| 6691108 | Li | Feb 2004 | B2 |
| 20020156763 | Marchisio | Oct 2002 | A1 |
| 20030061214 | Alpha | Mar 2003 | A1 |
| 20030225763 | Guilak et al. | Dec 2003 | A1 |
| 20040044669 | Brown et al. | Mar 2004 | A1 |
| 20040054670 | Noff et al. | Mar 2004 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20060059132 A1 | Mar 2006 | US |
