Patent Application
20100057438
This application is based upon and claims the benefit of priority from prior Chinese Patent Application No. 200810214667.6, filed Sep. 1, 2008, the entire contents of which are incorporated herein by reference.
1. Field of the Invention
The present invention relates to information processing technology, and particularly to a phrase-based statistics machine translation method and system.
2. Description of the Related Art
Machine translation technologies are mainly categorized as rule-based machine translation technologies and corpus-based machine translation technologies.
In the corpus-based machine translation technologies, the main translation resources come from a corpus repository. The corpus-based machine translation technologies are further categorized as example-based machine translation technologies and statistics-based machine translation technologies. In the statistics-based machine translation technologies, the phrase-based statistics machine translation (SMT) method is one of the main automatic machine translation methods.
The basic translation unit of the phrase-based statistics machine translation method is phrase, and the translation knowledge used therein consists of phrase table and language model obtained from parallel bilingual corpora in a corpus repository. The phrase table consists of bilingual phrase pairs in the parallel bilingual corpora. Herein, the phrase is defined as several continuous words.
The process of conventional phrase-based statistics machine translation mainly comprises the following steps: first, a phrase table is searched by using exactly matching method, so as to find all completely matched bilingual phrase pairs corresponding to an input sentence; then, based on the bilingual phrase pairs and a language model, all possible combinations of translation fragments in a target language are found for the input sentence, and the one having the highest score is selected from the all possible combinations by using a statistics method, as the correct target language translation of the input sentence.
The input unit 11 is an interface of the system 10 with the outside, and the system 10 obtains an input sentence to be translated from the outside through the input unit 11.
The searching unit 12 performs phrase exactly matching. Specifically, it searches a phrase table stored in the phrase table storing unit 15 for all completely matched bilingual phrase pairs corresponding to the input sentence by using exactly matching method.
Further, the translation generating unit 13 generates the correct target language translation of the input sentence. Specifically, it finds all possible translations in a target language for the input sentence based on the bilingual phrase pairs searched by the searching unit 12 and a language model stored in the language model storing unit 16, and selects the one having the highest score from the all possible translations by using a statistics model as the correct target language translation of the input sentence.
The target language translation generated by the translation generating unit 13 is output through the output unit 14.
(This means “I found the end of her story very exciting” in English.), the system of
<->I found, (P2)
<->her, (P3)
<->the end of the story, and (P4)
<->very exciting. Moreover, based on the four bilingual phrase pairs, the system obtains the final translation “I found her the end of the story very exciting” by using the statistics model.
It can be seen from the above that in the conventional phrase-based statistics machine translation system, with respect to an input sentence to be translated, the exactly matching method is used to search a phrase table for completely matched bilingual phrase pairs to obtain the translation of the input sentence. The condition of the exactly matching method is that two matched phrases must be completely identical.
However, the size of the parallel bilingual corpus in a pre-constructed corpus repository is limited generally, and may not cover long phrases. Thus for long phrases in the input sentence to be translated, it is very difficult to find out completely matched bilingual phrase pairs in the phrase table by using the exactly matching method. Therefore, in the translation process, a long phrase can only be split into several short phrases for matching one by one.
However, because a long phrase contains more context information than a short phrase, the quality of the translation in the target language for an input sentence generated based on the matching of short phrases is usually lower than that generated based on the matching of long phrases.
According to one aspect of the present invention, there is provided a phrase-based statistics machine translation method, comprising: for phrases in an input sentence, performing fuzzy matching in a pre-constructed phrase table.
According to another aspect of the present invention, there is provided a phrase-based statistics machine translation system, comprising a phrase fuzzy matching unit configured to, for phrases in an input sentence, performing fuzzy matching in a pre-constructed phrase table.
Next, a detailed description of each embodiment of the present invention will be given with reference to the drawings.
As shown in
At step 310, phrase fuzzy matching is performed.
Specifically, at the step, a pre-constructed phrase table is searched for identical or the most similar bilingual phrase pair for each phrase in the input sentence by using a phrase fuzzy matching method, and the most similar bilingual phrase pair is modified, thus obtaining the correct translation of each phrase.
At step 315, a target language translation of the input sentence is generated.
Specifically, all possible translations in the target language for the input sentence are found based on the bilingual phrase pairs obtained at step 310 and a pre-constructed language model, and the one having the highest score is selected therefrom by using a statistics model, as the correct target language translation of the input sentence.
At step 320, the generated target language translation is output.
The process of the above step 310 will be described in detail below.
In the present embodiment, the process of phrase fuzzy matching is implemented according to the concept of Example-Based Machine Translation (EBMT). The main process of the EBMT method is as follows: first, an example sentence repository is searched for the example sentence similar to the input sentence; then, differences between the similar example sentence and the input sentence are recognized; and finally, the differences in the similar example sentence are eliminated based on a translation model, thus generating the translation of the input sentence. For the detailed information about the EBMT method, referring to Harold Somers, “Review Article: Example-based Machine Translation”, 1999, Machine Translation, 14(2): 113-157.
As shown in
For example, referring to (This means “I found.”),
(This means “the end of her story.”) and
(This means “very exciting.”), for the phrase
(This means “I found.”), a completed matched bilingual phrase pair “(P1)
<->I found” is found; for the phrase
(This means “the end of her story.”), the most similar bilingual phrase pair “(S3)
<->the end of the story” is found; and for the phrase
(This means “very exciting.”), a completed matched bilingual phrase pair “(P4)
<->very exciting” is found.
For a long phrase such as (This means “the end of her story.”) that has no completed matched bilingual phrase pair in the phrase table, the process of searching for the most similar bilingual phrase pair thereof is as follows: first, a plurality of similar candidate bilingual phrase pairs containing most identical words to those in the long phrase are found from the phrase table; and then, for each of the plurality of similar candidate bilingual phrase pairs, an editing distance between it and the long phrase is calculated, wherein the editing distance is the number of inserting, deleting and replacing operations required for transforming the source language phrase in the similar candidate bilingual phrase pair to the long phrase; and finally, the similar candidate bilingual phrase pairs having the shortest editing distance from the long phrase are selected as the most similar bilingual phrase pairs of the long phrase.
For example, referring to (This means “the end of her story.”), a plurality of similar candidate bilingual phrase pairs “(S1)
<->plot of the story”, “(S2)
<->the end of the film” and “(S3)
<->the end of the story” are found in the phrase table.
In this case, for each of the candidate bilingual phrase pairs (S1), (S2) and (S3), the editing distance between it and the long phrase is calculated, thus obtaining: the editing distance between (S1) and the long phrase is 2, i.e., such two operations as the insertion of
(This means “her that.”) and the replacement of
(This means “plot.”) with
(This means “end.”) need to be executed in the source language phrase of (S1); the editing distance between (S2) and the long phrase is also 2, i.e., such two operations as the insertion of
(This means “her that.”) and the replacement of
(This means “film.”) with
(This means “story.”) need to be executed in the source language phrase of (S2); and the editing distance between (S3) and the long phrase is 1, i.e., only such an operation as the insertion of
needs to be executed in the source language phrase of (S3).
Thus, the bilingual phrase pair “(S3) <->the end of the story” having the shortest editing distance from the long phrase
(This means “the end of her story.”) can be obtained as the most similar bilingual phrase pair of the long phrase.
At step 415, for each of the long phrases in the input sentence, for which no completely matched bilingual phrase pair is found but the most similar bilingual phrase pair is found, the differences between the most similar bilingual phrase pair found therefor and the long phrase are recognized. That is, different words between the source language phrase in the most similar bilingual phrase pair and the long phrase are recognized.
Specifically, at this step, one of the following methods can be used according to specific circumstances to determine whether the words in the source language phrase in the most similar bilingual phrase pair are identical to those in the long phrase:
1) The source language phrase in the most similar bilingual phrase pair and the long phrase are compared with each other on words directly to see whether the words are consistent.
2) If the long phrase is in English, the source language phrase in the most similar bilingual phrase pair and the long phrase are compared with each other on the base form of words to see whether the base form of the words are consistent.
3) By using a synonym dictionary, it is checked whether the different words between the source language phrase in the most similar bilingual phrase pair and the long phrase express a same meaning.
For example, if the most similar bilingual phrase pair found for the long phrase (This means “the end of her story.”) in the example of
<->end of the novel”, then although
therein is a different word to the
(This means “story.”) in the long phrase literally, if it is defined in the synonym dictionary that
(This means “novel.”) and
(This means “story.”) belong to synonyms, then they express a same meaning, thus
(This means “novel.”) and
(This means “story.”) are not considered to be different parts herein.
4) By using a translation dictionary, it is checked whether the different words between the source language phrase in the most similar bilingual phrase pair and the long phrase express a same meaning.
Likewise, if the most similar bilingual phrase pair found for the long phrase (This means “the end of her story.”) in the example of
<->end of the novel”, then if it is found in the translation dictionary that
(This means “story.”) can be translated into “story” or “novel”, and
(This means “novel.”) can be translated into “novel”, then
(This means “novel.”) and
(This means “story.”) can be considered to belong to words having a same meaning but not considered to be different parts.
At step 420, for each of the long phrases in the input sentence, for which no completely matched bilingual phrase pair is found but the most similar bilingual phrase pair is found, the differences in the most similar bilingual phrase pair to the long phrase are modified to obtain the target language translation of the long phrase.
That is, the different words in the most similar bilingual phrase pair to those of the long phrase are modified. Specifically, the words having different meanings in the source language phrase in the most similar bilingual phrase pair to those of the long phrase are modified first, so that the modified source language phrase is consistent with the long phrase, then the corresponding words in the target language phrase in the most similar bilingual phrase pair are modified, thus obtaining the target language translation of the long phrase.
For example, for the most similar bilingual phrase pair “(S3)<->the end of the story” found for the long phrase
(This means “the end of her story.”) in the example of
(This means “her.”), firstly the word
(This means “her.”) is inserted in front of the word
(This means “that.”) in the source language phrase of (S3) so that the amended source language phrase is consistent with the long phrase, then the dictionary is looked up to obtain “
->her”, and based on this, the corresponding word in the target language phrase of (S3) is modified according to the amended source language phrase, i.e., the second “the” in the target language phrase is replaced with “her”, thus a correct target language translation “the end of her story” of the long phrase is obtained.
Therefore, referring to (This means “I found the end of her story very exciting.”), based on the following bilingual phrase pairs obtained through phrase fuzzy matching: (P1)
<->I found, (P5)
<->the end of her story and (P4)
<->very exciting, the final target language translation “I found the end of her story very exciting” having the highest score for the input sentence can be obtained by using a statistics model.
The above is a detailed description of the phrase-based statistics machine translation method of the present embodiment. In the present embodiment, by performing fuzzy matching on phrases, high quality translations can be generated for long phrases in the input sentence, thus the translating of the input sentence can be implemented based on the long phrases, which can effectively increase the quality of the translation with respect to the translation systems based on phrase exactly matching. Further, it can be seen by comparing the translation obtained based on phrase exactly matching in the example of
In addition, it should be noted that, although in the process of
Under the same inventive concept, the present invention provides a phrase-based statistics machine translation system, which will be described below in conjunction with the drawings.
The input unit 61 is an interface of the system 60 with the outside, and the system 60 obtains an input sentence to be translated from the outside through the input unit 61.
The phrase fuzzy matching unit 62 performs fuzzy matching for the phrases in the input sentence in a pre-constructed phrase table stored in the phrase table storing unit 65, so as to find the target language translations of the phrases.
The translation generating unit 63 finds all possible translations in a target language for the input sentence based on the matching result of the phrase fuzzy matching unit 62 and a pre-constructed language model stored in the language model storing unit 66, and selects the one having the highest score by using a statistics model as the correct target language translation of the input sentence.
Further, the target language translation generated by the translation generating unit 63 is output through the output unit 64.
The phrase fuzzy matching unit 62 will be described in detail below.
Specifically, as shown in
The bilingual phrase searching unit 622 searches the phrase table stored in the phrase table storing unit 65 for the identical or the most similar bilingual phrase pair, according to the input sentence.
Specifically, for each of long phrases for which no identical bilingual phrase pair is found, the bilingual phrase searching unit 622 finds a plurality of similar candidate bilingual phrase pairs containing most identical words to those in the long phrase from the phrase table for the long phrase; for each of the plurality of similar candidate bilingual phrase pairs, calculates an editing distance between it and the long phrase, wherein the editing distance is the number of inserting, deleting and replacing operations required for transforming the source language phrase in the similar candidate bilingual phrase pair to the long phrase; and selects the similar candidate bilingual phrase pair having the shortest editing distance from the long phrase as the most similar bilingual phrase pair of the long phrase.
The difference recognizing unit 623, for each long phrase for which the most similar bilingual phrase pair is found among the plurality of long phrases, recognizes the differences between the most similar bilingual phrase pair and the long phrase. That is, the words having different meanings between the source language phrase in the most similar bilingual phrase pair and the long phrase are recognized.
Specifically, for each long phrase for which the most similar bilingual phrase pair is found among the plurality of long phrases, the difference recognizing unit 623 recognizes the words having different meanings between the source language phrase in the most similar bilingual phrase pair and the long phrase directly or by using a synonym dictionary/translation dictionary.
The modifying unit 624, for each long phrase for which the most similar bilingual phrase pair is found among the plurality of long phrases, modifies the differences in the most similar bilingual phrase pair to the long phrase, so as to obtain the target language translation of the long phrase.
Specifically, for each long phrase for which the most similar bilingual phrase pair is found among the plurality of long phrases, the modifying unit 624 modifies the words having different meanings in the source language phrase in the most similar bilingual phrase pair to those of the long phrase, so that the modified source language phrase is consistent with the long phrase, and then modifies the corresponding words in the target language phrase in the most similar bilingual phrase pair according to the modified source language phrase.
In addition, it should be noted that, although the phrase fuzzy matching unit 62 is implemented based on the example-based machine translation method in the present embodiment, it is not limited to this, and in other embodiments, the phrase fuzzy matching unit can be implemented by using any presently known or future knowable translation concept.
The above is a detailed description of the phrase-based statistics machine translation system of the present embodiment.
The phrase-based statistics machine translation system 60 and its components can be implemented with specifically designed circuits or chips or be implemented by a computer (processor) executing corresponding programs.
While the phrase-based statistics machine translation method and system of the present invention have been described in detail with some exemplary embodiments, these embodiments are not exhaustive, and those skilled in the art may make various variations and modifications within the spirit and scope of the present invention. Therefore, the present invention is not limited to these embodiments; rather, the scope of the present invention is solely defined by the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 200810214667.6 | Sep 2008 | CN | national |
