Patent Application
20120303352
Embodiments of the present disclosure relate generally to methods, apparatus and computer program products for assessing a translation and, more particularly, to methods, apparatus and computer program products for assessing a translation following performance of the translation so as to identify, for example, one or more segments of a source language document that may be problematic for translators.
Global organizations, among many others, depend on document translations. Translation in industrial sectors such as utilities, manufacturing, and transportation require mastery of various technical disciplines, and translation errors or ambiguities can lead to financial and other adverse consequences. Some publication policies prescribe best practices for translating technical documentation into the languages of the receiving nations. These best practices usually permit authors or other document providers to exert control over the translation in a manner that balances cost with translation quality. However, this practice offers little control to an organization that produces line-of-business documents in only one language, especially when that organization's business model depends on foreign customers to translate received documents independently. According to this practice, source language documents are translated into target language documents by parties other than the owner of the source language document even though the owner of the source language document retains a proprietary interest in the quality of the translation notwithstanding the limited knowledge by the owner of the source language document of the target language.
In the absence of control over the translation itself, it could be beneficial for the owner of a source language document to draft the document so as to be more readily translatable. Translatability of a document denotes those properties of a source language document that increase the potential for successful translation of the source language document. Translation quality also depends on translatability and several different techniques have been developed for assessing translation quality, typically in the context of the prediction of translation costs in advance of the actual translation. For example, round-trip translation may be applied casually to machine-translation (MT) systems. In round-trip translation, source language (SL) input is translated into target language (TL) output by an MT system. This output is then re-translated from the TL back into the initial SL, and the final translation product is then compared to the original input to assess the translation quality of the MT system. Human judgment may determine when round-trip translation inputs and outputs are semantically equivalent or divergent. Although once thought to be an indicator of translation quality, especially when evaluators lack TL knowledge, round-trip translation quality assessment is now considered less helpful since round-trip translation fails to differentiate the distinct SL-TL and TL-SL contributions to the final translation product.
Regarding the relationship between translatability and translation quality, the relationship or correlation is suggested by the dependency between translatability assessment and post-editing costs. In this regard, translatability assessment may be used to predict translation costs. Typically, when translatability scores match translation capabilities, pre- and post-editing cost estimates are minimal. Otherwise, more time and effort are deemed necessary for an acceptable translation product. In either case, translation quality is predicted as a function of SL translatability and translation cost. Understanding of this relationship is useful when deciding how to effect a translation and which technologies to apply when human translation is prohibitively expensive or otherwise infeasible.
Some study has been undertaken to understand the formal parameters of translatability, that is, those properties of SL input that increase the potential for successful translation. In this regard, it has been suggested that authoring or pre-processing SL input with a controlled language (CL) enhances translatability. In this regard, translatability assessment typically identifies SL properties that act as impediments to translation. Usually these properties are aspects of SL non-compliance with CL specifications. Typically, non-compliance implicates lexical and grammatical restrictions that neutralize marked features of the SL from which the CL is adapted. In this way, the approach first assesses SL inputs with respect to an idealized, unmarked CL, which figures as a proxy for the actual TL. These studies eventually led to translatability assessment independent of the TL involved. Other studies employ machine learning to assess the translatability of SL inputs and reformulate them as necessary to enhance translatability. In general, the objective of these forms of translatability assessment is to predict the time and cost required for translation.
As such, translatability assessment techniques have been generally utilized prior to translation so as to determine, for example, the manner in which to execute a translation task. As such, the translatability assessment techniques described above may facilitate a determination as to how to effect a translation and which technologies to apply in an instance in which human translation is prohibitively expensive or otherwise unfeasible. However, translatability assessment techniques have not been widely utilized for purposes other than for pre-translation guidance in order to, for example, predict translation costs.
Methods, apparatus and computer program products are provided in accordance with embodiments of the present disclosure in order to assess a translation following performance of the translation. The methods, apparatus and computer program products of one embodiment may determine input segments of a source language document that may prove to be problematic from a translatability standpoint. As such, methods, apparatus and computer program products of the present disclosure may provide feedback to the author or owner of the source language document that may influence the generation of subsequent source language documents so as to have improved translatability.
In one embodiment, a method of assessing a translation is provided that includes aligning, with a processor, input segments of a source language document with corresponding output segments of a target language document. For each input segment, the method identifies variations between the output segments corresponding to a respective input segment. In this regard, the identification of the variations includes the identification of a reference translation and one or more output variants for the respective input segment. For example, the reference translation may be the output segment that most frequently corresponds to the respective input segment. The method of this embodiment also determines the one or more input segments having corresponding output variants that fail to satisfy a control limit for translation variation.
The method of one embodiment may also provide feedback regarding the one or more input segments having corresponding output variants that fail to satisfy a control limit for translation variation. As such, the recipient of the feedback, such as the author or owner of the source language document, can take the feedback into account during the production of other source language documents to improve the translatability of those other source language documents. In one embodiment, the determination of the one or more input segments having corresponding output variants that fail to satisfy the control limit for translation variation may include the determination of a measurement of similarity between each output variant and the reference translation. The measurement of similarity may, in turn, be determined by determining a longest common subsequence between each output variant and the reference translation. Further, the measurement of similarity may be determined by determining a similarity metric based upon recall and precision of the longest common subsequence between each output variant and the reference translation. In one embodiment, the control limit is based upon the similarity metric.
In one embodiment, a computing device for assessing a translation is provided that includes a processor configured to align input segments of a source language document with corresponding output segments of a target language document. For each input segment, the processor is configured to identify variations between the output segments corresponding to a respective input segment. In this regard, the identification of the variations includes the identification of a reference translation and one or more output variants for the respective input segment. For example, the reference translation may be the output segment that most frequently corresponds to the respective input segment. The processor of this embodiment is also configured to determine the one or more input segments having corresponding output variants that fail to satisfy a control limit for translation variation.
The processor of one embodiment may also be configured to provide feedback regarding the one or more input segments having corresponding output variants that fail to satisfy a control limit for translation variation. As such, the recipient of the feedback, such as the author or owner of the source language document, can take the feedback into account during the production of other source language documents to improve the translatability of those other source language documents. In one embodiment, the determination of the one or more input segments having corresponding output variants that fail to satisfy the control limit for translation variation may include the processor's determination of a measurement of similarity between each output variant and the reference translation. The measurement of similarity may, in turn, be determined by the processor determining a longest common subsequence between each output variant and the reference translation. Further, the measurement of similarity may be determined by the processor's determining a similarity metric based upon recall and precision of the longest common subsequence between each output variant and the reference translation. In one embodiment, the control limit is based upon the similarity metric.
In one embodiment, a computer program product for assessing a translation is provided that includes at least one computer-readable storage medium having computer-executable program code portions stored therein. The computer-executable program code portions include program code instructions for aligning input segments of a source language document with corresponding output segments of a target language document. For each input segment, the computer-executable program code portions include program code instructions for identifying variations between the output segments corresponding to a respective input segment. In this regard, the identification of the variations includes the identification of a reference translation and one or more output variants for the respective input segment. For example, the reference translation may be the output segment that most frequently corresponds to the respective input segment. The computer-executable program code portions of this embodiment also include program code instructions for determining the one or more input segments having corresponding output variants that fail to satisfy a control limit for translation variation.
The computer-executable program code portions of one embodiment also include program code instructions for providing feedback regarding the one or more input segments having corresponding output variants that fail to satisfy a control limit for translation variation. As such, the recipient of the feedback, such as the author or owner of the source language document, can take the feedback into account during the production of other source language documents to improve the translatability of those other source language documents. In one embodiment, the program code instructions for determining the one or more input segments having corresponding output variants that fail to satisfy the control limit for translation variation may include program code instructions for determining a measurement of similarity between each output variant and the reference translation. The measurement of similarity may, in turn, be determined by program code instructions for determining a longest common subsequence between each output variant and the reference translation. Further, the measurement of similarity may be determined by program code instructions for determining a similarity metric based upon recall and precision of the longest common subsequence between each output variant and the reference translation. In one embodiment, the control limit is based upon the similarity metric.
In accordance with embodiments of the present disclosure, a method, apparatus and computer program product are provided in order to assess a translation and to identify input segments of a source language document that may be problematic from a translatability standpoint. As such, authors, owners or other providers of source language documents may take into account the input segments that have poor translatability in order to subsequently produce other source language documents that are more readily translatable. However, the features, functions and advantages that have been discussed may be achieved independently and the various embodiments of the present disclosure may be combined in the other embodiments, further details of which may be seen with reference to the detailed description and drawings.
Having thus described embodiments of the present disclosure in general terms, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein:
Embodiments of the present disclosure now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments are shown. Indeed, these embodiments may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like numbers refer to like elements throughout.
A method, apparatus and computer program product are provided according to one embodiment of the present disclosure for assessing a translation of a source language document following the generation or production of the translation. Based upon the assessment of the translation, feedback may be provided to the author or owner of the source language document to indicate input segments of the source language document that are problematic from a translatability standpoint, such as those input segments that lend themselves to a plurality of different translations. Based upon this feedback, the source language document may be revised or other source language documents may be subsequently created that take into account the results of the translatability assessment so as to create source language documents that are more consistently and accurately translated.
While the methods, apparatus and computer program products of embodiments of the present disclosure may be utilized in a variety of situations, the methods, apparatus and computer program products of one embodiment are useful in an instance in which the author or owner of the source language document does not perform or otherwise have control over the translation of the source language document. For example, the author or owner of the source language document may create and provide a monolingual document to another party, such as a customer, a partner or the like. The other party may then translate the source language document independent of any input or control by the author or owner of the source language document. As a result of its authorship or ownership of the source language document, however, the author or owner of the source language document still has an interest in the quality of the translation to ensure that the content of the source language document is accurately and consistently reproduced in the target language. By acting upon feedback provided in accordance with embodiments of the present disclosure, the author or owner of a source language document may work to improve the translatability of subsequent source language documents, thereby reducing the risks associated with poor translations of the source language documents.
The methods, apparatus and computer program products of embodiments of the present disclosure generally identify input elements of the source language document that have poor translatability based upon the analysis of textual properties of a parallel pair of source language and target language documents. As shown in operation 10 of
In the alignment process, an input segment of the source language document is aligned or matched with an output segment of the target language document that represents the same sentence, phrase or the like as does the input segment. Various alignment techniques may be utilized, such as that described at http://champollion.sourceforge.net. For example, an alignment technique may accept a parallel document pair, such as a source language document and a corresponding target language document, as an input and produce a bisegmentation relation that identifies mutual translation correspondences between segments of each document, such as between an input segment of the source language document and a corresponding output segment of the target language document. As noted, the granularity of the bisegmentation relations may vary from words, collocations, phrases, sentences, or other textual units. In one embodiment, for example, an alignment technique may utilize a length-based probabilistic algorithm supplemented with a domain-specific source language-target language lexical resource to produce sentence alignments. See, for example, Peng Li, et al., “Fast-Champollion: A Fast and Robust Sentence Alignment Algorithm”, Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010).
For each input segment, the method may identify variations between the output segments that correspond to the respective input segment as shown in block 12 of
One of the input segments of the source language document, that is, “Present ADI pitch attitude is within the red RA regions” has only a single corresponding output segment and therefore has no translation variations and, as a result, superior translatability. However, the other three input segments of the English language source document have two or more corresponding output segments in the Mandarin language target document. As such, these input segments that have multiple corresponding output segments have poorer translatability. Generally, however, some variation in the output segments of a target language document may be tolerable, while more substantial translation variations may be considered intolerable and indicative of poor translatability of the corresponding input segments of the source language document.
The relationship between input segments of a source language document and the corresponding output segments of a target language document that is reflected in Table 1 need not be presented to a user, but the underlying information regarding the corresponding output segments of the target language document and the frequency with which each of the corresponding output segments appears within the target language document may be utilized when assessing the translatability of the source language document. In order to assess the translation variations, the output segments of a target language document are reviewed to identify instances in which different output segments correlate to the same input segment. In this regard, those input segments of the source language document that have a single corresponding output segment are identified by the method to have no output variants. However, for each input segment of the source language document that has two or more corresponding output segments in the target language document, the method identifies a reference translation and one or more output variants. See operation 12 of ) that occurs most frequently, i.e., five times, and is identified as the reference translation, while the other corresponding output segment (
) occurs less frequently, i.e., one time, and is identified as an output variant. As another example, the “Traffic aircraft is providing altitude information” input segment has a corresponding output segment (
) that occurs most frequently, i.e., six times, and is identified as the reference translation, while the two other corresponding output segments occur less frequently, i.e., four and three times, and are identified as output variants.
Thereafter, the method may determine the one or more input segments of the source language document that have corresponding output variants that fail to satisfy a control limit for translation variation, as shown in block 14 of
In this regard, each output segment that corresponds to a respective input segment may be construed as a string of words and the similarity between the output segments varies directly based upon the length of the subsequence commonality between the strings of words. In this embodiment, output segments that have longer subsequence commonality will be considered more similar than output variants that have shorter subsequence commonality. For example, a common subsequence of reference translation X is any output variant Y that exhibits the word sequence of X with zero or more elements omitted. Expressed in terms of abstract sequences X, Y and Z, Z is regarded as a common subsequence of X and Y if Z is a subsequence of X and Y. For example, if X equals {A, B, C, B, D, A} and Y equals {B, D, C, A, B}, the sequence {B, C, A} is the common subsequence of X and Y. See, for example, Thomas H. Cormen, et al., “Introduction to Algorithms,” Third Edition, MIT Press (2009). By way of example and without limitation or intent for aircraft or functional use, Table 2 represents the output segments (TL output) of a Mandarin language target document that correspond to an input segment of “Traffic aircraft is providing altitude information” from an English language source document.
As shown, the output segments may be tokenized in order to break the output segments into a plurality of words or other lexical units. By way of example, the first output segment may serve as the reference translation with the second and third output segments being output variants of the reference translation. While the second and third output segments share a common subsequence with the first output segment for the words in sequential positions 0 and 1, the method may determine the longest common subsequence (LCS) for each output variant relative to the reference translation. In this regard, the longest common subsequence for the second output variant relative to the reference translation is the words in sequential positions 0, 1, 3 and 4. Similarly, the longest common subsequence for the third output variant relative to the reference translation involves the words in sequential positions 0, 1, 4 and 5. In general, for any two output segments X and Y with X being the reference translation, the longest common subsequence of X and Y denoted LCS (X, Y) is the maximum count of words that Y shares in common with X and which occur in Y in the same sequential order, but not necessarily consecutively, as they appear in X.
In one embodiment, the determination of the measurement of similarity may include the determination of a similarity metric based upon the recall and precision, such as the weighted harmonic mean of the recall and precision, of the longest common subsequence (LCS) between each output variant and the reference translation. In this embodiment, the control limit may, in turn, be based upon the similarity metric. As described by Chin-Yew Lin, et al., “Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics”, Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL 2004), for a reference translation X of length m and an output variant Y of length n, the weighted harmonic mean of the recall Rlcs for the LCS may be defined as:
Additionally, the weighted harmonic mean of the precision Plcs for the LCS may be defined as:
Additionally, a weighting value β may be defined as:
Although a similarity metric may be determined based upon the recall and precision of the longest common subsequence in various manners, the method of one embodiment may determine a similarity metric Flcs (X, Y) as follows:
By way of example and with reference to the reference translation, i.e., TL output (1), and the output variants, i.e., TL outputs (2) and (3), of Table 2, the similarity metric Flcs (X, Y) is 0.8 for TL output (2) relative to the reference translation and 0.73 for TL output (3) relative to the reference translation in an instance in which the weighting value β equals one. Thus, the similarity metric of this embodiment takes into consideration word count variations between the output variants and the reference translation and confirms human intuition that, from among the output variants with the same LCS, the output variant having the same number of words as the reference translation has less variance from the reference translation than does an output variant that has a different number of words than the reference translation. As such, the longest common in-sequence n-gram information factored into the foregoing equation for the similarity metric Flcs (X, Y) provides a target language output comparison metric having sensitivity for the empirical facts of linear precedence.
As noted above, the method may then utilize the similarity metric in order to define the control limit that establishes whether a translation variation is tolerable or intolerable. In one embodiment, the similarity measures for the plurality of output segments are presumed to be a normally-distributed random variable that are aggregated so as to determine a control limit for translation variation between a source language document and the target language document. Thus, output segments of the target language document that satisfy the control limit may be considered to be tolerable or acceptable even if those output segments vary somewhat from the reference translation, while output segments that fail to satisfy the control limit may be considered intolerable as a result of their excessive variation relative to the reference translation.
While the control limit may be based upon the similarity metric in a variety of different manners, one example of the relationship between the similarity metric and the control limit is provided herein for purposes of example, but not of limitation. In this example, vi is an output variant that occurs in a parallel document pair, that is, a pair consisting of a source language document and a corresponding target language document, with a total of m output variants, excluding those output segments that serve as reference translations. Additionally, xi is the LCS-based similarity measure obtained from Flcs(vi, ri) in an instance in which ri is the reference translation for vi. In this example, the method may determine the arithmetic mean of the sum of all the differences between the similarity estimates for each xi and its predecessor xi−1 according to the following equation:
In this regard, the foregoing equation determines the moving range (MR) of translation variation across the parallel document pair. This moving range value quantifies the average translation variation. The control limit for translation variation may, in turn, be based upon the moving range MR and, in one embodiment, the control limit may be determined as the product of the moving range MR and the multiplier 2.66. In this regard, the multiplier 2.66 may be obtained by dividing 3 by the anti-biasing constant for n=2 as described, for example, in Douglas Montgomery, “Introduction to Statistical Quality Control”, John Wiley & Sons (2005).
Once a control limit has been established for translation variation, such as 2.66 MR, the method of one embodiment may compare the similarity measure xi for each output variant vi with the control limit in order to determine the output variants, if any, that exceed the control limit and which will, therefore, be considered to exceed the tolerable levels of translation variation established by the control limit. In an instance in which one or more input segments of a source language document have output segment(s) that exhibit an intolerable translation variation, the method may provide feedback to the author or owner of the source language document as shown in operation 16 of
By way of a further example,
As described above, a control limit for translation variation may then be determined and the output variants may be compared to the control limit to determine if the output variants vary excessively. See operations 32 and 34 of
The methods described above and illustrated, for example, in
As noted above, the processor 40 may operate under control of a computer program product. In this regard, the computer program product for performing the methods of embodiments of the present disclosure includes a computer-readable storage medium, such as a non-volatile, non-transitory storage medium, and computer-readable program code portions, such as a series of computer instructions, embodied in the computer-readable storage medium.
In this regard,
Accordingly, blocks or steps of the flowchart support combinations of means for performing the specified functions and combinations of steps for performing the specified functions. It will also be understood that each block or step of the flowchart, and combinations of blocks or steps in the flowchart, can be implemented by special purpose hardware-based computer systems which perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.
Many modifications and other embodiments of the present disclosure set forth herein will come to mind to one skilled in the art to which these embodiments pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the present disclosure is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
