1. Field
Embodiments relate to creating parallel text corpora which may be used, for example, in machine translation, machine learning, search technology, system of understanding of natural languages in AI. In particular, embodiments relate to automatic creation of aligned parallel natural languages corpora and tagged parallel corpora. Such electronic content may be available, for example, on the Internet and in other electronic resources.
2. Background
There is existing technology that generates parallel text corpora. A parallel text corpus refers to texts consisting of two or more parts—a text in one language and its translation in another language that is a translation of the first text. Parallel texts corpus may contain texts in two or more languages. An aligned parallel text additionally comprises a mapping (correspondence) of a portion of the first text into a portion of its translation, where the portion may be a sentence, a paragraph, or another part of the texts. An example of aligned parallel text is a translation memory or other databases of translations which can be created, for example, by translation agencies or by individual human translators. Applications, such as machine translation, machine learning, search technologies, system of understanding of natural languages in AI, may employ connected parallel texts, and, more importantly, parallel texts which comprise logical relations between sentences, such as referential relations, anaphoras, connectors and the like. Aligned and tagged parallel texts are very useful for these applications as well.
Usual methods of aligning parallel texts are chiefly manual, or based on heuristics, for example, aligning formally by boundaries of sentences which are identified by punctuation marks. But, such methods may be not sufficiently precise, because text formatting may complicate assumptions about sentence boundaries and because there are cases when one sentence is translated into two or more sentences in another language. Additionally, it is desirable to obtain tagged parallel texts where grammatical, lexical, and even syntactical and semantic features, as well as syntactic relationships or/and semantic relationships, are identified, and where grammatical and lexical meanings, deep or surface slots may be determined and searchable. While US Application Publication Number US20060217963 A1 mentions the use of Interlingua representation in connection with translation memory, it does not provide an effective way to generate and compare such representations, which are described as tree structures.
The preferred embodiment preliminarily divides a text into fragments where a fragment may be a paragraph, a sentence or any other portion of a text; generates a hypotheses about correspondence between at least one fragment in one language and at least one fragment in another language; verifies hypotheses about correspondence between the fragments; and selecting at least one hypothesis.
There are various ways to verify the hypothesis about the correspondence between a fragment (e.g., a sentence) in one language and a fragment in another language. In one embodiment, a heuristic method using only two-language dictionaries may be used to estimate the degree of correspondence between the fragment in one language and the corresponding fragment in another language. In another embodiment, verifying the hypothesis about the correspondence between a fragment in one language and a fragment in another language includes performing a lexico-morphological analysis of each element of the fragment using linguistic descriptions; performing a syntactic analysis using linguistic descriptions to generate at least one syntactic structure for each sentence; building a semantic structure for each sentence; estimating the degree of correspondence between semantic structures of the sentence in one language and the corresponding sentence in another language.
If the degree of correspondence is sufficiently significant, the hypothesis may be considered confirmed, and the correspondence between the sentence in one language and the corresponding sentence in another language would be established. The system may associate and save the generated syntactic and semantic structures in connection with the corresponding sentence. Accordingly the sentences of the parallel texts and, if desired, individual words would be tagged with their linguistic descriptions and semantic structures generated during the analysis. As result, aligned tagged parallel text corpus is provided.
As noted, parallel text copra are useful in many applications. In machine translation, it may be effectively employed for resolving ambiguities and lexical choices in a sentence by means of comparison its semantic structure with semantic structures of corresponding sentences in various languages. Additionally, for each corpus of natural language texts that were analyzed, the system indexes lexical meanings and values of linguistic parameters of each syntactic structure and each semantic structure and its semantic parameters associated with sentences in the corpus. The tagged, indexed parallel text corpora can be searched more productively including using intelligent semantic searching. Thus, generally, the disclosed system and method employs a full-fledged automatic syntactic and semantic analysis when aligning and marking texts making it possible to index and store syntactic and semantic information about each sentence, as well as parsing results and lexical choices, including results obtained when resolving ambiguities.
The preferred embodiments are implemented as a computer system that includes hardware and software as discussed in more detail below. The computer system can be a server, a personal computer, a mobile device, a distributed architecture, or any other computing environment suitable for supporting the functionality discussed herein.
The preferred embodiments relate to techniques and systems for aligning parallel natural language text corpora.
At step 220, hypotheses about correspondence between the fragments in one language to the fragments in another language are generated. The hypotheses may be generated using information about the size of fragments, punctuation symbols, formatting, capital letters, headings, numbering of text parts and paragraphs, and other characteristics of the texts. The system may generate several hypotheses about the fragment correspondence, since there may be several alternative ways of dividing the parallel texts into pairs of matching fragments.
At step 230, the system verifies the hypotheses about correspondence between the fragments in one language and fragments in another language. There are alternative approaches to verifying the hypothesis about the correspondence between a fragment in one language and a fragment in another language. In one embodiment, a heuristic method, using only two-language dictionaries, estimates the degree of correspondence between the fragments. The two-language dictionary may be created from a usual dictionary as a normalized one-to-one word dictionary, which can include all lexical values, homonyms and parts of speech. It also can include all word forms; otherwise, each word in a sentence should be subjected to morphological analysis. As part of verifying a hypothesis about the correspondence of the two fragments, an estimation of correspondence value is calculated. This value may be based on the relationship of the percentage of words with a translation in the corresponding fragment found in the dictionary to the sizes of the fragments and, optionally, to other parameters.
Additionally, a presence of corresponding word combinations (331) in the two fragments may increase the confidence in the hypothesis, which may be reflected in the estimation value mentioned above. Some other heuristic factors, such as the presence of specific words, such as dates, numbers, abbreviations, names, e.g., family names and geographical names, and the like may be taking into consideration in determining the probability of the selection of a hypothesis. After all the estimations, including those discussed above, reflecting the segment correspondence had been taken into account, the degree of correspondence is calculated (341). In calculating the degree of correspondence value, the previously-discussed partial estimations are adjusted by using weight coefficients which may be selected on the basis of heuristics or training. If the accumulate estimation is sufficiently high (351), the hypothesis is confirmed, otherwise the system attempts to change (361) the boundaries of the fragment (for example, to unite two adjacent fragments) and then to attempt to verify the new hypothesis. There may be a limitation on the number of hypotheses related to a fragment that may be attempted. If the best hypothesis for the fragment still does not have a sufficiently high degree of correspondence, the fragment is marked by a color, so as to indicate that it should be correct manually. Referring to
Another embodiment involves analyzing exhaustive linguistic descriptions for verifying hypotheses 230 about the correspondence between a fragment in one language and a fragment in another language. The method of analyzing natural languages was disclosed in the U.S. patent application Ser. No. 11/548,214, filed on 10 Oct. 2006, now U.S. Pat. No. 8,078,450. It includes a lexico-morphological analysis, syntactic and semantic analysis and building a semantic structure for each sentence. The implementation of the method of analyzing natural languages allows retrieving and saving all lexico-morphological, syntactical and semantic information about a sentence, its parts, words and their relations in the sentence, meanings of words and meaning of the whole sentence, and even relations between sentences.
In one preferred embodiment the fragments are selected as sentences. As a result of the above processing, each sentence is associated with the generated syntactic and semantic structure. Preferably the structures are saved in connection with the sentences, so that the texts are annotated with the information and structures ascertained during the analysis Additionally, for each corpus of natural language texts that was analyzed, the system performs an indexing operation to index lexical meanings and values of linguistic parameters of each syntactic structure and each semantic structure and its semantic parameters associated with sentences in the corpus. The indexed text corpora are useful for information searching. The tagged indexed parallel text corpora are useful for intelligent semantic search.
As noted, the preferred system employs automatic syntactic and semantic analysis when aligning and marking texts. It indexes and stores syntactic and semantic information about each sentence, as well as parsing results and lexical choices, including results obtained when resolving ambiguities. The system analyzes sentences using linguistic descriptions of a given natural language to reflect the real complexities of the natural language, rather than simplified or artificial descriptions. A principle of integral and purpose-driven recognition, where hypotheses about the structure of a part of a sentence are verified within the hypotheses about the structure of the whole sentence, is implemented during the analysis stage. It avoids analyzing numerous parsing of anomalous variants.
The linguistic descriptions may include morphological descriptions 301, syntactic descriptions 302, lexical descriptions, 303 and semantic descriptions 304. In one implementation, a plurality of linguistic models and knowledge about natural languages may be arranged in a database and applied for analyzing each text sentence. Such a plurality of linguistic models may include morphology models, syntax models, grammar models and lexical-semantic models. In a particular implementation, integral models for describing the syntax and semantics of a language are used in order to recognize the meanings of the sentence, analyze complex language structures, and correctly convey information encoded in the sentence.
When analyzing the meaning of the sentence, syntactic analysis 320 realizes a two-step analysis algorithm (e.g., rough syntactic analysis and precise syntactic analysis) implemented to make use of linguistic models and knowledge at various levels, to calculate probability ratings and to generate the most probable syntactic structure, e.g., a best syntactic structure.
Accordingly, a rough syntactic analysis is performed on the sentence to generate a graph of generalized constituents for further syntactic analysis. All the possible surface syntactic models for each element of lexical-morphological structure are applied, and all the possible constituents are built and generalized to represent all the possible variants of parsing the sentence syntactically.
Following the rough syntactic analysis, a precise syntactic analysis is performed on the graph of generalized constituents to generate one or more syntactic trees to represent the source sentence. In one implementation, generating the syntactic tree comprises choosing between lexical options and choosing between relations from the graphs. Many prior and statistical ratings may be used during the process of choosing between lexical options, and in choosing between relations from the graph. The prior and statistical ratings may also be used for assessment of parts of the generated tree and for the whole tree. In one implementation, the one or more syntactic trees may be generated in order of decreasing assessment. Thus, the best syntactic tree may be generated first. Non-tree links are also checked and generated for each syntactic tree at this time. If the first generated syntactic tree fails, for example, because of impossibility to establish non-tree links, the second syntactic tree is taken as the best, etc.
This novel two-step syntactic analysis approach ensures that the meaning of the source sentence is accurately represented by the best syntactic structure chosen from the one or more syntactic trees. Advantageously, the two-step analysis approach follows a principle of integral and purpose-driven recognition, i.e., hypotheses about the structure of a part of a sentence are verified using all available linguistic descriptions within the hypotheses about the structure of the whole sentence. This approach avoids a need to analyze numerous parsing anomalies or variants known to be invalid.
With reference to
The disclosed analysis methods ensure that the maximum accuracy in conveying or understanding the meanings of the sentence is achieved.
Semantic hierarchy may include semantic notions or semantic entities referred to herein as “semantic classes”. The semantic classes may be arranged into a semantic hierarchy comprising hierarchical parent-child relationships. In general, a child semantic class inherits most properties of its direct parent and all ancestral semantic classes. For example, semantic class SUBSTANCE is a child of semantic class ENTITY and at the same time it is a parent of semantic classes GAS, LIQUID, METAL, WOOD MATERIAL, etc.
Each semantic class in the semantic hierarchy is supplied with a deep model. The deep model of the semantic class is a set of the deep slots. Deep slots reflect the semantic roles of child constituents in various sentences with objects of the semantic class as the core of a parent constituent and the possible semantic classes as fillers of deep slots. The deep slots express semantic relationships between constituents, including, for example, “agent”, “addressee”, “instrument”, “quantity”, etc. A child semantic class inherits and adjusts the deep model of its direct parent semantic class.
Semantic descriptions 304 are language-independent. Semantic descriptions 304 may provide descriptions of deep constituents, and may comprise a semantic hierarchy, deep slots descriptions, a system of semantemes, and pragmatic descriptions.
A system of semantemes represents a set of semantic categories. As an example, a semantic category “DegreeOfComparison” can be used to describe the degrees of comparison expressed by various forms of adjectives, for example, “easy”, “easier” and “easiest”. So, the semantic category “DegreeOfComparison” may include such semantemes as, for example, “Positive”, “ComparativeHigherDegree”, “SuperlativeHighestDegree”, among others. As another example, a semantic category “RelationToReferencePoint” can be used to describe an order as before or after a reference point relative to some event or object, etc., and its semantemes may include, “Previous”, “Subsequent”, and the order may be spatial or temporal in a broad sense. As yet another example, “EvaluationObjective”, as a semantic category, may describe an objective assessment, such as “Bad”, “Good”, etc.
The systems of semantemes include language-independent semantic attributes that express semantic characteristics as well as stylistic, pragmatic and communicative characteristics. Semantemes can also be used to express an atomic meaning that finds a regular grammatical and/or lexical expression in a language. By purpose and usage, semantemes may be divided into various kinds, including, but not limited to, grammatical semantemes, lexical semantemes, and classifying grammatical (differentiating) semantemes.
Grammatical semantemes 1132 are used to describe grammatical properties of constituents when transforming a syntactic tree (a language dependent object) into a semantic structure. Lexical semantemes 1134 describe specific properties of objects (for example, “being flat” or “being liquid”) and are used in the deep slot descriptions 1120 as restriction for deep slot fillers (for example, for the verbs “face (with)” and “flood”, respectively). Classifying grammatical (differentiating) semantemes 1136 express differentiating properties of objects within a single semantic class. For example, in the semantic class “HAIRDRESSER” the semanteme <<RelatedToMen>> is assigned to the lexical meaning “barber”, unlike other lexical meanings which also belong to this class, such as “hairdresser”, “hairstylist”, etc.
Pragmatic descriptions 1140 are used to assign a corresponding theme, style or genre to texts and objects of the semantic hierarchy. For example, “Economic Policy”, “Foreign Policy”, “Justice”, “Legislation”, “Trade”, “Finance”, etc.
With reference to
Word-inflexion description 910 describes how the main form of a word form may change according to its case, gender, number, tense, etc. and broadly includes all possible forms for a given word. Word-formation 930 describes which new words may be generated involving a given word. The grammemes are units of the grammatical systems 920 and, as shown by a link 922 and a link 924, the grammemes can be used to build the word-inflexion description 910 and the word-formation description 930.
Referring to
There may be various types of assessments and their combinations for estimating of correspondence of semantic structures. For example, the structural similarity of semantic trees (structure), presence of corresponding non-tree links, correspondence of the semantic classes for the main parts of sentence and their semantic roles (deep slots), such as, for example, Agent (Subject), Predicate (Head), Object, etc., and subordinate parts of the sentence, such as Instrument, Goal, Direction, Location, Time, etc. So, the degree of correspondence may be expressed by a number of assessments. A semantic structure is a directional acyclic graph, with nodes that are assigned the elements of the Semantic Hierarchy. In certain simple cases, it may be a tree (semantic tree) representing semantic structure of the parsed sentence. However, certain constituents of a sentence may be connected by non-tree links.
The non-tree links may reflect various linguistic phenomena, such as, ellipsis and coordination. They are presented in syntactic structures which are generated during various steps of analysis. Ellipsis is a language phenomenon which is represented by the absence of core constituents. An example of an elliptical English sentence is “The president signed the agreement and the secretary [signed] the protocol”.
Ellipsis can also be related to the coordination. Coordination is a language phenomenon which is presented in sentences with enumeration and/or a coordinating conjunction, such as “and”, “or”, “but”, etc. A simple example of a sentence with coordination—“John, Mary and Bill came home”. In this case only one of coordinated child constituent is attached in the surface slot of a parent constituent during building the graph of the generalized constituents. During the coordination processing, the linear order and multiple filling possibility of the surface slot are determined. If the attachment is possible, a proform which refers to the common child constituent is created and attached. The other examples of non-tree links are referential and structural control, among others.
Referring to
A graph matching algorithm (342) is adapted to process pairs of sentences of two languages. The sentence graph nodes are mapped to produce a partial set of node pairs (some nodes do not have a pair). A similarity score algorithm (344) goes through the mapped set of nodes of acyclic graphs and calculates nodes distance in terms of, for example, values between zero and one. In this case, zero represents a complete match and one represents that there is no relationship between the nodes of two structures in the semantic hierarchy. For example, the notions represented by the nodes may be equal, non-vocabulary text maybe exactly equal or case-insensitive equal, one node may be a parent of another one in the semantic hierarchy, or the nodes may be further removed but still related in the hierarchy. The distance values may be calculated based on training data by using an optimization procedure. For example, the Differential Evolution optimization algorithm may be used. Finally, when all the node distances have been computed, they are added to produce the final distance (346), which represents the degree of correspondence.
If the degree of correspondence is sufficient for the hypothesis to be considered confirmed (348), the correspondence between the sentence in one language and the corresponding sentence in another language has been established. If the hypothesis is not confirmed, the system tries to change the boundaries of the fragment and to verify the next hypothesis. If the best hypothesis for the fragment had not been found, the fragment is marked by color, and the user is provided with an option to correct it manually. As a result of applying this process to each pair of fragments, aligned tagged parallel text corpus is obtained.
Marking may be displayed to a user or may be hidden. Such marking may be used for indexing and searching. When marking is displayed to a user, various notations may be used. For example, an XML-like or a tree-like notation may be used. More specific notation may be provided, for example, as subscript. Some features may be selected by a user for displaying, such as, parts of speech, syntactic roles, semantic roles, semantic classes, etc.
.” Since these sentences are the translations of each other, they have the same semantic structures where the trees match, and semantic classes and deep slots coincide. Also, both sentences have the same non-tree links.
In addition to automated matching of fragments, the user interface provide for manual correction of the alignment. For example, after the first pass of the program, there may be fragments that have been aligned incorrectly.
The reason of the incorrectness is that there are three short sentences in the German section translated as one English sentence (line 5). So, the fragments in German and in English parts should be correlated to properly identify parallel fragments. This can be accomplished by aligning English fragment (line 5) by German fragment (line 4) by means of the actions Mark and Match; and then by merging German fragments (lines 4-6) by means of the action Merge. Alternatively it can be accomplished by merging German fragments (lines 4-6) by means of the action Merge, and then aligning English fragment (line 5) by the obtained German fragment (line 4) by means of the actions Mark and Match. Another alternative is to merge German fragments (lines 4-6) by means of the action Merge, and then to move up English fragment (line 5) to obtained German fragment (line 4) by means of actions Up.
The first two approaches in the previous paragraph allow merging of fragments which are not located side by side. In one embodiment there are special buttons or menu items in the user interface to facilitate such actions.
The hardware 1400 also typically receives a number of inputs and outputs for communicating information externally. For interface with a user or operator, the hardware 1400 may include one or more user input devices 1406 (e.g., a keyboard, a mouse, imaging device, scanner, microphone) and a one or more output devices 1408 (e.g., a Liquid Crystal Display (LCD) panel, a sound playback device (speaker)). To embody the present invention, the hardware 1400 typically includes at least one display device.
For additional storage, the hardware 1400 may also include one or more mass storage devices 1410, e.g., a removable disk drive, a hard disk drive, a Direct Access Storage Device (DASD), an optical drive (e.g. a Compact Disk (CD) drive, a Digital Versatile Disk (DVD) drive) and/or a tape drive, among others. Furthermore, the hardware 1400 may include an interface with one or more networks 1412 (e.g., a local area network (LAN), a wide area network (WAN), a wireless network, and/or the Internet among others) to permit the communication of information with other computers coupled to the networks. It should be appreciated that the hardware 1400 typically includes suitable analog and/or digital interfaces between the processor 1402 and each of the components 1404, 1406, 1408, and 1412 as is well known in the art.
The hardware 1400 operates under the control of an operating system 1414, and executes various computer software applications, components, programs, objects, modules, etc. to implement the techniques described above. Moreover, various applications, components, programs, objects, etc., collectively indicated by application software 1416 in
In general, the routines executed to implement the embodiments of the invention may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as a “computer program.” A computer program typically comprises one or more instruction sets at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processors in a computer, cause the computer to perform operations necessary to execute elements involving the various aspects of the invention. Moreover, while the invention has been described in the context of fully functioning computers and computer systems, those skilled in the art will appreciate that the various embodiments of the invention are capable of being distributed as a program product in a variety of forms, and that the invention applies equally to actually effect the distribution regardless of the particular type of computer-readable media used. Examples of computer-readable media include but are not limited to recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD-ROMs), Digital Versatile Disks (DVDs), flash memory, etc.), among others. Another type of distribution may be implemented as Internet downloads.
While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative and not restrictive of the broad invention and that this invention is not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art upon studying this disclosure. In an area of technology such as this, where growth is fast and further advancements are not easily foreseen, the disclosed embodiments may be readily modified or re-arranged in one or more of its details as facilitated by enabling technological advancements without departing from the principals of the present disclosure.
This application is a continuation-in-part of the U.S. patent application Ser. Nos. 11/690,102 and 11/690,104, both filed on 22 Mar. 2007, which are currently co-pending, and which are continuations-in-part of U.S. patent application Ser. No. 11/548,214, filed on 10 Oct. 2006, now U.S. Pat. No. 8,078,450. The disclosures of the above applications are incorporated herein by reference to the extent that such disclosures are not inconsistent with this application.
Number | Date | Country | |
---|---|---|---|
Parent | 11690102 | Mar 2007 | US |
Child | 13464447 | US | |
Parent | 11548214 | Oct 2006 | US |
Child | 11690102 | US | |
Parent | 11690104 | Mar 2007 | US |
Child | 11548214 | US | |
Parent | 11548214 | Oct 2006 | US |
Child | 11690104 | US |