Patent Grant
11604930
With the increasing extent and usage of the internet, opportunities for one user to exchange documents (e.g., correspond with, conduct business, exchange information) with another user are constantly increasing. In some cases, exchanged documents may be in a language that one user does not understand and translation of the documents is required. A mere translation of the text in the document may not be sufficient for a complete understanding of the document because contextual information in the structure or layout of the document may be lost in the translation process.
In general, one or more embodiments the invention relate to a method for generating an editable translated electronic document from an input image of an original document with a first layout. The method includes: segmenting the input image to generate a first region comprising first untranslated text; extracting, from the first region, the first untranslated text and a first layout information; generating an editable output data comprising the first untranslated text and the first layout information; translating the first untranslated text into a translated text; editing the output data to include the translated text; and generating, using the first layout information, the translated electronic document comprising the translated text and a second layout that is identical to the first layout.
In general, one or more embodiments of the invention relate to non-transitory computer readable medium (CRM) storing computer readable program code for generating an editable translated electronic document from an input image of an original document with a first layout. The computer readable program code causes a computer to: segment the input image to generate a first region comprising first untranslated text; extract, from the first region, the first untranslated text and a first layout information; generate an editable output data comprising the first untranslated text and the first layout information; translate the first untranslated text into a translated text; edit the output data to include the translated text; and generate, using the first layout information, the translated electronic document comprising the translated text and a second layout that is identical to the first layout.
In general, one or more embodiments of the invention relate to a system for generating an editable translated electronic document from an input image of an original document with a first layout. The system includes: a memory; and a processor coupled to the memory. The processor: segments the input image to generate a first region comprising first untranslated text; extracts, from the first region, the first untranslated text and a first layout information; generates an editable output data comprising the first untranslated text and the first layout information; translates the first untranslated text into a translated text; edits the output data to include the translated text; and generates, using the first layout information, the translated electronic document comprising the translated text and a second layout that is identical to the first layout.
Other embodiments of the invention will be apparent from the following description and the appended claims.
Specific embodiments of the invention will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.
In the following detailed description of embodiments of the invention, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.
Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create a particular ordering of the elements nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before,” “after,” “single,” and other such terminology. Rather the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.
In general, embodiments of the invention provide a method, a non-transitory computer readable medium (CRM), and a system for processing an image of an original document written in a first language with a first layout, translating the text of the original document into a different language, and generating/outputting an editable translated version of the original document in a second language with a second layout that is identical to the first layout of the original document.
In other words, when translating the text in the original document into a different language, retaining the structure of the original document may help improve a user's understanding of the original document. The original document conveys information in an understandable manner by including one or more text objects (e.g., characters, text strings, titles, paragraphs, captions, tables, etc.) and non-text objects (e.g., charts, graphs, images, clipart, or logos) in a structured format. For example, a block of text describing a picture may be placed next to the picture as a caption. In the translated electronic version of the original document, the translated text may be placed in the same position relative to the picture such that the user understands that the translated text is the caption for the picture.
In one or more embodiments, one or more processes applying a combination of visual and semantic analyses are executed to segment the image of the original document and obtain one or more regions of the original document. Each region contains an object (e.g., untranslated text or non-text object) and a layout information that describes the object and/or how the object is oriented or positioned in the structure of the original document. By processing and preserving the structure of the original document when translating the original text, it is possible to generate an editable translated version of the original document that would be easier for a user to understand.
The system 100 includes the buffer 102. The buffer 102 may be implemented in hardware (i.e., circuitry), software, or any combination thereof. The buffer 102 is configured to store an input image 104 including one or more original documents. The input image 104 may be obtained (e.g., downloaded, scanned, imaged, etc.) from any source by the imagine engine 110. The input image 104 may be a single image or multiple images of a single-page document or a multi-paged document. The input image 104 may be of any size and in any format (e.g., BMP, JPEG, PNG, GIF, etc.). The input image 104 includes a first region comprising a first untranslated text. The first untranslated text may be part of the main body of an original document (e.g., free flowing text block) or embedded within other graphical elements of the original document (e.g., text embedded within a graphic).
The buffer 102 is also configured to store an output data 106. The output data 106 may be editable (i.e., items may be altered, added, or removed) and may include untranslated text, translated text, non-text objects, and layout information of regions extracted from the input image 104. Layout information may include a shape, a size, a width, a height, a relative position, a number of rows (e.g., rows in a table or lines in a paragraph), a number of columns (e.g., columns in a table or text columns in a page), a font, a text attribute (e.g., justification, boldness, strike-through, etc.), a line spacing, a margin, a text direction (e.g. horizontal or vertical), a wrap type (i.e., how text flows around objects), or any appropriate information that describes the layout of a region and/or an object within the region. The output data 106 may be generated, edited, and/or deleted by the imaging engine 110, segmenting engine 112, image processing engine 114, translation engine 116, output engine 118, or any combination thereof.
The buffer 102 is also configured to store the editable translated version of the original document as a translated electronic document (TED) 108. The TED 108 may be editable and include one or more text objects (e.g., a text character, a text string, a title, a paragraph, a table) and non-text objects (e.g., an image, a graphic, a chart, a graph, etc.). The TED 108 may be generated by the output engine 118. The TED 108 may be a single-paged document or a multi-paged document. Further, the TED 108 may be of any size and in any format (e.g., PDF, OOXML, ODF, HTML, etc.). Generally, the TED 108 contains the same content as the original document, in a second layout that is identical to the first layout of the original document, but with at least a portion of an original untranslated text replaced with a translated text.
The system 100 further includes the imaging engine 110. The imaging engine 110 may be implemented in hardware (i.e., circuitry), software, or any combination thereof. The imaging engine 110 is configured to obtain the input image 104. For example, the imagining engine 110 may scan an original document in a scanner, image the original document with a digital camera or with PIC device, or download an image file from a storage device or server to obtain the input image 104. Alternatively, the imaging engine 110 may obtain the input image 104 from any source and is not limited to the above methods. Furthermore, the imaging engine 110 may generate, edit, and/or delete any appropriate layout information in the buffer 102.
The system 100 further includes the segmenting engine 112. The segmenting engine 112 may be implemented in hardware (i.e., circuitry), software, or any combination thereof. The segmenting engine 112 is configured to segment the input image 104 and generate one or more regions. For example, the segmenting engine 112 may perform one or more visual or semantic analyses to determine one or more regions of the input image 104. Visual or semantic analysis may be applied using a system, a program, a software, or any combination thereof that is able to accurately parse regions of the input image 104.
Each region may include a text object, a non-text object, or any combination thereof. A region may be further segmented into additional regions based on the objects within the region (e.g., cells of a table). The regions of the input image 104 may be stored in the buffer 102. Furthermore, the segmenting engine 112 may generate, edit, and/or delete the layout information in the buffer 102.
The system 100 further includes the image processing engine 114. The image processing engine 114 may be implemented in hardware (i.e., circuitry), software, or any combination thereof. The image processing engine 114 is configured to perform image processing on the regions of the input image 104 to extract untranslated text and/or layout information from these regions and store the text and/or layout information in the buffer 102. The image processing engine 114 may perform optical character recognition (OCR) or intelligent character recognition (ICR) to recognize and extract text from the regions of the input image 104. Furthermore, based on the image processing of the input image 104 or the regions of the input image 104, the imaging engine 110 may generate, edit, and/or delete any appropriate text or layout information in the buffer 102.
The system 100 further includes the translation engine 116. The translation engine 116 may be implemented in hardware (i.e., circuitry), software, or any combination thereof. The translation engine 116 is configured to translate text from the first language into a second language different from the first language. In some cases, the translation engine 116 translates the first language into a different form of the first language (e.g., from traditional Chinese characters to simplified Chinese characters). The translation engine 116 may comprise a local process, a cloud-based translation service, or any combination thereof.
In one or more embodiments, the translation engine 116 translates the untranslated text stored in the output data 106 from the first language into a second language and include the translated text in the output data 106. The translation engine 116 may translate the translated text from the second language back into the first language and include the re-translated text in the output data 106. Furthermore, the translation engine 116 may generate, edit, and/or delete any appropriate text or layout information in the buffer 102.
The system 100 further includes the output engine 118. The output engine 118 may be implemented in hardware (i.e., circuitry), software, or any combination thereof. The output engine 118 is configured to control interactions between the buffer 102 and the various engines 110, 112, 114, 116 to generate the TED 108. For example, the output engine 118 uses the translated text, the layout information, and/or the non-text objects included in the output data 106 to generate the TED 108 with an identical layout as the first layout of the original document. Furthermore, the output engine 118 may generate, edit, and/or delete any appropriate text or layout information in the buffer 102.
Although the system 100 is shown as having six components 102, 110, 112, 114, 116, and 118, in other embodiments of the invention, the system 100 may have more or fewer components. Further, the functionality of each component described above may be split across components or combined in a single component. Further still, each component 102, 110, 112, 114, 116, and 118 may be utilized multiple times to carry out an iterative operation.
In one or more embodiments, the system 100 may work as a component of a larger document processing system. For example, the system 100 may be coupled to another product or system (e.g., a PIC device, a multi-function peripheral, etc.) that generates input images or manages/manipulates electronic documents.
In STEP 200, an input image 104 of an original document with a first layout is segmented by the segmenting engine 112 to generate a first region comprising a first untranslated text. Prior to segmentation, the input image 104 may be obtained (e.g., downloaded, scanned, imaged, etc.) by the imaging engine 110 from any source.
In one or more embodiments, discussed below with respect to
In STEP 202, the image processing engine 114 extracts the first untranslated text and a first layout information from the first region. Image processing may include OCR, ICR, or any appropriate image processing technique to recognize and extract text and/or layout information from an image. The imaging engine 114 may extract a text-content information (e.g., text characters, text strings, etc.), text styling attributes (e.g., font, size, color, boldness, italics, etc.), and/or other appropriate properties of the first untranslated text. The first layout information may include a position in the first region, a position in the input image 104, a text justification, a number of columns, a number of rows or lines, a line spacing, etc.
As a first non-limiting example, a region may correspond to an introduction section of the original document and thus may include a title and a paragraph (i.e., collectively the first untranslated text). In addition to the first untranslated text comprising the title and the paragraph, the image processing engine 114 may extract a first layout information comprising a spacing of the title from the paragraph and different styling attributes (e.g., boldness and font size) for both the title and the paragraph.
As a second non-limiting example, a region may correspond to a table of information in the original document, each cell of the table including text (i.e., collectively the first untranslated text) and/or non-text characters. The image processing may recognize the table and extract the first untranslated text along with a first layout information comprising the size, position, number of rows, number of columns of the table, the relative location of each text string within the table, etc.
As a third non-limiting example, a region may correspond to a graphical object containing text (i.e., the first untranslated text is embedded in a graphic). The image processing engine 114 may recognize the text embedded in the graphic and extract the first untranslated text along with a first layout information comprising a relative location within the graphic, an orientation, a skew, etc.
In STEP 204, an editable output data (i.e., the output date 106 discussed above in reference to
In STEP 206, the first untranslated text is extracted from the output data 106. The first untranslated text may be compiled into a translation list. The translation list may comprise one or more text strings that correspond to various parts of the first untranslated text. Constructing the translation list may be a recursive process if data is nested within the output data 106 (e.g., a paragraph is broken into sentences, a sentence is broken into words, cells of a table, etc.). The translation list may be reduced in size by consolidating identical entries.
In the first non-limiting example of a region comprising a title and a paragraph, the translation list may contain two text strings: one for the title; and one for the paragraph. Alternatively, a plurality of text strings may be used to represent one or a combination of the untranslated texts in the paragraph (e.g., one text string for each word or sentence).
In the second non-limiting example of a region comprising a table, the translation list may comprise a plurality of text strings that correspond to each cell of the table that contains text.
In the third non-limiting example of a region comprising a graphic containing text, the translation list may comprise one or more text strings that correspond to each instance of text embedded in the graphic.
Once the translation list has been compiled, the translation list may be submitted to a translation engine 116, as discussed above in reference to
In STEP 208, the translation engine 116 translates the first untranslated text into a translated text. Subsequently, the translation engine may output the translated text as a single output to reduce processing overhead and/or network latency and congestion. The translation engine 116 may also construct a translation map that associates each untranslated text string in the translation list with the corresponding translated text string in the translated text. The translation map may be reduced in size by consolidating identical entries.
In STEP 210, the output data 106 is edited to include the translated text. The output data 106 may be edited to also include the translation map to describe the relationship between the first untranslated text and the translated text.
In STEP 212, the output engine 118 generates, using the first layout information, the TED 108 comprising the translated text and a second layout that is identical to the first layout. The first region may be recreated in the TED 108 using the output data 106 (e.g., the translated text, the first layout information, the translation list, the translation map, and/or any non-text objects from the first region). The output engine 118 may use the translation list and the translation map to replace the first untranslated text with the corresponding translated text.
The replacement of the first untranslated text may be performed in the order of the text strings of the translation list (i.e., each text string is identified in the first untranslated text and sequentially replaced). Alternatively, if a particular text string is repeated throughout the first untranslated text (e.g., identical cells of a table), the particular string may be replaced multiple times before proceeding to the next text string in the translation list. However, the invention is not limited to the order of operations described above and replacing the translated text may be performed using any appropriate method and in any appropriate order.
The output engine 118 uses the first layout information to retain the original formatting of the original document when replacing the first untranslated text. The output engine 118 may use the same shape, width, height, relative position, or any combination thereof when recreating the first region. Furthermore, the output engine 118 may use the same font, and/or text attributes (e.g., justification, boldness, strike-through, etc.) when replacing the first untranslated text with the translated text.
In the first non-limiting example of a region comprising a title and a paragraph, the output engine 118 uses the translation map and the first layout information to recreate the spacing and relative position of the title and the paragraph. For example, if the original title was bolded and centered above the original paragraph, the translated title may have the same bolded font and may be centered over the translated paragraph in the same manner as the original document. Accordingly, the user can easily recognize the emphasized text in the TED 108 as the title of the TED 108.
In the second non-limiting example of a region comprising a table, the output engine 118 uses the translation map and the first layout information to recreate the original table using the same number of rows and columns before replacing the first untranslated text in each cell with the corresponding translated text. Accordingly, the user can reference the table in the TED 108 using the same table coordinates as the original document.
In the third non-limiting example of a region comprising an image containing text, the output engine 118 uses the translation map and the first layout information to recreate the image with the translated text. The output engine 118 may directly modify the image (i.e., by pixel manipulation) to include the translated text or may simply position a text box containing the translated text to cover the first untranslated text in the image.
In general, the output engine 118 creates the TED 108 with a second layout that is identical to the first layout of the original document. Alternatively, slight differences in the structure of the TED 108 may be introduced by changing one or more elements of the first layout information to ensure the second layout is identical to the first layout. In this application, an identical layout means that the relative relationships between text objects and non-text objects are the same in both the first and second layout. Even if slight differences exist between a region in the first layout and the corresponding region in the second layout (e.g., change in height/width of the region), the first and second layout may still be identical.
For example, an untranslated text string, contained within an untranslated text box in the original document, is translated into a corresponding translated text string that is longer than the original untranslated text string. The output engine 118 may create a translated text box in the TED 108 that wraps the translated text onto an additional line. Thus, the translated text box in the identical second layout may have the same width but a larger overall height compared to the untranslated text box in the first layout. Similarly, the output engine 118 may adjust the absolute position of objects around the translated text box in the identical second layout to retain the relative positioning of objects in the first layout of the original document.
In one or more embodiments, the first untranslated text may be retained in the TED 108. For example, the first untranslated text may be replaced by the corresponding translated text using a feature that tracks edits in the TED 108 (i.e., the first untranslated text is visible in the TED 108 but is marked or tagged for deletion).
After the translated text has replaced the first untranslated text, the first untranslated text may be added to the TED 108 as an annotation. For example, the output engine 118 may retain the first untranslated text in the TED 108 by adding one or more comment bubbles containing the first untranslated text. Alternatively, the output engine 118 may include a re-translated text (i.e., a translation of the translated text into the language of the untranslated text) into the TED 108 by adding one or more comment bubbles containing the re-translated text. In addition, the output engine 118 may retain the first untranslated text in the margins (e.g., as a footnote or header) of the TED 108.
By having the text of the original document appear in both the translated and untranslated/re-translated forms in the TED 108, a user may more easily confirm the accuracy of the translation engine 116. Furthermore, the user may be able to revise parts of the TED 108 if needed (e.g., an obvious translation error occurs).
Furthermore, the output engine 118 may generate, using the first layout information, an editable untranslated electronic document that is output with the TED 108. The untranslated electronic document is editable and comprises the untranslated text in the first layout. In other words, the untranslated electronic document is an editable electronic form of the original document. By outputting the TED 108 and the untranslated electronic document, the user can directly compare an editable electronic version of the original document and the TED 108.
In STEP 250, the segmenting engine 112 segments the input image 104 to generate the plurality of regions (i.e., a first region, a second region, etc.) and creates a region list that includes each of the plurality of regions.
In STEP 252, the image processing engine 114 extracts an untranslated text and a layout information from each of the plurality of regions (i.e., the first untranslated text, the second untranslated text, the first layout information, the second layout information, etc.) in the region list, includes each of the plurality of untranslated texts and each of the plurality of layout information in the output data 106. Each of the plurality of untranslated texts is extracted from the output data 106, without the corresponding layout information, and compiled into a translation list.
In STEP 254, the translation list is submitted to the translation engine 116 as a single instruction or as multiple instructions and the translation engine 116 returns the translated text as a single output. The translated text includes a translation of each untranslated text.
In STEP 256, the translation engine 116 creates a translation map that associates (i.e., maps) each of the plurality of untranslated texts with the corresponding translated text.
In STEP 258, the output engine 118 iteratively processes each region in the region list.
In STEP 260, the output engine 118 selects the next region to be processed from the region list.
In STEP 262, the output engine 118 determines if the selected region contains untranslated text (e.g., by referencing the translation list, parsing the selected region, etc.). If the determination in STEP 262 is NO, the process continues to STEP 268.
If the determination in STEP 262 is YES, the process continues to STEP 264 where the output engine 118 determines if the translation map comprises the untranslated text from STEP 262. If the determination in STEP 264 is NO (e.g., the translation engine failed), the process continues to STEP 268 and the untranslated text remains in the TED 108. In one or more embodiments, the untranslated text may be annotated with a text attribute or special character to indicate that the translation step failed.
If the determination in STEP 264 is YES, the process continues to STEP 266 where the output engine 118 edits the output data 106 to include, using the translation map, the translated text that corresponds to each of the plurality of untranslated texts.
In STEP 268, the output engine 118 completes processing of the currently selected region.
In STEP 270, the output engine 118 determines if the processing of each of the plurality of regions in the region list is complete. If the determination in STEP 270 is NO, the process continues to STEP 260 where the next region is selected for processing.
If the determination in STEP 270 is YES, the process continues with STEP 272 where the output engine 118 generates, using each of the plurality of layout information extracted by the various engines 110, 112, 114, 116, and 118, the TED 108 comprising the translated text and a second layout that is identical to the first layout based upon the extracted layout information. Each of the plurality of regions included in the region list may be recreated in the TED 108 using the output data 106 (e.g., translated text, layout information, the translation list, the translation map, and/or any non-text objects from each region). The output engine 118 may use the translation list and the translation map to replace each of the plurality of untranslated texts with the corresponding translated text. The TED 108 may be output to the user in any format.
In general, the output engine 118 creates the TED 108 with a second layout that is identical to the first layout of the original document. Alternatively, slight differences in the second layout may be introduced (e.g., a change in length of a text string during translation into another language) but the output engine 118 may maintain the first layout and relationship between text objects and non-text objects by changing one or more elements of the first layout information. For example, when a translated text string is longer than the original first untranslated text string, the output engine 118 may wrap the translated text onto an additional line (i.e., increase the height of the translated text box) in order to maintain the width of the first untranslated text box.
The method of generating the translated electronic document according to one or more embodiments may be implemented on virtually any type of computing system, regardless of the platform being used. For example, the computing system may be one or more mobile devices (e.g., laptop computer, smartphone, personal digital assistant, tablet computer, or other mobile device), desktop computers, servers, blades in a server chassis, or any other type of computing device or devices that includes at least the minimum processing power, memory, and input and output device(s) to perform one or more embodiments of the invention. For example, as shown in
The computing system 600 may also include one or more input device(s) 608, such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, imaging engine (e.g., a scanner or digital camera, a PIC device), or any other type of input device. Further, the computing system 600 may include one or more output device(s) 610, such as a screen (e.g., a liquid crystal display (LCD), a plasma display, touchscreen, cathode ray tube (CRT) monitor, projector, or other display device), a printer, external storage, or any other output device. One or more of the output device(s) may be the same or different from the input device(s). The computing system 600 may be connected to a network 612 (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) via a network interface connection (not shown). The input and output device(s) may be locally or remotely (e.g., via the network 612) connected to the computer processor(s) 602, memory 604, and storage device(s) 606. Many different types of computing systems exist, and the aforementioned input and output device(s) may take other forms.
Software instructions in the form of computer readable program code to perform embodiments of the invention may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium such as a CD, DVD, storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that when executed by a processor(s), is configured to perform embodiments of the invention.
Further, one or more elements of the aforementioned computing system 600 may be located at a remote location and be connected to the other elements over a network 612. Further, one or more embodiments of the invention may be implemented on a distributed system having a plurality of nodes, where each portion of the invention may be located on a different node within the distributed system. In one embodiment of the invention, the node corresponds to a distinct computing device. Alternatively, the node may correspond to a computer processor with associated physical memory. The node may alternatively correspond to a computer processor or micro-core of a computer processor with shared memory and/or resources.
One or more of the embodiments of the invention may have one or more of the following advantages and improvements over conventional document processing and translation technologies: facilitating the exchange of information between users with different native languages; generating an editable translated electronic document while retaining the first layout of the original document to reduce the loss of contextual information during the translation process; retaining the untranslated text in the translated electronic document (e.g., by generating one or more electronic documents with the text in both the first and second languages); reducing contextual translation errors in translated electronic documents; and reducing processing and communication costs by performing a single translation process. One or more of the above advantages may improve a user's understanding of the contents of the original document and/or allow a user fluent in both the first and second languages to verify the translation results and easily edit the translated electronic document to increase the accuracy of the translation.
Although the disclosure has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that various other embodiments may be devised without departing from the scope of the present invention. Accordingly, the scope of the invention should be limited only by the attached claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7359849 | Palmquist | Apr 2008 | B2 |
| 7969596 | Mathieson et al. | Jun 2011 | B2 |
| 8144990 | Englund et al. | Mar 2012 | B2 |
| 8508787 | Minhas | Aug 2013 | B2 |
| 20030200078 | Luo et al. | Oct 2003 | A1 |
| 20120245922 | Kozlova | Sep 2012 | A1 |
| 20130282599 | Kang | Oct 2013 | A1 |
| 20140006007 | Sumita | Jan 2014 | A1 |
| Number | Date | Country |
|---|---|---|
| 1058896 | Aug 2005 | EP |
| Number | Date | Country | |
|---|---|---|---|
| 20210097143 A1 | Apr 2021 | US |
