Patent Grant
10867136
This application claims the benefit under 35 USC 119(a) of Korean Patent Application No. 10-2016-0086036 filed on Jul. 7, 2016, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.
The following description relates to an automated interpretation method and apparatus.
Developments of the Internet and information telecommunication (IT) technology have enabled people to receive contents in various languages. Also, with globalization in business, recognition and translation technologies for content translation and communication among users using various languages have been of interest.
Due to differences in the words that are used or the expression types of sentences, different recognitions and translations may be derived from voice inputs having similar or same meanings.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is the Summary intended to be used as an aid in determining the scope of the claimed subject matter.
In one general aspect, an automated interpretation method includes encoding a voice signal in a first language to generate a first feature vector, decoding the first feature vector to generate a first language sentence in the first language, encoding the first language sentence to generate a second feature vector with respect to a second language, decoding the second feature vector to generate a second language sentence in the second language, controlling a generating of a candidate sentence list based on any one or any combination of the first feature vector, the first language sentence, the second feature vector, and the second language sentence, and selecting, from the candidate sentence list, a final second language sentence as a translation of the voice signal.
The generating of the candidate sentence list may include acquiring a candidate sentence, from a database, determined to correspond to any one or any combination of the first feature vector, the first language sentence, the second feature vector, and the second language sentence from a database.
The acquiring of the candidate sentence may include retrieving respective elements determined similar to any of the first feature vector, the first language sentence, the second feature vector, and the second language sentence from a plurality of elements stored in the database based on one or more approximate nearest neighbor (NN) algorithms.
The generating of the candidate sentence list may include any one or any combination of acquiring a first interpretation result matching a first language feature vector, from a database, determined similar to the first feature vector, acquiring a second interpretation result matching a previous recognized sentence, from the database, determined similar to the first language sentence, acquiring a third interpretation result matching a second language feature vector, from the database, determined similar to the second feature vector, and acquiring a fourth interpretation result matching a previous translation sentence, from the database, determined similar to the second language sentence.
The generating of the candidate sentence list may further include adding any previous translation sentences corresponding to any of the first interpretation result, the second interpretation result, the third interpretation result, and the fourth interpretation result to the candidate sentence list, and adding the second language sentence to the candidate sentence list.
The acquiring of the second interpretation result may include converting the first language sentence into a vector, and determining which of plural previous recognized sentences, from the database, are similar to the first language sentence based on the vector.
The acquiring of the fourth interpretation result may include converting the second language sentence into a vector, and determining which of plural previous translation sentences, from the database, are similar to the second language sentence based on the vector.
The selecting of the final second language sentence may include calculating scores of candidate sentences included in the candidate sentence list based on the second feature vector, and selecting a candidate sentence, from the candidate sentence list, having a highest of the calculated scores to be the final second language sentence.
The generating of the first feature vector may include sampling the voice signal in the first language based on a predetermined frame length, generating respective input vectors corresponding to frames, sequentially inputting the respective input vectors to an encoder used for voice recognition, and determining the first feature vector to be an output from the encoder for the sequentially input respective input vectors.
The generating of the first language sentence may include inputting the first feature vector to a decoder used for voice recognition, generating a predetermined number of sentence sequences based on probabilities of sub-words sequentially output from the decoder, and selecting a sentence sequence having a highest score among the predetermined number of sentence sequences to be the first language sentence.
The generating of the second feature vector may include dividing the first language sentence into a plurality of sub-words, sequentially inputting input vectors respectively indicating the plurality of sub-words to an encoder used for machine translation, and determining the second feature vector to be an output from the encoder for the sequentially input input vectors.
The generating of the second language sentence may include inputting the second feature vector to a decoder used for machine translation, generating a predetermined number of sentence sequences based on probabilities of sub-words sequentially output from the decoder, and selecting a sentence sequence having a highest score among the predetermined number of sentence sequences to be the second language sentence.
The method may further include storing the first feature vector, the first language sentence, and the second feature vector in a database, and storing any one or any combination of the second language sentence and the final second language sentence corresponding to the first feature vector, the first language sentence, and the second feature vector in the database.
In one general aspect, one or more embodiments may include a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform any one or any combination of operations and processes discussed herein.
In one general aspect, an automated interpretation method includes encoding a first language sentence in a first language to generate a feature vector with respect to a second language, decoding the feature vector to generate a second language sentence in the second language, controlling a generating of a candidate sentence list based on any one or any combination of the feature vector and the second language sentence, and selecting, from the candidate sentence list, a final second language sentence from the candidate sentence list.
The method may further include encoding a voice signal in the first language to generate a first feature vector, and decoding the first feature vector to generate the first language sentence.
The generating of the candidate sentence list may include any one or any combination of acquiring a first translation result matching a second language feature vector, from a database, determined similar to the feature vector, and acquiring a second translation result matching a sentence, from the database, determined similar to the second language sentence.
The generating of the candidate sentence list may further include adding any previous translation sentences corresponding to any of the first translation result and the second translation result to the candidate sentence list, and adding the second language sentence to the candidate sentence list.
The selecting of the final second language sentence may include calculating scores of candidate sentences included in the candidate sentence list based on the feature vector, and selecting a candidate sentence, from the candidate sentence list, having a highest of the calculated scores to be the final second language sentence.
The generating of the feature vector may include dividing the first language sentence into a plurality of sub-words, sequentially inputting input vectors respectively indicating the plurality of sub-words to an encoder used for machine translation, and determining the feature vector to be an output from the encoder for the sequentially input input vectors.
The generating of the second language sentence may include inputting the feature vector to a decoder used for machine translation, generating a predetermined number of sentence sequences based on probabilities of sub-words sequentially output from the decoder, and selecting a sentence sequence having a highest score among the predetermined number of sentence sequences to be the second language sentence.
The method may further include storing the feature vector in a database, with the first language sentence being stored in the database, and storing any one or any combination of the second language sentence and the final second language sentence corresponding to the first language sentence and the feature vector in the database.
In one general aspect, an automated interpretation apparatus includes a voice recognizer configured to generate a first language sentence by decoding a first feature vector, and configured to generate the first feature vector with respect to a first language by recognition encoding a voice signal that is in the first language, a translator configured to generate a second language sentence in a second language by decoding a second feature vector, and configured to generate the second feature vector with respect to the second language by translation encoding the first language sentence that is in the first language, and a processor configured to select, as a translation of the voice signal, a final second language sentence from a candidate sentence list generated based on any one or any combination of the first feature vector, the first language sentence, the second feature vector, and the second language sentence.
The voice recognizer may include a recognition decoder configured to perform the decoding of the first feature vector to generate the first language sentence and a recognition encoder configured to perform the recognition encoding of the voice signal to generate the first feature vector, and the translator may include a translation decoder configured to perform the decoding of the second feature vector to generate the second language sentence and a translation encoder configured to perform the translation encoding of the first language sentence to generate the second feature vector.
The processor may be further configured to include the recognition encoder, the recognition decoder, the translation encoder, and the translation decoder, and the recognition encoder may implement a neural network, of one or more neural networks of the automated interpretation apparatus, configured to determine the first feature vector based on the voice signal, the recognition decoder may implement a neural network, of the one or more neural networks of the automated interpretation apparatus, configured to determine the first language sentence based on the first feature vector, the translation encoder may implement a neural network, of the one or more neural networks of the automated interpretation apparatus, configured to determine the second feature vector based on the first language sentence, and the translation decoder may implement a neural network, of the one or more neural networks of the automated interpretation apparatus, configured to determine the second language sentence based on the second feature vector.
The automated interpretation apparatus may further include a memory having a database, and the processor may be configured to acquire a candidate sentence determined to correspond to any one or any combination of the first feature vector, the first language sentence, the second feature vector, and the second language sentence from the database.
The processor may be configured to retrieve respective elements determined similar to any of the first feature vector, the first language sentence, the second feature vector, and the second language sentence from a plurality of elements stored in the database based on one or more approximate nearest neighbor (NN) algorithms.
The processor may be configured to acquire any one or any combination of a first interpretation result matching a first language feature vector, from a database, determined similar to the first feature vector, a second interpretation result matching a previous recognized sentence, from the database, determined similar to the first language sentence, a third interpretation result matching a second language feature vector, from the database, determined similar to the second feature vector, and a fourth interpretation result matching a previous translation sentence, from the database, determined similar to the second language sentence from a database.
The processor may be configured to add any previous translation sentences corresponding to any of the first interpretation result, the second interpretation result, the third interpretation result, and the fourth interpretation result to the candidate sentence list, and may be configured to add the second language sentence to the candidate sentence list.
The processor may be configured to convert the first language sentence into a vector and may be configured to determine which of plural previous recognized sentences, from the database, are similar to the first language sentence based on the vector.
The processor may be configured to convert the second language sentence into a vector and may be configured to determine which of plural previous translation sentences, from the database, are similar to the second language sentence based on the vector.
The translator may be configured to calculate scores of candidate sentences included in the candidate sentence list based on the second feature vector, and the processor may be configured to select a candidate sentence, from the candidate sentence list, having a highest of the calculated scores to be the final second language sentence.
The processor may be configured to sample the voice signal in the first language based on a predetermined frame length, configured to generate respective input vectors corresponding to frames, configured to sequentially input the respective input vectors to an encoder used for voice recognition, and configured to determine the first feature vector to be an output from the encoder for the sequentially input respective input vectors.
The processor may be configured to input the first feature vector to a decoder used for voice recognition, configured to generate a predetermined number of sentence sequences based on probabilities of sub-words sequentially output from the decoder, and configured to select a sentence sequence having a highest score among the predetermined number of sentence sequences to be the first language sentence.
The processor may be configured to divide the first language sentence into a plurality of sub-words, configured to sequentially input input vectors respectively indicating the plurality of sub-words to an encoder used for machine translation, and configured to determine the second feature vector to be an output from the encoder for the sequentially input input vectors.
The processor may be configured to input the second feature vector to a decoder used for machine translation, configured to generate a predetermined number of sentence sequences based on probabilities of sub-words sequentially output from the decoder, and configured to select a sentence sequence having a highest score among the predetermined number of sentence sequences to be the second language sentence.
The processor may be configured to store the first feature vector, the first language sentence, and the second feature vector in a database and may be configured to store any one or any combination of the second language sentence and the final second language sentence corresponding to the first feature vector, the first language sentence, and the second feature vector in the database.
In one general aspect, an automated interpretation system includes a translator configured to generate a second language sentence in a second language by decoding a feature vector and configured to generate the feature vector with respect to the second language by translation encoding a first language sentence that is in a first language, and a processor configured to select a final second language sentence, as a translation of the first language sentence, from a candidate sentence list generated based on any one or any combination of the feature vector and the second language sentence.
The automated interpretation system may further include a voice recognizer configured to generate the first language sentence by decoding a first feature vector generated by recognition encoding a voice signal in the first language.
The automated interpretation system may further include a memory having a database, and the processor may be configured to acquire any one or any combination of a first translation result matching a second language feature vector, from the database, determined similar to the feature vector and a second translation result matching a sentence, from the database, determined similar to the second language sentence.
The processor may be configured to add any previous translation sentences corresponding to any of the first translation result and the second translation result to the candidate sentence list, and may be configured to add the second language sentence to the candidate sentence list.
The translator may be configured to calculate scores of candidate sentences included in the candidate sentence list based on the feature vector, and the processor may be configured to select, from the candidate sentence list, a candidate sentence having a highest of the calculated scores to be the final second language sentence.
The processor may be configured to divide the first language sentence into a plurality of sub-words, configured to sequentially input input vectors respectively indicating the plurality of sub-words to an encoder used for machine translation, and configured to determine the feature vector to be an output from the encoder for the sequentially input input vectors.
The processor may be configured to input the feature vector to a decoder used for machine translation, configured to generate a predetermined number of sentence sequences based on probabilities of sub-words sequentially output from the decoder, and configured to select a sentence sequence having a highest score among the predetermined number of sentence sequences to be the second language sentence.
The processor may be configured to store the feature vector in a database, along with the first language sentence stored in the database, and may be configured to store any one or any combination of the second language sentence and the final second language sentence corresponding to the first language sentence and the feature vector in the database.
In one general aspect, an automated interpretation system includes one or more processors configured to perform voice recognition of an input voice signal, perform an initial translation of a recognition result of the voice recognition, and compare results of the initial translation and previous results of select previous translations to determine a final translation of the input voice signal, with the comparison including comparing information for one or more of information derived in the voice recognition and information derived in the initial translation with information stored in a database for one or more previous sentence translation results to identify the select previous translations from plural previous translations whose information is recorded in the database.
The comparison may include comparing information for one or more of a first feature vector derived in the voice recognition, a first language sentence derived in the voice recognition, a second feature vector derived in the initial translation, and a second language sentence derived in the initial translation with information stored in the database to identify the select previous translations.
The one or more processors may be included in a same mobile device.
Other features and aspects will be apparent from the following detailed description, the drawings, and the claims.
Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same or like elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.
The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order. Also, descriptions of features that are known in the art may be omitted for increased clarity and conciseness.
The features described herein may be embodied in different forms, and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.
Terms such as first, second, A, B, (a), (b), and the like may be used herein to describe components. Each of these terminologies is not used to define an essence, order, or sequence of a corresponding component but used merely to distinguish the corresponding component from other component(s). For example, a first component may be referred to a second component, and similarly the second component may also be referred to as the first component.
It should be noted that if it is described in the specification that one component is “connected,” “coupled,” or “joined” to another component, a third component may be “connected,” “coupled,” and “joined” between the first and second components, although the first component may be directly connected, coupled or joined to the second component. In addition, it should be noted that if it is described in the specification that one component is “directly connected” or “directly joined” to another component, a third component may not be present therebetween. Likewise, expressions, for example, “between” and “immediately between” and “adjacent to” and “immediately adjacent to” may also be construed as described in the foregoing.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the,” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and/or “including,” when used herein, specify the presence of stated features, integers, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, operations, elements, components, and/or groups thereof.
Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains based on an understanding of the present disclosure. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The following example embodiments may be applicable to provide recognition and translation in/as an automobile, a television (TV), a mobile phone, and other electronic devices, depending on embodiment. Example embodiments may be implemented as various types of products such as a personal computer (PC), a laptop computer, a tablet computer, a smartphone, smart home appliances, a wearable device, and the like. In examples, embodiments include non-transitory computer readable media including interpretation application(s), instructions, or software to be executed in/by one or more processors of such a smartphone, mobile device, smart home system, wearable device, and the like embodiments. Example embodiments include a global conferencing hardware, or are configured to provide a translated transcription of audio and/or video conferences, or a corresponding method or non-transitory computer readable media causing one or more processors to be configured to implement the same. Also, example embodiments may include providing interpretations for communications between a driver and a passenger of a vehicle, such as public transportation automobiles, busses, trains, escalators, or people movers, as only examples, or other announcement or public auditory statements. Hereinafter, such non-limiting example embodiments will be described in more detail with reference to the accompanying drawings. Like reference numerals in the drawings denote like elements.
The user 110 requests the automated interpretation apparatus 130 to interpret a voice signal A expressed by the first language using the second language. In this example, the voice signal A may be a voice signal of the first language. The user 110 may interact with a user interface of the automated interpretation apparatus 130 to request the recognition and/or translation, or the recondition and/or translation operations of the automated interpretation apparatus 130 may automatically or continuously operate, e.g., in a background operation of the underlying device, or the user 110 may selectively implement both/either the user interface request and the automatic implementations.
When the voice signal A is input to the automated interpretation apparatus 130 in operation 101, the agent 133 of the automated interpretation apparatus 130 recognizes the voice signal A and generates a sentence A of the first language. As noted above, the agent 133 may include hardware to convert the audible voice into a digital signal, for example the agent 133 may include one or microphones, ADCs, and parsers, or any or any combination of the microphone, ADCs, and parsers may be external of or included elsewhere in the automated interpretation apparatus 100. The agent 133 may recognize the voice signal A by providing the corresponding audio frames to one or more of the example recognition models of the agent 133, such as the acoustic and/or language models, and by decoding the results of the recognition model(s) as the sentence A of the first language. The sentence A may be finalized as text-type data, for example. As only examples, the recognition model(s) may be respective trained neuro networks. In operation 102, the automated interpretation apparatus 130 requests the translator 136 to translate the sentence A.
In operation 103, the translator 136 generates a sentence A′ of the second language as an initial translation result of the sentence A and provides the results of the translation, e.g., sentence A′, back to the agent 133. The agent 133 provides a sentence B′ and a sentence C′ of the second language, e.g., previously stored in a database, to the translator 136 such that the translator 136 selects an optimal translation result from the sentence A′, the sentence B′, and the sentence C′ in operation 104. Here, agent 133 may determine that both sentences B′ and C′ are associated with respective speech similar to that of the voice signal A, e.g., from among other multiple sentences. The sentence B′ and the sentence C′ may be previously stored in the database as a result of previous interpreting operations by the automated interpretation apparatus 130, as only a non-limiting example. For example, when an interpretation request for a voice signal B was previously processed, the automated interpretation apparatus 130 may have generated the sentence B′ and stored the generated sentence B′ in the database. Also, the sentence C′ may have been stored in the database in another previous process of interpreting a voice signal C. In an example, such sentences may be stored in one or more databases in a categorical or searchable manner, as only examples, so the agent 133 can associate previously stored sentences with information about a currently translated sentence and forward information of the same to the translator 136 for operation 104, for example.
In operation 105, the translator 136 selects an optimal translation result, among sentence A′, sentence B′, and sentence C′ determined to have been acquired based on the determined similar speeches, to be the final translation result of the sentence A and transfers the final translation result back to the agent 133. For example, the translator 136 determines that the sentence B′ is a better translation result for voice signal A in comparison to the initial translation result of sentence A′.
An optimal translation result for speech of a first language from candidate sentences of the second language may be more readily provided compared to a performance of only directly translating the content A of the first language into that of the second language. In addition, when the candidate sentences are determined to be candidates based on their determined relatedness to the speech in the first language and/or relatedness to an initial translation of the speech in the first language, with such relatedness determinations being based on various considerations, a final translation result for the speech of the first language may be more robust and accurate than mere singular or direct translations of the speech of the first language into the second language using the translation model implemented by the translator 136, e.g., the translation model that is implemented to derive the initial translation of the speech of the first language. Thus, in one or more embodiments, the translator 136 may collectively consider a current translation result and translation results accumulated in previous translation processes and selects one of the collectively considered translation results having a highest determined score to be a final translation result, thereby providing a more robust and accurate translation result for the speech in the first language.
In operation 106, the agent 133 transfers a final interpretation result to the user 110. The agent 133 may provide the final interpretation result to the user 110 in a form of text in the second language, such as through a display of the automated interpretation apparatus 130. In an example, the agent 133 may provide the final interpretation result to the user 110 audibly through a voice synthetization process of the agent 133 and a speaker of the automated interpretation apparatus 130. For example, the agent 133 may provide the final interpretation result to the user 110 in a form of voice in the second language based on text to speed (TTS) technology of the agent 133.
Depending on embodiment, the agent 133 and the translator 136 may each be implemented in or through a user terminal or in a server, such as a remote server. The agent 133 and the translator 136 may operate in the user terminal such as a smartphone, as only an example. The agent 133 and the translator 136 may also or alternatively operate in the server, such as by receiving either voice recordings of speech or audio frames from a remote terminal. The server implementation may also consider candidate sentences from different local and/or remote terminals when determining the best translation for input speech. Also, the agent 133 may operate in the user terminal and the translator 136 may operate in the server, e.g., with the user terminal forwarding recognition results to the translator 136 and the translator 136 returning results to the agent 133 in accordance with operations of
In an example, a configuration and operation of an automated interpretation apparatus 130 configured to perform both recognition and translation will be described with reference to
Referring to
The voice recognizer 210 may include an encoder 211 and a decoder 213 for voice recognition. The translator 230 may include an encoder 231 and a decoder 233 for machine translation.
The automated interpretation apparatus 200 may collect operative results and/or outputs of the encoder 211 and the decoder 213, included in the voice recognizer 210, and operative results and/or outputs of the encoder 231 and the decoder 233, included in the translator 230, and may store the such operative respective results and/or outputs in the database 250 while performing respective recognition and/or translation operations. The operative results and/or outputs of the voice recognizer may include, for example, abstracted voice information of an input voice signal, e.g., separated or extracted voice, acoustic, phoneme, morpheme, syntax, and/or context information from sampled voice frames, as only examples, and a voice recognition result for the input voice signal. The operative results and/or outputs of the translator 230 may include, for example, abstracted sentence information for the voice recognition result, e.g., such separated or extracted information for translation considerations, and a translation result for the voice recognition result based on the abstracted sentence information. Here, these examples of abstracted voice information and abstracted sentence information are only non-limiting examples, as other voice signal, acoustic information, and sentence or contextual information may be separated or extracted by operations of each of the voice recognizer 210 and translator 230, for example.
For example, when a voice signal of a user saying “?” is input to, or received by, the automated interpretation apparatus 200, the automated interpretation apparatus 200 provides features extracted from the voice signal to the voice recognizer 210 and acquires a voice recognition result from the voice recognizer 210. The voice recognition result may be a first language sentence corresponding to a voice of the user, which in this example may be a voice recognition in the Korean language.
The automated interpretation apparatus 200 provides the voice recognition result to the translator 230 and acquires an initial translation result from the translator 230. For example, the initial translation result of the voice recognition may be “I'll go to Gangnam?”. In addition to “I'll go to Gangnam?”, various other initial translation results of the translator 230 may also be generated. For example, one or more various initial translation results whose determined accuracy scores are above a first threshold may be selected. The automated interpretation apparatus 200 selects the initial translation result output by the decoder 233 of the translator 230 to be a candidate translation or a candidate sentence. The candidate translation is stored in, for example, an n-best candidate sentence list.
The automated interpretation apparatus 200 searches the database 250 for information elements similar to the operative results and/or outputs of the encoder 211, the encoder 231, the decoder 213, and the decoder 233 generated in the processes of recognizing and translating the input “?” voice signal. The searched information elements may be one of stored abstracted voice information, stored the voice recognition results, stored abstracted sentence information, and stored the translation results, e.g., from previous interpretation operations. In an example, the automated interpretation apparatus 200 may search the database 250 for final translation sentences, e.g., stored as the illustrated translation results in the database 250, matching or corresponding to the searched information elements that are found to be similar. For example, based on the found/determined similar information elements, the stored previous final translations of “How do I get to length Gangnam?” and “How do I get to Gangnam Station?” may be identified and added to the n-best candidate translations list, as candidate translations.
As such, the n-best candidate sentence list includes a sentence corresponding to the initial translation result and one or more other translation sentences acquired from the database 250. For example, the candidate sentence list may include a current translation result and previous translation results. Thus, in the present example, the candidate sentence list includes the sentences, “I'll go to Gangnam?”, “How do I get to length Gangnam?”, and “How do I get to Gangnam Station?”.
The automated interpretation apparatus 200 scores each of the candidate sentence list to which the final sentences corresponding to similar translation results are added to acquire a final score for each of the candidate sentences. The automated interpretation apparatus 200 may calculate the respective final scores for the candidate sentences based on speech corresponding to a current interpretation target, i.e., the speech currently being interpreted. Here, the automated interpretation apparatus may have previously respectively scored any of the candidate sentences from the stored previous translation results, i.e., when performing their respective final translation operations for their respective previous interpretation targets. Accordingly, herein, when performing interpretation operations for a current interpretation target the automated interpretation apparatus 200 may be considered as recalculating or rescoring, i.e., calculating or scoring again, such previous translation results, but this time based on the current interpretation target. For example, the automated interpretation apparatus 200 may use the decoder 233 to recalculate the final scores for each of the candidate sentences based on the current interpretation target. In this example, the decoder 233 may assign a higher score to a translation result previously stored in the database 250 in comparison to the initial translation result obtained in a current translation process.
As another example, and as will be discussed in greater detail further below,
In such examples, rescoring may be performed based on probability values given for each word (or generated as a result of a translation model, for example) in a process of decoding speech corresponding to the interpretation target. However, depending on embodiment, differing rescoring schemes may be employed, such as by replacing or interpolating, into a form of a weighted sum, a word probability value of another language model or an n-gram-based probability value (in consideration of a domain, a user, a nation, etc.), as only examples.
Thus, returning to
A scenario of translating voice signals using the automated interpretation apparatus 200 will be further described with reference to the bellow Table 1, for example.
?
?
?
When a voice signal 1 “?” is input to the automated interpretation apparatus 200, an initial translation result “How do I get to Gangnam?” of the translator 230 is determined to be a candidate translation, for example, in the ‘n-best’ column of Table 1 and then, stored in the candidate sentence list for the voice signal 1 because prestored information elements may be absent in the database 250, e.g., as other translations may not have been previously performed so there may be no initially stored information elements to search through database 250. In this example, the initial translation result may be provided as a final translation result of the automated interpretation apparatus 200.
When a voice signal 2 “?” is later input, the automated interpretation apparatus 200 generates an initial translation result “Tell us delicious jajangmyen home.” using the translator 230 and determines the initial translation result to be a candidate translation for voice signal 2. The automated interpretation apparatus 200 verifies whether an information element similar to the interpretation outputs exists in the database 250 in an initial translation process. For example, the automated interpretation apparatus 200 verifies whether any information elements similar to the abstracted voice information, the voice recognition result, the abstracted sentence information, and the translation result for voice signal 2 are present in the database 250. Because the database 250 does not yet include any information elements similar to the current voice signal 2 generated interpretation process outputs, no further candidate translations are additionally selected. Thus, in this example, the initial translation result is also provided as the final translation result of the automated interpretation apparatus 200 for voice signal 2.
When a voice signal 3 “?” is input, the automated interpretation apparatus 200 generates an initial translation result “How do I get to length Gangnam?” in the translator 230 and determines the initial translation result to be a candidate translation for voice signal 3. The automated interpretation apparatus 200 searches the database 250 for additional candidate translations. For example, the automated interpretation apparatus 200 may search the abstracted voice information, the voice recognition results, the abstracted sentence information, and the translation results from interpretation results for either or both of the voice signal 1 and voice signal 2. The automated interpretation apparatus 200 may search such previous interpretation results for information elements similar to the results or outputs generated in the current interpretation process of voice signal 3. Then, the previous translation sentence results corresponding any of the found matching or similar information in the abstracted voice information, the voice recognition results, the abstracted sentence information, and the translation results of database 250 are added to the candidate translations list. For example, the automated interpretation apparatus 200 may add a previous final translation sentence “How do I get to Gangnam?” to the candidate translation list, as a candidate translation, after determining that there are similarities between information elements for the current voice signal 3 and one or more stored information elements corresponding to the voice signal 1. In this example, the candidate sentence list includes the candidate translations “How do I get to length Gangnam?” and “How do I get to Gangnam?”. The automated interpretation apparatus 200 calculates scores of the candidate translations included in the candidate sentence list with respect to the voice signal 3, and selects the candidate translation, for example, “How do I get to Gangnam?”, as having a highest score from the candidate translations, to be the final translation result for the voice signal 3.
When a voice signal 4 “?” is input, the automated interpretation apparatus 200 generates an initial translation result “I'll go to Gangnam?” and determines the initial translation result to be a candidate translation for voice signal 4. The automated interpretation apparatus 200 searches for similar information elements in the stored interpretation results for voice signal 1, the stored interpretation results for voice signal 2, and the stored interpretation results for voice signal 3, in the database 250, based on the results and outputs generated up through the initial translation process in the current interpretation process for voice signal 4. Based on results of these searches the automated interpretation apparatus 200 determines that the final translation sentences “How do I get to Gangnam” and “How do I get to length Gangnam?”, respectively corresponding to the translation results of the voice signal 1 and the voice signal 2, are also candidate translations for voice signal 4. In this example, one of these determined candidate translations, for example, “How do I get to Gangnam?”, having a determined highest score among all of the determined candidate translations, e.g., as included in a candidate sentence list for voice signal 4, is selected to be the final translation result for voice signal 4.
When searching the database 250 for the information element or similar information elements, the automated interpretation apparatus 200 may use various algorithms separately or in combination. For example, the automated interpretation apparatus 200 may use an approximate k-nearest neighbor (k-NN) algorithm, or respective such algorithms, to quickly determine or retrieve from the database 250 information elements that are similar to the outputs of the encoder 211, the encoder 231, the decoder 213, and the decoder 233 from a current interpretation process. In an example, the automated interpretation apparatus 200 may also, or alternatively, use a locality sensitive hashing (LSH) algorithm and/or a greedy filtering algorithm for comparing outputs of the encoder 211, encoder 231, the decoder 213, and decoder 233 to information stored in the database 250 to identify additional candidate translations from previous interpretation operations. In addition, results of such searching of the database 250 may also be stored in the database 250 and available for refining current or future searches of the database 250.
In addition, or alternatively, the comparison by the automated interpretation apparatus 200 between outputs of the encoder 211, encoder 231, the decoder 213, and decoder 233 to information in the database 250 may include a method of determining whether a similarity exists between feature vectors, for example, a first feature vector and/or a second feature vector determined in a current interpretation process, and feature vectors in the database 250. For example, the abstracted voice information, the recognition results, and the abstracted sentence information may have been converted into respective information in high-dimensional vector forms, and respectively stored in the database 250. In an example, where the encoder 211 and decoder 213, as only example, include a neural network, e.g., recurrent neural network, and implement neural network learning to process voice recognition, a vector form generated as an intermediate result or a phoneme sequence may be a by-product of the neural network voice recognition process.
Here, as only another example, the automated interpretation apparatus 200 may calculate Euclidean distances between feature vectors stored in the database 250 and one or more feature vectors generated in current recognition or translation processes. Hereinafter, a feature vector generated in the current recognition or translation process is also referred to as a target feature vector. As only an example, the similarity determinations or comparisons for determining the candidate translations may be based on a set principle that a similarity between a target feature vector and a feature vector stored in the database 250 increases according to a decrease in a Euclidean distance therebetween, alternatively or in addition similarity determinations or comparisons may be based on a set principle that similarity decreases according to an increase in the Euclidean distance.
In addition, or alternatively, the automated interpretation apparatus 200 may determine whether the target feature vector is similar to the feature vectors stored in the database 250 based on a determined cosine similarity. For example the automated interpretation apparatus 200 may determine that a similarity between the target feature vector and the feature vector stored in the database 250 increases as the cosine similarity between the target feature vector and the feature vector stored in the database 250 is closer to ‘1’. Since a threshold for verifying the similarity may not easily be determined, the automated interpretation apparatus 200 may arrange the feature vectors stored in the database 250 in a descending order of similarity and determine that feature vectors corresponding to a predetermined percentage (%) in a highest (similarity) ranking or a predetermined number of the highest (ascended) arranged feature vectors are similar to the target feature vector and then store or identify the corresponding previous translation results for those determined feature vectors in the candidate translation list.
As only an example, a method of determining whether a similarity between sentences, for example, a first language sentence and a second language sentence is present may include the automated interpretation apparatus 200 determining whether the similarity between the sentences is present based on a term frequency-inverse document frequency (TF-IDF). The TF-IDF may be a statistic value indicating a frequency or an importance of a predetermined word in a document among a document group including a plurality of documents. A term frequency (TF) is a value indicating a frequency of a predetermined word in a document. Also, it is determined that an importance of the word in the document increases according to an increase in the value. When the word is frequently used through the document group overall, it is indicated that the word is commonly used. The foregoing example may be indicated by a document frequency (DF), and an inverse number of the DF may be an inverse document frequency (IDF). The TF-IDF may be a value obtained by multiplying the TF by the IDF.
In this example, the automated interpretation apparatus 200 converts a sentence into a vector based on the TF-IDF and compares similarities between vectors. Through this, the automated interpretation apparatus 200 determines whether a sentence generated in the current recognition or translation process is similar to sentences stored in the database 250. The sentence generated in the current recognition or translation process may also be referred to as a target sentence. Since a threshold for verifying the similarity may not be easily determined, the automated interpretation apparatus 200 may arrange the sentences stored in the database 250 in a descending order based on their similarity and determine that sentences corresponding to a predetermined percentage in a highest rank or a predetermined number of the highest arranged sentences are similar to the target sentence and then store or identify the corresponding previous translation results for those feature vectors in the candidate translation list.
Although not shown, features extracted from a voice signal may be additionally stored in the database 250 and the extracted feature may be additionally used to select the candidate translations, for example, the n-best candidate translations. For example, using feature vectors generated during the example neural network voice recognition process, similarity between two voice signals of different lengths or of changed lengths between speeches may be determined. In this example, similar portions may be compared to each other or a dynamic time warping scheme may be performed on a whole sequence so as to obtain a transformation between two sequences. Through this, a similarity between voice sequences having different lengths, for example, may be verified, and that similarity may be used to select candidate translations.
Referring to
In
An encoder 211 of the voice recognizer 210 may include a neural network 212, and a decoder 213 of the voice recognizer 210 may include a neural network 214. Also, an encoder 231 of the translator 230 may include a neural network 232, and a decoder 233 of the translator 230 may include a neural network 234. The encoder 211, decoder 213, encoder 231, and/or decoder 233 of
When the encoder 211, the encoder 231, the decoder 213, and the decoder 233 are each configured with a neural network, a learning process may be performed with each of the encoder 211, the encoder 231, the decoder 213, and the decoder 233 in advance to interpreting or translating a current voice input. In this example, operations of training the encoder 211, the encoder 231, the decoder 213, and the decoder 233 may be understood as an operation of determining weights or parameters of the neural network through learning operations. The learning operations may respectively be performed during manufacture and/or post manufacture using training data, and may also be available to be updated during operation of the automated interpretation apparatus
In an example, in response to an input of a voice signal of the first language “?”, the automated interpretation apparatus extracts features from the voice signal. A method of extracting features from the voice signal using the automated interpretation apparatus will be described in more detail with reference to
In response to an input of the features extracted from the voice signal, the encoder 211 encodes the extracted features and generates a first feature vector, for example, a real number vector of {‘2.542’, ‘0.827’, . . . , ‘5.936’}. The decoder 213 decodes the first feature vector generated by the encoder 211 and generates a first language sentence, for example, a sentence “?” as a voice recognition result. The decoder 213 outputs the first language sentence sub-word or word units. The sub-word may be understood as a sequence of characters frequently used in a common sentence, such as phonemes or syllables, as only examples. The neural network 212 of the encoder 211 and the neural network 214 of the decoder 213 will be described in greater detail with reference to
The decoder 213 decodes the first feature vector and generates an m-best list including m candidate sentences of the first language. The decoder 213 generates the m-best list of the first language based on, for example, a beam search algorithm. In this example, m may be a complexity of the beam search algorithm. The m-best list includes sentences corresponding to candidates for the voice recognition, in contrast to the n-best list of sentence (and/or phrase) candidates described with reference to
The m-best list includes first language sentences, for example, “?”, “
?”, “
”, and “
?”. Each sentence (and/or phrase) included in the m-best list also has a score or a probability value, for example, 0.6, 0.05, 0.2, and 0.1 that may be stored along with the sentence.
The voice recognizer 210 further includes a rescoring model 215. The rescoring model 215 scores the sentences or ranks the sentences based on their scores. The rescoring model 215 outputs one best sentence (or phrase) among the m sentences as a result of the voice recognition.
The encoder 231 of the translator 230 encodes the first language sentence “?” and generates a second feature vector. The encoder 231 uses the neural network 232 to encode the first language sentence to be the second feature vector.
The decoder 233 decodes the second feature vector and generates m-best list of the second language including m candidate sentences of the second language. The m-best list includes sentences (and/or phrases) corresponding to candidates for an initial translation, in contrast to the n-best list of sentence (and/or phrase) candidates described with reference to
In one example, an input dimension of the encoder 231 may be a dimension of a dictionary including sub-words of the first language and an output dimension of the decoder 233 may be a dimension of a dictionary including sub-words of the second language. The dimension of the dictionary may be the number of the sub-words included in the dictionary. An example configuration and operation of the neural network 232 included in the encoder 231 and an example configuration and operation of the neural network 234 included in the decoder 233 will be described with reference to
The translator 230 may further include a rescoring model 235. For example, the rescoring model 235 selects a final sentence based on a value of probability or confidence that an original sentence for each candidate sentence is correctly translated into a translated sentence and an average of values of probabilities or confidences that the translated sentence is correctly translated into the original sentence. Also, the rescoring model 235 determines scores calculated in a process of decoding candidate sentences of the second language to be scores of the candidate sentences of the second language. For example, the score for each candidate sentence may be a value of probability or confidence that the original sentence was correctly translated into the corresponding candidate sentence. In this example, the rescoring model 235 may also be referred to as a ranking model.
Though not limited thereto, the rescoring model 235 may output only one best or highest scoring sentence among the m sentences. For example, the rescoring model 235 may output a candidate sentence “I'll go to Gangnam?” corresponding to the highest score, for example, 0.5 as an initial translation result.
The illustrated example input vectors x1, x2, . . . , xi, . . . , xL, e.g., generated corresponding to respective frames in
Thus, in an example, decoder portion of the neural network 610 may sequentially outputs sub-words y1, y2, . . . , yi, . . . , yL included in a first language sentence. The output sub-words may then be re-input to the decoder portion of the neural network 610 so as to then be used for recognizing a subsequent sub-word, e.g., as a temporal feedback. The decoder portion of the neural network 610 may generate a predetermined number of sentence (and/or phrase) sequences and may select a sentence sequence (or phrase) having a highest set score among the number of sequences to be a first language sentence, e.g., the sentence “?”. The recognized first language sentence corresponds to a voice recognition result and is stored in a database, which may be the same database the output voice abstract information is stored in, or may be a different database.
The neural network 232 of ’, a binary ‘1’ is input to a node corresponding to ‘
’ among a plurality of nodes of the input layer 810 and a binary ‘0’ is input to remaining nodes. A one-hot vector input to the input layer 810, for example, a vector to which the binary ‘1’ is mapped to correspond to ‘
’ is propagated through the hidden layer 820 to the output layer 830.
The neural network 232 of the encoder of ’, a one-hot vector to which a binary ‘1’ is mapped to correspond to ‘
’ is input to the input layer 810. In this example, when the one-hot vector is propagated through the hidden layer 820 to the output layer 830, nodes included in the hidden layer 820 additionally receive outputs of nodes having propagated a first sub-word in the hidden layer 820.
As such, a feature vector finally output in response to the sequential input of the sub-words of the first language to the input layer 810 of the neural network 232 is generated. The generated feature vector corresponds to abstracted sentence information and may be stored in a separate database, such as the database 250 of
The neural network 234 of the decoder of
The input layer 850 of the neural network 234 receives the feature vector finally output from the encoder. The feature vector is propagated through the hidden layer 860 to the output layer 870. A dimension of the output layer 870 corresponds to a dimension of a dictionary including sub-words of the second language. Each node of the output layer 870 may correspond to a sub-word of the second language, and an output value of each node may indicate a probability or confidence that the sub-word of the corresponding node is a correctly translated output. In an example, the automated interpretation apparatus performing the beam search algorithm selects a predetermined number of, for example, three candidate sub-words and the number being determined by the decoder in a descending order. For example, three sub-words, from among 30000 sub-words, corresponding to the top three scores or probabilities P1-1, P1-2, and P1-3 are propagated to a subsequent operation.
When the candidate sub-words are selected, a subsequent candidate sub-word is decoded to correspond to each of the candidate sub-words. In this example, similarly to the neural network 232 of the encoder of
In the aforementioned method, a sequence of the candidate sub-words is generated and a candidate sentence (or phrase) of the second language is constructed based on the sequence of the candidate sub-words. When the predetermined number of, for example, three candidate sub-words are selected every time that one sub-word is decoded, a number of final candidate sentences exponentially increases. To prevent or minimize such phenomenon, in one or more embodiments pruning may be applied to each such decoding process. The pruning is a selective removal performed to maintain the predetermined number of candidate sentences. For example, through the pruning, three candidate sentences are selected from nine candidate sentences generated by decoding up to the second sub-word and propagated to a subsequent process.
When a sub-word is selected in one process, a hidden layer of a subsequent process is changed by the selected sub-word. For example, an embedding vector indicating the selected sub-word may be applied to an internal status of nodes included in the hidden layer of the subsequent process.
Candidate sentences are obtained as translation results corresponding to different/previous speeches and thus, the scores of the candidate sentences are to be recalculated, i.e., calculated again, based on a speech corresponding to a current interpretation target. In one example, scores of the candidate sentences are recalculated based on abstracted sentence information previously generated to correspond to a speech that is a current translation target. A process of rescoring a first candidate sentence, for example, the above example “I'll go to Gangnam” of Table 1 in the example second language for a subsequent speech signal is described in more detail below.
Abstracted sentence information input to an input layer of the neural network 234 is propagated through a hidden layer to an output layer. Nodes included in the output layer correspond to sub-words, for example, “Gangnam”, “I”, and “will” (corresponding to the “'II” of “I'll”) of the second language. An output value of each of the nodes indicates a probability or confidence that a sub-word of the corresponding node is translated correctly and should be output.
When a first sub-word of a first candidate sentence is “I”, an automated interpretation apparatus selects a probability P1-1 output in a node corresponding to “I” to calculate a score of the first candidate sentence. When the node corresponding to the first sub-word is selected, the output layer of the neural network 234 outputs probabilities of second sub-words. As such, by sequentially selecting nodes in the output layer based on sub-words included in the first candidate sentence, the automated interpretation apparatus calculates the score of the first candidate sentence.
Based on the aforementioned method, the automated interpretation apparatus recalculates the scores of the candidate sentences. The automated interpretation apparatus selects a candidate sentence having a highest score among the scores recalculated on the first candidate sentence through an nth candidate sentence to be a final translation sentence (phrase) for the current input speech signal, for example.
The automated interpretation apparatus stores the first feature vector, the first language sentence, and the second feature vector from a current interpretation process in a database, such as the database 250 of
The operations of
The machine translation apparatus stores the feature vector in a database. Also, the machine translation apparatus stores one or both of the second language sentence and the final second language sentence matching/corresponding to the feature vector in the database.
Herein, though the interpretation processes have been discussed with respect to translation of information in a first language to sentences or phrases in a second language, embodiments are not limited thereto. In one or more examples, the illustrated translators in the automated interpretation apparatus or machine translation apparatus may be representative of plural translators, each translators being configured as discussed to translate sentence information from the first language or another language into the second language or other languages, e.g., other than the above English second language examples. Multiple different translation processes may also be performed selectively and/or simultaneously. In addition, the different translation processes may further be automatically or selectively performed as automatic background processes of the underlying device to provide results of such translation operations to a user when/if the user desires or the underlying interactive agent of the device determines the user would need or desire.
The memory 1910 includes a volatile memory and a non-volatile memory to store information received through the bus 1950, for example. The memory 1910 includes a database configured to store a first feature vector, a first language sentence, a second feature vector, and a second language sentence generated in a process of automated interpretation, such as in any or any combination of processes discussed above with respect to
The processor 1920 may perform an operation of the agent 133 described with reference to
The processor 1920 may acquire candidate sentences from the database of the memory 1910 based on any one or any combination of the first feature vector, the first language sentence, the second feature vector, and the second language sentence of a current recognition and/or translation operation, for example. The processor 1920 may transfer the candidate sentences and the second feature vector to a decoder of the translator 1940 and receive scores of the candidate sentences from the decoder of the translator 1940, e.g., as determined or calculated by the decoder of the translator 1940. The processor 1920 may also select a final sentence from the candidate sentences based on the scores.
In one example, the voice recognizer 1930 and the translator 1940 are implemented independently of the processor 1920. In this example, the voice recognizer 1930 and the translator 1940 are implemented using processor or computing resources differing from the processor 1920, and may be implemented by the same processor or computing resources or by different processor or computing resources. In addition, in an embodiment, the voice recognizer 1930 and the translator 1940 are located external or remote from the respective automated interpretation apparatuses 1900 and communicate with the respective automated interpretation apparatuses 1900 through a wired or wireless network, for example. The user interface 1960 illustrated in
In an example, the voice recognizer 1930 and the translator 1940 are implemented through the processor 1920 and the memory 1910, such as through recognition and translation modeling. For example, one or more neural networks included in the voice recognizer 1930, including an example where respective neural networks are included in an encoder and a decoder of the voice recognizer 1930, and/or one or more neural networks included in the translator 1940, including an example where respective neural networks are included in an encoder and a decoder of the translator 1940, may be stored in the memory 1910. In an example, each of the neural networks may be stored in the memory 1910 in a form of executable object file or execution file, as only examples. In addition, parameters for each of the neural networks may also be stored in the memory 1910. In such examples, the processor 1920 loads the neural networks from the memory 1910 and applies the parameters for each of the neural networks, thereby implementing recognition of the voice recognizer 1930 and translation of the translator 1940. In another example, the processor 1920 loads the neural networks from the memory 1910 and applies the parameters for each of the neural networks, thereby implementing the encoder and the decoder of the voice recognizer 1930 and the encoder and the decoder of the translator 1940.
In another example, the processor 1920 may encode frames of the sampled voice signal in a first language and generate the first feature vector with respect to the first language. The processor 1920 may then decode the first feature vector and generate the first language sentence in the first language. The processor 1920 may encode the first language sentence with respect to a second language and generate the second feature vector with respect to the second language. The processor 1920 may then decode the second language vector and generate the second language sentence in the second language. The processor 1920 may select a final second language sentence from a candidate sentence list, e.g., generated by the processor 1920 based on any one or any combination of the first feature vector, the first language sentence, the second feature vector, and the second language sentence. Referenced outputs or results generated in voice recognition processes and referenced outputs or results generated in machine translation processes may be transferred to the memory 1910. In addition, though embodiments may discuss that any of such outputs or generated results may be transferred between the processor 1920, the voice recognizer 1930, and/or the translator 1940, embodiments also include the respective processor 1920, voice recognizer 1930, and/or translator 1940 storing their respective outputs or results to local caches, the memory 1910, or any other memories so as to be available for acquiring or requesting from such local caches, the memory 1910, or other memories by any of the processor 1920, voice recognizer 1930, and/or translator 1940.
The agent 133, translator 136, automated interpretation apparatus 130, automated interpretation apparatus 200, voice recognizer 210, encoder 211, decoder 213, translator 230, encoder 231, decoder 233, database 250, rescoring model 215, rescoring model 235, neural network 212, neural network 214, neural network 232, neural network 234, machine translation apparatus 300, translator 310, database 330, neural network 610, neural network 710, automated interpretation apparatuses 1900, memories 1910, processors 1920, voice recognizer 1930, translator 1940, bus 1950, user interface 1960, and display 1970 in
In one or more embodiments, the methods and processes illustrated in
Instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler. In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computer using an interpreter. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.
The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computers.
As non-exhaustive examples only, and in differing embodiments, an automated interpretation apparatus as described herein may be a mobile device, such as a cellular phone, a smart phone, a wearable smart device (such as a ring, a watch, a pair of glasses, a bracelet, an ankle bracelet, a belt, a necklace, an earring, a headband, a helmet, or a device embedded in clothing), a portable personal computer (PC) (such as a laptop, a notebook, a subnotebook, a netbook, or an ultra-mobile PC (UMPC), a tablet PC (tablet), a phablet, a personal digital assistant (PDA), a digital camera, a portable game console, an MP3 player, a portable/personal multimedia player (PMP), a handheld e-book, a global positioning system (GPS) navigation device, or a stationary device, such as a desktop PC, a high-definition television (HDTV), a DVD player, a Blu-ray player, a set-top box, or a home appliance, or any other mobile or stationary device configured to perform wireless or network communication. For example, such automated interpretation discussed herein may be implemented in hardware, such as a mobile device, television, or PC, implementing video conferencing, such as to output and display subtitles in real time with a concurrent video conference. The automated interpretation apparatus or system according to one or more embodiments may be a vehicle, a public transportation kiosk or interface, or other user interface. In another example, a mobile device according to one or more embodiments may be configured to automatically interpret public announcements, such as in public transportation systems or audible public warning systems. In one example, a wearable device is a device that is designed to be mountable directly on the body of the user, such as a pair of glasses or a bracelet. In another example, a wearable device is any device that is mounted on the body of the user using an attaching device, such as a smart phone or a tablet attached to the arm of a user using an armband, or hung around the neck of the user using a lanyard. These examples are for demonstrative purposes and should not be interpreted as limiting of application or implementation of the automated interpretation apparatus or system.
While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application t that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2016-0086036 | Jul 2016 | KR | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5809461 | Inazumi | Sep 1998 | A |
| 7224981 | Deisher et al. | May 2007 | B2 |
| 7697760 | Kwok et al. | Apr 2010 | B2 |
| 7769580 | Gemello et al. | Aug 2010 | B2 |
| 7904294 | Rose et al. | Mar 2011 | B1 |
| 8078469 | White et al. | Dec 2011 | B2 |
| 8296123 | Thayer et al. | Oct 2012 | B2 |
| 8352243 | Jang et al. | Jan 2013 | B2 |
| 8775177 | Heigold | Jul 2014 | B1 |
| 8775181 | Ehsani et al. | Jul 2014 | B2 |
| 8930194 | Newman et al. | Jan 2015 | B2 |
| 9704257 | Tuzel | Jul 2017 | B1 |
| 20010029449 | Tsurufuji | Oct 2001 | A1 |
| 20040148170 | Acero | Jul 2004 | A1 |
| 20050060160 | Roh | Mar 2005 | A1 |
| 20060173684 | Fischer | Aug 2006 | A1 |
| 20060271370 | Li | Nov 2006 | A1 |
| 20070203688 | Fuji | Aug 2007 | A1 |
| 20080027705 | Koga | Jan 2008 | A1 |
| 20080077391 | Chino | Mar 2008 | A1 |
| 20080086300 | Anismovich | Apr 2008 | A1 |
| 20080133245 | Proulx | Jun 2008 | A1 |
| 20080154577 | Kim et al. | Jun 2008 | A1 |
| 20080288474 | Chin | Nov 2008 | A1 |
| 20090171662 | Huang | Jul 2009 | A1 |
| 20090182549 | Anisimovich | Jul 2009 | A1 |
| 20100088096 | Parsons | Apr 2010 | A1 |
| 20110046941 | Manuel-Devados | Feb 2011 | A1 |
| 20110097693 | Crawford | Apr 2011 | A1 |
| 20120078607 | Sumita | Mar 2012 | A1 |
| 20120323554 | Hopkins et al. | Dec 2012 | A1 |
| 20130103382 | Kim | Apr 2013 | A1 |
| 20130103695 | Rarrick | Apr 2013 | A1 |
| 20130138589 | Yu et al. | May 2013 | A1 |
| 20140229158 | Zweig | Aug 2014 | A1 |
| 20140330551 | Bao | Nov 2014 | A1 |
| 20150134336 | Huang | May 2015 | A1 |
| 20150227505 | Morimoto | Aug 2015 | A1 |
| 20160078020 | Sumita | Mar 2016 | A1 |
| 20160117316 | Le | Apr 2016 | A1 |
| 20160275667 | Modica | Sep 2016 | A1 |
| 20160350290 | Fujiwara | Dec 2016 | A1 |
| 20170031901 | Song | Feb 2017 | A1 |
| 20170228382 | Haviv | Aug 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 2 220 285 | Jan 1990 | GB |
| 2518318 | Mar 2015 | GB |
| 4756499 | Jun 2011 | JP |
| 10-0853173 | Aug 2008 | KR |
| 10-1134467 | Apr 2012 | KR |
| 10-2015-0069188 | Jun 2015 | KR |
| 10-1543992 | Aug 2015 | KR |
| Entry |
|---|
| Alkhouli, Tamer, et al. “Vector Space Models for Phrase-Based Machine Translation.” Syntax, Semantics and Structure in Statistical Translation (2014): 1-10. (10 pages, in English). |
| Cho, Kyunghyun, et al. “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches.” arXiv preprint arXiv:1409.1259 103-111 (2014). (9 pages, in English). |
| Hatakoshi, Yuto, et al. “Rule-Based Syntactic Preprocessing for Syntax-Based Machine Translation.” Syntax, Semantics and Structure in Statistical Translation (2014): 34-42. (9 pages, in English). |
| Singla, Karan, et al. “Reducing the Impact of Data Sparsity in Statistical Machine Translation.” Syntax, Semantics and Structure in Statistical Translation 51-56 (2014). (6 pages, in English). |
| Extended European Search Report dated Aug. 31, 2017 in counterpart European Application No. 17155528.7 (10 pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20180011843 A1 | Jan 2018 | US |
