Patent Grant
10606942
Speech translation devices serve the purpose of bridging the language barrier between two (or more) human beings who do not understand each others language. This involves an arrangement where at least one speech translation system in at least one language direction is constructed from the combination of (1) a speech recognition engine, (2) a machine translation engine, and (3) an output generation interface that presents the output to the listener in the other language (such output may be presented by synthetic speech, output text on linked web clients, heads-up displays, projectors, special audio-speakers, etc.). The purpose of such a system is to provide human-to-human communication, i.e. to transmit ideas—information expressed by one speaker, to the listener in an other language. Prior patent references that disclose speech translation system include: U.S. Pat. Nos. 5,712,957; 6,324,510; 6,963,837; 8,090,570; 8,204,739; U.S. Pub. No. 2007/0271088; U.S. Pub. No.
There are many situations where the information transmitted in the process of a dialog is further used and processed in various information processing tasks. Currently, such information typically has to be entered, added or reentered into a different system of a human-machine interface, and is thus wasteful and time-consuming. The recognized dialog information, however, could also be exploited directly to a human-machine interface that “listens” to the human-human conversation. This application discloses in one general aspect an apparatus that becomes a component of a speech-translation device, and extends its capabilities to inform and direct various ancillary machine services, implicitly or explicitly, but as a side effect of the human-to-human communication. In so doing, it saves time, and renders information gathering more effective and efficient.
For example, in a speech translation device used in medical missions, two speakers (a doctor and a patient) might be discussing ailments that the patient might be experiencing. In the course of such a dialog, the doctor may ask questions, such as: “Do you have any fever”, “any headache”, or “do you have any allergies”, that are all part of the diagnosis. Rather than entering the responses into a patient report or diagnosis and rather than writing down the resulting treatment, embodiments of the present invention can be used to generate such reports automatically, during the human-to-human dialog. Embodiments of the present invention can improve the effort of a knowledge worker by simplifying and unifying the work flow by extracting information as a side effect in a human-to-human dialog, or permit a rapid and efficient multimodal transfer or correction of such information during the human-human interaction.
In addition, the technique can be applied as a post-editing step based on a human-human interaction record. One important aspect of such a combined human-human and human-machine dialog is the treatment of errors. Not all information will be recognized and translated correctly, or subsequently correctly extracted and filled into the desired record. In such cases, human intervention may be necessary to verify and correct the resulting record. In certain embodiments, various multi-modal error repair techniques can be employed to correct the record.
In another general aspect, the present invention is directed to systems and method for resolving ambiguities occurring during speech translation of a human-to-human dialog between a first speaker speaking a first language and a second speaker speaking a second language. When an ambiguity is detected as part of either the speech recognition or translation processes, a disambiguation query is issued to the participants (e.g., the speakers). The disambiguation query is phrased so that a response resolves the ambiguity and permit the speech translation process to continue with higher confidence. A touch screen display may be used to issue disambiguation query and to receive a speaker's response.
Various embodiments of the present invention are described herein by way of example in conjunction with the following figures, wherein:
Similarly, in the other direction, input speech from Speaker 2 in Language 2 is input to an Automatic Speech Recognition engine (ASR 2) for Language 2. ASR 2 converts the input speech in Language 2 to text in Language 2 using, for example, conventional automatic speech recognition technologies. The text output from ASR 2 is input to Machine Translation engine (MT 2), which translates the text in Language 2 to Language 1. The translated speech in Language 1 from MT2 may be then output on the text display 14 and/or output in speech in Language 1 by the speaker 16 using a text-to-speech synthesizer (TTS 1) for Language 1. Multiple additional arrangements in additional language directions may be added depending on application and deployment.
In addition, each direction includes an information extraction module, IEM 1 for Language 1 and IEM 2 for Language 2. IEM 1 receives the text output from the ASR 1 in Language 1 and IEM 2 receives the text output in Language 2 from the ASR 2. The IEMs are configured to extract relevant information from the human-to-human dialog.
In one embodiment, the IEMs process their respective text inputs from the respective ASRs by formatting and parsing steps (PARS) to parse the textual sentence(s) from the ASR engines by a semantic grammar. In various embodiments, the IEMs may use a Phoenix parser to parse the text from the ASR engines. More details about Phoenix parsers may be found in W. Ward, “Extracting Information in Spontaneous Speech,” Proceedings of International Conference on Spoken Language, 1994 and W. Ward, “Understanding Spontaneous Speech: the Phoenix System,” Proceedings of 1CASSP '91, 1991, both of which are incorporated herein by reference in their entirety.
In another embodiment, instead of parsing, the IEMs use a spoken term detection or keyword spotting (KWS) module, which only extracts keywords instead of semantic frames, and copies them into the report. This keyword function might also be given by a named entity tagging module that identifies names in a text string. This is particularly useful where named entities (proper names, place names, medications, foods, etc.) are to be identified and filled in the form/report. Named entities play an important role in extracting names for requirements such as the preparation of patient records, hospital registration, refugee registration for disaster relief, and many more. More details about KWS modules may be found in L. Ratinov and D. Roth, “Design Challenges and Misconceptions in Named Entity Recognition,” CoNLL (2009) and R. Finkel, T. Grenager, and C. Manning, “Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling,” Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL 2005), pp. 363-370, which are incorporated herein by reference in their entirety.
In another embodiment, the IEMs employ an information retrieval (IR) module. The IR modules IR1 and IR2 may extract relevant documents or responses from a large remote database of documents, including the World Wide Web, as shown in
The speech translation device 12 shown in
In other embodiments, a distributed computing system may be used. For example, as shown in
Various embodiments of the present invention can also be expanded by a multimodal interaction interface (MMII), as shown in
Transferring information from the human-to-human dialog to the forms should ideally always be done automatically, but multimodal interaction provides for a robust recovery mechanism from errors. In such a case, two different types of errors may be identified: omissions and erroneous information. In the case of omissions, the computer device (e.g., computer device 32 in
As mentioned above, the system could be used with multiple speakers.
In various embodiments, the present invention can provide:
In the example of
In various embodiments, the form may also permit data entry by other modes besides speech extraction. For example, through a touch screen interface 14, for example, the user (e.g., the doctor) may click on the “male” and “female” fields in the form if the doctor is able to determine the sex of the patient without a question-answer dialog. Similarly, if the patient is male, the “not pregnant” response may be automatically populated.
In addition, in various embodiments, instead of Speaker 1 enunciating in Language 1 a question relevant to one of the fields of the form, e.g., asking in English “What is your age?,” the user could activate (e.g., via the touch screen interface) an icon, e.g., “Ask” input 110, which when activated causes the device 12 to audibly output in the selected second language (Spanish in this example, see the language selection input 91) a prerecorded question designed to elicit the information for the associated field (e.g., the age field 96). For example, by activating the “Ask” input 100 associated with the age field 96, the device 12 may audibly output a question such as “¿c7,Cuál es su edad?” (“What is your age” in Spanish) without the user having to first enunciate the question in Language 1 (e.g., English). Other fields in the form 94 could have associated “Ask” buttons that cause the device 12 to audibly output prerecorded questions designed to elicit the desired information for their associated field, as shown in the example of
Similarly, in various embodiments, the interface 89 may provide buttons or inputs (not shown) where, when activated by the user, prerecorded instructional or explanatory information—as opposed to a question—is enunciated for Speaker 2 (e.g., the patient) in Language 2 (e.g., Spanish). Such prerecorded explanatory information can explain an aspect of the procedure to the patient, for example. For example, instead of the doctor (e.g., Speaker 1) saying in English, “I will now check your blood pressure” and then having the question translated into Language 2 (e.g., Spanish), the doctor could merely activate the associated tell button for blood pressure, in which case the device may audibly output “Ahora voy a revisar su presión arterial” (Spanish for “I will now check your blood pressure”), thereby obviating the need for the doctor to first audiblize the explanation in Language 1 and then have it translated to Language 2. The prerecorded questions and explanations can save time and effort, particularly during repetitive queries such as patient's personal data (name, age, etc.) or the collection of vital signs.
In other embodiments, instead of having the questions and answers prerecorded, the device 12 may produce the questions and/or answers synthetically. For example, the user could generate his/her own questions or explanations in Language 1 for various fields of the form 94, being input and stored in the device 12 via speech or text, for example. The device 12 can translate this input as explained above into Language 2 when needed.
Of course,
It should be noted that not always will the extraction of the information from the human-human dialog be accurate or correct. The device 12 in various embodiments may provide means for repairing or correcting the information in the form. These means can be other than speech extraction, i.e., other input modes (e.g., allowing multi-modal input for the form). For example, using the touch screen interface 14, the user (e.g., Speaker 1) could drag information from either of the fields 90, 92 to appropriate field in the form 94 by a touch and drag gesture or input.
Alternatively or additionally, in other embodiments different modes of form correction or data entry can be used. For example, in various embodiments the user (e.g., the doctor) could click on or otherwise selected extracted data in the form 94 to edit the data in the form. For example, in various embodiments the user could edit (or delete) the selected data via a keyboard (including a touch screen keyboard), by respeaking the data for speech extraction, by a handwriting gesture on the touch screen interface 14, or by any other suitable means. Similarly, the information can be entered independently without explicitly asking a question related to the form 94, as mentioned above. For example, the doctor may see that the patient is female, and just click or type the information on the form 94 directly, thus bypassing the human-human dialog altogether at this stage. In this case, the user (e.g., Speaker 1) may choose to push on the field directly and activate speech recognition in either language to enter the information directly. Alternatively, the information for the field in the form 94 may be typed or spelled or handwritten. U.S. Pat. Nos. 5,712,957 and 5,855,000, which are incorporated herein by reference in their entirety, provide details on various multi-modal correction techniques.
An applications processor 204 may execute an operating system as well as software applications, including the speech recognition and translation modules described herein (e.g., the ASR, MT, TTS and IEM modules shown in
Various other sensors may be included in certain embodiments. A magnetic sensor 226 may sense magnetic fields near the device. For example, the magnetic sensor 226 may be used by various apps and/or system functionality to implement a compass. An accelerometer 228 and gyroscope 230 may provide data describing movement of the device. For example, data from the accelerometer 228 and gyroscope 230 may be used to orient the display of the touch screen 232 (e.g., portrait versus landscape). The device 200 may be powered by a battery 234, which may, in turn, be managed by a power management integrated circuit (PMIC) 236. An I/O transceiver 238 may manage wired communications between the device and other devices, for example, according to the Universal Serial Bus (USB) or any other suitable standard. A connector 239 may facilitate wired connections. In some embodiments, connections via the connector 239 and I/O transceiver 238 may provide power for charging the battery 234.
Returning to the example form in
In various embodiments, questionable information may be determined by way of confidence measures that use both acoustic confidences, as well as complementary information that confirms plausibility of the extracted information. The questionable information in the form may be called out, for example, such as by highlighting. For example, a doctor may enter the gender of the patient directly, but confidence measures may highlight the information if acoustic confidences and speaker/gender ID suggest that the information could be in error. Low confidence information may be highlighted or otherwise called out in the form as well as in the text of the speech translation dialog window. Other sources of low confidence may be derived from the detection of Out-of-Vocabulary (OOV) items, acoustic, semantic and translation ambiguities. Low confidence regions may be highlighted and the ambiguity explained (similar word, alternate meaning, missing word, etc.).
In the example of
In addition, in various embodiments, the speech translation device 12 permits the definition of the forms themselves. For example, the form(s) may be read in from a template form (e.g., a spreadsheet such as Excel) and generate a form (as shown in the example of
Additionally, via the transceiver 206 or the transceiver 218, the device 12 may be able to share a form (stored in database 26), completed or not, by email or electronic patient record upload, for example, thereby enabling incremental and instantaneous updates of information records at centralized databases from mobile portable devices. In various embodiments, the user may initiate such wireless transmission or uploading of the form by activating the share icon 116. The uploaded/transmitted form may or may not include the actual dialog data as well as pertinent multimodal user interaction, queries and edits, thus providing granular information on the entire human-human encounter and interaction.
In various embodiments, the device may operate (e.g., translate speech and extract the form-fillable information) in simultaneous and consecutive translation mode. In simultaneous mode, the device uses the transcript and translation of an ongoing continuous dialog in real-time without the use of push buttons or other inputs to toggle between speakers, and without waiting for the translation to complete. In this case, the device extracts the relevant and pertinent information into the forms as above and error correction works the same. In consecutive translation mode, the speakers take turns speaking and speak consecutively. They may use a button or other input on the device (such as on the touch screen interface 14) to transition from one translation channel (e.g., translation from Language 1 to Language 2) to the other (e.g., translation from Language 2 to Language 1). The interface may selectively enable or disable the speech translation and/or the form windows for the convenience of the user.
In addition, as shown in the example of
At the translation level, the of the interactive disambiguation module 40 may request disambiguation based on input from the MT engine (e.g., MT1 or 2) in various embodiments if several output translations are possible and of similar likelihood (there is at least one other translation that is within a threshold scoring difference of the highest scoring translation), or if there is one more likely output (e.g., there are no alternative output translations within the threshold scoring difference from a highest scoring output translation), the highest scoring output has a low confidence in the target language and is likely to be wrong. In the later case (b), a query back to the user in his/her (source) language may be issued, alerting the speaker of the likely problem and requesting a restatement or a paraphrase for clarity. In the former case (a), two or more alternatives need to be considered. In this case, the system may present to the user these two cases embedded in a carrier sentence in the source language that disambiguates the two uses of the work in the target language. For example, if the user speaks the following sentence in English—“This is my nail”—the word “nail” may be considered ambiguous and may be translated in Spanish as “clavo” or as “uña”. If the translation probabilities of both outcomes are comparable within a threshold of plausibility, the system can request disambiguation from the user. In various embodiments, this may be then done by searching for phrases that contain the word in question in either of its usage in a training database. For example, the speaker's devices may be connected via a data network to a remote computer system that search a remote training database. The remote computer system may detect the disambiguation or one of the speaker's devices may detect the disambiguation. In alternative embodiments, instead of a training database a traditional dictionary could be used, which could be stored on the speaker's device or remotely.
On the source side, the system is then in a position to ask the user: “did you mean nail as in: “they used nails to fix the door”, or as in “she painted her nails red”. If the user then selects one or the other solution, the system will adopt the selected translation meaning, and increase the translation probabilities for this word. Presentation of the two ambiguous options can be given graphically on the display or by a verbal disambiguation phrase.
The code for the disambiguation module may be stored in the RAM 212 and/or non-volatile data storage 216, and may be executed by the application processor 204, for example. See
If not ambiguity is detected as step 50, the process flows to step 54, where it is determined whether a disambiguation query is necessary based on the translated speech of Speaker 1 from Language 1 to Language 2. This may be done as described previously. If a disambiguation query is needed, the process flows to step 52 where the disambiguation query is issued. Of course, if no disambiguation query is needed due to translation either, the system continues with the normal speech recognition and translation process at step 56. As mentioned above, the disambiguation queries at step 52 may have different formats based on the type of ambiguity, e.g., recognition versus translation.
The above-described disambiguation capability can be used with or without the information extraction capability according to various embodiments.
In the above description, the information extraction process was described in the context of a bi-lingual dialog. In other embodiments, the information extraction process could be performed in the process of a mono-lingual dialog. Similar to the bi-lingual mode, the device in the mono-lingual mode may perform speech recognition on the speech utterances by the speakers and extract key information in the recognized speech to fill in the various and applicable fields of the form 28. Using the same example as above from the bi-lingual mode, the doctor may ask in the first language (e.g., English) “What is your age?” The patient may answer in the same language (i.e., a mono-lingual dialog) “I am fifty years old.” The key information, here “50,” can be extracted from the dialog and filled-in the age field 96 of the form 94 (see
In addition, as before, the device 12 may be able to share the form, completed or not, by email or electronic patient record upload, thereby enabling incremental and instantaneous updates of information records at centralized databases from mobile portable devices. The uploaded/transmitted form may or may not include the actual dialog data as well as pertinent multimodal user interaction, queries and edits, thus providing granular information on the entire human-human encounter and interaction
Accordingly, pursuant to various embodiments, the present invention is directed to a device for extracting information from a human-to-human dialog between at least a first speaker and a second speaker. The device may comprise at least one microphone, a screen display, at least one programmable processor and at least one data storage unit for storing digital data. The at least one programmable processor is in communication with the at least one microphone and the screen display. Further, the at least one programmable processor is programmed to: (i) automatically recognize speech by the first speaker received by the at least one microphone; (ii) automatically recognize speech by the second speaker received by the at least one microphone; (iii) extract at least information from the recognized speech by the second speaker; and (iv) enter the extracted information from the recognized speech by the second speaker into an electronic form that is stored in the at least one data storage unit of the computer system and displayed in a graphical user interface on the screen display.
According to various implementations: the first speaker speaks a first language; the second speaker speaks a second language that is different from the first language; and the at least one programmable processor is further programmed to: (i) automatically translate the recognized speech by first speaker in the first language to the second language; (ii) automatically translate the recognized speech by second speaker in the second language to the first language; (iii) extract at least information from the recognized speech by the second speaker by extracting at least information from the translation of the recognized speech by the second speaker translated to the first language; and (iv) enter the extracted information by entering the extracted information from the translation of the recognized speech by the second speaker translated to the first language into the electronic form stored in the at least one data storage unit. In addition, the processor may be further programmed to: (v) extract at least information from the recognized speech by the first speaker in the first language; and (vi) enter the extracted information from the recognized speech by the first speaker in the first language into the electronic form.
According to various implementations, the processor is programmed to extract the information from the translation of the recognized speech by the second speaker translated to the first language by parsing the translation by a semantic grammar. In addition, the processor may be further programmed to retrieve one or more documents related to the extract information from a remote database. In addition, the processor is programmed to extract the information from the translation of the recognized speech by the second speaker translated to the first language by detecting one or more keywords in the translation. Additionally, the processor may be further programmed to solicit feedback from at least one of the first speaker and the second speaker prior to entering the extracted information in the electronic form. Also, the at least one programmable processor may be programmed to recognize and received an edit to extracted information in the electronic form input via the screen display by a user of the device.
In another general aspect, the present invention is directed to a computer-based device for extracting information from a human-to-human dialog between at least a first speaker and a second speaker. The device comprises at least one microphone, a screen display, and at least one data storage unit for storing digital data. The device also includes a first automatic speech recognition module for automatically recognizing speech by the first speaker received by the at least one microphone. The device further includes a second automatic speech recognition module for automatically recognizing speech by the second speaker received by the at least one microphone. In addition, the device includes an information extraction module in communication with the first and second automatic speech recognition modules, the at least one microphone and the screen display. The information extraction module is for: (i) extracting at least information from the recognized speech by the second speaker; and (ii) entering the extracted information from recognized speech by the second speaker into an electronic form that is stored in the at least one data storage unit and displayed on a graphical user interface on the screen display.
According to various implementations: the first speaker speaks a first language; the second speaker speaks a second language that is different from the first language; and the device further comprises: (i) a first machine translation module in communication with the first automatic speech recognition module, wherein the first machine translation module is for automatically translating the recognized speech by first speaker in the first language to the second language; and (ii) a second machine translation module in communication with the second automatic speech recognition module, wherein the second machine translation module is for automatically translating the recognized speech by second speaker in the second language to the first language. In such an implementation, the information extraction module is for: (i) extracting at least information from the recognized speech by the second speaker by extracting at least information from the translation of the recognized speech by the second speaker translated into the first language; and (ii) entering the extracted information by entering the extracted information from translation of the recognized speech by the second speaker translated to the first language into the electronic form stored in the at least one data storage unit.
In various implementations, the information extraction module is further for: (i) extracting at least information from the recognized speech by the first speaker in the first language; and (ii) entering the extracted information from the recognized speech by the first speaker in the first language into the electronic form. The information extraction module may extract the information from the translation of the recognized speech by the second speaker translated to the first language by parsing the translation by a semantic grammar. Also, the device may comprise an information retriever module for retrieving one or more documents related to the extract information from a remote database. In addition, the information extraction module may extract the information from the translation of the recognized speech by the second speaker translated to the first language by detecting one or more keywords in the translation. The device may further comprise a multimodal interaction interface to solicit feedback from at least one of the first speaker and the second speaker prior to entering of the extracted information in the electronic form.
In yet another general aspect, the present invention is directed to a computer-implemented method for extracting information during a human-to-human dialog between at least a first speaker and a second speaker speaking. The method may comprise the steps of: (i) receiving, by at least one microphone of a computer-based information extraction device, speech by the first and second speakers during the dialog; (ii) automatically recognizing, by the computer-based information extraction device, the speech by the first speaker; (iii) automatically recognizing, by the computer-based information extraction device, the speech by the second speaker in the second language; (iv) extracting, by the computer-based information extraction device, at least information from the recognized speech by the second speaker; and (v) entering, by the computer-based information extraction device, the extracted information from the recognized speech by the second speaker into an electronic form stored in at least one data storage unit of the information extraction device.
In various implementations, the method may further comprise the step of displaying the form on a screen display of the computer-based information extraction device. Also, in a situation where the first speaker speaks a first language and the second speaker speaks a second language that is different from the first language, and the method may further comprise the steps of: (i) automatically translating, by the computer-based information extraction device, the recognized speech by first speaker in the first language to the second language; (ii) and automatically translating, by the computer-based information extraction device, the recognized speech by second speaker in the second language to the first language. Also, the step of extracting at least information may comprise extracting by the computer-based information extraction device at least information from the translation of the recognized speech by the second speaker translated to the first language. And the step of entering the extracted information may comprise entering, by the computer-based information extraction device, the extracted information from the translation of the recognized speech by the second speaker translated to the first language into the electronic form stored in the at least one data storage unit of the information extraction device
In various implementations, the method further comprises: extracting at least information from the recognized speech by the first speaker in the first language; and entering the extracted information from the recognized speech by the first speaker in the first language into the electronic form. Extracting the information from the translation of the recognized speech by the second speaker translated to the first language may comprise parsing the translation by a semantic grammar. The method may further comprise retrieving, by the computer-based information extraction device, one or more documents related to the extract information from a remote database. Extracting the information from the translation of the recognized speech by the second speaker translated to the first language may comprise detecting one or more keywords in the translation. The method may also further comprise the step of soliciting, by the computer-based information extraction device, feedback from at least one of the first speaker and the second speaker prior to entering the extracted information in the electronic form. In addition, the method may further comprise: (i) determining, by the computer-based information extraction device, whether there exists an ambiguity in the recognized speech of the first speaker; (ii) determining, by the computer-based information extraction device, whether there exists an ambiguity in the translation of the recognized speech of the first speaker in the first language into the second language; and (iii) upon a determination by the computer-based information extraction device that there is ambiguity in either (a) the recognized speech of the first speaker or (b) the translation of the recognized speech of the first speaker in the first language into the second language, issuing by the computer-based information extraction device a disambiguation query to the first speaker via the screen display of the computer-based information extraction device, wherein a response to the disambiguation query resolves the ambiguity. The disambiguation query issued to the first speaker may be different when the ambiguity is in the recognized speech of the first speaker than when the ambiguity is in the translation of the recognized speech of the first speaker in the first language into the second language. Also, the determination of whether there exists an ambiguity in the recognized speech of the first speaker may be based upon a plurality of factors, including: (i) an acoustic confidence score in the recognized speech of the first speaker; (ii) a context of the electronic form; and (iii) a language context given by a translation of one or more utterances from the second speaker from the second language to the first language. Determination of whether there exists an ambiguity in the translation of the recognized speech of the first speaker in the first language into the second language also may be based upon a plurality of factors, including: whether there are one or more alternative output translations within a threshold scoring difference of a highest scoring output translation; and whether, if there are no alternative output translations within the threshold scoring difference of the highest scoring output translation, the score for the highest scoring output translation is below a minimum threshold.
In yet another general aspect, the present invention is directed to a computer-implemented method of resolving ambiguity in a speech translation of a human-to-human dialog between at least a first speaker speaking a first language and a second speaker speaking a second language. The method may comprise the steps of: (i) recognizing, by a computer-based speech translation system, speech by the first speaker in the first language; (ii) determining, by the computer-based speech translation system, whether there exists an ambiguity in the recognized speech of the first speaker; (iii) translating, by the computer-based speech translation system, the recognized speech of the first speaker in the first language into the second language; (iv) determining, by the computer-based speech translation system, whether there exists an ambiguity in the translation of the recognized speech of the first speaker in the first language into the second language; and (v) upon a determination by the computer-based speech translation system that there is ambiguity in either (a) the recognized speech of the first speaker or (b) the translation of the recognized speech of the first speaker in the first language into the second language, issuing by the computer-based speech translation system a disambiguation query to the first speaker via a user-interface of the speech translation system, wherein a response to the disambiguation query resolves the ambiguity.
In yet another general aspect, the present invention is directed to a device for extracting information from a human-to-human dialog between at least a first speaker and a second speaker. The device may comprise: at least one microphone; a screen display; and at least one programmable processor and at least one data storage unit for storing digital data. The at least one programmable processor is in communication with the screen display and the at least one microphone. And the at least one programmable processor may be programmed to: (i) automatically recognize speech by the first and second speakers, received by the at least one microphone, during a dialog between the first and second speakers; (ii) output recognized speech of the first and second speakers on a first portion of a graphical user interface that is displayed on the screen display during the dialog between the first and second speakers; and (iii) output on a second portion of the graphical user interface that is displayed on the screen display a form with information extracted from the dialog between the first and second speakers. The at least one programmable processor may be further programmed to extract information from the dialog between the first and second speakers for outputting on the second portion of the graphical user interface in the form. Also, for a situation where the first speaker is speaking a first language and the second speaker is speaking a second language, the at least one programmable processor may be programmed to: (i) translate the recognized speech of the first speaker to the second language; (ii) translate the recognized speech of the second speaker to the first language; and (iii) display on the first portion of the graphical user interface the translations of the recognized speech of the first and second speakers. In addition, the at least one programmable processor may be programmed to recognize and received an edit to extracted information input via the screen display by a user of the device.
It will be apparent to one of ordinary skill in the art that at least some of the embodiments described herein may be implemented in many different embodiments of software, firmware, and/or hardware. The software and firmware code may be executed by a processor circuit or any other similar computing device. The software code or specialized control hardware that may be used to implement embodiments is not limiting. For example, embodiments described herein may be implemented in computer software using any suitable computer software language type, using, for example, conventional or object-oriented techniques. Such software may be stored on any type of suitable computer-readable medium or media, such as, for example, a magnetic or optical storage medium. The operation and behavior of the embodiments may be described without specific reference to specific software code or specialized hardware components. The absence of such specific references is feasible, because it is clearly understood that artisans of ordinary skill would be able to design software and control hardware to implement the embodiments based on the present description with no more than reasonable effort and without undue experimentation.
Moreover, the processes associated with the present embodiments may be executed by programmable equipment, such as computers or computer systems and/or processors. Software that may cause programmable equipment to execute processes may be stored in any storage device, such as, for example, a computer system (nonvolatile) memory, an optical disk, magnetic tape, or magnetic disk. Furthermore, at least some of the processes may be programmed when the computer system is manufactured or stored on various types of computer-readable media.
It can also be appreciated that certain process aspects described herein may be performed using instructions stored on a computer-readable medium or media that direct a computer system to perform the process steps. A computer-readable medium may include, for example, memory devices such as diskettes, compact discs (CDs), digital versatile discs (DVDs), optical disk drives, or hard disk drives. A computer-readable medium may also include memory storage that is physical, virtual, permanent, temporary, semipermanent, and/or semitemporary.
A “computer,” “computer system,” “host,” “server,” or “processor” may be, for example and without limitation, a processor, microcomputer, minicomputer, server, mainframe, laptop, personal data assistant (PDA), wireless e-mail device, cellular phone, pager, processor, fax machine, scanner, or any other programmable device configured to transmit and/or receive data over a network. Computer systems and computer-based devices disclosed herein may include memory for storing certain software modules or engines used in obtaining, processing, and communicating information. It can be appreciated that such memory may be internal or external with respect to operation of the disclosed embodiments. The memory may also include any means for storing software, including a hard disk, an optical disk, floppy disk, ROM (read only memory), RAM (random access memory), PROM (programmable ROM), EEPROM (electrically erasable PROM) and/or other computer-readable media. The software modules and engines described herein can be executed by the processor (or processors as the case may be) of the computer devices that access the memory storing the modules.
In various embodiments disclosed herein, a single component may be replaced by multiple components and multiple components may be replaced by a single component to perform a given function or functions. Except where such substitution would not be operative, such substitution is within the intended scope of the embodiments. Any servers described herein, for example, may be replaced by a “server farm” or other grouping of networked servers (such as server blades) that are located and configured for cooperative functions. It can be appreciated that a server farm may serve to distribute workload between/among individual components of the farm and may expedite computing processes by harnessing the collective and cooperative power of multiple servers. Such server farms may employ load-balancing software that accomplishes tasks such as, for example, tracking demand for processing power from different machines, prioritizing and scheduling tasks based on network demand and/or providing backup contingency in the event of component failure or reduction in operability.
The computer systems may comprise one or more processors in communication with memory (e.g., RAM or ROM) via one or more data buses. The data buses may carry electrical signals between the processor(s) and the memory. The processor and the memory may comprise electrical circuits that conduct electrical current. Charge states of various components of the circuits, such as solid state transistors of the processor(s) and/or memory circuit(s), may change during operation of the circuits.
While various embodiments have been described herein, it should be apparent that various modifications, alterations, and adaptations to those embodiments may occur to persons skilled in the art with attainment of at least some of the advantages. The disclosed embodiments are therefore intended to include all such modifications, alterations, and adaptations without departing from the scope of the embodiments as set forth herein.
This application is a continuation of U.S. patent application Ser. No. 15/338,750, filed Oct. 31, 2016, which is a continuation of U.S. patent application Ser. No. 14/985,300, filed Dec. 30, 2015, now U.S. Pat. No. 9,514,130, which is a continuation of U.S. patent application Ser. No. 13/760,535, filed Feb. 6, 2013, now U.S. Pat. No. 9,257,115, which claims the benefit of, under 35 U.S.C. § 119(e), U.S. provisional patent application Ser. No. 61/608,334, entitled “SPEECH TRANSLATION DEVICE WITH INFORMATION EXTRACTION,” filed Mar. 8, 2012, all of which are hereby incorporated herein by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5712957 | Waibel et al. | Jan 1998 | A |
| 5855000 | Waibel et al. | Dec 1998 | A |
| 6324510 | Waibel et al. | Nov 2001 | B1 |
| 6963837 | Finke et al. | Nov 2005 | B1 |
| 7124085 | Junqua et al. | Oct 2006 | B2 |
| 7149347 | Wnek | Dec 2006 | B1 |
| 7660400 | Bangalore et al. | Feb 2010 | B2 |
| 7907705 | Huff et al. | Mar 2011 | B1 |
| 8090570 | Waibel et al. | Jan 2012 | B2 |
| 8204739 | Waibel et al. | Jun 2012 | B2 |
| 10318623 | Waibel | Jun 2019 | B2 |
| 20050135571 | Bangalore et al. | Jun 2005 | A1 |
| 20060173682 | Manabe et al. | Aug 2006 | A1 |
| 20070033005 | Cristo et al. | Feb 2007 | A1 |
| 20070043552 | Omi et al. | Feb 2007 | A1 |
| 20070118351 | Sumita | May 2007 | A1 |
| 20070271088 | Waibel et al. | Nov 2007 | A1 |
| 20080077387 | Ariu | Mar 2008 | A1 |
| 20080120091 | Waibel et al. | May 2008 | A1 |
| 20080126087 | Chou | May 2008 | A1 |
| 20080235014 | Oz | Sep 2008 | A1 |
| 20100076972 | Baron et al. | Mar 2010 | A1 |
| 20110307241 | Waibel et al. | Dec 2011 | A1 |
| 20120323574 | Wang et al. | Dec 2012 | A1 |
| Number | Date | Country |
|---|---|---|
| HII-120176 | Apr 1999 | JP |
| 2000-010578 | Jan 2000 | JP |
| 3411198 | May 2003 | JP |
| 2005-141089 | Jun 2005 | JP |
| 3890326 | Mar 2007 | JP |
| 2011-524991 | Sep 2011 | JP |
| Entry |
|---|
| Canadian Office Action, Canadian Patent Application No. 2,872,790, dated Sep. 20, 2017, four pages. |
| Annex to Form PCT/ISA/206 Communication Relating to Results of the Partial International Search, PCT/US2013, 028831, dated May 17, 2013. |
| Australian First Examination Report, Australian Patent Application No. 2013230453, dated Mar. 3, 2016, 2 pages. |
| Azzini et al., “Application of Spoken Dialogue Technology in a Medical Domain”, Text, Speech and Dialogue, Springer Berlin Heidelberg, Berlin, vol. 2448, Jan. 1, 2006, pp. 449-452. |
| Azzini et al., “First steps toward an adaptive spoken dialogue system in medical domain”, Eurospeech, Scandinavia, vol. 2, Jan. 1, 2001, 4 pages. |
| Canadian Office Action, Canadian Patent Application No. 2,872,790, dated Sep. 23, 2016, 5 pages. |
| Eck et al.: “Jibbigo: Speech-to-Speech translation on mobile devices”, Spoken Language Technology Workshop {SLT), IEEE, Piscataway, NJ, USA, Dec. 12, 2010, pp. 165-166. |
| European Patent Office, Examination Report, European Patent Application No. 13710719.9, dated May 19, 2017, give pages. |
| Finkel, J.R. et al., “Incorporating Non-local Information into Information Extraction Systems by Gibbs Sampling,” Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, Jun. 2005, pp. 363-370, Ann Arbor, Michigan, USA. |
| Japanese Office Action, Japanese Patent Application No. 2014-560989, dated Mar. 29, 2016, 8 pages. |
| Korea Intellectual Property Office, Office Action, Korean Patent Application No. 10-2014-7028069, dated Jun. 20, 2016, ten pages. |
| Narayanan, 5., et al.: “Transonics: a speech to speech system for English-Persian interactions”, Automatic Speech Recognition and Understanding, 2003, ASRU 2003 IEEE Workshop on St. Thomas, VI, USA, Piscataway, NJ, USA, Nov. 30-Dec. 3, 2003, pp. 670-675. |
| PCT International Search Report and Written Opinion, PCT Application No. PCT/U52013/028831, dated Oct. 10, 2013, sixteen pages. |
| Ratinov, L. et al., “Design Challenges and Misconceptions in Named Entity Recognition,” Proceedings of the Thirteenth Conference on Computational Natural Language Learning, Jun. 2009, pp. 147-155, Boulder, Colorado, USA. |
| Reithinger, “Robust Information Extraction in a Speech Translation System,” Proceedings of EuroSpeech-99, pp. 2427-2430, 1999 {4 pages). |
| State Intellectual Property Office of the People's Republic of China, First Office Action, Chinese Patent Application No. 2013800203743, dated Jun. 2, 2016, twenty pages. |
| United States Advisory Action, U.S. Appl. No. 13/760,535, dated Aug. 20, 2015, 3 pages. |
| United States Office Action, U.S. Appl. No. 13/760,535, dated Apr. 28, 2015, 11 pages. |
| United States Office Action, U.S. Appl. No. 13/760,535, dated Dec. 3, 2014, 10 pages. |
| United States Office Action, U.S. Appl. No. 14/985,300, dated Feb. 25, 2016, six pages. |
| Ward, W., “Extracting Information in Spontaneous Speech,” Proceedings of International Conference on Spoken Language Processing, Sep. 18-22, 1994, pp. 83-86, Yokohama, Japan. |
| Ward, W., “Understanding Spontaneous Speech: The Phoenix System,” Proceedings of ICASSP '91, 1991, pp. 365-367. |
| Number | Date | Country | |
|---|---|---|---|
| 20190251156 A1 | Aug 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61608334 | Mar 2012 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15338750 | Oct 2016 | US |
| Child | 16395001 | US | |
| Parent | 14985300 | Dec 2015 | US |
| Child | 15338750 | US | |
| Parent | 13760535 | Feb 2013 | US |
| Child | 14985300 | US |
