Patent Grant
7562006
1. Field of the Invention
The present invention relates to a dialog supporting device which supports an on-going dialog made between people.
2. Description of the Related Art
Conventionally, translation devices have been developed with the purpose of supporting on-going dialog in different languages, which is made between ordinary people and local people on trips abroad or the like. Such a translation device is represented by certain translation devices, by which a translation scheme using original/translation pairs of example sentences or example usages is implemented in small information processing apparatuses such as Personal Digital Assistances (PDAs). Such devices have thousands of examples in order to cover general travel conversation. However, the devices have a problem in usability when a user actually uses the device, since the user has to select a desired example from a list of the examples by sight. This problem is noticeable, especially when such a device has a small display for displaying the list of examples, which makes it difficult for the user to view many examples at once. Moreover, in general use of the translation device, one dialog with a dialog partner requires more than several examples, so that the translation device completes the dialog taking much more time than the user expects. Therefore, in order to achieve the final purpose of supporting dialog between people, there is a need for a function by which the user can speedily select the desired example from a great number of examples.
As a method for solving the above problem, there has been proposed a technique of narrowing down candidates for a next utterance of a user, using example dialog models or a corpus of conversation training histories, as disclosed in Japanese Unexamined Patent Application Publication No. 2003-30187, for instance.
When such a translation device is actually used, two users who speak respective different languages have dialog using respective translation devices alternately. When the translation device has an example expressing exactly what one user wishes to say to the other user (hereinafter, referred to also as “a partner”, “a dialog partner”, or “a partner user”), the former user simply selects the example. However, each of the examples in the translation device is generally a short sentence in order to be applied to various dialog situations. The reason for a short sentence as mentioned above is that a long example expresses rather specific situations and thereby applications of the sentence are considerably limited. Therefore, the user has to express what to say, by sequentially selecting a plurality of such short examples.
However, although it is possible to shorten a time required to select examples using, for instance, the technique disclosed in the above-mentioned Japanese Unexamined Patent Application Publication No. 2003-30187 or the like, it is not possible to completely eliminate a time lag between selection of a certain example and selection of a next example. Therefore, in the actual use, there is a problem that the time required to complete dialog is eventually increased, due to increase of utterances caused by misunderstandings and hasty conclusions, which will be explained in detail below.
For instance, in the case where a user A who speaks Japanese has dialog with a user B who speaks English and the user A wishes to say to the user B, “What time does a train bound for an airport leave?”, only the following examples are related to what the user wishes to say in the translation device.
Thus, in a view of the above problems, an object of the present invention is to provide a dialog supporting device which can support dialog to be speedily completed, by reducing the number of utterances caused by misunderstandings and hasty conclusions.
In order to achieve the above object, a dialog supporting device according to the present invention supports dialog. The device includes: an example database operable to hold first examples and second examples which are used in the dialog, the respective first examples being associated with the respective second examples; an utterance receiving unit operable to receive an utterance of a user; an example selecting unit operable to (i) select a first example by searching the example database based on the utterance received by the utterance receiving unit, and (ii) obtain a second example associated with the selected first example; an utterance difference calculating unit operable to (i) calculate a difference between the utterance and the first example selected by the example selecting unit, and (ii) obtain difference information representing the calculated difference; and a sending unit operable to send the second example obtained by the example selecting unit and the difference information obtained by the utterance difference calculating unit, to a dialog supporting device of a dialog partner of the user.
According to the dialog supporting device of the present invention, by using the difference information, it is possible to notify a dialog partner of a user of when the user completes utterances. Therefore, interruption during the utterances is prevented, so that the dialog using the dialog supporting devices go on smoothly. In particular, it is possible to prevent situations where the user makes a too early reply to the partner from kindness and friendliness which unfortunately confuses the dialog eventually causing discomfort of both of the user and the partner after the dialog.
The disclosure of PCT application No. PCT/JP2006/311786 filed Jun. 13, 2006 and Japanese Patent Application No. 2005-197923 filed Jul. 6, 2005, including specifications, drawings and claims are incorporated herein by reference in its entirety.
These and other objects, advantages and features of the present invention will become apparent from the following description thereof taken in conjunction with the accompanying drawings that illustrate specific embodiments of the invention. In the Drawings:
The dialog supporting device according to the present embodiments of the present invention supports dialog and includes: an example database operable to hold first examples and second examples which are used in the dialog, the respective first examples being associated with the respective second examples; an utterance receiving unit operable to receive an utterance of a user; an example selecting unit operable to (i) select a first example by searching the example database based on the utterance received by the utterance receiving unit, and (ii) obtain a second example associated with the selected first example; an utterance difference calculating unit operable to (i) calculate a difference between the utterance and the first example selected by the example selecting unit, and (ii) obtain difference information representing the calculated difference; and a sending unit operable to send the second example obtained by the example selecting unit and the difference information obtained by the utterance difference calculating unit, to a dialog supporting device of a dialog partner of the user.
Thereby, the user can notify a dialog partner of how much has already been told among what the user wishes to say (utterances received by the utterance receiving unit), so that it is possible to support the dialog to be speedily completed by reducing the number of utterances caused by misunderstandings and hasty conclusions.
Further, the example selecting unit may be further operable to select another first example related to the utterance, based on the difference between the utterance and the firstly-selected first example. Thereby, it is possible to sequentially gives the remaining part (rest) of what the user wishes to say to the partner, and also possible to notify the partner of the difference information at that time.
Furthermore, the example selecting unit may be operable to calculate a similarity which represents a degree of similarity between the utterance and the selected first example, using dynamic programming in a unit of any one of a character, a word, and a letter indicating pronunciation.
Still further, the utterance difference calculating unit may be operable to calculate the difference, by comparing the utterance and the first example selected by the example selecting unit in a unit of any one of a character, a word, and a letter indicating pronunciation.
Still further, the dialog supporting device may further include: a receiving unit operable to receive a second example and difference information sent from the dialog supporting device of the partner user; and an utterance outputting unit operable to output the second example and the difference information received by the receiving unit. Thereby, the user is able to know how much has already received among what the partner wishes to say to the user, so that the user can determine whether or not the received utterances should be replied. Thus, it is possible to support the dialog to be speedily completed by reducing the number of utterances caused by misunderstandings and hasty conclusions.
Still further, the utterance outputting unit may be operable to display the difference information received by the receiving unit, by a numeric character. Still further, the utterance outputting unit may be operable to display the difference information received by the receiving unit, by a figure. Thereby, it is possible for the user to immediately learn the difference information by sight, so that stress in waiting for a next utterance is reduced.
Still further, the utterance outputting unit may be operable to convert the difference information received by the receiving unit, into qualitative language expression, and output the resulting language expression. Still further, the utterance outputting unit may be operable to generate language expression corresponding to a filler based on the difference information received by the receiving unit, and output the resulting language expression. Thereby, it is possible to reduce stress of the user waiting for a partner's next utterance.
Still further, the dialog supporting device may further include: a candidate notifying unit operable to notify the user of candidates of the first example selected by the example selecting unit; and a selection instruction receiving unit operable to receive an instruction from the user to select one first example from the candidates of the first example, wherein the example selecting unit is operable to (i) extract candidates for a first example from the example database based on the utterance, and (ii) select one first example from the candidates according to the instruction received by the selection instruction receiving unit.
Still further, the utterance receiving unit is operable to receive the utterance (i) via continuous utterance recognition processing using a language model derived from the first example held in the example database, or (ii) via a keyboard under limitation by which only words included in the first examples held in the example database can be inputted.
Moreover, a dialog supporting system according to the present embodiments of the present invention includes a first dialog supporting device and a second dialog supporting device. The dialog supporting system supports dialog using the first and second dialog supporting devices. The first dialog supporting device includes: an example database operable to hold first examples and second examples which are used in the dialog, the respective first examples being associated with the respective second examples an utterance receiving unit operable to receive an utterance of a user; an example selecting unit operable to (i) select a first example by searching the example database based on the utterance received by the utterance receiving unit, and (ii) obtain a second example associated with the selected first example; an utterance difference calculating unit operable to (i) calculate a difference between the utterance and the first example selected by the example selecting unit, and (ii) obtain difference information representing the calculated difference; and a sending unit operable to send the second example obtained by the example selecting unit and the difference information obtained by the utterance difference calculating unit, to the second dialog supporting device. The second dialog supporting device includes: a receiving unit operable to receive the second example and the difference information sent from the first dialog supporting device; and an utterance outputting unit operable to output the second example and the difference information received by the receiving unit.
Moreover, a dialog supporting system according to the present embodiments of the present invention includes a first dialog supporting device, a second dialog supporting device, and a server connecting the first and second dialog supporting devices. The dialog supporting system supports dialog using the first and second dialog supporting devices. The first dialog supporting device includes: an utterance receiving unit operable to receive an utterance of a user; and a sending unit operable to send the utterance received by the utterance receiving unit to the server. The server includes: an example database operable to hold first examples and second examples which are used in the dialog, the respective first examples being associated with the respective second examples; an example selecting unit operable to (i) select a first example by searching the example database based on the utterance sent from the first dialog supporting device, and (ii) obtain a second example associated with the selected first example; an utterance difference calculating unit operable to (i) calculate a difference between the utterance and the first example selected by the example selecting unit, and (ii) obtain difference information representing the calculated difference; and a sending unit operable to send the second example obtained by the example selecting unit and the difference information obtained by the utterance difference calculating unit, to the second dialog supporting device. The second dialog supporting device includes: a receiving unit operable to receive the second example and the difference information from the server; and an utterance outputting unit operable to output the second example and the difference information received by the receiving unit.
Moreover, the present invention can be realized not only as the dialog supporting device, but also as: a dialog supporting method having steps performed by the units in the dialog supporting device; a computer program product which, when loaded into a computer, allows the computer to execute the steps; and the like. Furthermore, it is obvious that the program product can be distributed via a recording medium such as a CD-ROM or a transmitting medium such as the Internet.
The following describes preferred embodiments according to the present invention with reference to the drawings.
First Embodiment
The dialog supporting device according to the first embodiment is an device which supports dialog between people. The dialog supporting device has an example database 101, an utterance receiving unit 102, an example selecting unit 103, a sending unit 104, an utterance difference calculating unit 105, a receiving unit 106, and an utterance outputting unit 107.
Here, the utterance receiving unit 102 corresponds to the utterance receiving unit. The example selecting unit 103 corresponds to a set of the example selecting unit, the candidate notifying unit, and the selection instruction receiving unit. The utterance difference calculating unit 105 corresponds to the utterance difference calculating unit. The sending unit 104 corresponds to the sending unit. The receiving unit 106 corresponds to the receiving unit. The utterance outputting unit 107 corresponds to the utterance outputting unit.
The example database 101 holds examples used in dialog. The utterance receiving unit 102 receives an utterance of the user and converts the utterance into utterance information. From the example database 101, the example selecting unit 103 sequentially selects an example most similar to the utterance information, and provides a translation of the selected example. The utterance difference calculating unit 105 calculates a difference between (i) the utterance information and (ii) the example selected by the example selecting unit 103, and provides information regarding the difference (hereinafter, referred to as “difference information”). To a dialog supporting device of the partner user, the sending unit 104 sends the translated example provided from the example selecting unit 103 and the difference information. The receiving unit 106 receives a translated example and difference information outputted from the dialog supporting device of the partner user. The utterance outputting unit 107 outputs the received difference information and translated example to the partner user. Note that examples may is sent and received directly between the sending unit 104 of the user and the receiving unit 106 of the partner user or directly between the sending unit 104 of the partner user and the receiving unit 106 of the user, or indirectly via a server.
Next, a structure by which the above-structured dialog supporting device supports dialog in different languages is described in detail. It is assumed that the user is a native Japanese speaker, and the partner user is a native English speaker.
” of utterance No. 16 in language 1 is selected by the example selecting unit 103, then an associated translated example “I want to go to the airport.” in language 2 is outputted.
The utterance receiving unit 102 receives a user's utterance as character data using continuous utterance recognition processing or a keyboard, and provides the character data as utterance information to a next stage. Here, the utterance information is a character string corresponding to the user's utterance. When the utterance receiving unit 102 converts a user's utterance, using the continuous utterance recognition processing, accuracy of the utterance recognition is able to be increased by deriving language models from the examples held in the example database 101. On the other hand, when the utterance receiving unit 102 receives a user's utterance inputted by the keyboard, the characters are able to be efficiently inputted, by limiting input-able words to the words included in the examples in language 1 held in the example database 101.
Firstly, the example selecting unit 103 sets utterance information corresponding to an utterance inputted from the utterance receiving unit 102, to entire utterance information U (Step S301). Next, the entire utterance information U is set to partial utterance information (not-yet-translated remaining utterance information) u, and already-translated utterance information v is cleared to be empty. In addition, via the sending unit 104, a mode of the dialog supporting device of the partner user is changed to a receiving state (Step S302). The example selecting unit 103 calculates a degree of similarity (hereinafter, referred to as “a similarity degree”) between the partial utterance information u and each of the examples in language 1 in the example database 101 (Step S303). Here, the example selecting unit 103 calculates, as the similarity degree, a distance between the partial utterance information u and each of the examples, using dynamic programming as shown in ,
,
} and a sentence 2={
,
,
} a[3][3]=1.
Next, from all examples in language 1, the example selecting unit 103 selects examples whose similarity degrees are equal to or less than a threshold value β, and sets the selected examples to example candidates E (Step S304). Note that the threshold value β is determined based on the number of examples held in the example database 101 and the number of examples whose expression are similar. Then, the example selecting unit 103 determines whether or not the example candidates E are empty (Step S305). As a result, if the example candidates E are not empty (No at Step S305), then the partial utterance information u and the example candidates E are presented to the user, so that the user selects a desired example from the example candidates E (Step S306). Then, after selecting an example e by the user, an example which is a translation of the example e is provided to the sending unit 104 (Step S307). Next, the example selecting unit 103 updates the partial utterance information u and the already-translated utterance information v as u←U−v and v←v+u, respectively, to be provided to the utterance difference calculating unit 105 (Step S308). Then, it is determined whether or not the partial utterance information u is empty (Step S310), and if the partial utterance information u is empty (Yes at Step S310), then the processing is ended, while if the partial utterance information u is not empty (No at Step S310), then the processing returns to the similarity degree calculation (Step S303).
On the other hand, if the determination (Step S305) is made that the example candidates E are empty (No at Step S305), the partial utterance information u is shortened from its end in order to select examples most similar to the partial utterance information u from the example database 101 (Step S309). Here, a unit of the shortening is a character, a word, or a clause. For instance, in the case of u=“”, when the partial utterance information u is shortened on a clause basis, the partial utterance information u becomes “
” since the original u is {
,
,
,
}. Then, the processing returns again to the similarity degree calculation (Step S303) and repeats calculation of a similarity degree between the partial utterance information u and each of the examples in language 1 in the example database 101.
The utterance difference calculating unit 105 calculates difference information (U−v)/U based on the entire utterance information U and the already-translated utterance information v, and provides the resulting difference information to the sending unit 104. The difference information is calculated based on, for instance, the number of characters, the number of words, the number of letters representing pronunciation of the entire utterance information U and the already-translated utterance information v. Assuming the calculation is performed based on the number of characters, if the entire utterance information U=“” and the already-translated utterance information v=“
”, the resulting difference information becomes (14−5)/14≈0.64.
To the dialog supporting device of the partner, the sending unit 104 sends translated examples in language 2 in the example database 101 sequentially provided from the example selecting unit 103. Here, each of the translated examples is added with each difference information provided from the utterance difference calculating unit 105. The sent information is received by the receiving unit 106 of the partner's dialog supporting device, and outputted to the partner via the utterance outputting unit 107. The utterance outputting unit 107 has a display device such as a liquid crystal display (LCD) or audio synthesis processing.
The following describes processing performed by the above-structured dialog supporting devices in order to support dialog in different languages. Here, as shown in
For instance, as shown in ”, the utterance is transformed by utterance recognition processing of the utterance receiving unit 102, into utterance information represented by a character string “
” and displayed on the display area 505 as shown in
” at Step S301, partial utterance information u is set to “
” at Step S302, already-translated utterance information v is set to “ ”, and a mode of the partner's PDA 502 is changed to a receiving state via the user's sending unit 104. At the receiving state, a display area 701 displays a translated example of the last example which has recently been selected by the partner's dialog supporting device, and a display area 701 displays, as histories, translated examples of all examples which have already been selected sequentially by the partner's dialog supporting device. Note that, by pressing the utterance input button 504 to start inputting of an utterance, this receiving state of the PDA 502 is released and back to the state shown in
Next, by gradually shorting the partial utterance information u, the example selecting unit 103 calculates each similarity degree between the partial utterance information (remaining seech information u and each of the examples at Steps S303, S304, S405, and S309, in order to eventually obtain example candidates E={,
} corresponding to the partial utterance information u=“
”. At Step S306, a value of the partial utterance information u is reflected in displaying on the display area 505, by being underlined as shown in
” as a desired example e, a translated example “I want to go to the airport.” is provided to the sending unit 104 to be sent. At Step S308, the already-translated utterance information v is set to “
” and the partial utterance information u is set to U−v (u=U−v), in other words, the partial utterance information u is updated to “
”. These information are provided to the utterance difference calculating unit 105, and from these information, the utterance difference calculating unit 105 calculates difference information (U−v)/U=(14−5)/14≈0.64. The resulting difference information is displayed on the display area 511, and the sending unit 104 outputs the translated example “I want to go to the airport.” and the difference information “0.64” to the PDA 502. These information received by the receiving unit 106 of the PDA 502 are displayed by the utterance outputting unit 107 on the display areas 701 and 702, respectively, as shown in
Next, in the case where at Step S310 the partial utterance information u is “” and then the processing returns to Step S303, example candidates E={
,
,
} are obtained at Step 305 in the above-described manner, and the display areas 505 and 507 display at Step 306 the partial utterance information u and the example candidates E, respectively, as shown in
” in the above-described manner as shown in
” and the partial utterance information u is updated to “ ”. These information are outputted to the utterance difference calculating unit 105, and from these information the utterance difference calculating unit 105 obtains difference information (U−v)/U=(14−14)/14=0. The resulting difference information is displayed on the display area 511 and also on the display area 512. The user 2 can learn that the display area 702 now displays all that the user 1 wishes to say. Thus, it is possible to objectively judge to reply to the display area 702, so that the user 2 can replay to the user 1 with minimum utterances.
Note that in the above description, the difference information is represented by a numeric value as one example, but may be expressed by a figure such as bars 1201 and 1202 as shown in
Second Embodiment
Note also that, as shown in
Third Embodiment
Note also that, as shown in
In such a structure, when the difference information sent from the partner's dialog supporting device is not 0, the utterance outputting unit 107 may use the difference information as a control parameter for generating regularly or at random a sentence corresponding to a filler, such as “”, “
”, or “And please wait,” in the filler database 1801 and present the user with the generated sentence until the user's dialog supporting device receives a next example from the partner's dialog supporting device.
In this structure of
Although the first to third embodiments have been described assuming that the user speaks Japanese and the partner user speaks English when the dialog supporting devices are used, the present invention can be applied to dialog in any languages such as the following situation where a user speaks Chinese and a partner user speaks English. In this situation, in the same manner as shown in
” of utterance No. 16 in language 1 is selected by the sentence selecting unit 103, an associated translated example “I want to go to the airport.” in lo language 2 is outputted.
For instance, as shown in ”, the utterance is transformed by utterance recognition processing of the utterance receiving unit 102, into utterance information represented by a character string “
”, and then displayed on the display area 505 as shown in
” at Step S301, partial utterance information u is set to “
”, already-translated utterance information v is set to “ ” at Step S302, and a mode of the PDA 502 of the partner user 2 is changed to a receiving state via the sending unit 104 of the user's dialog supporting device. At the receiving state, a display area 701 displays a translated example of the last example which has recently been selected by the partner's dialog supporting device, and a display area 701 displays, as histories, translated examples of all examples which have already been selected sequentially by the partner's dialog supporting device. Note that, by pressing the utterance input button 504 to start inputting of an utterance, the receiving state of the PDA 502 is released and back to the state of
Next, by gradually shorting the partial utterance information u, the example selecting unit 103 calculates each similarity degree between the partial utterance information (remaining utterance information) u and each of the examples at Steps S303, S304, S405, and S309, in order to eventually obtain example candidates E={,
} corresponding to the remaining partial utterance information u=“
”. At Step S306, a value of the partial utterance information u is reflected in displaying on the display area 505, by being underlined as shown in
” as an example e, at Step S307 a translated example “I want to go to the airport.” is provided to the sending unit 104 to be sent. At Step S308, already-translated utterance information is set to “
” and partial utterance information u is set to U−v (u=U−v), in other words, the partial utterance information u is updated to “
”. These information are provided to the utterance difference calculating unit 105, and from these information the utterance difference calculating unit 105 calculates difference information (U−v)/U=(14−4)/14≈0.71 using the number of characters. The resulting difference information is displayed on the display area 511, and the sending unit 104 sends the translated example “I want to go to the airport.” and the difference information “0.71” to the PDA 502. These information received by the receiving unit 106 of the PDA 502 are displayed by the utterance outputting unit 107 on the display areas 701 and 702 as shown in
Next, in the case where at Step S310 the partial utterance information u is “” and then the processing returns to Step S303, example candidates E={
,
,
?} are obtained at Step 305 in the above-described manner, and the display areas 505 and 507 display at Step S306 the partial utterance information u and the example candidates E, respectively, as shown in
As shown in ?” in the same manner as descried above, a translated example “What time does the train leave?” is displayed on the display area 701 and 702 and also outputted as utterance from the PDA 502 as shown in
”, and the partial utterance information u is updated to “ ”. These information are provided to the utterance difference calculating unit 105, and from these information the utterance difference calculating unit 105 calculates difference information (U−v)/U=(14−14)/14)=0. The resulting difference information is displayed on the display area 511 and also on the display area 512. The user 2 can learn that the display area 702 displays all the user 1 wishes to say. Thus, it is possible to objectively judge to reply to the display area 702, so that the user 2 can reply to the user 1 with minimum utterances.
Note also that in the above description, the difference information is represented by a numeric value as one example, but the difference information may be expressed by a figure such as bars 1201 and 1202 as shown in
Note also that, as shown in
Note also that, as shown in ”, “
”, or “And please wait,” in the filler database 1801 and present the user with the generated sentence until the user's dialog supporting device receives a next example from the partner's dialog supporting device. In the structure of
Fourth Embodiment
Note also that, as shown in
Fifth Embodiment
Note also that, as shown in
Note also that the present embodiments have been described, as one example, the cases of dialog between Japanese and English, and dialog between Chinese and English, but the present invention can be applied to dialog in any other languages and does not depend on specific languages.
Although only the exemplary embodiments of the present invention have been described in detail above, those skilled in the art will be readily appreciate that many modifications are possible in the exemplary embodiments without materially departing from the novel teachings and advantages of the present invention. Accordingly, all such modifications are intended to be included within the scope of this invention.
The dialog supporting device according to the present invention has a function of speedily completing utterances of dialog participants, and is suitable for a translation application software and the like used in a mobile telephone, a mobile terminal, and the like. Further, the present invention can be applied to a public street terminal, a public guide terminal, and the like. Furthermore, if the present invention is used for dialog between the same language, the present invention is also applied to a chatting system using fixed sentences.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2005-197923 | Jul 2005 | JP | national |
This is a continuation application of PCT application No. PCT/JP2006/311786 filed Jun. 13, 2006, designating the United States of America.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6523000 | Ando et al. | Feb 2003 | B1 |
| 6535842 | Roche et al. | Mar 2003 | B1 |
| 20020120436 | Mizutani et al. | Aug 2002 | A1 |
| 20040002848 | Zhou et al. | Jan 2004 | A1 |
| 20050131673 | Koizumi et al. | Jun 2005 | A1 |
| 20050149316 | Ushioda et al. | Jul 2005 | A1 |
| 20060004560 | Whitelock | Jan 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| 2000-242642 | Sep 2000 | JP |
| 2003-030187 | Jan 2003 | JP |
| 2003-288339 | Oct 2003 | JP |
| 2005-107597 | Apr 2005 | JP |
| 0049523 | Aug 2000 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20070244687 A1 | Oct 2007 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2006/311786 | Jun 2006 | US |
| Child | 11785368 | US |
