Patent Application
20210174790
The present invention relates to an operation assistance device for vehicles that operates to obtain recommendation information suitable for an occupant composition including human relationships and provide the recommendation information for assisting various vehicle operations performed by one or more occupants.
An onboard device is known, which is configured to: collect voices in a vehicle interior; specify seating positions of occupants; estimate a speaker among the occupants present in the vehicle interior on the basis of the collected voices and the specified seating positions; estimate content of conversation on the basis of the collected voices; estimate an occupant composition on the basis of the specified seating positions, the estimated speaker, and the estimated conversation content; estimate action purposes of the occupants on the basis of the estimated conversation content and the estimated occupant composition; and determine a recommended service on the basis of the estimated occupant composition and the estimated action purposes (Patent Document 1). Specifically, the onboard device operates to: identify individuals from voiceprint patterns of the collected voices; specify the owner from the seating positions and boarding frequencies of the individuals who can be identified; specify the relationships with the identified individuals from the collected voices using a conversation keyword; register the relationships as speaker pattern data; and estimate the occupant composition using the registered speaker pattern data.
[Patent Document 1] JP2012-133530A
In the above prior art, however, data has to be collected several times in order to create highly accurate speaker pattern data, and there is thus a problem in that it takes a considerable time to provide an optimal recommended service (also referred to as recommendation information) to the occupants.
A problem to be solved by the present invention is to provide an operation assistance device for vehicles that operates to obtain recommendation information suitable for an occupant composition including human relationships in a short time and provide the recommendation information for assisting various vehicle operations performed by one or more occupants.
The present invention includes preliminarily acquiring conversation voice data of a plurality of persons to specify speakers, analyzing the acquired conversation voice data for each of the specified speakers to extract a predetermined keyword, specifying a speech pattern of each of the speakers on the basis of the keyword for each of the speakers, specifying conversation content on the basis of the conversation voice data of the plurality of persons, and quantifying and obtaining direct human relationships among the plurality of persons from the specified speech pattern and the specified conversation content. Then, when the plurality of persons gets on a vehicle, assistance information for a vehicle operation determined to be recommended is determined from the persons and the quantified human relationships. The above problem is thus solved.
According to the present invention, information on the human relationships obtained before boarding is used; therefore, the recommendation information suitable for the occupant composition including the human relationships can be obtained in a short time and can be provided in a timely manner for assisting various vehicle operations performed by one or more occupants.
Hereinafter, one or more embodiments of the present invention will be described with reference to the drawings. The operation assistance device for vehicles 1 according to one or more embodiments of the present invention operates to obtain recommendation information suitable for an occupant composition including human relationships and provide the recommendation information for assisting various vehicle operations performed by one or more occupants. Although not particularly limited, in an example for facilitating the understanding of the present invention, when the occupants of a vehicle VH1 are composed of an occupant A and an occupant B and both the occupants A and B are in a human relationship between fishing partners, a destination suitable for a fishing spot is displayed as an option or automatically set as the destination on an onboard navigation device, or a radio of a fishing program is displayed as an option or automatically played on an onboard audio device. Additionally or alternatively, when the occupants of a vehicle VH2 are composed of an occupant C and an occupant D and both the occupants C and D are in a human relationship between a boss and a subordinate of the same company, a destination such as a business trip place or a restaurant for lunch is displayed as an option or automatically set as the destination on an onboard navigation device, or a radio of an economic program is displayed as an option or automatically played on an onboard audio device.
As used herein, the term “human relationship” refers to a relationship between a specific person and another specific person determined by the present or past experiences in the social life. Although not particularly limited, in an example for facilitating the understanding of the present invention, human relationships can be classified into relationships among family members such as parents, children, husbands, and wives, relationships among relatives such as cousins, relationships of these families, relatives, and others, relationships among positions in organizations, such as bosses, subordinates, colleagues, classmates, seniors, and juniors in organizations such as companies and schools, relationships among members of the same hobby or entertainment, relationships among boyfriends, girlfriends, lovers, and other friends, and relationships among others. In one or more embodiments of the present invention, the occupant composition of a vehicle means including such human relationships.
Although not particularly limited, in an example for facilitating the understanding of the present invention, the “vehicle operations” as used herein include various operations for a vehicle performed by one or more occupants including the driver, such as a driving operation for a vehicle (such as an accelerator operation, a brake operation, a transmission lever operation, or a steering operation), an operation of a navigation device, an operation of an audio device, an operation of a car air conditioner, and an operation of adjusting a seat position, which are performed by one or more occupants.
As used herein, the “recommendation information suitable for the occupant composition” is instruction information for a vehicle or an onboard device for realizing a highly possible or preferred operation that can be considered from the human relationships among the occupants in the above-described vehicle operations performed by one or more occupants. Although not particularly limited, in an example for facilitating the understanding of the present invention, examples of the recommendation information when the occupants A and B composing the occupants are in a human relationship between fishing partners include instruction information for a destination setting operation on the onboard navigation device and instruction information for a channel selection operation on the audio device.
As used herein, the term “assistance” for vehicle operations encompasses not only presenting options to an occupant when the occupant performs manual operation, but also autonomous (automated) operations performed by the operation assistance device for vehicles 1 without manual operation performed by an occupant. In the case in which the operation assistance device for vehicles 1 operates to autonomously perform a vehicle operation on the basis of the recommendation information, when an occupant has a favorable impression on the recommendation information, the number of vehicle operations to be performed by the occupant can be reduced. When an occupant has a negative impression, the occupant can cancel the autonomous vehicle operation by performing a different manual operation than the autonomous vehicle operation.
Thus, the operation assistance device for vehicles 1 according to one or more embodiments of the present invention operates to obtain recommendation information suitable for an occupant composition including human relationships and provide the recommendation information for assisting various vehicle operations performed by one or more occupants and is characterized by preliminarily obtaining the human relationships by analysis or estimation before boarding, obtaining the recommendation information using the human relationships in a short time after boarding, and providing the recommendation information for assisting the vehicle operations.
To this end, as illustrated in
The operation assistance device for vehicles 1 according to one or more embodiments of the present invention is configured as a computer installed with hardware and software. Specifically, the operation assistance device for vehicles 1 is configured to include a read only memory (ROM) that stores programs, a central processing unit (CPU) that executes the programs stored in the ROM, and a random access memory (RAM) that serves as an accessible storage device. A micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like can be used as the operation circuit as substitute for or in addition to the CPU. The above-described human relationship analysis unit 1A, human relationship storage unit 1B, assistance information determination unit 1C, vehicle information learning unit 1D, and operation tendency storage unit 1E achieve respective functions, which will be described later, by the software established in the ROM.
First, the operation assistance device for vehicles 1 according to one or more embodiments of the present invention is based on the assumption that a plurality of persons who can be the occupants owns respective terminals TD1, TD2, TD3, . . . (also collectively referred to as a “terminal TD” or “terminals TD,” hereinafter) that are carried or used on a daily basis. The terminals TD of this type for use include a smartphone, a mobile phone, a detachable onboard device, a vehicle remote control key (such as the Intelligent Key (registered trademark)), and a voice recognition user interface (such as the Amazon Echo Dot (registered trademark)). Each terminal TD according to one or more embodiments of the present invention has a computer function. More specifically, each terminal TD has a microphone for inputting conversation voice data, a communication function for transmitting the input conversation voice data to the human relationship analysis unit 1A of the operation assistance device for vehicles 1 according to one or more embodiments of the present invention, and a position detection function such as the function of a GPS receiver for detecting the current position of the terminal TD. Each terminal TD transmits its own ID, the current position, and the collected conversation voice data to the voice acquisition section 11 of the human relationship analysis unit 1A via a wireless communication network such as the Internet.
As illustrated in
The voice acquisition section 11 operates to execute transmission and reception of information with the above-described plurality of terminals TD via a wireless communication network such as the Internet. In particular, the ID, current position, and collected conversation voice data of each terminal TD are input to the voice acquisition section 11 (step S21 of
On the basis of the ID, current position, and collected conversation voice data of each terminal TD which are input to the voice acquisition section 11, the conversation group estimation section 12 operates to estimate who are in conversation with whom and a group (cluster) of persons who are in conversation with one another in terms of the conversation voice data which is input to a specific terminal TD. In this operation, the voiceprint data of the owner of each terminal TD (or a specific person associated with each terminal TD, here and hereinafter), which is preliminarily registered, is checked to specify which conversation voice data represents whose voice. For example,
The input conversation voice data collected by a terminal TD includes not only the conversation voice data of a plurality of persons who are actually in conversation with one another but also the conversation voice data of persons who are not involved in the conversation. The conversation group estimation section 12 therefore operates to execute a combination extraction process based on the positional information of a terminal TD and a combination extraction process based on speech periods thereby to estimate the persons of a group who are actually in conversation with one another in the conversation voice data which is input to the terminal TD. That is, for a plurality of the conversation voice data that are collected by a terminal TD at the same time, the combination extraction process based on the positional information of the terminal TD is used to extract combinations in which the positional information associated with the collection of voices indicates a distance of a threshold or less, on the basis of the ID, current position, and collected conversation voice data of the terminal TD which are input to the voice acquisition section 11, thereby executing estimation of a provisional conversation group based on the positional information of the terminal TD (step S22 of
For example, as illustrated in
For one or more conversation voice data estimated to belong to the same group which is extracted by the above-described combination extraction process based on the positional information (step S22 of
For example,
The direct human relationship analysis section 13 is responsible for functions of analyzing the acquired conversation voice data for each of the specified speakers to extract a predetermined keyword, specifying a speech pattern of each of the speakers on the basis of the keyword for each of the speakers, specifying conversation content on the basis of the conversation voice data of a plurality of persons, and analyzing and quantifying direct human relationships among the plurality of persons from the specified speech pattern and the specified conversation content to obtain the direct human relationships. This analysis is executed on the basis of the conversation voice data which belongs to the same conversation group estimated by the above-described conversation group estimation section 12 (step S31 of
The keyword extraction process is a process of extracting a plurality of keywords (predetermined words), which are preliminarily registered, from the conversation voice data belonging to the same conversation group using a known voice detection process. The analysis process for the category of conversation content is a process of classifying the keywords extracted by the keyword extraction process into categories to which the keywords belong. The keyword extraction process and the analysis process for the category of conversation content are performed by referring to a category dictionary stored in the human relationship database 15.
The analysis process for the speech pattern is a process of classifying the keywords extracted by the keyword extraction process into speech patterns to which the keywords belong. This process is performed by referring to a speech pattern dictionary stored in the human relationship database 15.
In the analysis process for the speech pattern, as illustrated in
The combining process is used to quantify the human relationship between the targeted persons by combining numerical values related to the occurrence frequencies calculated through the analysis process for the category of conversation content and the analysis process for the speech pattern and store the quantified human relationship in the human relationship database 15 as a direct human relationship. As previously described, the term “human relationship” as used herein refers to a relationship between a specific person and another specific person determined by the present or past experiences in the social life. Although not particularly limited, in an example for facilitating the understanding of the present invention, human relationships can be classified into relationships among family members such as parents, children, husbands, and wives, relationships among relatives such as cousins, relationships of these families, relatives, and others, relationships among positions in an organization, such as bosses, subordinates, colleagues, classmates, seniors, and juniors in organizations such as companies and schools, relationships among members of the same hobby or entertainment, relationships among boyfriends, girlfriends, lovers, and other friends, and relationships among others.
The quantification of such a direct human relationship is performed by combining numerical values related to the occurrence frequencies calculated through the analysis process for the category of conversation content and the analysis process for the speech pattern. Although not particularly limited, the quantification is performed using a probability value or the like on the basis of a human relationship quantification map, which is preliminarily stored in the human relationship database 15, such that the probability of a relationship between a boss and a subordinate in a company organization is 70%, for example, because the analysis results of the categories of conversation content indicate that the occurrence frequency of the conversation content classified into “business” is high as illustrated in
On the basis of the quantified direct human relationship, the indirect human relationship estimation section 14 operates to estimate and quantify an indirect human relationship between unanalyzed persons among the persons stored in the human relationship database 15. The above-described direct human relationship is analyzed and quantified on the basis of the actual conversation voice data and therefore referred to as the “direct” human relationship. In contrast, the indirect human relationship estimation section 14 operates to estimate the quantification of the human relationship between persons who have not been actually in conversation with each other on the basis of the data of the quantified direct human relationship. In this sense, the human relationship between persons who have not been actually in conversation with each other is referred to as an “indirect” human relationship.
The indirect human relationship estimation section 14 operates to execute a reading process for the direct human relationships (step S41 of
The statistical process for the direct human relationships is performed with consideration for the mutual relationships among values that are quantified by the direct human relationship analysis section 13 and accumulated in the human relationship database 15. Specifically, a combination of three persons whose values of the direct human relationships are known is extracted from the human relationship database 15, and in the values of three human relationships among the extracted three persons, two values are assumed and the remaining one value is recorded. This process is statistically performed on a large number of combinations accumulated in the human relationship database 15 on the assumption that two human relationships are V1 and V2, and a probability value P (V3|V1, V2) for obtaining a remaining one human relationship V3 can thereby be calculated. This probability value P is recorded in the human relationship database 15.
The combination extraction process for unanalyzed persons is used to extract a combination of two persons who have not been actually in direct conversation with each other, as described above. For the two persons, therefore, a quantified value Vn of the direct human relationship is not stored in the human relationship database 15. For example, it is assumed that the two persons are a person Z and a person X, as illustrated in
In the calculation process for an indirect human relationship, the values V1 and V2 of the direct human relationships between the estimated relaying person and the two persons X and Z extracted by the combination extraction process for unanalyzed persons are referred to from the human relationship database 15. Then, the value Vn of the human relationship between the persons on the assumption of the values V1 and V2 of the two human relationships referred to is calculated as the value of an indirect human relationship, that is, as the value which maximizes the probability value V3 obtained by the statistical process in step S42 of
The human relationship storage unit 1B includes the human relationship database 15. As described above, the human relationship database 15 stores the voiceprint data associated with the ID of the owner of each terminal TD, the category dictionary illustrated in
The vehicle information learning unit 1D includes the vehicle information learning section 17, which executes an acquisition process for occupant information (step S51 of
The acquisition process for occupant information is a process of acquiring information as to who are on board the vehicle. For example, the acquisition process for occupant information can be used to specify an occupant when the occupant connects the terminal TD to some device equipped in the vehicle, specify an occupant by detecting that the positional information of the terminal TD is in close vicinity of the positional information of the vehicle, and/or specify an occupant by face recognition on an image acquired from a camera equipped in the vehicle. The reference process for human relationships is a process of referring to the human relationship database 15 for occupants acquired by the occupant information acquisition process to acquire the value of the human relationship between the occupants.
The acquisition process for vehicle information is a process of acquiring vehicle control information, vehicle state information, and other vehicle information. For example, the acquisition process for vehicle information is used to acquire vehicle information such as a driving operation for the vehicle (such as an accelerator operation, a brake operation, a transmission lever operation, or a steering operation) performed by an occupant, a destination that is set in the navigation device, an operation of the audio device, an operation of the car air conditioner, the current position of the vehicle, the moving trajectory of the vehicle, the current date and time, and the elapsed time after boarding.
The combining process is a process of combining the vehicle information acquired by the acquisition process for vehicle information and the information acquired by the reference process for the human relationships and storing the combined information in the operation tendency database 18 as the operation information on the human relationships. For example, provided that a person A and a person B (the direct human relationship or the indirect human relationship is V1) are on board, when the destination which is set using the navigation device is a specific fishing spot, the human relationship V1 and the destination are stored in the operation tendency database 18 together with the occurrence frequency. Persons A, B, C, etc. may be stored in addition to the human relationships.
The operation tendency storage unit 1E includes the operation tendency database 18 and operates to accumulate the human relationship and the operation information, which are obtained by the vehicle information learning section 17, in association with each other.
The assistance information determination unit 1C includes the assistance information determination section 16 and operates to specify a plurality of occupants on board the vehicle and determine the assistance information for the vehicle operation, which is determined to be recommended in accordance with the human relationships among the plurality of occupants, on the basis of the direct human relationships and indirect human relationships accumulated in the human relationship database 15. The assistance information determination section 16 operates to execute an acquisition process for occupant information (step S61 of
The acquisition process for occupant information is a process similar to the acquisition process for occupant information executed by the vehicle information learning section 17 (step S51 of
The acquisition process for vehicle information is a process similar to the acquisition process for vehicle information executed by the vehicle information learning section 17 (step S53 of
The reference process for operation tendency is a process of acquiring information on the operation performed after a determination is made that the values of the human relationships among the occupants are similar to one another or the values of the human relationships among the occupants are similar to those in the vehicle information up to the present time with reference to the operation tendency database 18 having data accumulated by the vehicle information learning section 17. For example, provided that a person A and a person B (the direct human relationship or the indirect human relationship is V1) are on board and the operation tendency database 18 accumulates the operation tendency information that the destination which is set using the navigation device is a specific fishing spot, when the human relationship is V1 or a value similar to V1, the operation tendency information for setting the destination of the navigation device to the specific fishing spot is referred to.
The determination/output process for assistance information is used to determine the assistance information for the vehicle operation, which is determined to be recommended in accordance with the human relationships among a plurality of occupants. As described above, the “recommendation information suitable for the occupant composition” is instruction information for a vehicle or an onboard device for realizing a highly possible or preferred operation that can be considered from the human relationships among the occupants in the vehicle operations performed by one or more occupants. Examples of the recommendation information when occupants A and B composing the occupants are in a human relationship between fishing partners include instruction information for a destination setting operation on the onboard navigation device and instruction information for a channel selection operation on the audio device. The term “assistance” for vehicle operations encompasses not only presenting options to an occupant using a display and/or a speaker when the occupant performs manual operation, but also autonomous (automated) operations performed by the operation assistance device for vehicles 1 without manual operation performed by an occupant.
The flow of information processing according to one or more embodiments of the present invention will then be described.
First, on a daily basis, the human relationships between a specific person and other specific persons are analyzed and quantified using the human relationship analysis unit 1A and the terminals TD1, TD2, and TD3 carried by a plurality of persons who can be occupants, and the quantified human relationships are accumulated in the human relationship database 15. Specifically, as illustrated in
Then, in step S23 of
Then, as illustrated in
Through the processes up to step S36, the quantified values of the human relationships among the persons who have been actually in conversation with one another are accumulated in the human relationship database 15, but there are also human relationships among unanalyzed persons. The indirect human relationship estimation section 14 therefore operates to execute a reading process for the direct human relationships in step S41 of
On the other hand, the vehicle information learning unit 1D operates to accumulate information as to which kind of vehicle operation is actually performed by the occupant composition including the human relationships, and this information is provided for determining the assistance information for the vehicle operation. That is, as illustrated in
Then, the assistance information determination unit 1C operates to specify a plurality of occupants on board the vehicle and determine the assistance information for the vehicle operation, which is determined to be recommended in accordance with the human relationships among the plurality of occupants, on the basis of the direct human relationships and indirect human relationships accumulated in the human relationship database 15. Specifically, the assistance information determination section 16 operates to execute an acquisition process for occupant information in step S61 of
As heretofore described, according to the operation assistance device for vehicles 1 in one or more embodiments of the present invention, the direct human relationships among a plurality of persons who can be occupants are preliminarily accumulated. When a plurality of persons gets on the vehicle, a plurality of occupants on board is specified, and the assistance information for the vehicle operation determined to be recommended in accordance with the human relationships among the plurality of occupants is determined on the basis of the direct human relationships and indirect human relationships accumulated in the human relationship database 15. Thus, the assistance information for appropriate vehicle operations can be provided in a short time after boarding.
According to the operation assistance device for vehicles 1 in one or more embodiments of the present invention, the human relationships among persons who are not actually in conversation with one another are quantified and estimated from the human relationships among persons who have been actually in conversation with one another, and errors can therefore be avoided, such as a lack of the assistance information for the vehicle operation due to inappropriate combination of occupants. Moreover, when estimating the indirect human relationships, the indirect human relationships are obtained by statistically processing the data of the direct human relationships among persons who have been actually in conversation with one another, and the accuracy can therefore be enhanced.
According to the operation assistance device for vehicles 1 in one or more embodiments of the present invention, the actually performed vehicle operation is stored in association with the human relationships at that time, and this is reflected on the assistance information for the vehicle operation; therefore, the assistance information for the vehicle operation determined to be recommended in accordance with the human relationships among the occupants can be made closer to a more realistic operation.
According to the operation assistance device for vehicles 1 in one or more embodiments of the present invention, when the same conversation group is extracted, speakers of a conversation voice data set of a plurality of conversation voice data in which the speech positions are not more than a predetermined distance are grouped into a conversation group. Thus, the accuracy of specifying the conversation group and therefore the accuracy of analyzing the human relationships can be improved.
According to the operation assistance device for vehicles 1 in one or more embodiments of the present invention, when the same conversation group is extracted, speakers of a conversation voice data set of a plurality of conversation voice data in which the speech periods do not overlap for a predetermined time or more are estimated as a conversation group. Thus, the accuracy of specifying the conversation group and therefore the accuracy of analyzing the human relationships can be improved.
According to the operation assistance device for vehicles 1 in one or more embodiments of the present invention, the conversation voice data of a plurality of persons who can be occupants is detected using a terminal capable of collecting voices even when the persons are not on board the vehicle and, therefore, the conversation voice data of a plurality of persons can be collected on a daily basis.
In the above-described operation assistance device for vehicles 1, the human relationship analysis unit 1A is configured to include the indirect human relationship estimation section 14, but the indirect human relationship estimation section 14 may be omitted as necessary. Moreover, the above-described operation assistance device for vehicles 1 is configured to include the vehicle information learning unit 1D and the operation tendency storage unit 1E and the assistance information for the vehicle operation is determined using the operation tendency information in steps S64 and S65 of
The above voice acquisition section 11, conversation group estimation section 12, direct human relationship analysis section 13, and indirect human relationship estimation section 14 correspond to the human relationship analysis unit according to the present invention. The above human relationship database 15 corresponds to the human relationship storage unit according to the present invention. The above assistance information determination section 16 corresponds to the assistance information determination unit according to the present invention. The above vehicle information learning section 17 corresponds to the vehicle information learning unit according to the present invention. The above operation tendency database 18 corresponds to the operation tendency storage unit according to the present invention.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2017/041460 | 11/17/2017 | WO | 00 |
