The present disclosure relates generally to a vehicle infotainment system and, more particularly, to an automated control algorithm for recommending and administering content delivery to vehicle occupants.
Vehicle information and entertainment systems are becoming more prevalent in the marketplace (referred to herein as infotainment systems). Beyond radios and navigation devices, today's vehicles are equipped to video players, game consoles, multiple displays, external networking connections and more. These types of features are controlled traditionally by their users. What is lacking is a more automated manner of controlling these devices and the content they deliver to the vehicle occupants.
Furthermore, certain content may not be suitable for consumption by all of the vehicle occupants or playable by all audio/visual equipment in various locations in the vehicle. In such cases, it is desirable for the system to determine the interpersonal relationships amongst the occupants and provide recommendations regarding content or delivery thereof in accordance with the interpersonal relationships of the vehicle occupants. This section provides background information related to the present disclosure which is not necessarily prior art.
An infotainment system is provided for delivering content to multiple occupants of a vehicle. The infotainment system includes: an occupant detector configured to receive characteristic data for occupants of the vehicle and generate a profile for each occupant of the vehicle; a recommendation engine that analyzes the profiles of the vehicle occupants; and a content delivery engine that deliver content to one or more of the vehicle occupants in accordance with the analysis of the profiles of the vehicle occupants.
Characteristic data for occupants may be received from one or more sources, including sensors disposed in the vehicle, an occupant's mobile computing device in data communication with the infotainment system and a personal profile of an occupant retrieved remotely from a data source outside the vehicle.
Based on the analysis of profiles, the content delivery engine can change the content being delivered and/or change delivery medium for content being delivered via the infotainment system.
In one aspect of the disclosure, the recommendation engine determine seat positions for each occupant and the content delivery engine delivers content to one or more of the vehicle occupants in accordance with the seat positions of the vehicle occupants.
In another aspect, the recommendation engine determines an interpersonal relationship between the two or more occupants and the content delivery engine delivers content to one or more of the vehicle occupant in accordance with the interpersonal relationship.
This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features. Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.
The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure. Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.
The computer memory 15 includes storage for algorithms contemplated herein, content usage history, and passenger profile information (which can include passenger history). The storage of such items can be in a database or other data structure arrangement.
Wireless transceiver 17 can include Near Field Communications (NFC), Bluetooth, LTE, HSPA+, 802.11(a, g, n), Zigbee, or other wireless communication technologies. These allow both the car to communicate with remote systems via a wide area network such as the internet and also communicate with local computing devices of vehicle occupants (smartphones, laptops, etc.). The vehicle also may include wired connections that allow local occupant computing devices to be plugged into the car for additional communication and device battery charging.
The infotainment system 10 is further configured to receive inputs from additional sources. First, the infotainment system 10 receives inputs from conventional vehicle sensors 8, such as a speedometer, an odometer, a fuel tank sensor, etc. In one example, weight of a vehicle occupant is determined from weight sensors disposed in the each vehicle seat. The weight sensors may be part of a vehicle safety subsystem and interfaced via the vehicle bus or internal communication network 19 to the infotainment system 10. In another example, a sensor is placed in the headrest of a seat in order to detect height of a user. The headrest sensor can also be used to detect and/or sync up with child seats for detection of additional occupant characteristics. Inputs from other types of sensors are contemplated by this disclosure.
The vehicle may be further configured with additional sensors 6 for capturing characteristics of vehicle occupants. For example, the vehicle may be equipped with one or more cameras that can be used to detect facial images of the vehicle occupants. Facial recognition algorithms can then be applied to the facial images from the cameras to determine an identity or mood of a vehicle occupant. In another example, microphones positioned proximate to each seat can captured voice input. Voice recognition algorithms can then be applied to captured voice data to determine the identity, tone, mood, or linguistic background of the vehicle occupant. In yet another example, doors of the vehicle may be equipped with capacitive sensors that can read capacitive signatures of persons opening the door. Other types of biometric sensors and sensing schemes may be employed to determine the identity of vehicle occupants. It is understood that the different sensors discussed above may interface directly with the system or via the vehicle bus depending upon whether the sensors are provided by the vehicle manufacturer or are aftermarket components.
With reference to
Upon detecting the new occupant, the control algorithm will first identify the occupant and retrieve any available characteristics about the occupant as indicated at 32. Occupant characteristics can be retrieved from a variety of sources. For example, characteristics for an occupant can be determined or derived from input from one or more in-vehicle sensors as noted above.
In another example, the occupant may carry a personal mobile computing device, such as a phone, a smartphone, a portable media player, a handheld global positioning device, a laptop computer, etc. Such computing devices may be queried by the infotainment system to gather information about the occupant. In some instances, the queried information can be identifying information for the device user. Queried information may further include media content residing on the device, such as music, movies, podcasts and the like, as well as applications residing on the device or used by the occupant, such as newspapers, games, social networks, and the like. The infotainment system may also query past media consumption including partial consumption (e.g., two of three rear seat passengers have started watching but did not finish a particular TV episode). Mobile computing devices may also provide phone contacts, email, and schedule of occupants. Schedule of occupants is important in the vehicle context as the control algorithm can suggest preparatory content or activity for upcoming meetings or next up event. For example, if a meeting is upcoming in 30 minutes and the meeting request in the person's calendar has an attached agenda or other content, the control algorithm can use such information to give high priority to display or output of such related and time sensitive information. Mobile computing devices may be interface with the infotainment system via a wireless data link (e.g., Bluetooth) or a wired connection to an input port of a docking station. The use of docking stations is a good technique for associating the user of the mobile device with the user's location in the vehicle.
In yet another example, characteristics for an occupant can be determined or derived from a personal profile or other information retrieved from a remote data source, such as a web server, outside the vehicle. Persons routinely post information about themselves, such as who they work for, movies they like, food or restaurants they like (which the control algorithm can use to suggest restaurants on the planned travel route), other persons they know and the like, on blogs or social networking sites. In one implementation, such information can be retrieved using a search engine via a cellular or satellite data link by the infotainment system. Another of user characteristics accessible online are comments and logs of previous locations they have been to or services they have used on review sites or geotagging websites. Other types of data sources are contemplated by this disclosure.
One additional feature of the system is to identify occupants who may not be as tech-savvy by building the occupant history of not having or using a personal mobile computing device. A tag can be added to such a user's control algorithm history as a user who prefers simple interaction. The system may ask this type of user to input some helpful characteristics directly into the infotainment system because the system cannot locate as much about them from their device or internet profiles.
Regardless of the source type, characteristics about an occupant are accumulated and stored locally at 33 in a database for subsequent processing. Such processing includes updating a history database to keep current the user profiles of occupants and non-occupants. This provides current, past, and future (e.g. future scheduled appointments) profiles of vehicle participants. An exemplary user profile may include content such as “User=Bob”—“current preferred option=silence”—“music=metal, then rock, then blues”—“business=Panasonic”—“geologger=no”—“yelp reviewer=yes, likes Tony's pizza, dislikes Holiday Inn”—“mood=frown”—“voice=quiet”—“family member=Father” “temperature=cool”. Various types of data structures for storing user profiles are contemplated by this disclosure.
From the recently acquired occupant characteristics and stored profile information, the infotainment system can formulate at 34 suggestions regarding content selection and/or content delivery. An exemplary use case may involve a child playing loud rap music while alone in the vehicle. When a parent enters the car, music being played in the vehicle is changed automatically to a genre enjoyed by both parent and child such as classic rock. In another exemplary use case, user profiles for the vehicle occupants indicate each occupant enjoys playing a particular game (e.g., Scrabble), where the user preference was learned from a social media site (e.g., “likes” on Facebook). The system may in turn recommend that occupants play this particular game. The system may further operate to download the particular game from a software application source external to the vehicle if not currently available in the vehicle's local repository. Content suggestions are implemented at 35 either automatically or in response to user selection as further described below.
Functionality of the control algorithm may be divided amongst different software modules as shown in
In another more robust embodiment, the current vehicle status may include additional vehicle information. For example, the control algorithm may query the current graphical user interface (GUI mode of the vehicle. Various GUI modes may be tailored to specific users with skins, text size, voice vs. touch control, and other types of car friendly GUI modes. The current GUI mode provides the algorithm with an indication of what GUI mode to propose in the content suggestion hierarchy list.
The control algorithm may also query the current content being used in the car at the various input/output locations (e.g. shown in
The control algorithm may further query additional vehicle status information that may allow certain mode types to be recommended over others. Internal car temperature, external car temperature, window up/down location information, car velocity, car acceleration, car gear, navigation device destination, time of day, car layout of AV equipment, and other sensor data is collected and used to formulate content recommendations.
An exemplary scheme for maintaining the vehicle occupant map is depicted in
Given an occupant's identity, the occupant's relationship with the other occupants is retrieved or otherwise determined as indicated at 54. In one exemplary embodiment, the driver classifies the relationship with each new occupant as they enter the vehicle. In an alternative embodiment, the relationship may be determined automatically from an occupant's profile retrieved from a social network. More advanced techniques of acquiring relationship information are contemplated and may include inferring relationships from conversations between occupants using known artificial intelligent technology. An occupant's relationship may be defined in relation to the vehicle driver or owner, such as spouse, child, parent, cousin, co-worker, friend, etc. Once a relationship between two persons has been determined, this relationship is stored locally, along with other identifying information for the occupant, for subsequent use by the infotainment system.
When a new occupant enters the vehicle, the stored relationships are retrieved and used to update the relationship field for the occupant in the occupant map. If the new occupant is not identified with a certain confidence level or their relationship to the other occupants in unknown, the occupant's relationship designation may be set to a “polite” mode. In a polite mode, content delivery may be defaulted to playing jazz music at a low volume. Jazz content could be retrieved from a driver's portable device or from a local radio station. Privacy settings may be set to high so email delivery is terminated. And any online profiles containing personal information are logged off. In the polite mode, the occupant detector 62 would actively monitor and record characteristic data for the new occupant; whereas, in other modes, the occupant detector 62 may discontinue such monitoring activities once each occupant is positively identified.
Maintaining the vehicle occupant map may include building a timeline of when seats are occupied or anticipated to be occupied as indicated at 55. In addition to actual entry and exit times, the automated control algorithm may learn anticipated entry and exit times for occupants. For example, the system may learn anticipated entry and exit times for a passenger being driven home from school. Content selection and delivery can then be tailored based upon the anticipated entry and exit times. In the case of a child being picked up from school, content over an hour in duration may not be recommended for an occupant whose anticipated exit time is less than hour. Other scenarios which account for anticipated entry and exit times are readily envisioned.
Navigational information retrieved, for example, from GPS device 16 can also be used to update the vehicle occupant map as indicated at 56. For instance, a vehicle's point of origination or destination may be used to designate the atmosphere during the trip (e.g., a sober atmosphere is designated when traveling to or from a funeral home). In another instance, an occupant's current interest may be correlated to topics related to the destination.
Lastly, the occupant map may track the current interests for each occupant. In a simplified embodiment, an overall user profile is maintained for each known occupant of the vehicle. Techniques for building such user profiles are readily known in the art. When a new occupant enters the vehicle, the user profile for the occupant is retrieved at 57 and used to update the current interest field in the occupant map as indicated at 58.
In a more robust embodiment, additional contemporaneous information may be used to determine an occupant's current interest. In some embodiments, the system may listen to conversations between vehicle occupants using in-vehicle microphones as indicated at 59. From these conversations, a current topic of interest can be determined and used to update the current interest field in the occupant map. In some embodiments, the system may query at 60 the mobile devices associated with an occupant. Recently accessed content or applications on the device may indicate the occupant's current interest. For example, was the occupant recently watching a news report on a current event? If so, the current interest field in the occupant map may be updated with the current event. Current interests may also be learned by querying a calendar on a personal mobile device for activities preceding or following occupancy in the vehicle. For example, if the occupant is scheduled to attend an upcoming music concert, then the occupant's current interests may include the artist featured at the concert.
With continued reference to
The commonality of interests amongst the occupants is also evaluated at 43. To so do, current interests for each occupant can be retrieved from the vehicle occupant map and compared to each other. When a current interest from one occupant matches or is substantially similar to a current interest of another occupant, the current interest is added to a list of common occupant interests. Each interest on the list may be further assigned a relevance rating indicative of how relevant the interest is to the vehicle occupants. In one embodiment, the relevance rating is assigned based on the number of occupants having this common interest. For example, an interest common to all occupants is assigned a rating of five, an interest common to all but one occupant is assigned a rating of four and so forth with an interest common to at least two occupants having a minimum rating of one. Activities currently occurring in the vehicle may also be added to the list and assigned a high priority rating. For example, if the occupants are already watching a particular movie or playing a particular game, this would constitute a current interest to the occupants and be added to the list.
Next, the listing of interests is translated at 44 into a listing of content recommendations. In a simplified embodiment, the listing of occupant interests is compared to a directory of available content. For instance, the vehicle may be equipped with a video player and an associated database of available content which is viewable via the video player by the vehicle occupants. When an entry in the list of occupant interests correlates closely with an entry in the database of available content, the correlated content, along with its relevance rating, is added to the listing of content recommendations. Techniques for correlating interests to a directory of available content are readily known in the art.
In other embodiments, recommendation engines may be used to generate a listing of content recommendations. Recommendation engines (and recommender systems) are known in the art, for example, commercially available from Panasonic Automotive Systems Company, Amazon.com, Google and others. These recommendation engines are implemented by the processor 14 from memory 15, and can be updated from time to time via the Internet. These recommendation engines take the context and history information and provide related output of automobile settings or related content. They use neural networks, statistical modeling, semantic analysis (including vectorial semantic analysis) and continuously update based on changing global criteria (such as weather patterns, road construction, and other related features). The vehicle can include multiple recommendation engines and give each hierarchy rankings based on user selections of outputs in the past. Further, companies providing the recommendation engine could pay the vehicle owner or manufacturer for the right or ability to be the recommendation engine of choice.
Formulating content suggestions can further include a combination of neural network for sensory input interpretation, relational database query, and rule based or probabilistic modeling of the satisfaction of each occupant to see what are the most likely options of car usage. Relational databases are used to intersect user's profiles and preferences and construct a set of items of interest, and the last pass is using a decision tree style rule based algorithm or a probabilistic satisfaction function maximization to select the best candidates for adapting the services/content rendered and the human interface through which they are rendered. Before presenting or otherwise implementing content recommendations in step 46, each recommendation is checked at 45 for potential conflicts.
Second, a particular recommendation may not be supported by the available hardware in the vehicle as indicated at 73. For example, the recommendation to play a particular game may require that each participant have access to a keypad input. If one or more of the vehicle occupants does not have access to a keypad input (e.g., a person without an associated personal mobile device or sitting in the back seat), then the recommendation to play the game may be removed from the listing of recommendations.
Third, a particular recommendation may not be supported by the current seating arrangement of the occupants as indicated at 74. For example, the recommendation to play a game may dictate a preferred seating arrangement for the participants. For cooperative games, the participants may be required to sit next to each other, for example, in the back seat; whereas, other types of games, the participants may be separated from each other (i.e., one in front seat and one in rear seat). If the vehicle occupants are not arranged properly, the recommendation to play the particular game may be removed from the listing of recommendations.
Fourth, a particular recommendation may not be suitable for the known travel itinerary as indicated at 75. For example, the duration of a recommended movie may exceed the anticipated travel time to the destination. In another example, the occupant map may indicate that a child is expected to be entering the vehicle shortly. In this case, content recommendations not suitable children may be removed from the listing of recommendations.
Fifth, a particular recommendation may conflict with parental controls established by the vehicle owner as indicated at 76. For example, there may be a parental control that specifies only G rated movies or content may be delivered when the relationship amongst occupants is designated as family or kid-friendly. Content recommendations having a different rating would be removed the listing of recommendations. In another example, content delivery to a particular seat location or associated device(s) may be blocked based on such parental controls.
Similarly, a particular recommendation may conflict with privacy settings established by a vehicle occupant as indicated at 77. For example, there may be privacy settings that control delivery of email messages to the driver or another vehicle occupant. Email may be displayed on a center console when the relationship amongst the occupants is designated as family but directed to a headset worn by the driver or to the driver's mobile device when the relationship amongst the occupants is designated as business associates or something else other than family.
In an exemplary embodiment, conflict checking may be implemented using a rule-based approach, where one or more rules are defined for each conflict check. Each content recommendation is then compared at 71 against each of the conflict rules. When a particular recommendation does not comply with the rule, it may be removed from the list of content recommendations as indicated at 78 or some other corrective action may be taken to ensure compliance with the rule; otherwise, the recommendation is retained at 79 on the final listing of content recommendations. It is envisioned that each rule may further define one or more suggested corrective actions that can be taken when a given rule is not met. The vehicle owner may define a global set of rules which apply to the vehicle. Each vehicle occupant may further define rules that are applicable to themselves or to the vehicle when that occupant is the driver. While a few exemplary rules have been set forth above, it is readily understood that other types of conflict checks and associated rules fall within the scope of this disclosure.
In a robust embodiment, content recommendations may be assigned a delivery mode as further described in relation to
Each content recommendation is then checked at 102 against a repository of preset modes stored in the system. Each preset mode defines vehicle settings for implementing that mode. For example, preset modes related to delivering a movie may include an action mode and a relax mode. In the action mode, content is delivered to a display viewable to multiple occupants with the audio volume and display brightness set to high; whereas, in a relax mode, content may be delivered only to smaller displays individually associated with an occupant with the audio volume and display brightness set to low. In this way, the action mode creates a different environment within the vehicle for viewing the content than the relax mode.
For each preset mode, the vehicle settings are compared against the vehicle settings of a given content recommendation. If the vehicle settings are compatible, then the preset mode is assigned at 103 to the given content recommendation. To the extent that more than one preset mode is compatible with a given content recommendation, then each compatible present mode is associated with the content recommendation. That is, the content recommendation may be delivered in different ways (i.e., having different vehicle settings). Preset modes may be created and pre-installed in the system by the vehicle manufacturer. Alternatively, preset modes may be downloaded periodically to the vehicle from a source external (e.g., social network) to the vehicle.
Content recommendations are also checked at 104 against any historical modes stored in the system. Historical modes are delivery modes that have been previously implemented by the system and are typically defined by the vehicle owner; these historical modes also define vehicle settings and are assigned at 105 in the same manner as described above. If none of the preset or historical modes are compatible with a given content recommendation, then the content recommendation may be assigned a suitable default mode at 106. Default modes may be defined for the different types of content. That is, there may be a default modes for delivering music, movies, email, driving directions, etc.
Content recommendations may be implemented automatically by the content delivery engine without input from the driver or another vehicle occupant under certain criteria as indicated at 82. For example, when the listing of content recommendations contains a single entry, the recommendation may be implemented automatically at 83 by the content delivery engine. Assuming all of the vehicle occupants have a common interest in jazz music, a corresponding recommendation to tune the radio to a local jazz station is implemented by the content delivery engine. In another example, two or more recommendations which are highly correlated to the interests of the vehicle occupants may be implemented automatically without input from the driver. Other criteria may be defined for implementing recommendations automatically without input from the vehicle occupants.
Conversely, the listing of content recommendations or a portion thereof may be presented for user selection. To do so, a determination is made at 84 as to whom will provide selection input. In some instances, only a single occupant is presented with the listing of recommendations as indicated at 86. For example, priority may be given to the driver, a parent regardless of seat location or the boss. Depending upon a privacy setting for the selector, options may be presented visually on a display in the center console or in a private manner such as audibly via a headset worn by the selector.
Alternatively, the listing of recommendations may be presented to two or more vehicle occupants. In this instance, the content delivery engine evaluates at 87 whether a suitable interface is available to present to the listing of recommendations to the vehicle occupants. Recommendations may be presented to occupants having a personal mobile device or another type of user interface (e.g., keypad and display) associated with their seat location. If such an interface is available for the applicable occupants, the listing of recommendation may be presented individually to each occupant as indicated at 88. It is understood that certain vehicle occupants may be excluded from those presented with recommendations (e.g., small children or infants). If an interface is not available for one or more of the applicable occupants, then the listing of recommendations is presented collectively to the occupants (e.g., on a shared display) as indicated at 89.
Occupants provide input as to which recommendations they prefer. Given input from the occupants, the content delivery engine 66 selects one or more options from the listing of recommendations. In one embodiment, the content delivery engine 66 selects the option receiving the most votes as shown in
Privacy features are one important aspect of this system. Exemplary use cases are further described in relation to
Continuing with the scenario above, the new occupant may not be recognized by the system but is recognized by the driver as a co-worker. If the driver is not concerned with privacy of the email message or would like to share the contents with the new occupant, the driver may supply a gesture command to the personal portable device (see
In another exemplary scenario, the driver and passenger may be viewing different context on a dual view display as shown in
Location specific recommendation is another important aspect of this system.
In another exemplary use case, the system may determine that a group of co-workers have entered the vehicle near lunch time. Given each occupant food or restaurant preferences as determined from their user profile, a listing of restaurant choices may be generated that is most appealing to the particular grouping of co-workers. The listing of restaurant is presented to the vehicle occupants as shown in
In yet another example, the group of co-workers may share a common client meeting which is learn by the system from an occupant's calendar. The meeting event on the calendar may further include a presentation document. Given the relationship between the occupants and the common interest in the meeting, the system can then operate to display the presentation on one more displays accessible to the vehicle occupants. Occupants could in turn edit, comment or rehearse the presentation while in the vehicle.
The ability to provide suggestions that cater to multiple users in a setting where the infotainment system can adapt to a single user experience or multiple experiences on individual mobile devices is an important aspect of this disclosure. Private content sessions provided by a car or third party through the car system is one way for the infotainment system to generate revenue and provide better or more useful connectivity than just the mobile device by itself.
The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.