Patent Application
20250110983
The present disclosure relates, at least in part, to systems and methods for generating and providing media content and, more particularly (but not exclusively), to systems and methods for generating and providing media content based on contextual events regarding a vehicle.
Mapping and navigation systems can provide media content during a trip to an occupant on-board a vehicle. For example, in one approach, a system provides content to occupants based on their vicinity to a particular place associated with the content. However, such content may have little relevance to the actual situation of the occupants or the vehicle. This irrelevant content can cause annoyance to the occupants. Consequently, such content suffer from low user engagement and negatively affect occupant's experience. Additionally, providing content in this manner would take up a significant amount of processing and navigation resources which reduces overall system efficiency without any real benefit to the occupants.
It is therefore a first non-exclusive object of the disclosure to provide a method and a system which solves the above problem. A method and a system disclose herein may facilitate providing media content to an occupant that is relevant and addresses a specific, real need of the occupant or the vehicle. This in turn may improve system efficiency for the provided media content while also enhancing user experience. Specifically, some of the disclosed techniques leverage extensive data that is only available locally at the vehicle (e.g., OBD or ECU data). Historically, such data has not been utilized to tailor content delivery to a particular user or vehicle. Rather, such data has typically been used for diagnostic purposes.
According to a first aspect, there is provided a method comprising: obtaining first data associated with a current use state of a vehicle, analysing the first data to identify a contextual event regarding the vehicle, in response to identifying the contextual event, requesting second data comprising media content data, generating, based on the media content data, a media content item relevant to the identified contextual event, and providing the media content item for consumption by an occupant of the vehicle. The use state may be a state relating to an operational parameter of the vehicle and/or occupant parameter. Operational parameters may include at least one of vehicle sensor output, Vehicle Identification Number (VIN) data, navigation data, one or more codes read from an Electronic Control Unit (ECU) or On-Board Diagnostics (OBD) port etc. Occupant parameter may include the state of the occupants, e.g., measured from a vehicle sensor, and/or taken from a user device in operational communication with the vehicle and/or a server. The contextual event may be an incorrect tyre pressure, certain fuel status, light bulb status, air conditioning status, desired food/drink by the occupants, desire to visit certain clothing store.
In an embodiment, the first data comprises data indicative of a state of the vehicle (e.g., indicative of a state of vehicle motion, a state of one or more vehicle components, a state of one or more vehicle passengers, a state of one or more detectable objects or people external to the vehicle, a state of environmental or ambient conditions external to the vehicle, etc.). For example, in an embodiment, the first data comprises at least one of location data, velocity data, diagnostic data, vehicle control data, physical health data, mood data, speech data or image data indicative of a state of occupants of the vehicle.
In an embodiment, the method comprises determining a severity and/or urgency of the contextual event, and generating the media content item relevant to the identified contextual event based on the determined severity and/or urgency. The severity and/or urgency may be determined based on a threshold value and/or a predetermined scale. The advantage of determining severity and/or urgency is that a priority level for addressing a specific need associated with an occupant or the vehicle may be determined and a media content item most relevant to the identified contextual event may be generated. For instance, when the contextual event relates to a mechanical issue with the vehicle, the severity and/or urgency may be determined as high or maximum, thus generating a media content item specifically relevant to the identified contextual event may be prioritised.
In an embodiment, the method comprises identifying a severity and/or urgency value for the contextual event, and comparing the identified severity and/or urgency value with a pre-determined threshold value.
In an embodiment, the method comprises identifying a severity and/or urgency value for the contextual event, and comparing the identified severity and/or urgency value with a predefined severity and/or urgency scale comprising a minimum value and a maximum value.
In an embodiment, the method comprises determining whether the occupant of the vehicle is a passenger or a driver of the vehicle, and determining the format of the media content item based on whether the occupant of the vehicle is a passenger or a driver of the vehicle. The format may be an audible format and/or a visual format. Additionally, the format may be an interactive format. This may facilitate delivery of the media content in a manner that is least distractive to a person such as a driver or delivery of the media content in a manner that is most engaging for a passenger of the vehicle.
In an embodiment, the method comprises receiving at least one query from the user with respect of the media content item, analysing the at least one query based on a machine learning model, providing a response to the user query based on the output of the machine learning model, and generating a variant of the media content item based on the at least one query and/or the output of the machine learning model, sorting the generated variant in a media content item data set, collecting data indicative of a response by one or more occupants to the generated variant. The newly generated variant of the media content item may include the response to the query so that the new variant pre-emptively answers an occupant's query when it is delivered in future.
In an embodiment, the method comprises detecting an error code associated with the vehicle in the first data, wherein the error code is indicative of a fault in the vehicle. This may allow proactive detection of a fault in the vehicle and prompt guidance to a place of business where the fault can be remedied.
In an embodiment, the second data comprising media content data comprises a media content queue sequence. The advantage of a queue sequence is that an order by which the media content is to be delivered may be determined based on the identified contextual events.
In an embodiment, the method comprises modifying the media content queue sequence based on a determined severity and/or urgency of the contextual event. This allows providing media content associated with a severe and/or urgent contextual event out of order from the predetermined sequence so that needs of the occupants can be addressed dynamically and promptly.
The present disclosure provides systems and methods for providing media content for consumption to occupants of a vehicle. The media content may be provided in audio format (e.g., while the vehicle is being driven), and the media content may be provided in video format (e.g., while the vehicle is stationary).
“Media content” may refer to multimedia information that may be transmitted, received, stored, or output (e.g., displayed) in a manner consistent with the described techniques. Media or multimedia information may be or include a combination of content forms that may be played or provided in a presentation (e.g., for consumption by a user), such as text (e.g., raw text or hyperlinks), audio (e.g., speech or music), image(s), video(s), scene data or models for rendering 3D scenes, 3D renderings (e.g., rendered from scene data), or haptic information for generating haptic feedback. When provided by way of an output device (e.g., a display, speaker), media content may include consumable or observable audible, visual, or tactile aspects. Media content may be or include interactive media that enables a user to control or manipulate the way the interactive media is presented (e.g., video games). Media content may be delivered for real-time output (e.g., live streamed), or for storage and subsequent retrieval and output. Examples of media content include movies; shows; recordings, streams, or broadcasts of events (e.g., sporting events, concerts, etc.); video clips (e.g., available via social media); video games (e.g., including cut scenes); or extended reality content.
At step 11, a first data associated with a current use state of a vehicle, such as the vehicle 102 shown in
In step 13, the first data is analysed (e.g., by the control circuitry 101 of the system 1001 shown in
In step 15, in response to identifying the contextual event, second data comprising media content data is requested (for example by a system implemented in the vehicle or a remote server such as a Demand Side Platform (DSP)). In an embodiment, the control circuitry 101 shown in
In step 17, a media content item relevant to the identified contextual event is generated based on the media content data. The generated media item may be generated based on one or more data set stored locally or remotely. In an embodiment, the control circuitry 101 shown in
In step 19, the media content item is provided for consumption by an occupant of the vehicle (e.g., by the control circuitry 101 and/or at the output system 117 shown in
The operational parameters may include at least one of location, velocity, or radius parameter to a particular point of interest or region of interest. As illustrated in
In some embodiments, at least some of these data may be obtained and/or determine by analysing error codes that are readable from the OBD-II port. The operational parameters and/or the occupant parameters may be utilised to order, filter and/or prioritise certain media content for providing to occupants. The operational parameters of the vehicle provide a perspective from the vehicles point of view as to the current context and state of the vehicle. Generally vehicles have more than 100s of ECUs and data is collected from them continuously to understand the state of the vehicle. Some of these feed into error codes that are readable from the OBD-II port. In some embodiments, data from OBD-II port is collected and transmitted by using a wired connection and/or a wireless communication to the vehicle's infotainment headend.
As one example, one or more attachment devices may be plugged into OBD-II port and the data may be read wirelessly via any appropriate wireless communication standard (such as Bluetooth (BT)). By using BT, data or codes from the OBD-II port can be made available to the vehicle infotainment headend. This may require a conversion process that can leverage a BT OBD-II port scanner.
In some embodiments, other communication standards such as Wi-Fi versions may be used. Alternatively, a basic connection via a Rasperry Pi, and an open source PhyOBD OBD-II (SAE-J1979) compliant scan tool software may be written entirely in Python which can be utilized to obtain the readings. Codes or data obtained in this manner can be broad, anything from a faulty catalytic converter to a burned out tail light. Moreover, data from other sensors such as tire-pressure monitoring system can also be aggregated using these methods.
In some embodiments, all ECU codes may be communicated via Controller Area Network buses, Ethernet Bus or FPD Link within the network and eventually be made accessible to the headend that navigation systems interact with.
In some embodiments, media content data requests are sent from the vehicle when error codes are detected (by mapping codes to particular need or business type) as well as other basic needs such as low fuel tank level or tire pressure below acceptable limits. Media content data associated with each request may be stored in a queue to be provided for consumption by occupants.
In some embodiments, a level of severity and/or urgency of each requests may be determined. The level of severity and urgency may be determined by a vehicle head end unit or a supply server. In one example when a serious issue such as an issue with the vehicle lighting system or an engine issue is detected, a media content data including information regarding a repair shop will be more appropriate and thus will have a higher priority in the queue. Any identified contextual event indicative of such serious need from the vehicle may lead to prioritizing media content data that can help with more serious issues. Thus, a port reading may be converted to a media content request based on identifying a contextual event (such as incorrect air pressure in tires etc.). In another example, low tire pressure means that the severity is high, and a media content data associated with such low tire pressure may be pulled up even if its position in the media content queue is low.
In some embodiments, assessing the contextual event may be mapped based on the received codes and the severity of the request may be mapped based on where the data read falls under the acceptable range for the data. For instance, when a recommended tire pressure range is 29-30 psi and the current tire pressure is 21 psi, a request associated with this contextual event may be determined as an urgent request. In some embodiments, a context specific threshold may be set in the system. For example, a threshold of 25 psi may be set specifically in relation to tire pressure. When a current read of air pressure in one or more tires is 21 psi, indicating that the current read value is below the threshold of 25 psi, the system may determine a request associated with this contextual event as urgent. In some embodiments, multiple thresholds may be set in the system. For example, a first threshold of 29 psi may be set as the lowest pressure in a recommended range and a second threshold pf 25 psi may be set as an urgent threshold. A “moderate” urgency may be assigned for tire pressures between the first and second thresholds. Urgent requests may, in addition to signaling prioritization of media content associated with these request (for example moving to a higher position in a media content queue sequence), signal selecting/sorting of media content data based on a radius distance from the current GPS location to a business sponsoring the media content data in order to make an attempt to find the nearest business that can assist and resolve an identified situation.
In some embodiments, any identified issues that is associated with a recall, for example a backup camera not working (against NHTSA rules) which may be detected/communicated via FPD-link may be marked as high severity by default.
Alternatively and/or in addition to the previous examples, occupant's reactions to media content items provided to the occupants may be identified by using in-cabin monitoring (e.g. computer vision) such as analyzing gestures, smiling, frowning, thumbs up/down or even subtle and simple gestures such as head nod. Such data may be associated with a media content and interpreted as the media content being liked or disliked and the navigation to the provider of the media content may be initiated or ignored.
If the vehicle is an autonomous vehicle (AV), then the drive decision to a location associated with a provided media content data may be commands issued to both the navigation and drive-by-wire system.
In some embodiments, subtle gestures may be used by drivers since they are busier cognitively due to being focused on driving, versus passengers. When allowed by the user, conversation that is happening within the vehicle cabin will also be analyzed by a microphone detection using Natural Language Processing (NLP) methods to determine media content data more appropriate. For example, when there is a conversation about food that is happening in cabin, then a media content data including information about food may be queued (i.e. prioritized) ahead of another media content data including information about a furniture store.
In some embodiments, media contents may be turned off by an occupant (such as a driver) at any time or can be filtered based on a business type. An occupant may turn off different one or more categories of media contents, for instance he/she may want to receive audio media content including information regarding gas stations, drive thru restaurants, coffee shops but not from book stores. In addition, media content being in audio format may be buffered and may only be provided when it is detected that there is no navigation instruction being currently played out and no human chatter is detected in the cabin of the vehicle.
In some embodiments, media contents may be pushed by a Supply Side Platform (SSP) and pulled by a Demand Side Platform (DSP).
In some embodiments, the order of media content in the queue may be modified based on a particular identified contextual event and optionally based on a level of severity and/or urgency of the contextual event. For instance, if the contextual event is identified as low fuel level or incorrect air pressure in tires, the media content associated with Exxon may be prioritized in the queue sequence. This may only happen if a metadata associated with the Exxon media content indicates that Exxon is a fuel station or that there is an air pump at the upcoming Exxon station respectively.
In some embodiments, a threshold value may be set as a threshold for indicating severity and/or urgency of an identified contextual event. The threshold value may be an integer value, for instance n=3. In this case, contextual events may be pre-categorized and assigned at least one severity and/or urgency value. For instance, a contextual event such as a tyre puncture may be assigned a severity and/or urgency value x=4 where x is the severity and/or urgency value and another contextual event such as wiper fluid being below a recommended level may be assigned a severity and/or urgency value x=1. When a contextual event (such as the aforementioned examples) is identified the severity and/or urgency value is compared to the threshold value and if it is determined that the severity and/or urgency value of the identified contextual event meets certain condition (such as being above the threshold value, that is x>n), then it is determined that the identified contextual event requires urgent attention. In other examples, the severity and/or urgency value of the identified contextual event may be compared to a predetermined scale to determine whether identified contextual event requires urgent attention. The predetermined scale may be a numerical scale indicating least urgent to most urgent as the magnitude of values in the scale increases, for instance the value of 0 being least urgent and a maximum value of 100 being most urgent. It should be appreciated that the examples provided herein are merely exemplary and non-limiting and in other embodiments, the threshold value may have other types or forms of values such as non-integer value, Boolean value etc.
In some embodiment, dynamic premium is used for prioritizing certain media content. The Supply Side Platform may charge a higher premium since the Supply Side Platform will be bidding specifically for a media content that either was not already in the queue or had a lower position in the queue. For example, when the in-cabin monitoring system detects chatter about tacos, a media content associated with a Mexican restaurant or Taco Bell may be desired and since it was in the buffered queue, SSP may ask for a higher bid price.
In some embodiments, the in-car monitoring system is used to determine whether an occupant of the vehicle is a passenger or a driver of the vehicle, and the format of the media content item is determined based on whether the occupant of the vehicle is a passenger or a driver of the vehicle. For example, the in-car monitoring system may use images obtained from a camera in the cabin to determine whether the occupant of the vehicle is a passenger or a driver of the vehicle. Alternatively other methods such as audio triangulation using multiple microphones may be used to determine the location of the occupant in the cabin and consequently determine whether the occupant is a passenger or driver. The format may be an audible format and/or a visual format. An audible format may be more suitable when the occupant is a driver, as audible media contents do not require the driver to look away from the road. However a visual format may be more suitable when the occupant is a passenger because visual formats such as a video clip may be most attention grabbing and engaging for the occupant.
In some embodiments, a response period is set so that after a media content item is provided to occupants of a vehicle the head unit listens for a response from the occupants. The response may be an immediate explicit verbal skip/next, navigate to the business or stop command. Stop means stopping all media contents after that point in time. If there is no response received during the response period, it is assumed that the occupant(s) were not interested in that specific media content and the next buffered media content will be in line for delivery. Visual response can be also collected from the vehicle occupants during the methods described above.
In some embodiments, a limit on the rate at which media content is provided to occupant(s) is set. The limit may be set for the duration of a drive (since the destination is known based on navigation system). This may insure that a frequency of the media content provided to the occupants is reasonable thus balancing a nuisance factor for the occupant(s) and the desire of the fee paying sponsor to provide media contents.
In some embodiments, a media content in video format may be converted to audio only format. information can be extracted from the media content in video format (for example by analyzing frames of the video) and converted to an audio only format suitable for providing using speakers in the vehicle. In some embodiments a video stream (i.e. a media content in video format) is split into video and audio data and only the latter is rendered. Alternately a text-to-speech is used to produce audio data base don a sub-title portion of the video stream.
In some embodiments, occupants such as driver or passengers may interact with a media content provided to them in the vehicle. As one example, the occupant(s) may ask questions about the content or information included in the media content, for instance when the media content provides information about a particular business, the occupants may request further information about the business. Returning to the Exxon example, the questions may be, e.g., “Is there an ATM at the station?,” “Is there an air pump?, “Do you have ethanol free unleaded gasoline and at what price?,” “Do you accept Apple Pay?,” “Is there a restroom?,” “Is the restroom wheelchair accessible?,” etc.
Queries from the occupants about a media content item may be transmitted to a Chat Agent of a server that will provide appropriate answered to the queries. In one example, the questions may be analysed and answered by using a Business specific Language Model (BLLM) using data sets provided by a particular business. Business specific data sets and the BLLM model/chat agent may be abstracted from each other. The BLLM may be invoked via an API accessible by the server. Optionally the BLLM model/chat agent may be accessible by other applications via an API server.
In some embodiments, information about a place of business may be accessed or obtained from available sources on the internet. For example, information may be obtained from the Google Map business details, or the review/comment sections populated by other users.
The driver of the vehicle may not read that sort of information while driving nor would they have time to do so to make a selection of whether they want to direct the navigation to a business. Therefore, the systems and methods according to the present disclosure may facilitate providing information to the driver without causing distraction.
In some embodiments, a specialized BLLM specific to a business may be used to avoid funky or biased answers (generally seen for large models). Domain-specific pre-training, model alignment, and supervised fine-tuning may be used to ensure that user questions about a specific business can be answered in an expert fashion. As one example, a McDonalds specific BLLM may be used which is an expert on all McDonalds related questions such as meals, burgers, drinks, prices, business hours, drive thru specifics but will not be able to answer and even ignore questions about avocados or climate change.
In some embodiments, when a statistically high number of questions are coalescing around the same topic, for example 20% of the questions are related to the presence of an ATM, a media content that is intended to be delivered to occupants in the future may be modified to include the answer to that question. In one example, the information relating to the most asked question is simply appended to the original media content.
In another embodiment, an answer to a question may be included in the media content based on comments from other users in a review section. For instance, if multiple comments make a referral to commenters' views about the food rating of a particular place of business when the asked question concerns food rating, the media content may be modified to include information about food rating.
In some embodiments, a media content may be modified based on a general profile of the driver/passengers. If the driver often asks a fuel price after a media content containing information about a fuel station is provided, a modified version of the media content that includes the price at the fuel station will be generated and delivered to vehicle before the particular question is asked. This will require a metadata field associated with the media content which is dynamic that keeps the fuel price and updates its value by checking the provider's current data store. In some embodiment, the BLLM may define what metadata fields will be dynamic and need to be updated per time, per location and context.
In some embodiments, a media content may be modified based on the condition of the vehicle or the other contextual parameters. For instance, if the vehicle had incorrect tire pressure (based on data collected from the vehicle), a media content will be modified to include information about the presence of an air pump equipment at a particular place of business. In another example, when there is a big lottery prize currently, a media content can be modified to include the amount of the lottery prize. This means, a business may have several versions of the base media content and multiple variants that were generated based on interactions of users with a Chat Server and the BLLM model using the data store and enhanced data models of the business.
In some embodiments, interaction with the Business LLM Chat Agent can be done not only by voice but also on a smart device chat interface, such as a smartphone. This assumes that smart devices that are in cabin are wirelessly connected to the headend (for example using Bluetooth) and also receive the media content. In this embodiment, they will be able to respond via text using the BLLM's chat interface. In some embodiments, this functionality may be limited to only the passengers. In another related embodiment, BLLM chat interface may process the requests from the aggregate driver and passengers, collate votes if needed and answer them accordingly. In another embodiment, queued media contents may pull in variants of the media content based on the questions answered by the chat agent of BLLMs from the vehicles that are in the same vicinity of one vehicle, based on GPS measurements. For instance, if a traffic jam exists at a location, and questions are received by a plurality of vehicles relating to a coffee shop in an upcoming rest stop, a first vehicle may be queued with media content variants which answers these very questions pre-emptively.
Additionally, a business can monitor the effectiveness of each media content variant (with respect to the base media content) at any time to modify their campaign by weighting more towards a variant versus another. Metadata of each variant, about how they were generated (based on what sort of driver/passenger input) will be recorded and shared as metadata. The weightings of a variant may go down if the BLLM based modified content may lose its relevance (and even eventually phased out), in terms of questions not being asked or not being popular anymore.
In a first step, OBD/ECU codes are obtained from a vehicle's ECU 123. At step 124, the OBD/ECU codes may be transmitted by using a wired connection or wireless connection to a head end unit 125 of the vehicle. An in-can monitoring system 127 implemented in the cabin of the vehicle may collect image data from at least one camera in the cabin and/or audio data from at least one microphone located in the cabin. The image data and the audio data may be associated with occupants in the vehicle. For example, data collected by in-can monitoring system 127 may indicate the occupants age, gender and other physical characteristics. At step 126, the collected data including the OBD/ECU codes and the video/audio data may be transmitted to a sever (e.g. DSP) 129. Optionally, additional data such as profile data associated with the vehicle and/or the occupants, input from the vehicle and/or the occupants, speed data, navigation specific data or other type of data associated with a current use state of a vehicle may also be transmitted to the DSP 129. The DSP 129 may analyse the data to identify a contextual event regarding the vehicle. Subsequently, at step 128, the DSP 129 may transmit a request to an Advertising Supply Sever (SSP) 131 for an audio advertisement data. At step 130, the SSP 131 then provides data including audio advertisement data to the DSP 129. In some embodiments, if the supplied data contains advertisement data in video format, the supplied data is converted to audio format as explained before. Content in video format may be played for non-driver occupants in the vehicle such as passengers, on displays such as those that are on the back of head rest or their smart devices. The DSP 129 sort and prioritize the audio advertisement data based on the identified contextual event and generate based on the audio advertisement data an audio advertisement item for delivery to the occupants of the vehicle and transmits the data to the head end at step 132. The in-cabin monitoring system 127 may measure the advertisement impact by analysing the reaction of the occupants to the provided advertisement item and reports metrics such as advertisement specific metrics and frequency of delivery of ads to the SSP 131, either directly or via DSP 129 as show in step 134. The SSP 131 reports the advertisement specific metrics to an Advertising Business Data Store 133 for future management and reference at step 136. The Advertising Business Data Store 133 may also allow bidding of advertisement from various sponsors and vendors at the stage (i.e. step 138) where the SSP 131 provides data including audio advertisement data to the DSP 129.
In some embodiments, after providing the audio advertisement item to the occupants, the head end unit 125 may listen for questions from the occupants for a period of time and relay any questions from the occupants to an LLM based chat agent 135 at step 141. At step 143, the LLM based chat agent 135 may analyse the question and relay an answer to the head end unit for delivery to the occupants. The chat agent 135 may also modify the audio advertisement item (i.e. base variant) to generate a new variant of the audio advertisement item which may include information about the answer provided to the occupant. At step 145, the chat agent may record the metadata of the newly generated variant and update the advertisement inventory in the SSP 131 with the new variant. At step 147, the SSP 131 may transmit the metadata associated with the variant to the Advertising Business Data Store 133. This allows measuring the user's interest to that specific variant. For instance, if the user had a more favourable reaction to a certain variant of the media content, a feedback based on this reaction may be relayed to relevant business. Optionally in future the SSP 131 may provide content media similar to a variant with positive reaction more frequently to the user. In some embodiments, the LLM based chat agent 135 may build data models using the data in the Advertising Business Data Store 133 at step 149. The process of Audio advertisement pulling and bidding and Audio advertisement pushing may then refresh and repeat upon receiving operational parameters or occupant parameters, for example is steps 151 and 153.
The processes described above are intended to be illustrative and not limiting. One skilled in the art would appreciate that the steps of the processes discussed herein may be omitted, modified, combined, and/or rearranged, and any additional steps may be performed without departing from the scope of the disclosure. More generally, the above disclosure is meant to be illustrative and not limiting. Only the claims that follow are meant to set bounds as to what the present disclosure includes. Furthermore, it should be noted that the features and limitations described in any one example may be applied to any other example herein, and flowcharts or examples relating to one example may be combined with any other example in a suitable manner, done in different orders, or done in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and/or methods described above may be applied to, or used in accordance with, other systems and/or methods.
