Patent Application
20170267251
This application relates in general to vehicular safety, and in particular, to a system and method for providing context-specific vehicular driver interactions.
Alert drivers are an essential requirement for safe roads. Unfortunately, due to constant demands of modern life, such as insufficient time for sleep, long work hours, and long commutes, many people drive even when they are too tired to focus on the road, not having the option to stay off the roads despite their tired state. Thus, one poll in the United States has estimated that 60% of adult drivers have driven while feeling drowsy, and more than one third has actually fallen asleep at the wheel in the past year. Such drivers may fail to react in time to road conditions, other vehicles, and pedestrians on the roads, and are at an increased risk of being in a potentially-fatal car accident. Such risk further increases if a driver falls asleep at the wheel entirely. Consistently, the National Highway
Traffic Administration conservatively estimates that at least 100,000 police-reported crashes in the United States are a result of driver fatigue, resulting in an estimated 1,550 deaths.
Multiple technologies exist that attempt to prevent the fatigue-related crashes, though none of them are ideal. For example, the Nad-Zapper™ Anti-Sleep Alarm includes a motion-detector wearable behind a driver's ear that sounds an alarm when the driver's head tilts forward at a certain speed, waking the driver. However, the alarm is activated only after the driver falls asleep and is at risk of losing control of the vehicle, thus failing to prevent the dangerous situation of the driver falling asleep from taking place. Further, until the next episode of falling asleep, the alarm does nothing to keep the driver awake.
Similarly, other systems, such as the driver alert system produced by Ford® Motor Company of Dearborn Mich., evaluate variations in lateral position of the vehicle, steering wheel angle, and velocity to determine if the driver has lost control of the vehicle. However, such technologies do not detect micro-sleep, sleep which lasts only a few seconds, while the car is on a straight road and may not attempt to awaken the driver before an accident occurs. Further, such systems do not attempt to keep the driver who needs to stay on a road awake before the driver actually loses control of the vehicle.
Likewise, other systems, such a system researched by Volvo® Group of Gothenburg, Sweden, perform a visual analysis of the driver using cameras in the vehicle and process images to detect signs of drowsiness in the driver's face. Upon detecting signs of drowsiness, such systems provide a warning to the driver that the driver is drowsy on the assumption that the driver will take a break sufficient enough to rest. Such systems do no directly increase the driver's alertness and as rest may not be possible for a driver in certain situations, the warnings of such systems may be ignored by the driver.
Accordingly, there is a need for a way to measure a level a driver's alertness and increase that level of alertness to allow when the driver is drowsy.
Interacting with the driver based on the driver's context can keep help keep the driver alert. The context can be determined by determining driver characteristics including driver interests and by monitoring the circumstances surrounding the driver, such as the state of the driver using sensors included in the vehicle, the state of the vehicle, and the information about the driver's current locale. The characteristics and the monitored circumstances define the context of driver. Information of interest to the driver is obtained and is used to generate actions that are recommendable to the driver based on the driver's context. The actions are used to keep the driver alert.
In one embodiment, a system and method for performing context-specific actions towards a vehicular driver are disclosed. A context of a driver of a vehicle is determined, including: determining a state of the driver; determining a state of the vehicle; and determining one or more characteristics of the driver. One or more actions are recommended to be performed by the system with respect to the driver based on the context. One or more of the recommended actions are performed.
Still other embodiments of the present invention will become readily apparent to those skilled in the art from the following detailed description, wherein is described embodiments of the invention by way of illustrating the best mode contemplated for carrying out the invention. As will be realized, the invention is capable of other and different embodiments and its several details are capable of modifications in various obvious respects, all without departing from the spirit and the scope of the present invention. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive.
Driver alertness, and consequently, safety on the roads, can be improved by customizing interactions with the driver to be based on the driver's current context.
The servers 11 further execute a context module 18 that determines that driver's context, a set of circumstances associated with the driver at a particular moment of time. The context is the represented in a context graph 19 that is stored in the context module 18 and that can be additionally stored in backup storage media 20. The context graph 19 is a semantic graph 19 (the context graph 19 is here forth referred to in the description below as a “semantic graph 19”). In addition to the state of the driver 14, the semantic graph 19 reflects the state 21 of the vehicle 15, such as the speed of the vehicle or repetitions per minute of the vehicle's engine, any objects in front of the car, conditions of the road on which the vehicle 15 is driving, and whether vehicle maintains the position with regards to lanes on a road (which can be determined using sensors within the vehicle), though other kinds of vehicles state information is also possible. In addition, the storage 20 can include spatial data 22 regarding the driver's locale, such as a particular city or county, though other kinds of geographical locales are possible. The spatial data 22 can include data about restaurants, shops, and other points of interest in the locale. The spatial data 22 can be obtained by the context module 18 from one or more webpages 21 accessible through an Internetwork 23, such as the Internet or a cellular network. The context module 18 can receive the vehicle's current location, which can be obtained using a GPS receiver built into the vehicle and transmitted via the wireless transceiver built into the vehicle 15 that can connect to the Internetwork 23, and taken identify those points of interest that are proximate to the drivers of location. The context module 18 can include the points of interest in to the semantic graph 19.
In a further embodiment, the sensors of the vehicle 15, such as the camera or other sensors, can measure a driver's load (not shown), such whether the load is high and the driver needs to focus attention on driving and should not be distraction; a normal load, when some driver attention is required for driving and addition load for conversation is permitted; or low, when the driver is parked or idling at a stoplight. The load can be measured by evaluating the eye blinking rate (which can be determined as described below), by measuring saccade, or pupil dilation, though other ways to evaluate the driver load are possible. The data from the sensors can be transmitted by the wireless transceiver to the sensors and incorporated into the semantic graph 19. The semantic graph 19 can be generated from the driver state 13, the vehicle state 21, the locale information 22, and other information, as described in commonly-assigned U.S. Pat. No. 9,208,439, to Roberts et al., issued Dec. 8, 2015, the disclosure of which is incorporated by reference. Other ways to create the semantic graph 19 are possible.
The servers 11 further execute a personal data module 24 that collects information about the driver and, based on the collected information, learns characteristics of the driver, such as current and historical interests of the driver, though other characteristics of the driver are possible; the characteristics are represented profile 25 of the driver, as further described below with reference to
The extracted data items are compared by the personal data module 24 to a hierarchy of topics 28 (which can include topics grouped in different categories) can be stored the storage 20, though other kinds of comparisons are possible. Based on the comparison, the topics in the hierarchy 28 that are associated with each data item are identified. Based on the identified topics, the data items can be classified in the uniform parameter space. In particular, in one embodiment, a representational vector can be generated from the identified topics for each of the data items. Such a vector describes the classification of the document in terms of the hierarchical topics and defines a point in high dimensional vector space unique to the content of that data item. The vectors can be weighed based on the age of the data item, with vectors for more recent data items been weighed more heavily. The weighed vectors are combined into a single vector that functions as a profile 25 of the driver in this description vector that describes the driver's current and historical interests. The fields in the vector correspond to numeric values related to the topics included in the hierarchy 28. In a further embodiment, the personal data module 24 can combine the vectors associated with multiple users to form population priors 29.
The priors 29 can be created through techniques such as clustering and unsupervised learning, though other techniques are also possible. For example, the priors 29 can be constructed through collaborative filtering rule, which can be used to make recommendations to combine the profiles 25 based on similarity. Similarly the population priors could be constructed based on the other information associated with the drivers 13, such as the age of the driver, and other data in their profile 25. Still other ways to create the population priors are possible. The population prior closest to a profile 25 of a particular driver can be used instead of the driver profile 25 for recommending actions, as further described below.
The semantic graph 19 and the profile 25 (or a closest prior) of a driver 14 are merged together by a recommender 30 executed by the servers 11 into a single vector space “current context” vector 31 representing the driver's current context, which covers both the driver's personal characteristics (such as his interests), the driver's state, vehicle state, and the locale. Further, the recommender 30 has access to a list of possible actions 32 that could be taken with respect to the driver 14. Such actions can include particular conversation patterns to be executed with the driver 14 and other actions. For example, such actions can include: conversing with the driver 14 on topics such as a social networking post made by a social networking connection of the driver 14; asking the driver 14 if the driver 14 would like to have a news story read to him or her, to hear about a particular point of interest nearby. Still other possible actions in the list 32 are possible. The possible actions are used to generate parameterized, recommendable actions 33 that can be recommended for implementation.
To generate the recommendable actions 33, the recommender 30 extracts recent data items representing current information associated with the driver from the web content 26, such as the recent social network of connections of the driver 14, recent news stories, and uses the extracted content to parameterize the possible actions. The actions 33 are further generated based on current context vector 31. Thus, for example, if a possible action 32 is having a conversation with the driver, the driver's interest indicated in the vector 31 include cooking, and the extracted current information includes a social networking post about cooking, a generated action could be a conversation about the extracted social networking post.
Each of the generated recommendable actions are represented by a characterization vector that describes the action in high-dimensional vector space. The vector space corresponds to a representation of the hierarchy of topics 28. For example, talking about a point of interest that is a Chinese/Asian Fusion restaurant might have high values in its description vector for “Asian Fusion Cuisine” and “Chinese Cuisine.” Similarly, a piece of content intended-to-be-played when the user is drowsy might have a high value for “drowsy.” If such a piece of content were also relevant to posts made by a close friend, the piece of content might also have a “CloseFriend” value close to 1, as opposed to 0 for a non-close friend item.
Further, at least some of the generated actions 32 can be associated with a triggering condition 34 that must be fulfilled before the action is implemented, as further described below. For example, an action that includes conversing with the driver 14 may not be implemented until being triggered by the driver's cognitive load being low enough to safely conduct the conversation. Similarly, conversing about a particular point of interest can be triggered by the point of interest being nearby.
The recommender 30 analyzes characterization vectors of the generated actions and ranks them. The ranking can be performed in a plurality of ways. For example, the characterization vectors can be compared based on their closeness to the driver profile 25. For example, in one embodiment, the values in the slots of the characterization vectors are multiplied by the values in the corresponding slots in the vector that is the driver's profile 25, and then these values are summed to give a score for the item. The recommender 30 compares the scores for the different vectors and ranks the vectors for the recommendable actions 33 based on the comparison. In a further embodiment, the actions can be ranked based on novelty, which describes whether the action has been done before, and recency, which describes how recently a particular action has been done before. In a still further embodiment, multiple rankings using multiple techniques can performed, with the results being differentially weighed and combined. The weights can be optimized using machine learning.
Actions 33 whose vectors are of a certain rank, such as the top two scoring vectors, are recommended by the recommender 30 for execution. In a further embodiment, other actions 33 of other ranks could be also used. The generation and recommendation of the data items can be done as described in commonly-assigned U.S. Patent Application No. 2015/0142785, published May 21, 2015, by Roberts et al., the disclosure of which is incorporated by reference, and as described in “Activity-based serendipitous recommendations with the Magitti mobile leisure guide”, by Belotti et al., CHI 2008, 5 Apr. 2008, the disclosure of which is incorporated by reference.
The recommended actions 33 are implemented by an action module 35 implemented by the servers 11. The action module 35 includes a natural language component 35 that engages into natural language conversations with the driver 14. The conversation can be performed as described in commonly-assigned U.S. Patent Application Publication No. 2015/0293904, published Oct. 15, 2015. The recommender 30 can provide the recommended actions 33 to the action module 35 in a variety of forms, such as a form of a serialized JSON objects, which, in addition to the description of the actions 33, the current state 13 of the driver 14, and any information necessary to implement the action. Thus, the provided information can include triggers 34 for taking the action at relevant engagement points (such a driver being drowsy and nearby point of interest); contextual information about driver's state of alertness (with the information being updated as the information changes in real-time) retrieved from the semantic graph 19; personal data from the profile 25 of the driver 14, and extracted web content 26 such as social networking updates and sports and entertainment which can be used to implement the recommended actions 33. The action module analyzes the driver state 13 and other provided information to recognize when a triggering condition 34 has taken place and performs a recommended action associated with the triggering condition 34 that has occurred.
The natural language component 35 can support both conversation prompts and dialog acts. A conversation prompt causes the component 36 to invoke one of the predefined patterns like reading a social networking post or playing a game. Dialog acts provide for simpler interactions from other modules such as confirming a musical selection. For example, if a recommendation is made for an upbeat tune to keep a driver from feeling drowsy, the recommender 30 can issue a request to ConfirmTune(X) where X is the recommended tune given the driver's current state and known preferences. The request causes the natural language component 36 to ask the question of the driver and provide the driver's acknowledgement or denial of the suggested music.
The action module 36 interacts with the driver 14 through a driver interface 37 located in the vehicle through the Internetwork 23 to perform the recommended action. In one embodiment, the driver interface 37 can be a software component and utilize onboard computer systems that are integrated into the vehicle 15, such as a trip or a navigational computer, rearview monitors, and other components. In a further embodiment, the driver interface 37 can include hardware components that are exclusively part of the system 10 and not used by other onboard vehicle components. The driver interface 37 can include a visual display, such as in a form of an animated ring that changes shape and opacity when the interface delivers speech, though other visual representations of the interface are also possible. Upon decision of the action module 35 to engage in a particular conversation, the action module 35 transmits the text to be spoken to the driver 14 to the agent interface 37 within the vehicle, which performs text-to-speech (“TTS”) conversion, such as using a commercially using the available TTS software like Nuance® produced by Nuance Communications, Inc. of Burlington, Mass., though other ways to perform the text-to-speech conversion. The received speech is a natural language speech. The speech is delivered through speakers (not shown) integrated into the vehicle 15. Driver 14 responses are picked up through one or more microphones connected to the driver interface 13, and can be used to further interact with the driver 14. The interface 37 performs basic thresholding and other needed audio processing on the driver speech before performing speech-to-text conversion using an appropriate speech-to-text conversion software, and sending the text to the natural language component 36 for analysis and for possibly continuing the conversation or taking another action. The interface 37 can also include other components for taking actions, such as a light that can be flashed at the user to wake the user up. Other components in the driver interface 37 are further possible.
As mentioned above, the servers 11 include multiple modules for carrying out the embodiments disclosed herein. The modules can be implemented as a computer program or procedure written as source code in a conventional programming language and is presented for execution by the central processing unit as object or byte code. Alternatively, the modules could also be implemented in hardware, either as integrated circuitry or burned into read-only memory components, and each of the servers can act as a specialized computer. For instance, when the modules are implemented as hardware, that particular hardware is specialized to perform the communications described above and other computers cannot be used. Additionally, when the modules are burned into read-only memory components, the computer storing the read-only memory becomes specialized to perform the operations described above that other computers cannot. The various implementations of the source code and object and byte codes can be held on a computer-readable storage medium, such as a floppy disk, hard drive, digital video disk (DVD), random access memory (RAM), read-only memory (ROM) and similar storage mediums. Other types of modules and module functions are possible, as well as other physical hardware components. For example, the servers 11 can include other components found in programmable computing devices, such as input/output ports, network interfaces, and non-volatile storage, although other components are possible. The servers 11 and the storage 20 can be a part of a cloud-computing environment or be dedicated servers.
Tailoring actions to be taken towards a driver based on his context and interests helps to keep the driver alert and driving safely.
A context of a driver can include multiple components.
Estimating the directions in which the driver looks can help to determine how drowsy the driver is.
Based on the driver's context and the driver's characteristics, an action can be recommended for execution with regards to the driver.
While the invention has been particularly shown and described as referenced to the embodiments thereof, those skilled in the art will understand that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.
