Patent Grant
7039551
The present invention relates generally to automotive safety and more specifically to collision warning systems based upon driver distraction levels.
According to reports by the U.S. National Highway Traffic Safety Administration, an estimated 20% of all reported automobile crashes each year involve some form of driver distraction. Thus, there has been significant interest recently regarding the issue of driver distraction, as well as concerns that many of the telematic features being planned for automobiles may lower their safety through increased driver distraction. How to measure driver distraction, how to model driver cognitive workload, and how to predict distractive effects are all open, important research issues.
Actions such as pushing a button or adjusting a knob often require the driver to glance away from the roadway in order to accurately locate the button or knob, e.g., to adjust the heating system or to change radio stations. While the driver is glancing away to perform the secondary task, the driver's attention to the primary task of driving the vehicle is distracted, and thus secondary tasks are associated with an increased driver distraction level.
Because of the high level of danger associated with driver distraction, it would be desirable to provide a system that utilizes measures of driver distraction derived from driver manipulation of controls within a vehicle to adjust various parameters of the vehicle in order to account for the driver's lack of attention to driving. It would be further desirable to provide a system that may be applied in situations beyond driving to those involving operators of various types of equipment such as manufacturing or process-related equipment.
In a first aspect, the present invention provides a method, an apparatus, and a computer program product for calculating an indicator of operator distraction for an equipment operator. In a method aspect, the invention comprises a step of receiving distraction-correlated indicator information regarding the operator from devices around an operator. After the receiving step, the invention performs a step of estimating a distraction level from the information. Subsequently, a step is performed for producing an output indicative of the distraction level in order to facilitating a threat assessment decision.
In a further aspect, in the step of producing an output, the output produced ensures that when the distraction of the operator is high, the threat assessment decision is made conservatively.
In a still further aspect, the activity information is received from at least one device selected from the group comprising vision-based sensing systems, range-finding systems, and operator-manipulated devices.
In a yet further aspect, the present invention further comprises a step of grouping the activity information into complex task types comprising sets of related activities. Subsequently, a step is performed for determining an elapsed time since the start of the first activity in a complex task. Next, a step of determining the delay time since the most recent activity in the complex task is performed. Next, the delay since the most recent activity is compared with a minimum task quiescent period, and when the delay exceeds the minimum task quiescent period, resetting the elapsed time and the delay. During the performance of a complex task, the type and the elapsed time of the complex task are compared with other information in order to produce the output for facilitating the threat assessment decision.
In a still further aspect, in the receiving step, the activity information includes a duration of a current activity.
In a further aspect, the step of correlating is performed by a controller selected from the group consisting of fuzzy rule-based controllers and rule-based controllers.
In another aspect, in the step of receiving operator activity information, the activity information is based on secondary activities performed by the operator of a vehicle.
In an additional aspect, the present invention further comprises a step of adjusting the operational parameters of the vehicle to accommodate the driver's distraction.
In a still further aspect, in the step of adjusting the operational parameters of the vehicle, the operational parameters adjusted are collision warning system parameters.
In yet another aspect, in the step of receiving operator activity information, the activity information is based on secondary activities performed by the operator of equipment selected from a group consisting of manufacturing equipment and process control equipment.
In another aspect, the present invention further comprises a step of adjusting the operational parameters of the equipment to accommodate the operator's distraction.
Each of the steps of the method discussed above typically corresponds to operations of a software or hardware module for performing the function. In other embodiments, the steps may be incorporated onto a computer readable medium to provide a computer program product. Also, the steps discussed above also correspond to means in an apparatus for calculating an indicator of operator distraction for an equipment operator.
The objects, features and advantages of the present invention will be apparent from the following detailed descriptions of the various aspects of the invention in conjunction with reference to the following drawings.
a) shows the membership functions for the elapsed time of complex radio tasks;
b) shows the membership functions for the elapsed time of complex heating, ventilation, and air conditioning (HVAC) tasks;
c) shows the membership functions for the elapsed time of radio control activations (simple tasks);
d) shows the membership functions for the elapsed time of HVAC control activations (simple tasks);
a) is a table depicting the fuzzification rules for the radio group (R);
b) is a table depicting the fuzzification rules for the HVAC group (H);
c) is a table depicting the fuzzification rules for the turn signal group (T);
a) is a graph depicting the membership functions for the varying levels of driver distraction;
b) is a graph depicting the membership functions for the varying levels of the confidence factor;
a) is a graph depicting the relative contribution of HIGH to the combined weighted rule output;
b) is a graph depicting the relative contribution of MEDIUM to the combined weighted rule output;
c) is a graph depicting the center of gravity after all of the weighted rule outputs have been combined;
The present invention relates generally to equipment operation safety and more specifically to collision warning systems based upon driver distraction levels. The following description, taken in conjunction with the referenced drawings, is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications, will be readily apparent to those skilled in the art, and the general principles defined herein, may be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. Furthermore it should be noted that unless explicitly stated otherwise, the figures included herein are illustrated diagrammatically and without any specific scale, as they are provided as qualitative illustrations of the concept of the present invention.
In order to provide a working frame of reference, first a glossary of terms used in the description and claims is given as a central resource for the reader. Next, a discussion of various physical aspects of the present invention is provided. Finally, a discussion is provided to give an understanding of the specific details.
(1) Glossary
Before describing the specific details of the present invention, it is useful to provide a centralized location in which various terms used herein and in the claims are defined. The glossary provided is intended to provide the reader with a general understanding of the intended meaning of the terms, but is not intended to convey the entire scope of each term. Rather, the glossary is intended to supplement the rest of the specification in more clearly explaining the terms used.
Complex task: A complex task is a task that involves multiple manipulations of various objects (multiple performances of simple tasks). For example, to tune a radio to a particular station, a user must first turn on the radio (simple task 1), adjust the station (simple task 2), and adjust the volume (simple task 3). Because these simple tasks belong to the same overall operation, the overall operation is referred to as a complex task. The overall time required to perform a complex task is measured from the start of the first individual task or control activation until a “quiescent” period has passed since the most recent individual task or control activation has ended. The time required to complete a complex task is referred to as the elapsed time, and the threshold “quiescent” period that signals the end of a complex task is referred to as the minimum task quiescent period.
Control activation: A control activation is the performance of a single task. For example, turning on the radio in a car is considered a single control activation. Control activations range from those involving a relatively instantaneous action such as flipping a switch to those involving a longer action such as turning a knob or depressing a button for a period of time. The length of time it takes to complete control activation is referred to as an activation time.
Distraction-correlated indicator information: Distraction-correlated indicator information is information regarding an operator that may be correlated with a level of distraction. Optimally, this type of information would be derived directly from the operator's brain function. Generally, however, this type of information is derived from indirect sources such as the operator's physical motions such as eye movements, body position, or other body movements as well as from the operator's manipulation of various objects in their surroundings in the performance of secondary activities.
Equipment—This term is used to indicate any equipment, where the effective operation of the equipment is a desired goal. For example, equipment may be an automobile such as a car or truck, another type of vehicle such as a boat or an airplane, or equipment may be another type of equipment, such as manufacturing equipment, etc.
Primary activity: Primary activities are those considered critical for the proper performance of the task an operator is attempting to accomplish. For example, in the case of a vehicle, driving is considered the primary task.
Secondary activity: Secondary activities are those considered non-critical or auxiliary to the primary activity. Secondary activities require the shifting of attention away from primary activities. For example, adjusting the heating system of a vehicle is not critical to the driver's ability to drive, but it requires that attention be taken away from the primary activity of driving, increasing the likelihood of collision.
Simple task: A simple task is the performance of a simple control activation.
Threat assessment decision: A threat assessment decision is a decision made in automatic response to a particular circumstance. For example, if a the driver of an automobile is swerving, a warning may sound in order to alert the driver that he or she may be driving recklessly. In another example, as a car gets very close to another vehicle, a warning may sound to alert the driver to avoid a potential collision. The output of the present invention is used to provide information to such systems to assist in the threat assessment decision in order to determine whether to warn the driver of potential danger. For example, if the level of driver distraction was considered to be HIGH and the confidence in the measure of distraction was MEDIUM to HIGH, the system may react more conservatively, e.g. signaling a potential collision earlier. The system may be adjusted to match various driver profiles. For example, a driver who enjoys driving aggressively could adjust the system so that activities that would normally cause the system to warn a less aggressive driver would not cause a warning for the more aggressive driver. As a result, the system can be tailored to the driving style of a particular user.
(2) Physical Aspects
The present invention has three principal “physical” aspects. The first is a system for determining operator distraction, typically in the form of a computer system operating software or in the form of a “hard-coded” instruction set. This system may be incorporated into various, devices such as a vehicular warning system, and may be coupled with a variety of sensors that provide information regarding an operator's distraction level. The second physical aspect is a method, typically in the form of software, operated using a data processing system (computer). The third principal physical aspect is a computer program product. The computer program product generally represents computer readable code stored on a computer readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape. Other, non-limiting examples of computer readable media include hard disks, read only memory (ROM), and flash-type memories. These aspects will be described in more detail below.
A block diagram depicting the components of a computer system used in the present invention is provided in
An illustrative diagram of a computer program product embodying the present invention is depicted in
(3) Discussion
The present invention provides a mechanism for generating an indication of operator distraction, and more particularly for measuring the level of distraction of vehicle drivers. Specifically, a technique for estimating an operator's distraction level for tasks that involve glancing away from the vehicle's forward path is presented. It may be based on fuzzy logic rules for estimating the distraction level due to pressing buttons, turning knobs, etc. Such actions are often parts of multi-action complex tasks and, as such, provide predictive information regarding near-term distraction levels. The present invention is described herein in the context of its application for drivers of motor vehicles, but a wide variety of other applications involving the measurement of human distraction by monitoring secondary activities will be readily ascertainable by one of skill in the art. With regard to the application for drivers of motor vehicles, the present invention is particularly beneficial for collision warning systems because it allows for the adjustment of a threat assessment decision.
If the system determines that the driver has a high distraction level with moderate to high confidence in the estimate of the distraction level, then it will make a more “conservative” decision in evaluating the best response to the current threat assessment. If the driver is distracted, they need more time to respond to a potential threat alert in order to take the correct action, so an earlier warning/response is desired. However, if the driver is attentive and has adequate time to respond, it may be better to wait until the threat level increases further before taking a response action. Producing earlier alerts when the driver is attending requires that a threat assessment decision be made with less information. It is also important that the threat assessment system not react prematurely and take a responsive action before a driver would normally respond. Such responses are considered “false alarms” and are perceived as a significant nuisance by the driver. Since different drivers exhibit different response characteristics, it may be desirable to have different warning settings or preferences available to accommodate variations in the normal driving behavior range, e.g., a young male may have a different setting than that of an elderly female. Overall safety is enhanced by producing earlier threat warnings/responses when driver distraction is high.
As may be understood by reference to the glossary above, button pushes and knob adjustments are generally referred to as control activations. A single control activation, not followed by another control activation, is classified as a simple secondary task. Many secondary tasks require multiple action (i.e. multiple button pushes), and are referred to as complex secondary tasks. An example of a complex secondary task is hunting around to find a radio station with a preferred song or type of music.
During a simple task, typically the driver glances to locate the control, activates the control, and then returns their glance to the forward view. Once the control is activated, the amount of remaining distraction to predict is considered insignificant. One difficulty is that a portion of the distraction of the simple secondary task occurs before the event that can be easily measured (i.e. the button push). This lack of predictive capability is not a critical shortcoming because the number of accidents caused by simple tasks has not been found to be significant.
Complex tasks, on the other hand, require multiple control activations and involve eye-hand coordination and/or visual feedback; typically requiring many short sampling glances away from the forward view. It is precisely these complex tasks that are known to increase the risk of accidents. Fortunately, complex tasks do contain significant predictive information about successive glances and control activations.
Data utilized in the examples herein is from Dingus, T. A., Antin, J. F., Julse, M. C., and Wierwille, W. W., “Attentional demand requirements of an automobile moving-map navigation system,” Transportation Research, 1989, A23(4), 301–315, incorporated herein by reference in its entirety for reference purposes only; without regard to any critical aspect of the present invention. This data characterizes the temporal behavior of complex secondary tasks involved in automobile driving, and separates them into two groups: radio controls, and heating, ventilation, and air-conditioning (HVAC) controls. Complex tasks for each group are characterized by their statistical behavior (i.e., mean and variance). A complex task is defined as a series of control activations followed by a quiescent period. If the amount of elapsed time since the end of the most recent control activation is less than the quiescent period, Tq, then the activations are considered to belong to the same complex task. On the other hand, once the quiescent period, Tq, has been exceeded, the complex task is considered complete. The driver distraction level of a complex task is modeled as “HIGH” after the first control activation, decaying with elapsed time, and increasing with successive control activations. The degree of driver distraction resulting from each successive control activation within a complex task (when taken alone) is considered to decrease with the total elapsed time, although the total distraction continues to increase.
As the length of time since the first activation of a control within a complex task increases, there is a decreasing degree of future predictability of the driver distraction level. Therefore a second output is modeled, encompassing a decreasing confidence factor.
Various controllers such as a rule-based controllers, analog, and/or digital circuits may be used in the performance of the present invention, as will be readily apparent to one of skill in the art. However, it is desirable that a fuzzy logic-based controller be used, as will be described below. The domain of knowledge regarding complex tasks is incorporated into a set of fuzzy logic rules as follows.
t1st(i) is the elapsed time since the start of the first control activation of a complex task (for group i);
τ(i) is the duration of the current control activation (for group i). This duration is generally measured by the amount of time a particular button was pushed or knob adjusted; and
Tq(i) is the minimum task quiescent period (for group i).
Timeline diagrams depicting two complex tasks are depicted in
The controls are separated into groups, each having similar rules, but capturing the different statistics (i.e., mean and variance) of each group. For each group of controls, two inputs are measured: the elapsed task time t1st(i) for each complex task, and the duration τ(i) for each control activation. These two inputs are then fuzzified and applied to the fuzzy controller.
The complex task is considered complete when the time since the end of the last control activation in group i, exceeds the maximum quiescent period Tq(i) 310, or when the driver takes a particular action that clearly indicates a high attentiveness level. It should be noted that t1st(i) is also reset to zero in this case.
An example of a desirable fuzzification of the inputs from this set of data is depicted in
In terms of these fuzzy inputs, a set of rules is defined for each group that determines the system outputs: the driver distraction level (D) and the confidence factor (C). The rules for group i are of the form:
R1: If t1st(i) is OFF and τ(i) is LONG then D(i) is LOW.
Rn: If t1st(i) is STRONG and τ(i) is FAULT then D(i) is NONE.
These statements are represented as a compact table of rules in
For each group, all of the rules are evaluated to determine the relative contribution of each rule. For the radio group driver distraction level there are 20 rules to evaluate:
μR1=min(μt
. . .
μR20=min(μt
Recall that rule 1 for this case was:
R1: If t1st(i) is OFF AND τ(i) is LONG THEN D(i) is LOW
This rule is evaluated by determining the degree of membership of t1st(R) and the degree of membership of τ(R). There are many possible fuzzy algebras to implement “AND”, as will be appreciated by one skilled in the art. For this example, the algebra of the minimum operator for the “AND” or intersection operation is used.
Thus, if at the current time, t1st(R)=4.8, and τ(R)=1.4, then membership of OFF, μt
μR1=min(μt
The other 89 rules are evaluated in a similar fashion. The distribution of the 90 rules used is depicted below in Table 1.
The output classification rules are presented next. As with the inputs, membership functions are defined for the outputs. The output functions used in conjunction with the present invention are depicted by the charts shown in
To determine the driver distraction level, D, in this example, the rules are combined as follows (again, note that other fuzzy algebras are possible). For each of the 45 driver distraction rules, the outputs are weighted by their rule membership. For example, if rule 16 and rule 17 were:
R16: If t1st(R) is STRONG AND τ(R) is LONG THEN D(R) is MEDIUM μR16=0.64
and
R17: If t1st(R) is STRONG AND τ(R) is NORMAL THEN D(R) is HIGH μR17=0.36,
then the combination would include a 0.64 weighted contribution of MEDIUM, a 0.36 weighted contribution of HIGH, and similar corresponding weighted contributions of the other 43 rules.
There are many ways to combine the weighted contributions as will be apparent to one of skill in the art. The combination of the weighted contributions is referred to as a rule connective operation. The connective operation of maximum is used in this example aspect. The maximum of the weighted membership output is computed for all the rules.
After all of the weight rule outputs have been combined, the center of mass of the connected result is determined. The center of mass occurs where there are equal areas under the curve to the left and to the right of the center of gravity (CoG) dividing line [over the range of distraction or confidence, e.g. (0–100)]). In the example depicted in
The output is converted into a crisp output (defuzzified) by determining the operator distraction output whose membership is maximum for the value of the center of gravity. With reference to
A flow chart depicting the steps in a method aspect of the present invention is depicted in
As noted previously, the present invention may be applied in many different situations. A few example situations include systems which incorporate distraction into the operating parameters for heavy equipment, systems which account for operator distraction in assembly lines, and systems for determining distraction in the context of vehicles other than automobiles, such as for use with other terrestrial vehicles such as trucks as well as aircraft and watercraft. The primary requisite for application of the present invention is that there be an indirect distraction measure (e.g. analogous to the use of controls described herein). Conversely, in some situations, the present invention may be adapted to provide a measure of attention.
This application claims the benefit of priority to provisional application 60/354,452, titled Information “Driver Distraction Level”, filed with the United States Patent and Trademark Office on Feb. 4, 2002.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5224566 | Stepanian et al. | Jul 1993 | A |
| 5344324 | O'Donnell et al. | Sep 1994 | A |
| 5642467 | Stover et al. | Jun 1997 | A |
| 5745031 | Yamamoto | Apr 1998 | A |
| 5801667 | Shimizu et al. | Sep 1998 | A |
| 5978737 | Pawlowski et al. | Nov 1999 | A |
| 6185514 | Skinner et al. | Feb 2001 | B1 |
| 6263282 | Vallancourt | Jul 2001 | B1 |
| 6580973 | Leivian et al. | Jun 2003 | B1 |
| 6675130 | Kanevsky et al. | Jan 2004 | B1 |
| 20020091473 | Gardner et al. | Jul 2002 | A1 |
| 20020128751 | Engstrom et al. | Sep 2002 | A1 |
| 20030004615 | Hagelin | Jan 2003 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20030149545 A1 | Aug 2003 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60354452 | Feb 2002 | US |
