The present invention generally relates to eye monitoring and, more particularly, relates to monitoring an eye, such as one or both eyes of a driver of a vehicle, with a light illuminated video imaging system.
Video imaging systems have been proposed for use in vehicles to monitor the driver and/or passengers in the vehicle. Some proposed video imaging systems include one or two cameras focused on the driver of the vehicle to capture images of the driver's face and allow for determination of various facial characteristics of the driver including the position, orientation, and movement of the driver's eyes, face, and head. By knowing the driver facial characteristics, such as the driver's eye position and gaze, vehicle control systems can provide enhanced vehicle functions.
A vehicle control system can monitor one or both eyes of the driver and determine a condition in which the driver appears to be drowsy, and can take further action to alert the driver of the driver drowsy condition. The vehicle control system can also determine if the driver is distracted, and take action to minimize the distraction. Further, the vehicle control system can monitor the driver eye gaze and control various vehicle systems, such as the radio and entertainment system, and provide enhanced control of such systems.
Some proposed vision-based imaging systems that monitor the eye(s) of the driver of a vehicle require infrared (IR) illumination along with visible light filters to control scene brightness levels inside of the vehicle cockpit. The ability to discern facial characteristics is a considerable requirement in the case of driver monitoring. Current techniques for illuminating the driver's face with infrared illumination include providing a light source focused on the driver's eye(s). Other techniques for illuminating the driver's face propose employing multiple light sources of different wavelengths, arranged in a generally concentric ring, for generating a bright eye pupil and dark eye pupil effect.
Some proposed eye monitoring techniques process video images taken during both the bright eye pupil and dark eye pupil illuminations, and process the difference in the acquired bright and dark eye pupil images. By taking the difference in images acquired during bright and dark illumination, glare and other undesirable effects caused by bright images can be removed to allow for enhanced identification of the target eye. While combined dark and bright eye pupil illuminations have been proposed, such techniques typically require a large number of light sources of multiple wavelengths arranged in the shape of a concentric ring adjacent to and surrounding the video imaging camera.
It is therefore desirable to provide for an effective method for illuminating an eye and generating images of the eye to allow for enhanced eye monitoring of a person in the vehicle. In particular, it is desirable to provide for a more cost affordable system that offers flexible packaging of the light source illuminators and is easy and economical to manufacture.
In accordance with the teachings of the present invention, a system and method are provided for monitoring an eye. According to one aspect of the present invention, the system includes a video imaging camera oriented to generate images of an eye. The camera is aligned along an imaging axis. A first light source is located at an angle less than 2.5 degrees from the imaging axis for illuminating the eye. A second light source is arranged at an angle greater than 4.5 degrees from the imaging axis for illuminating the eye. The video imaging camera acquires a first image when the first light source illuminates the eye, and generates a second image when the second light source illuminates the eye. The system further includes a processor for processing the first image and the second image.
According to another aspect of the present invention, the method includes arranging a video imaging camera oriented along an imaging axis to generate images of an eye. The method also includes arranging a first light source at an angle less than 2.5 degrees from the imaging axis, and a second light source at an angle greater than 4.5 degrees from the imaging axis. The method also includes the steps of illuminating an eye with the first light source, and generating a first image of the eye while the eye is illuminated with the first light source. The method further includes the steps of illuminating the eye with the second light source, and generating a second image of the eye while the eye is illuminated with the second light source. Further, the method includes processing the first and second images.
These and other features, advantages and objects of the present invention will be further understood and appreciated by those skilled in the art by reference to the following specification, claims and appended drawings.
The present invention will now be described, by way of example, with reference to the accompanying drawings, in which:
Referring now to
The eye monitoring system 20 is shown mounted generally in a mid-region of the dash 14 in the front region of the vehicle cockpit 12. The eye monitoring system 20 may be mounted in any of a number of various locations within the vehicle 10 which allow for light illumination and acquisition of video images of one or both eyes of the driver 18 of the vehicle 10. For example, the video imaging camera 20 may be mounted in the steering assembly 16, or mounted elsewhere in the dash 14, or may be mounted in the instrument cluster. While the eye monitoring system 20 is shown and described herein in connection with a single video imaging camera, it should be appreciated that two or more video imaging cameras may be employed, without departing from the teachings of the present invention.
The eye monitoring system 20 is mounted to the dash 14 such that the light illumination sources illuminate the eye(s) of the driver and the video imaging camera captures successive video image frames of the region where the driver 18 of the vehicle 10 is expected to be located during normal vehicle driving. More particularly, the video image captures the driver's face including one or both eyes 22 and the surrounding ocular features generally formed in the area referred to as the ocular adnexa. The acquired video images are then processed for tracking characteristics of one or both eyes 22 of the driver 18. Each video frame image may be processed to determine whether one or both eyes 22 of the driver 18 are in an open position or a closed position to determine a driver drowsiness condition. The video frame images may be processed to determine a driver distraction condition, an inattentive driver, or other conditions.
Referring to
As shown in
The video imaging camera 30 is focused on one or both eyes 22 of the driver of the vehicle. In the embodiment shown, video imaging camera 30 is focused on one eye 22. The video imaging camera 30 is aligned along an imaging axis 38, which generally defines a centerline of the focus of the camera 30 onto the eye 22. Line 33 defines the centerline between the first light illumination source 32 and the eye 22, while line 35 defines the centerline between the second light illumination source 34 and the eye 22.
The eye monitoring system 20 employs the first and second light illumination sources 32 and 34 for illuminating the target eye 22, such that the video imaging camera 30 is able to capture images of the eye 22 and its pupil in bright and dark illuminated conditions. The first plurality of IR LEDs 32A-32C of light source 32 are shown configured in a straight line to one side of the video imaging camera 30. The first plurality of IR LEDs 32A-32C are mounted close to the video imaging camera 30 such that the centerline 33 of first light source 32 is at an angle θB less than 2.5 degrees from the imaging axis 38. According to another embodiment, the first light source 32 is arranged at an angle θB less than 2.3 degrees from the imaging axis 38. By arranging the first light source 32 at angle θB less than 2.5 degrees from the imaging axis 38, the first light source 32 is substantially coaxial with the imaging axis 38 such that the resultant light illumination produces a bright eye image.
The second plurality of IR LEDs 34A-34C of light source 34 are shown configured in a straight line on the opposite side of the video imaging camera 30. The second plurality of IR LEDs 34A-34C are mounted close to the video imaging camera 30 such that the centerline 35 of the second light source 34 is at an angle θD greater than 4.5 degrees from the imaging axis 38. According to a further embodiment, the second light source 34 is arranged at an angle θD greater than 4.6 degrees from the imaging axis 38. By arranging the second light source 34 at angle θD greater than 4.5 degrees from the imaging axis 38, the second light source 34 is substantially non-coaxial with the imaging axis 38 such that the resultant light illumination produces a dark eye image.
The first and second light sources 32 and 34 may include any of a number of commercially available lighting sources. In one embodiment, the light sources 32 and 34 each generate a light source at substantially the same frequency. While each of the first and second light sources 32 and 34 are shown made up of three LEDs, it should be appreciated that any of a number of one or more light lighting devices may be employed for each of light sources 32 and 34, without departing from the teachings of the present invention.
In the example shown, the first plurality of IR LEDs 32A-32C are spaced at a distance of less than twenty-five millimeters (25 mm) from the center of the video imaging camera 30. In this example, the second plurality of IR LEDs 34A-34C are located at a distance YD greater than fifty millimeters (50 mm) from the video imaging camera 30. The video imaging camera 30 light source 32 and 34 are displaced from the eye 22 by a distance X of about six hundred fifteen millimeters (615 mm) according to this example.
The first and second light sources 32 and 34 are illuminated such that only one of the first and second light sources 32 and 34 is on at a time, according to one embodiment. The first plurality of LEDs 32A-32C of first light source 32 are turned on to illuminate the eye 22 to enable the video imaging camera 30 to capture a bright first image of the eye 22. With the first plurality of LEDs 32A-32C turned off, the second plurality of LEDs 34A-34C of light source 34 are then turned on to illuminate the eye 22 so as to allow the video imaging camera 30 to generate a dark second image of the eye 22. The first and second video images captured during bright and dark images of the eye are then processed to discern information about the eye 22 and/or condition of the driver 18.
Referring to
The vision processor 40 is shown having a frame grabber 46 for receiving the video output frames on line 52 generated by the video imaging camera 30. Vision processor 40 also includes a video processor 42 for processing the acquired video frames. The vision processor 40 includes memory 44, such as random access memory (RAM), read-only memory (ROM), and other memory as should be readily apparent to those skilled in the art. The vision processor 40 may be configured to perform one or more routines for controlling the actuation of first and second light illumination sources 32 and 34, controlling the video imaging camera 30, processing the acquired video images, and applying the processed information to vehicle control systems.
The vision processor 40 has a camera control function via control RS-232 logic 56 which allows for control of the video imaging camera 30 via camera control signals on line 54. Control of the video imaging camera 30 may include automatic adjustment of the pointing orientation of the video imaging camera 30. For example, the video imaging camera 30 may be repositioned to focus on an identifiable feature, and may scan a region in search of an identifiable feature, including the driver's face and, more particularly, one or both eyes 22. Control may also include adjustment of focus and magnification as may be necessary to track an identifiable feature. Thus, the eye monitoring system 20 may automatically locate and track an identifiable feature, such as one or both of the driver's eyes 22.
The vision processor 40 further includes a timing sync pulse generator 48 coupled to video processor 42. The timing sync pulse generator 48 produces timing sync output signals on line 50 to control the activation of the first and second light illumination sources 32 and 34. That is, the timing sync output signals control activation of light sources 32 and 34 by sequentially turning light sources 32 and 34 on and off, so that only one light source is on at a time. The vision processor 40 monitors the timing sync output signals and processes the video output signals as images that were acquired either with the first lighting source 32 or second lighting source 34.
Referring to
In
The captured bright and dark video images of eye 22 may be processed to determine whether the eye 22 is in an open or closed position, indicative of a drowsy driver condition, or may be evaluated to determine if the driver is inattentive or distracted. This can be achieved by monitoring the sclera 26 and/or the iris 24 of the eye 22 and determining when at least one of the sclera 26 and iris 24 are not visible in the video image due to complete covering of the eyelid to determine a driver drowsiness condition. Driver inattention or distraction can be determined by monitoring the position of the iris 24, as well as positioning of the surrounding facial characteristics.
Referring to
With the first light source 32 turned on, routine 70 captures a frame 2 image to create a bright pupil effect image in step 78. In step 80, the LED bank 2 (34A-34C) of the second light 34 source is turned off in step 80. In step 82, routine 70 generates a timing/sync pulse. Next, in step 84, the LED bank two of the second light source 34 is turned on in step 84. With the second light source 34 turned on, routine 70 captures a frame 3 image to create the dark pupil effect image in step 86. Next, in step 88, routine 70 turns the LED bank 2 of the second light source 34 off in step 88, and then generates a timing/sync pulse in step 90, before returning to step 76. Accordingly, control routine 70 controls the timing for activating the first and second light sources 32 and 34 and capturing images generated during illumination of the first source 32 and then the second light source 34.
Referring to
The processed eye images may also be used in a driver gaze application as shown in step 114. In the driver gaze application, routine 100 determines a driver gaze in step 116. The determined driver gaze may be employed to adjust the following distance of the vehicle in step 118 and/or utilize appropriate driving cues in step 120.
The processed eye images may further be used in a driver drowsy application as shown in step 108. In the driver drowsy application, routine 100 determines if the driver is drowsy in decision step 110. If the driver is determined to be drowsy, routine 100 initiates one or more countermeasures in step 112. These countermeasures may include an audible warning in step 112A, initiating a haptic seat in step 112B, increasing vehicle following distance in step 112C, initiating cold air conditioning (A/C) in step 112D, and/or activating peppermint olfactory stimulation in step 112E.
Accordingly, the eye monitoring system 20 of the present invention advantageously illuminates and acquires bright and dark images of the eye to allow for easy monitoring of the driver of the vehicle. The eye monitoring system 20 of the present invention is economical and easy to manufacture in a single package, and easy to deploy in a vehicle. It should be appreciated that the eye monitoring systems may be employed in various applications employed both on and off a vehicle.
It will be understood by those who practice the invention and those skilled in the art, that various modifications and improvements may be made to the invention without departing from the spirit of the disclosed concept. The scope of protection afforded is to be determined by the claims and by the breadth of interpretation allowed by law.