Patent Application
20080166052
1. Field of the Invention
The present invention relates to a face condition determining device and an imaging device for monitoring and imaging a vehicle driver using an in-vehicle camera and determining if the driver is, for example, drowsy while driving as a part of a fail-safe image processing technology for preventing the occurrence of an accident.
2. Description of the Related Art
In recent years, a camera technology for monitoring a vehicle driver in order to prevent the occurrence of an accident has been increasingly materialized, to which improvement in the speed and image quality of a digital camera has been contributing.
An example of an image processing device for detecting eye conditions of the driver and the like is recited in No. H04-174309 of the Japanese Patent Applications Laid-Open, a basic structure of which is shown in
In the device, the image data of the driver's face is converted into binary data in the A/D converter 34. The eyeball position defining circuit 36 detects the continuity of white pixels or black pixels in the binarized image data in horizontal and vertical directions to thereby detect the eyeball position and face width of the driver. The iris detecting circuit 37 detects the iris part of the eyeball. The drowsy/inattentive driving determining circuit 38 determines if the driver has his/her eyes open or closed based on the iris detection result, and further determines if the driver is, for example, drowsy or inattentively driving based on a result of the determination. This technology is utilized to give a warning when the driver is drowsy while driving or inattentively driving.
The conventional image processing device thus described is effective only when the face is looking forward while being imaged. When the vehicle is actually driven, however, the position and angle of the driver's face changes because he/she, in one position for too long, feels weary or drowsy. As a result, accuracy in detecting the face width and the eye position is deteriorated.
Therefore, a main object of the present invention is to improve an accuracy when a face area and eye blinks are detected.
A face condition determining device according to the present invention comprises:
a brightness signal extractor for extracting a brightness signal of image data comprising continuous frame images;
a resizing processor for resizing the brightness signal into a size demanded when a face area of a photographic subject in the brightness signal is detected;
a memory in which the resized brightness signal for at least one frame is stored;
a face area detector for reading the resized brightness signal from the memory and detecting the face area of the photographic subject in the brightness signal;
a particular section detector for detecting a particular section in the face area;
a motion detector for extracting a difference between the image data of the particular section in a current frame of the image data and the image data of the particular section in the previous frame of the image data read from the memory as motion information of the particular section; and
a face condition determiner for determining a face condition of the photographic subject based on the motion information of the particular section.
In the constitution, the face area and the particular section (an eye section or a mouth section) are detected at the same time for each frame, and the face condition is determined by the face condition determiner based on the motion information of the particular section. As a result, the condition of the particular section can be accurately determined. Thus, the face determining device has an advantage in that it is determined in a stable manner that the driver is drowsy.
A face condition determining device according to the present invention comprises:
a memory in which image data is stored;
a resizing processor for resizing the image data read from the memory into a size demanded when a face area of a photographic subject in the image data is detected and storing the resized image data again in the memory;
a face area detector for detecting the face area of the photographic subject in the resized image data read from the memory;
a motion vector detector for detecting a motion vector for each basic block in the image data read from the memory or the resized image data;
a particular section motion information calculator for estimating a particular section in the face area and calculating a variation of the motion vector for each frame in the estimated particular section based on the motion vector for each basic block detected by the motion vector detector; and
a face condition determiner for determining a face condition of the photographic subject based on the variation of the motion vector for each frame of the particular section.
In the constitution, the motion vectors of the face area and the particular section (an eye section or a mouth section) are detected at the same time by each frame so that the face condition is determined by the face condition determiner based on the motion vector of the particular section. As a result, the condition of the particular section can be accurately determined. Thus, the face determining device has an advantage in that it is stably determined that the driver is drowsy.
In the face condition determining device thus constituted, the resizing processor preferably trims or partially enlarges the face area of the image data to thereby generate the image data for which the motion vector is extracted by the motion vector detector. Accordingly, when the motion vector detector extracts the motion vector for each basic block, the size of the face area can be large enough in comparison to a size adopted in the processing of the basic block.
According to the present invention, the vehicle driver is continuously monitored and imaged with the in-vehicle camera so that the motion information or the motion vector of the face area and the particular section (eyes or a mouth) are detected at the same time, and the fact is thereby stably detected that the driver is, for example, drowsy while driving through the judgments on the motion of the eyes or mouth. According to the present invention, a monitor camera system for the vehicle driver, which can be used as a fail-safe technology for preventing the occurrence of an accident, can be provided.
According to the face condition determining device of the present invention, the variation of the motion of the eyes or mouth is estimated concurrently with the detection of the face area while the vehicle driver is continuously monitored and imaged with the in-vehicle camera so that the fact is stably detected that the driver is drowsy, for example. The face condition determining device is useful as a monitor camera system for the vehicle driver which can be used as a fail-safe technology for preventing the occurrence of an accident.
These and other objects of the invention will become clear by the following description of preferred embodiments of the invention and be specified in the claims attached hereto. A number of benefits not recited in this specification will come to the attention of the skilled in the art upon the implementation of the present invention.
Hereinafter, preferred embodiments of a face condition determining device according to the present invention are described in detail referring to the drawings.
The brightness signal extractor 21 extracts a brightness signal from the image data AD-converted by the AD converter 4, REC601 (STD signal generated by the processing of the DSP5), REC656 data, or input format image data of the display device 10. As pre-processing to be executed before image data is inputted to the face condition determining device 11, the brightness signal extractor 21 extracts the brightness signal from the image signal.
The resizing processor 22 filters and downsizes the brightness signal extracted by the brightness signal extractor 21. The memory 23 stores the resized image data (brightness signal) for at least one frame. The face area detector 24 accesses the resized image data stored in the memory 23 and detects the face area and the size and tilt of the face to thereby generate face area information. The particular section generator 25 generates information of the particular section of the face such as the eyes, nose, cheek or mouth as a frame signal based on the face area information of the face area detector 24. The motion detector 26, as update processing of moving-image frames, extracts a difference between the particular section information of a current frame obtained by the particular section information generator 25 in current frame data outputted from the resizing processor 22 and the particular section information of the previous frame read from the memory 23 as a motion information. The CPU interface 27 is connected to the CPU 7 and controls the system operation of the respective processing units through a control program. The CPU 7 comprises, as a part of its function, a face condition determiner for determining a face condition based on the face area information by the face area detector 24 and the particular section motion information by the motion detector 26.
Next, the operation of the image processing device including the face condition determining device thus constituted is described. First, a typical recording/reproducing operation executed when a moving image is obtained is described. When an imaging light enters the two-dimensional image sensor 1 via the lens in the lens unit 8, an image of the photographic subject is converted into an electrical signal by a photo diode or the like, and an imaging video signal, which is an analog continuous signal, is generated in the two-dimensional image sensor 1 in accordance with horizontal and vertical drives synchronizing with a drive pulse from the timing generator 2, and then, outputted. The 1/f noise of the imaging video signal outputted from the two-dimensional image sensor 1 is appropriately reduced by the sample hold circuit (CDS) of the CDS/AGC circuit 3, and the noise-reduced video signal is automatically gain-controlled by the AGC circuit of the CDS/AGC circuit 3. The imaging video signal thus processed is supplied to the AD converter 4 from the CDS/AGC circuit 3. The AD converter 4 converts the supplied imaging video signal into image data (RGB data). The obtained image data is supplied to the DSP 5. The DSP 5 executes various types of processing (bright-signal processing, color-separation processing, color-matrix processing, data compression, resizing and the like). The DSP 5 resizes the processed image data into a display size, and then, outputs the resized image data to the display device 10. The image data is transmitted to and recorded in the recording medium 9 in the case where the recording operation is selected. When the foregoing series of operation thus described with respect to the image of an arbitrary one frame is repeatedly executed in parallel as continuous moving-image frame processing, the moving image is outputted.
Next, the operation of the face condition determining device 11 is described in detail. The brightness signal extractor 21 generates brightness-signal data in accordance with the image data. The brightness-signal data is used when the face area and the motion are detected. The brightness signal extractor 21 may generate the brightness-signal data based on the brightness-signal data of REC601 (STD signal generated by the processing of the DSP 5), REC656 data or the image data in compliance with an input format of the display device 10 in place of the image data.
The brightness-signal data outputted from the brightness signal extractor 21 is supplied to the resizing processor 22, where the image is resized. Next, the resizing processing is described. The brightness-signal data outputted from the brightness signal extractor 21 does not define the size of the image. In the face area detection implemented by the face area detector 24 and the motion detection implemented by the motion detector 26, on the contrary, the size of the image to be processed in the respective processes is defined. Therefore, the resizing processor 22 resizes the brightness-signal data inputted with an arbitrary image size into the image size defined in the face area detection and the motion detection. The resizing processor 22 filters and downsizes the brightness-signal data to thereby adjust the image size. The resizing processor 22 stores the resized brightness-signal data (hereinafter, referred to as resized image data) in the memory 23.
The face area detector 24 reads the resized image data stored in the memory 23, and detects the face area in the resized image data and extracts the size and tilt of the face. The CPU 17 confirms via the CPU interface 27 that face area detection information is detected by the face information detector 24, and instructs the particular section information generator 25 to generate the particular section information. The particular section information generator 25 generates the particular section information based on the instruction from the CPU 7. More specifically, the particular section information generator 25 identifies a particular section of the face (an eye section, a nose/cheek section, a mouth section or the like) based on the face area detection information detected by the face area detector 24 and generates the particular section information (frame information or the like) indicating the particular section, and then, supplies the generated information to the motion detector 26. The motion detector 26 detects each particular section in the resized image data of the current frame supplied from the resizing processor 22 and each particular section in the previous frame read from the memory 23 based on the particular section information. Further, the motion detector 26 extracts the difference between the image data in each particular section of the current frame and the image data in each particular section of the previous frame read from the memory 23 as the motion information of each particular section. The motion information of each particular section is extracted when the moving image frame is updated. The motion detector 26 supplies the extracted motion information to the CPU 7 via the CPU interface 27.
The operations of the respective processing units are executed based on a sequence operation by each frame through a control program executed by the CPU 7. It is assumed that the image data shown in
When it is thus determined that the face area A0 is at a fixed position, the face condition determiner determines whether or not the motion information in the eye section A1 is at least a predetermined threshold value (this threshold value is also a value specific to this variation amount) as shown in
Further, the face condition determiner focuses on the motion information in the eye section A1 when the eyes blinks are detected. The face condition determiner compares the motion information in the eye section A1 to a predetermined threshold value (this threshold value is a value specific to this motion information), and regards the number of times (the number of pulses) the motion information becomes at least the predetermined threshold value as the number of blinks as a result of the comparison. Based on the foregoing findings, the face condition determiner counts the number of pulses to thereby detect the number of blinks per unit time.
Further, the face condition determiner focuses on an absolute value of the motion information in the eye section A1 when the eyes blinks are detected. The face condition determiner memorizes the past record of an integrated value per unit time of the absolute value of the motion information. Then, the face condition determiner compares the integrated value currently calculated to the record. When it is confirmed that the current integrated value is less in comparison to the record, the face condition determiner determines that the number of blinks is decreasing.
Further, the face condition determiner focuses on an absolute value of the integrated value. The face condition determiner calculates a variation amount of the absolute value per frame or every several frames. Then, the face condition determiner determines that the speed at which eyes are blinked is decreasing when the calculated variation amount decreases over time.
Further, the face condition determiner focuses on the motion information in the mouth section A3 at the time when the eyes blinks are detected. The face condition determiner compares the motion information in the mouth section A3 to a predetermined threshold value (this threshold value is a value specific to this variation amount). When it is determined from a result of the comparison that the motion information in the mouth section A3 is at most the predetermined threshold value, the face condition determiner determines that the photographic subject is engaged in conversation because the motion information in the mouth section A3 randomly changes in the state where the face area A1 is substantially fixed in the frame.
Further, the face condition determiner determines whether or not the motion information in the mouth section A3 randomly changes when it is determined that the photographic subject is engaged in conversation. Further, the face condition determiner compares the integrated value per unit time of the absolute value of the motion information (absolute value of differential value) in the mouth section A3 to its record when it is determined that the motion information in the mouth section A3 randomly changes. When it is determined that the integrated value is less in comparison to the record, the face condition determiner determines that the photographic subject gradually talks less.
The face condition determiner determines whether or not the photographic subject is in a drowsy state based on one or the combination of two judgments: the judgement that the number of blinks becomes less and, at the same time, the speed at which the eyes are blinked is decreasing and the judgment that he/she gradually talks less. More specifically, the face condition determiner determines that the photographic subject is in a drowsy state when it is determined that the number of blinks becomes less and, at the same time, the speed at which the eyes are blinked is decreasing.
As described, according to the present preferred embodiment constituted in such a manner that the face area and the motion of the particular section (eyes or a mouth) in the face area can be detected with respect to an arbitrary image at the same time, the condition of the particular section can be accurately determined. As a result, the fact can be accurately detected that the driver is, for example, drowsy while driving. Further, even in the case when a vehicle is actually driven and the driver's face is tilted because he/she, in one position for too long, feels weary or drowsy, the face area, the eye blinks and the motion of the mouth can be accurately detected.
A face condition determining device according to a preferred embodiment 2 of the present invention is described in detail referring to the drawings.
The description of the operation of the imaging device according to the present preferred embodiment thus constituted, which is basically similar to that of the preferred embodiment 1, is omitted.
Referring to reference numerals in
Next, the operation of the face condition determining device according to the present preferred embodiment thus constituted is described. The image data fetched into the DSP 56 is subjected to the pre-processing such as the black-level adjustment and the gain adjustment by the pre-processor 41, and written in the memory 58 via the memory controller 42. The image data processor 43 reads the image data written in the memory 58 via the memory controller 42 and executes the brightness-signal processing and the color-signal processing thereto, and writes the resulting image data back into the memory 58 via the memory controller 42 as the brightness data and color-difference data (or RGB data).
The resizing processor 45 reads the original image data from the memory 58 via the memory controller 42 and horizontally and vertically resizes the read image data, and writes the resized image data back into the memory 58.
The face area detector 46 reads the resized image data for detecting the face area from the memory 58 via the memory controller 42, and detects the information such as the face area, and the size and tilt of the face. Further, in parallel with the detection, the compression/extension and motion vector detector 44 periodically reads the resized image data or the full image data before the resizing process from the memory 48 via the memory controller 42, and compresses the inputted moving-image frame data and writes the compressed image data back into the memory 18 so that the compressed image data is stored in a memory space. At the time, the compression/extension and motion vector detector 44 detects the motion vector as intermediate processing in the moving-image compression, and also outputs the motion vector for each basic block obtained as a result of the detection of the motion vector. The obtained motion vectors is be stored either in the memory 58 via the memory controller 42 or in an internal register of the compression/extension and motion vector detector 44. The respective components execute the before-mentioned operations based on the sequence operation of each frame. The sequence operation is executed based on the control program executed by the CPU 57.
The resizing processor 45 generates the image data to be displayed by horizontally and vertically resizing the relevant image data into a size optimum for the display in the entire surface thereof, and outputs the generated image data to be displayed to the display processor 47.
In the foregoing process, the face condition is determined by the CPU 57 as follows. The CPU 57 executes the predetermined control program to thereby:
These types of processing are executed by the particular section motion information calculator and the face condition determiner of the CPU 57. Below are given details.
The information such as the face area and the size and tile of the face obtained by the face area detector 46 and the information such as a resizing factor in the resizing processor 45 are inputted to the CPU 57. The CPU 17 estimates the particular section such as eyes, a nose, mouth or cheek in the face image of the original image based on these pieces of information. In relation to any of the estimated particular sections, the compression/extension and motion vector detector 4 has already written the motion vector information for each basic block in the memory 58 or the register of the compression/extension and motion vector detector 44. Then, the CPU 57 reads the motion vector information for each basic block of the estimated particular section from the memory 58 or the compression/extension and motion vector detector 44. The CPU 57 extracts the variation of the motion vector per frame of the particular section based on the foregoing information to thereby generate the particular section motion information. The function of generating the particular section motion information by the CPU 57 serves as the particular section motion information calculator.
The CPU 57 determines the face condition such as the driver being in a the drowsy state or the like based on the particular section motion information extracted by itself (particular section motion information calculator) and the face area information extracted by the face area detector 46. The function of determining the face condition by the CPU 17 serves as the face condition determiner.
It is assumed that such an image data as shown in
At the time, the CPU 57 (face condition determiner) counts the number of pulses at the time when the value of the motion information on the time axis in the eye section A1 is at least the predetermined threshold value to thereby extract the information showing how many times the eyes are blinked per unit time.
Further, the CPU 57 (face condition determiner) determines whether or not the integrated value per unit time of the absolute value of the motion information on the time axis in the eye section A1 at the time is reduced in comparison to the integrated value in the past record. The CPU 57 determines that the number of blinks is decreasing when the reduction is detected as a result of the determination.
Further, the CPU 57 (face condition determiner) determines whether or not the variation amount per frame of the integrated value per unit time at the time is reduced. The CPU 57 determines that the speed at which the eyes are blinked is decreasing when a result of the determination shows that the variation amount per frame is reduced.
The CPU 57 (face condition determiner) determines whether or not the value of the motion information on the time axis in the mouth section A3 at the time is at most a predetermined threshold value. The CPU 57 determines that the face area in the frame is at a fixed position when the value of the motion information is at most the predetermined threshold value. The CPU 57 further determines that the photographic subject is engaged in conversation in the case where the value of the motion information on the time axis in the mouth section A3 at the time randomly changes.
The CPU 57 (face condition determiner) determines whether or not the value of the motion information on the time axis in the mouth section A3 at the time randomly changes. In the determination, the CPU 57 determines whether or not the integrated value per unit time of the absolute value of the motion information on the time axis in the mouth section A3 is reduced in comparison to the past record. The CPU 57 determines that the driver talks less when a result of the determination shows that the integrated value is reduced in comparison to the past record.
Based on the foregoing determinations, the CPU 57 (face condition determiner) determines whether or not the photographic subject is in a drowsy state based on one or the combination of two judgments: the judgment that the number of blinks becomes less and, at the same time, the speed at which the eyes are blinked is decreasing and the judgment that he/she gradually talks less. More specifically, the face condition determiner determines that the photographic subject is in a drowsy state when it is determined that the number of blinks is decreasing and the photographic subject talks less.
As described, according to the present preferred embodiment constituted in such a manner that the face area and the motion of the particular section (eyes or a mouth) in the face area can be detected with respect to an arbitrary image at the same time, the condition of the particular section can be accurately determined. As a result, the fact can be accurately detected that the driver is, for example, drowsy while driving. Further, even in the case when the vehicle is actually driven and the driver's face is tilted because he/she, in one position for too long, feels weary or drowsy when the vehicle is actually driven, the face area, the eye blinks and the motion of the mouth can be accurately detected.
While there has been described what is at present considered to be preferred embodiments of this invention, it will be understood that various modifications may be made therein, and it is intended to cover in the appended claims all such modifications as fall within the true spirit and scope of this invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2007-002230 | Jan 2007 | JP | national |
| 2007-002231 | Jan 2007 | JP | national |
