The present invention relates to the visual processing and object recognition technology, and particularly to an image processing apparatus and method for calculating and processing a visual image, and a human face detection system using the apparatus and method.
For a long time, machines passively receive information from various inputting devices, such as a keyboard, inputted text file, and so on, while do not actively acquire information from outside. In order to allow machines to contact with outside and actively acquire information therefrom, the machine vision technology has been developed.
Up to now, the development of the machine vision technology has undergone a very long process. Based on persistent effort of researchers, new information technology and medium manner appear, which facilitate a development of an efficient and friendly man-machine conversation model, and the new man-machine conversation model no longer depends on the conventional inputting devices. In addition, an improving of price quality (i.e., price to performance ratio) and a lowering of a video acquisition cost facilitate the computer vision system developing to desktop system and embedded system, which means that the computer vision system can be installed in every electronic system. In the future, an intelligent electronic product having high-level vision system can make people's life more convenient.
Human face visual processing is an important content of the computer vision processing. In a human face analysis process, identification, state, intention or other information of a user can be acquired from an image (e.g., an image taken by a digital camera) thereof, and then the computer makes a corresponding response according to the acquired information (for example, analyzing mood through observing face expression and giving a corresponding response).
Human face analysis includes face detection and face recognition. Initially, the face analysis mainly focuses on human face recognition field. Compared with the human biological feature recognition system such as fingerprint, retina, iris, gene, palmprint, and so on, human face recognition is relatively direct and friendly. Human face recognition technology is widely used in various fields, such as criminal identification recognition, verification of status card and driving license, monitor of bank and custom, automatic door guard system, video meeting, study on the intellectualized robot, medicine, and so on. In addition, Human face recognition technology can be used to trace focus in photography.
In the early years, the face recognition method is performed basis on an acquired front face image or on a presupposition for easily acquiring a face image. With a widening of an application of the human face analysis and an increasing of a requirement of exploiting practical system, this face recognition method under the presupposition condition can not meet the practical requirement. People require exactly and quickly detecting whether or not a human face image exists in real complex background images, if detected, returns a position, dimension, pose of the human face. Except for being used to focus, the information can also be used to assist quickly recognizing the identification of the human. If the human face detecting technology tends to practice, an on-line detecting is necessarily required. Therefore, a detecting speed is a key factor considered by each practical system.
Presently, literatures in out country and abroad have various human face detecting methods, and many international meetings and periodicals relate to a thesis of the human face detecting. The human face detecting technology begins to be widely used in new man-machine interface, searching basis on context, digital video processing, image on-line detecting, and other fields.
For example, P. Viola and M. J. Jones release a paper named “Robust Real-Time Face Detection” in “International Journal of Computer Vision”, 57(2), 137-154, 2004. In this paper, P. Viola set forth a rectangle is used as a feature vector of the human face detection, named a rectangle feature. The rectangle feature is sensitive to some simple configurations such as edge, line segment, and can be configured for describing a direction (e.g., horizontal, vertical, diagonal) of a given configuration. Referring to
Referring to
Referring to
In
wherein i(x′, y′) is an original image of a point (x′, y′) and is a color value of the point (x′, y′). With respect to a grayscale image, a color value is from above zero to about 255. With respect to a color image, it can be transformed into the grayscale according to a color space of the human face, thereby facilitating acquiring a corresponding color value.
The ii(x, y) can be computed by the formulas (2) and (3).
s(x, y)=s(x1, y−1)+i(x, y) (2);
ii(x, y)=ii(x−1, y)+s(x, y) (3);
wherein s(x, y) is a cumulative row sum of the point (x, y) along x direction, named as a row integral sum, and is defined in a formula (4):
wherein s(x, 0)=0, ii(0, y)=0.
If a size of an image is equal to m×n, thus an integral matrix (i.e., the integrals of all of the pixels in the image) is defined in a formula (5):
every pixel of the image is retrieving through visited only once, and is iterated m×n×2 times, thereby obtaining the integral matrix described in the formula (5).
Referring to
The pixels sum of a region D=ii4+ii1−(ii2+ii3) (6);
According to the first feature template shown in
The pixels sum of the region A=ii4+ii1−(ii2+ii3);
The pixels sum of the region B=ii6+ii3−(ii4+ii5);
Therefore, the feature template shown in FIG. 5=(ii4−ii3)−(ii2−ii1)+(ii4−ii3)−(ii6−ii5) (8).
From above, the computing of the feature value of the rectangle feature only relates to integral images of end points of the rectangle feature, and does not relate to coordinate values. Therefore, no matter what is the size of the rectangle feature, a time for computing the feature value is a constant, and all computing processes are involved in algorithms
According to the above formula (8), in
In above-described human face image processing method, P. Viola set forth that the rectangle in images is used as the feature vector for human face detection, then the feature value of the rectangle feature is computed by the integral image, and according to the feature value of the rectangle feature in various positions of the image, the image of human face is judged.
The method set forth by P. Viola has improved the human face image processing speed. However, the method has some following shortcomings.
Firstly, in computing the integral matrix of the pixels of the image, the method can only computes one pixel ii(x, y) at a time, but can not calculate a number of pixels simultaneously, therefore, the computing speed is relatively slow.
Secondly, in computing the integral matrix of the pixels of the image, the integral matrix value of each pixel ii(x, y) is iterated many times so as to cause the reiteration of the computing process. In addition, the larger a range of an image, and the larger a computing quantity, thereby lowering the computing speed.
Thirdly, in computing the integral matrix of the pixel of the image, because the computing quantity is relatively large, a high speed processor and a large-capacity memory are required to perform the computing process, as a result, a high cost is caused and the method is not easy integrated with an integrate circuit.
Therefore, how to improve human face image processing method set forth by P. Viola is an important problem for human face image processing technology, so as to compute the integral matrix of the acquired image with a faster speed, lower the capacity of the required memory, easier integrate with the integrate circuit, realize miniaturization and modular of human face detection system.
An image processing apparatus with a quickly computing speed is provided.
An image processing apparatus with a quickly computing speed and having a simple configuration is provided.
An image processing method with a quickly computing speed is provided.
A human face detection system with a quickly computing speed is provided.
A human face detecting method with a quickly computing speed is provided.
The image processing apparatus includes a first register, a second register, and an adder. The first register includes M numbers of stored values, and an initial value of each stored value is zero. The second register reads I numbers of pixels from a sub-window having M×N numbers of pixels in turn, I is a positive integer that greater than 1 and less than M×N, M and N are positive integer respectively that greater than 1. The adder adds the I numbers of pixels read by the second register and the corresponding I numbers of stored values stored in the first register to acquire I numbers of adding calculation values. I numbers of adding calculation values are returned to the first register and are saved as I numbers of stored values.
An image processing method includes the steps of: providing a sub-window image containing M×N numbers of pixels, M and N being positive integers that greater than 1; providing a first register containing M numbers of stored values; reading I numbers of pixels from the sub-window image, I being a positive integer that greater than 1 and less than M×N; and performing a first operation, adding stored values of I numbers of pixels read by the second register and I numbers of stored values in the first register to acquire I numbers of adding calculation values, returning the I numbers of adding calculation values to the first register and saving the I numbers of adding calculation values as I numbers of stored values of the first register.
A human face detection system includes an image processing unit, an image integral computing unit and a human face detection unit. The image processing unit is used to divide a sub-window image into M×N numbers of pixel regions and evaluate each of the pixel regions, thereby outputting a processed result. The image integral computing unit includes multiple registers for parallel computing the processed result, thereby outputting a computed result. The human face detection unit is used to detect and judge whether or not the computed result contains a human face image.
A human face detection method includes the steps of: providing a sub-window image; dividing the sub-window image into M×N numbers of pixels and evaluating each of the pixel to acquire a processed result; computing the processed result by an image integral computing unit comprising multiple registers, thereby acquiring an integral matrix, an accumulated value of pixels and a sum of square value of pixels; according to the integral matrix, computing a given rectangle region of the sub-window image to acquire a feature value corresponding to the given rectangle region; according to the feature value, the accumulated value of pixels, and the sum of square value of pixels, applying a feature value quick detection step to the sub-window image and acquiring a detected result; according to the detected result, computing M×N numbers of pixels of the sub-window image one by one, thereby acquiring M×N numbers of feature values; and according to M×N numbers of feature values, applying a feature value general detection step to the sub-window image.
The above-described image processing apparatus additionally includes an accumulated value register, and an initial value of a stored value stored in the accumulated value is zero. The adder adds the I numbers of pixels and the stored value in the accumulated value register to acquire an accumulated value. The accumulated value is returned to the accumulated register and saved as a stored value.
The above-described image processing apparatus additionally includes a discrete quantity register and a discrete quantity computing unit. An initial value of a stored value stored in the discrete quantity register is zero. The discrete quantity computing unit is configured for apply a discrete quantity calculation to I numbers of pixels and the stored value stored in the discrete quantity register, thereby acquiring a discrete quantity calculation value. The discrete quantity calculation value is returned to the discrete quantity register and is saved as a stored value of the discrete quantity register.
Multiple registers are used to simultaneously/parallel process and compute the stored values of the pixels in the sub-window image, the computing speed of the image integral computing unit is multiplicatively quickened. In the process of the image integral computing unit computing the integral matrix of image pixels, the multi-bit registers are used to register intermediate results, thus the stored value of each pixel II(x, y) does not need to be recalculated by the reiteration manner, greatly saving the reiteration steps and quickening the computing speed of the image integral computing unit. Because the computing speed is greatly quickened, a relative low speed processor and low capacity memory can be applied to the present human face detection system, thereby lowering a cost of the system. In addition, the image integral computing unit has a simple configuration and is easy to integrate with the integrate circuit, so as to miniaturize the human face detection system and lower the cost thereof, thereby achieving a better practicability.
The feature value quick detection unit and the feature value general detection unit cooperatively detect the sub-window image, the sub-window images which do not contain the human face image are rapidly eliminated by the feature value quick detection unit; the feature value quick detection unit and the feature value general detection unit can simultaneously run the detection process, so the detecting speed and efficiency of the human face detection system are greatly improved, thereby causing the human face detection system having better practicability and better performance of on-line detection. In addition, the feature value calculator can be made of an integrate circuit, so a compact structure and a high computing speed are acquired. The processed sub-window image is saved in the image memory by the local storage manner, which facilitates sequential units access the sub-window image data, thereby improving a running efficiency of the human face detection system
These and other features and advantages of the various embodiments disclosed herein will be better understood with respect to the following description and drawings, in which like numbers refer to like parts throughout, and in which:
An exemplary embodiment of an image processing technology is provided. The image processing technology includes an image processing apparatus and an image processing method applied in processing a computer vision image. The image processing technology improves the method of calculating the feature value of the rectangle feature by the integral image provided by P. Viola, thereby having a rapidly computing speed, low capacity memory, and easy to integrate with the integrate circuit.
Referring to
Referring to
Referring to
An initial value of a stored value of the first memory 127 is zero, and the first memory 127 is used to receive the 24×24 numbers of pixels having feature values outputted by the image processing unit 110, and the 24×24 numbers pixels are saved as 24×24 numbers of stored values. Because the first memory 127 employs a local storage mode, and the image integral computing unit 120 does not require storing or accessing image information to an external memory, thus a high computing speed is achieved.
Initial values of stored values of the four store bits I1 to I4 contained in the second register 122 are zero, according a certain order, he second register 122 every time reads the stored values of four pixels from the stored values of the 24×24 numbers of pixels stored in the first memory 127. For example, according to a row column order, firstly, pixels from (0, 0) to (0, 23) are read in that order; secondly, pixels from (1, 0) to (1, 23) are read in that order; in such manner, until the pixel (23, 23) is read. In addition, according to a column row order, firstly, pixels from (0, 0) to (23, 0) are read in that order; secondly, pixels from (0, 1) to (23, 1) are read in that order; in such manner, until the pixel (23, 23) is read. Four pixels are read every time and are separately saved as the stored values of the store bits I1 to I4 of the second register 122. Four stored values of the second register 122 are separately output to the adder 123. In the present embodiment, in a first period of time, the second register 122 reads the pixels from (0, 0) to (0, 3) from 24×24 numbers of pixels according to the row column order.
The adder 123 can be an integrated structure. According to a function in the image integral computing unit 120, the adder 123 includes a first adder 1231 and a second adder 1232. In the first period of time, the first adder 1231 receives the stored values of four pixels (i.e., the pixels from (0, 0) to (0, 3)) separately saved in the four store bits (i.e., the store bits from I1 to I4) of the second register 122, and pluses each of the above four stored values and a corresponding stored value (the initial value of the stored value is zero) saved in the four store bits from I1 to II 4 of the first register 121. That is, the stored value of the pixel (0, 0) saved in the store bit I1 pluses the stored value saved in the store bit II 1; the stored value of the pixel (0, 1) saved in the store bit I2 pluses the stored value saved in the store bit II 2; the stored value of the pixel (0, 2) saved in the store bit I3 pluses the stored value saved in the store bit II 3; and the stored value of the pixel (0, 3) saved in the store bit I4 pluses the stored value saved in the store bit II 4. Except for being separately returned to the four store bits from II 1 to II 4 of the first register 121 and separately saved as four corresponding stored values, the above four adding calculation values are also separately returned to the four store bits corresponding to the pixels from (0, 0) to (0, 3) of the second memory 128 and separately saved as four corresponding stored values (the initial value of each of the stored value is zero). The second adder 1232 receives the four stored values of four pixels from (0, 0) to (0, 3) saved in the four store bits from I1 to I4 of the second register 122, and runs an adding calculation, i.e., plus the above four stored values of the four pixels saved in the four store bits from I1 to I4, thereby acquiring and outputting an accumulated value. The accumulated value plus a stored value (an initial value of the stored value is zero) saved in the accumulated value register 124, and the adding calculation value is returned to the accumulated value register 124 and is saved as a stored value. In addition, the accumulated value is also returned to the multiplier 126 and a multiplicative square calculation is run, thereby acquiring a square value. The square value pluses the stored value (an initial value of the stored value is zero) saved in the sum of square register 125, and the acquired adding calculation value is returned to the sum of square register 125 and is saved as a stored value.
The second memory 128 is used to store an adding calculation values, i.e., sums of the stored values of the store bits from I1 to I4 of the second register 122 and the stored values of the store bits from II0 to II23 of the first register 121 computed by the first adder 1231, into store bits of the second memory 128 according to an input order, thereby forming a stored value integral matrix.
The accumulated value register 124 is used to store the accumulated value of the stored values of pixels of the sub-window image including 24×24 numbers of pixels.
The sum of square register 125 is used to store a sum of square value of the stored values of pixels of the sub-window image including 24×24 numbers of pixels.
The human face detection unit 130 receives the stored value integral matrix saved in the second memory 128, the accumulated valued saved in the accumulated value register 124 and the sum of square value saved in the sum of square register 125; and compares the aforesaid values with the reference value prestored in the database to detect whether or not the sub-window image contains a human face and to judge a position, size, expression, status and other human face information in the sub-window image.
Stored values of twenty-four store bits in the twenty-fourth row are separately equal to twenty-four adding calculation values by separately adding the stored values of the pixels from (0, 0) to (0, 23), pixels from (1, 0) to (1, 23) . . . the pixels from (23, 0) to (23, 23), thereby constructing a 24×24 numbers of stored value integral matrix. The stored value saved in the accumulated value register 124 is an accumulated value of the pixels from (0, 0) to (23, 23). The stored value saved in the sum of square register 125 is a sum of square of respective accumulated values of one hundred and forty-four groups of stored values, i.e., the stored values of the pixels (0, 0) to (0, 3), . . . the stored values of the pixels (23, 20) to (23, 23).
In a process of the image integral computing unit 120 computing the stored value integral matrix, the first register 121 registers the accumulated value of the stored values of each row, the stored value integral matrix of each pixel ii(x, y) do not need to be recalculated by a reiteration manner, but the stored value saved in the second register 122 plus the stored value saved in the first register 121 is performed to acquire a desired integral matrix value. This computing manner greatly reduces reiteration steps and quickens the computing speed of the image integral computing unit 120.
In the first embodiment of the present invention, in order to rapidly recognize the human face image regions and non-face image regions, the image integral computing unit 120 provides a method for computing an average value and a standard deviation value of stored values of pixels of the sub-window image including 24×24 numbers of pixels. By computing the average value and the standard deviation which describes a random discrete quantity of pixels, simple background patterns such as the sky image 10 shown in
A given stored value of a pixel is denoted by II(x, y), wherein x=0˜23, y=0˜23, the average value of the stored values of the pixels in the sub-window image is denoted by A, the standard deviation value of the stored values of pixels is denoted by B, the stored value of the accumulated value register 124 is denoted by IIh, the stored value of the sum of square register 125 is denoted by SQ, the store bit of the second register 122 is denoted by I and I=4, the accumulated value of the second register 122 each time is denoted by Σ, and Σ is an output value of the adder 123 each time, the number of the pixels of the sub-window image is denoted by W, and W=24×24, therefore, the number of Σ denoted by K, and K=W/I=144.
Σ0=II(0,0)+II(0,1)+II(0,2)+II(0,3);
Σ1=II(0,4)+II(0,5)+II(0,6)+II(0,7);
. . .
Σ143=II(23,20)+II(23,21)+II(23,22)+II(23,23);
According to
IIh=Σ0+Σ1+Σ2+ . . . +Σ143 (9);
SQ=Σ02+Σ12+Σ22+ . . . +Σ1432 (10)
According to probability theory, with respect to the random discrete quantity of the pixels in the image:
In the first embodiment, the method of computing the standard deviation value is different from the conventional method. Because the second register 122 is used to read the sub-window image, therefore, the random discrete quantity of the sub-window image is computed on the basis of I stored values of I bits of pixels stored in the second register 122.
In the first embodiment, the standard deviation value is computed by sub-regions (a size is I bits of pixels), which is capable of representing a distribution status of the random discrete quantity of a bigger sub-region in the sub-window image. Therefore, compared with only computing a distribution status of the random discrete quantity of the smallest region of the prior art, the present method judges whether or not the human face exists with a rapidly speed and a high veracity. When the size of the sub-region is smaller, the feature value of the image rectangle is more unclear, therefore, it is easier to judge whether or not the human face exists in the larger sized image region.
In the present invention, the average value A and the standard deviation value B of the stored values of the pixels of the sub-window image are not limited to be computed by such method. For example, other computing method in the prior art can also be used to compute the average value A and the standard deviation value B.
Referring to
In the present embodiment, the number of the sub-window image is not limited to be 24×24, 8×8, 16×16, 48×48, 64×64 format, even can be 16×24, 24×48, 48×64 format.
In the present embodiment, the number of the stored values of the second register 222 is not limited to be a fixed value. The number of the stored values of the second register 222 can be less than that of the first register 221, e.g., the number is four, eight, twelve; or equal to that of the first register 221, e.g., the number is twenty-four; or even larger than that of the first register 221, e.g., the number is forty-eight or sixty-four. When the accumulated value register 224 and the sum of square register 225 are self-governed register, the number of the stored values of the second register 222 is a multiple or a submultiple of the number of the stored values of the first register 221, thereby saving the store bits of the registers and avoiding a waste of the store bits. In addition, when the number of the stored values of the second register 222 is larger than the multiple or submultiple of the number of the stored values of the first register 221, so long as practical used store bits are set, redundant store bits can be left unused or set in other use.
Referring to
Referring to
Furthermore, a human face detection system applying the aforesaid image integral computing unit 120 is additionally provided. Referring to
The image data stored in the DRAM 142 is generally a whole image as shown in
The main control unit 111 is a central processing unit of the human face detection system 100, and is configured for receiving external image data and communication signals of external systems; controlling a running of the image processing unit 110, image integral computing unit 120 and the human face detection unit 130; complete controlling processing, computing and detecting of the image data; and returning the detected result of the image data to the external systems. In the first embodiment, according to the predetermined coordinate values of images, the main control unit 111 reads the image data stored in the DRMA 142 by the DMA 141; intercepts a sub-window image with a fixed size from a whole image of the DRMA 142; controls the re-sampling unit 113 to re-sample the read image data; divides the sub-window image into M×N image regions, evaluates each of the image regions with different feature vectors by a series of feature templates (e.g., a rectangle feature); processes the sub-window image including M×N numbers of image regions into another sub-window image including M×N numbers of pixels which have feature values (e.g., the sub-window image including 24×24 numbers of pixels), thereby facilitating the computing and processing of sequential units. The processed sub-window image is saved in the image memory 112 by the main control unit 111. The processed sub-window image is saved in the image memory 112 by the local storage manner, so it facilitates sequential units accessing the image data of the sub-window image, thereby improving a running efficiency of the human face detection system 100.
The main control unit 111 controls the image integral computing unit 120 reading the image memory 112, and inputs the processed M×N numbers of pixels stored in the image memory 112 into the image integral computing unit 120. The image integral computing unit 120 performs an integral matrix, a pixel accumulating and a pixel sum of square computing process, and separately saves the computed result into the accumulated value register 124, the sum of square register 125 and the second memory 128.
According to the integral matrix value, the pixel accumulated value and the pixel sum of square value computed by the integral calculating unit 120, the human face detection unit 130 controlled by main control unit 111 performs a judging process. The feature value calculator 131 of the human face detection unit 130 is used to compute the feature value of a given region in the sub-window image. According to the description of
Referring to
According to the feature value of the given rectangle region of the sub-window image computed by the feature value calculator 131, the accumulated value and the sum of square value computed by the integral calculating unit 120, the main control unit 111 controls the feature value quick detection unit 132 running a first step of judging process of human face recognition to eliminate the sub-window images which do not contain human face image. Regarding a generic image, the number of the sub-window images which contain human face image is generally less than the number of the sub-window images which do not contain human face image. Therefore, in the present embodiment, the feature value quick detection unit 132 is used to quickly detect and eliminate the sub-window images which do not contain human face image so as to quicken the detecting speed of the human face detection system 100. Referring to
Referring to
Referring to
Referring to
Referring to
Referring to
Alternatively, the feature value quick detection unit 132 can directly detect all of the sub-window images in a whole image, eliminate the sub-window image which do not contain the human face image, and deposit the sub-window image which has a probability of containing the human face image into a memory. Similarly, the feature value general detection unit 133 can direct access and detect the sub-window image which has a probability of containing the human face image into deposited in the memory. In this detecting process, the feature value quick detection unit 132 does not need to wait for the feature value general detection unit 133 finishing its detection process, that is, the feature value quick detection unit 132 and the feature value general detection unit 133 can run the detection process simultaneously, thereby the detecting efficiency is greatly improved.
In summary, in the embodiments of the present invention, multiple registers are used to simultaneously/parallel process and compute the stored values of the pixels in the sub-window image, the computing speed of the image integral computing unit is multiplicatively quickened. In the process of the image integral computing unit computing the integral matrix of image pixels, the multi-bit registers are used to register intermediate results, thus the stored value of each pixel II(x, y) does not need to be recalculated by the reiteration manner, greatly saving the reiteration steps and quickening the computing speed of the image integral computing unit. Because the computing speed is greatly quickened, a relative low speed processor and low capacity memory can be applied to the present human face detection system, thereby lowering a cost of the system. In addition, the image integral computing unit has a simple configuration and is easy to integrate with the integrate circuit, so as to miniaturize the human face detection system and lower the cost thereof, thereby achieving a better practicability.
Regarding the human face detection system of the present invention, the feature value quick detection unit and the feature value general detection unit cooperatively detect the sub-window image, the sub-window images which do not contain the human face image are rapidly eliminated by the feature value quick detection unit; the feature value quick detection unit and the feature value general detection unit can simultaneously run the detection process, so the detecting speed and efficiency of the human face detection system are greatly improved, thereby causing the human face detection system having better practicability and better performance of on-line detection. In addition, the feature value calculator can be made of an integrate circuit, so a compact structure and a high computing speed are acquired. The processed sub-window image is saved in the image memory by the local storage manner, which facilitates sequential units access the sub-window image data, thereby improving a running efficiency of the human face detection system.
The above description is given by way of example, and not limitation. Given the above disclosure, one skilled in the art could devise variations that are within the scope and spirit of the invention disclosed herein, including configurations ways of the recessed portions and materials and/or designs of the attaching structures. Further, the various features of the embodiments disclosed herein can be used alone, or in varying combinations with each other and are not intended to be limited to the specific combination described herein. Thus, the scope of the claims is not to be limited by the illustrated embodiments.
Number | Date | Country | Kind |
---|---|---|---|
097107576 | Mar 2008 | TW | national |