Patent Application
20230353885
The present disclosure relates to an image processing system, and more particularly, to an image processing system for achieving the automatic white balance of a subject's face.
When a scene is recorded by a camera, the colors shown on an image may depend greatly on a light source that illuminates the scene. For example, when a white object is illuminated by yellow sunlight, the white object may appear yellow instead of white in the image. To correct such color shifts caused by the colors of light sources, the camera may perform an automatic white balance (AWB) operation to correct the white colors shown in the image.
Furthermore, if a person is shown in an image, a color shift of the person's skin can be noticeable since human faces tend to be observed in detail. Therefore, cameras may further perform another automatic white balance operation to correct the skin color. However, people may wear sunglasses or masks when being photographed, so the automatic white balance operation to correct the skin color can be challenging as the colors of the sunglasses and/or the masks may also be regarded as skin color. For example, if the face of a subject is partially covered by a blue mask, the facial AWB operation may tend to adjust the blue color, so the subject's face in a displayed image may appear too yellow after the facial AWB calibration.
One embodiment of the present disclosure provides an image processing system. The image processing system includes an image sensor and a processing device. The image sensor is configured to capture an image. The processing device is configured to detect a face shown in the image, estimate locations of a plurality of facial features of the face, determine at least one non-occluded region of the face according to occluding conditions of the plurality of facial features, and perform a facial white balance operation on the image according to color data derived from within the non-occluded region in which at least one facial feature is visible.
Another embodiment of the present disclosure provides a method for processing an image. The method comprises capturing, by an image sensor, an image; detecting a face shown in the image; estimating locations of a plurality of facial features of the face; determining at least one non-occluded region of the face according to occluding conditions of the facial features, wherein at least one of the facial features in the at least one non-occluded region is visible; and performing a facial white balance operation on the image according to color data derived from within the non-occluded region.
Since the image processing system and the method for processing images can detect the non-occluded regions of the face and sample the skin color within the non-occluded regions of the face for performing the facial white balance operation not affected by the colors of a mask and sunglasses that occlude the face, and thus, the facial white balance operation can correct the skin color more accurately.
A more complete understanding of the present disclosure may be derived by referring to the detailed description and claims when considered in connection with the Figures, where like reference numbers refer to similar elements throughout the Figures.
The following description accompanies drawings, which are incorporated in and constitute a part of this specification, and which illustrate embodiments of the disclosure, but the disclosure is not limited to the embodiments. In addition, the following embodiments can be properly integrated to complete another embodiment.
References to “one embodiment,” “an embodiment,” “exemplary embodiment,” “other embodiments,” “another embodiment,” etc. indicate that the embodiment(s) of the disclosure so described may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Further, repeated use of the phrase “in the embodiment” does not necessarily refer to the same embodiment, although it may.
In order to make the present disclosure completely comprehensible, detailed steps and structures are provided in the following description. Obviously, implementation of the present disclosure does not limit special details known by persons skilled in the art. In addition, known structures and steps are not described in detail, so as not to unnecessarily limit the present disclosure. Preferred embodiments of the present disclosure will be described below in detail. However, in addition to the detailed description, the present disclosure may also be widely implemented in other embodiments. The scope of the present disclosure is not limited to the detailed description, and is defined by the claims.
In step S210, the image sensor 110 may capture an image IMG0, and in step S220, the processing device 120 may detect a face shown in the image IMG0. In some embodiments, there may be more than one face shown in the image IMG0. In such case, the processing device 120 may detect multiple faces and choose one face from the detected faces which occupies the greatest area in the image IMG0 for the facial white balance operation.
In step S230, the processing device 120 may further estimate the locations of facial features of the face F1. In some embodiments, the processing device 120 may also detect the face outline of the face in step S220 during the face detection, and the processing device 120 may estimate the locations of the facial features within the face outline. The facial features may correspond to eyes, a nose, and lips of the face F1. In some embodiments, the step S230 may be performed based on an artificial intelligence model or a machine learning model, and the artificial intelligence model or the machine learning model may be trained to detect faces shown in the image and predict the most likely locations of the facial features in the faces. That is, even if some of the facial features are covered by a mask or sunglasses, the locations of the facial features can still be estimated.
In the present embodiment, the image processing system 100 may be incorporated into a mobile device, and the processing device 120 may include a central processing unit of the mobile device. In addition, to operate the artificial intelligence model or the machine learning model for face detection, the processing device 120 may further include multiple processing units that can be used for parallel computing so as to accelerate the speed of face detection. However, the present disclosure is not limited thereto. In some other embodiments, other types of face detection algorithms may be implemented to detect the faces and estimate the locations of the facial features, and the processing device 120 may omit the processing units or include some other types of processing units according to the computational requirements.
In step S240, after the locations of the facial features are obtained by estimation, the processing device 120 may further determine at least one non-occluded region of the face according to occluding conditions of the facial features.
In sub-step S242, the processing device 120 may define feature points on the face outline of the face F1, and in sub-step S244, the processing device 120 may further define a plurality of feature lines according to the feature points defined in sub-step S242. For example, the processing device 120 may locate the coordinates of the feature points along the face outline of the face F1, and define the feature lines by connecting the corresponding feature points. In the present embodiment, the processing device 120 may scan the face one feature line at a time to see if any of the facial features is occluded along the feature line. In this way, the boundaries of the non-occluded region can be determined in step S246.
In the present embodiment, there is a boundary between a non-occluded region and an occluded region which is somewhere between a first feature line and a second feature line that is adjacent to the first feature line if at least one facial feature is occluded along the first feature line and no facial feature is occluded along the second feature line. That is, the processing device 120 may determine whether at least one facial feature is occluded along a first feature line and no facial feature is occluded along a second feature line of the feature lines, and if affirmative, the processing device 120 would choose one of the first feature line, the second feature line, or a line between the first feature line and the second feature line to be a boundary of a non-occluded region.
In some embodiments, since a mask is a commonly worn object that may cover a face, the method 200 may scan regions likely covered by a mask with a higher priority. For example, normally, a mask may cover a lower part of a face; therefore, in step S246, the processing device 120 may start the detection of occluding conditions from a feature line L1 at the bottom of the face F1. If the processing device 120 determines that a facial feature of the face F1 is occluded along the bottom feature line L1, it may mean that the face F1 is covered by a mask. In such case, the processing device 120 may choose the feature line L1 as a lower boundary of the occluded region R1 covered by the mask, and the processing device 120 proceeds to detect occluding conditions of the facial features that are above the bottom feature line L1 so as to find an upper boundary of the occluded region R1.
In the example of
Furthermore, since sunglasses are another commonly worn object that may cover the face, the method 200 may give a higher priority to the regions corresponding to eyes likely covered by the sunglasses on a face. In the example of
Since the boundaries of the occluded regions may also be the boundaries of the non-occluded regions, the boundaries of the non-occluded regions can be found after the boundaries of the occluded regions are detected. In step S250, after the boundaries of the non-occluded regions NR1 and NR2 are determined, the processing device 120 may perform a facial white balance operation on the image IMG0 according to the color data derived from within the non-occluded regions NR1 and NR2 of the face F1. In such case, since the color data used for facial white balance will only be derived from within the non-occluded regions NR1 and NR2 of the face F1, the colors of the mask and the sunglasses will not be taken into account, and thus, the facial white balance operation can correct the skin color more accurately. In some embodiments, the image IMG0 may be divided into a number of blocks, and the color data may be derived from calculating the average color values of R. G, and B in those blocks within the non-occluded regions NR1 and NR2.
In addition, since colors of objects outside of the faces may also be shifted by the light sources and require corrections, a non-facial white balance operation, or a global white balance operation, may also be performed. For example, in step S260, the processing device 120 may perform the non-facial white balance operation on the image IMG0. In some embodiments, since the non-facial white balance operation can be performed by deriving the color data from the whole image without performing face detection, step S260 may be performed before the face detection in step S220 or may be performed in parallel with step S220.
In step S270, the results of the facial white balance operation and the non-facial white balance operation may be combined to generate a final image IMG1. In some embodiments, the results of the facial white balance operation and the non-facial white balance operation may be combined using weightings related to the area occupied by the faces in the image IMG0. For example, if the area of the faces occupies most of the image IMG0, then the weighting of the facial white balance operation will be greater. On the other hand, if the area of the faces takes up only a small portion of the image IMG0, then the weighting of the facial white balance operation will be smaller and the weighting of the non-facial white balance operation will be greater.
Furthermore, as shown in
In some embodiments, the encoding operation in step S280 can be performed by the processing device 120. However, the present disclosure is not limited thereto. In other embodiments, the image processing system 100 may further include an image encoder for encoding the final image IMG1 in step S280.
In addition, in the present embodiment, the image processing system 300 may further include an image signal processor 350. The image signal processor 350 may downscale an image IMG0 captured by an image sensor 310, and a processing device 320 may detect a face using a downscaled image IMG0 received from the image signal processor 350. By reducing the size of the image IMG0, a computing load required by face detection and facial features estimation in steps S220 and S230 may also be reduced. Furthermore, the image signal processor 350 may further be used to provide white balance statistics for the processing device 320 to perform the facial white balance operation and the non-facial white balance operation.
In summary, the image processing system and the method for processing images provided by embodiments of the present disclosure can detect non-occluded regions of a face and derive color data from within the non-occluded regions of the face for the facial white balance operation. In so doing, the facial white balance operation will not be affected by the colors of a mask and/or sunglasses on the face and can thus correct the skin color more accurately.
Although the present disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the disclosure as defined by the appended claims. For example, many of the processes discussed above can be implemented in different methodologies and replaced by other processes, or a combination thereof.
Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. As one of ordinary skill in the art will readily appreciate from the present disclosure, processes, machines, manufacture, compositions of matter, means, methods or steps, presently existing or later to be developed, that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein, may be utilized according to the present disclosure. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods and steps.
