This application claims the priority benefit of Taiwan application serial no. 102115729, filed on May 2, 2013. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification.
1. Field of the Invention
The present invention generally relates to an auto focus (AF) technique, and more particularly, to an AF method and an AF apparatus adopting a stereoscopic image processing technique.
2. Description of Related Art
Generally, an AF technique refers to that a digital camera moves its lens to change a distance between the lens and an object to be photographed, and repeatedly calculates a focus evaluation values (referred to as focusing values in brief hereinafter) of a captured image according to the position of the lens, until the maximum focusing value is determined. To be specific, the maximum focus value of a lens allows a clearest image of the object to be photographed at the current position of the lens.
However, in the hill-climbing technique or regression technique adopted by the existing AF technique, every focusing action requires the lens to be continuously moved and multiple images to be captured to search for the maximum focus value. Thus, it is very time-consuming. Moreover, when a digital camera moves its lens, the lens may be moved too much therefore has to be moved back and forth. As a result, a phenomenon named “Breathing” may be produced. The phenomenon of breathing refers to the change of angle of view of a lens when shifting the focus and therefore destroys the stability of the image. Presently, an AF technique adopting the stereoscopic vision technique for processing images is provided. This AF technique may effectively shorten the focusing time and eliminate the phenomenon of breathing and can increase the focusing speed and image stability therefore becomes increasingly popular in the related field.
However, generally speaking, when the current 3D coordinate position information of each pixel in an image is obtained through image processing of the present stereoscopic vision technique, the position of each point in the image cannot be determined precisely. Moreover, since it is difficult to identify relative depth or precisely determine depth information of each point in a texture-less or flat area, etc., “holes” may be produced in a 3D depth map. In addition, if the AF technique is applied to a handheld electronic device (for example, a smart phone), in order to minimize the size of the product, a stereo baseline of the product has to be reduced as much as possible. As a result, precise positioning may become even more difficult, and more holes may be produced in the 3D depth map. Moreover, the execution of subsequent image focusing procedures may be affected. Therefore, it is an important issue for related researchers to give consideration to focusing speed, stability of images and accuracy of focus positioning of the AF technique.
The present invention is directed to an auto focus (AF) method and an AF apparatus which offer fast speed of auto focusing, optimal image stability and optimal focus positioning accuracy.
The present invention provides an AF method adapted to an AF apparatus which has a first image sensor and a second image sensor. The AF method includes following steps. At least one target object is selected and photographed by the first image sensor and the second image sensor to perform a procedure of three-dimensional (3D) depth estimation, so as to generate a 3D depth map. A block covering at least one initial focusing point is selected according to at least one initial focusing point of the target object. The 3D depth map is queried to read pieces of depth information of a plurality of pixels in the block. It is determined whether the pieces of depth information of the pixels are enough to operate. If yes, a first statistics operation is performed on the pieces of depth information of the pixels to obtain a piece of focusing depth information. If not, the position of the block is moved or the size of the block is enlarged to obtain the piece of focusing depth information. A focusing position regarding the target object is obtained according to the piece of focusing depth information, and the AF apparatus is driven to perform an AF procedure according to the focusing position.
In an embodiment of the present invention, the step of determining whether the depth information of the pixels is enough to operate includes following steps. It is determined whether the piece of depth information of each pixel is a piece of valid depth information, and if yes, the pixel is determined to be a valid pixel. Moreover, it is determined whether a quantity of the valid pixels or a ratio between the valid pixels and the pixels is greater than a predetermined ratio threshold.
In an embodiment of the present invention, after the step of enlarging the size of the block, the AF method further includes determining whether the size of the block is greater than a predetermine range threshold. If not, it is returned to the step of determining whether the pieces of depth information of the pixels are enough to operate. If yes, it is determined that the focusing is failed, and the AF apparatus is driven to perform a pan-focusing procedure, or to perform an AF procedure of contrast type focusing, or does not perform focusing.
In an embodiment of the present invention, the method for selecting the at least one target object includes following steps. At least one click signal for selecting the at leas one target object is received from a user through the AF apparatus, or an object detecting procedure is executed through the AF apparatus to automatically select the at least one target object, and a coordinate position of the at least one initial focusing point is obtained.
In an embodiment of the present invention, when the at least one target object are a plurality of target objects, the step of obtaining the focusing position regarding the target objects includes following steps. The pieces of focusing depth information of the target objects are calculated to obtain average focusing depth information. A focal range is calculated according to the average focusing depth information. It is determined whether the target objects are all within the focal range. If yes, the focusing position regarding the target objects is obtained according to the average focusing depth information.
In an embodiment of the present invention, when the at least one target object are a plurality of target objects, the AF method further includes following steps. A target object position discrete test is executed and it is determined whether the coordinate positions of the target objects are discrete.
In an embodiment of the present invention, the target object position discrete test is a standard deviation test, a variance test or an entropy test.
In an embodiment of the present invention, when it is determined that the coordinate positions of the target objects are discrete, the step of obtaining the focusing position regarding the target objects includes following steps. A maximum target object is selected from the target objects, wherein the maximum target object has characteristic focusing depth information. The focusing position regarding the target objects is obtained according to the characteristic focusing depth information.
In an embodiment of the present invention, when it is determined that the coordinate positions of the target objects are convergent, the step of obtaining the focusing position regarding the target objects includes following steps. Each piece of focusing depth information of the target objects is obtained. A second statistics operation is performed on the pieces of focusing depth information to obtain characteristic focusing depth information, wherein the second statistics operation is a mod operation. The focusing position regarding the target objects is obtained according to the characteristic focusing depth information.
In an embodiment of the present invention, the first statistics operation is a mean operation, a mod operation, a median operation, a minimum value operation or a quartile operation.
The present invention provides an AF apparatus including a first image sensor, a second image sensor, a focusing module and a processing unit. The first image sensor and the second image sensor photograph at least one target object. The focusing module controls a focusing position of the first image sensor and the second image sensor. The processing unit is coupled to the first image sensor, the second image sensors and the focusing module, wherein the processing unit includes a block depth estimator and a depth information determination module. The block depth estimator performs a procedure of 3D depth estimation to generate a 3D depth map, selects a block covering at least one initial focusing point according to the at least one initial focusing point of the target object, and queries the 3D depth map for reading pieces of depth information of a plurality of pixels in the block. The depth information determination module is coupled to the block depth estimator, and determines whether the pieces of depth information of the pixels are enough to operate. If not, the block depth estimator moves the position of the block or enlarge the size of the block for reading the pieces of depth information of the pixels in the block. If yes, the processing unit drives the block depth estimator to perform a first statistics operation on the pieces of depth information of the pixels to obtain a piece of focusing depth information. The processing unit obtains a focusing position regarding the at least one target object according to the piece of focusing depth information and drives the AF apparatus to perform an AF procedure according to the focusing position.
According to the above descriptions, in the AF method and the AF apparatus provided by the present invention, a 3D depth map is generated through a stereoscopic image processing technique. Besides, the piece of depth information of each pixel in the 3D depth map is determined and statistics operations are performed to obtain the focusing position. Thus, the AF apparatus and the AF method provided by the present invention not only can be performed within a single image shooting period, but also resolve the problem of focusing error caused by depth information “holes” in the 3D depth map. Moreover, the AF apparatus and the AF method provided by the present invention can also execute different statistics operation methods on the piece of depth information of each pixel in the block to calculate a piece of suitable focusing depth information. Thereby, the AF apparatus and the AF method provided by the present invention have a faster speed of auto focusing and optimal image stability, and also have optimal focus positioning accuracy.
These and other exemplary embodiments, features, aspects, and advantages of the invention will be described and become more apparent from the detailed description of exemplary embodiments when read in conjunction with accompanying drawings
The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
Reference will now be made in detail to the present preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the description to refer to the same or like parts.
In the present embodiment, the processing unit 150 is coupled to the first image sensor 110, the second image sensor 120, the focusing module 130 and the storage unit 140. The processing unit 150 controls the first image sensor 110, the second image sensor 120 and the focusing module 130, and stores related information in the storage unit 140. The processing unit 150 also drives the block depth estimator 151 and the depth information determination module 152 to execute related instructions.
First, in step S110, at least one target object is selected. To be specific, in the present embodiment, a method of selecting the target object is, for example, to receive at least one click signal for selecting the target object from the user through the AF apparatus 100 to select the target object, and to obtain a coordinate position of at least one initial focusing point IP (shown in
Then, in step S120, the first image sensor 110 and the second image sensor 120 are used to photograph the target object, and perform a procedure of three-dimensional (3D) depth estimation to generate a 3D depth map. Details of the step S120 are described below with reference of
Then, in step S122, the block depth estimator 151 of the processing unit 150 performs 3D depth estimation according to the first image and the second image. In detail, the block depth estimator 151 of the processing unit 150 performs image processing through a stereoscopic vision technique to obtain a 3D coordinate position of the target object in the space and depth information of each point in the image. Then, in step S123, after the block depth estimator 151 of the processing unit 150 obtains the piece of initial depth information of each point, the block depth estimator 151 synthesise all pieces of depth information into the 3D depth map, and stores the 3D depth map in the storage unit 140 to be used in subsequent steps.
However, generally speaking, the 3D depth map generated in the step S123 probably has a plurality of holes HL (shown in
Referring to
In detail, when the related calculation of the 3D depth estimation is performed, a specific value can be given to the pixels on the part of region in the 3D depth map where the aberration cannot be calculated. Moreover, in a subsequent calculation process, the pixels having the specific value are regarded as invalid pixels, and not included in the calculation. For example, a value range of a 10-bit pixel format image falls between 1 and 1023, and the processing unit 150 can set a value of 1023 on the pixel without the piece of valid depth information and set values of 0-1020 on the pixels having valid depth information. In this way, the depth information determination module 152 can quickly determine whether each pixel is a valid pixel, though the invention is not limited thereto. Those with ordinary skill in the art can select other suitable definition of the valid pixels according to an actual requirement, which is not repeated.
Then, in the step S153, it is determined whether the number of the valid pixels or a ratio between the number of the valid pixels and the number of the pixels in the block is greater than a predetermined ratio threshold through the depth information determination module 152. If yes, the step S154 is executed, by which it is determined that the pieces of depth information of the pixels is enough to operate. In detail, the predetermined ratio threshold can be a suitable pixel number or a numerical percentage. For example, the predetermined ratio threshold can be a numerical percentage of 30%, which represent that, the depth information determination module 152 determines that the depth information of the pixels is enough to operate when the ratio between the number of the valid pixels and the number of the pixels in the block is greater than 30%. Moreover, subsequent operations are performed according to a depth information statistic histogram of the block. It should be noticed that the above numerical ratio range is only used as an example, and a threshold value and a range magnitude thereof are not limited by the invention.
However, on the other hand, referring to
On the other hand, when the depth information determination module 152 determines that the pieces of depth information of the pixels is enough to operate, a step S156 shown in
To be specific, the mean operation refers to that the average depth information of the valid pixels in the block is taken as the piece of focusing depth information of the subsequent auto focusing steps. Further, when a distribution the depth information of the valid pixels in the block is not even, the average depth information can be taken as the piece of focusing depth information to cause a focusing effect for balancing each pixel. However, an disadvantage thereof is that it is unable to correctly focus when the pieces of depth information of valid pixels is extremely uneven or difference between the pieces of depth information of pixels is too large. The mod operation is to take the piece of valid depth information with the largest number of pixels in the block as the piece of focusing depth information. The median operation is to take a middle value of the pieces of valid depth information in the block as the piece of focusing depth information, which takes both of the focusing characteristics of the mean operation and the mod operation into account.
The minimum value operation is to take the closest piece of valid depth information in the block as a reference to determine the piece of focusing depth information. However, such operation method is easy to be influenced by noises when only using the minimum value for calculation. The quartile operation is to take a first quartile or a second quartile of the pieces of valid depth information in the block as the piece of focusing depth information. Further, when the first quartile of the pieces of valid depth information in the block is taken as the piece of focusing depth information, it has a similar effect with the method of taking the closest piece of valid depth information in the block as the piece of focusing depth information. However, the influence of noises would be avoided. When the second quartile of the pieces of valid depth information in the block is taken as the piece of focusing depth information, it has a similar effect with the method of taking the middle value of the valid depth information in the block as the piece of focusing depth information.
It should be noticed that although the aforementioned statistic operation method is taken as an example to describe the method of the first statistic operation, the invention is not limited thereto. Those with ordinary skill in the art can select other suitable statistic operation methods to obtain the piece of focusing depth information of the target object according to an actual requirement, which is not repeated.
Then, after the piece of focusing depth information is obtained, a step S160 is executed, by which a focusing position regarding the target object is obtained according to the focusing depth information through the processing unit 150. In detail, the step S160 is to, a depth table may be queried according to the piece of focusing depth information, so as to obtain the focusing position regarding the target object. For example, while executing AF procedure, the focusing module 130 controls steps of a stepper motor in the AF apparatus 100 or controls a current value of a voice coil motor to respectively adjust zoom lenses of the first image sensor 110 and the second image sensor 120 to desired focusing positions, and then performs focusing. Therefore, by calibrating the stepper motor or the voice coil motor, the AF apparatus 100 can obtain a corresponding relationship between the steps of the stepper motor or the current value of the voice coil motor and clear depth of the target object in advance, and the corresponding data can be recorded in the depth table, and stored in the storage unit 140. In this way, the steps of the stepper motor or the current value of the voice coil motor corresponding to the piece of focusing depth information can be queried according to currently obtained focusing depth information of the target object, and focusing position information of the target object is obtained accordingly.
Then, in step S170, the processing unit 150 drives the AF apparatus 100 to execute the AF procedure according to the focusing position. In detail, since the focusing module 130 controls the focusing positions of the first image sensor 110 and the second image sensor 120, after obtaining the focusing position regarding the target object, the processing unit 150 can drive the focusing module 130 of the AF apparatus 100, and accordingly adjust the zoom lenses of the first image sensor 110 and the second image sensor 120 to the focusing positions, so as to complete the AF procedure.
In this way, a 3D depth map is generated through the stereoscopic vision image processing technique, and the piece of depth information of each pixel in the 3D depth map is determined. Moreover, the statistics operation is performed to obtain the focusing position. In this way, the AF apparatus and the AF method of the invention not only can be performed within a single image shooting period, but also resolve the problem of focusing error caused by depth information holes HL in the 3D depth map. Moreover, the depth information of each pixel in the block can be suitably processed by executing different statistics operation methods, so as to calculate the suitable piece of focusing depth information. Therefore, the AF apparatus 100 and the AF method provided by the present invention have a faster speed of auto focusing and optimal image stability, and also have optimal focus positioning accuracy.
Moreover, it should be noticed that the difference between the AF method of the present embodiment and the AF method of the embodiment of
Referring to
Then, in the step S562, it is determined whether the coordinate positions of the target objects are discrete, and different methods of obtaining the focusing position are selected accordingly. In detail, in the present embodiment, the characteristic focusing depth information calculation module 154 is coupled to the block depth estimator 151 and the position discrete test module 153 to obtain the piece of focusing depth information of each target object, and accordingly obtain related characteristic focusing depth information. For example, when it is determined that the coordinate positions of the target objects are discrete, the step S563 is executed, by which a maximum target object of the target objects is selected through the characteristic focusing depth information calculation module 154, wherein the maximum target object has the characteristic focusing depth information. On the other hand, when it is determined that the coordinate positions of the target object are convergent, the step S564 is executed to obtain the piece of focusing depth information of each target object.
Then, in the step S565, a second statistic operation is performed on the pieces of focusing depth information to obtain the characteristic focusing depth information, wherein the second statistic operation is, for example, a mod operation. For example, a method of executing the mod operation is to calculate the piece of focusing depth information of a target object which has the most valid pixels of the target objects that are covered by the block, though the invention is not limited thereto. In other embodiments, those with ordinary skill in the art can select other method for executing the mod operation according to an actual requirement. For example, when the numbers of invalid pixels covered by different target objects are the same, the method for executing the mod operation can also calculate the piece of focusing depth information of the target object with a maximum surface area and perform follow up operations, which is not repeated.
Then, in the step S566, the focusing position regarding the target objects is obtained according to the characteristic focusing depth information obtained in the step S563 or the step S565. In the present embodiment, the method of the step S566 has been described in detail in the step S160 of the embodiment of
As described above, in the AF apparatus and the AF method provided by embodiments of the invention, a 3D depth map is generated through the stereoscopic vision image processing technique, and the piece of depth information of each pixel in the 3D depth map is determined. Moreover, the statistics operation is performed to obtain the focusing position. In this way, the AF apparatus and the AF method of the invention not only can be performed within a single image shooting period, but also resolve the problem of focusing error caused by depth information “holes” in the 3D depth map. Moreover, the AF apparatus and the AF method of the invention can also suitably process the depth information of each pixel in the block by executing different statistics operation methods, so as to calculate the suitable piece of focusing depth information. Therefore, the AF apparatus and the AF method provided by the present invention have a faster speed of auto focusing and optimal image stability, and also have optimal focus positioning accuracy.
It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the invention without departing from the scope or spirit of the invention. In view of the foregoing, it is intended that the invention cover modifications and variations of this invention provided they fall within the scope of the following claims and their equivalents.
Number | Date | Country | Kind |
---|---|---|---|
102115729 | May 2013 | TW | national |