1. Field of the Invention
The subject matter disclosed generally relates to auto-focusing electronically captured images.
2. Background Information
Photographic equipment such as digital cameras and digital camcorders may contain electronic image sensors that capture light for processing into still or video images, respectively. Electronic image sensors typically contain millions of light capturing elements such as photodiodes.
Many image capturing devices such as cameras include an auto-focusing system. The process of auto-focusing includes the steps of capturing an image, processing the image to determine whether it is in focus. A feedback signal is generated from the process to vary a position of a focus lens (“focus position”).
An auto focus image system that includes a pixel array coupled to a focus signal generator. The pixel array captures an image that has a plurality of edges. The generator generates a focus signal that is a function of a plurality of edge-sharpness measures for the plurality of edges. The generator compares a sequence of gradients across the edge with one or more reference sequences of gradients and/or reference curves defined by data retrieved from a non-volatile memory. The generator may reject or de-emphasize the edge using result of the comparison. The edge sharpness measure is a quantity whose unit is a positive or negative, integer or non-integer power of a unit of length. It may be measured from the edge and/or a reference sequence/curve matched to the edge, or may be retrieved for the matched reference sequence/curve from a non-volatile memory.
Aspects of the invention include the following:
1. A method for evaluating a degree of sharpness of an image on basis of a plurality of edges within said image, comprising:
curve-fitting a gradient profile of an edge among said plurality of edges with a sequence of two or more reference gradients, among which at least two reference gradients have different gradient values;
performing a comparison of said sequence with said gradient profile; and,
making a determination, based at least in part on a result of said comparison, of a weight of one or more quantities from said gradient profile and/or said sequence in contributing to an evaluation of said degree of sharpness.
2. The method of aspect 1, wherein said degree of sharpness is a focus signal used to generate a focus drive signal to move a focus lens.
3. The method of aspect 1, wherein said result is a binary result whose use in said determination consists of either the one or more quantities contribute to said evaluation or not.
4. The method of aspect 1, wherein said result can take any one of more than two values and said weight is a function of said result.
5. The method of aspect 1, wherein said at least two reference gradients are retrieved from a volatile memory.
6. The method of aspect 1, wherein said at least two reference gradients are computed from a formula retrieved from a non-volatile memory.
7. The method of aspect 1, wherein each one of said one or more quantities has a unit that is a positive or negative, integer or non-integer power of a unit of distance, given that each image sample has a unit of energy and distance between any two image samples has a unit of distance.
8. The method of aspect 7, wherein said one or more quantities includes a quantity computed from a plurality of image samples of said edge.
9. The method of aspect 7, wherein said one or more quantities includes a quantity computed from a plurality of gradients of said edge.
10. The method of aspect 7, wherein said one or more quantities includes a quantity associated with said sequence and/or a reference curve and retrieved from a non-volatile memory.
11. The method of aspect 7, wherein said one or more quantities includes an edge width.
12. The method of aspect 1, wherein said sequence is computed from a reference curve defined by data retrieved from non-volatile memory.
13. The method of aspect 12, wherein said data include a formula and its coefficients.
14. The method of aspect 1, wherein said sequence is retrieved from a non-volatile memory.
15. The method of aspect 1, wherein said sequence is curve-fitted to at least two gradients on one side of a peak gradient of the gradient profile but not to any gradient on another side of the peak gradient.
16. The method of aspect 1, wherein said sequence is curve-fitted to gradients on both sides of the peak gradient.
17. The method of aspect 1, wherein said comparison comprises:
extracting a parameter from the gradient profile; and,
comparing the parameter extracted from the gradient profile with a predetermined parameter retrieved from a non-volatile memory for the sequence of reference gradients.
18. The method of aspect 1, wherein said comparison comprises:
forming pairs between said reference gradients and gradients of said gradient profile; and,
comparing between said reference gradients and said gradients of said pairs.
21. An image capture system that evaluates a degree of sharpness of an image on basis of a plurality of edges in the image, comprising:
a memory or a bank of registers that stores (a memory means for storing) data that define one or more reference gradient curves and/or sequences of reference gradients.
22. The image capture system of claim 21, wherein said data define a spacing of a reference peak from a reference gradient in at least one of said one or more sequences.
23. The image capture system of claim 21, further comprising:
an evaluation unit, said evaluation unit being configured to compare a sequence of reference gradients or reference gradient curve defined in said memory with a gradient profile across one of said plurality of edges, said sequence of reference gradients or reference gradient curve being one among said one or more reference gradient curves or sequences.
24. The image capture system of claim 23, wherein said evaluating unit compares a parameter of the sequence of reference gradients or reference gradient curve with a parameter measured from the gradient profile.
25. The image capture system of claim 24, wherein said evaluation unit comprises an arithmetic circuit.
26. The image capture system of claim 25, wherein said evaluation unit comprises a memory.
27. The image capture system of claim 21, further comprising:
a parameter extraction unit, said parameter extraction unit being configured to extract from said gradient profile one or more parameters that describe said gradient profile.
28. The image capture system of claim 21, further comprising:
a sequence selector, said sequence selector being configured to select one or more among said plurality of sequences on basis of said one or more parameters.
29. A method for evaluating a degree of sharpness of an image on basis of a plurality of edges within said image, comprising:
curve-fitting a gradient profile of an edge among said plurality of edges with a sequence of two or more reference gradients, among which at least two reference gradients have different gradient values;
performing a comparison of said sequence with said gradient profile under a predetermined criterion; and,
making a determination, based at least in part on a result of said comparison, on how to use one or more quantities from said gradient profile and/or said sequence in evaluating said degree of sharpness.
30. The method of claim 29, further comprising:
generating a focus signal from said plurality of edges via modifying a contribution from said edge on basis of said determination.
31. The method of claim 30, wherein said edge is prevented from influencing said generating of said focus signal if said comparison finds that said gradient profile is not sufficiently similar to said sequence under said predetermined criterion.
32. The method of claim 30, wherein said comparison produces a graded result and said focus signal receives contribution of an edge width of said edge to a relative extent compared with other edges, said relative extent depending on said graded result.
Disclosed is an auto focus image system that includes a pixel array coupled to a focus signal generator. The pixel array captures an image that has at least one edge with a width. The focus signal generator may generate a focus signal that is a function of the edge width and/or statistics of edge widths. An auto focus image system that includes a pixel array coupled to a focus signal generator. The pixel array captures an image that has at least one edge with a width. The generator generates a focus signal that is a function of the edge width and various statistics of edge width. The generator may eliminate an edge having an asymmetry of a gradient of an image signal. The generator may also eliminate an edge that fails a template for an associated peaking in the gradient. A processor receives the focus signal and/or the statistics of edge widths and adjusts a focus position of a focus lens. The edge width can be determined by various techniques including the use of gradients. A histogram of edge widths may be used to determine whether a particular image is focused or unfocused. A histogram with a large population of thin edge widths is indicative of a focused image.
Referring to the drawings more particularly by reference numbers,
The focus signal generator 120 receives a group of control signals 132 from the processor 112, in addition, and may output signals 134 to the processor 112. The output signals 134 may comprise one or more of the following: a focus signal 134, a narrow-edge count, and a set of numbers representing a statistics of edge width in the image. The processor 112 may generate a focus control signal 136 that is sent to the drive motor/circuit 118 to control the focus lens 104. A focused image is ultimately provided to the display 114 and/or stored in the memory card 116. The algorithm(s) used to adjust a focus position may be performed by the processor 112.
The pixel array and circuits 108, A/D Converter 110, focus signal generator 120, and processor 112 may all reside within a package. Alternately, the pixel array and circuits 108, A/D Converter 110, and focus signal generator 120 may reside within a package 142 as image sensor 150 shown in
The EDWM unit 206 may transform the input image such that the three signals of the image, red (R), green (G) and blue (B) are converted to a single image signal. Several techniques can be utilized to transform an image to a single image. RGB values can be used to calculate a luminance or chrominance value or a specific ratio of RGB values can be taken to form the single image signal. For example, the luminance value can be calculated with the equation Y=0.2126*R+0.7152*G+0.0722*B, where Y is luminance value. The single image signal may then be processed by a Gaussian filter or any lowpass filter to smooth out image data sample values among neighboring pixels to remove a noise.
The focus signal generator 120, 120′, 120″ is not limited to grayscale signal. It may operate on any one image signal to detect one or more edges in the image signal. Or it may operate on any combination of the image signals, for example Y, R-G, or B-G. It may operate on each and every one of the R, G, B image signals separately, or any one or more combinations thereof, to detect edges. It may form statistics of edge widths for each of the R, G, B image signals, or any combination thereof. It may form a focus signal from statistics of edge widths from one or more image signals.
The focus signal generator includes an edge detector to identify an edge in an image signal. The edge detector may use a first-order edge detection operator, such as Sobel operator, Prewitt operator, Roberts Cross operator, or Roberts operator. The edge detector may use a higher-order edge detection operator to identify the edge, for example a second order operator such as a Laplacian operator. The edge detector may use any one of the known edge detection operators or any improved operator that shares a common edge detection principle of any of the known operators.
Where the edge detector uses a first-order edge detection operator, a gradient (i.e. first derivative) of the image signal is computed. There are various methods available to calculate the gradient, including using any one of various first order edge detection operators such the Sobel operator, the Prewitt operator, the Roberts Cross operator, and the Roberts operator. The Roberts operator has two kernels which are single column or single row matrices: [−1 +1] and its transpose. The Roberts Cross operator has two kernels which are 2-by-2 matrices: [+1, 0; 0, −1] and [0, +1; −1, 0], shown in the format of [<first-row vector; second-row vector; third-row vector] like in Matlab. The Prewitt and the Sobel operator are basically have the same kernels, [−1, 0, +1] taking gradient in a direction of the row and its transpose taking gradient in a direction of the column, further multiplied by different lowpass filter kernels performing lowpass filterings perpendicular to the respective gradient directions. Gradients across the columns and the rows may be calculated to detect vertical and horizontal edges respectively, for example using a Sobel-X operator and a Sobel-Y operator, respectively. Sobel X-operator at pixel location [k, q] where k is a row number and q is a column number, is given by the equation Sx[k, q]=U[k, q+1]−U[k, q−1]. Sobel Y-operator at the same location is given by the equation Sy[k,q]=U[k+1,q]−U[k−1,q], where U is an image signal of the processed image.
Where the edge detector uses a second-order operator, a second derivative (such as the Laplacian) of the image signal is computed.
Each pixel may be tagged either a horizontal edge (‘H’) or a vertical edge (‘V’) if either vertical or horizontal gradient magnitude exceeds a predetermined lower limit (“elimination threshold”), e.g. 5 for an 8-bit image, or no edge if neither is true. This lower limit eliminates spurious edges due to gentle shading or noise. A pixel may be tagged a vertical edge if its horizontal gradient magnitude exceeds its vertical gradient magnitude by a predetermined hysteresis amount or more, e.g. 2 for an 8-bit image, and vice versa. If both gradient magnitudes differ less than the hysteresis amount, the pixel gets a direction tag same as that of its nearest neighbor that has a direction tag already determined. For example, if the image is scanned from left to right in each row and from row to row downwards, a sequence of inspection of neighboring pixels may be the pixel above first, the pixel above left second, and the pixel on the left third, and the pixel above right last. Applying this hysteresis helps to ensure that adjacent pixels get similar tags if each of them has nearly identical horizontal and vertical gradient magnitudes.
The image, gradients and tags may be scanned horizontally for vertical edges, and vertically for horizontal edges. Each group of consecutive pixels in a same row, having a same horizontal gradient polarity and all tagged for vertical edge may be designated a vertical edge if no adjacent pixel on left or right of the group are likewise. Likewise, each group of consecutive pixels in a same column having a same vertical gradient polarity and all tagged for horizontal edge may be designated a horizontal edge if no adjacent pixel above or below the group satisfies the same. Thus horizontal and vertical edges may be identified.
Each edge may be refined by removing pixels whose gradient magnitudes are less than a given fraction of the peak gradient magnitude within the edge.
Edge width may be calculated in any one of known methods. One method of calculating edge width is simply counting the number of pixels within an edge. An alternate method of calculating edge width is shown in
Another alternative edge width calculation method is to calculate a difference of the image signal across the edge (with or without edge refinement) and divide it by a peak gradient of the edge.
Alternatively, edge width may be a distance between a pair of positive and negative peaks (or interpolated peak(s)) of the second order derivative of the image signal across the edge. Other alternatives are possible, to be described under the heading “edge-sharpness measure” further into this specification.
It will be seen further into this specification under the heading “edge-sharpness measure” that there are other alternatives than a width, which is merely one example of a edge-sharpness measure that is essentially independent of illumination of the scene.
Although each edge may be assigned to one prescribed direction (e.g. vertical direction or horizontal direction) or another, perpendicular, prescribed direction (e.g horizontal direction or vertical direction) and may have its edge width measured in a direction perpendicular to that assigned edge direction, the boundaries between regions of different image signal values in the image from which these edges arise may not be and usually are not aligned perfectly with either prescribed directions. In
For purposes of calculating a focus signal from edge widths, the edge widths measured in one or the other of those prescribed directions are to be corrected by reducing them down to be widths in directions perpendicular to directions of the respective edges. The Edge Detection and Width Measurement Unit 206 performs such a correction on edge widths. As shown in
Each horizontal or vertical edge's edge width may be corrected for its slant from either the horizontal or vertical orientation (the prescribed directions), respectively.
By way of example,
From step 502 to step 506, a slant angle φ is found. For each vertical edge, at step 502, locate the column position where the horizontal gradient magnitude peaks, and find the horizontal gradient x. At step 504, find where the vertical gradient magnitude peaks along the column position and within two pixels away, and find the vertical gradient y.
At step 506, find the slant angle φ=tan−1 (y/x). At step 506, the slant angle may be found by looking up a lookup table. Although steps 502 to 506 present one specific procedure and method to find the slant angle, other procedures and methods known in the art may be used instead.
Finally, at step 508, scale down the edge width by multiplying with cos(φ), or with an approximation thereto as one skilled in the art usually does in practice.
A first modification of the process shown in
A second modification is to calculate a quotient y/x between a vertical gradient y and a horizontal gradient x to produce a quotient q, then use q to input to a lookup table that has entries for various values of q. For each value of q, the lookup table returns an edge width correction factor. The edge width correction factor may be an approximation to cos(tan−1(q)) to within 20%, preferably within 5%.
For finding the slant angle φ (or an approximation thereto such that the correction factor is accurate to within 20%) and subsequently the correction factor cos(φ) (or an approximation thereto), or to directly find the correction factor without finding the slant angle φ (as in the first and second modifications), the values of x and y may be obtained in steps 502 to 506, but other methods may be employed instead.
A third modification is to perform the following for each one of a plurality of pixels in the edge: (a) find horizontal gradient x and vertical gradient y both for a pixel, (b) find q=y/x for this pixel, and (c) find a correction factor that corresponds to q, for instance cos(tan−1(q)) or an approximation thereto to within 20%. Finally, find the correction factor for the edge width by averaging across the correction factor from each of the plurality of pixels. The average may be a weighted average, such as one in which a pixel that has a larger horizontal gradient is given a larger weight than another pixel that has a lesser horizontal gradient.
Other modifications are possible along these directions or other.
Adjacent edges may be prevented altogether from contributing to a focus signal, or have their contributions attenuated, if their peak gradient magnitudes are below a predetermined fraction of an adjacent wider edge's peak gradient magnitude.
The significant decline, e.g. 20% or greater, in peak gradient magnitude for a narrower edge adjacent to a wider edge having an opposite-signed gradient gives a hint that the blurred image is not well focused, and thus the narrower edge should not be relied upon as an indication that the blurred image is sharp.
Likewise, mutually adjacent edges of alternating gradient polarities should not be relied upon for such indication even if their edge width are small as long as they are in close proximity to each other, e.g. no more than 1 pixel apart (“minimum edge gap”). The minimum edge gap is in terms of a number of pixels, e.g. 1, or 2, or in between.
Furthermore, given that one edge may have been eliminated due to having a peak gradient less than the elimination threshold, two successive edges having an identical gradient polarity and spaced no more than two times the minimum edge gap plus a sharp_edge_width (sharp_edge_width is a number assigned to designate an edge width of a sharp edge) apart may be used as a condition for eliminating or demoting a contribution from one or both of the two mutually adjacent edges. either.
The Edge Detection and Width Measurement Unit 206 may execute the following algorithm for eliminating closely-packed narrower edges based on a screen threshold established from a wider edge, and a modulation screen flag that can be turned on and off.
For each edge, the screen threshold and screen flag to be used for the immediate next edge of an opposite polarity are determined according to the process of the flowchart shown in
Given the screen threshold and screen flag, an edge may be eliminated unless one of the following conditions is true: (a) the screen flag is off for this edge, (b) a peak gradient magnitude of the edge is not smaller than the screen threshold for this edge. To conditions (a) and (b) may be added condition (c) the edge width is not less than sharp_edge_width+1, where a number has been assigned for sharp_edge_width to designate an edge width of a sharp edge, and where the “+1” may be varied to set a range of edge widths above the sharp_edge_width within which edges may be eliminated if they fail (a) and (b). For the example shown in
The image input by the focus signal generator 120 may have pixels laid out in a rectangular grid (“pixel grid”) rotated at 45 degrees with respect to a rectangular frame of the image. In this case, the X- and Y-directions of the edge detection operations and width measurement operations may be rotated likewise.
In the above description, sharpness of image of an edge is represented by a width of the edge measured from a sequence of gradients across the edge with the gradients oriented across the edge, there are alternatives that work on similar principle. In essence, what allows the focus signal generated in this manner is that the individual edges contributes a quantity (hereinafter “edge-sharpness measure”) that is independent of scaling the image data by, for example, 20%, or essentially independent, such as changes by not more than 5% for 20% scaling down of the image data, thus helping to make the focus signal independent of or far less dependent on illumination of the scene of the image or reflectivity of objects in the scene compared with the conventional contrast detection method.
In the present focus signal generator 120, any edge-sharpness measure that has the above characteristic of being independent of or essentially independent of 20% scaling down of the image data in addition is a good alternative to the width measured from a gradient or interpolated gradient to another gradient or interpolated gradient of a same gradient value.
The alternative edge-sharpness measure preferably has a unit that does not include a unit of energy. The unit of the edge-sharpness measure is determined on basis two points: (a) each sample of the image data on which the first-order edge-detection operator operates on has a unit of energy, (b) distance between samples has a unit of length. On basis of points (a) and (b), a gradient value has a unit of a unit of energy divided by a unit of length. Likewise, contrast across the edge or across any undivided portion of the edge has a unit of energy. Therefore the contrast is not a good edge-sharpness measure, as the unit reveals that it is affected by illumination of the scene and reflectivity of the object. Neither is peak gradient of the edge, because the unit of the peak gradient has a unit of energy in it, indicating also that it is responsive to a change in illumination of the scene. On the other hand, peak gradient of the edge divided by a contrast of the edge is a good edge-sharpness measure, as it has a unit of the reciprocal of a unit of length. As another example, the count of gradients whose gradient values exceeds a certain predetermine fraction of the peak gradient is a good edge-sharpness measure, as the count is simply a measure of distance quantized to the size of the spacing between contiguous gradients, hence having a unit of length.
It is here noted that, in the generation of the edge-sharpness measure, a gradient may be generated from a first-order edge detection operator used to detect the edge, or may be generated from a different first-derivative operator (i.e. gradient operator). For example, while the Sobel operator (or even a second-order edge detection operator, such as a Laplacian operator) may be used to detect the edge, the Roberts operator whose kernels are simply [−1, +1] and its transpose, which is simply subtracting one sample of the image data from the next sample in the orientation of the gradient operator, with the resulting gradient located midway between the two samples. Edges may be detected with a higher-order edge detection operator than first-order independently of one or more derivative operators used in generating the edge-sharpness measure or any of the shape measures described in the next section.
Viewing it another way, the edge-sharpness measure should have a unit of a power of a unit of length, for example a square of a unit of length, a reciprocal of a unit of length, the unit of length itself, or a square-root of a unit of length.
Any such alternative edge-sharpness measure can replace the edge width in the focus signal generator 120.
To correct for a slant of the edge, the correction factor as described above with reference to
Several examples of alternative edge-sharpness measures are described below with reference to the drawings in
In the example of
One edge-sharpness measure may involve merely one side of the edge but not the other. For example, in
The below method of qualifying of edges may be performed in the Edge Detection & Width Measurement Unit 206 and is described below with reference to
In this method, the gradient profile is compared with a sequence of reference gradients. Each of the reference gradients has a gradient value and a spacing to the next reference gradient in the sequence. The reference gradients are generated under a predefined relationship between them. For example, the sequence may be generated from and/or stored as a sequence in a lookup table. As another example, the sequence may be defined by way of a mathematical formula stored in a non-volatile memory.
The reference gradient curve 3402 may be defined via a mathematical formula stored in a non-volatile memory. The formula may use different mathematical expressions for different sections of the reference gradient curve 3402. For example, it may use a hyperbola formula to define a range from a reference peak of the reference gradient curve 3402 to 50% down from the reference peak, and a third degree polynomial on each side of the curve to define a range further down from 50% below the reference peak. Preferably, the mathematical formula is parameterized by (or associated with) one or more parameters, for example any one or more among a reference width, an edge slant angle (that can be used to match a slant angle of an edge), a peak gradient level, a zoom factor (of a zoom lens on the camera), an aperture size (of the lens of the camera), etc. Coefficients that define the formula may be stored in a lookup table (e.g. in a nonvolatile memory, or loaded into a read-writable memory such as an SRAM or a DRAM from a nonvolatile memory) and indexed (i.e. keyed) by various combinations of the parameter values.
Alternatively, the reference gradients may be defined via and/or in a lookup table. The lookup table may specify more than one sequence. Each sequence is specified in the lookup table with a gradient value for each reference gradient and a spacing to the next reference gradient value in the sequence. If the spacing is constant throughout the sequence, it may be specified once only for the entire sequence. If the spacing is constant for the entire lookup table for all sequences, it may be specified once only for the lookup table. Alternatively, the spacing may assume a default value, for example, 1.0. A reference peak may be specified for the sequence, to have a peak gradient level and a relative position (i.e. spacing and whether it is before/after) to a reference gradient in the sequence.
Further reference gradients may be interpolated from the reference gradients generated directly from the source (e.g. the lookup table or the mathematical formula) as a predefined reference sequence generator. It may be useful to interpolate a reference gradient to where the gradient profile has a steep slope. For example, in
On the other hand, gradients may be interpolated from the gradient profile to where a reference gradient is positioned relative to the gradient profile. It may be useful to interpolate an interpolated gradient to where a reference gradient is. For example, interpolated gradient 3514 (marker “□”) is interpolated from gradients 3502, 3504, 3506 to be near reference gradient 3402. Interpolated gradients are likewise marked “□” in
The lookup table may store a plurality of sequences, each for a different combination of parameter values. Each sequence may be parameterized by (or associated with) one or more parameters, for example any one or more among a reference width, an edge slant angle, a peak gradient level, a zoom factor (of a zoom lens on the camera), an aperture size (of the lens of the camera), etc. For example, a sequence stored in the lookup table may correspond to a zoom factor of 2 times, an aperture of F3.5, a width of 4.5 at 50% down from peak, an edge slant angle of 30 degrees, whereas another sequence may correspond to a zoom factor of 0.5 times, an aperture of F2.8, a width of 3.7 at 50% down from peak, and an edge slant angle of 0 degree.
The lookup table may be stored in a memory, which may be a RAM (SRAM or DRAM) or a non-volatile memory, for example a flash memory, or simply a bank of registers. The lookup table may be stored on a nonvolatile memory outside the focus signal generator but within the image capture system 102.
A shorter sequence may be extracted from a sequence stored in the lookup table such that the extracted sequence consists of a proper subset of the reference gradients of the sequence stored in the lookup table. Spacings between the extracted reference gradients, as well as other data such as the relative positions of the extracted reference gradients with respect to the reference peak may be extracted along. For example, a sequence of 30 reference gradients stored in the lookup table may have a uniform spacing of 0.2 between successive reference gradients and may have a reference peak that coincides with the 16th reference gradient in the sequence, whereas an extraction extracts the 7th, the 13th and the 22nd reference gradients to form a shorter sequence consisting of these three reference gradients. In the extracted sequence of this example, a spacing from the first reference gradient to the second reference gradient is 1.2, a spacing from the second reference gradient to the third reference gradient is 1.8, and the reference peak is noted to lie between the second and third reference gradients and at a spacing of 0.6 from the second reference gradient.
The sequence of reference gradients may be curve-fitted to the gradient profile or a portion thereof. For example, the sequence may be fitted to the gradient profile from the peak gradient and 50% down, as
All the reference gradients may be shifted in position together relative to the gradient profile to achieve a better curve-fitting. For example, a gradient profile may have a first, a second, a third and a fourth gradients and may have a uniform spacing of 1.0 from one gradient to the next, whereas a sequence of reference gradients consisting of a first, a second, a third and a fourth reference gradients and having a uniform spacing of 1.0 from one reference gradient to the next may be positioned relative to the gradient profile such that the first reference gradient is between the first and second gradients and at spacing of 0.2 from the first gradient, the second reference gradient is between the second and third gradients and at spacing of 0.2 from the second gradient, and so on, as shown in
If a reference peak location is provided along with the sequence of reference gradients, the sequence may be aligned to the gradient profile by aligning the reference peak to a position determined from the gradient profile in a predetermined manner. There are more than one way to determine this position. In one way, this position can be a midpoint between two gradient(s)/interpolated gradient(s), one on each of the two sides of the gradient profile at a predetermined gradient level, which may be specified as a predetermined percentage down from the gradient level of the peak gradient or an interpolated peak, as shown in
The gradient profile is compared with the sequence to generate a comparison result under a predefined manner, which may or may not be same or similar to that for the criterion of curve-fitting. Interpolated gradients/reference gradients may be included in the comparison, especially where doing so can improve precision and/or accuracy for the comparison. One approach to the comparison is to evaluate how a quantity measured from the gradient profile relates to a quantity associated with the sequence. For instance, a width of the gradient profile may be compared with a reference width of the sequence. Both widths may be measured at a gradient level, which may be predetermined as a predetermined percentage down from the peak gradient or an interpolated peak. For example, in
Another approach to compare the gradient profile and the reference sequence is to compare between individual gradient and reference gradient in pairs. As shown in
One or more sequences may be selected to curve-fit the gradient profile. If more than one sequence is selected, a sequence that best fits the gradient profile may be chosen to have its comparison result reported. Sequence(s) may be selected on basis of one or more parameters associated with the gradient profile. An example is a width of the gradient profile measured at a predetermined percentage down from the peak gradient or an interpolated peak. Another example is a one-sided width of the gradient profile measured from an interpolated peak position (or a midpoint between two interpolated gradients of same gradient value) to a gradient/interpolated gradient at a predetermined percentage down from the peak gradient or the interpolated peak. A third example is a slant angle of an edge associated with the gradient profile or giving rise to the gradient profile. A fourth example is an area under the gradient profile (down to a certain percentage down from the peak/interpolated peak). A fifth example is a spacing of the interpolated peak position from the peak gradient. A sixth example is a zoom lens zoom factor. A seventh example is a size of lens aperture.
If the comparison result indicates that there is dissimilarity beyond a threshold, the focus signal generator may de-emphasize or reject altogether an edge associated with the gradient profile from entering a calculation for a focus signal or edge count or focus control. Where the edge is to be de-emphasized, a reduced weight is assigned to the edge than otherwise for the edge's participation in forming the focus signal. This is similar to how the Width Filter de-emphasizes an edge by applying lesser weight to its contribution to the focus signal. Thus, if the area A3 of the gradient profile of the edge as shown in
As shown in an embodiment in
It is noted that, in this disclosure, a quantity from an edge, such as a gradient level, is said to be normalized when it is divided by, by default unless otherwise specified, either a peak gradient value of the edge or gradient value of an interpolated peak. For example, in
Where the edge contributes to the focus signal, the edge sharpness measure is a quantity whose unit is a power of a unit of length. The power may be positive or negative, integer or non-integer. The edge-sharpness measure may be measured from the edge and/or a reference sequence/curve matched to the edge in any manner described under the preceding section under the heading of “Edge-sharpness measures,” or may be retrieved for a matched reference sequence/curve from a non-volatile memory.
Aside from computing the edge-sharpness measure from the gradient profile of the edge, the edge-sharpness measure can also be computed from a sequence of reference gradients matched to the gradient profile according to any one of the measurement methods described above for computing a edge-sharpness measure because the sequence of reference gradients is a sequence of gradients just like the gradient profile, which is clear from each of the reference sequences marked in “x” in
Where the edge is permitted to contribute to the focus signal, the edge may contribute one or more edge-sharpness measures. As a weighted average involves a sum between various contributed quantities and, as is well known in science, all quantities in a sum must have the same unit, e.g. the unit of length or the square of the unit of length or the reciprocal of the unit of length, etc., if two or more edge-sharpness measures are contributed by the edge then they must be converted to the same unit. For example, if the edge-sharpness measures being contributed consists of a width and an area, then the width may be squared first before being used in the weighted average along with the area.
Below describes a function of length filter 212. Broadly defined, length filter 212 creates a preference for edges that each connects to one or more edges of a similar orientation. A group of edges that are similarly oriented and mutually connected within the group (“concatenated edge”) is less likely to be due to noise, compared with an isolated edge that does not touch any other edge of similar orientation. The more edges of a similar orientation thus concatenated together, the lesser the chance of them being due to noise. The probability of the group being due to noise falls off exponentially as the number of edges within the group increases, and far faster than linearly. This property can be harnessed to reject noise, especially under dim-lit or short-exposure situations where the signal-to-noise ratio is weak, e.g. less than 10, within the image or within the region of interest. The preference may be implemented in any reasonable method to express such preference. The several ways described below are merely examples.
A first method is to eliminate edges that belong to vertical/horizontal concatenated edges having lengths lesser than a concatenated length threshold. The concatenated length threshold may be larger when the region of interest is dimmer. For example, the concatenated length threshold may start as small as 2, but increases to 8 as a signal-to-noise ratio within the region of interest drops to 5. The concatenated length threshold may be provided by the processor 112, 112′, 112″, for example through a ‘length command’ signal, shown in
A second method is to provide a length-weight in the length filter 212 for each edge and apply the length-weight to a calculation of focus signal in the focus signal calculator 210. An edge that is part of a longer concatenated edge receives a larger weight than one that is part of a shorter concatenated edge. For example, the length-weight may be a square of the length of the concatenated edge. Thus, a contribution of each edge towards the focus signal may be multiplied by a factor A/B before summing all contributions to form the focus signal, where B is a sum of the length-weights of all edges that enter the focus signal calculation, and A is a length-weight of the edge. Likewise, the edge-width histogram, which may be output as part of signals 134, may have edges that are members of longer concatenated edges contribute more to the bins corresponding to their respective edge width, thus preferred, instead of all edges contribute the same amount, e.g. +1. Thus, for example, each edge may contribute A/C, where C is an average value of A across the edges. Similarly, the narrow-edge count may have edges that are members to longer concatenated edges contribute more. Thus, for example, the contribution from each edge may be multiplied by A/D, where D is an average of A among edges that are counted in the narrow-edge count.
A group of N vertical (horizontal) edges where, with the exception of the top (leftmost) and the bottom (rightmost) ones, each edge touches two other vertical (horizontal) edges, one above (to the left of) itself, the other below (to the right of) itself, is a vertical (horizontal) concatenated edge of length N. The top (leftmost) edge needs only touch one edge below (to the right of) itself. The bottom (rightmost) edge needs only touch one edge above (to the left of) itself.
In a situation (not shown) where a vertical (horizontal) concatenated edge has two or more branches, i.e. having two edges in a row (column), the length may be defined as the total number of edges within the concatenated edge. Alternately, the length may be defined as the vertical (horizontal) distance from a topmost (leftmost) edge therein to a bottommost (rightmost) edge therein plus one.
There are other possible ways to define a concatenated length other than the above proposals. For example, a definition of a length for a concatenated edge shall have a property that the length is proportional to the number of member edges within the concatenated edge at least up to three. This is to be consistent with the previously stated reasoning that more edges being mutually connected by touching each other exponentially reduces a probability that the concatenated edge is caused by a noise, and as such the length should express a proportionality to the number of member edges within the concatenated edge up to a reasonable number that sufficiently enhances a confidence in the concatenated edge beyond that for a single member. The length filter 212 may de-emphasize or eliminate and thus, broadly speaking, discriminate against an edge having a concatenated length of one. The length filter 212 may discriminate against an edge having a concatenated length of two. The length filter 212 may discriminate against an edge having a concatenated length of three, to further reduce an influence of noise. The length filter 212 may do any one of these actions under a command from the processor.
Although shown in
In an alternate embodiment of a focus signal generator, the fine switch 220 may be removed so that the focus signal calculation unit 210 receives a first set of data not filtered by the width filter 209 and a second set filtered, and for each calculates a different focus signal, gross focus signal for the former, fine focus signal for the latter, and outputs both to the processor 112, 112′.
Refer next to
In addition, the Width Filter 209 may calculate a total count of the edges whose edge widths fall within the narrow-edge range and output as part of output signals 134. Narrow-Edge Count may be input to and used by the focus system controller (processor 112) to detect a presence of sharp image and/or for initiating tracking.
Referring next to the focus signal calculator 210 of
A focus control system may use the gross focus signal to search for the nearest sharp focus position in a search mode. It can move the focus position away from the current focus position to determine whether the gross focus signal increases or decreases. For example, if the gross focus signal increases (decreases) when the focus position moves inwards (outwards), there is a sharp focus position farther from the current focus position. The processor 112, 112′, 112″ can then provide a focus drive signal to move the focus lens 104 in the direction towards the adjacent sharp focus position.
A focus control system may use the fine focus signal to track an object already in sharp focus to maintain the corresponding image sharp (thus a “tracking mode”) despite changes in the scene, movement of the object, or movement of the image pickup apparatus. When an object is in sharp focus, the fine focus signal level is stable despite such changes. Hence a change in the fine focus signal suggests a change in focus distance of the object from the image pickup apparatus. By “locking” the focus control system to a given fine focus signal level near the minimum, for example between 2.0 to 2.5 in this example, in particular 2.1, any shift in the fine focus signal level immediately informs the processor 112, 112′, 112″ of a change in the focus distance of the object. The processor 112, 112′, 112″ can then determine a direction and cause the focus lens 104 to move to bring the fine focus signal level back to the “locked” level. Thus the image pickup apparatus 102, 103, 103′, 103″ is able to track a moving object.
A focus control system, e.g. as implemented in algorithm in processor 112, 112′, 112″, may use narrow-edge count to trigger a change from a search mode to a tracking mode. In the tracking mode, the focus control system uses the fine focus signal to “lock” the object. Before the focus position is sufficiently near the sharp focus position for the object, the focus control system may use the gross focus signal to identify the direction to move and regulate the speed of movement of the lens. When a object is coming into sharp focus, narrow-edge count peaks sharply. The processor 112, 112′, 112″ may switch into the tracking mode and use the fine focus signal for focus position control upon detection of a sharp rise in the narrow-edge count or a peaking or both. A threshold, which may be different for each different sharp focus position, may be assigned to each group of objects found from an end-to-end focus position “scan”, and subsequently when the narrow-edge count surpasses this threshold the corresponding group of objects is detected. For a stationary scene, e.g. for still image taking, an end-to-end focus position scan can return a list of maximum counts, one maximum count for each peaking of the narrow-edge count. A list of thresholds may be generated from the list of maximum counts, for example by taking 50% of the maximum counts.
Referring to
The processor 112′ may internally generate a focus signal and/or a narrow-edge count in addition to the functions included in the processor 112 of
The pixel array 108, A/D Converter 110, color interpolator 148, and generator 120′ may reside within a package 142, together comprising an image sensor 150′, separate from the processor 112′.
The main pixel array 2808 may be covered by a color filter array of a color mosaic pattern, e.g. the Bayer pattern. The optical lowpass filter 2808 prevents the smallest light spot focused on the pixel array 2808 from being too small as to cause aliasing. Where a color filter of a mosaic pattern covers the pixel array 2808, aliasing can give rise to color moiré artifacts after a color interpolation. For example, the smallest diameter of a circle encircling 84% of the visible light power of a light spot on the main pixel array 2808 (“smallest main diameter”) may be kept larger than one and a half pixel width but less than two pixel widths by use of the optical lowpass filter. For example, if the main pixel array 2808 has a pixel width of 4.5 um, whereas the smallest diameter is 2.0 um without optical lowpass filtering, the optical lowpass filter 2840 may be selected to make the light spot 6.7 um or larger in diameter.
The auxiliary pixel array 108″ may comprise one or more arrays of photodetectors. Each of the arrays may or may not be covered by a color filter array of a color mosaic pattern. The array(s) in auxiliary pixel array 108″ outputs image(s) in analog signals that are converted to digital signals 130 by A/D Converter 110. The images are sent to the focus signal generator 120. A color interpolator 148 may generate the missing colors for images generated from pixels covered by color filters. If auxiliary pixel array 108″ comprises multiple arrays of photodetectors, each array may capture a sub-image that corresponds to a portion of the image captured by the main pixel array 2808. The multiple arrays may be physically apart by more than a hundred pixel widths, and may or may not share a semiconductor substrate. Where the pixel arrays within auxiliary pixel array 108″ do not share a semiconductor substrate, they may be housed together in a package (not shown).
Main A/D Converter 2810 converts analog signals from the Main Pixel Array 2808 into digital main image data signal 2830, which is sent to the processor 112, where the image captured on the Main Pixel Array 2808 may receive image processing such as color interpolation, color correction, and image compression/decompression and finally be stored in memory card 116.
An array of photodetectors in the auxiliary pixel array 108″ may have a pixel width (“auxiliary pixel width”) that is smaller than a pixel width of the main pixel array 2808 (“main pixel width”). The auxiliary pixel width may be as small as half of the main pixel width. If an auxiliary pixel is covered by a color filter and the auxiliary pixel width is less than 1.3 times the smallest spot of visible light without optical lowpass filtering, a second optical lowpass filter may be inserted in front of the auxiliary array 108″ to increase the smallest diameter on the auxiliary pixel array 108″ (“smallest auxiliary diameter”) to between 1.3 to 2 times as large but still smaller than the smallest main diameter, preferably 1.5. The slight moiré in the auxiliary image is not an issue as the auxiliary image is not presented to the user as the final captured image.
The shaded region in
The auxiliary pixel array 108″, A/D Converter 110, focus signal generator 120 together may be housed in a package 142 and constitute an auxiliary sensor 150. The auxiliary sensor 150 may further comprise a color interpolator 148.
The auto-focus image pickup system 102, 102′, 103, 103′, 103″ may include a computer program storage medium (not shown) that comprises instructions that causes the processor 112, 112′, 112″ respectively, and/or the focus signal generator 120, 120′ to perform one or more of the functions described herein. By way of example, the instructions may cause the processor 112 or the generator 120′ to perform a slant correction for an edge width in accordance with the flowchart of
While a memory card 116 is shown as part of system 102, any nonvolatile storage medium may be used instead, e.g. hard disk drive, wherein images stored therein are accessible by a user and may be copied to a different location outside and away from the system 102.
One or more parameters for use in the system, for instance the sharp_edge_width, may be stored in a non-volatile memory in a device within the system. The device may be a flash memory device, the processor, or the image sensor, or the focus signal generator as a separate device from those. One or more formulae for use in the system, for example for calculating the concatenated length threshold, or for calculating beta may likewise be stored as parameters or as computer-executable instructions in a non-volatile memory in one or more of those devices.
While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.
This application is a continuation-in-part of U.S. patent application Ser. No. 13/430,684 filed on Mar. 26, 2012, which is a non-provisional of U.S. Provisional Patent Application No. 61/467,379 filed on Mar. 25, 2011 and a continuation-in-part of PCT Application No. PCT/IB2011/052524 filed on Jun. 9, 2011.
Number | Date | Country | |
---|---|---|---|
61467379 | Mar 2011 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 13430684 | Mar 2012 | US |
Child | 14040619 | US | |
Parent | PCT/IB2011/052524 | Jun 2011 | US |
Child | 13430684 | US |