Patent Application
20090110274
The present disclosure relates generally to data processing, and more specifically to techniques for enhancing the quality of captured images.
Digital and regular photography are widely used to capture images of scenes. In many instances, the captured images may have poor quality due to imbalance lighting. For example, a scene may be illuminated uniformly but may not reflect back a uniformly illuminated image to a camera. The imbalance lighting may result in some regions of the image being very dark and other regions being very bright. In general, imbalance lighting may result in images having poor contrast, loss of details, and/or other traits of poor quality. It is desirable to improve the quality of such images.
Techniques for improving the quality of images are described herein. In an aspect, a first/original histogram of intensity values may be obtained for an input image and may be diffused to obtain a second/diffused histogram with better intensity coverage. The first histogram may correspond to a probability density function (PDF) of intensity values for all pixels in the input image.
In one design, diffusion may be achieved by filtering the first histogram for multiple iterations with a diffusion function, which may be obtained based on a filter function and a diffusion control function. The filter function may control the rate and/or characteristics of the diffusion. The diffusion control function may control shifts in positions of lobes in the first histogram. A transformation function may be determined based on the first and second histograms. For example, a first cumulative distribution function (CDF) may be obtained based on the first histogram, and a second CDF may be obtained based on the second histogram. The transformation function may then be determined based on the first CDF, an inverse function of the second CDF, and possibly other CDFs. An output image may be generated by mapping each pixel value in the input image to a corresponding pixel value in the output image based on the transformation function.
Various aspects and features of the disclosure are described in further detail below.
Each pixel may be associated with luminance and chrominance information or color information. The luminance information conveys the intensity or grey scale of the pixel and is denoted as Y for luminance component. The chrominance information conveys color information and may be provided as Cr for red chrominance component and Cb for blue chrominance component. The color information may be provided by red (R), green (G), and blue (B) components. Y. Cr and Cb may be converted to R, G and B, and vice versa, via known matrix transformations.
The image quality enhancement techniques described herein may be applied to any component used to convey an image. For example, the techniques may be applied to only luminance component, to each of the luminance and chrominance components, to each of the color components, etc. For clarity, the techniques are described below for luminance component, which is also commonly referred to as intensity, grey scale, etc.
In image 100, the luminance or intensity of a pixel at coordinate x, y may be denoted as Pin(x,y) and may be represented with B bits, where B may be any integer value. Pin(x,y) may then be within an intensity range of 0 through N, or 0≦Pin(x, y)≦N, where N=2B−1. An intensity range is a range of all possible values for intensity. For example, if B=8, then Pin(x,y) is within an intensity range of 0 through 255.
A CDF Hin(n) may be obtained based on the PDF hin(n). For each value of n, Hin(n) is equal to the summation of hin(n) from 0 to n. Hin(n) is indicative of the area under the curve of hin(n) from 0 to n and is equal to N for the largest value of n, or Hin(N)=N, with the normalization factor Q described above.
In general, the pixels within an image may have any intensity values, which may be dependent on the captured scene, lighting conditions, etc. Imbalance lighting may cause the image to have poor quality, which may be quantified by various traits. As one example, the image may have some regions that are very dark and some regions that are very bright, which may result in loss of details. As another example, many pixels in the image may have intensity values that fall within a small portion of the intensity range, which may then cause the image to have less contrast. Poor image quality may also be given by other traits.
The quality of an image may be enhanced by utilizing more of the intensity range. For example, if the image does not significantly occupy a middle portion of the intensity range, then this portion may be used to show more details that would otherwise be buried in too dark or too bright regions. As another example, if the image occupies mostly the middle portion of the intensity range and has poor contrast, then the lower and higher portions of the intensity range may be used to improve contrast.
The techniques described herein may improve image quality by re-mapping the luminance component of an input image using a function that results in an output image having better coverage across the intensity range. In one design, which is referred to as histogram diffusion, an original histogram of intensity values is obtained for the input image and is diffused to generate a diffused histogram with better intensity coverage. The term “diffuse” refers to flatting, spreading, or stretching of a shape to make the shape more flat. The diffusion may retain the general shape of the original histogram while extending the breadth of intensity values used for the output image. In other designs, the original histogram may be modified or mapped to another histogram based on some mapping function that can retain the general shape of the original histogram while extending the intensity coverage. In any case, the output image may be generated based on the new/diffused histogram. For clarity, image quality enhancement using histogram diffusion is described below.
A new histogram 312 corresponding to a PDF of h′out(n) may be obtained by diffusing original histogram 310. Histogram 312 also has a single lobe that spans from intensity values of aout to cout and is centered at an intensity value of bout. The lobe in histogram 312 may be stretched so that (cout−aout)>(cin−ain), where aout may be less than ain and/or cout may be greater than Cin. The center of the lobe in histogram 312 may be maintained at the same position so that bout=bin or may be shifted lower or higher relative to the center of the lobe in histogram 310. In any case, stretching the lobe in histogram 312 may improve contrast in an output image generated based on histogram 312.
A new histogram 322 corresponding to a PDF of hout″(n) may be obtained by diffusing original histogram 320. Histogram 322 also has two lobes. The lower lobe spans from intensity values of aout1 to cout1 and is centered at an intensity value of bout1. The higher lobe spans from intensity values of aout2 to cout2 and is centered at an intensity value of bout2. The lower lobe in histogram 322 may be stretched so that (cout1−aout1)>(cin1−ain1), where aout1 may be less than ain1 and/or cout1 may be greater than cin1. The center of the lower lobe in histogram 322 may be maintained at the same position so that bout1=bin1 or may be shifted lower or higher relative to the center of the lower lobe in histogram 320. The higher lobe in histogram 322 may also be stretched so that (cout2−aout2)>(cin2−ain2), where aout2 may be less than ain2 and/or cout2 may be greater than cin2. The center of the lower lobe in histogram 322 may be maintained at the same position so that bout2=bin2 or may be shifted lower or higher relative to the center of the higher lobe in histogram 320. In any case, stretching the two lobes in histogram 322 may improve details in an output image generated based on histogram 322.
Histogram diffusion may be performed in various manners. In one design, histogram diffusion is achieved by filtering an original histogram in an iterative manner based on a filter function f(l) and a diffusion control function d(n). The filter function may determine the rate and/or other characteristics of diffusion and may be defined with 2L+1 taps, where L may be any integer value. The diffusion control function may be defined to achieve desired shifts in the positions of lobes, as described below. A diffusion function g(n,l) may be defined based on the filter function f(l) and the diffusion control function d(n), as follows:
g(n+l)=f(l)·d(n), for l=−1, . . . , −L,
g(n)=f(0)·d(n), for l=0, and
g(n+l)=f(l)/d(n), for l=+1, . . . , +L. Eq (1)
As shown in equation set (1), diffusion coefficients g(n+l), for n=0, . . . , N and l=0, ±1, . . . , ±L, are functions of intensity value/histogram index n as well as tap index l. In particular, g(n+l) is dependent on filter coefficient f(l) at tap index l and diffusion control value d(n) for intensity value n. The multiplication by d(n) for negative l and division by d(n) for positive l are used to control shifts in positions of lobes.
The diffusion of the original histogram may be iteratively performed as follows:
where hk(n) is a diffused PDF after k iterations.
In general, the filter function f(l) may have any response and include any number of taps. A lowpass filter response may be used to diffuse or flatten lobes in the original histogram. The filter response may be selected to achieve any desired amount of flattening, e.g., to flatten narrow lobes and retain wide lobes. An odd number of taps (e.g., 3, 5, etc.) may simplify computation. The filter function may also have a programmable delay, which may be used to control shifts in positions of lobes. In one example design, the filter function f(l) is a 3-tap averaging function that may be given as follows:
f(0)=f(−1)=f(+1)=1. Eq (3)
For the 3-tap filter function shown in equation (3), the diffusion of the original histogram may be iteratively performed as follows:
In equations (2) and (4), the PDF h0(n) used for the first iteration may be set equal to the PDF hin(n) of the input image, or h0(n)=hin(n). For each iteration k, a new histogram value hk(n) may be computed for each integer value of n based on the histogram values from the prior iteration k−1. For the smallest intensity value of n=0, hk-1(n−1) may be set to either zero or hk-1(n). For the largest intensity value of n=N, hk-1(n+1) may be set to either zero or hk-1(n). To reduce memory usage, the computation for each iteration may be performed “in-place”. In one design, h n) is written over hk-1(n) after hk-1(n) has been retrieved and used to compute hk(n+1). In another design, the computation proceeds from n=0 to 255, and hk(n) is written over hk-1(n−1) in memory. The computation may alternate, e.g., from n=0 to 255 for odd k and from n=255 down to 0 for even k, so that hk(n) may be stored in alternating memory locations in odd and even iterations. In any case, the computation for equation (2) or (4) may be performed for K iterations, or k=1, . . . , K, where K may be any suitably selected value. The PDF hK(n) generated by the last iteration may be provided as a final diffused PDF hdif(n), or hdif(n)=hK(n).
In equation (4), the diffusion for a given intensity value n is achieved by filtering histogram value hk-1(n) for that intensity value n as well as histogram values hk-1(n−1) and hk-1(n+1) for two nearest intensity values n−1 and n+1. The filtering has an effect of flattening the shape of the histogram. In the limit, as k approaches a large value or infinity, the diffused PDF may approach a totally flat histogram, which is a histogram with a constant value for all values of n, or hflat(n)=1 for n=0, . . . , N.
The diffusion control function d(n) controls the shifting of the shape of the histogram. In equation (4), if d(n) is larger than one, then hk-1(n−1) is given more weight and hk-1(n+1) is given less weight than hk-1(n), which results in a shift toward larger n. Conversely, if d(n) is smaller than one, then hk-1(n−1) is given less weight and hk-1(n+1) is given more weight than hk-1(n), which results in a shift toward smaller n. If d(n) is equal to one, then hk-1(n−1), hk-1(n) and hk-1(n+1) are given equal weight, which results in no shift. The direction of the shift as well as the rate of the shift of the shape of the histogram may thus be controlled by the diffusion control function. To avoid a division by d(n) in equation (4), the reciprocal of d(n), or 1/d(n), may be pre-computed for each value of n and stored. Thereafter, hk(n) may be computed by multiplying hk-1(n−1) with d(n) and multiplying hk-1(n+1) with 1/d(n).
After all K iterations are completed, PDF hK(n) from the last iteration is provided as the diffused PDF hdif(n). A CDF Hdif(n) may then be obtained for the diffused PDF hdif(n), as follows:
A transformation function F1(n) may be defined as follows:
F
1(n)=Hif−1(Hin(n)), Eq (6)
where Hin(n) is the CDF of the input image, and
Hdif−1(n) is an inverse function of Hdif(n).
The inverse function may be obtained in various manners. In one design, Hdif(n) is approximated with a polynomial, which may then be inverted in a manner known in the art. In another design, a look-up table uses Hdif(n) as an index and stores the intensity value n corresponding to Hdif(n). In this design, interpolation may be performed to obtain all possible integer values of Hdif(n) from 0 to N. A rounded value of n may then be stored for each possible integer value of Hdif(n). The inverse function may also be obtained in other manners.
Transformation function F1(n) may be determined for each value of n as follow. First, Hin(n) is obtained for n. The inverse function of Hdif(n) for this value of Hin(n) is then provided as F1(n). Transformation function F1(n) may be used to map the intensity value of each pixel in the input image to another intensity value for that pixel in an output image. The mapping with F1(n) would result in the mapped intensity values in the output image achieving a PDF of hdif(n).
A transformation function F2(n) may be defined as follows:
where Hflat(n)=n is a CDF for the totally flat PDF hflat(n)=1.
Transformation function F2(n) is obtained based on Hin(n) and is further capped to limit the amount of deviation from the totally flat PDF. If F2 (n)=Hin(n), then mapping with F2(n) would result in the mapped intensity values achieving a PDF of hin(n). If F2(n)=Hflat(n), then mapping with F2(n) would result in the mapped intensity values achieving the totally flat PDF. For F2(n) defined as shown in equation (7), mapping with F2(n) would result in the mapped intensity values achieving a PDF that is between hin(n) and hflat(n). In equation (7), F2(n) is equal to Hin(n) when Hin(n)=n, is pulled up toward flat distribution when Hin(n)<n, and is pulled down toward flat distribution when Hin(n)>n.
A transformation function F3(n) may be defined as follows:
F
3(n)=(1−α)·F1(n)+α·F2(n), Eq (8)
where α is a weighting factor.
Transformation function F3(n) is a weighted average of transformation functions F1(n) and F2(n). Factor α determines the weights given to F1(n) and F2(n). If α=0, then F3(n) is equal to F1(n), only the diffused CDF Hdif(n) is used, and mapping with F3(n) would result in the mapped intensity values achieving a PDF of hdif(n). If α=1, then F3(n) is equal to F2(n), only the original CDF Hin(n) with capping is used, and mapping with F3(n) would result in the mapped intensity values achieving a PDF between hin(n) and hflat(n). An α value closer to 0 may produce more details in the output image whereas an α value closer to 1 may produce more contrast in the output image. α may be set to 0.5 to give equal weight to F1(n) and F2(n) or may be set to some other value.
A transformation function F4(n) may be defined as follows:
Equation (9) sets F4(n) equal to F3(n) but limits F4(n) to a range of 0.7n to 2n. This limits the amount of change to the histogram for the output image.
The intensity values of pixels in the original image may be mapped to new intensity values in the output image, as follows:
P
out(x,y)=F(Pin(x,y)), Eq (10)
where F(n) may be equal to F1(n), F2(n), F3(n), F4(n) or some other transformation function. The mapping for each pixel may be achieved by (a) determining the intensity value of that pixel in the input image, or np=Pin(x, y), (b) determining the transformation function value F(np) corresponding to this pixel intensity value np, and (c) providing the transformation function value as the intensity value for the corresponding pixel in the output image, or Pout(x,y)=F(np).
The diffusion control function d(n) may be defined in various manner. In one design, the diffusion control function is defined to push lobes at low intensity toward the middle of the intensity range and to push lobes at high intensity toward the middle. The rate of push for each end may be dependent on the percentage of pixels with intensity values in that end.
In one design, outages are computed for low and high ends, as follows:
where slow is the percentage of pixels with intensity values less than or equal to Nlow, and
shigh is the percentage of pixels with intensity values greater than or equal to Nhigh.
Nlow determines the range of intensity values within which a lobe will be pushed higher. Conversely, Nhigh determines the range of intensity values within which a lobe will be pushed lower. Nlow and Nhigh may be any suitably selected values. For example, Nlow may be equal to 39 and Nhigh may be equal to 216, which would then result in a lobe within the 40 lowest intensity values being pushed higher and a lobe within the 40 highest intensity values being pushed lower. Other values may also be used for Nlow and Nhigh.
A function d1(n) may be defined as follows:
Equation (12) results in d1(n) having a value of one for n=0 then linearly decreasing to zero at n=Nlow, then remaining at zero until n=Nhigh, and then linearly increasing to one at n=N.
A function d2(n) may be defined as follows:
d2(n)=d1(n). Eq (13)
If slow<S then d2 (n)=d (n)·(slow/S)2, for n=0, . . . , Nlow.
If Shigh<S then d2(n)=d1(n)·(Shigh/S)2, for n=Nhigh, . . . , 1 N.
In equation set (13), d2(n) is initially set to d1(n), is then boosted by (slow/S)2 in the low end if slow is less than a threshold S, and is boosted by (shigh/S)2 in the high end if shigh is less than S. S determines whether or not to boost as well as the amount of boost. S may be set to a suitable value, e.g., S=0.2·N or some other value.
The diffusion control function d(n) may then be defined as follows:
where Nhaft=(N+1)/2 is a middle intensity value in the intensity range from 0 to N.
In equation set (14), a value of one is initially added to d2(n) to obtain a nominal value of one for d(n). The values of d2(n)+1 in the upper half of the intensity range are then inverted. d(n) is greater than or equal to one for n in the lower half of the intensity range. This results in hk-1(n−1) being multiplied with a value of one or greater, hk-1(n+1) being multiplied with a value of one or smaller, and a lobe within the range of 0 to Nlow being shifted higher. Conversely, d(n) is less than or equal to one for n in the upper half of the intensity range. This results in hk-1(n−1) being multiplied with a value of one or smaller, hk-1(n+1) being multiplied with a value of one or greater, and a lobe within the range of Nhigh to N being shifted lower.
For clarity, histogram diffusion is described below for an example input image. In this example, diffusion is performed based on the histogram diffusion function shown in equation (4) and with the diffusion control function obtained from equations (11) to (14).
For clarity, a specific histogram diffusion function hk(n) operating on three histogram values of hk-1(n−1), hk-1(n), and hk-1(n+1) with three coefficients of d(n), 1, and 1/d(n) has been described in equation (4). Histogram diffusion may also be performed based on other functions. For example, a histogram diffusion function may operate on more than three histogram values, e.g., two, four, five, or some other number of histogram values. A histogram diffusion function may also use other filter functions instead of the filter function shown in equation (3). In general, a histogram diffusion function may operate on any number of histogram values with any set of coefficients. Different diffusion rate and/or characteristics may be obtained with different numbers of histogram values and/or different sets of coefficients.
For clarity, a specific diffusion control function d(n) has been described above in equations (11) through (14). Other diffusion control functions may also be used to achieve different shift characteristics.
For clarity, specific transformation functions F1(n) through F4(n) have been described above. Transformation function F1(n) in equation (6) is based solely on the diffused histogram hdif(n), transformation function F2(n) in equation (7) is based on the input histogram hin(n) and the totally flat histogram hflat(n), and transformation function F3(n) in equation (8) is based on the diffused histogram, the input histogram, and the totally flat histogram. Other transformation functions may also be defined based on the diffused histogram, the input histogram, the totally flat histogram, and/or some other histogram. The various histograms may also be combined in other manners to obtain other transformation functions.
In one design, histogram diffusion may be applied to each input image to be processed. In another design, histogram diffusion may be selectively applied to a given input image, e.g., depending on its histogram. A preliminary test may be performed to determine whether the input image is a candidate for histogram diffusion. For example, slow and/or shigh may be computed for the input image and compared against a predetermined threshold. Histogram diffusion may be performed for the input image if slow and/or shigh exceeds the threshold.
A transformation function may be determined based on the first and second histograms (block 520). For example, the transformation function may be (a) determined based on the first CDF and an inverse function of the second CDF, e.g., as shown in equation (6), (b) determined based further on a CDF for a totally flat histogram, e.g., as shown in equations (7) and (8), and (c) limited to within a predetermined range, e.g., as shown in equation (9). The transformation function may also be determined in other manners. An output image may be generated based on the input image and the second histogram, e.g., by mapping each pixel value in the input image to a corresponding pixel value in the output image based on the transformation function (block 522).
The image quality enhancement techniques described herein may have certain desirable characteristics such as:
The techniques described herein may be used to improve intensity coverage and may provide the following advantages:
The image quality enhancement techniques described herein may be used for communication, computing, networking, personal electronics, etc. For example, the techniques may be used for wireless communication devices, handheld devices, gaming devices, computing devices, consumer electronics devices, computers, etc. An example use of the techniques for a wireless communication device is described below.
Wireless device 600 is capable of providing bi-directional communication via a receive path and a transmit path. On the receive path, signals transmitted by base stations are received by an antenna 612 and provided to a receiver (RCVR) 614. Receiver 614 conditions and digitizes the received signal and provides samples to a digital section 620 for further processing. On the transmit path, a transmitter (TMTR) 616 receives data to be transmitted from digital section 620, processes and conditions the data, and generates a modulated signal, which is transmitted via antenna 612 to the base stations.
Digital section 620 includes various processing, interface and memory units such as, for example, a modem processor 622, a reduced instruction set computer/digital signal processor (RISC/DSP) 624, a controller/processor 626, an internal memory 628, a graphics processing unit (GPU) 632, a video/camera processor 634, a display processor 636, and an external bus interface (EBI) 638. Modem processor 622 may perform processing for data transmission and reception, e.g., encoding, modulation, demodulation, and decoding. RISC/DSP 624 may perform general and specialized processing for wireless device 600. Controller/processor 626 may direct the operation of various processing and interface units within digital section 620. Internal memory 628 may store data and/or instructions for various units within digital section 620.
GPU 632 may perform graphics processing for 2-dimensional and/or 3-dimensional graphics. Video/camera processor 634 may receive data for images from a camera 644 and may perform processing on data for still images, moving videos, moving texts, etc., for camera and video applications such as camcorder, video playback, video conferencing, etc. Display processor 636 may perform processing to facilitate the display of videos, graphics, images, and texts on a display unit 646. EBI 638 may facilitate transfer of data between digital section 620 and a main memory 648. The image quality enhancement techniques described herein may be implemented by GPU 632 on images for graphics, video/camera processor 634 on input images, display processor 636 for output images, etc.
Digital section 620 may be implemented with one or more processors, DSPs, micro-processors, RISCs, etc. Digital section 620 may also be fabricated on one or more application specific integrated circuits (ASICs) and/or some other type of integrated circuits (ICs).
The image quality enhancement techniques described herein may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. For a hardware implementation, the processing units used to perform the techniques may be implemented within one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, a computer, or a combination thereof.
For a firmware and/or software implementation, the techniques may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The firmware and/or software codes may be stored in a memory (e.g., memory 628 or 648 in
An apparatus implementing the techniques described herein may be a stand-alone unit or may be part of a device. The device may be (i) a stand-alone integrated circuit (IC), (ii) a set of one or more ICs that may include memory ICs for storing data and/or instructions, (iii) an ASIC such as a mobile station modem (MSM), (iv) a module that may be embedded within other devices, (v) a cellular phone, wireless device, handset, or mobile unit, (vi) etc.
The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
