Embodiments of the invention relate to recovering a haze-free image given a single photograph as an input.
In almost every practical scenario in which an image is being captured by a camera, the light reflected from a surface is scattered in the atmosphere before it reaches the camera. This may be due to the presence of aerosols, such as dust, mist, and/or fumes, which deflect light from its original course of propagation. In long distance photography or foggy scenes, this process can have a substantial effect on the image, in which contrasts may be reduced and surface colors may become faint. Such degraded photographs often lack visual vividness and appeal, and moreover, they may offer a poor visibility of the scene contents. This effect may be an annoyance to amateur, commercial, and/or artistic photographers, as well as undermine the quality of underwater and/or aerial photography. This may also be the case for satellite imaging, which is used for many purposes including, for example, cartography and web mapping, land-use planning, archeology, and/or environmental studies.
As discussed in more detail below, in this process light, which should have propagated in straight lines, is scattered and replaced by previously scattered light, called the “airlight” (See Koschmieder, H., “Theorie der horizontalen sichtweite,” in Beitr. zur Phys. d. freien Atm., 171-181, 1924, hereinafter “Koschmieder 1924,” the contents of which are hereby incorporated herein by reference in their entirety). This can result in a multiplicative loss of image contrasts, as well as an additive term due to this uniform light. As described in more detail below, a model that is commonly used to formalize the image formation in the presence of haze factors the degraded image into a sum of two components: the airlight contribution and the unknown surface radiance. Algebraically these two, three-channel color vectors are convexly combined by the transmission coefficient, which is a scalar specifying the visibility at each pixel. Recovering a haze-free image using this model requires the determination of the three surface color values, as well as the transmission value at every pixel. Since the input image provides three equations per pixel, the system is ambiguous and the transmission values cannot be determined. A formal description of this ambiguity is described below. However, intuitively it follows from the inability to answer the following question based on a single image: are we looking at a deep red surface through a thick white medium, or is it a faint red surface seen at a close range or through a clear medium. In the general case, this ambiguity, which is referred to as the “airlight-albedo ambiguity,” holds for every pixel and cannot be resolved independently at each pixel given a single input image.
A need, therefore, exists, for a technique for resolving this ambiguity and, as a result, de-hazing a captured image, using only the information obtainable from the captured image itself.
Embodiments of the present invention provide an improvement by providing, among other things, a new method for estimating the optical transmission in hazy scenes given a single input image. Based on this estimation, the scattered light can be eliminated to increase scene visibility and recover haze-free scene contrasts. In the approach of one embodiment, a refined image formation model may be formulated that can account for surface shading in addition to the transmission function. This allows ambiguities in the data to be resolved by searching for a solution in which the resulting shading and transmission functions are locally statistically uncorrelated. A similar principle can be used to estimate the color of the haze. Results, which are discussed below, demonstrate the ability of embodiments described herein to remove the haze layer, as well as provide a reliable transmission estimate that can be used for additional applications, such as image refocusing and/or novel view synthesis.
According to one aspect, a method of recovering a haze-free image given a single input image is provided. In one embodiment, the method may include: (1) receiving, from an image capture device, a single input image I comprising a plurality of pixels; (2) modeling a surface radiance vector J of the input image I as a product of a surface albedo coefficient R and a shading factor l; (3) determining, by a processor for each of the plurality of pixels, a value of a transmission t of the pixel, such that a covariance CΩ between the transmission t and the shading factor l is minimized; and (4) recovering, by the processor, the surface radiance vector J based at least in part on the determined value of the transmission t for each pixel.
According to another aspect, an apparatus for recovering a haze-free image given a single input image is provided. In one embodiment, the apparatus may include a processor configured to: (1) receive, from an image capture device, a single input image I comprising a plurality of pixels; (2) model a surface radiance vector J of the input image I as a product of a surface albedo coefficient R and a shading factor l; (3) determine, for each of the plurality of pixels, a value of a transmission t of the pixel, such that a covariance CΩ between the transmission t and the shading factor l is minimized; and (4) recover the surface radiance vector J based at least in part on the determined value of the transmission t for each pixel.
According to yet another aspect, a computer program product for recovering a haze-free image given a single input image is provided. The computer program product contains at least one computer-readable storage medium having computer-readable program code portions stored therein. The computer-readable program code portions of one embodiment may include: (1) a first executable portion for receiving, from an image capture device, a single input image I comprising a plurality of pixels; (2) a second executable portion for modeling a surface radiance vector J of the input image I as a product of a surface albedo coefficient R and a shading factor l; (3) a third executable portion for determining, for each of the plurality of pixels, a value of a transmission t of the pixel, such that a covariance CΩ between the transmission t and the shading factor l is minimized; and (4) a fourth executable portion for recovering the surface radiance vector J based at least in part on the determined value of the transmission t for each pixel.
Having thus described embodiments of the invention in general terms, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein:
Embodiments of the present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the inventions are shown. Indeed, embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like numbers refer to like elements throughout.
Overview:
Embodiments of the present invention provide a method for recovering a haze-free image given a single photograph as an input. In particular, according to one embodiment, the input image may be interpreted through a model that accounts for surface shading in addition to the scene transmission. Based on this refined image-formation model, the image can be broken into regions of a constant albedo, and the airlight-albedo ambiguity can be resolved by deriving an additional constraint, wherein the surface shading and medium transmission functions may be locally statistically uncorrelated. In one embodiment, the shading component may vary significantly compared to the noise present in the image. A graphical model may further be used to propagate the solution to pixels in which the signal-to-noise ratio falls below an admissible level that can be derived analytically (discussed below). The airlight color can also be estimated using this uncorrelation principle. The method of one embodiment may be passive. In other words, according to one embodiment, the method may not require multiple images of the scene, any light-blocking based polarization, any form of scene depth information, or any specialized sensors or hardware. The method of one embodiment may have the minimal requirement of a single image acquired by an ordinary consumer camera, or similar image capture device. In addition, according to one embodiment, the method of embodiments described herein may not assume the haze layer to be smooth in space, i.e., discontinuities in the scene depth or medium thickness are permitted. As shown in
Image Degradation Model:
Light passing through a scattering medium is attenuated along its original course and is distributed to other directions. This process is commonly modeled mathematically by assuming that along short distances there is a linear relation between the fraction of light deflected and the distance traveled. More formally, along infinitesimally short distances dr the fraction of light absorbed is given by βdr where β is the medium extinction coefficient due to light scattering. Integrating this process along a ray emerging from the viewer, in the case of a spatially varying β, gives
where r is an arc-length parametrization of the ray. The fraction t is called the transmission and expresses the relative portion of light that managed to survive the entire path between the observer and a surface point in the scene, at r(d), without being scattered. In the absence of black-body radiation the process of light scattering conserves energy, meaning that the fraction of light scattered from any particular direction is replaced by the same fraction of light scattered from all other directions. The equation that expresses this conservation law is known as the Radiative Transport Equation (See Rossum M. V., et al., “Multiple scattering of classical waves: microscopy, mesoscopy and diffusion,” vol. 71, 313-371, 1999, the contents of which are hereby incorporated herein by reference in their entirety). Assuming that this added light is dominated by light that underwent multiple scattering events, allows for the approximation of the added light as being both isotropic and uniform in space. This constant light, known as the “airlight” (Koschmieder 1924) or also as the “veiling light,” can be used to approximate the true in-scattering term in the full radiative transport equation to achieve the following simpler image formation model
I(x)=t(x)J(x)+(1−t(x))A, (2)
where this equation is defined on the three RGB color channels. I stands for the observed image, A is the airlight color vector, J is the surface radiance vector at the intersection point of the scene and the real-world ray corresponding to the pixel x=(x, y), and t(x) is the transmission along that ray. This degradation model is commonly used to describe the image formation in the presence of haze. As with similar works using this degradation model, a goal of embodiments of the present invention is to recover J, which is an image showing the scene through a clear haze-free medium. By that, other effects the haze may have on the scene, such as a change in overall illumination which in turn affects the radiant emittance, are not eliminated. Also, embodiments of the present invention assume that the input image I is given in the true scene radiance values. These radiance maps can be recovered by extracting the camera raw data or inverting the overall acquisition response curve, as described in Debevec, P. E., et al., “Recovering high dynamic range radiance maps from photographs,” In ACM SIGGRAPH 1997, 369-378, the contents of which are hereby incorporated herein by reference in their entirety.
This model (2) explains the loss of contrasts due to haze as the result of averaging the image with a constant color A. If the contrasts in the image are measured as the magnitude of the image's gradient field, a scene J seen through a uniform medium with t(x)=t<1 yields
∥∇I∥=∥t∇J(x)+(1−t)∇A∥=t∥∇J(x)∥<∥∇J(x)∥ (3)
Constant Albedo Images:
The airlight-albedo ambiguity exists in each pixel independently and gives rise to a large number of undetermined degrees of freedom. To reduce the amount of this indeterminateness, embodiments of the present invention simplify the image locally by relating nearby pixels together. This can be done in two steps. In the first step, the unknown image J can be modeled as a pixelwise product of surface albedo coefficients and a shading factor, Rl, where R is a three-channel RGB vector of surface reflectance coefficients and l is a scalar describing the light reflected from the surface. This refined model can be used to simplify the image by assuming that R(x) is piecewise constant. In this section, one of these localized sets of pixels x∈Ω that share the same surface albedo, i.e., pixels in which R(x)=R for some constant vector R, is considered. At these pixels, the standard image formation model (2) becomes
I(x)=t(x)l(x)R+(1−t(x))A. (4)
Instead of having three unknowns per pixel in J(x), there are now only two unknowns l(x) and t(x) per pixel plus another three constants in R. Embodiments of the present invention may proceed by breaking R into a sum of two components, one parallel to the airlight A and a residual vector R′ that lies in the linear sub-space which is orthogonal to the airlight, R′∈A⊥. In terms of these normalized components, the equation above becomes
I(x)=t(x)l′(x)(R′/∥R′∥+ηA/∥A∥)+(1−t(x))A, (5)
where l′=∥R′∥l and η=R,A
/(∥R′∥∥A∥) measuring the component that is mutual to the surface albedo and the airlight. By
•,•
the standard three-dimensional dot-product in the RGB space can be denoted. In order to obtain independent equations, embodiments of the present invention can project the input image along the airlight vector, which results in a scalar given by
IA(x)=I(x),A
/∥A∥=t(x)l′(x)η+(1−t(x))∥A∥, (6)
and project along R′, which equals to the norm of the residual that lies within A⊥, i.e.,
IR′(x)=√{square root over (∥I(x)∥2−IA(x)2)}{square root over (∥I(x)∥2−IA(x)2)}=t(x)l′(x). (7)
The transmission t can then be written in terms of these two quantities as
t(x)=1−(IA(x)−ηIR′(x))/∥A∥. (8)
In this equation the airlight-albedo ambiguity is made explicit; assuming for the moment that there is some estimate for the airlight vector A, the image components IA(x) and IR′(x) give two constraints per pixel x∈Ω, while there are two unknowns per pixel l(x) and t(x) plus an additional unknown constant η. A third equation cannot be obtained from l since, according to the model of embodiments described herein, any vector in A⊥ yields the same equation as IR′. Thus, this model (5) reduces the pointwise ambiguity into a problem of determining a single scalar η for all x∈Ω. Finding this number, that expresses the amount of airlight in R′, enables the recovery of the transmission from (8) and ultimately the output image J according to (2).
A technique for resolving the airlight-albedo ambiguity in this reduced form according to one embodiment is now described. According to this embodiment, the transmission t depends on the scene depth and the density of the haze, β(x), while the shading l depends on the illumination in the scene, surface reflectance properties, and scene geometry. Therefore it is reasonable to expect that the object shading function l and the scene transmission t do not exhibit a simple local relation. This leads to the assumption that the two functions are statistically uncorrelated over Ω, meaning that CΩ(l, t)=0, where the sample covariance CΩ is estimated in the usual way
and the mean by
A simpler formulation can be obtained by expressing this lack of correlation between t and 1/l′ instead (if X and Y are two independent random variables then so are ƒ(X) and g(Y) for every pair of continuous functions ƒ and g), where the latter can be given by
If a new variable is defined as
h(x)=(∥A∥−IA)/IR′, (12)
substitute (8) and (11) into the covariance operator, and extract η, embodiments of the present invention finally get that
This function can be computed given the input image and, as explained earlier, η allows for the computation of the transmission from (8) and the recovery of the unknown image J according to (2). The procedure described here is, in essence, an independent component analysis (See Hyvrinen, A., et al., “Independent component analysis: Algorithms and applications,” Neural Networks 13, 411-430, 2000, the contents of which are hereby incorporated herein by reference in their entirety). Unlike certain models, equation (13) fully determines η and decorrelates l and t. This is true as long as the denominator CΩ(IR′,h) is non-zero, which can be ensured in a manner described below.
In
Noise Estimation:
Unfortunately, as one of ordinary skill in the art will recognize, no measurement comes free of noise, including the input image I. This error, or lack of data, in the input can play a role in this inverse problem of recovering a faint signal masked by haze. This noise introduces an uncertainty into the estimation of η by (13) and, therefore, into t as given by (8). In order to use the estimated transmission later, with the right amount of confidence, the amount of error contained in the estimated transmission may need to be assessed. In one embodiment, this can be done by modeling the error present in I as an additive white Gaussian noise. More specifically, it can be assumed that I=Ĩ+ξ, where ξ is a three dimensional vector of random Gaussian variables, which is independent both component-wise and pixel-wise, with zero mean and variance σ2. The propagation of ξ, starting from I through the calculation of η up to t, is described below, and an expression that is linear in ξ is obtained. This linearity may mean that, at the end, a Gaussian noise whose variance σt2 can be computed may be left. In this section, how this information is factored into the estimation of the final t, in accordance with one embodiment of the present invention, is described.
Along the derivation of the error estimate, given in detail below, several conditions which allow for the truncation of Taylor expansions are assumed to hold. Here these conditions are listed, along with a brief explanation as to why they are likely unavoidable and not merely artificial byproducts of the error formulation or analysis. The first condition may require the noise variance σ2 to be less than the three components of I/10 and also less than IR′/6, which is the haze-free component of the input. These are basic signal-to-noise requirements stating that data must be available and that pixels which are too opaque, compared to σ, contain little or no information about J. The second condition may require the variance of the noise term ξCR′ to be less than (CΩ(IR′,h)/5)2, where ξCR′ is the noise present in CΩ(IR′,h), which is defined below. This condition breaks down when |Ω| is too small and there is not enough averaging to reduce the variance in ξCR′ or when there is not enough variation in l, which appears both in IR′ and h. Intuitively, these conditions ensure the stability of the statistics EΩ and CΩ over Ω and the availability of enough multiplicative variation in IR′, which is needed to resolve airlight-albedo ambiguity. These conditions are used in the way predictions are made in embodiments described herein and are explicitly quantified by this noise model.
The above is not meant to justify or assume that any of these conditions actually hold. On the contrary, every pixel that does not meet these error estimation criteria may be omitted. And as the discussion above suggests, these pixels may be discarded for not containing reliable information on top of violating conditions needed to evaluate the error they contain. The set of these discarded pixels can be denoted by . As shown below, the estimate of t at the rest of the pixels x∉
may contain zero-mean Gaussian noise with a variance σt2, defined below. This error estimate can be used, as described in the next section, to extrapolate t for all x, needed for computing the output image J, when embodiments of the present invention are extended to handle images with multiple albedo values.
Given an input image, a σ that best approximates the variance of the noise present in the input image may need to be supplied. In one embodiment, this may be left for the user to set. In the experiments described herein a σ= 1/100- 1/200 was used (for images with pixel values ranging between zero and one), though other values of σ are contemplated within the scope of embodiments of the invention. Pixels with low signal-to-noise ratios were either discarded or received a high σt2, meaning that, according to embodiments described herein, this parameter need not be precise.
Multi-Abedo Images:
In the previous section the restricted case in which it was assumed that all the pixels correspond to the same surface, i.e., a constant albedo value R(x)=R, and hence result in a constant airlight-albedo mutual component η was considered. In order to handle general images, containing surfaces with different reflectance values, according to one embodiment described herein, these evaluations can be performed on a pixel basis using robust estimators defined on neighboring pixels. For that pixels can be related based on their similarity in albedo using the following shading and airlight invariant measure. A naive haze-free image can be computed by I′=I−AIA/∥A∥ (which may contain negative values) and projected onto the two chroma channels U and V of the YUV color space. Since embodiments described herein are not interested in differentiating pixels based on their shading, a single chroma channel can be extracted based on the ratio of these projections
θ(x)tan−1(I′,U
/
I′,V
). (14)
This maps pixel color into angles where distances can be measured by the length of the shortest arc, i.e.,
d(θ1,θ2)=min{|θ1−θ2|,2π−|θ1−θ2|}. (15)
This similarity measure can be used to define the weights of the following w-estimators (See Andrews et al., Robust Estimates of Location: Survey and Advances, Princeton University Press; London, Oxford University Press, 1972, the contents of which are hereby incorporated herein by reference in their entirety)
where the weighting function w(x,y) is given by exp(−d(θ(x),θ(y))2/σθ2) the normalizing weight Wx=Σy∈Ωx w(x,y), and Ωx is the window of pixels centered around x excluding pixels from . In the testings described herein σθ is set to ⅛ and windows of 24-by-24 pixels, which can be enlarged in case of noisy images, are used. EΩ and CΩ are replaced in (13) with these robust estimators and η(x) is computed at every pixel x∉
. The transmission can be computed at these pixels according to (8), and this primary estimate can be denoted by {circumflex over (t)}. As noted earlier, below the estimated noise variance σt2 present in these estimated transmission values is computed.
Embodiments of the present invention can proceed with a statistical smoothing to account for the noise present in {circumflex over (t)} and to fill in the transmission values at pixels x∈. The airlight contribution is fully contained in IA and can therefore be used as a prior to derive the spatial regularity of t. Embodiments of the present invention assume independence between these dependencies and formalize them using a Gauss-Markov random field model (See Pérez, P., “Markov random fields and images,” in CWI Quarterly, vol. 11, 413-437 1998, the contents of which are hereby incorporated herein by reference in their entirety), defined by
where Nx is the set of pixel x's four nearest-neighbors in the lattice and use σs=⅕. The links defined by this model are illustrated in
According to embodiments described herein, this model can be maximized by solving the linear system resulting from d log P/dt=0 and taking this optimum to be the final t which can be used to recover the final haze-free output image J from (2).
Estimating the Airlight Color:
According to one embodiment described herein, the principle of uncorrelation, described above, can be applied once again in order to estimate the airlight color vector A. One possibility may be to restrict to small windows of constant albedo and search for a vector A that yields the least correlated t and l. This is based on a need for consistency between the equations derived and the input pixels; the airlight vector used in equations (8), (11), and (12) should be identical to the airlight present in the input pixels so that the t and l resulting from the computations described herein will indeed be uncorrelated. If this is not the case, the η computed from (13) and meant to achieve zero correlation, is meaningless. Thus given an initial guess for A, C(J, t)2 can be minimized by updating A's components using the steepest descent method. This process can be done within small windows of 24-by-24 pixels and may take a few seconds to perform. In one embodiment, the most haze-opaque pixel can be used as an initial guess.
This three-dimensional minimization can be reduced to a one dimensional search if two or more regions that correspond to different but uniform albedos are selected. As indicated by (4), in such regions all the pixels lay within a two-dimensional linear sub-space spanned by the vectors R and A. A principal component analysis will not extract the directions of these vectors since their coefficients tl and −t are not independent. Nevertheless, embodiments of the present invention can use this analysis in order to find the sub-spaces themselves by omitting the least-active component of the three. This can be applied in each of the given regions, indexed by i, and the two spanning components can be denoted by vi1 and vi2. Since A is expected to be contained in each of these sub-spaces, its direction can be recovered by intersecting these planes. This can be done by searching for a vector with the highest projection onto all these sub-spaces, i.e.,
The solution to this problem, according to the Lagrange-multipliers rule, can be given by the eigenvector that corresponds to the highest eigenvalue of the 3-by-3 matrix given by Σivi1(vi1)T+vi2(vi2)T. The search described above, where C(J, t)2 is minimized, can now be used, in accordance with an embodiment described herein, to find the magnitude of A. This is a one-dimensional optimization problem which can be solved using a naive search and can take less than one second.
Results:
In
In
The predicted transmission function t(x) may be used for other applications in addition to haze removal. In particular, in
e−∫λβ(s)ds=e−λ∫β(s)ds=(e∫β(s)ds)λ=tλ. (19)
Derivation of the Noise Estimates:
The following provides an example of the derivation of an estimation for the noise present in the transmission t, given by (8). As described above, this analysis is based on modeling the noise present in the input data I as an additive white Gaussian noise with zero mean and variance σ2, given by
I=Ĩ+ξ, (20)
where Ĩ is the true image and the noise ξ is a three-dimensional vector (corresponding to the RGB color channels) with each component at each pixel being an independent Gaussian variable ξ˜N(0,σ2). This breaks IA into signal and noise terms,
IA=Ĩ+ξ,A
/∥A∥=ĨA+ξA, (21)
with ĨA=Ĩ,A
/∥A∥ and ξA=
ξ,A
/∥A∥. Similarly, for IR′ one gets
and by assuming that the noise ξ is less than Ĩ the quadratic noise terms can be neglected. This leaves
IR′=√{square root over ({tilde over (I)}R′2+2ξ,Ĩ−ĨAA
,)} (23)
and since ĨR′2=∥Ĩ−ĨAA∥2ξ may need to be less than ĨR′ (and not than ĨR′2) in order to use a first-order Taylor approximation for √{square root over (x)} centered around ĨR′2. This gives
IR′≈ĨR′+(Ĩ,ξ
−ĨAξA)/ĨR′=ĨR′+ξR′, (24)
where ξR′ can be written more explicitly as
ξR′=ξ,Ĩ−AĨA/∥A∥
/ĨR′. (25)
Based on the same assumption, ξ<ĨR′, a first-order Taylor approximation can be used for 1/x around ĨR′ in order to remove the noise term from the denominator of h
where {tilde over (h)}=(∥A∥−ĨA)/ĨR′, and
ξh=−(ξA+ξR′{tilde over (h)})/ĨR′=ξ,A({tilde over (h)}ĨA/ĨR′−1)/∥A∥−Ĩ{tilde over (h)}/ĨR′
/ĨR′. (27)
The covariance terms gives
where, by taking into account that ξR′{tilde over (h)}+ξhĨR′=−ξA, the noise term becomes
ξCR′=−EΩ(ξA)−EΩ(ξR′)EΩ({tilde over (h)})−EΩ(ξh)EΩ(ĨR′. (29)
Similarly, one gets
CΩ(IA,h)≈CΩ(ĨA,{tilde over (h)})+ξCA, (30)
where
ξCA=EΩ(ξA{tilde over (h)}+ξhĨA)−EΩ(ξA)EΩ({tilde over (h)})−EΩ(ξh)EΩ(ĨA). (31)
Finally, under the assumption that CΩ(ĨR′{tilde over (h)}) is larger than ξCR′, a first-order Taylor expansion of 1/x around CΩ(ĨR′{tilde over (h)}) can again be used to get
and {tilde over (η)} is the noiseless CΩ(ĨA, {tilde over (h)})/CΩ(ĨR′, {tilde over (h)}) that is being estimated. Note that EΩ and CΩ are the mean and variance of the pixels sample in Ω and have nothing to do with the moments of ξ. Finally, from (8) one gets
t={tilde over (t)}+(ξA−{tilde over (η)}ξR′−ĨR′ξη−ξηξR′)/∥A∥={tilde over (t)}+ξt, (34)
where {tilde over (t)}=1−(ĨA−{tilde over (η)}ĨR′)/∥A∥ and ξt=(ξA−{tilde over (η)}ξR′−ĨR′ξη)/∥A∥.
Here the quadratic noise term can be dropped, under the assumption that ξR′<ĨR′, i.e., ξ is small.
Along this derivation, several assumptions were made which are quantified now. In order for aξ+ξ2 to still behave like a Gaussian, i.e., have a distribution with a similar shape and a variance of ≈a2σ2,σ may need to be less than a/10. In the derivation (23), this condition may require that σ<Ĩ/10≈I/10 for each of the RGB components and at each pixel. This is also the case in (34) where σ may need to be less than 1/10. Similarly, in order to remove the noise from √{square root over (a+ξ)} and still maintain an approximately Gaussian distribution, σ should be less than a/3. In the derivation of embodiments described herein, this condition arises in (24) and may require a σ<ĨR′/6≈IR′/6. The ratio between a and ξ that allows for the movement of the noise from the denominator in 1/(a+ξ) and still ending up with approximately a Gaussian variable is a σ<a/5. This applies to the two cases that were encountered above, the first in (26), which gives a less restrictive condition a σ<(Ĩ−ĨAA)/5 than what there already is, and the second case in (32) that may require
Varξ(ξCR′)<(CΩ(ĨR′,{tilde over (h)})/5)2≈(CΩ(IR′,h)/5)2, (35)
where Varξ is the variance with respect to the noise randomness. These conditions were summarized above and an explanation was given as to how this uncertainly is accounted for in the estimation of t in accordance with an embodiment described herein.
The estimated transmission noise term ξt is now a linear combination of ξ. Therefore its variance can be computed according to the following arithmetic rules for Gaussian variables: (i) aξ(x)+bξ(x)˜N(0, (a+b)2σ2), and (ii) aξ(x)+bξ(y)˜N(0, (a2+b2)σ2) for any scalars a,b. The variance of ξA, ξR′, and ξh can be computed according to the first rule as no different pixels get mixed together. Mixing occurs in ξCR′ and ξCh where the second rule is applied. The first rule can also be used to calculate the variance of the intermediate noise terms ξA{tilde over (h)}+ξhĨA in (31) and for ξA−{tilde over (η)}ξR′, found in ξt.
Apparatus for De-Hazing
Referring to
In particular, as shown, in one embodiment, the apparatus 900 may, itself, include an image capture device 910 configured to capture a digital image, from which a haze-free image may be recovered in accordance with embodiments described herein. Alternatively, while not shown, according to another embodiment, the image capture device 910 may be external to and in electronic communication with (e.g., via a Universal Serial Bus (USB), or similar, port) the apparatus 900. An example of the former instance may include a PDA having both image capturing and data processing capabilities. An example of the latter instance may include where a digital camera is used to capture a digital image, which may be uploaded to a PC or laptop, wherein the PC or laptop performs the de-hazing process of embodiments described herein.
The apparatus 900 can further generally include means, such as a processor 920 for performing or controlling the various functions of the apparatus 900. In particular, the processor 920 may be configured to perform the processes discussed in more detail above and below with regard to
In one embodiment, the processor 920 is in communication with or includes memory 930, such as volatile and/or non-volatile memory that stores content, data or the like. For example, the memory 930 may store content transmitted from, and/or received by, the entity. Also for example, the memory 930 may store software applications, instructions or the like for the processor to perform steps associated with operation of the apparatus 900 in accordance with embodiments of the present invention. In particular, the memory 930 may store software applications, instructions or the like for the processor to perform the operations described above and below with regard to
For example, according to one embodiment, the memory 930 may store one or more modules for instructing the processor 920 to perform the operations including, for example, a receiving module, a modeling module, a determining module, and a recovering module. In one embodiment, the receiving module may be configured to receive, from the image capture device 910, a single input image I comprising a plurality of pixels, wherein the input image I further comprises a transmission t, a surface radiance vector J, and an airlight color vector A. The modeling module may be configured to instruct the processor 920 to model the surface radiance vector J as a product of a surface albedo coefficient R and a shading factor l. The determining module may be configured to instruct the processor 930 to determine, for each of the plurality of pixels, a value of the transmission t, such that a covariance CΩ between the transmission t and the shading factor l is minimized. Finally, the recovery module may be configured to instruct the processor 930 to recover the surface radiance vector J based at least in part on the determined value of the transmission t for each pixel.
In addition to the memory 930, the processor 920 can also be connected to at least one interface or other means for displaying, transmitting and/or receiving data, content or the like. In this regard, the interface(s) can include at least one communication interface or other means for transmitting and/or receiving data, content or the like (not shown), as well as at least one user interface that can include a display 940 and/or a user input interface. The user input interface (not shown), in turn, can comprise any of a number of devices allowing the entity to receive data from a user, such as a keypad, a touch display, a joystick or other input device.
Method of De-Hazing Using a Single Input Image:
Reference is now made to
In order to remove the haze of the received image I, the apparatus 900 (e.g., the processor 920 executing the modeling module) may, at Block 1002 model the surface radiance vector J as a product of a surface albedo coefficient R and a shading factor l (See equation (4)). The apparatus 900 (e.g., processor 920) may then, at Block 1003 break the surface albedo coefficient R into a sum of two components, a first component parallel to the airlight color vector A and a second component comprising a residual vector R′, wherein the residual vector R′ comprises an amount of airlight expressed by an unknown scalar value η(See equation (5)).
The apparatus 900 (e.g., processor 920) may then project the input image I onto the airlight color vector A to create an airlight projection IA, as well as along the residual vector R′ to create a residual projection IR′ (See equations (6) and (7), respectively). (Blocks 1004 and 1005). A value for η may then be selected for each pixel of the input image I (e.g., by the processor 920), such that a covariance CΩ between the transmission t and the shading factor l is minimized, wherein η is based at least in part on the airlight projection IA and the residual projection IR′ (See equations (12), (13) and (16)). (Block 1006).
A value of the transmission t for each pixel can then be determined (e.g., by the processor 920) based at least in part on the selected value for η (Block 1007), and the surface radiance vector J (i.e., a haze-free image) can be recovered based at least in part on the determined value of the transmission t for each pixel (Block 1008).
As described above, embodiments of the present invention provide a method for estimating the transmission in hazy scenes based on minimal input—a single image. This new approach is physically sound and relies only on the assumption that the transmission and surface shading are locally uncorrelated. The use of robust statistics can enable coping with complicated scenes containing different surface albedos, and the use of an implicit graphical model can make it possible to extrapolate the solution to pixels where no reliable estimate is available. Despite the challenges in resolving the ambiguities involved in this problem, the images produced by the method of embodiments described herein offer a lucid view of the scene and regain contrast that is typical to haze-free scenes.
The method of one embodiment solves a non-linear inverse problem. Accordingly, its performance depends, to some extent, on the quality of the input data. An estimation for the noise present in the input may be derived and statistical extrapolation may be used to cope with large errors. Nevertheless, insufficient signal-to-noise ratio or the absence of multiplicative variation in significant portions of the image may negatively impact the method of embodiments described herein. These scenarios are easy to find, however, and in
While haze removal from images provides a clearer view of the scene, it may not be regarded as a default preference. On the contrary, the presence of haze can be desired for many reasons. For example, it is known to be an important monocular depth cue (See Coon, D, Psychology: A Modular Approach To Mind And Behavior, Wadsworth Pub Co, July, 2005, the contents of which are hereby incorporated herein by reference in their entirety). In addition, painters use a technique called atmospheric perspective where they wash out regions of the painting using white paint to create an illusion of depth.
Recent theories of perceptual transparency (See Singh, M., et al., “Toward a perceptual theory of transparency,” No. 109, 492-519, 2002, the contents of which are hereby incorporated herein by reference in their entirety) suggest that human observers perceive the medium transmission based on the contrast ratio of a pattern seen through a semi-transparent filter and through a clear medium. Although it may not be assumed here that any haze-free data is available, variations in surface shading may also be relied on in order to disambiguate and require a significant C(IR′,h).
As described above and as will be appreciated by one skilled in the art, embodiments of the present invention may be configured as a method or electronic device/apparatus. Accordingly, embodiments of the present invention may be comprised of various means including entirely of hardware, entirely of software, or any combination of software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. Any suitable computer-readable storage medium may be utilized including hard disks, CD-ROMs, optical storage devices, or magnetic storage devices.
Embodiments of the present invention have been described above with reference to block diagrams and flowchart illustrations of methods, apparatuses (i.e., systems) and computer program products. It will be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by various means including computer program instructions. These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus, such as processor 920 discussed above with reference to
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus (e.g., processor 920 of
Accordingly, blocks of the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions. Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these embodiments of the invention pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the embodiments of the invention are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe exemplary embodiments in the context of certain exemplary combinations of elements and/or functions, it should be appreciated that different combinations of elements and/or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and/or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
This application claims priority to, and incorporates in its entirety by reference, U.S. Application Ser. No. 61/202,807 entitled “Single Image Dehazing,” filed on Apr. 8, 2009.
Number | Name | Date | Kind |
---|---|---|---|
7873235 | Steiger | Jan 2011 | B2 |
7941004 | Zhu et al. | May 2011 | B2 |
20100067823 | Kopf et al. | Mar 2010 | A1 |
20110043603 | Schechner et al. | Feb 2011 | A1 |
Number | Date | Country | |
---|---|---|---|
20100259651 A1 | Oct 2010 | US |
Number | Date | Country | |
---|---|---|---|
61202807 | Apr 2009 | US |