Many approaches exist that enable high-quality imaging systems with good image capture quality. These systems, however, fail to provide a complete solution imaging system that includes a given capture system and an associated quality rendering system that can extract the best color settings associated with a given scene.
Color Constancy in the Human Visual System (HVS) and Illuminant Approximation
Color constancy has been described in Chakrabarti, A., Hirakawa, K., & Zickler, T. (2011), Color Constancy with Spatio-Spectral Statistics. IEEE Transactions on Pattern Analysis and Machine Intelligence, and refers to a property of color descriptors allowing for inference of appropriate colors in spite of changes in lighting. This is therefore a perceptual notion that colors and their perception fundamentally don't change as images change in a scene during subjective testing. As such, colors of various objects are usually perceived uniformly by the Human Visual System in spite of the fact that such objects may change colors as they are being observed, or as they undergo different ambient lighting conditions from different illuminants. Stockman, A., & Brainard, D. (2010). Color vision mechanisms. OSA Handbook of Optics (3rd edition, M. Bass, ed), 11.1-11.104. hence, a very powerful quality of biological vision involves maintaining a robust estimate of the color of objects even as such objects change their perceived color values. This perceptual quality is very powerful and has been the focus of much research on how to emulate color constancy.
Described aptly in (Stockman & Brainard, 2010), “When presented in the same context under photopic conditions, pairs of lights that produce the same excitations in the long-, middle-, and short-wavelength-sensitive (L-, M-, and S-) cones match each other exactly in appearance. Moreover, this match survives changes in context and changes in adaptation, provided that the changes are applied equally to both lights.” Biological color constancy, however, has its limits and typically falls short under various conditions. As a fun example, optical illusions are presented in (Lotto's Illusion), where perceptions of different illumination levels, colors, and light sources are presented as examples. As illuminant changes happen drastically in the field-of-view, humans are more likely to confuse color quality. Estimating an illuminant is a good starting point, but compensating for the illuminant is a problem in itself, and has been the subject of much research, see Barnard, K., Cardei, V., & Funt, B. (2002). A comparison of computational color constancy algorithms—Part I: Methodology and experiments with synthesized data. IEEE Trans. Image Processing, 11 (9), 972-984, and Gijsenij, A., Gevers, T., & Weijer, J. (2011). Computational Color Constancy: Survey and Experiments. IEEE Trans. Image Processing, 20 (9), 2475-2489.as example approaches to computing illuminant estimators. Biological vision, while powerful, isn't perfect and can be fooled and confused by a variety of circumstances: it is relatively easy to fool people's visual capacities as well as other animals' by camouflaging colors, and by changing the first and second-order statistics of different objects.
Additionally, estimating the illuminant is an inherently underdetermined problem, i.e., with significantly more scene parameters than there are degrees of freedom in the data. Estimating the illuminant is, hence, an inherently nonlinear problem, with many degrees of freedom. It is therefore important to make some simplifying assumptions before proceeding:
1) The notion of color constancy in the HVS has its limitations and may not be very appropriate to applications where changes in lighting occur rapidly and frequently, such circumstances being very different than what the HVS or even biological vision may be used to.
2) Maintaining a notion of color constancy is less important than maintaining a sense of color consistency. At very high frame rates, with lighting changes occurring suddenly and abruptly, it becomes essential to maintain a notion of color consistency across frames, while not necessarily maintaining color constancy across the entire dataset, potentially comprised of millions of frames.
3) Cameras are not analogous to biological eyes. Cameras may over-expose, under-expose, or use a series of rolling exposures in ways that biological vision is not capable of. Moreover, cameras may be underexposed completely, such that only very little light is incident onto the sensors, and then having depth perception accomplished for whatever little information is present, or having the cameras significantly overexposed, such that significantly darker regions are adequately exposed. While the HVS may use autofocus, vary the aperture, and adapt to changing lighting, the speed with which it does so may hinder high-end machine vision applications. It is important in accordance with one or more embodiments of the invention to be able to preferably vary camera settings at speeds that are significantly greater than possible by the HVS.
4) There is a need to synthetically predict/compute images at different exposures in real-time and at a very high frame rates, given a set of observations. In accordance with this approach presented in accordance with one or more embodiments of the present invention, the entire relevance of the HVS becomes marginalized, relatively to computational models of scene analysis that can extra critical information about the response of the cameras to different illuminants. Again, this is a departure from the usual HVS-inspired approaches.
Therefore, in accordance with one or more embodiments of the present invention, an overview of illuminant estimation, and how such illuminant estimation is applicable to modifying a scene's trichromatic feature set, adapting and compensating for such illuminants will be provided. Various embodiments of the present invention are provided, based on the notion of chromaticity maximization and hue value consistency, including the concepts of false colors, as well as identifying metallic data profiles as well as various other specularities that may produce such false colors. Comparisons between true color-based profiles and their gray-ish counterparts may then be made. Irradiance histograms and their relationship to auto-exposure bracketing are presented. Algorithms for applying an inventive synthetic image formation approach to auto-exposure bracketing (AEB) are also presented. A control algorithm that enables such an AEB approach is provided in accordance with one or more embodiments of the invention. This AEB approach may be applied to segmentation itself, and thus such segmentation can be improved by integrating synthetic exposure estimates associated with chromaticity maximization/hue consistency computation. Therefore, it is suggested in accordance with the various embodiments of the invention, that by maximizing chromaticity, one can ensure that the true hue that is associated with a given pixel or object is accurately estimated. In doing so, Chromaticity maximization is one way of maintaining hue and color consistency.
Still other objects and advantages of the invention will in part be obvious and will in part be apparent from the specification and drawings.
The invention accordingly comprises the several steps and the relation of one or more of such steps with respect to each of the other steps, and the apparatus embodying features of construction, combinations of elements and arrangement of parts that are adapted to affect such steps, all as exemplified in the following detailed disclosure, and the scope of the invention will be indicated in the claims.
This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
For a more complete understanding of the invention, reference is made to the following description and accompanying drawings, in which:
One or more embodiments of the invention will now be described, making reference to the following drawings in which like reference numbers indicate like structure between the drawings.
Illuminant Estimation
Image colors may vary significantly as a function of an illuminant incident on one or more given surfaces that may be under observation. As has been recognized by the inventors of the present invention, successfully estimating this illuminant color is essential in the analysis of a given object's true color value. In accordance with one or more embodiments of the invention, it has been determined that color is inherently perceived as a function of incident illuminant light (or an ensemble of illuminant light sources) as well as the surface's relationship to that ensemble.
Chromaticity Maximization with the Camera Response Function
As noted above, the overall goal of Chromaticity maximization is to devise a means for maintaining color consistency across scenes with different lighting conditions and under different illuminants. To accomplish this goal, in accordance with one or more embodiments of the present invention, a camera response function is first computed and then used to synthetically create profiles of various pixels at different exposures. Then, the exposure that maximizes chromaticity is determined and is therefore employed in accordance with one or more embodiment of the invention.
Overview of the Camera Response Function
A camera response function relates scene radiance to image brightness. For a given camera response function the measured intensity, Z, is given by Debevec, P. E., & Malik, J. (1997). Recovering High Dynamic Range Radiance Maps from Photographs. SIGGRAPH 97:
Zij=f(Eitj) Equation 1
where f encapsulates the nonlinear relationship between sensor irradiance, E, and the measured intensity of a photosensitive element (pixel) over an exposure time, t (Debevec & Malik, 1997) is given by Z.
The camera response function can be used to convert intensity to irradiance by recovering the inverse of the response, denoted as f-−1:
f−1(Zij)=Eitj Equation 2
Taking the natural logarithm of both sides, g represents the inverse log function defined by:
g(Zij)=ln(f−1(Zij))=ln(Ei)+ln(tj) Equation 3
Letting Zmin and Zmax define the minimum and maximum pixel intensities of N pixels and P images, g can be solved by minimizing the following quadratic objective function:
where the first term satisfies Equation 3 and the second term imposes a smoothness constraint on the second derivative, such that:
g″(z)=g(z−1)+2g(z)−g(z+1) Equation 5
The weighting function, w, is given by Granados, M., Ajdin, B., Wand, M., Christian, T., Seidel, H.-P., & Lensch, H. P. (2010), Optimal HDR reconstruction with linear digital cameras, CVPR (pp. 215-222) San Francisco: IEEE.
Once the inverse camera response function is known, a map of the scene irradiance may be obtained from P images by weighting exposures, which produce intensities closer to the middle of the response function or another targeted set of criteria in intensity, chromaticity or any combination thereof:
For a color sensor, the camera response function may be developed separately for each channel assuming that the channels respond equally to achromatic light. This means:
Z|Z∈{R,G,B} Equation 8
where the response is given by:
Rij=fR(EtΔtj)
Gij=fG(EiΔtj)
Bij=fB(EiΔtj) Equation 9
and the inverse response at a common exposure is given as:
gR(Rij)=ln(ERi)+ln(Δtj)
gG(Gij)=ln(EGi)+ln(Δtj)
gB(Bij)=ln(EBi)+ln(Δtj) Equation 10
Creating Synthetic Photographs from Exposures
Once the irradiance of each of the channels is known, the log-inverse camera response function may be used to compute the response of the sensor to a user-specified exposure, Δt, creating virtual photographs from a series of observations at different exposure values. An example of this process is depicted in
Hence, a set of new channels, Ri, Gi, Bi, can be computed from an observed set of exposures, such that:
Ri∈gR(Ri)=ln(ERi)+ln(Δt)
Gi∈gG(Gi)=ln(EGi)+ln(Δt)
Bi∈gB(Bi)=ln(EBi)+ln(Δt) Equation 11
Equation 11 may be written more efficiently as:
Ri=gR−1(ln(ERiΔt))=fR(ERiΔt)
Gi=gB−1(ln(EGiΔt))=fG(EGiΔt)
Bi=gB−1(ln(EBiΔt))=fB(EBiΔt) Equation 12
Chromaticity Maximization and Color Consistency
With the ability to reconstruct images/pixels of a scene at any exposure value with an acceptable degree of accuracy, in accordance with various embodiments of the invention, a method with which one can extract the prevalent Hue value by viewing the region around the maximum Chromaticity that is associated with a given pixel is provided. Using the Camera Response Function, a method for maintaining robust color consistency throughout a scene by maximizing Chroma and plotting the values of hue that are associated with different Chroma values in the neighborhood of maximum Chroma is provided.
Maximizing Chromaticity
The exposure value(s) that maximize Chromaticity, Cij, for a given pixel, within the constraints of a camera's response function, by maximizing the Chromaticity for a pixel at location (i,j) may be provided as:
where Δtij is the exposure value that maximizes Chromaticity at location (i,j).
Note that, Cij is given by (Hanbury, 2008):
Cij=√{square root over (αij2+βij2)} Equation 14
where αij and βij are given by:
Computing the exposure that would maximize the Chromaticity in Equation 13, then calculating the hue associated therewith allows for finding a hue that is relatively constant across multiple lighting intensities of the same illuminant, since each different intensity will have a set of exposure values that is associated with it. This approach may also be extended to multiple illuminants as will be described in greater depth below, by estimating the illuminant value that is associated with a given scene.
Different Tones and their Apparent Color
Depending on an exposure value that is configured in one or more camera settings, different objects can appear to have colors that are very dissimilar from the perceived colors by observers. A problem arises in the definition of a perceived color in a field-of-view and how it relates to the colors that are associated with the object. However, observing the three channels of a pixel across a whole range of exposure values provides a better idea as to the object's true color, and not just the apparent color. Assuming that the lux that is incident at a certain scene is known, and assuming that the exposures that are associated with a given process to reconstruct lux are also known, it then becomes a significantly simpler problem to address estimation of the correct color. Hence, looking at a given set of exposure values and their associated images is a good way of identifying the different apparent colors that are associated with such images. An example Camera Response Function is presented in
Approximating the Camera Response Function with B-Splines
Another approach is to approximate the CRF with B-spline polynomials, such that a C2 continuous graph is presented. This is important to reduce the overall amount of data that is used to store the camera response function. The CRF may also be approximated with a best-fit log function.
Adding Context to Chromaticity Maximization—Dealing with Gray and Metallic Objects
Defining Relationships that are invariant to color temperature is essential to the identification of regions with high-reflectivity. Not every pixel will have high Chromaticity values; specifically pixels that are either too gray with naturally low Chromaticity, and pixels that belong to regions that are highly specular, both exhibit low Chromaticity. In photography, High Dynamic Range (HDR) imaging is used to control such cases, de-emphasizing specularities and over-exposing very dark regions. The intended consequence in both cases defines higher Chromaticity in both of these regions. We instead suggest a twist to HDR photography that not only allows it to run much faster, but makes it significantly friendlier to regions of higher Chromaticity, while de-emphasizing regions that are too bright (specular regions) or too dark. As an example,
Eliminating Reflectivity with the Camera Response Function
To identify reflectivity, it is important to address regions with such high reflectivity through the utilization of the CRF. Since colors can be returned that are associated with Exposure Values (EVs) that have been artificially enhanced. Note that the camera response function has a logarithmic nature that introduces various irradiance data reduction errors when the CRF, specifically in regions with dense irradiance information. This necessitates a non-uniform spacing of the exposures, Δt, used to reconstruct the irradiances, as illustrated in
Irradiance Sensitivity and Exposure Settings
Looking at the CRF's first derivative, the irradiance sensitivity is maximized where the first derivative is also maximized. As an example,
The maximum sensitivity, S, in irradiance as determined by the maximum of the first derivative, occurs at approximately Z=205 or g(Z)=0.5, where g represents the inverse CRF in accordance with a particular embodiment of the invention. However, this does not mean that in all situations the value to be chosen should be 205. Rather, a constraint will preferably be put in place, as will be described below, to address image quality and bring the value down to a range that is more consistent with visually acceptable image quality. So, chroma maximization will preferably be attempted selectively, i.e. maximizing chromaticity within a given set of constraints, one of which is maximum intensity that is associated with the data.
S=f′(g(z)) Equation 16
The log-exposure that produces the most sensitivity to a given irradiance can be computed as:
ln(Δt)=g(arg max(f′(g(z)))−ln(E) Equation 17
Irradiance Histogram Computation
The current approach to computing the irradiance histogram via the calculation of the CRF transforms the individual red, green, and blue irradiances into a grayscale irradiance using the grayscale conversion coefficients typically applied to pixel values, Ez=0.3Er+0.59Eg+0.11Eb. If the CRF were linear, this would be equivalent to the grayscale irradiance computed by first converting the pixels to grayscale, then applying the CRF,
Ez=g(z)=g(0.3R+0.59G+0.11B) Equation 18
Since the CRF is non-linear, in accordance with embodiments of the invention, the grayscale irradiance should either be computed from a grayscale conversion of the demosaiced RGB values or taken directly from the Bayer pattern.
Both approaches assume that the grayscale irradiance adequately represents the per-channel irradiance. Although this assumption may be practical to simplify the process of synthetic data creation, it still discounts the fact that the differences between the irradiance channels drastically affect hue. Another option is to create an irradiance histogram over all channels simultaneously, such that the modes reflect the most dominant irradiances of all channels.
Relationship Between Modes and Exposure Settings
In an irradiance histogram, a mode may be present around one or more local maxima. As an example,
In one exception, highlighted in
Unrecoverable Irradiance
A problem with this approach occurs if the exposures that are used for computing the irradiances cannot recover certain ranges of irradiances. This can occur when the scene changes too drastically and the exposures clip or saturate one or more of the source images. One option is to set such pixels' associated irradiances to either zero or 255, representing a means with which to mitigate the associated effects. This is a reasonable assumption to make since the associated source values, i.e. from the source images at different exposure settings, are too extreme on one side of the intensity range or the other. The rationale behind this issue stems from the fact that, given proper exposures, irradiances that are deemed unrecoverable are too low, representing pixels that are near zero in intensity, and hence have very little pixel information that is associated with them. These irradiances can also be too high, i.e. representing specularities in the field of view where the pixel's intensity in one or more of the three channels is too high for the observations that are associated with an exposure range. In this case, a multilinear constraint on dichromatic planes (Toro & Funt, 2007) may be used to first estimate the illuminant and then remove illuminant effects from the scene, to prevent artificial colors from occurring. Either way, recovering the irradiance would then require more than the method that is described so far and would require an understanding of the illuminant that is associated with the scene, as well as eliminating it.
Histogram Cases
Cases of different histogram distributions, as well as the lighting conditions that are associated with them, are presented in
Auto Exposure Bracketing
Auto Exposure bracketing is a means of capturing high dynamic range (HDR) images from a set of low-dynamic range images at different exposure settings. This is a prevalent technique in photography in which multiple images are taken, typically three, one with an optimal exposure setting, with a second image being taken at a lower EV value, and a third image being taken at a higher EV value. The idea is to try and get as much dynamic range of the scene as possible with a single image, and then use the other images to fill in any missing information, by over-exposing dark regions or under-exposing really bright regions.
As mentioned earlier, High Dynamic Range imaging is critical for photography. HDR is used in a number of instances. Auto-exposure bracketing techniques are very prevalent in industry (Canon U.S.A., Inc., 2012), academia (Robertson, Borman, & Stevenson, 2003), and in patents and patent applications (Yeo & Tay, 2009).
Some Common Assumptions and Weaknesses
Almost every AEB technique requires a known intensity histogram (Ohsawa, 1990). Some may also require a known irradiance histogram (Guthier, Kopf, & Effelsberg, 2012) as well. However, if the histogram is not adequately covering pertinent values in the scene, then the AEB technique will converge on the wrong settings. Also, if the histogram covers regions belonging to the background, instead of the region of interest, the AEB algorithm will converge on exposures that don't capture the dynamic range of values of the targeted ROI.
A Control Algorithm for Recovering Max Chromaticity Images
Therefore, in accordance with one or more embodiments of the present invention, a new approach that can take advantage of chromaticity maximization under resilient hue conditions, while maintaining a robust exposure bracketing algorithm is presented. This approach is depicted in
Control Algorithm for Adaptive Pixel-Based Irradiance Estimation
A control algorithm provided in accordance with one or more embodiments of the present invention can now be used to describe the scene.
UE—Under exposed: stands for scenes that are underexposed and whose exposure values need to be increased to compensate for such a case.
OE—Over exposed: stands for scenes that are overexposed and whose exposure values need to be decreased to compensate for too much irradiance in the scene.
WE—Well-exposed: stands for scenes that are well-defined with a quality dynamic range defined by exposures that cover most of the scene.
LL—Low lux: this stands for scenes (generally under 500 Lux) with a limited dynamic range that requires fewer exposures to recover irradiance.
NM—Night mode: this is an extension of the low-lux case (under 20 Lux) where a dual mode sensor may choose to turn on infrared LEDs for operating the system. NM can be accessed from the LL or SM states.
SM—Scan mode: represents a state where the system searches through the set or subset of available exposures to optimize settings for a new scene.
In accordance with an embodiment of the invention, additionally defined transition criteria based on the presence of underexposed as well as over exposed pixels, the mean of the irradiance distribution, the support window for the hue distribution, and the overall chroma distribution are preferably employed. As is shown in
Example Source and Output Images
As an example,
A second example is shown in
Ensembles of Illuminants for a Given Scene and their Temperature Estimation
Estimating the illuminant is critical to accounting for it in a given scene and correcting the RGB color space, per the effects of the illuminant (Gijsenij, Gevers, & Weijer, 2011). Black body radiators are ones that emit light when excited at different temperatures. One can look at regular light sources as black body radiators. These vary from red to white, according to the temperature that is associated with their light color, in Kelvins. Typical temperature ranges vary from 2000K (red) to 9000K (blue/white or white with blue tint).
A set of illuminants may be defined based on values that are associated with their respective color temperatures (usually defined in Kelvins). In a manner similar to what has been defined in (Barnard, Cardei, & Funt, 2002), each illuminant may be defined according to its Chromaticity map, and may be expanded to include a more substantive set of illuminants. As a result, to find an approximate illuminant, the Veronese map of degree n may be computed, such that
c(x)Tv=0 Equation 19
where c(x) is an observed color belonging to a single material and v is a normal vector associated with the dichromatic plane of the material. The projection of the observed color is projected as:
d(x;w)T=M(w)Tc(x)
d(x;w)=M(w)Tc(x) Equation 20
where w is a vector representing the color of the illuminant and d(x;w) represents the projection of the observed color c(x) onto a 2-D subspace using the 3×2 projection matrix M(w). The 2-D subspace is orthogonal to the light vector.
d(x;w)Tu=0 Equation 21
where u is a vector in the 2-D subspace that consists of the coefficients of a linear combination.
where vn(d)=[d1nd1n-1d21d1n-2d22 . . . d2n]T
ui is one vector in a collection of n vectors that represent all of the materials in the scene.
vn(d)=[d1nd1n-1d21d1n-2d22 . . . d2n]T Equation 23
where d1 and d2 are the 2-D coordinates of the projected color onto the subspace that is orthogonal to the light vector and vn(d) is the Veronese map of degree n and n is the number of materials in the scene.
where Λn consists of the Veronese maps for all of the colors under consideration.
Equation 25 can now be rewritten as:
where an is the set of coefficients associated with each Veronese map in Λn.
As mentioned in, Toro, J., & Funt, B. B. (2007). A multilinear constraint on dichromatic planes for illumination estimation. IEEE Trans. Image Processing, 16 (1), 92-97, a candidate color, w, is the actual color of the light if the smallest eigenvalue of matrix, ΛnTΛn, is equal to zero. Hence, the approximate illuminant is identified as the one corresponding to the smallest eigenvalue. To choose from the different materials, one approach can be applied where scenes are subdivided into various regions, based on both color as well as texture. A system for performing this approach may comprise any number of region classification schemes, and in a preferred embodiment, a system such as that described in U.S. patent application Ser. No. 12/784,123, currently pending, the contents thereof being incorporated herein by reference. Hence, an image is preferably broken up into regions of dominant texture or color, with multiple segments representing different blocks of data being present in any given image. For each block, the illuminant estimation process is conducted, such that the temperature of the illuminant is computed, as well as the associated RGB values. The color space of the image is then rectified to mitigate the illuminant's effects. The rest of the software stack follows from this step, to reconstruct segments followed by depth reconstruction, as described in U.S. patent application Ser. Nos. 12/784,123; 12/784,022; 13/025,038; 13/025,055; 13/025,070; 13/297,029; 13/297,144; 13/294,481; and 13/316,606, the entire contents of each of these application being incorporated herein by reference.
Example Approach to Illuminant Estimate
Referring next to
If the inquiry at step 1225 is instead answered in the positive, and it is therefore determined that all desired candidate illuminants have been considered, processing then passes to step 1230 where the number of materials and block size are configured. Next, at step 1235, a projection matrix for the two-dimensional subspace orthogonal to a light vector may be computed, preferably employing the relevant equations described above. Then for each pixel defined in the two dimensional subspace, at step 1240, the pixel color is then preferably projected onto the subspaces of illuminants preferably according to Equation 5 described above, and at step 1245 a Veronese map from the result of the projections may then be computed, preferably employing Equation 8 noted above. Processing then continues to step 1250 where it is questioned whether the processing associated with steps 1240 and 1245 has been performed for each pixel (or substantially each pixel) in a defined block, or section of the image under observation. If this inquiry is answered in the negative, and it is determined that processing has not been completed for all desired pixels, processing returns to step 1240 for addressing a next pixel.
If on the other hand, if the inquiry at step 1250 is answered in the affirmative, and it is therefore determined that processing has been completed for all desired pixels, processing then passes to step 1255, where a matrix is preferably then built from the Veronese projections of all the pixels in a given block of the image, preferably employing Equation 10 described above. At a next step 1260, the resultant matrix is then multiplied by its transpose, the eigen values of the results being computed, and the smallest eigenvalue is obtained, preferably in accordance with Equation 11 described above. Processing then passes to step 1265 where it is questioned whether processing for all desired candidate illuminants (or substantially all desired candidate illuminants) for the current block has been completed. If the inquiry at step 1265 is answered in the negative, and therefore it is determined that processing has not been completed for all desired candidate illuminants for the current block, processing returns to step 1235 for a next of the desired candidate illuminant values for the current block.
If on the other hand, the inquiry at step 1265 is answered in the affirmative, thus confirming that processing has been completed for all of the desired candidate illuminants for the current block, processing then passes to step 1270 where the relative error is preferably computed for each illuminant by dividing the smallest eigenvalue of each illuminant by the eigenvalue of the set. Processing then passes to step 1275, where it is questioned whether processing for all blocks (or substantially all blocks) in the image has been completed. If this inquiry is answered in the negative, and it is therefore determined that processing for all blocks in the image has not been completed, processing once again returns to step 1235 for a next block in the image.
If on the other hand the inquiry at step 1275 is answered in the affirmative, and it is therefore determined that processing for all blocks in the image has been completed, processing then passes to step 1280 where an illuminant that minimizes relative error across all of the blocks is chosen.
Many other variants to this approach are contemplated in accordance with the various embodiments of the invention. For the sake of consistency and to enable visualization, the next following description should be considered applicable to two-dimensional as well as three-dimensional visualizations of the resultant light vectors in RGB space, while the three-dimensional implementation will be described.
Three-Dimensional Visualization of the Light Vector Relative to the RGB Space
Referring next to
Integration of Exposure Bracketing Features into Segmentation
As noted above, one potential problem that is associated with segmentation under challenging lighting conditions is that of maintaining color consistency as pixels are being tracked in a field-of-view. In accordance with one or more embodiments of the present invention, through the user of Chroma maximization, color consistency can be achieved and maintained even as color constancy is no longer plausible throughout a scene. A clear distinction should be drawn between color constancy, i.e. maintaining a constant color across the entire span of frames, and color consistency, where a given color is allowed to change, but not across adjacent frames. By computing max Chroma through simulating row lines or exposure values, one can now move on to the problem of segmentation. Specifically, given that different objects may appear with a similar hue or even Chroma under similar lighting conditions, it is left to the use and manipulation of exposure settings to identify differences in different materials and the way that such differences may be exploited in segmentation. In accordance with one or more embodiments of the invention, quality of color-based segmentation accuracy can be accomplished by computing which exposure value(s) would provide for Chroma maximization, per the description of the Camera Response Function at different irradiances. Not only can segmentation now be accomplished based on texture and color as described in U.S. patent application Ser. Nos. 12/784,123; 12/784,022; 13/025,038; 13/025,055; 13/025,070; 13/297,029; 13/297,144; 13/294,481; and 13/316,606, the entire contents of each of these application being incorporated herein by reference, wherein both texture and color are used to segment in stereo images, but one can also define or profile the evolution of such a color across multiple exposure settings, given only a few observations. Such information is then preferably integrated into segmentation and used to further enhance pixel and segment-level discriminability. Thus, a pixel may be classified and clustered by its response at different exposures.
Light Slicing
Referring next to
Color Consistency Towards Feature Constancy
One can now start to look at what features to extract from better discriminability and color quality. These features are presented below, as determined by the inventors of the present invention:
1. Prevalent or Persistent Hue Value:
A prevalent or persistent hue can be extracted from the Chroma maximization graphs that are associated with the camera response functions of the R, G, and B channels, (see
2. Chroma Profile/Max Chromaticity Identification and The Notion of False Colors:
Typical surfaces with good color quality exhibit good Chroma responses at a range of exposure settings, and may possibly be approximated with a Gaussian curve centered around an optimal exposure. This is especially true for skin tone, where skin tone modeling in the past has focused on Gaussian Mixture Models. For good colors well within a camera's dynamic range, and with adequate lighting, a large support window of high Chromaticity is typically the case (see
3. Illuminant Estimate:
It now becomes critical to divide up the scene into regions of relatively constant illuminant estimates. Breaking up the scene into ensembles of illuminants has been presented above. However, here, the contribution of an illuminant ensemble map to the computation of depth in a field-of-view is also presented and integrated into the process that has been defined to compute overall depth. One can also mitigate the effects of a given illuminant in the scene. Illuminant estimation for an example illuminant estimation approach has been described above.
4. Irradiance:
As disclosed above, irradiance is a measure of reconstructed brightness incident on a particular pixel in an image. It can be used to detect the amount of light in a scene. Irradiance is also used to compute the maximum Chroma as well as the hue values that are associated with a given pixel. Irradiance is also used for detecting a given feature's constancy in the scene.
Putting it all Together
A novel technique for pixel superresolution is therefore provided in which multiple pixels can be extracted from a smaller number of observations. A multi-dimensional feature vector, v, is given by:
v={h,c,i,t} Equation 27
Where h represents the hue feature vector, c represents the Chromaticity feature vector, i represents the irradiance feature vector, and t represents texture as a feature vector. All of these vectors have been described earlier with the exception of t, the texture feature vector. However, using texture has been defined extensively in earlier works by the assignee of the present application, as noted above.
Extraction of Feature Vectors
One of the challenges that is facing all of segmentation is the capability to accurately organize the scene by means of color quality, and texture features, such that different regions in the scene can be organized via color/texture characteristics either in the spatial or frequency domain. Motion features can also be extracted to increase scene segmentation. In accordance with the various embodiments of the invention, the inventors therefore propose an entirely novel approach to scene segmentation through pixel superresolution via features that are extracted from the camera response function.
The above-described process may also be generally applicable to photography, allowing for a high accuracy HDR type process producing images as well as videos with more vivid colors, and less sensitive to difficult lighting conditions.
Application—High Dynamic Range Video and Photography
Given all of the features and the segmentation components presented above, users have the ability to interactively enable HDR features with this approach. Alternatively, an HDR video scheme can be developed that utilizes the various features that have been described above. Specifically, regions of common synthetic exposures may be exposed more uniformly, and various stable segments may be under of over-exposed based on selection criteria defined by the user. Users can also autofocus on different parts of the video in real-time, given that every component of the video is being tracked in real-time.
5. Conclusions
A novel approach to AEB as well as segmentation and disparity computation has been presented that takes into account the utilization of the Camera Response Function to synthetically create different images, maximize chromaticity, and create a notion of color consistency. This is used to produce a robust segmentation/disparity estimation algorithm. Illuminant estimation may also be used for compensating color differences that are impacted by the effect of incident illuminants. The approach has multiple other applications include HDR photography and real-time HDR video.
Furthermore, while the invention has been primarily described related to a general imaging system, the various embodiments of the invention may also be applicable to any other platform that may be used for imaging, including but not limited to mobile devices, devices employing a Graphical Processing Unit (GPU), display on 3D screens, display on 2D screens, various types of photography employing one or multiple single frame or video cameras, and the like.
It will thus be seen that the objects set forth above, among those made apparent from the preceding description, are efficiently attained and, because certain changes may be made in carrying out the above method and in the construction(s) set forth without departing from the spirit and scope of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
It is also to be understood that this description is intended to cover all of the generic and specific features of the invention herein described and all statements of the scope of the invention which, as a matter of language, might be said to fall there between.
Number | Name | Date | Kind |
---|---|---|---|
5454043 | Freeman | Sep 1995 | A |
5504524 | Lu et al. | Apr 1996 | A |
5544050 | Abe et al. | Aug 1996 | A |
5581276 | Cipolla et al. | Dec 1996 | A |
5594469 | Freeman et al. | Jan 1997 | A |
5699441 | Sagawa et al. | Dec 1997 | A |
5767842 | Korth | Jun 1998 | A |
5887069 | Sakou et al. | Mar 1999 | A |
5990865 | Gard | Nov 1999 | A |
6002808 | Freeman | Dec 1999 | A |
6072494 | Nguyen | Jun 2000 | A |
6075895 | Qiao et al. | Jun 2000 | A |
6115482 | Sears et al. | Sep 2000 | A |
6128003 | Smith et al. | Oct 2000 | A |
6141434 | Christian et al. | Oct 2000 | A |
6147678 | Kumar et al. | Nov 2000 | A |
6181343 | Lyons | Jan 2001 | B1 |
6195104 | Lyons | Feb 2001 | B1 |
6204852 | Kumar et al. | Mar 2001 | B1 |
6215890 | Matsuo et al. | Apr 2001 | B1 |
6222465 | Kumar et al. | Apr 2001 | B1 |
6240197 | Christian et al. | May 2001 | B1 |
6240198 | Rehg et al. | May 2001 | B1 |
6252598 | Segen | Jun 2001 | B1 |
6256033 | Nguyen | Jul 2001 | B1 |
6256400 | Takata et al. | Jul 2001 | B1 |
6269172 | Rehg et al. | Jul 2001 | B1 |
6323942 | Bamji | Nov 2001 | B1 |
6324453 | Breed et al. | Nov 2001 | B1 |
6360003 | Doi et al. | Mar 2002 | B1 |
6363160 | Bradski et al. | Mar 2002 | B1 |
6377238 | McPheters | Apr 2002 | B1 |
6389182 | Ihara et al. | May 2002 | B1 |
6394557 | Bradski | May 2002 | B2 |
6400830 | Christian et al. | Jun 2002 | B1 |
6434255 | Harakawa | Aug 2002 | B1 |
6442465 | Breed et al. | Aug 2002 | B2 |
6456728 | Doi et al. | Sep 2002 | B1 |
6478432 | Dyner | Nov 2002 | B1 |
6509707 | Yamashita et al. | Jan 2003 | B2 |
6512838 | Rafii et al. | Jan 2003 | B1 |
6526156 | Black et al. | Feb 2003 | B1 |
6553296 | Breed et al. | Apr 2003 | B2 |
6556708 | Christian et al. | Apr 2003 | B1 |
6571193 | Unuma et al. | May 2003 | B1 |
6590605 | Eichenlaub | Jul 2003 | B1 |
6600475 | Gutta et al. | Jul 2003 | B2 |
6608910 | Srinivasa et al. | Aug 2003 | B1 |
6614422 | Rafii et al. | Sep 2003 | B1 |
6624833 | Kumar et al. | Sep 2003 | B1 |
6674877 | Jojic et al. | Jan 2004 | B1 |
6674895 | Rafii et al. | Jan 2004 | B2 |
6678425 | Flores et al. | Jan 2004 | B1 |
6681031 | Cohen et al. | Jan 2004 | B2 |
6683968 | Pavlovic et al. | Jan 2004 | B1 |
6757571 | Toyama | Jun 2004 | B1 |
6766036 | Pryor | Jul 2004 | B1 |
6768486 | Szabo et al. | Jul 2004 | B1 |
6788809 | Grzeszczuk et al. | Sep 2004 | B1 |
6795567 | Cham et al. | Sep 2004 | B1 |
6801637 | Voronka et al. | Oct 2004 | B2 |
6804396 | Higaki et al. | Oct 2004 | B2 |
6829730 | Nadeau-Dostie et al. | Dec 2004 | B2 |
6857746 | Dyner | Feb 2005 | B2 |
6901561 | Kirkpatrick et al. | May 2005 | B1 |
6937742 | Roberts et al. | Aug 2005 | B2 |
6940646 | Taniguchi et al. | Sep 2005 | B2 |
6944315 | Zipperer et al. | Sep 2005 | B1 |
6950534 | Cohen et al. | Sep 2005 | B2 |
6993462 | Pavlovic et al. | Jan 2006 | B1 |
7039676 | Day et al. | May 2006 | B1 |
7046232 | Inagaki et al. | May 2006 | B2 |
7050606 | Paul et al. | May 2006 | B2 |
7050624 | Dialameh et al. | May 2006 | B2 |
7058204 | Hildreth et al. | Jun 2006 | B2 |
7065230 | Yuasa et al. | Jun 2006 | B2 |
7068842 | Liang et al. | Jun 2006 | B2 |
7095401 | Liu et al. | Aug 2006 | B2 |
7102615 | Marks | Sep 2006 | B2 |
7129927 | Mattson | Oct 2006 | B2 |
7170492 | Bell | Jan 2007 | B2 |
7190811 | Ivanov | Mar 2007 | B2 |
7203340 | Gorodnichy | Apr 2007 | B2 |
7212663 | Tomasi | May 2007 | B2 |
7221779 | Kawakami et al. | May 2007 | B2 |
7224830 | Nefian et al. | May 2007 | B2 |
7224851 | Kinjo | May 2007 | B2 |
7233320 | Lapstun et al. | Jun 2007 | B1 |
7236611 | Roberts et al. | Jun 2007 | B2 |
7239718 | Park et al. | Jul 2007 | B2 |
7257237 | Luck et al. | Aug 2007 | B1 |
7274800 | Nefian et al. | Sep 2007 | B2 |
7274803 | Sharma et al. | Sep 2007 | B1 |
7289645 | Yamamoto et al. | Oct 2007 | B2 |
7295709 | Cootes et al. | Nov 2007 | B2 |
7296007 | Funge et al. | Nov 2007 | B1 |
7308112 | Fujimura et al. | Nov 2007 | B2 |
7340077 | Gokturk et al. | Mar 2008 | B2 |
7340078 | Shikano et al. | Mar 2008 | B2 |
7342485 | Joehl et al. | Mar 2008 | B2 |
7346192 | Yuasa et al. | Mar 2008 | B2 |
7348963 | Bell | Mar 2008 | B2 |
7359529 | Lee | Apr 2008 | B2 |
7372977 | Fujimura et al. | May 2008 | B2 |
7379563 | Shamaie | May 2008 | B2 |
7391409 | Zalewski et al. | Jun 2008 | B2 |
7394346 | Bodin | Jul 2008 | B2 |
7412077 | Li et al. | Aug 2008 | B2 |
7415126 | Breed et al. | Aug 2008 | B2 |
7415212 | Matsushita et al. | Aug 2008 | B2 |
7421093 | Hildreth et al. | Sep 2008 | B2 |
7423540 | Kisacanin | Sep 2008 | B2 |
7444001 | Roberts et al. | Oct 2008 | B2 |
7450736 | Yang et al. | Nov 2008 | B2 |
7460690 | Cohen et al. | Dec 2008 | B2 |
7477758 | Piirainen et al. | Jan 2009 | B2 |
7489308 | Blake et al. | Feb 2009 | B2 |
7489806 | Mohri et al. | Feb 2009 | B2 |
7499569 | Sato et al. | Mar 2009 | B2 |
7512262 | Criminisi et al. | Mar 2009 | B2 |
7519223 | Dehlin et al. | Apr 2009 | B2 |
7519537 | Rosenberg | Apr 2009 | B2 |
7574020 | Shamaie | Aug 2009 | B2 |
7590262 | Fujimura et al. | Sep 2009 | B2 |
7593552 | Higaki et al. | Sep 2009 | B2 |
7598942 | Underkoffler et al. | Oct 2009 | B2 |
7599547 | Sun et al. | Oct 2009 | B2 |
7606411 | Venetsky et al. | Oct 2009 | B2 |
7612813 | Hunter | Nov 2009 | B2 |
7614019 | Rimas Ribikauskas et al. | Nov 2009 | B2 |
7620316 | Boillot | Nov 2009 | B2 |
7646372 | Marks et al. | Jan 2010 | B2 |
7660437 | Breed | Feb 2010 | B2 |
7665041 | Wilson et al. | Feb 2010 | B2 |
7676062 | Breed et al. | Mar 2010 | B2 |
7720282 | Blake et al. | May 2010 | B2 |
7721207 | Nilsson | May 2010 | B2 |
7804998 | Mundermann Lars et al. | Sep 2010 | B2 |
20010001182 | Ito et al. | May 2001 | A1 |
20010030642 | Sullivan et al. | Oct 2001 | A1 |
20020041327 | Hildreth et al. | Apr 2002 | A1 |
20020064382 | Hildreth et al. | May 2002 | A1 |
20020090133 | Kim et al. | Jul 2002 | A1 |
20020140633 | Rafii et al. | Oct 2002 | A1 |
20020176010 | Wallach et al. | Nov 2002 | A1 |
20040183775 | Bell | Sep 2004 | A1 |
20050002074 | McPheters et al. | Jan 2005 | A1 |
20050083314 | Shalit et al. | Apr 2005 | A1 |
20050105775 | Luo et al. | May 2005 | A1 |
20050190443 | Nam et al. | Sep 2005 | A1 |
20050286756 | Hong et al. | Dec 2005 | A1 |
20060093186 | Ivanov | May 2006 | A1 |
20060101354 | Hashimoto et al. | May 2006 | A1 |
20060136846 | Im et al. | Jun 2006 | A1 |
20060139314 | Bell | Jun 2006 | A1 |
20060221072 | Se et al. | Oct 2006 | A1 |
20070055427 | Sun et al. | Mar 2007 | A1 |
20070113207 | Gritton | May 2007 | A1 |
20070132721 | Glomski et al. | Jun 2007 | A1 |
20070195997 | Paul et al. | Aug 2007 | A1 |
20070211165 | Yaguchi | Sep 2007 | A1 |
20070263932 | Bernardin et al. | Nov 2007 | A1 |
20070280505 | Breed | Dec 2007 | A1 |
20080002878 | Meiyappan et al. | Jan 2008 | A1 |
20080005703 | Radivojevic et al. | Jan 2008 | A1 |
20080013793 | Hillis et al. | Jan 2008 | A1 |
20080037875 | Kim et al. | Feb 2008 | A1 |
20080052643 | Ike et al. | Feb 2008 | A1 |
20080059578 | Albertson et al. | Mar 2008 | A1 |
20080065291 | Breed | Mar 2008 | A1 |
20080069415 | Schildkraut et al. | Mar 2008 | A1 |
20080069437 | Baker | Mar 2008 | A1 |
20080104547 | Morita et al. | May 2008 | A1 |
20080107303 | Kim et al. | May 2008 | A1 |
20080120577 | Ma et al. | May 2008 | A1 |
20080178126 | Beeck et al. | Jul 2008 | A1 |
20080181459 | Martin et al. | Jul 2008 | A1 |
20080219501 | Matsumoto | Sep 2008 | A1 |
20080219502 | Shamaie | Sep 2008 | A1 |
20080225041 | El Dokor et al. | Sep 2008 | A1 |
20080229255 | Linjama et al. | Sep 2008 | A1 |
20080240502 | Freedman et al. | Oct 2008 | A1 |
20080244465 | Kongqiao et al. | Oct 2008 | A1 |
20080244468 | Nishihara et al. | Oct 2008 | A1 |
20080267449 | Dumas et al. | Oct 2008 | A1 |
20080282202 | Sunday | Nov 2008 | A1 |
20090006292 | Block | Jan 2009 | A1 |
20090027337 | Hildreth | Jan 2009 | A1 |
20090037849 | Immonen et al. | Feb 2009 | A1 |
20090040215 | Afzulpurkar et al. | Feb 2009 | A1 |
20090060268 | Roberts et al. | Mar 2009 | A1 |
20090074248 | Cohen et al. | Mar 2009 | A1 |
20090077504 | Bell et al. | Mar 2009 | A1 |
20090079813 | Hildreth | Mar 2009 | A1 |
20090080526 | Vasireddy et al. | Mar 2009 | A1 |
20090085864 | Kutliroff et al. | Apr 2009 | A1 |
20090102788 | Nishida et al. | Apr 2009 | A1 |
20090102800 | Keenan | Apr 2009 | A1 |
20090103780 | Nishihara et al. | Apr 2009 | A1 |
20090108649 | Kneller et al. | Apr 2009 | A1 |
20090109036 | Schalla et al. | Apr 2009 | A1 |
20090110292 | Fujimura et al. | Apr 2009 | A1 |
20090115721 | Aull et al. | May 2009 | A1 |
20090116742 | Nishihara | May 2009 | A1 |
20090116749 | Cristinacce et al. | May 2009 | A1 |
20090150160 | Mozer | Jun 2009 | A1 |
20090153366 | Im et al. | Jun 2009 | A1 |
20090153655 | Ike et al. | Jun 2009 | A1 |
20090180668 | Jones et al. | Jul 2009 | A1 |
20090183125 | Magal et al. | Jul 2009 | A1 |
20090183193 | Miller, IV | Jul 2009 | A1 |
20090189858 | Lev et al. | Jul 2009 | A1 |
20090208057 | Wilson et al. | Aug 2009 | A1 |
20090222149 | Murray et al. | Sep 2009 | A1 |
20090228841 | Hildreth | Sep 2009 | A1 |
20090231278 | St Hilaire et al. | Sep 2009 | A1 |
20090244309 | Maison et al. | Oct 2009 | A1 |
20090249258 | Tang | Oct 2009 | A1 |
20090262986 | Cartey et al. | Oct 2009 | A1 |
20090268945 | Wilson et al. | Oct 2009 | A1 |
20090273563 | Pryor | Nov 2009 | A1 |
20090273574 | Pryor | Nov 2009 | A1 |
20090273575 | Pryor | Nov 2009 | A1 |
20090278915 | Kramer et al. | Nov 2009 | A1 |
20090295738 | Chiang | Dec 2009 | A1 |
20090296991 | Anzola | Dec 2009 | A1 |
20090315740 | Hildreth et al. | Dec 2009 | A1 |
20090316952 | Ferren et al. | Dec 2009 | A1 |
Entry |
---|
Freeman, W. T. et al., “The Design and Use of Steerable Filters”, IEEE Transactions of Pattern Analysis and Machine Intelligence V. 13, (Sep. 1991),891-906. |
Simoncelli, E.P. et al., “Shiftable Multi-scale Transforms”, IEEE Transactions on Information Theory V. 38, (Mar. 1992),587-607. |
Simoncelli, E.P. et al., “The Steerable Pyramid: A Flexible Architecture for Multi-Scale Derivative Computation”, Proceedings of ICIP-95 V. 3, (Oct. 1995),444-447. |
Chen, J et al., “Adaptive Perceptual Color-Texture Image Segmentation”, IEEE Transactions on Image Processing, v. 14, No. 10, (Oct. 2005),1524-1536 (2004 revised draft). |
Halfhill, Tom R., “Parallel Processing with CUDA”, Microprocessor Report, Available at http://www.nvidia.com/docs/IO/55972/220401_Reprint.pdf,(Jan. 28, 2008). |
Farber, Rob “CUDA, Supercomputing for the Masses: Part 4, The CUDA Memory Model”, Under the High Performance Computing section of the Dr. Dobbs website, p. 3 available at http://www.ddj.com/hpc-high-performance-computing/208401741, 3. |
Rajko, S et al., “HMM Parameter Reduction for Practice Gesture Recognition”, Proceedings of the International Conference on Automatic Gesture Recognition, (Sep. 2008). |
Hinton, Geoffrey et al., “A Fast Learning Algorithm for Deep Belief Nets”, Neural Computation, V. 18, 1527-1554. |
Susskind, Joshua M., et al., “Generating Facial Expressions with Deep Belief Nets”, Department of Psychology, Univ. of Toronto I-Tech Education and Publishing, (2008),421-440. |
Bleyer, Michael et al., “Surface Stereo with Soft Segmentation.”, Computer Vision and Pattern Recognition. IEEE, 2010, (2010). |
Chen, Junqing et al., “Adaptive perceptual color-texture image segmentation.”,The International Society for Optical Engineering, SPIE Newsroom, (2006),1-2. |
Forsyth, David A., et al., “Stereopsis”, In Computer Vision a Modern Approach Prentice Hall, 2003, (2003). |
Harris, Mark et al., “Parallel Prefix Sum (Scan) with CUDA”, vol. 39 in GPU Gems 3, edited by Hubert Nguyen, (2007). |
Hirschmuller, Heiko “Stereo Vision in Structured Environments by Consistent Semi-Global Matching”, Computer Vision and Pattern Recognition, CVPR 06, (2006),2386-2393. |
Ivekovic, Spela et al., “Dense Wide-baseline Disparities from Conventional Stereo for Immersive Videoconferencing”, ICPR. 2004, (2004),921-924. |
Kaldewey, Tim et al., “Parallel Search on Video Cards.”, First USENIX Workshop on Hot Topics in Parallelism(HotPar '09), (2009). |
Kirk, David et al., “Programming Massively Parallel Processors a Hands-on Approach”, Elsevier, 2010, (2010). |
Klaus, Andreas et al., “Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure”, Proceedings of ICPR 2006. IEEE, 2006, (2006),15-18. |
Kolmogorov, Vladimir et al., “Computing Visual Correspondence with Occlusions via Graph Cuts”, International Conference on Computer Vision. 2001., (2001). |
Kolmogorov, Vladimir et al., “Generalized Multi-camera Scene Reconstruction Using Graph Cuts.”, Proceedings for the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. 2003., (2003). |
Kuhn, Michael et al., “Efficient ASIC Implementation of a Real-Time Depth Mapping Stereo Vision System”, Proceedings of 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. Taipei, Taiwan: IEEE, 2009., (2009). |
Li, Shigang “Binocular Spherical Stereo”, IEEE Transactions on Intelligent Transportation Systems(IEEE) 9, No. 4 (Dec. 2008), (Dec. 2008),589-600. |
Marsalek, M et al., “Semantic hierarchies for visual object recognition”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR '07. MN: IEEE, 2007, (2007),1-7. |
Metzger, Wolfgang “Laws of Seeing”, MIT Press, 2006, (2006). |
Min, Dongbo et al., “Cost Aggregation and Occlusion Handling With WLS in Stereo Matching”, Edited by IEEE. IEEE Transactions on Image Processing 17(2008), (2008),1431-1442. |
“NVIDIA: CUDA compute unified device architecture, prog. guide, version 1.1”, NVIDIA, (2007). |
Remondino, Fabio et al., “Turning Images into 3-D Models”, IEEE Signal Processing Magazine, (2008). |
Richardson, Ian E., “H.264/MPEG-4 Part 10 White Paper”, White Paper/www.vcodex.com, (2003). |
Sengupta, Shubhabrata “Scan Primitives for GPU Computing”, Proceedings of the 2007 Graphics Hardware Conference. San Diego, CA, 2007, (2007),97-106. |
Sintron, Eric et al., “Fast Parallel GPU-Sorting Using a Hybrid Algorithm”, Journal of Parallel and Distributed Computing (Elsevier) 68, No. 10, (Oct. 2008),1381-1388. |
Wang, Zeng-Fu et al., “A Region Based Stereo Matching Algorithm Using Cooperative Optimization”, CVPR (2008). |
Wei, Zheng et al., “Optimization of Linked List Prefix Computations on Multithreaded GPUs Using CUDA”, 2010 IEEE International Symposium on Parallel & Distributed Processing(IPDPS). Atlanta, (2010). |
Wiegand, Thomas et al., “Overview of the H.264/AVC Video Coding Standard”, IEEE Transactions on Circuits and Systems for Video Technology 13 , No. 7 (Jul. 2003),560-576. |
Woodford, O.J. et al., “Global Stereo Reconstruction under Second Order Smoothness Priors”, IEEE Transactions on Pattern Analysis and Machine Intelligence(IEEE) 31, No. 12, (2009),2115-2128. |
Yang, Qingxiong et al., “Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling”, IEEE Transactions on Pattern Analysis and Machine Intelligence(IEEE) 31, No. 3, (Mar. 2009),492-504. |
Zinner, Christian et al., “An Optimized Software-Based Implementation of a Census-Based Stereo Matching Algorithm”, Lecture Notes in Computer Science(SpringerLink) 5358, (2008),216-227. |
“PCT Search report”, PCT/US2010/035717, (dated Sep. 1, 2010),1-29. |
“PCT Written opinion”, PCT/US2010/035717, (dated Dec. 1, 2011),1-9. |
“PCT Search report”, PCT/US2011/49043, (dated Mar. 21, 2012), 1-4. |
“PCT Written opinion”, PCT/US2011/49043, (dated Mar. 21, 2012), 1-4. |
“PCT Search report”, PCT/US2011/049808, (dated Jan. 12, 2012), 1-2. |
“PCT Written opinion”, PCT/US2011/049808, (dated Jan. 12, 2012), 1-5. |
“Non-Final Office Action”, U.S. Appl. No. 12/784,123, (dated Oct. 2, 2012), 1-20. |
“Non-Final Office Action”, U.S. Appl. No. 12/784,022, (dated Jul. 16, 2012), 1-14. |
Tieleman, T et al., “Using Fast weights to improve persistent contrastive divergence”, 26th International Conference on Machine Learning New York, NY ACM, (2009),1033-1040. |
Sutskever, I et al., “The recurrent temporal restricted boltzmann machine”, NIPS, MIT Press, (2008),1601-1608. |
Parzen, E “On the estimation of a probability density function and the mode”, Annals of Math. Stats., 33, (1962),1065-1076. |
Hopfield, J.J. “Neural networks and physical systems with emergent collective computational abilities”, National Academy of Sciences, 79, (1982),2554-2558. |
Culibrk, D et al., “Neural network approach to background modeling for video object segmentation”, IEEE Transactions on Neural Networks, 18, (2007),1614-1627. |
Benggio, Y et al., “Curriculum learning”, ICML 09 Proceedings of the 26th Annual International Conference on Machine Learning, New York, NY: ACM, (2009). |
Benggio, Y et al., “Scaling learning algorithms towards AI. In L. a Bottou”, Large Scale Kernel Machines, MIT Press,(2007). |
Battiato, S et al., “Exposure correction for imaging devices: An overview”, In R. Lukac (Ed.), Single Sensor Imaging Methods and Applications for Digital Cameras, CRC Press,(2009),323-350. |
U.S. Appl. No. 12/028,704, filed Feb. 2, 2008, Method and System for Vision-Based Interaction in a Virtual Environment. |
U.S. Appl. No. 13/405,319, filed Feb. 26, 2012, Method and System for Vision-Based Interaction in a Virtual Environment. |
U.S. Appl. No. 13/411,657, filed Mar. 5, 2012, Method and System for Vision-Based Interaction in a Virtual Environment. |
U.S. Appl. No. 13/429,437, filed Mar. 25, 2012, Method and System for Vision-Based Interaction in a Virtual Environment. |
U.S. Appl. No. 13/562,351, filed Jul. 31, 2012, Method and System for Tracking of a Subject. |
U.S. Appl. No. 13/596,093, filed Aug. 28, 2012 Method and Apparatus for Three Dimensional Interaction of a Subject. |
U.S. Appl. No. 11/567,888, filed Dec. 7, 2006, Three-Dimensional Virtual-Touch Human-Machine Interface System and Method Therefor. |
U.S. Appl. No. 13/572,721, filed Aug. 13, 2012, Method and System for Three-Dimensional Virtual-Touch Interface. |
U.S. Appl. No. 12/784,123, filed Mar. 20, 2010, Gesture Recognition Systems and Related Methods. |
U.S. Appl. No. 12/784,022, filed May 20, 2010, Systems and Related Methods for Three Dimensional Gesture Recognition in Vehicles. |
U.S. Appl. No. 13/025,038, filed Feb. 10, 2011, Method and Apparatus for Performing Segmentation of an Image. |
U.S. Appl. No. 13/025,055, filed Feb. 10, 2011, Method and Apparatus for Disparity Computation in Stereo Images. |
U.S. Appl. No. 13/025,070, filed Feb. 10, 2011, Method and Apparatus for Determining Disparity of Texture. |
U.S. Appl. No. 13/221,903, filed Aug. 31, 2011, Method and Apparatus for Confusion Learning. |
U.S. Appl. No. 13/189,517, filed Jul. 24, 2011, Near-Touch Interaction with a Stereo Camera Grid and Structured. |
U.S. Appl. No. 13/297,029, filed Nov. 15, 2011, Method and Apparatus for Fast Computational Stereo. |
U.S. Appl. No. 13/297,144, filed Nov. 15, 2011, Method and Apparatus for Fast Computational Stereo. |
U.S. Appl. No. 13/294,481, filed Nov. 11, 2011, Method and Apparatus for Enhanced Stereo Vision. |
U.S. Appl. No. 13/316,606, filed Dec. 12, 2011, Method and Apparatus for Enhanced Stereo Vision. |
Number | Date | Country | |
---|---|---|---|
20140267612 A1 | Sep 2014 | US |