This invention relates generally to stereo imaging and more particularly to a new Graphic Processing Unit (GPU)-based stereo algorithm that addresses many difficulties typically associated with stereo imaging. The algorithm is considered as belonging to a family of surface-disparity algorithms, where the scene is treated as a series of slowly-varying surfaces, with homogenized color/texture. The algorithm also preferably includes a real-time component, dubbed residual stereo compute, which addresses real-time constraints and the minimization of compute load in real-time, by only analyzing changes in the image, as opposed to the entire image. Finally, the algorithm also preferably addresses another chronic problem in stereo imaging, aliasing due to texture regions.
With the ascent of new parallel computing platforms, such as the use of GPUs, as presented in NVIDIA: CUDA compute unified device architecture, prog. guide, version 1.1, 2007 and various accelerated processing units (APUs), real-time high-quality stereo imaging has become increasingly feasible. GPUs are comprised of a number of threaded Streaming multiprocessors (SMs), each of which is, in turn, comprised of a number of streaming processors (SPs), with example architectures presented in David Kirk and Wen-Mei W. Hwu, Programming Massively Parallel Processors A Hands-on Approach.: Elsevier, 2010.
The human visual system is very hierarchical, and visual recognition is performed in layers, first by recognizing the most basic features of an image, and then recognizing higher-level combinations of those features. This process continues until the brain recognizes an adequately high-level representation of the visual input.
There are many different approaches to stereo imaging. in accordance with the present invention, segment-based approaches will be mainly utilized, and may also be referred to as surface stereo. This is because segment-based approaches best resemble the human visual system. Such algorithms are ones in which the 3D field-of-view is treated as a set of smooth, slowly varying surfaces as set forth in Michael Bleyer, Carsten Rother, and Pushmeet Kohli, “Surface Stereo with Soft Segmentation,” in Computer Vision and Pattern Recognition, 2010. Segment-based approaches have emerged in recent years as an alternative to many region-based and pixel-based approaches and have outperformed in accuracy on the Middlebury dataset almost any other algorithm. The Middlebury set is widely considered the reference dataset and metric for stereo/disparity computation algorithms as set forth in (2010) Middlebury Stereo Vision Page. [Online at vision.middlebury.edu/stereo/].
There are many reasons why such methods today represent the more dominant approaches in stereo imaging, see Andreas Klaus, Mario Sormann, and Konrad Karner, “Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure,” in Proceedings of ICPR 2006, 2006, pp. 15-18. Segment-based approaches address semi-occlusions very well. They are also more robust to local changes. Other pixel and region-based approaches blur edges, causing ambiguity between background and foreground regions, as well as potentially removing smaller objects, as noted in Ines Ernst and Heiko Hirschmuller, “Mutual Information based Semi-Global Stereo Matching on the GPU,” in Lecture Notes in Computer Science, vol. 5358, 2008, pp. 228-239. A cross-based local approach as set forth in Jiangbo Lu, Ke Zhang, Gauthier Lafruit, and Francky Catthoor, “REAL-TIME STEREO MATCHING: A CROSS-BASED LOCAL APPROACH,” in 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, 2009 represents an implementation of such approaches on the GPU, but is still impractical because it exhibits weaknesses at regions of high texture and regions with abrupt changes in color/intensity. However, many segment-based approaches are therefore tedious, inaccurate and require a significant amount of computation, even on the GPU.
Therefore, it would be beneficial to provide an improved segment-based approach that overcomes the drawbacks of the prior art.
Therefore, in accordance with various embodiments of the present invention, an inventive system and method has been developed for disparity estimation in stereo images associated with current stereo algorithms. The implementation of a preferred embodiment of the invention may utilize a GPU or other dedicated processing unit, allowing for significant improvements in performance, both in accuracy and efficiency, but may be employed on any appropriate computing platform.
In accordance with the various embodiments of the present invention, a novel surface/segment-based approach for computing disparity estimation is provided, a real-time approach to computing stereo on the residual image as opposed to the entire image is described, and a means for addressing textured regions, which has been a major drawback of previous stereo algorithms, is finally presented.
Therefore, in accordance with various embodiments of the present invention, the following will be presented:
Implementation may preferably be on a Graphical Processing Unit (GPU), with comparisons being highlighted with existing methods and other prevalent algorithms for disparity computation. The inventive approach provides a significant improvement over existing depth estimation algorithms. Its preferred GPU-based implementation presents a number of novel interpretations of former algorithms, as well as realizations of new algorithms, ranging from texture segmentation, to disparity computation.
Still other objects and advantages of the invention will in part be obvious and will in part be apparent from the specification and drawings.
The invention accordingly comprises the several steps and the relation of one or more of such steps with respect to each of the others, and the apparatus embodying features of construction, combinations of elements and arrangement of parts that are adapted to affect such steps, all as exemplified in the following detailed disclosure, and the scope of the invention will be indicated in the claims.
For a more complete understanding of the invention, reference is made to the following description and accompanying drawings, in which:
One or more embodiments of the invention will now be described, making reference to the following drawings in which like reference numbers indicate like structure between the drawings.
In accordance with a particular embodiment of the present invention,
As is discussed in M. Marszalek and C. Schmid, “Semantic hierarchies for visual object recognition,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR '07, MN, 2007, pp. 1-7, a 3D field-of-view is preferably defined as an ensemble of smooth, well-behaved surfaces, varying slowly over space and time. Constancy in registration lends itself conceptually to spatio-temporal constancy, and allows some simplifying assumptions to be made for scene segmentation, including segment-based robustness as well as integrity across frames. Specifically, if a segment in a reference image is accurately matched to its counterpart in a slave image, then estimating the associated disparity value becomes significantly simplified. Good segmentation, however, poses many challenges, including context, shading, gradient changes, etc. Also, correctly matching the segment is no trivial task. Once a segment is robustly tracked in one image, a real challenge arises in trying to match the tracked segment to a corresponding segment in a second image To ensure a level of constancy, segmentation has to also be maintained over a number of frames (time), so that regions are properly tracked. This simplifies the process of firstly computing the correct disparity, and then tracking the segment's disparity across frames.
The Relevance of Segmentation to Disparity Computation—Segmentation, as part of disparity computation, is very crucial, as it allows for a hierarchical approach to the scene and field-of-view (FOV) organization. This approach has been attempted before, see in David A. Forsyth and Jean Ponce, “Stereopsis,” in Computer Vision A Modern Approach.: Prentice Hall, 2003 and in M. Marszalek and C. Schmid, “Semantic hierarchies for visual object recognition,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR '07, MN, 2007, pp. 1-7. Two properties, relevant to segments, which are carried across scenes are very important: cluster (or segment) number, and the segment's associated first and second-order moments. The fundamental idea is to track such segments across time (consecutive frames) and across space (reference and left frames). The two properties together comprise spatio-temporal constancy.
Governing principles of disparity generation—Initially, a simplifying assumption may be made that entire segments share a common disparity, again borrowing aspects of the surface-based approach noted in Marszalek noted above. This concept will be utilized to iteratively refine a disparity estimate of what will be considered sub-segments of the actual segment. Rigid segmentation may lead to local inaccuracies, and may fundamentally change depth calculation. The inventive approach in accordance with various embodiments of the present invention assumes that an initial disparity estimate provides for a coarse disparity calculation of a given segment. This disparity measurement may be allocated to all pixels in the segment, including ones that are semi-occluded (not visible by one of the two sensors).
Spatio-Temporal Segmentation and Tracking
One of the main advantages of attempting segmentation employing a GPU or other appropriate processing platform is the ability for Single-Instruction-Multiple Data (SIMD) operations to simultaneously update different regions in an image. This significantly simplifies a set of rules being used during a region-growing phase of segmentation.
The Image as a Cluster Map—Clustering may include one or more aspects as set forth in U.S. patent application Ser. No. 12/784,123, filed May 20, 2010, titled “Gesture Recognition Systems and Related Methods”, currently pending, the contents thereof being incorporated herein by reference. Further refinements of this basic system are set forth herein. Therefore, in accordance with embodiments of the present invention, the following processes may be employed.
Every pixel in a frame, i, presented as pi(x,y), is assigned a cluster number, ci(x,y), such that:
ci(x,y)=x·y−y mod x Equation 1
In Equation 1, clusters are sequentially numbered left-to-right, and top-to-bottom, such that:
ci(x,y)=ci(x,y−ε)+ε, Equation 2
where ε is an integer number. At this stage in the segmentation algorithm, pixels may begin to be connected, such that two pixels pi(x,y) and pi(x+1,y) are connected if:
Comprising the three channels, r, g, and b respectively, for a given frame i. A similar implementation has also been developed for HSV-based segmentation. HSV offers a number of perceptually driven properties for segmentation, which may be useful, especially for mitigating issues with shading. This connectivity assigns to the pixels a cluster number that is lowest in its neighborhood. As mentioned earlier, priority is set in descending order, from left to right and from top to bottom. Recall the assumption that all pixels belonging to the same surface are at the same disparity. Surface discontinuities can either refer to a depth discontinuity or a color/texture discontinuity.
Implementation in CUDA—In accordance with the various embodiments of the invention, a key pixel for each cluster is assigned, such that a cluster is defined by that key pixel. Since the implementation is multi-threaded, each pixel in a cluster therefore points to the same pixel to simplify and homogenized the process of tracking clustered/segmented regions. By pointing to the key pixel, clusters preferably have two attributes: 1) a cluster number that is associated with that key pixel, and 2) a stopping criterion, beyond which the cluster is terminated (see implementation above.) The assumption that all pixels belonging to the same surface are at the same disparity value is easily violated without an iterative approach. Specifically, such surfaces have to be further segmented into regions, for more accurate disparity calculation. Surface discontinuities, among other things, can refer to depth discontinuity or a color/texture discontinuity.
Semi-occlusions, Pixels having more than One Disparity Value—Semi-occluded pixels are described in Dongbo Min and Kwanghoon Sohn, “Cost Aggregation and Occlusion Handling With WLS in Stereo Matching,” IEEE Transactions on Image Processing, vol. 17, pp. 1431-1442, 2008, and Vladimir Kolmogorov and Ramin Zabih, “Computing Visual Correspondence with Occlusions via Graph Cuts,” in International Conference on Computer Vision, 2001, as pixels that are visible in only one image (from one sensor), but not from the other. Typically, stereo algorithms have difficulty handling semi-occlusions. In accordance with the present invention semi-occlusions are defined as locations that contain more than one disparity value (a left and right-image disparity value), since there is a foreground and a background value associated with the pixel location. It has been determined by the inventors of the present invention that it is very appropriate in such a case to address semi-occlusions with an inspiration from Gestalt—looking at it from a scene organization standpoint. Specifically, if a pixel belongs to a segment, that pixel will take on the segment's disparity value, even if its actual location is semi-occluded in the reference image. This ultimately means that there are two, not one, disparity estimates for semi-occlusions. The one that is relevant to the segment is chosen. A simple example that is inspired by biology, in which our visual system similarly compensates for semi-occlusions, is illustrated by looking at objects that are only partially occluded, and yet our eyes are capable of reconstructing the entire object at the correct depth. A whole school of psychology has evolved around this principle, known as Gestalt theory, in which our brains are described as having the capacity to reconstruct complex forms, from simpler, and sometimes incomplete, constituents, as further described in Vladimir Kolmogorov, Ramin Zabih, and Steven Gortler, “Generalized Multi-camera Scene Reconstruction Using Graph Cuts,” in Proceedings of the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, 2003.
Slanted Regions-Iterative Segmentation—For spatially-dominant regions, spanning large areas, in accordance with embodiments of the invention, an iteration may be used in which the segment itself is broken up across both the rows and columns, in essence, further segmenting the clusters into smaller ones, and re-evaluating these smaller clusters. By enforcing over-segmentation in the disparity space, a more refined disparity estimate may be obtained. Segmenting clusters across rows provides for the vertical tilt (or slant) of a cluster, while segmenting across the columns provides for the horizontal tilt (or slant) of a given cluster.
Refining the iteration—Given that iterations may be executed in both the vertical and horizontal directions, these iterations can be combined together to produce one result including both iterations. One possibility is to average both iterations, such that the new estimated disparity is given by Equation 4:
Segmenting Texture—One of the fundamental drawbacks and arguments against stereo imaging has been the performance in regions that are highly-textured. Such regions may contribute to false positives in the disparity computation, since many similarity metrics will confuse regions which look spatially similar. In accordance with embodiments of the present invention, texture segmentation may be approached generatively, in which a texture primitive is in itself comprised of a number of color primitives. First, a look at alternative approaches in texture segmentation is presented to see how they may affect stereo/disparity computation.
Texture, described extensively in the literature, (see Wolfgang Metzger, Laws of Seeing.: MIT Press, 2006.), is comprised of a fundamental building element, called a texton, or texture primitive. Extracting texture primitives from images is a challenging task and has been the subject of on-going research. Some approaches utilize a Gabor (or wavelet) multi-scale filter bank to decompose a texture into its primitives, establishing spatial periodicity and extracting a texture primitive that can be explicitly expressed. For example, in Metzger, dominant spatial texture orientations of a grayscale version of an image are extracted, and multiscale frequency decomposition is attempted. This is accomplished with a steerable pyramid decomposition process as set forth in Junging Chen, Pappas T. N., Mojsilovic A., and Rogowitz B. E., “Adaptive perceptual color-texture image segmentation,” IEEE Transactions on Image Processing, vol. 4, no. 0, pp. 1524-1536, October 2005, in which frequency decomposition is accomplished with four orientation subbands: horizontal, vertical, and the two diagonals, is shown in
Computationally, these are very expensive methods, and they make many unrealistic assumptions about knowing a priori dominant orientations which are associated with a texture primitive. Furthermore, such approaches suffer in the presence of a dominant gradient which is associated with the texture, i.e. if the texture has a gradient component that spatially and/or temporally varies. Such approaches negatively impact disparity/stereo computation for two reasons: 1) as mentioned above, regions with similar responses to texture decomposition may behave similarly to a region, or pixel-based similarity metric, and 2) texture segmentation and classification may be computationally prohibitive when combined with stereo, and the rest of the algorithm.
Approaching Texture Segmentation Generatively—Therefore, in accordance with various embodiments of the present invention, a different approach to texture segmentation is preferably employed that is very useful for disparity computation, and adopts the concept of “emergence” from the Gestalt school of psychology, see Vladimir Kolmogorov, Ramin Zabih, and Steven Gortler, “Generalized Multi-camera Scene Reconstruction Using Graph Cuts, in Proceedings of the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, 2003. In accordance with embodiments of the invention, a texton, or texture primitive may be viewed as a set of color primitives, combined together to produce a perceptually consistent spatially periodic or slowly-varying texture primitive, which, in itself, comprises texture. These color primitives define and comprise the texture spatial primitives. Many techniques have been developed to evaluate texture. In accordance with embodiments of the invention, texture is reviewed in the sense of interleaving a series of color primitives through slowly varying them over time. This is a generative approach, in which it is determined how a particular texture is formed in the first place, and then the texture is represented as a weighted linear combination of a color primitive, as well as its gradient that is associated with that color primitive. Pixels belonging to the same color primitive may be clustered independently, such that, for any given texture, a texture primitive is defined as a linear combination of clusters, comprised of these color primitives. A texture primitive, T, over a window, W, may be given by:
Tx,y=Tx+ε
Where ε1 and ε2 are values which represent periodicity in the texture primitive.
Also, T is represented by:
Tx,y=C0(x,y)+C1(x,y)+ . . . +CN−1(x,y)
To successfully compute disparity for textured regions, the inventive approach implicitly mitigates texture by looking at variations in both scale and intensity, as gradual changes in themselves. Together, changes in color, intensity, and lighting, make extracting a texture primitive quite challenging, mainly due to the permutation of combinations that is perceptually easy to identify, yet very difficult to describe in closed form. Instead, the inventive approach is less concerned with defining texture primitives and more concerned with generatively reproducing texture primitives from more fundamental spatially varying primitives. Such primitives constitute interleaving segments that constitute a perceptually-visible texture. This concept extends to address primitives that are disjoint, such as those that may include, for example, a checkerboard pattern as shown in
To summarize, the underlying concepts governing texture segmentation include 1) Texture color primitives can link up correctly even in the presence of a gradient associated with them, to form a consistent segment; and 2) Interleaving two or more such segments together constitutes texture. Segmentation Rules: as set forth in the Ser. No. 12/784,123 reference noted above, may include 1) the concept of spatio-temporal constancy; 2) the lowest cluster number rule; and 3) that segments ultimately live and die in the field-of-view.
Residual Image Compute-Stereo Codec
This is a computationally expensive algorithm that can be significantly improved for real-time performance in accordance with various embodiments of the invention, in which computation is only performed on the changes in the image, referred to as the residual image. The overall approach requires a similar one to typical video codec encoding schemes, such as the H.264 encoding standard as presented in W. T. Freeman and E. Adelson, “The design and use of steerable filters,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, pp. 891-906, September 1991 or Thomas Wiegand, Gary J. Sullivan, Gisle Bjøntegaard, and Ajay Luthra, “Overview of the H.264/AVC Video Coding Standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 560-576, July 2003. Such standards have emerged as the new compression standard associated with video codecs. H.264 has flexibility, as a video compression standard, which Metzger allows for real-time gains in bandwidth. H.264 enables this great reduction by exploiting both intra-frame and inter-frame redundancies and reducing the overall compute load required to represent an image.
Since changes in a video stream are gradual, an image is mostly unchanged in real-time, transitioning from one frame to another. It is logical to assume in accordance with embodiments of the invention that the FLOPS/computational load would be diminished in real-time, since relatively very few pixels change between frames. As such, a similar method is adopted in accordance with embodiments of the present invention where a reference image I is utilized, and only the residual information between consecutive images is preferably processed in real-time. In H.264, an image is segmented into blocks of non-uniform size. This concept is transferred over in accordance with embodiments of the present invention, utilizing an existing segmented cluster map instead. However, a more complex memory architecture may also be employed, comprised of a reference image, a cluster map, a segmented reference image, and a depth map. In accordance with embodiments of the present invention, a reference memory architecture may be utilized comprised of all these images, and a residual architecture comprised of the difference between these reference images and subsequent images in a video stream. As is shown in
In accordance with the present invention, a process for computing the various required valued resembles that suggested for the H.264 and other video encoding schemes. Thus, two main sections are presented:
The term temporally stable is preferably used to describe segments which have very little change in their overall statistics and depth values. In the section of the algorithm set forth in
In accordance with various embodiments of the invention, during the process, starting with a reference, Ic cluster map (see
Associated Pixel Flags and Pixel Memory Architecture—As is next depicted in
Segment Stability: denotes the temporal stability of the segment that the current pixel is associated with.
Cluster ID Number: denotes the cluster number associated with a the current pixel.
Cluster Affinity: Denotes the relationship of the cluster with a key pixel in another cluster.
Key Pixel: denotes whether the current pixel is a key pixel of a cluster or segment.
Temporal Stability: denotes the temporal stability of the current pixel.
Edge: denotes whether the current pixel is an edge pixel.
Edge: another flag for an edge pixel.
!Background: denotes whether the current pixel is not a background pixel.
Note that pixels are background pixels, temporally stable pixels, or temporally unstable pixels.
Segmentation in Real-time—Under real-time conditions, segmentation is only attempted on any temporally unstable residual pixels. Clustering follows that step, such that pixels with common color and spatial proximity are clustered together. Agglomeratively connecting such clusters to previously existing segments is then performed. At this point, any clusters/segments whose size/statistics have been significantly changed undergo disparity decomposition. Such segments are considered unstable, and will have their disparity recomputed. Otherwise, the remaining, stable, segments do not contribute to the residual data, and are part of the reconstructed frame. Disparity decomposition is not performed on temporally stable segments, and instead, a composite depth map is formed, comprised of temporally stable (and thus already pre-computed segments) as well as the computed segments.
Such an approach to real-time segmentation and depth calculation requires a memory architecture in global memory to support this approach. The architecture proposed in accordance with various embodiments of the present invention contains both reference as well as residual portions, again in a manner similar to typical video codecs such as H.264. This architecture is then utilized at different steps in the analysis to significantly reduce compute load and enhance accuracy.
Thus the following set of guidelines governing segmentation and disparity computation in real-time are realized. If there is no change in the image, the overall FLOPS count should drop very dramatically. Any segment that has significantly changed is deemed temporally unstable and will have its disparity recalculated. The depth map is a composite map comprised of temporally stable segments as well as temporally unstable ones
Implementation in CUDA—
To implement this approach, the inventive architecture accounts for the number of flags noted above that define the stability of the pixels, as well as their relationship to the corresponding clusters and segments. A pixel's cluster number is then assessed; the pixel either receives a new cluster number (cluster number is elevated) if it is temporally unstable, or maintains the same cluster number that is associated with a segment. This enables changes at the pixel level to affect the segment. Conversely, if the segment is found to be temporally stable, all pixels belonging to that segment will have their temporal stability flag updated accordingly, even if the pixels are temporally unstable. Thus, to deem a pixel temporally unstable requires that both the segment stability flag and the pixel stability flag be enabled. Therefore, if a segment is deemed temporally unstable, every pixel in that segment is deemed unstable in spite of the fact that some of them might have been deemed stable earlier in the algorithm.
Computing Residual Statistics—The overall concept of residual-only compute lends itself to one or more statistics calculations for the cluster map, an essential step in segmentation. Because residual statistics are maintained on the block level, a first step utilizes block statistics accumulators, which must be merged after a final refinement iteration. Subsequent stages will occur during residual segmentation to compute statistics solely on temporally unstable pixels. Computing the statistics of a set of data is inherently a serial operation. Many GPU-based implementations associated with statistics are presented in Ian E. G. Richardson, H.264/MPEG-4 Part 10 White Paper, 2003, where a merge-sort implementation is presented. Eric Sintron and Ulf Assarson, “Fast Parallel GPU-Sorting Using a Hybrid Algorithm,” Journal of Parallel and Distributed Computing, vol. 68, no. 10, pp. 1381-1388, October 2008. Another approach, presented in Sintron and Assarson, utilizes a linked list prefix computations, implemented on GPUs. In Zheng Wei and Joseph Jaja, “Optimization of Linked List Prefix Computations on Multithreaded GPUs Using CUDA,” in 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), Atlanta, 2010, a parallel search is presented however, they are mostly aimed at a homogeneous data set, with the idea of starting with a large data set, and then condensing, or reducing the data set with intermediate statistics, until the final statistics are calculated. Other techniques include stream compaction Tim Kaldewey, Jeff Hagen, Andrea Di Blas, and Eric Sedlar, “Parallel Search On Video Cards,” in First USINIX Workshop on Hot Topics in Parallelism (HotPar '09), 2009, and scan/scatter algorithms Shubhabrata Sengupta, Mark Harris, Yao Zhang, and John D. Owens, “Scan Primitives for GPU Computing,” in Proceedings of the 2007 Graphics Hardware Conference, San Diego, CA, 2007, pp. 97-106. Any of these techniques can work for sorting through the clusters and compacting the data into a few segments. In accordance with the invention, support for sort/compaction across multiple portions of the inventive algorithm have been provided, since statistics computation is very critical for real-time implementation. In such a case, a different approach is taken from that presented in the prior art. The compute load that is associated with the statistics is preferably distributed so that a parallel implementation becomes feasible. Note that if APUs are available, an alternative approach could be used in which an X-86 processor (for instance), sharing memory with a number of ALUs, may perform the intermediate calculations. The performance would even be enhanced further in that case because a shared memory CPU does provide an enhanced ability in handling all the relevant serial operations.
Statistics Implementation—Statistics are first accumulated on the block level into two separate buffers, as is displayed in
Because the initial cluster numbers are constrained by the spatial boundaries of the block, the initialization kernel will merge the spatial statistics on the block level. The integration of statistics components is integrated into, both, the linking and refinement steps, illustrated in
Disparity Decomposition—The goal of disparity computation is to decompose the image into a series of surfaces that are present at different disparity values. The right, reference image IR is subtracted from a shifted version of the left image. However, instead of looking at per-pixel metrics, such as described in accordance with the prior art noted above, in accordance with an embodiment of the invention utilizes a segment-based disparity estimate, and tries to represent the best disparity value that is associated with a given segment.
In accordance with this embodiment of the invention, a left image I1 is shifted one pixel at a time, while subtraction between left and right images is performed with the shifted versions of the left image. Every shift then represents a new disparity. The resulting set of difference images then constitutes a disparity decomposition of every segment in the image. Any zero-pixels represent regions (or segments) in the image that are candidates for the correct disparity. The computation of a difference image is presented as:
For any given segment Si, such that Si⊆S,
For a given disparity, di,S:
Where d denotes the current shift, and τc is the threshold that is associated with a current color channel.
So, Ñ⊆N
For any given segment, disparity decomposition is a means of reducing the candidate disparity values that can be associated with the segment (based on a similarity metric). The goal is to determine a disparity that best matches a given segment. A similarity metric is used in which the total number of overlapping pixels can zero out regions in the segment during the subtraction phase of disparity decomposition. For a given disparity, the more pixels in the segment that are zeroed out, the closer the segment is to the correct disparity. The similarity metric for estimating segment disparity is given by:
Where {tilde over (D)}i,S represents the disparity estimate for a given segment S, and {tilde over (S)}i(x,y,D) represents the portion of the segment that is present at step D, in a given disparity decomposition step. Every segment in the image can be considered a sequence that is comprised of the reference image's pixels, such that {sn}n=1N where N is the size of the segment in the reference image. For a given disparity, we can define a sub-sequence {{tilde over (s)}n}n=1Ñ
The goal is to have sn and {tilde over (s)}n overlap nearly entirely at one of the candidate disparity values. The appropriate disparity is estimated as one that maximizes the ratio of the subsequence, relative to its sequence. Equation 7 above can actually be represented as:
If, on the other hand, the inquiry at step 1630 is answered in the negative, and it is therefore determined that a stability condition does not exist, processing then continues at step 1640 where differences between left and right images are computed at each disparity level. Next, at step 1650, a cluster counter is incremented if the difference computed in step 1640 is below a predetermined threshold determined empirically. Then, at step 660, these difference results are stored in a compressed form, comprising (1-bit/disparity). So, for 32 disparity values, a 32-bit image is used. For 64 disparity values, a 64-bit image is used, and so on.). Finally, processing then passes to step 1660 as described above.
Violating the Subsequence Criterion—The rule, of looking at similarity as an overlap ratio, based on a number of spatio-temporal criteria, presented above can be violated in a number of special cases. For instance, this can happen if a smaller segment undergoing disparity decomposition overlapped a much larger segment with similar spatial characteristics. For such a case, penalizing the non-overlapped region would present one means of mitigating such a problem. Another case can occur if such a segment belongs to a textured pattern, occurring at a spatially periodic setting. In this case, agglomeratively growing the regions (see earlier section on segmentation) would present a possible solution. Then, textured regions would cluster together before disparity decomposition occurs (see section below on disparity estimation of textured regions).
Implementation in CUDA—To accomplish disparity decomposition in accordance with the various embodiments of the invention, an efficient shift-difference kernel may be implemented, as will be shown below. Each block may read one row of source image data to shared memory. As a result, the difference between left and right image pixels can be computed for several disparities at once because all of the necessary data is available in shared memory. Because only one bit is necessary to store the result of the threshold on the difference result, the results can be compressed into a single 32-bit image in which each pixel contains the results from all disparities. This inventive method not only reduces storage requirements, but also reduces the number of required read/write operations on global memory. An extension of this method can be used to encode 64 bits or 128-bit disparity differences, stored in a 64 or 128-bit image.
The block may be preferably organized in three dimensions with the z-dimension representing the disparity. To maximize performance, the size of shared memory may preferably be kept under 4 KB to allow up to four blocks to be swapped out per streaming multiprocessor (SM). Depending on the GPU (or APU), shared memory may differ. In a particular exemplary implementation, the size of the z-dimension will be set to eight and each thread will calculate the difference for four disparity values in a loop if the maximum disparity is set to 32. The results of the differencing operation are stored in shared memory at the end of each iteration, per the description of disparity decomposition that was presented earlier. After all iterations have been executed, the final 32-bit values will be written to Global Memory, as a disparity image ID(x,y). For larger images, the size of the z-dimension may be reduced to maintain the size of the shared memory used per block.
In addition to computing the difference between the source images, these intersection pixels are preferably counted with respect to the clusters in the left and right images to determine how much of each cluster in one image intersects with clusters in the other image.
A system in which the algorithm of
The buffer containing the cluster numbers in the correct order will be copied to texture memory so it can quickly be accessed by all thread blocks.
Composite Disparity Real-time Analysis—In real-time, the pixel architecture is again utilized, such that preferably, only temporally unstable segments have their disparity computed. As such disparity decomposition is reduced to segment-disparity decomposition. The result is a composite disparity image, comprised of already temporally stable segments and the newly computed/merged temporally unstable segments, on which disparity decomposition has been attempted.
Disparity Estimation of Textured Regions—Texture Disparity “Emergence”—The inventive segmentation algorithm is capable of agglomeratively adjoining spatially disjoint regions through an inter-cluster criterion. Growing such inter-cluster region thresholds gradually through kernel iterations allows for combining multiple smaller clusters into one larger, disjoint cluster. The advantage is an ability to segment textured regions, which are characterized by spatially periodic regions (see earlier section on texture). Although these regions are clearly disjoint, they are actually easily assembled together after disparity decomposition. The best disparity estimates will solidly remain associated with the correct cluster. This is accomplished through an emergence of a texture pattern from its constituent primitives after disparity computation.
The concept of emergence in texture segmentation and the subsequent disparity computation is consistent with Gestalt psychology, or Gestaltism. In Gestalt theory, the brain is holistic, with self-organizing features, which, when combined together form more complex objects. As such, many Gestalt theorists argue that objects emerge from their constituent parts, hence the concept of emergence, presented by Metzger, noted above. In the inventive implementation of texture segmentation, a similar approach is adapted. The inventive approach to texture segmentation is Gestalt-inspired, and allows for this emergence of an object, such as those highlighted in
Both figures highlight a highly-textured region where conventional region-based and pixel-based disparity computation techniques perform poorly. In both images, a checkerboard sequence is estimated at the correct disparity. This is because the checkerboard sequence is really comprised of two objects representing all the white squares and all the black ones. Segmentation is first accomplished on the back and white squares separately. These squares are then agglomeratively adjoined to create larger white and black clusters. The disparities of each of the two objects is then computed. A depth-based clustering step is utilized to segment a foreground object that is highly textured, comprised of the entire checkerboard sequence.
Therefore, in accordance with various embodiments of the present invention, an algorithm for disparity computation that preferably runs on a GPU and other appropriate platforms, and performs very well in real-time conditions is presented. A residual compute portion of the algorithm reduces the FLOPs count dramatically, by exploiting a residual architectural component, and creating composite intermediate images for the segmentation as well as disparity computation. Texture is mitigated with a Gestalt-inspired technique that emphasizes the emergence of texture at the correct disparity by correctly estimating the disparities of its constituents. Examples of mitigating other chronic issues associated with region and pixel-based techniques have also been shown.
It will thus be seen that the objects set forth above, among those made apparent from the preceding description, are efficiently attained and, because certain changes may be made in carrying out the above method and in the construction(s) set forth without departing from the spirit and scope of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.
It is also to be understood that this description is intended to cover all of the generic and specific features of the invention herein described and all statements of the scope of the invention which, as a matter of language, might be said to fall there between.
This application is continuation of U.S. application Ser. No. 16/812,346 filed Mar. 8, 2020 to El Dokor et al., titled “Apparatus and Method for Segmenting an Image”, currently pending, which is a continuation of U.S. application Ser. No. 14/148,761 filed Jan. 7, 2014 to El Dokor et al., titled “Apparatus and Method for Segmenting an Image”, now U.S. Pat. No. 10,586,334, which is a divisional of U.S. application Ser. No. 13/025,038 filed Feb. 10, 2011 to El Dokor et al, titled “Method and Apparatus for Performing Segmentation of an Image”, now U.S. Pat. No. 8,665,093 which in turn claims the benefit of U.S. Provisional Patent Application Ser. No. 61/379,706 filed Sep. 2, 2010 to El Dokor et al., titled “Imaging”, the contents of each of these applications being incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
5454043 | Freeman | Sep 1995 | A |
5544050 | Abe et al. | Aug 1996 | A |
5581276 | Cipolla et al. | Dec 1996 | A |
5594469 | Freeman et al. | Jan 1997 | A |
5699441 | Sagawa et al. | Dec 1997 | A |
5767842 | Korth | Jun 1998 | A |
5887069 | Sakou et al. | Mar 1999 | A |
5990865 | Gard | Nov 1999 | A |
6002808 | Freeman | Dec 1999 | A |
6072494 | Nguyen | Jun 2000 | A |
6075895 | Qiao et al. | Jun 2000 | A |
6115482 | Sears et al. | Sep 2000 | A |
6128003 | Smith et al. | Oct 2000 | A |
6141434 | Christian et al. | Oct 2000 | A |
6147678 | Kumar et al. | Nov 2000 | A |
6181343 | Lyons | Jan 2001 | B1 |
6195104 | Lyons | Feb 2001 | B1 |
6204852 | Kumar et al. | Mar 2001 | B1 |
6215890 | Matsuo et al. | Apr 2001 | B1 |
6222465 | Kumar et al. | Apr 2001 | B1 |
6240197 | Christian et al. | May 2001 | B1 |
6240198 | Rehg et al. | May 2001 | B1 |
6252598 | Segen | Jun 2001 | B1 |
6256033 | Nguyen | Jul 2001 | B1 |
6256400 | Takata et al. | Jul 2001 | B1 |
6269172 | Rehg et al. | Jul 2001 | B1 |
6323942 | Bamji | Nov 2001 | B1 |
6324453 | Breed et al. | Nov 2001 | B1 |
6360003 | Doi et al. | Mar 2002 | B1 |
6363160 | Bradski et al. | Mar 2002 | B1 |
6377238 | McPheters | Apr 2002 | B1 |
6389182 | Ihara et al. | May 2002 | B1 |
6394557 | Bradski | May 2002 | B2 |
6400830 | Christian et al. | Jun 2002 | B1 |
6434255 | Harakawa | Aug 2002 | B1 |
6442465 | Breed et al. | Aug 2002 | B2 |
6456728 | Doi et al. | Sep 2002 | B1 |
6478432 | Dyner | Nov 2002 | B1 |
6509707 | Yamashita et al. | Jan 2003 | B2 |
6512838 | Rafii et al. | Jan 2003 | B1 |
6526156 | Black et al. | Feb 2003 | B1 |
6553296 | Breed et al. | Apr 2003 | B2 |
6556708 | Christian et al. | Apr 2003 | B1 |
6571193 | Unuma et al. | May 2003 | B1 |
6590605 | Eichenlaub | Jul 2003 | B1 |
6600475 | Gutta et al. | Jul 2003 | B2 |
6608910 | Srinivasa et al. | Aug 2003 | B1 |
6614422 | Rafii et al. | Sep 2003 | B1 |
6624833 | Kumar et al. | Sep 2003 | B1 |
6658136 | Brumitt | Dec 2003 | B1 |
6674877 | Jojic et al. | Jan 2004 | B1 |
6674895 | Rafii et al. | Jan 2004 | B2 |
6678425 | Flores et al. | Jan 2004 | B1 |
6681031 | Cohen et al. | Jan 2004 | B2 |
6683968 | Pavlovic et al. | Jan 2004 | B1 |
6757571 | Toyama | Jun 2004 | B1 |
6766036 | Pryor | Jul 2004 | B1 |
6768486 | Szabo et al. | Jul 2004 | B1 |
6788809 | Grzeszczuk et al. | Sep 2004 | B1 |
6795567 | Cham et al. | Sep 2004 | B1 |
6801637 | Voronka et al. | Oct 2004 | B2 |
6804396 | Higaki et al. | Oct 2004 | B2 |
6829730 | Nadeau-Dostie et al. | Dec 2004 | B2 |
6847728 | Tao et al. | Jan 2005 | B2 |
6857746 | Dyner | Feb 2005 | B2 |
6901561 | Kirkpatrick et al. | May 2005 | B1 |
6937742 | Roberts et al. | Aug 2005 | B2 |
6940646 | Taniguchi et al. | Sep 2005 | B2 |
6944315 | Zipperer et al. | Sep 2005 | B1 |
6950534 | Cohen et al. | Sep 2005 | B2 |
6993462 | Pavlovic et al. | Jan 2006 | B1 |
7039676 | Day et al. | May 2006 | B1 |
7046232 | Inagaki et al. | May 2006 | B2 |
7050606 | Paul et al. | May 2006 | B2 |
7050624 | Dialameh et al. | May 2006 | B2 |
7058204 | Hildreth et al. | Jun 2006 | B2 |
7065230 | Yuasa et al. | Jun 2006 | B2 |
7068842 | Liang et al. | Jun 2006 | B2 |
7095401 | Liu et al. | Aug 2006 | B2 |
7102615 | Marks | Sep 2006 | B2 |
7129927 | Mattson | Oct 2006 | B2 |
7170492 | Bell | Jan 2007 | B2 |
7190811 | Ivanov | Mar 2007 | B2 |
7203340 | Gorodnichy | Apr 2007 | B2 |
7212663 | Tomasi | May 2007 | B2 |
7221779 | Kawakami et al. | May 2007 | B2 |
7224830 | Nefian et al. | May 2007 | B2 |
7224851 | Kinjo | May 2007 | B2 |
7233320 | Lapstun et al. | Jun 2007 | B1 |
7236611 | Roberts et al. | Jun 2007 | B2 |
7239718 | Park et al. | Jul 2007 | B2 |
7257237 | Luck et al. | Aug 2007 | B1 |
7274800 | Nefian et al. | Sep 2007 | B2 |
7274803 | Sharma et al. | Sep 2007 | B1 |
7289645 | Yamamoto et al. | Oct 2007 | B2 |
7295709 | Cootes et al. | Nov 2007 | B2 |
7296007 | Funge et al. | Nov 2007 | B1 |
7308112 | Fujimura et al. | Nov 2007 | B2 |
7340077 | Gokturk et al. | Mar 2008 | B2 |
7340078 | Shikano et al. | Mar 2008 | B2 |
7342485 | Joehl et al. | Mar 2008 | B2 |
7346192 | Yuasa et al. | Mar 2008 | B2 |
7348963 | Bell | Mar 2008 | B2 |
7359529 | Lee | Apr 2008 | B2 |
7372977 | Fujimura et al. | May 2008 | B2 |
7379563 | Shamaie | May 2008 | B2 |
7391409 | Zalewski et al. | Jun 2008 | B2 |
7394346 | Bodin | Jul 2008 | B2 |
7412077 | Li et al. | Aug 2008 | B2 |
7415126 | Breed et al. | Aug 2008 | B2 |
7415212 | Matsushita et al. | Aug 2008 | B2 |
7421093 | Hildreth et al. | Sep 2008 | B2 |
7423540 | Kisacanin | Sep 2008 | B2 |
7444001 | Roberts et al. | Oct 2008 | B2 |
7450736 | Yang et al. | Nov 2008 | B2 |
7460690 | Cohen et al. | Dec 2008 | B2 |
7477758 | Piirainen et al. | Jan 2009 | B2 |
7489308 | Blake et al. | Feb 2009 | B2 |
7489806 | Mohri et al. | Feb 2009 | B2 |
7499569 | Sato et al. | Mar 2009 | B2 |
7512262 | Criminisi et al. | Mar 2009 | B2 |
7519223 | Dehlin et al. | Apr 2009 | B2 |
7519537 | Rosenberg | Apr 2009 | B2 |
7551770 | Harman | Jun 2009 | B2 |
7574020 | Shamaie | Aug 2009 | B2 |
7590262 | Fujimura et al. | Sep 2009 | B2 |
7593552 | Higaki et al. | Sep 2009 | B2 |
7598942 | Underkoffler et al. | Oct 2009 | B2 |
7599547 | Sun et al. | Oct 2009 | B2 |
7606411 | Venetsky et al. | Oct 2009 | B2 |
7614019 | Rimas Ribikauskas et al. | Nov 2009 | B2 |
7620316 | Boillot | Nov 2009 | B2 |
7646372 | Marks et al. | Jan 2010 | B2 |
7660437 | Breed | Feb 2010 | B2 |
7665041 | Wilson et al. | Feb 2010 | B2 |
7676062 | Breed et al. | Mar 2010 | B2 |
7720282 | Blake et al. | May 2010 | B2 |
7721207 | Nilsson | May 2010 | B2 |
7804998 | Mundermann et al. | Sep 2010 | B2 |
20010001182 | Ito et al. | May 2001 | A1 |
20010030642 | Sullivan et al. | Oct 2001 | A1 |
20020012459 | Oh et al. | Jan 2002 | A1 |
20020041327 | Hildreth et al. | Apr 2002 | A1 |
20020061131 | Sawhney | May 2002 | A1 |
20020064382 | Hildreth et al. | May 2002 | A1 |
20020090133 | Kim et al. | Jul 2002 | A1 |
20020140633 | Rafii et al. | Oct 2002 | A1 |
20040183775 | Bell | Sep 2004 | A1 |
20040240725 | Xu | Dec 2004 | A1 |
20050002074 | McPheters et al. | Jan 2005 | A1 |
20050083314 | Shalit et al. | Apr 2005 | A1 |
20050105775 | Luo et al. | May 2005 | A1 |
20050190443 | Nam et al. | Sep 2005 | A1 |
20050286756 | Hong et al. | Dec 2005 | A1 |
20050286758 | Zitnick, III | Dec 2005 | A1 |
20060093186 | Ivanov | May 2006 | A1 |
20060101354 | Hashimoto et al. | May 2006 | A1 |
20060136846 | Im et al. | Jun 2006 | A1 |
20060139314 | Bell | Jun 2006 | A1 |
20060221072 | Se et al. | Oct 2006 | A1 |
20070055427 | Sun et al. | Mar 2007 | A1 |
20070086646 | Yang | Apr 2007 | A1 |
20070109300 | Li | May 2007 | A1 |
20070113207 | Gritton | May 2007 | A1 |
20070132721 | Glomski et al. | Jun 2007 | A1 |
20070195997 | Paul et al. | Aug 2007 | A1 |
20070263932 | Bernardin et al. | Nov 2007 | A1 |
20070280505 | Breed | Dec 2007 | A1 |
20080002878 | Meiyappan et al. | Jan 2008 | A1 |
20080005703 | Radivojevic et al. | Jan 2008 | A1 |
20080013793 | Hillis et al. | Jan 2008 | A1 |
20080037875 | Kim et al. | Feb 2008 | A1 |
20080052643 | Ike et al. | Feb 2008 | A1 |
20080059578 | Albertson et al. | Mar 2008 | A1 |
20080065291 | Breed | Mar 2008 | A1 |
20080069415 | Schildkraut et al. | Mar 2008 | A1 |
20080069437 | Baker | Mar 2008 | A1 |
20080104547 | Morita et al. | May 2008 | A1 |
20080107303 | Kim et al. | May 2008 | A1 |
20080120577 | Ma et al. | May 2008 | A1 |
20080178126 | Beeck et al. | Jul 2008 | A1 |
20080181459 | Martin et al. | Jul 2008 | A1 |
20080219501 | Matsumoto | Sep 2008 | A1 |
20080219502 | Shamaie | Sep 2008 | A1 |
20080225041 | El Dokor et al. | Sep 2008 | A1 |
20080229255 | Linjama et al. | Sep 2008 | A1 |
20080240502 | Freedman et al. | Oct 2008 | A1 |
20080244465 | Kongqiao et al. | Oct 2008 | A1 |
20080244468 | Nishihara et al. | Oct 2008 | A1 |
20080267449 | Dumas et al. | Oct 2008 | A1 |
20080282202 | Sunday | Nov 2008 | A1 |
20090006292 | Block | Jan 2009 | A1 |
20090027337 | Hildreth | Jan 2009 | A1 |
20090037849 | Immonen et al. | Feb 2009 | A1 |
20090040215 | Afzulpurkar et al. | Feb 2009 | A1 |
20090060268 | Roberts et al. | Mar 2009 | A1 |
20090074248 | Cohen et al. | Mar 2009 | A1 |
20090077504 | Bell et al. | Mar 2009 | A1 |
20090079813 | Hildreth | Mar 2009 | A1 |
20090080526 | Vasireddy et al. | Mar 2009 | A1 |
20090085864 | Kutliroff et al. | Apr 2009 | A1 |
20090102788 | Nishida et al. | Apr 2009 | A1 |
20090102800 | Keenan | Apr 2009 | A1 |
20090103780 | Nishihara et al. | Apr 2009 | A1 |
20090108649 | Kneller et al. | Apr 2009 | A1 |
20090109036 | Schalla et al. | Apr 2009 | A1 |
20090110292 | Fujimura et al. | Apr 2009 | A1 |
20090115721 | Aull et al. | May 2009 | A1 |
20090116742 | Nishihara | May 2009 | A1 |
20090116749 | Cristinacce et al. | May 2009 | A1 |
20090150160 | Mozer | Jun 2009 | A1 |
20090153366 | Im et al. | Jun 2009 | A1 |
20090153655 | Ike et al. | Jun 2009 | A1 |
20090180668 | Jones et al. | Jul 2009 | A1 |
20090183125 | Magal et al. | Jul 2009 | A1 |
20090183193 | Miller, IV | Jul 2009 | A1 |
20090189858 | Lev et al. | Jul 2009 | A1 |
20090208057 | Wilson et al. | Aug 2009 | A1 |
20090222149 | Murray et al. | Sep 2009 | A1 |
20090228841 | Hildreth | Sep 2009 | A1 |
20090231278 | St. Hilaire et al. | Sep 2009 | A1 |
20090244309 | Maison et al. | Oct 2009 | A1 |
20090249258 | Tang | Oct 2009 | A1 |
20090262986 | Cartey et al. | Oct 2009 | A1 |
20090268945 | Wilson et al. | Oct 2009 | A1 |
20090273563 | Pryor | Nov 2009 | A1 |
20090273574 | Pryor | Nov 2009 | A1 |
20090273575 | Pryor | Nov 2009 | A1 |
20090278915 | Kramer et al. | Nov 2009 | A1 |
20090295738 | Chiang | Dec 2009 | A1 |
20090296991 | Anzola | Dec 2009 | A1 |
20090315740 | Hildreth et al. | Dec 2009 | A1 |
20090316952 | Ferren et al. | Dec 2009 | A1 |
20110003241 | Ignatov et al. | Feb 2011 | A1 |
20110025827 | Shpunt | Feb 2011 | A1 |
20110081042 | Kim et al. | Apr 2011 | A1 |
Entry |
---|
Freeman, W. T. et al., “The Design and Use of Steerable Filters”, IEEE Transactions of Pattern Analysis and Machine Intelligence V. 13, (Sep. 1991), 891-906. |
Simoncelli, E.P. et al., “Shiftable Multi-scale Transforms”, IEEE Transactions on Information Theory V. 38, (Mar. 1992), 587-607. |
Simoncelli, E.P. et al., “The Steerable Pyramid: A Flexible Architecture for Multi-Scale Derivative Computation”, Proceedings of ICIP-95 V. 3, (Oct. 1995),444-447. |
Chen, J et al., “Adaptive Perceptual Color-Texture Image Segmentation”, IEEE Transactions on Image Processing, v. 14, No. 10, (Oct. 2005), 1524-1536 (2004 revised draft). |
Halfhill, Tom R., “Parallel Processing with CUDA”, Microprocessor Report, Available at http://www.nvidia.com/docs/IO/55972/220401_Reprint.pdf, (Jan. 28, 2008). |
Farber, Rob “CUDA, Supercomputing for the Masses: Part 4, The CUDA Memory Model”, Under the High Performance Computing section of the Dr. Dobbs website, p. 3 available at http://www.ddj.com/hpc-high-performance-computing/208401741, 3, 2008. |
Rajko, S et al., “HMM Parameter Reduction for Practice Gesture Recognition”, Proceedings of the International Conference on Automatic Gesture Recognition, (Sep. 2008). |
Hinton, Geoffrey et al., “A Fast Learning Algorithm for Deep Belief Nets”, Neural Computation, V. 18, 1527-1554, 2006. |
Susskind, Joshua M., et al., “Generating Facial Expressions with Deep Belief Nets”, Department of Psychology, Univ. of Toronto I-Tech Education and Publishing, (2008),421-440. |
Bleyer, Michael et al., “Surface Stereo with Soft Segmentation.”, Computer Vision and Pattern Recognition. IEEE, 2010, (2010). |
Chen, Junqing et al., “Adaptive perceptual color-texture image segmentation.”, The International Society for Optical Engineering, SPIE Newsroom, (2006), 1-2. |
Forsyth, David A., et al., “Stereopsis”, In Computer Vision A Modern Approach Prentice Hall, 2003, (2003). |
Harris, Mark et al., “Parallel Prefix Sum (Scan) with CUDA”, vol. 39, in GPU Gems 3, edited by Hubert Nguyen, (2007). |
Hirschmuller, Heiko “Stereo Vision in Structured Environments by Consistent Semi-Global Matching”, Computer Vision and Pattern Recognition, CVPR 06, (2006),2386-2393. |
Ivekovic, Spela et al., “Dense Wide-baseline Disparities from Conventional Stereo for Immersive Videoconferencing”, ICPR. 2004, (2004), 921-924. |
Kaldewey, Tim et al., “Parallel Search On Video Cards.”, First USENIX Workshop on Hot Topics in Parallelism (HotPar '09), (2009). |
Kirk, David et al., “Programming Massively Parallel Processors A Hands-on Approach”, Elsevier, 2010, (2010). |
Klaus, Andreas et al., “Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure”, Proceedings of ICPR 2006. IEEE, 2006, (2006), 15-18. |
Kolmogorov, Vladimir et al., “Computing Visual Correspondence with Occlusions via Graph Cuts”, International Conference on Computer Vision. 2001., (2001). |
Kolmogorov, Vladimir et al., “Generalized Multi-camera Scene Reconstruction Using Graph Cuts.”, Proceedings for the International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. 2003., (2003). |
Kuhn, Michael et al., “Efficient ASIC Implementation of a Real-Time Depth Mapping Stereo Vision System”, Proceedings of 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. Taipei, Taiwan: IEEE, 2009., (2009). |
Li, Shigang “Binocular Spherical Stereo”, IEEE Transactions on Intelligent Transportation Systems (IEEE) 9, No. 4 (Dec. 2008), (Dec. 2008),589-600. |
Marsalek, M et al., “Semantic hierarchies for visual object recognition”, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR '07. MN: IEEE, 2007, (2007), 1-7. |
Metzger, Wolfgang “Laws of Seeing”, MIT Press, 2006, (2006). |
Min, Dongbo et al., “Cost Aggregation and Occlusion Handling With WLS in Stereo Matching”, Edited by IEEE. IEEE Transactions on Image Processing 17 (2008), (2008), 1431-1442. |
“NVIDIA: CUDA compute unified device architecture, prog. guide, version 1.1”, NVIDIA, (2007). |
Remondino, Fabio et al., “Turning Images into 3-D Models”, IEEE Signal Processing Magazine, (2008). |
Richardson, Ian E., “H.264/MPEG-4 Part 10 White Paper”, White Paper/www.vcodex.com, (2003). |
Sengupta, Shubhabrata “Scan Primitives for GPU Computing”, Proceedings of the 2007 Graphics Hardware Conference. San Diego, CA, 2007, (2007),97-106. |
Sintron, Eric et al., “Fast Parallel GPU-Sorting Using a Hybrid Algorithm”, Journal of Parallel and Distributed Computing (Elsevier) 68, No. 10, (Oct. 2008), 1381-1388. |
Wang, Zeng-Fu et al., “A Region Based Stereo Matching Algorithm Using Cooperative Optimization”, CVPR, (2008). |
Wei, Zheng et al., “Optimization of Linked List Prefix Computations on Multithreaded GPUs Using CUDA”, 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS). Atlanta, (2010). |
Wiegand, Thomas et al., “Overview of the H.264/AVC Video Coding Standard”, IEEE Transactions on Circuits and Systems for Video Technology 13, No. 7. (Jul. 2003), 560-576. |
Woodford, O.J. et al., “Global Stereo Reconstruction under Second Order Smoothness Priors”, IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE) 31, No. 12, (2009),2115-2128. |
Yang, Qingxiong et al., “Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling”, IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE) 31, No. 3, (Mar. 2009),492-504. |
Zinner, Christian et al., “An Optimized Software-Based Implementation of a Census-Based Stereo Matching Algorithm”, Lecture Notes in Computer Science (SpringerLink) 5358, (2008),216-227. |
“PCT Search report”, PCT/US2010/035717, (dated Sep. 1, 2010), 1-29. |
“PCT Written opinion”, PCT/US2010/035717, (dated Dec. 1, 11), 1-9. |
“PCT Search report”, PCT/US2011/49043, (dated Mar. 21, 2012), 1-4. |
“PCT Written opinion”, PCT/US2011/49043, (dated Mar. 21, 2012), 1-4. |
“PCT Search report”, PCT/US2011/049808, (dated Jan. 12, 2012), 1-2. |
“PCT Written opinion”, PCT/US2011/049808, (dated Jan. 12, 2012), 1-5. |
“Non-Final Office Action”, U.S. Appl. No. 12/784,123, (dated Oct. 2, 2012), 1-20. |
“Non-Final Office Action”, U.S. Appl. No. 12/784,022, (dated Jul. 16, 2012), 1-14. |
Tieleman, T et al., “Using Fast weights to improve persistent contrastive divergence”, 26th International Conference on Machine Learning New York, NY ACM, (2009), 1033-1040. |
Sutskever, I et al., “The recurrent temporal restricted boltzmann machine”, NIPS, MIT Press, (2008), 1601-1608. |
Parzen, E “On the estimation of a probability density function and the mode”, Annals of Math. Stats., 33, (1962), 1065-1076. |
Hopfield, J.J. “Neural networks and physical systems with emergent collective computational abilities”, National Academy of Sciences, 79, (1982),2554-2558. |
Culibrk, D et al., “Neural network approach to background modeling for video object segmentation”, IEEE Transactions on Neural Networks, 18, (2007), 1614-1627. |
Benggio, Y et al., “Curriculum learning”, ICML 09 Proceedings of the 26th Annual International Conference on Machine Learning, New York, NY: ACM, (2009). |
Benggio, Y et al., “Scaling learning algorithms towards AI. In L. a Bottou”, Large Scale Kernel Machines, MIT Press, (2007). |
Battiato, S et al., “Exposure correction for imaging devices: An overview”, In R. Lukac (Ed.), Single Sensor Imaging Methods and Applications for Digital Cameras, CRC Press, (2009), 323-350. |
Yang, Q et al., “Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation and Occlusion Handling”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, Issue 3, 2009, pp. 1-16. |
U.S. Appl. No. 12/028,704, filed Feb. 2, 2008, Method and System for Vision-Based Interaction in a Virtual Environment. |
U.S. Appl. No. 13/405,319, filed Feb. 26, 2012, Method and System for Vision-Based Interaction in a Virtual Environment. |
U.S. Appl. No. 13/411,657, filed Mar. 5, 2012, Method and System for Vision-Based Interaction in a Virtual Environment. |
U.S. Appl. No. 13/429,437, filed Mar. 25, 2012, Method and System for Vision-Based Interaction in a Virtual Environment. |
U.S. Appl. No. 13/562,351, filed Jul. 31, 2012, Method and System for Tracking of a Subject. |
U.S. Appl. No. 13/596,093, filed Aug. 28, 2012 Method and Apparatus for Three Dimensional Interaction of a Subject. |
U.S. Appl. No. 11/567,888, filed Dec. 7, 2006, Three-Dimensional Virtual-Touch Human-Machine Interface System and Method Therefor. |
U.S. Appl. No. 13/572,721, filed Aug. 13, 2012, Method and System for Three-Dimensional Virtual-Touch Interface. |
U.S. Appl. No. 12/784,123, filed Mar. 20, 2010, Gesture Recognition Systems and Related Methods. |
U.S. Appl. No. 12/784,022, filed May 20, 2010, Systems and Related Methods for Three Dimensional Gesture Recognition in Vehicles. |
U.S. Appl. No. 13/025,038, filed Feb. 10, 2011, Method and Apparatus for Performing Segmentation of an Image. |
U.S. Appl. No. 13/025,055, filed Feb. 10, 2011, Method and Apparatus for Disparity Computation in Stereo Images. |
U.S. Appl. No. 13/025,070, filed Feb. 10, 2011, Method and Apparatus for Determining Disparity of Texture. |
U.S. Appl. No. 13/221,903, filed Aug. 31, 2011, Method and Apparatus for Confusion Learning. |
U.S. Appl. No. 13/189,517, filed Jul. 24, 2011, Near-Touch Interaction with a Stereo Camera Grid and Structured Tessellations. |
U.S. Appl. No. 13/297,029, filed Nov. 15, 2011, Method and Apparatus for Fast Computational Stereo. |
U.S. Appl. No. 13/297,144, filed Nov. 15, 2011, Method and Apparatus for Fast Computational Stereo. |
U.S. Appl. No. 13/294,481, filed Nov. 11, 2011, Method and Apparatus for Enhanced Stereo Vision. |
U.S. Appl. No. 13/316,606, filed Dec. 12, 2011, Method and Apparatus for Enhanced Stereo Vision. |
Number | Date | Country | |
---|---|---|---|
61379706 | Sep 2010 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 13025038 | Feb 2011 | US |
Child | 14148761 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 16812346 | Mar 2020 | US |
Child | 17871971 | US | |
Parent | 14148761 | Jan 2014 | US |
Child | 16812346 | US |