Patent Application
20150294470
The present invention relates to a three dimensional video processing system. In particular, the present invention is directed towards a method to generate and visualize the distribution of disparities in a stereo sequence over time.
Three dimensional (3D) video relies on at least two views of a single image, with each view originating from a different position. For example, humans see a scene with two eyes separated from each other by a certain distance, resulting in a different angle of view of an object. The brain computes the difference between these two angles and generates an estimated distance of the object. Likewise, in 3D video, two different camera angles are captured simultaneously of a scene. A computer then processes the image and determines an object depth primarily in response to the distance in pixels between a pixel in a first image and the corresponding pixel in the second image. This distance is referred to as disparity. The disparity map of a stereo pair gives a distance value for each pixel, which corresponds to the horizontal offset between matching points in the left view and right view images. In some applications, it is desired to study the disparities along time. As such, visualizing the disparity map in a sequential fashion (one frame at a time) can damage both productivity and quality of the work. It would be desirable to be able to distill and visualize the information provided in the disparity map over many frames simultaneously.
In one aspect, the present invention involves a method comprising the steps of receiving a video stream comprising a plurality of 3D images, determining at least one disparity value for each of said plurality of 3D images, weighting each of said at least one disparity values with a confidence value to generate a plurality of weighted disparity values, normalizing each of said plurality of weighted disparity values to generate a plurality of normalized disparity values, and generating a graphical representation of said plurality of normalized disparity values where each of said plurality of normalized disparity values corresponds to a different time in said video stream
In another aspect, the invention also involves an apparatus comprising an input wherein said input is operative to receive a video stream comprising a plurality of 3D images, a processor for determining at least one disparity value for each of said plurality of 3D images, weighting each of said at least one disparity values with a confidence value to generate a plurality of weighted disparity values, normalizing each of said plurality of weighted disparity values to generate a plurality of normalized disparity values, and an output for receiving said plurality of normalized disparity values from said processor where each of said plurality of normalized disparity values corresponds to a different time in said video stream.
In another aspect, the invention also involves a method of processing a 3D video signal comprising the steps of receiving a video stream comprising a plurality of paired images, wherein said paired images consist of two images and wherein each of said images having different perspectives of the same scene, determining at least one disparity value for each of said paired images by determining the difference in the location of objects within each of said images, weighting each of said at least one disparity values with a confidence value to generate a plurality of weighted disparity values, normalizing each of said plurality of weighted disparity values to generate a plurality of normalized disparity values, generating a graphical representation of said plurality of normalized disparity values where each of said plurality of normalized disparity values corresponds to a different time in said video stream.
The characteristics and advantages of the present invention will become more apparent from the following description, given by way of example. One embodiment of the present invention may be included within an integrated video processing system. Another embodiment of the present invention may comprise discrete elements and/or steps achieving a similar result. The exemplifications set out herein illustrate preferred embodiments of the invention, and such exemplifications are not to be construed as limiting the scope of the invention in any manner.
Referring to
The source of a 3D video stream 110, such as a storage device, storage media, or a network connection, provides a time stream of two images. Each of the two images is a different angular view of the same scene. Thus, the two images will have slightly different characteristics in that the scene is viewed from different angles separated by a horizontal distance, similar to what would be seen by each individual eye in a human. Each image may contain information not available in the other image due to some objects in the foreground of one image hiding information available in the second image due to camera angle. For example, one view taken closer to a corner would see more of the background behind the corner than a view take further away from the corner. This results in only one point being available for a disparity map and therefore generating a less reliable disparity map.
The processor 120 receives the two images and generates a disparity value for a plurality of points in the image. These disparity values can be used to generate a disparity map, which shows the regions of the image and their associated image depth. The image depth of a portion of the image is inversely proportional to the disparity value. The processor then stores these disparity values on a memory 130 or the like.
After further processing by the processor 120 according to the present invention, the apparatus can display to a user a disparity map for a pair of images, or can generate a disparity time comparison according to the present invention. These will be discussed in further detail with reference to other figures. These comparisons are then displayed on a display device, such as a monitor, or a led scale, or similar display device.
Referring now to
Once loaded into the exemplary 3D video processing system according to the present invention, the 3D video images are analyzed to calculate and record depth information 240. This information is stored in a storage media. After analysis, an analyst or other user will then review 250 the information stored in the storage media and determine if the some or all of the analysis must be repeated with different parameters. The analyst may also reject the content. A report is then prepared for the customer 260, and the report is presented to the customer 270 and any 3D video content is returned to the customer 280. The two pass processes permits an analyst to optimize the results based on a previous analysis.
Referring now to
Once loaded into the exemplary 3D video processing system according to the present invention, the 3D video images are analyzed to calculate and record depth information 340, generate depth map and perform automated analysis live during playback. This information is may stored in a storage media. An analyst will review the generated information. Optionally the system may dynamically down-sample to maintain real-time playback. A report may optionally be prepared for the customer 350, and the report is presented to the customer 360 and any 3D video content is returned to the customer 370.
Referring now to
Referring now to
First, the system receives the 3D video stream as a series of paired images 510. Each image in a pair represents a view of the scene as taken from a slightly different perspective. These images may be transmitted as part of a live 3D video stream. Alternatively, they can be transmitted via a media storage device, such as a hard drive, flash memory, or optical disk, or the images may be received from a remote storage location via a network connection.
The system then performs a disparity calculation and generates a disparity map 520. A disparity map, sometimes called a depth map, is an array of values that contains information relating to the distance of the surfaces of scene objects from a viewpoint. In one exemplary embodiment of the present disclosure, the values of the disparity map are stored as a “short integer” data type, hence the possible range of disparities is between −32768 and 32767.
The system then generates a confidence map using the generated disparity map 530. To improve upon the initial disparity estimates generated in the previous step, a subsequent refinement step is commonly employed. The accuracy of disparity map calculations inherently depend on the underlying image content. For some regions of an image, it may be difficult or impossible to establish accurate point correspondences. This results in disparity estimates of varying accuracy and reliability. A confidence map may then be generated which models the reliability of each disparity match. In an exemplary embodiment, the values of the confidence map are stored in an unsigned char type, and the values can vary from 0 for very low confidence up to 255 for very high confidence.
The system then generates a histogram weighted with the confidences of the disparity values 540. An array Hi of histograms, where the sub-index i indicates frame number, is computed for every disparity map with its associated confidence map. Within each histogram, the bins represent disparity values, and for every pixel's disparity value in the disparity map, its correspondent confidence value is added to the corresponding bin. The array Hi can be interpreted as a histogram weighted with the confidences of the disparity values. In our particular embodiment, the size of the histogram is 512 bins.
To generate Hi let Di be the disparity map and Ci its associated confidence map for the i-th frame, both expressed as a column vector. Let Hi be an array of size s, initialized to zeroes, which will contain the result of the procedure. Then, the procedure is as follows (note that the center of the histogram is s/2):
p=min(max(0, Di[j]+s/2), s−1)
H
i
[p]=H
i
[p]+C
i
[j]
H={H0, H1, . . . , HN} is the set of all the histograms in the video sequence.
d=L[(1−p)*(N−1)]
The system then normalizes the histogram 550. In order to visualize the histograms consistently on a video display device, they have to be normalized. In the exemplary embodiment of the present disclosure, the common variable d that will divide all the data in H is chosen using the steps of procedure 2. As d is not necessarily defined as the maximum value in H, during the normalization process, all the values greater than 1 will be clipped to 1.
To generate Ĥ, the normalized value of H, a percentage factor is applied that offsets the normalizing parameter from the peak (in an exemplary embodiment this value is set as 0.95).
The system then optionally applies user defined thresholds 560. The user s may set predefined thresholds which may indicate undesirable conditions, such as hyperconvergence or hyperdivergence. These thresholds may be indicated on the display by changing color of the histogram. For example, when the value of the histogram exceeds a certain threshold, the color is changed to red making easier for a user to recognize the condition is present.
The system then couples the histogram to a display device 570. The set of Ĥ is finally rendered on the screen. As the bins of Ĥ directly match to disparity values, different colors can be used to indicate if the disparity is between user-defined thresholds, like error and warning thresholds for hyper convergence and hyper divergence (see
Turning now to
The method according to the present invention may be practiced, but is not limited to, using the following hardware and software: SIT-specified 3D Workstation, one to three 2D monitors, a 3D Monitor (frame-compatible and preferably frame-sequential as well), Windows 7 (for workstation version), Windows Server 2008 R2 (for server version), Linux (Ubuntu, CentOS), Apple Macintosh OSX, Adobe Creative Suite software and Stereoscopic Player software.
It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input/output interfaces.
The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its spirit and scope.
All examples and conditional language recited herein are intended for informational purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.
Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.
Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herewith represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.
The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (“DSP”) hardware, read only memory (“ROM”) for storing software, random access memory (“RAM”), and nonvolatile storage.
Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context.
Although embodiments which incorporate the teachings of the present disclosure have been shown and described in detail herein, those skilled in the art can readily devise many other varied embodiments that still incorporate these teachings. Having described preferred embodiments for a method and system for disparity visualization (which are intended to be illustrative and not limiting), it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings.
This application claims the benefit of U.S. Provisional Patent Application No. 61/563261, filed Nov. 23, 2011 entitled “METHOD AND SYSTEM FOR DISPARITY VISUALIZATION” which is incorporated herein by reference.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2012/066580 | 11/27/2012 | WO | 00 |
