Patent Application
20120224037
Not applicable.
The present invention relates generally to displaying stereoscopic images on a display.
Viewing stereoscopic content on planar stereoscopic display sometimes triggers unpleasant feelings of discomfort or fatigue in the viewer. The discomfort and fatigue may be, at least in part, caused by limitations of existing planar stereoscopic displays. A planar stereoscopic display, no matter whether LCD based or projection based, shows two images with disparity between them on the same planar surface. By temporal and/or spatial multiplexing the stereoscopic images, the display results in the left eye seeing one of the stereoscopic images and the right eye seeing the other one of the stereoscopic images. It is the disparity of the two images that results in viewers feeling that they are viewing three dimensional scenes with depth information. This viewing mechanism is different from how eyes normally perceive natural three dimensional scenes, and may cause a vergence-accommodation conflict. The vergence-accommodation conflict strains the eye muscle and sends confusing signals to the brain, and eventually causes discomfort/fatigue.
The preferred solution is to construct a volumetric three dimensional display to replace existing planar stereoscopic displays. Unfortunately, it is difficult to construct such a volumetric display, and is likewise difficult to control such a display.
Another solution, at least in part, is based upon signal processing. The signal processing manipulates the stereoscopic image pair sent to the planar stereoscopic display in some manner. Although the signal processing cannot fundamentally completely solve the problem, the vergence-accommodation conflict can be significantly reduced and thereby reduce the likelihood of discomfort and/or fatigue.
What is desired is a display system that reduces the discomfort and/or fatigue for stereoscopic images.
The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.
The system provides a signal processing based technique to reduce the discomfort/fatigue associated with a three dimensional viewing experience. More specifically, given a planar stereoscopic display, the technique takes in a stereoscopic image pair that may cause viewing discomfort/fatigue, and outputs a modified stereoscopic pair that causes less or no viewing discomfort/fatigue.
One simplified example (merely for purposes of illustration) of a stereoscopic processing system for reducing viewer discomfort is illustrated in
As shown in
The disparity map estimation 200 outputs two disparity maps, L2R map 202 and R2L map 204. The L2R map 202 gives the disparity of each pixel in the L image, while the R2L map 204 gives the disparity of each pixel in the R image. The data also tends to indicate occlusion regions which may be accounted for in the new view image synthesis 400. The disparity map estimation 200 also provides matching errors of the two disparity maps, which provides a measure of confidence in the map data.
The adjustment based on the L2R map 202 and the R2L map 204 in the disparity map adjustment 300 may be controlled by a disparity selection 302. The disparity selection 302 may predict the discomfort based upon the estimated disparity in the image pairs 202, 204, viewing conditions, display characteristics, viewer preferences, and/or any other suitable data. Based upon this estimation, the amount of disparity may be modified. The modification may result in global modification, object based modification, region based modification, or otherwise. A modified set of disparity maps 310, 320 are created by the disparity map adjustment 300. The modified disparity maps may include the newview2R disparity map 310 and the newview2L disparity map 320 which are provided to the new view image synthesis 400.
The new view image synthesis 400 synthesizes the new views 120 and/or 130 based upon data from the disparity map adjustment 300, the disparity map estimation 200, and the input image pair 100, 110. It is to be understood that any suitable technique may be used to compute disparities or otherwise the disparities may be provided together with the video content, and one or more new images may be computed as a result.
While this technique of computing the disparity information is suitable for image content from a content provider from which disparity information is traditionally provided or computed, there are other types of content such as graphical elements that are rendered on the display that do not contain disparity information from the content provider. Some of the types of graphical elements includes captions, menus, and logos. Often, these graphical elements are inserted into the video stream or otherwise indicated to be displayed on the display, after the video content is obtained. For example, while obtaining three dimensional video content of a live football game, graphical overlays may be provided with scoring or other information that should also be displayed on the display. When graphical overlays are rendered on a traditional two-dimensional display, the horizontal and vertical positions for the graphical element are provided. When a graphical element is rendered on a stereoscopic display, the disparity information for the graphical element should be provided. That is to say that the graphical elements should be rendered on both the left view and the right view with the appropriate amount of horizontal displacement.
One approach for rendering the graphical elements is to use a fixed disparity value, such as for example zero disparity or a pre-determined negative disparity. Unfortunately, using a fixed disparity tends to result in a visual conflict at the boundaries where the graphical element and the three dimensional video content being display have substantially different disparities. This conflict of different disparities tends to make it hard to fuse both views together and ultimately tends to result in discomfort for the viewers.
A modified technique for the suitable presentation of a graphical element on a stereoscopic display should be based upon the local disparity in the region where the graphical element is to be displayed. In this manner, the local content of the region where the graphical element is to be displayed is used to determine the disparity to be used for the graphical element, and therefore the disparity of the local region and the graphical element will substantially match one another. The result of such a content-dependent technique for determining the disparity information is that fewer artifacts are created and a more comfortable viewing experience is achieved for the viewer.
Referring to
A local disparity analysis 630 may compute a disparity histogram of a local window where the geographical element is to be located. This aggregates disparity data that may be used for determining a suitable disparity for the graphical element.
Referring also to
Referring also to
The preferred disparity for the video content of the current frames together with one or more graphical elements 640 may be computed. In this manner, the disparities for the video images are determined and those regions of the image that correspond to graphical elements are likewise determined, both in a manner suitable to reduce viewer discomfort.
The disparity for the local window may be computed in any suitable manner, such as by using a histogram. For example, the disparity associated with the graphical element may be a mean disparity, a most frequent disparity, or a minimal disparity. The mean disparity may be determined as the mean disparity of substantially all the samples in the histogram. The most frequent disparity may be determined as the disparity value associated with the highest count in the histogram. The minimal disparity may be determined as the minimal disparity in the histogram. The minimal disparity in the histogram tends to ensure that the graphical element will be placed “on top of” the underlying content in terms of perceived depth, which is usually what a viewer would expect to observe for a graphical element.
In some cases, the local disparity will vary significantly between temporal frames, especially in the case of a shot boundary in the video. Accordingly, a temporal smoothing 660 of the disparity calculations 650 is preferably used to smooth out such effects. For example, assuming the disparity of the graphical element is determined to be d(t), the system may apply a temporal filter to reduce the disparity from changing to dramatically from frame to frame. As one particular embodiment, an infinite impulse response filter may be used: {circumflex over (d)}(t)=α•{circumflex over (d)}(t−1)+(1−α)•d(t). If a shot boundary is detected, the temporal smoothing may be turned off (or substantially reduced) to avoid false disparity information being leaked from one shot to the next shot, as defined by:
With the disparities for graphical elements being content-dependent on the particular region(s) of the display, together with temporal smoothing, the disparity estimates for rendering the images are determined 660 and the graphical elements are rendered on both the left image and the right image with the appropriate disparity values. For example, the final disparities estimate {circumflex over (d)}(t) for the on-screen graphical element(s) may be rendered and positioned. Assuming the intended spatial location of the left image is [Gx(t), Gy(t)], the same overlay may be placed on the right image as [Gx(t)+{circumflex over (d)}(t), Gy(t)]. The opacity of the graphical overlay may also be modified based on its disparity. For example, if there is a strong conflict between the scene disparities and the graphical disparities, the opacity of the graphical element may be reduced which reduces the visual discomfort.
The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.
