Patent Application
20150249817
The invention relates to a system and method for processing a three-dimensional [3D] image signal, the 3D image signal comprising a two-dimensional [2D] image signal and a 2D auxiliary signal, the 2D auxiliary signal enabling 3D rendering of the 2D image signal on a 3D display. The invention further relates to a 3D display device comprising the system.
Increasingly, display devices such as televisions, digital photo frames, tablets and smartphones comprise 3D displays to provide a user with a perception of depth when viewing content on such a device. For that purpose, such 3D display devices may, either by themselves or together with glasses worn by the user, provide the user with different images in each eye so as to provide the user with a perception of depth based on stereoscopy, i.e., a stereoscopic perception of depth.
3D display devices typically use content which contains depth information in order to establish the content on screen as having a degree of depth. The depth information may be provided implicitly in the content. For example, in the case of so-termed stereo content, the depth information is provided by the differences between a left and a right image signal of the stereo content. Together, the left and right image signal thus constitute a stereo 3D image signal. The depth information may also be provided explicitly in the content. For example, in content encoded in the so-termed image+depth format, the depth information is provided by a 2D depth signal comprising depth values which are indicative of distances that objects within the 2D image signal have towards a camera or viewer. Instead of depth values, also disparity values may be used, i.e., the 2D depth signal may be a 2D disparity signal, or in general, a 2D depth-related signal. The 2D image signal and the 2D depth-related signal together constitute an alternative to the stereo 3D image signal.
Essentially, a 3D image signal thus comprises of at least one 2D image signal and one 2D auxiliary signal, the latter being, e.g., a 2D depth-related signal, or a further 2D image signal which together with the 2D image signal constitutes a stereo 3D image signal.
With respect to the 3D displays themselves: so-termed autostereoscopic displays provide said stereoscopic perception of depth without needing the viewer to wear polarized or shutter-based glasses. For that purpose, optical components are used, such as lenticular lens arrays (or in general lenticular or barrier means), which enable the display to emit a viewing cone from each given point on the 3D display, the viewing cone comprising at least a left view and a right view of a scene. This enables the viewer to see a different image with each eye when positioned accordingly within the viewing cone. Certain autostereoscopic displays, sometimes referred to as automultiscopic displays, provide multiple views of the same scene, rather than only a left and a right view. This allows the viewer to assume multiple positions in the viewing cone, i.e., move left-right in front of the display, while still obtaining a stereoscopic perception of the scene.
Examples of such autostereoscopic displays are described in a paper by C. van Berkel et al entitled “Multiview 3D -LCD” published in SPIE Proceedings Vol. 2653, 1996, pages 32 to 39 and in GB-A-2196166. In these examples the autostereoscopic display comprises a matrix LC (liquid crystal) display panel which has rows and columns of pixels (display elements) and which acts as a spatial light modulator to modulate light from a light source. The display panel can be of the kind used in other display applications, for example computer display screens for presenting display information in two dimensional form. A lenticular sheet, for example in the form of a molded or machined sheet of polymer material, overlies the output side of the display panel with its lenticular elements, comprising (semi) cylindrical lens elements, extending in the column direction with each lenticular element being associated with a respective group of two, or more, adjacent columns of display elements and extending in a plane that runs parallel with the display element columns. In an arrangement in which each lenticule is associated with two columns of display elements, the display panel is driven to display a composite image comprising two 2D sub-images vertically interleaved, with alternate columns of display elements displaying the two images, and the display elements in each column providing a vertical slice of the respective 2D (sub) image. The lenticular sheet directs these two slices, and corresponding slices from the display element columns associated with the other lenticules, to the left and right eyes respectively of a viewer in front of the sheet so that, with the sub-images having appropriate binocular disparity, the viewer perceives a single stereoscopic image. In other, multi-view, arrangements, in which each lenticule is associated with a group of more than two adjacent display elements in the row direction and corresponding columns of display elements in each group are arranged appropriately to provide a vertical slice from a respective 2-D (sub-) image, then as a viewer's head moves a series of successive, different, stereoscopic views are perceived for creating, for example, a look-around impression.
Autostereoscopic displays of above kind may be used for various applications, for example in home or portable entertainment, medical imaging and computer-aided design (CAD).
The inventors have recognized that when viewing content of a 3D image signal on a 3D display, any high-detail features in the content are best displayed at a display depth which not too far from the display plane, i.e., at a relatively neutral display depth. A reason for this is that crosstalk such as optical crosstalk may occur between the stereoscopic sub-images perceived by the user. Such crosstalk typically leads to so-termed ghosting. Ghosting in general is distracting for a viewer. Ghosting in high-detailed features in the content is especially distracting for the viewer. By displaying the high-detailed features at a relatively neutral display depth, such ghosting is reduced.
Examples of such high-detailed features are subtitles, broadcaster logos, or graphical user interface (GUI) elements, or in general anything involving small text which needs to displayed to the user in a readable form. In general, such high-detailed features are henceforth referred to as overlays due to the subtitles, logos, etc, typically being overlaid over another type of content.
When such overlays are delivered separately from the 2D image signal, e.g., as separate layers or streams within or next to the 3D image signal, the overlays may be displayed at a depth which not too far from the display plane by assigning corresponding depth values to the overlays. Nevertheless, constraints have to be taken into account such that, e.g., subtitles are assigned a depth which places them in front of the content, i.e., nearer to the viewer, than the content underlying and surrounding the subtitles. The inventors have recognized that such depth assignment is much more difficult when the overlays have already been composited into the content, i.e., being hardcoded in the 3D image signal. Although it is possible to detect an overlay, e.g., using an overlay detection (also known as overlay segmentation) as known per se from the field of image analysis, such overlay detection is frequently imperfect. Hence, a detected overlay may not perfectly match the actual overlay in the 3D image signal. In particular, (sub)pixel-accurate overlay detection is difficult.
In principle, it is possible to already assign a relatively neutral display depth to overlays when generating the 3D image signal. For that purpose, a 2D-to-3D conversion may be used which inherently or by design attempts to assign said neutral display depth to the overlays. However, the inventors have recognized that depth estimation for small text like subtitles is difficult since they comprise separate thin structures. Consequently, it is difficult for a depth estimator to assign the same depth to such separate parts of the overlay. This can cause fluctuations (spatially and temporally) of the depth assigned to such overlays. These can be very distracting for a viewer.
It would be advantageous to provide a system and method which addresses the above concerns.
A first aspect of the invention provides a system for processing a three-dimensional [3D] image signal, the 3D image signal comprising a two-dimensional [2D] image signal and a 2D auxiliary signal, the 2D auxiliary signal enabling 3D rendering of the 2D image signal on a 3D display, the system comprising:
A further aspect of the invention provides a 3D display device comprising the system set forth.
A further aspect of the invention provides a method for processing a three-dimensional [3D] image signal, the 3D image signal comprising a two-dimensional [2D] image signal and a 2D auxiliary signal, the 2D auxiliary signal enabling 3D rendering of the 2D image signal on a 3D display, the method comprising:
A further aspect of the invention provides a computer program product comprising instructions for causing a processor system to perform the method set forth.
The above measures provide a user interface subsystem for enabling a user to define a user-defined 2D region in the 2D image signal. For example, the user may define the user-defined 2D region using a graphical user interface (GUI) to define a top side, bottom side, left side, and right side of a rectangular user-defined 2D region, or select the user-defined 2D region amongst a plurality of pre-determined 2D regions, etc. Based on the user-defined 2D region, a 2D region in the 2D auxiliary signal is defined. The 2D region corresponds to a display region on a display plane of the 3D display when 3D rendering the 2D image signal using the 2D auxiliary signal. A depth reduction parameter is obtained for the display region, with the depth reduction parameter representing a desired amount of depth reduction in the display region. Here, the term depth reduction refers to a reduction towards a neutral display depth when 3D rendering the 2D image signal on the 3D display. To enable the depth reduction to be established, an adjustment value is derived from the depth reduction parameter. Accordingly, the depth reduction in the display region can be established, namely by adjusting signal values of the 2D auxiliary signal within the 2D region based on the adjustment value.
The above measures have the effect that a 2D region is defined and an adjustment value is provided, which together enable establishing the depth reduction in the display region by adjusting signal values of the 2D auxiliary signal within the 2D region based on the adjustment value. This has the advantageous effect that when the 3D image signal comprises hard-coded overlays, the user is enabled to define the user-defined 2D region to include the hard-coded overlays, thereby causing the system to provide an adjustment value for a corresponding 2D region in the auxiliary signal, with the adjustment value enabling the depth of the hard-coded overlays in the display region to be reduced towards a neutral display depth. Accordingly, in case a depth estimator has assigned an erroneous depth to the hard-coded overlay, the erroneous depth can be reduced, thereby also reducing depth fluctuations typically associated with such an erroneous depth. Advantageously, it is not needed to rely on an (automatic) overlay detection which is frequently imperfect for the earlier mentioned reasons. Rather, the user is enabled to define the user-defined 2D region him/herself, i.e., manually.
It is noted that US 2011/0316991 A1 describes a stereoscopic display device including: a parallax adjustment section performing a parallax adjustment on each of a left-eye image and a right-eye image which are inputted; and a display section displaying the left-eye image and the right-eye image which are resultant of the parallax adjustment by the parallax adjustment section. The parallax adjustment section performs the parallax adjustment only on a region other than an OSD image region with an OSD image superposed therein in each of the left-eye image and the right-eye image. US 2011/0316991 A1 thus excludes the OSD image from parallax control. However, US 2011/0316991 does not disclose enabling a user to establish a user-defined 2D region so as to establish a depth reduction in a display region on the 3D display. Rather, US 2011/0316991 selects a fixed region, namely that of the OSD image, from which parallax control is to be excluded and a complementary region, namely the rest of the image, to which parallax control is to be applied. In fact, in US 2011/0316991, the OSD image is explicitly available to the system. Accordingly, US 2011/0316991 does not offer a solution for dealing with hardcoded overlays in a 3D image signal.
EP 2451176 A2 describes video communication method of a 3D video communication, which is said to include acquiring a plurality of 2D images corresponding to a talker using a 3D camera, adjusting a point of convergence of the plurality of 2D images using a preset feature point of the talker, detecting an object located between the talker and the 3D camera using the acquired plurality of 2D images, scaling an original sense of depth of the detected object to a new sense of depth, and generating a 3D talker image including the object with the new sense of depth and transmitting the 3D talker image to a 3D video communication apparatus of a listener. EP 2451176 A2, however, does not disclose enabling a user to establish a user-defined 2D region so as to establish a depth reduction in a display region on the 3D display. Rather, in EP 2451176, the object is detected automatically using an object detecting unit 135 [0077]. In fact, EP 2451176 makes use of the depth of the object to detect the object, i.e., the object is effectively located in 3D [0077-0082], and thereby relies on the distance having been correctly obtained by the 3D camera. It will be appreciated that this does not provide a solution for dealing with hardcoded overlays in a 3D image signal to which a depth estimator may have assigned an erroneous depth.
US 2012/0162199 A1 describes an apparatus and a method for displaying a 3D augmented reality. It is said that if the augmented reality is implemented as a 3D image, some of the 3D augmented information may be degraded in terms of information delivery efficiency. It is further said that the 3D effects may be selectively removed from a selected object of a 3D augmented reality image by providing an object area detecting unit to detect a first object area of a left image frame of a 3D image and a second object area of a right image frame of the 3D image based on a selected object of the 3D image, and a frame adjusting unit to adjust the left image frame and the right image frame to change a 3D effect of the selected object. However, in US 2012/0162199, the objects are known per se, i.e., defined by object information [see 0036, 0048], thereby enabling the apparatus to know which object is selected by the user, or even enabling the object to be automatically selected [see 0059]. Accordingly, instead of enabling a user to establish a user-defined 2D region so as to establish a depth reduction in a display region on the 3D display, US 2012/0162199 enables the user to directly select an object via the object information. US 2012/0162199 thus addresses the problem of how to remove a three-dimensional effect of an object defined by object information, and does not offer a solution when dealing with hardcoded overlays in a 3D image signal, i.e., for which such object information is not available.
The following describes optional aspects of the present invention.
Optionally, the region definer comprises an overlay detector for detecting an overlay in the 3D image signal, and the user interface subsystem is arranged for enabling the user to establish the user-defined 2D region based on the detected overlay. Although it has been recognized that an overlay detector may fail at perfectly detecting hardcoded overlays in a 3D image signal, a detected overlay may nevertheless be used as a basis for the user in defining the user-defined 2D region. For example, the detected overlay may guide the user towards the location of the hardcoded overlay, thereby enabling the user to quickly and conveniently define the user-defined 2D region. Another example is that, at times, the detected overlay may be sufficiently well detected, thereby enabling the user to define the user-defined 2D region directly based on the detected overlay.
Optionally, the user interface subsystem is arranged for using the detected overlay to:
Optionally, the user interface subsystem is arranged for enabling the user to specify the desired amount of depth reduction in the display region, thereby establishing the depth reduction parameter.
Optionally, the signal input is arranged for obtaining metadata indicative of a pre-defined 2D region, and the region definer is arranged for defining the 2D region based on the pre-defined 2D region.
Optionally, the depth processor is arranged for deriving an offset value from the depth reduction parameter to enable adjusting the signal values of the 2D auxiliary signal within the 2D region by applying said offset value to the signal values.
Optionally, the 2D auxiliary signal is a 2D depth-related signal, and the depth processor is arranged for deriving a gain value from the depth reduction parameter to enable adjusting the signal values of the 2D depth-related signal within the 2D region by applying the gain value to the signal values.
Optionally, the depth processor is arranged for adjusting the signal values of the 2D auxiliary signal within the 2D region based on the adjustment value so as to establish the depth reduction in the display region.
Optionally, the depth processor is arranged for adjusting the signal values of the 2D auxiliary signal within the 2D region based alpha-blending the signal values with a neutral depth value.
Optionally, the depth processor is arranged for establishing a gradual transition between the adjusted signal values within the 2D region and non-adjusted signal values outside the 2D region.
Optionally, the gradual transition is a substantially first-order linear transition or second-order non-linear transition.
Optionally, the system further comprises an image processor for:
Optionally, the image enhancement technique is at least one of the group of: a contrast enhancement, a sharpness adjustment, and a temporal filtering.
These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter. In the drawings,
a shows a 2D image signal comprising subtitles;
b shows a 2D depth signal corresponding to the 2D image signal;
a shows a GUI for enabling a user to establish a user-defined 2D region;
b shows the 2D depth signal reflecting the GUI;
a shows the user establishing the user-defined 2D region using the GUI;
b shows the region definer establishing the user-defined 2D region as the 2D region, and the depth processor establishing the depth reduction in the display region;
a shows a close-up view of the 2D depth signal without depth reduction;
b shows a close-up view of the 2D depth signal with depth reduction;
c illustrates the display region and a transition region; and
It should be noted that items which have the same reference numbers in different Figures, have the same structural features and the same functions, or are the same signals. Where the function and/or structure of such an item has been explained, there is no necessity for repeated explanation thereof in the detailed description.
The system 100 comprises a signal input 120 for obtaining the 2D image signal 122 and the 2D auxiliary signal 122. The system 100 further comprises a region definer 140 for defining a 2D region 142 in the 2D auxiliary signal, the 2D region corresponding to a display region on a display plane of the 3D display when 3D rendering the 2D image signal. The 2D region thus has a shape and a position. The 2D region may be constituted by region parameters describing an outline of the 2D region. The region parameters may be position parameters. The 2D region corresponds to a display region on a display plane of the 3D display when 3D rendering the 2D image signal. In other words, in a display region on the display plane of the 3D display, the depth as perceived by the user is established by the signal values of the 2D auxiliary signal within the 2D region. The display region is a region on the display plane in that it extends in width and in height on the display plane. The term display plane refers to the plane coinciding with the main light emitting surface of the 3D display and having a substantially same depth, i.e., corresponding to that of the main light emitting surface.
The system 100 further comprises a depth processor 160 for obtaining a depth reduction parameter 162, the depth reduction parameter representing a desired amount of depth reduction in the display region when 3D rendering the 2D image signal. The depth reduction parameter may be obtained internally, e.g., by being established by another part of the system or by being pre-set. The depth reduction parameter may also be obtained externally, e.g., from a user, as will be discussed further onward. The depth reduction parameter represents a desired amount of depth reduction in the display region when 3D rendering the 2D image signal. Here, the adjective desired refers to the depth reduction parameter having established in order to effect said depth reduction. The term depth reduction refers to depth within the display region being nearer to a neutral display depth, i.e., resulting in the content being established less protruding from, or caving into the 3D display.
The depth processor 160 is arranged for deriving an adjustment value from the depth reduction parameter for enabling establishing the depth reduction in the display region by adjusting signal values of the 2D auxiliary signal within the 2D region based on the adjustment value. Thus, the adjustment parameter is arranged for, when adjusting signal values of the 2D auxiliary signal within the 2D region based on the adjustment value, establishing the depth reduction in the display region. Consequently, the depth reduction is effected after said adjusting of the signal values.
It is noted that the depth processor 160 may be arranged for actually adjusting the signal values of the 2D auxiliary signal within the 2D region based on the adjustment value. This is in fact shown in
Although not shown in
The system 100 may further comprise a user interface subsystem 180 for enabling a user to establish a user-defined 2D region 182. For that purpose, the user interface subsystem 180 may be arranged for establishing a Graphical User Interface (GUI) on the 3D display 200 so as to enable the user to establish the user-defined 2D region 182 using the GUI. For example, the GUI may enable the user to define a vertical position on screen below which the depth is reduced. Effectively, the region below the vertical position constitutes the user-defined 2D region 182. The GUI may also enable the user to define a top side, bottom side, left side, and right side of a rectangular user-defined 2D region 182 using, e.g., position sliders corresponding to respective positions of the respective sides.
The region definer 140 may be arranged for defining the 2D region 142 based on the user-defined 2D region 182. For example, the region definer 140 may define the 2D region 142 to be equal to the user-defined 2D region 182. Hence, the user may have full control over the 2D region, and he/she can define the 2D region by establishing the user-defined 2D region 182. Alternatively, the region definer 140 may define the 2D region 142 based on the user-defined 2D region 182 by, e.g., initializing the 2D region with the user-defined 2D region 182, or using the user-defined 2D region 182 in any other suitable manner to define the 2D region 142. Essentially, the 2D region 142 thus constitutes a user configurable depth-reduced area within the 3D image signal.
Alternatively or additionally, the region definer 140 may comprise an overlay detector for detecting an overlay in the 3D image signal 122, 124, and the user interface subsystem 180 may be arranged for enabling the user to establish the user-defined 2D region 182 based on the detected overlay. For example, the user may be shown the detected overlay, i.e., in the form of an outline or position indicators, thereby enabling the user to base his/her establishing of the user-defined 2D region 182 on the detected overlay. The user interface subsystem 180 may also use the detected overlay to initialize the user-defined 2D region 182. Consequently, the detected overlay may provide an initial 2D region, and the user may adjust the initial 2D region so as to establish the user-defined 2D region 182. Alternatively or additionally, the user interface subsystem 180 may establish a grid for providing the user with snap-to-grid functionality when establishing the user-defined 2D region 182. Hence, the user may be guided towards establishing the user-defined 2D region 182.
Alternatively or additionally, the user interface subsystem 180 may be arranged for enabling the user to specify the desired amount of depth reduction in the display region, thereby establishing the depth reduction parameter 162. For example, the user may adjust a depth reduction slider. It is noted that instead of the user establishing the depth reduction parameter 162, the depth reduction parameter 162 may also be pre-set or by determined by the system 100. For example, the depth reduction parameter 162 may depend on an overall amount of depth in the 3D image signal.
The signal input 120 may be arranged for obtaining metadata indicative of a pre-defined 2D region, and the region definer 140 may be arranged for defining the 2D region based on the pre-defined 2D region. The pre-defined 2D region may be provided by earlier or previous system or device in the signal transmission chain. It is noted that the system 100 may, in turn, be arranged for providing the 2D region as defined by the region definer 140 and/or the adjustment value to a later or next system or device in the signal transmission chain, e.g., in the form of further metadata.
The depth processor 160 may be arranged for deriving an offset value from the depth reduction parameter 162 to enable adjusting the signal values of the 2D auxiliary signal 124 within the 2D region by applying said offset value to the signal values. The offset may be a depth-related offset in case the 2D auxiliary signal 124 is a 2D depth-related signal. As such, the offset value may be added and/or subtracted from depth-related signals of the 2D depth-related signal within the 2D region. The offset may also be a disparity offset in case the 2D auxiliary signal 124 is another 2D image signal. The disparity offset may be used to horizontally displace image values of the 2D auxiliary signal 124 in the 2D region. In case the 2D auxiliary signal 124 is a 2D depth-related signal, the depth processor 160 may also be arranged for deriving a gain value from the depth reduction parameter 162 to enable adjusting the depth-related values of the 2D depth-related signal within the 2D region by applying the gain value to the depth-related values. As such, the gain value may be multiplied with depth-related signals of the 2D depth-related signal within the 2D region. The depth processor 160 may be arranged for deriving both a gain value and an offset value from the depth reduction parameter 162. The offset may be first applied, and then the gain, or vice versa.
In case the depth processor 160 is arranged for adjusting the signal values of the 2D auxiliary signal within the 2D region based on the adjustment value, the depth processor 160 may perform said adjusting based on an alpha-blending the signal values with a neutral depth value. The alpha-value in the alpha-blending may be derived from the depth reduction parameter 162. It is noted that alpha-blending is known per se from the field of image processing. Furthermore, the depth processor 160 may be arranged for establishing a gradual transition between the adjusted signal values within the 2D region and non-adjusted signal values outside the 2D region. Hence, a transition region is established surrounding the 2D region in which the gradual transition is effected. Advantageously, a perception of breakup is avoided or reduced which would otherwise occur if an object extends both into the 2D region as well as outside of said region. The gradual transition may be a substantially first-order linear transition or second-order non-linear transition.
Although not shown in
It is noted that the term image signal refers to a signal representing at least one image. The image signal may also represent multiple images, e.g., a sequence of images such as a video sequence. Effectively, each image signal may thus constitute a video signal.
The operation of the system 100 and the method 300 may be further explained in reference to
a shows an example of the user interface subsystem 180 establishing a GUI on screen for enabling the user to establish a user-defined 2D region. The GUI comprises a slider termed “Border factor” which enables the user to specify the desired amount of depth reduction in the display region. The GUI further comprises four sliders enabling the user to establish multiple user-defined 2D regions, i.e., one at every side of the 2D image signal 122. If all four user-defined 2D regions are established by the user with a non-zero size, said regions together have a shape of a window frame and thus effectively constitute a single user-defined 2D region. It is noted that many alternatives are conceivable for enabling the user to establish the user-defined 2D region(s). By operating the sliders suitable, the user can thus establish the user-defined 2D region.
a shows the user completing the adjustments of the slider by establishing a user-defined 2D region at the bottom of the screen, i.e., comprising in this particular example all 89 image lines from the bottom of the 2D image signal 122.
The adjusted 2D depth signal 124a of
It will be appreciated that, in accordance with the present invention, a system may be provided for processing a 3D image signal, the 3D image signal comprising a two-dimensional 2D image signal and a 2D auxiliary signal, the 2D auxiliary signal enabling 3D rendering of the 2D image signal on a 3D display, the system comprising:
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments.
In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2009616 | Oct 2012 | NL | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2013/070926 | 10/8/2013 | WO | 00 |
