Patent Application
20120218256
The present invention relates to a method of generating a recommended depth value for use in displaying a graphics item over a three dimensional video; and to a method of operating a display device to display a graphics item over a three dimensional video.
Graphics items such as subtitles and on-screen display messages (OSDs) are often overlayed on top of video. In standard two-dimensional (2D) television, graphics items can simply be placed on top of the video. Whilst this might obscure video of interest, it does not generate any visual conflict. However, in the case of three-dimensional (3D) television, placing graphics items on top of the video without any consideration of the depth at which the graphics item is to be placed at can result in the graphics item itself being obscured by something in the 3D video image appearing in front of the graphics item. This causes a degree of visual dissonance that can significantly impact the overall impression of 3D television.
Some graphics items include “x,y” locations for placement of the graphics item on the screen and these are typically embedded into the graphics item information. 3D DVDs that are available today all place graphics items at a single, fixed depth (equivalent to the screen depth).
Some 3D video coding mechanisms utilize a grey-scale depth map. This is generated from views taken from different camera angles, and provides depth information for each pixel location, and is generated automatically by the encoding process.
The following references are believed to represent the state of the art:
www.displaydaily.com/2007/10/24/making-3d-movies-part-ii;
International Patent Application WO2008/115222 to Thomson Licensing;
International Patent Application WO2008/038205 to Koninklijke Philips Electronics N.V.;
International Patent Application WO2008/044191 to Koninklijke Philips Electronics N.V.;
Patent Abstracts of Japan JP 2004274125 to Sony Corp;
United States Patent Application US 2009/0027549 to Weisberger; and
United States Patent Application US 2008/0192067 to Koninklijke Philips Electronics N.V.
There is provided in accordance with an embodiment of the present invention a method of generating a recommended depth value for use in displaying a graphics item over a three dimensional video, the method including at a headend: receiving a three dimensional video including video frames; analyzing a sequence of said video frames of the three dimensional video in turn to produce a sequence of depth maps, each depth map in the sequence of depth maps being associated with timing data relating that depth map to a corresponding video frame in the sequence of video frames, each depth map including depth values, each depth value representing a depth of a pixel location in its corresponding video frame; selecting a region of depth maps in the sequence of depth maps; analyzing the region of the depth maps in the sequence to identify a furthest forward depth value for said region in the sequence of depth maps; and transmitting the furthest forward depth value as the recommended depth value for the region to a display device and region information describing the region.
Further, in accordance with an embodiment of the present invention, the method further includes: selecting at least one additional region of depth maps in the sequence of depth maps; analyzing the at least one additional region of the depth maps in the sequence to identify at least one additional furthest forward depth value for the at least one additional region; and transmitting the at least one additional furthest forward depth value as an additional recommended depth value for the at least one additional region and additional region information describing the at least one additional region.
Additionally, in accordance with an embodiment of the present invention, the method further includes: receiving details of the graphics item, the details comprising a two dimensional screen location where the graphics item is to be displayed and a size of the graphics item; and determining the region from the details of the graphics item.
Moreover, in accordance with an embodiment of the present invention, the method further includes: receiving a maximum depth value; comparing the furthest forward depth value with the maximum depth value; and transmitting the maximum depth value as the recommended depth value if the furthest forward depth value exceeds the maximum depth value, otherwise transmitting the furthest forward depth value as the recommended depth value.
Further, in accordance with an embodiment of the present invention, the method further includes: selecting an alternate region of depth maps in the sequence of depth maps; analyzing the alternate region of the depth maps in the sequence to identify an alternate furthest forward depth value for the alternate region; and transmitting the alternate furthest forward depth value as an alternate recommended depth value for the alternate region and alternate region information describing the alternate region.
Additionally, in accordance with an embodiment of the present invention, prediction of a change in depth value is used to identify the furthest forward depth.
Moreover, in accordance with an embodiment of the present invention, the recommended depth value is transmitted independently of the three dimensional video.
There is also provided in accordance with a further embodiment of the present invention, a method of operating a display device to display a graphics item over a three dimensional video, the method including: receiving the three dimensional video including video frames; receiving the graphics item; receiving a recommended depth value for use in displaying the graphics item over the three dimensional video; and displaying the graphics item over the three dimensional video at the recommended depth value, wherein the recommended depth value is generated according to a recommended depth value generation process executed at a headend, the recommended depth value generation process including: analyzing a sequence of the video frames in turn to produce a sequence of depth maps, each depth map in the sequence of depth maps being associated with timing data relating that depth map to a corresponding video frame in the sequence of video frames, each depth map including depth values, each depth value representing a depth of a pixel location in its corresponding frame; selecting a region of depth maps in the sequence of depth maps; analyzing the region of the depth maps in the sequence of depth maps to identify a furthest forward depth value for the region in the sequence of depth maps; and transmitting the furthest forward depth value as the recommended depth value for the region to the display device and region information describing the region.
Further, in accordance with an embodiment of the present invention, the recommended depth value generation process further includes at the headend: selecting an alternate region of depth maps in the sequence of depth maps; analyzing the alternate region of the depth maps in the sequence to identify an alternate furthest forward depth value for the alternate region; and transmitting the alternate furthest forward depth value as an alternate recommended depth value for the alternate region and alternate region information describing the alternate region to the display device; the method further including displaying the graphics item in a region of the three dimensional video described by the region information or in an alternate region of the three dimensional video described by the alternate region information.
Additionally, in accordance with an embodiment of the present invention, a determination is made by the display device as to which of the region and the alternate region to use for displaying the graphics item.
Moreover, in accordance with an embodiment of the present invention, the determination is based on information received from a user.
Further, in accordance with an embodiment of the present invention, the method further includes: comparing the recommended depth value with a maximum depth value; and causing the graphics item to disappear if the recommended depth value exceeds the maximum depth value.
Still further, in accordance with an embodiment of the present invention, the method further includes: comparing the recommended depth value and the alternate recommended depth value with a maximum depth value; and displaying the graphics item in the alternate region if the recommended depth value exceeds the maximum depth value but does not exceed the alternate recommended depth value.
Additionally, in accordance with an embodiment of the present invention, the recommended depth value generation process further includes at the headend: selecting at least one additional region of depth maps in the sequence of depth maps; analyzing the at least one additional region of the depth maps in the sub-sequence to identify at least one additional furthest forward depth value for the at least one additional region; and transmitting the at least one additional furthest forward depth value as an additional recommended depth value for the at least one additional region and additional region information describing the at least one additional region; the method further including: displaying the graphics item at a furthest forward depth value of the recommended depth value and the at least one additional recommended depth value.
The present invention will be understood and appreciated more fully from the following detailed description, taken in conjunction with the drawings in which:
Referring now to
Each frame of the 3D video data is associated with a time code that provides a time reference for identification and synchronization. For example, each frame may be associated with a SMPTE timecode (as defined in the SMPTE 12M specification of the Society of Motion Picture and Television Engineers (SMPTE)). The SMPTE time codes may in turn be mapped to MPEG presentation time stamps (PTS) (as defined in International Standard ISO/IEC 13818-1, Information technology—Generic coding of moving pictures and associated audio information: Systems—a metadata field in an MPEG-2 Transport Stream or Program Stream that is used to achieve synchronization of programs' separate elementary streams (for example video, audio, subtitles) when presented to the viewer)—using synchronized auxiliary data (as defined in ETSI TS 102 823 Digital Video Broadcasting (DVB); Specification for the carriage of synchronized auxiliary data in DVB transport streams.) Alternatively, each frame may be associated with an MPEG PTS.
The 3D video data may also be accompanied by camera metadata defining parameters of the camera(s) used to record the 3D video such as camera location, lens parameters, f-stop settings, focal distance, auto-focus information etc., and stereo camera parameters such as the relative separations and angle of the stereo cameras.)
The 3D video data and optional camera metadata is passed to depth extraction module 103, which is operable to produce a depth map for each frame of the 3D video data according to any known depth extraction technique (e.g. as described in “Depth Extraction System Using Stereo Pairs”, Ghaffar, R. et al., Lecture Notes in Computer Science, Springer Berlin/Heidelberg, ISSN 0302-9743, Volume 3212/2004). A depth map typically provides depth values (typically 24 or 32 bit) for pixel locations of a corresponding frame of 3D video. A depth value conveys the extracted estimate of the depth (or distance from the camera) of a pixel location in the 3D video frame. Each depth map is associated with timing data (e.g. SMPTE time code or MPEG PTS, as described previously) used to synchronize a depth map with the frame of 3D video from which it was generated. A depth map produced for each frame of the 3D video data represents a large amount of information. The depth maps and timing data 105 are stored in a memory (not shown) and then passed to processing module 107, together with processing parameters 117, when required for processing.
Processing parameters 117 enable the temporal and/or spatial filtering of the depth maps so that the quantity of data being processed can be reduced. The depth maps can be temporally filtered by processing selected sequences of depth maps corresponding to a selected sequence of the 3D video (e.g. if a graphics item is only to be displayed for a limited period of time). The depth maps can be spatially filtered by processing selected regions of depth maps (e.g. if a graphics item only overlays part of the 3D video). Examples of typical processing parameters 117 include:
Other processing parameters 117 will be apparent to someone skilled in the art. Processing module 107 processes depth maps in order to extract information that is used to place graphics in the 3D video and convey the information in an effective format to display device 111. This involves identifying areas/regions within 3D video frame sequences that will be used for graphics placement and a recommended depth at which graphics could be placed.
The output of processing module 107 is one or more streams of processed depth data and associated timing data 109 that defines a recommended depth at which to place graphics for one or more frame sequences of the 3D video; or one or more regions of the 3D video.
Graphics items typically include subtitles; closed captions; asynchronous messages (e.g. event notifications, tuner conflict notifications); emergency announcements; logo bugs; interactive graphics applications; information banners; etc.
The processed depth data and timing data 109 is transmitted from the headend to a display device 111 via a network 113. The processed depth data and timing data 109 can be compressed prior to transmission. Display device 111 is typically an integrated receiver decoder (IRD); set top box (STB), digital video recorder (DVR) etc. connected in operation to a display such as a television. Network 113 is typically a one-way or two-way communication network, e.g. one or more of a satellite based communication network; a cable based communication network; a terrestrial broadcast television network; a telephony based communication network; a mobile telephony based communication network; an Internet Protocol (IP) television broadcast network; a computer based communication network; etc.
The graphics data and the 3D video data 115 are also transmitted from the headend to display device 111 via network 113. The graphics data and the 3D video data 115 can be transmitted separately from the processed depth data and associated timing data 109. Typically, the graphics data and the 3D video data 115 are compressed prior to transmission.
Display device 111 receives the processed depth data and timing data 109 and the 3D video and graphics data 115. When graphics are to be rendered over the 3D video, the graphics data is provided to a rendering engine within display device 111 together with the processed depth data. The rendering engine then generates the appropriate images and combines them with the 3D video at the recommended depth indicated by the processed depth data using standard combination techniques. It will be remembered that the processed depth data is accompanied by timing data allowing it to be associated and synchronized with the 3D video.
For example, whilst a viewer is watching a program, display device 111 identifies a conflict between the event currently be viewed and an event to be recorded. In order to display a graphic message to communicate this tuner conflict to the viewer, display device 111 consults the processed depth data and timing data 109 and locates the region associated with the OSD message class for the tuner conflict. This provides display device 111 with screen location and depth information. This depth information is then provided to the graphics rendering software and hardware together with the graphic to be displayed. The graphics rendering software converts the graphics and location and depth information into images that are perceived by the viewer as being placed at the intended depth and location.
The operation of processing module 107 will now be described in more detail. As was mentioned previously, processing module 107 processes depth maps in order to extract information from the depth maps that is used to place one or more graphics items in the 3D video and convey the information in an effective format to display device 111. This information is typically includes a recommended depth at which to place each graphics item and information about the region that the recommended depth relates to. The region information can be in the form of a region tag (as mentioned previously); or a screen area (defined by region size and location as mentioned previously; or by a vector outline). Other ways of describing a region will be apparent to someone skilled in the art.
Referring to
Next (step 203), processing module 107 uses the details of the graphics item to derive the frame sequence, region location and size processing parameters. In certain embodiments, processing module 107 may not derive the region size and location parameters (e.g. if details of the graphics item were not provided). In such a case, processing module 107 would use the entire depth map rather than just a region of the depth map. Processing module 107 may alternatively or additionally use the depth maps of all the frames of the 3D video in the processing rather than just the depth maps from a sequence of frames.
Then (step 205), processing module 107 extracts the appropriate depth maps for the (derived) frame sequence and analyses the (derived) region of the extracted depth maps (step 207) in order to calculate a furthest forward depth of pixels in the 3D video where the graphics item is to be placed so as not to conflict with 3D video image. That is, if the graphics item was placed behind (or backwards from) the furthest forward depth, a depth disparity may occur (e.g. if a graphics item was overlaid on an object in the 3D video but at a depth behind that object, the object may look as if some pixels of the object had been removed.) Typically, calculating the furthest forward depth involves calculating the depth of pixels in the derived region for frames in the sequence and extracting the furthest forward depth identified in the calculation for use as the recommended depth for the graphics item.
Next (step 209), processing module 107 packages the processed depth data (including the recommended depth for a graphics item overlaying that region; and the region location and size if appropriate) together with the timing data for the processed depth data and transmits it to display device 111 (step 211).
The above described process is repeated for each graphics item that could be selected (e.g. by a user or operator or broadcaster) to overlay the 3D video.
It can therefore be seen that it is not necessary to transmit entire depth maps of a 3D video to display device 111. Rather, only processed depth data (which includes recommended depths for graphics items) is transmitted representing a substantial saving in the amount of data that is transmitted from the headend and therefore processed by display device 111.
The above described process is suitable, for example, for calculating the recommended depth for subtitles to accompany the 3D video. In the simplest case, the 3D video scene may have a static depth throughout the video (e.g. a newsreader sitting at a desk). In this case, it may be possible to process a single depth map (or a single (short) sequence of depth maps) in order to determine a recommended depth at which to place the subtitles (or any other graphic item). This depth value can then remain constant throughout the 3D video.
However, objects and people in the 3D video often move around resulting in changing and varying distances between the camera and the nearest object (hence resulting in varying depth maps). Whilst it would be possible to calculate and adapt the depth of a graphics item on a frame by frame basis, this may result in a graphics item moving extensively in depth, which may be visually unappealing. It is therefore desirable, in certain embodiments of the present invention, to minimize or eliminate the movement of certain graphics items.
As described above, by analyzing a sequence of depth maps, it is possible to identify the furthest forward depth and choose that depth as the recommended depth for the duration of the 3D video; or for the duration of a sequence/scene of the 3D video; or for the duration of display of the graphics item. In this way, the graphics item can be placed in the 3D video without changing in depth on a frame by frame basis.
In the case of subtitles, processing module 107 may extract all depth maps for the 3D video and compare a region of each extracted depth map to be used for the subtitles in order to calculate the furthest forward depth that the subtitles can be placed without conflicting with 3D video image. Alternatively, each subtitle to be displayed could be considered as a separate graphics item, each associated with a defined sequence of frames of the 3D video. Details of the screen location and size for each subtitle will typically already be known from a subtitle generation process but subtitling of live content (e.g. news) can also be handled. These details can be passed to processing module 107 in step 201.
In alternative embodiments, other processing parameters (as described above) can be provided to processing module 107 for processing module 107 to use when comparing depth maps to calculate the processed depth data. For example, a depth limit (maximum depth value) can be provided to processing module 107 so that if the calculated recommended depth exceeds the depth limit, the depth limit could be used as the recommended depth. Where region movement and/or region size adjustment and/or alternate region parameters have been provided, the (x,y) positioning of the graphics item could be controlled so that the graphics item could be moved and/or resized according to those parameters in order to find a screen location for the graphics item where the recommended depth (which will be recalculated for the new region location and/or size) is within the depth limit. This is an extension of the ideas described above, whereby the (x,y) position of the region is used to vary the z positioning, and an optimization is made for the motion in all three directions. The new region location and/or size would typically be transmitted with the recommended depth as part of the processed depth data.
Alternatively or additionally, the depth movement parameter could be used to enable the recommended depth to vary across a sequence of frames. Thus, for instance, a subtitle could move slowly towards the camera during its display, minimizing its overall movement and speed of movement whilst also avoiding conflict with any object in the video.
In some embodiments, other information relevant to the placement of the graphics item can also be included as part of the processed depth data. For example, it may be possible to place the graphics item in a different screen location (identified by the alternate region processing parameter) in which case the recommended depth for the graphics item for the alternate region can also be calculated and included in the processed depth data.
In certain embodiments, object tracking functionality (well known from the field of image vision processing) may be used to track objects in the 3D video. It may be desirable to include a graphics item associated with an object that is being tracked (e.g. labeling the ball as such in a fast moving game such as a golf or hockey; or labeling a player with their name). In these embodiments, the output of object tracking could be provided to the processing module 107 as the region location parameter (with a suitable offset to avoid conflicting with the actual object being tracked, e.g. using the region movement parameter) enabling the depths for a graphics item labeling the tracked object to be calculated.
Typically, the depth extraction and processing take place at the time the 3D video is ingested into the headend. This is often significantly in advance of the transmission. In such embodiments, the processed depth data can be provided in advance of the transmission of the 3D video. Alternatively, the processed depth data can be provided at the same time as the 3D video. The processed depth data is typically provided as a separate stream of data to the 3D video data.
In certain embodiments (e.g. for live broadcast events) processing module 107 processes the depth maps in real time. Due to the normal encoding delays at the headend, at least a second (and sometimes up to ten seconds) of the 3D video can be analyzed in order to calculate the processed depth data. Well known techniques such as regression analysis can be used to extrapolate depth motions of objects in the 3D video beyond those obtained from the analyzed depth maps and the extrapolated depth motions used to when calculating the recommended depths. The processed depth data is transmitted to display device 111 at the same time as the 3D video, typically once every second. However, in alternative embodiments the processed depth data may only be sent at less-regular intervals (e.g. every 5 s) with a check being performed more regularly (e.g. every 1 s) to see if the recommended depth has changed. If it has changed then it can be transmitted immediately.
It will be remembered that when graphics are to be rendered over the 3D video, the graphics data is provided to a rendering engine within display device 111 together with the processed depth data. The rendering engine then generates the appropriate images and combines them with the 3D video at the depth indicated by the processed depth data. In certain embodiments, display device 111 may be able to adapt where graphics items are placed based on user set preferences. For example, a user may be able to set a user preferred depth for graphics items that overrides the originally received recommended depth (assuming no conflict is indicated (e.g. where the user preferred depth is further forward than the transmitted recommended depth)). Furthermore, a user may also prefer to have a graphics item move to an alternate location on the screen (or disappear altogether) if the received recommended depth exceeds the user preferred depth (either backwards or forwards). Moreover, a user may prefer to always have the graphics displayed at the minimum depth, rather than see the graphics item moving in depth. Display device 111 may also be configured to make certain graphics items disappear if they conflict with other graphics items, or to correctly combine the graphics items at their relative depths if transparency options make such an alternative possible. The display device 111 may also be configured with different depth movement and/or region movement parameters.
The head-end may employ one of several alternative methods for region identification:
As mentioned previously, a region can be described using one or more of a range of methods depending on the expected usage of the region. These can include a region tag that is used by display device 111, e.g. “NN” to indicate a Now-Next banner region, or a screen area from regular shapes such as a square or rectangle to complex shapes defined using a vector outline. As such, each region could include a range of descriptive information.
It is not necessary that regions always correspond directly to the location of a graphics item (graphical asset). There may be cases (e.g. for complex interactive applications) where the range of different choices in graphics items that may be rendered at a user's request are so variable as to prevent simplistic regions for each graphics item. Instead, display device 111 may receive details of multiple regions (each with a recommended depth), and then, using information about the graphics item(s), identify the region(s) that the graphics item(s) will overlap with. Display device 111 would then choose an appropriate depth based on an analysis of the recommended depth for all regions. For example, display device 111 may use the graphics item to identify the screen area that the graphics item will cover, and then consult the set of regions. Display device 111 would then identify which region(s) is/are covered by the graphics item, and from this identification process extract the appropriate depth positioning information. Typically, the region(s) include descriptions that allow their screen location/area to be identified.
It is also worth noting that a ‘near plane’ and a ‘far plane’ can be defined for graphics items. A ‘near plane’ is the nearest plane to a user viewing a three dimensional video (i.e. out of the screen) in which to place a graphics item. A ‘far plane’ is the furthest plane from a user viewing a three dimensional video (i.e. into the screen) in which to place a graphics item.
It is appreciated that software components of the present invention may, if desired, be implemented in ROM (read only memory) form. The software components may, generally, be implemented in hardware, if desired, using conventional techniques. It is further appreciated that the software components may be instantiated, for example: as a computer program product; on a tangible medium; or as a signal interpretable by an appropriate computer.
It is appreciated that various features of the invention which are, for clarity, described in the contexts of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features of the invention which are, for brevity, described in the context of a single embodiment may also be provided separately or in any suitable subcombination.
It will be appreciated by persons skilled in the art that the present invention is not limited by what has been particularly shown and described hereinabove. Rather the scope of the invention is defined only by the claims which follow:
| Number | Date | Country | Kind |
|---|---|---|---|
| 0915658.9 | Sep 2009 | GB | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/IB2010/052664 | 6/15/2010 | WO | 00 | 5/11/2012 |
