The present invention relates to decoding of video data encoded by a scalable video coding. In particular, the present invention relates to parallelization of such a decoding.
A majority of present applications related to providing video content rely on a fixed format of the video signal, given by the spatial and temporal resolution of the encoded video sequence. Lately, however, a large variety of diverse client devices have emerged such as set-top boxes, computers, portable digital assistants (PDAs) or mobile phones. These devices differ substantially in terms of achievable spatial and temporal resolution and available computational power. Thus the video streams suitable for these devices are necessarily different. In order to provide a video service adapted for diverse viewing devices and channel bandwidths, the video stream must be encoded many times with different settings. Each combination of settings must yield a stream that targets the bandwidth of the channel carrying the stream to the consumer as well as the decode capability of the viewing device. If the original uncompressed stream is not available, the encoded stream must be transcoded, which means that it has to be decoded and then re-encoded with the appropriate new settings. This rapidly becomes prohibitively expensive, and requires considerable time and computational power.
Scalable video coding (SVC) provides an alternative to such a multiple encoding and/or transcoding. In particular, scalable video coding provides an encoded video stream, the parts of which can be removed in such a way that the resulting substream forms another valid video stream for a decoder. The substream represents the same content as the complete video stream, however, with a lower reconstruction quality. Thus the video may be encoded only once with a high efficiency codec. The resulting video stream would, when decoded, yield the full resolution video. Furthermore, if a lower resolution or bandwidth stream is needed, for instance, in order to reach further into the network or to target a lower performance device, a small portion of the encoded stream may be sent without any additional processing. This smaller substream is easier to decode and yields lower resolution video. In this way, the encoded video stream is capable of adapting itself to both the bandwidth of the channel and to the capabilities of the target device.
In general, the term “scalable video coding” refers to a set of tools facilitating the features described above rather than to a single video coding mechanism. The concept of video scalability has been the scope of research and standardization over past at least twenty years. Most of video coding standards such as H.262/MPEG-2, H.263, or MPEG-4 include tools for supporting selected scalability features. Lately, the recent video coding standard H.264/MPEG-4 AVC has also introduced an extension including a broad choice of tools supporting scalable video coding.
H.264/MPEG-4 AVC is a video coding standard based on hybrid coding. A video signal input to an encoder is a sequence of images called frames, each frame being a two-dimensional matrix of pixels. Each individual video frame is subdivided into smaller blocks consisting of a plurality of pixels. Typically, a macroblock (usually denoting a block of 16×16 pixels) is the basic image element for which the encoding is performed. However, various particular encoding steps may be performed for smaller image elements, denoted submacroblocks or simply blocks and having the size of, for instance, 8×8, 4×4, 16×8, etc. Usually, the encoding steps of a hybrid video coding include a spatial and/or a temporal prediction. Accordingly, each block to be encoded is first predicted using either the blocks in its spatial neighborhood or blocks from its temporal neighborhood, i.e. from previously encoded video frames. A block of differences between the block to be encoded and its prediction, also called block of prediction residuals or prediction error signal, is then calculated. Another encoding step is the transformation of a block of residuals from the spatial (pixel) domain into a frequency domain. The transformation aims at reducing the correlation of the input block. A further encoding step is quantization of the transform coefficients. In this step the actual lossy (irreversible) compression takes place. Usually, the compressed transform coefficient values are further losslessly compressed by means of an entropy coding. In addition, side information necessary for the reconstruction of the encoded video signal is encoded and provided together with the encoded video signal. This is, for example, information about the spatial and/or temporal prediction, amount of quantization, etc. The H.264/MPEG-4 AVC includes two functional layers, a Video Coding Layer (VCL) and a Network Abstraction Layer (NAL). The VCL provides the encoding functionality as briefly described above. The NAL encapsulates the encoded residuals together with the side information necessary for the decoding of video into standardized units, called NAL units, according to a prescribed syntax, and thus forms a bitstream of the encoded video signal.
The scalable video coding extension of H.264/MPEG-4 AVC supports the entire concept of the H.264/MPEG-4 AVC specification and, in addition provides various scalability tools. H.264/MPEG-4 AVC scalable video coding extension provides scalability temporally, spatially, and in terms of video quality. This means that it can yield decoded video at different frame rates, resolutions, or quality levels. In order to facilitate this, the scalable video coding extension has introduced a concept of layers. A base layer includes the lowest temporal, spatial and quality representation of the video stream. Enhancement layers encode additional information which, using the base layer as a starting point, can be employed to reconstruct higher quality, resolution, or temporal versions of the video during the decoding process. By decoding the base layer and a selected number of the subsequent enhancement layers required, a decoder can produce a video stream with certain desired characteristics.
Temporal scalability is provided in both H.264/MPEG-4 AVC and its SVC extension by means of hierarchical temporal prediction structure. Accordingly, the motion-compensated prediction is simply restricted to reference pictures with a temporal layer identifier smaller or equal to the temporal layer identifier of the picture to be predicted.
Spatial scalability in H.264/MPEG-4 SVC provides multiple layers, each spatial layer corresponding to a supported spatial resolution. In each spatial layer motion-compensated (inter) prediction or spatial (intra) prediction is employed in the same way as for the single layer encoding. However, in order to further improve coding efficiency, an additional inter-layer prediction mechanism is supported. The inter-layer prediction in H.264/MPEG-4 SVC includes i) inter-layer intra-prediction, which is prediction of reconstructed image signal of a layer based on reconstructed and upsampled image signal of a lower (reference) layer, ii) inter-layer motion prediction, which is prediction of macroblock modes and associated motion parameters of a layer from the corresponding information of a reference layer, and iii) inter-layer residual prediction, which is prediction of the residual (prediction error) signal of a layer from upsampled prediction error signal of a reference layer.
Quality scalability is a special case of spatial scalability with equal picture sizes of different layers, without upsampling. The refinement of higher enhancement layers is typically achieved by re-quantizing the residual signal in an enhancement layer with a smaller quantization step than the quantization step applied to the reference layer.
Further details on the H.264/MPEG-4 AVC and its scalable video coding extension are provided in Advanced Video Coding for Generic Audiovisual Services, ITU-T Rec. H.264 and ISO/IEC 14496-10 (MPEG-4 AVC), Version 8 and later, which is incorporated herein by reference.
In general, video codecs are used for compressing the video sequences in order to reduce the bandwidth required for transport of the encoded stream, or to reduce the storage space required to archive the encoded stream. However, sophisticated and bandwidth-efficient compression approaches are typically connected with increased computational requirements: the higher the compression ratio to be achieved, the more computational power is required. Fixing the tradeoff between bandwidth and computational requirements requires defining both the minimum channel bandwidth required for carrying the encoded stream and the minimum specification of the decoding device required to view the reconstructed stream. The computational requirements are especially critical due to a real-time nature of the video content and due to a possibly limited power and computational resources of the viewing devices.
The additional inter-layer dependencies introduced in the scalable video coding introduce additional delay in the decoding. As the decoding of each single layer is similar to a single layer decoding, a stream encoded by H.264/MPEG-4 SVC may be decoded by sequentially decoding each layer one by one. Such an arrangement is also termed as “multi-pass” since it passes multiple times through the same decoder.
As can be seen from
The above disadvantages can be overcome partially by decoding a scalable video coding stream in parallel.
However, the decoding system 900 of
In order to reduce the complexity of spatial scalable video decoding, document IIHong Shin, Haechul Choi, et. al., “Fast decoder for H.264 scalable video coding with selective up-sampling for spatial scalable video coding”, Optical Engineering Letters, vol. 47(7), SPIE, July 2008 discloses a method according to which the upsampling is performed on a per-macroblock basis, only for intra macroblocks (I_BL) having the intra base layer mode of H.264/SVC. With such a method, the complexity of decoding may be somewhat reduced. However, the method does not provide any means for reducing the initial latency.
Some other methods have been developed to support operation of a scalable video encoding and decoding. For instance, document US 2008/0165848 A1 discloses a method for performing upsampling in a scalable video coding. In particular, the upsampling approach such as interpolation or nearest neighbor copying is selected adaptively in accordance with the location of the pixel to be interpolated. This approach aims at improving the quality of decoded pictures. However it does not relate to a parallel decoding and provides no means for essentially reducing the initial latency and the decoding complexity.
Given these problems with the existing technology, it would be advantageous to provide a system capable of parallel decoding of a video stream encoded by a scalable video encoding with a decreased latency.
It is the particular approach of the present invention to decode multiple layers of a video stream encoded with a scalable video coding in parallel by starting decoding of an enhancement layer of a frame before decoding all macroblocks of the reference layer of the frame, wherein the second layer requires data from the first layer for decoding.
In accordance with a first aspect of the present invention, a method is provided for parallel decoding of a stream of data including a video signal encoded with a scalable video coding and including a plurality of layers, the video signal including a plurality of frames divided into blocks. The method comprises decoding block data of a first layer required for decoding a target block in a second layer and providing the block data required for decoding of the target block to a second layer decoder. It further comprises decoding the target block in the second layer before having decoded all blocks of the first layer and based on said required block data.
In accordance with a second aspect of the present invention, a decoder is provided for parallel decoding of a stream of data including a video signal encoded with a scalable video coding and including a plurality of layers, the video signal including a plurality of frames divided into blocks. The decoder includes a first layer decoder for decoding block data of a first layer required for decoding a target block in a second layer and for providing the block data required for decoding of the target block to a second layer decoder, and a second layer decoder for decoding the target block in the second layer before having decoded all blocks of the first layer and based on said required block data.
In accordance with a third aspect of the present invention, a computer program product comprising a computer-readable medium having a computer-readable program code embodied thereon is provided, the program code being adapted to carry out the present invention.
In accordance with a fourth aspect of the present invention, a semiconductor chip is provided implementing the decoder of the present invention.
The accompanying drawings are incorporated into and form a part of a specification to illustrate several embodiments of the present invention. These drawings together with the description serve to explain the principles of the invention. The drawings are only for the purpose of illustrating preferred and alternative examples of how the invention can be made and used and are not to be construed as limiting the invention to only the illustrated and described embodiments. Further features and advantages will become apparent from the following and more particular description of the various embodiments of the invention, as illustrated in the accompanying drawings, in which like reference numbers refer to like elements and wherein:
The present invention relates to a method and a device for parallel decoding of a video bitstream encoded by a scalable video coding. A stream encoded by a scalable video coding includes typically two or more subs-streams, each sub-stream comprising data of a particular layer. Parallelization of the decoding is provided at the sub-stream (layer) level. In particular, various scalable video coding layers may be decoded in parallel even if they are encoded utilizing inter-layer dependency. Such a parallelization at a sub-stream level may advantageously reduce the latency of decoding as well as decrease the memory requirements.
A method for parallel decoding of a video stream encoded with a scalable video coding, wherein the video stream includes at least two layers 210, 220 in which at least one block 225 is encoded by employing inter-layer prediction, comprises a step of decoding from a sub-stream including a first layer (reference layer) block data necessary for decoding of a target block 225 in a second layer (enhancement layer). As soon as this block data necessary for decoding of the second layer are decoded, the decoding of the target block may start. Starting of the decoding before all the blocks of the reference layer have been decoded enables reducing of the initial latency. The two layers 210 and 220 may be a base layer and an enhancement layer or two enhancement layers. The block data necessary for decoding of the target block may be, for instance, the reconstructed image pixels of the reference layers, or prediction error (residual) signal, or data related to type of prediction and motion information such as motion vectors. However, the present invention is not limited thereto and any block data that can be used for predicting block data of another layer may be employed instead. The requirement of a block data from a reference layer for decoding of another layer is typically indicated within the video stream encoded with the scalable video coding. Alternatively, such a requirement may be implicitly derived based on the received and already decoded data. Starting of the decoding of the target block as soon as the information required for its decoding is available enables a reduction of the initial decoding delay to less than one frame per layer.
In general, the decoded block information 215, possibly upsampled, may be made available by the reference layer decoder either by a direct pipelining it to the enhanced layer decoder or by writing the decoded block data into a memory buffer, to which the enhancer layer decoder has at least a reading access.
A subsystem such as 301, 302, . . . 30L includes all functions necessary for decoding of a layer of a scalable video coding bits-stream. These functions may differ according to the particular coding applied. For instance, in the context of H.264/MPEG-4 SVC, the decoding may include steps as described with reference to
In order to prepare the decoded data required for decoding of another layer, it may be necessary to upsample the data. The upsampling may be necessary either of the reconstructed image or of the prediction error signal (residuals). Furthermore, virtual base layer generation (VBL) may be generated in order to derive from the reference layer the block data required to decode the target block, the block data being related to motion estimation. Accordingly, motion vectors 811 are provided to the second layer decoding sub-system. Moreover, a type of prediction and partitioning for the motion estimation may be determined by a simple rescaling.
The modified upsampling is illustrated in
Decoding of the scalable video coding stream in parallel at the layer level provides another advantage of reducing the size of buffering means. For the decoding as described with reference to
Furthermore, the decoding of the prior art results in latency of decoding of L-1 number of layer frames before the decoding of the target layer (layer to be displayed in the final output) frame, where L is the number of layers in the SVC stream. The latency of the method described with reference to
The present invention is also applicable to decoding of more than two layers in parallel. In particular, the decoding of each enhancement layer may start block-wise as soon as the data of its reference layer required for decoding of a current block are available.
This enables, in a real time embedded environment, parallelization at the layer/sub-stream level, which allows reducing the initial latency of the decoding and requires smaller buffer capacity. Such a processing further reduces the processing delay, system cost, requirements of power by this reducing of the storage requirements.
In the prior art, the worst-case latency is L-1 number of frames, L being the number of frames, one per layer, in the SVC bitstream consisting of L layers in one access unit. The present invention may reduce this latency to at most (2*PicWidthlnMBRef+3) number of macroblocks per reference layer, where PicWidthlnMBRef is the number of macroblocks in one row of the frame in the reference layer.
Another embodiment of the invention relates to the implementation of the above described various embodiments using hardware and software. It is recognized that the various embodiments of the invention may be implemented or performed using computing devices (processors). A computing device or processor may for example be general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, etc. The various embodiments of the invention may also be performed or embodied by a combination of these devices.
Further, the various embodiments of the invention may also be implemented by means of software modules, which are executed by a processor or directly in hardware. Also a combination of software modules and a hardware implementation may be possible. The software modules may be stored on any kind of computer readable storage media, for example RAM, EPROM, EEPROM, flash memory, registers, hard disks, CD-ROM, DVD, etc.
Most of the examples have been outlined in relation to an H.264/AVC/SVC based video coding system, and the terminology mainly relates to the H.264/AVC terminology. However, this terminology and the description of the various embodiments with respect to H.264/AVC/SVC based coding is not intended to limit the principles and ideas of the invention to such systems. Also the detailed explanations of the encoding and decoding in compliance with the H.264/AVC standard are intended to better understand the exemplary embodiments described herein and should not be understood as limiting the invention to the described specific implementations of processes and functions in the video coding. Nevertheless, the improvements proposed herein may be readily applied in the video coding described.
Summarizing, a method and a decoder is provided for parallel decoding of a stream of data including a video signal encoded with a scalable video coding and including a plurality of layers. a block data of a first layer required for decoding a target block in a second layer are decoded at first. Then after the required data of the first layer has been decoded, the target block in the second layer is decoded based on said block data.
Number | Name | Date | Kind |
---|---|---|---|
5886736 | Chen | Mar 1999 | A |
20050259729 | Sun | Nov 2005 | A1 |
20080165848 | Ye et al. | Jul 2008 | A1 |
20110286526 | Nakagami et al. | Nov 2011 | A1 |
Entry |
---|
IIHong Shin, et al.; Fast decoder for H.264 scalable video coding with selective up-sampling for spatial scalable video coding; Optical Engineering; Jul. 2008; pp. 070502-1-070502-3; vol. 47(7). |
Number | Date | Country | |
---|---|---|---|
20110122944 A1 | May 2011 | US |