Patent Application
20070009024
The present invention is in the field of video encoding/decoding and, in particular, suitable for scalable video coding schemes.
Video sequences consisting of several subsequent frames or “single-pictures”, or, alternatively, one or more still pictures can be acquired with any desired resolution and accuracy. In general, the resolution and accuracy for acquiring a picture is determined by, for example, a digital camera, a camcorder, or computer equipment for generating virtual scenes.
In this context, the resolution is defined by the number of pixels of a picture. When the picture has a high number of pixels, it has a high resolution and, accordingly, a large size, when the area occupied by a pixel on a display screen is set to a certain value. The accuracy of a picture or, particularly, the accuracy of a pixel is defined by the number of bits required to encode this pixel. Generally, a high number of bits per pixel results in a higher accuracy for the pixel, or in a higher number of values representable by the pixels.
As a simple example, providing one bit per pixel means, when no compression algorithm is used that the pixel can only be white or black. It is clear that providing for example 3 bits per pixel enhances the accuracy of this pixel and, therefore, the picture quality, since the pixel can, for example, have one of 8 different grey values or colour values. Generally, the pixel accuracy or the total bit per pixel value is determined by a quantizer in the picture encoder.
It is to be noted here that a picture or video encoder/decoder scheme can be based on MPEG 4 or H.264/AVC. Naturally, several picture encoding/decoding schemes are also known in the context of JPEG. All these encoders have in common that the picture or a sequence of pictures are compressed to reduce redundancy in the picture file. A good way to do this is the application of motion-compensated prediction, which is known in the art. Additionally, most encoding schemes also rely on a lossy compression scheme, which is, in general, a quantization of—in the context of MPEG-4—block-based DCT coefficients. Naturally, a large quantizer step size results in lower quantized values, which can be represented by a smaller number of bits compared to quantized values obtained by a small quantizer step size. Such a small quantizer step size results in a higher range of quantized values so that a higher number of bits are required for representing them. In particular, when a subsequent entropy encoding of quantized values is used, one can state in general that smaller values result in a higher coding gain compared to higher values.
As stated above, it is possible to acquire pictures with any desired resolution (size) and accuracy (bits per pixel). Instead of the large progress in encoder compression gains, it is, of course, not possible to encode a large and very accurate picture and to simultaneously obtain a very small bit rate being small enough for a low-bandwidth transmission channel.
In this context, one has two general possibilities to reduce the number of bits per picture so that a picture can be transmitted via a transmission channel having a limited bandwidth. To this end, reference is made to
Another possibility is shown at the right hand side in
For reducing the resolution to obtain picture 31, one can use more or less sophisticated decimation filters such as filters combining two adjacent pixels in a row or in a column so that the resulting picture 31 is smaller than the original picture 30.
For reducing pixel accuracy, one can increase a quantizer step size in a picture encoder or simply delete a certain number of low significant bits per pixel until a certain bit rate is achieved.
Pictures 31 and 32 both have a certain subjective quality. When one looks at picture 31, one will see the very high pixel accuracy. However, the subjective impression is affected by the small size of picture 31. When one, however, looks at picture 32, one will be satisfied with the large size of this picture. Nevertheless, the reduced accuracy of the bits will forbid a fully satisfying subjective impression.
Naturally, all combinations of the two general possibilities shown in
The decisions shown in
The easiest way would be to select the target resolution identical to the source resolution. In such a case, only the bit rate needs to be decided. This procedure would correspond to the right alternative of
When, however, the bit rate of the target, i.e., the low-bandwidth channel or low-bandwidth replay device is too low so that the accuracy would be affected too much, changing the resolution by decimating, i.e., using the first alternative in
In many cases, however, the replay display is not small enough that a decimation from the size of picture 30 to the size of picture 31 will result in the required low number of bits per picture. In this case, in order to not impair the pixel accuracy too much, a resolution reduction to a target resolution has to be performed, which is normally smaller than the size of the intended replay device.
Naturally, this procedure is disadvantageous in that sometimes pictures become too small so that the very smallness of the picture dominates—and destroys—the subjective picture quality, which impression can not be compensated for by a high pixel accuracy.
It is the object of the present invention to provide an encoding/decoding concept which results in a better subjective quality while keeping a bit rate, which is low enough for the encoded data stream to be transmitted over a transmission channel having a limited bandwidth.
In accordance with a first aspect of the invention, this object is achieved by an apparatus for producing a data stream having encoded picture data representing a single picture or a plurality of pictures, comprising: a provider for providing the encoded picture data, a picture represented by the encoded picture data having a picture resolution defined by a number of pixels included in the picture; and a resizing information supplier for supplying resizing information, the resizing information defining a resizing resolution of the picture when replaying the picture, the resizing resolution being higher than the picture resolution.
In accordance with a second aspect of the invention, this object is achieved by an apparatus for processing a data stream having encoded picture data representing a single picture or a plurality of pictures, comprising: a receiver for receiving the data stream and resizing information, the resizing information defining a resizing resolution of the picture when replaying the picture, the resizing resolution being higher than a picture resolution of a picture in the encoded picture data; and a resizer for resizing a picture in response to the resizing information to obtain a resized picture based on the picture having a picture resolution, wherein a number of pixels of the resized picture is higher than a number of pixels of the corresponding picture in the picture resolution.
In accordance with a third aspect of the invention, this object is achieved by a method of producing a data stream having encoded picture data representing a single picture or a plurality of pictures, comprising: providing the encoded picture data, a picture represented by the encoded picture data having a picture resolution defined by a number of pixels included in the picture; and supplying resizing information, the resizing information defining a resizing resolution of the picture when replaying the picture, the resizing resolution being higher than the picture resolution.
In accordance with a fourth aspect of the invention, this object is achieved by a method of processing a data stream having encoded picture data representing a single picture or a plurality of pictures, comprising: receiving the data stream and resizing information, the resizing information defining a resizing resolution of the picture when replaying the picture, the resizing resolution being higher than a picture resolution of a picture in the encoded picture data; and resizing a picture in response to the resizing information to obtain a resized picture based on the picture having a picture resolution, wherein a number of pixels of the resized picture is higher than a number of pixels of the corresponding picture in the picture resolution.
In accordance with a fifth aspect of the invention, this object is achieved by a computer program having a program code for performing a method of producing a data stream having encoded picture data representing a single picture or a plurality of pictures, when the computer program runs on a computer, the method comprising: providing the encoded picture data, a picture represented by the encoded picture data having a picture resolution defined by a number of pixels included in the picture; and supplying resizing information, the resizing information defining a resizing resolution of the picture when replaying the picture, the resizing resolution being higher than the picture resolution.
In accordance with a sixth aspect of the invention, this object is achieved by a computer program having a program code for performing a method of processing a data stream having encoded picture data representing a single picture or a plurality of pictures, when the computer program runs on a computer, the method comprising: receiving the data stream and resizing information, the resizing information defining a resizing resolution of the picture when replaying the picture, the resizing resolution being higher than a picture resolution of a picture in the encoded picture data; and resizing a picture in response to the resizing information to obtain a resized picture based on the picture having a picture resolution, wherein a number of pixels of the resized picture is higher than a number of pixels of the corresponding picture in the picture resolution.
In accordance with a seventh aspect of the invention, this object is achieved by a data file or stream having encoded picture data representing a picture or a plurality of pictures, a picture represented by the encoded picture data having a picture resolution defined by a number of pixels included in the picture, and having resizing information, the resizing information defining a resizing resolution of a picture when replaying the picture, the resizing resolution being higher than the picture resolution.
In accordance with a seventh aspect of the invention, this object is achieved by a storage device having stored thereon a data file or stream having encoded picture data representing a picture or a plurality of pictures, a picture represented by the encoded picture data having a picture resolution defined by a number of pixels included in the picture, and having resizing information, the resizing information defining a resizing resolution of a picture when replaying the picture, the resizing resolution being higher than the picture resolution.
The present invention is based on the finding that the subjective quality of the displayed picture can be enhanced by resizing the picture by a certain degree on the decoder/display side of the encoding/decoding scheme. In this connection, resizing does not mean a simple zoom operation, which does not influence the resolution of this picture but only influences the area, a certain pixel occupies on a display screen. Instead, the resizing operation results in a higher resolution of the displayed picture or the sequence of displayed pictures. The pixels not included in the transmitted picture but included in the displayed picture are calculated before displaying the picture using information transmitted from an encoder to the decoder and, in particular, are derived from the transmitted pixels for example by interpolation. It has been discovered, however, that resizing for example by interpolation can not be performed by an unlimited degree. The subjective quality of a picture is only better than the subjective quality of the non-resized picture within a certain resizing range. The resizing range or, stated in other words, the amount of resizing giving an optimum subjective quality depends on a high number of different factors. These factors include the picture itself, the pixel accuracy of the low resolution picture, encoding algorithms used for encoding the low resolution picture and characteristics of the display device intended to display the resized picture. In view of this, the complicated interdependence between the above example factors makes it impossible to prescribe or predefine a certain resizing amount valid for all combinations of picture data resolution/accuracy and transmission/display settings.
In accordance with the present invention, resizing information is supplied, which is preferably generated by the user producing the data stream. The user or generator of the data stream having the reduced picture resolution can easily determine the optimum resizing information by, for example, performing some empirical tests. Although it might look as a heavy burden at a first glance, the determination of the optimum resizing parameter only has to be performed once for a similar picture material and similar encoder/decoder/display settings.
Supplying the optimum resizing information on the encoder-side and transmitting this resizing information preferably together with the encoded picture data to a decoder results in an optimum achievable subjective quality on decoder/display side. Thus, the inventive supply of the resizing information in addition to the encoded picture data helps to avoid any surprising subjective quality degradations incurred by a resizing operation, which increases the resolution too much. Such a situation can occur, when a display device is set to interpolate each incoming picture irrespective of the resolution of the incoming picture to the resolution of the display device so that a full-size video reproduction is obtained. Such a situation is especially disadvantageous, because the viewer does not appreciate the high size, but is annoyed by a quite shady picture showing too much interpolation artefacts.
On the other hand, the inventive picture data adaptive signalling of the resizing information reduces situations, which can occur, when the decoder operates too conservative. A straightforward procedure would be to do no resize at all. It is, however, clear that subjective quality enhancement possibilities are completely given away. This situation also occurs, when a default resizing degree of, for example 5%, is fixed in a display device. In this case, the user can see a slight improvement but will not profit from the full quality, which would have been possible and which is signalled in accordance with the present invention.
In the field of video encoding, there is a highly competitive market and, in addition, the restriction of more and more decreasing available transmission bandwidth resources because of the extremely growing need for digital transmission channels. In view of this, the ultimate goal—with respect to the scarce transmission resources—is to reduce bandwidth as far as possible. On the other hand, however, a product can only survive on the market place, when it has an improved quality compared to products of the competitors. An advantage over products of the competitors can be, of course, achieved, when one can say that one has the same bandwidth needs as the competitor's product but one has a better product quality than the competitor.
The only price to be paid for the optimum subjective quality obtained by the present invention is, on the encoder-side, the supply of the resizing information and, on the decoder-side an interpolator controllable by the resizing information. Additionally, the resizing information has to be transmitted. Since this information can be transmitted with a very low number of bits such as three bits for signalling eight percentage ratios between an increase of 0% to 100%, the additional need of channel capacity can be neglected. Additionally, any encoder-related efforts do not matter, since transmission efficiency and subjective display quality are the issues that count in a highly competitive market. Regarding the additional interpolator in the display device, it is to be noted here that display devices or decoder devices that do not have such interpolators, can easily be upgraded. For other display devices/decoders, no additional effort is caused by such an additional interpolator, since these devices already include such a device operating under a predetermined non-adaptive or user-initiated setting.
Preferred embodiments of the present invention are subsequently explained with respect to the accompanying drawings, in which:
a a preferred embodiment of the apparatus for producing on the encoder-side;
b a preferred embodiment of the apparatus for processing on the decoder-side;
The system in
Preferably, the resizing information and the encoded picture data are input into a data stream formatter 12. The device 12 forms a data stream and transmits the data stream via a transmission channel 13 to a data stream parser 14. The data stream parser 14 works as a receiver for receiving the data stream and the resizing information.
As outlined by a dashed line 15, the resizing information does not necessarily have to be transmitted together with the encoded picture data within a single data stream. Instead, the resulting information can also be transmitted via a separate transmission channel. Such an embodiment is preferable in cases, in which the transmission format of a data stream cannot be changed. Then, the resulting information can, for example, be transmitted via an additional data packet over the same transmission channel or via a completely separate transmission channel. In this case, the resulting information preferably includes an identification of the encoded picture data so that the decoder-side apparatus can allocate the correct resizing information to a certain data stream having encoded picture data.
The apparatus for processing the data stream on the decoder-side further includes a resizer 16 for resizing a picture in response to resizing information supplied by the receiver 14. The resizer 16 outputs a resized picture based on the picture having the picture resolution. In contrast to the picture having the picture resolution included in the data stream, the resized picture has a higher number of pixels than the number of pixels of the corresponding picture in the data stream transmitted via channel 13. The picture data generated by the resizer can then be input into a display device 17 to give a user of the display maximum subjective quality video. It is clear that the apparatus for processing including element 14 and 16 does not necessarily have to be included within the display 17 such as a monitor or a television set. Instead, the resizer can be a software-plug-in for a general purpose processor or an application program for application-specific processors which are, for example, located on a graphics board of a display drive device.
a shows a preferred embodiment of the apparatus for producing on the encoder-side of
Alternatively or additionally, the input device 11 also includes an auxiliary input. This input is operative to receive estimated resizing information adaptively derived by certain determination rules from the encoded picture data in the second resolution or the first resolution. As outlined above, there is a quite complicated interdependence between several factors to arrive at an optimum resolution. Nevertheless, it has been found out that the main factors are a target resolution or, preferably, target resolution (second resolution), source resolution (first resolution) and pixel accuracy. Thus, a quite good estimation value for the resizing operation can be automatically generated and input via the auxiliary input to the resizing information supplier 11. The resizing information is such that it results, when controlling the resizer 16 (
The data stream formatter 12 is implemented as a transmitter when a streaming mode is set. Preferably, the resizing information is transmitted ahead of the encoded picture data. In case of a file storing mode, the data stream formatter 12 is operative to write a video file into a storage device. In this case, it is preferred to write the resizing information together with the encoded video data into a single file.
As outlined above,
b illustrates a preferred embodiment of an inventive apparatus for processing a data stream having encoded data and resizing information. The encoded data are input into a decoder 19. The decoded picture data, which are in the picture resolution are input into the resizer 16, which preferably includes an interpolator. The interpolator receives, as a further input, the resizing information for performing an interpolator control. In this embodiment, the resizing information define the resolution at the output of the interpolator 16 in absolute terms or in relative terms with respect to the picture resolution of a picture input into the interpolator. The interpolator output can then be displayed on a display device 18.
Although
Subsequently, reference is made to
Naturally, the interpolation distortion included in a picture increases more and more, when the interpolation degree increases. This increase of interpolation distortion results in a degradation of subjective quality.
On the other hand, the interpolation results in a larger size of the picture, which leads to an increase of subjective quality. When both contradicting influences are added together, one will obtain a curve such as the
When the resizing degree is increased over the resizing value, then the optimum quality decreases and reaches the original subjective quality at a certain value corresponding to a resizing value, which should be the maximum resizing value. When the resizing degree is increased over the maximum resizing value, the whole interpolation was not worth doing, since the result is worse than the initial (small) picture without any additional processing. In view of this, it is preferred to select the resizing value as a value smaller than the maximum resizing value.
Preferably, the preferred resizing value resulting in the maximum of the subjective quality is determined and entered, as an additional information, into the data stream having the encoded data. Preferably, when the resizing information is not signalled separately from the encoded picture data, reserved fields provided in each standardized transmission format, which are reserved for “future application” can be used for entering the resizing information indicating the degree of resizing intended by the user.
Inventively, the resizing information or “virtual size” information is added to the video information so that a player uses such virtual size as the preferred playback resolution size for the file or stream coming from a streaming server. So it is the choice of the creator of the stream/file to define the best virtual size, for example based on the
The present invention is especially suitable for not doing playback in full screen. Then, the inventive device allows to scale the displayed picture to a pre-defined higher resolution instead of an original small or “stamp-sized” image. A predefined virtual size gives an impression of a better quality video. The inventive transmission of the virtual size, therefore, also obviates the need to adjust a replay size from the player itself, since the player is automatically adjusted by the transmitted resizing information.
A preferred application of the invention is in a browser plug-in like http://trailers.divx.com/plugin/index.html. For streaming purposes etc., the author can encode the video with a desired virtual size, and the player plug-in would automatically scale it to the resolution the author of the stream intended.
Depending on certain implementation requirements of the inventive methods, the inventive methods can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, in particular a disk, DVD or a CD having electronically readable control signals stored thereon, which cooperate with a programmable computer system such that the inventive methods are performed. Generally, the present invention is, therefore, a computer program product with a program code stored on a machine readable carrier, the program code being operative for performing the inventive methods when the computer program product runs on a computer. In other words, the inventive methods are, therefore, a computer program having a program code for performing at least one of the inventive methods when the computer program runs on a computer.
While the foregoing has been particularly shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various other changes in the form and details may be made without departing from the spirit and scope thereof. It is to be understood that various changes may be made in adapting to different embodiments without departing from the broader concepts disclosed herein and comprehended by the claims that follow.
