The present invention generally relates to the field of changing aspect ratios in relation to coded data images and more particularly to a method, an image processing device, an image display device and a computer program product for changing the size of presentation of an image data stream provided in an image data format.
In the field of presentation of visual information like for instance presentation of televised image data, it is customary to broadcast data to be displayed in a certain format. Here a certain aspect ratio of 4:3, i.e. the relationship between the width and height of the displayed image, is more or less standardized.
However nowadays there are often television screens having different aspect ratios of for instance 16:9, which provides a wider picture. There is even wider aspect movie material, and with future televisions e.g. employing front projectors even this material can become displayable.
In order to be able to show information on the whole of the screen when image data broadcast according to one aspect ratio is presented on a display having another aspect ratio, it is common to provide black bars, cut out information or perform other ways of tampering of the data in order to obtain the new aspect ratio. Such change can be performed through stretching the image horizontally and vertically that leads to loss of information, stretching the image only horizontally that leads to a distortion of the image and so called panoramic stretch, i.e. stretching horizontally with non-uniform zoom factor, which leads to objects placed at the side of an image being more distorted than objects in the center of the image.
One such way of tampering with the data in order to enable the conversion of one aspect ratio to another is described in WO 03/017649. This documents describes the insertion of image information from a previous picture into a present picture. In this way a new aspect ratio is provided without having to stretch the image.
There has furthermore been a movement towards providing different types of digital coding of video data, which format can be used also in transmitting video data. Such formats are among other MPEG2 and MPEG4.
It would in this case be beneficial to use background data that is present in such coding, if a stream including such data is present, to also allow a variation of the aspect ratio, such that an aspect ratio conversion can take place without having to distort the image shown or leave out important information, while at the same time using the whole of the display.
It is therefore an object of the present invention to allow a variation of the field of view for a coded image stream for a display, such that a field of view conversion can take place without having to distort the image shown or leave out important information, while at the same time using the whole of the display.
According to a first aspect of the present invention, this object is achieved by a method of changing the size of presentation of an image data stream provided in an image data format comprising the steps of:
a) obtaining an image data stream coded in a format and having a first original field of view to be presented in,
b) selecting at least parts of the image data stream,
c) obtaining, from selected image data, values of pixel regions from an area larger than the original field of view, and
d) changing the field of view by calculating an image to be displayed conforming to a second field of view based on the obtained data and values, such that image data that is intended to be presented in the first field of view can be displayed in the second field of view.
According to a second aspect of the invention, this object is also achieved by an image processing device for changing the size of presentation of an image data stream provided in an image data format and comprising:
at least one image decoding unit arranged to:
According to a third aspect of the present invention, this object is also achieved by an image display device for changing the size of presentation of an image data stream provided in an image data format and comprising:
a display unit, and
an image processing device comprising:
According to a fourth aspect of the present invention, this object is also achieved by a computer program product to be used on a computer for changing the size of presentation of an image data stream provided in an image data format and comprising computer program code for making the computer execute, when said code is loaded into the computer:
obtain an image data stream having a first original field of view to be presented in, select at least parts of the image data stream,
obtain, from selected image data, values of pixel regions from an area larger than the original field of view, and
change the field of view by calculating an image to be displayed conforming to a second field of view based on the obtained data and values, such that image data that is intended to be presented in the first field of view can be displayed in the second field of view.
Claims 2 and 15 are directed towards providing objects of the image data stream as pixels in different layers, where pixel regions outside the first field of view are provided in at least one layer and combining objects of at least some of the layers of the decoded image data stream including said one layer for providing an output data stream allowing presentation of image data. In this way it is possible to limit the manipulation of the image to be displayed only to those layers, which are affected by the change of field of view.
Claim 3 is directed towards displaying at least some of the image data in the stream on a display with the second field of view.
According to claims 4 and 16 the coded stream is an MPEG-4 image data stream and at least some of the pixel regions that are at least partially outside the original field of view are coded as a sprite. This allows the changing of field of view in a simple manner, since a sprite contains a large amount of information that can be used for changing the field of view.
According to claims 5 and 17 the selected image data information is processed regarding mapping of less satisfactory positions of pixels in the second field of view. This measure allows the change of field of view to be made more efficient if there are not enough pixels in the pixel regions outside the first field of view to be used for obtaining the second field of view or if the relationship between objects in different layers need to be considered in the change of the field of view, like if the geometrical relationship needs to be adjusted, for example if an object would be moving in and out of consecutive images. This can be annoying and distracting the viewer's attention if e.g. part of a person is periodically entering the image from outside, in which case it may be better to always position this person object inside our outside the field of view. Another example is where a person positioned on the edge of a 4:3 image is repositioned on the edge of the enlarged, e.g. 16:9 image.
Claims 6, 7, 8, 9, 10, 18, 19, 20, 21 and 22 are directed towards different ways of processing the selected image data provided outside the first field of view if there are not enough pixels in the pixel regions outside the first field of view to be used for obtaining the second field of view.
Claims 11 and 23 are directed towards processing the selected image data when the relationship between objects in different layers need to be considered in the change of the field of view.
According to claims 12 and 24 the first field of view corresponds to an aspect ratio of 4:3 and the second field of view corresponds to an aspect ratio of 16:9.
According to claims 13 and 25 the values of pixel regions outside the first field of view are provided in at least one different output data stream than the stream including the combined objects. In this way several streams can be combined for enlarging video shown on a display. This also enables the provision of video data to be shown that can be used for different types of screens having different types of aspect ratios.
The present invention has the advantage of allowing a variation of the field of view, for instance the aspect ratio, for a coded image stream for a display, such that a field of view conversion can take place without having to distort the image shown or leave out important information, while at the same time using the whole of the display. This change of field of view is furthermore possible using information already available in the image data stream. This change can therefore be performed without additional complex image processing. The invention is furthermore simple to implement with only slight variations of the decoder associated with the standardized coding format.
The general idea behind the invention is thus to obtain, from a selected image data stream, values of pixel regions from an area larger than the original field of view of the image data stream and change the field of view by calculating an image to be displayed conforming to a second field of view based on the obtained data and values. In this way image data that is intended to be presented with a first aspect ratio can be displayed with a second aspect ratio without distorting the content.
These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.
The present invention will now be explained in more detail in relation to the enclosed drawings, where
The present invention generally relates to the field of displaying information on a screen such as on a television screen. Reference will now be made to
The stream is divided according to frames, where a frame defines the format the stream is to be displayed in at a certain instant in time. The frame format or first original field of view is here normally provided corresponding to an aspect ratio of 4:3. The display 16 does however have another aspect ratio of 16:9, which means that the images to be displayed have to be manipulated in order to be shown on the display 16 and when the whole of the display is to be used. There are a number of methods that exist for manipulating images having a certain aspect ratio for provision to a screen having another aspect ratio, like for instance stretching, providing black bars on the sides of the image etc. With these known schemes it is however hard to use the whole of the screen without influencing the displayed content of the stream in a negative way. A stretching of the image in two directions leads to a loss of information, stretching in one direction, like for instance horizontally, leads to distortion of the objects in the image and a panoramic stretch leads to uneven distortion of the objects. All these effects can be found to be disturbing or annoying for a viewer of the stream.
The present invention is directed towards resolving this.
Before going into more details of the invention a short summary of some of the aspects of MPEG-4 will now be given. Objects to be displayed in a stream are coded into different layers. The coding of objects in a layer here follow close to the coding used in MPEG-2. Thus objects can be coded using motion vectors based on earlier or afterwards sent objects. Normally a layer is provided for each object and one layer is provided for background. The stream is divided according to frames, where each coded layer has some information for an object related to a specific frame. Frames determine what is to be displayed from the different layers at different points in time and for the frames the aspect ratio is also set, which can be seen as a field of view. All frames should then have the same aspect ratio. One of the layers has information that extends far outside the frame and that is a sprite layer. A sprite can then be static information that remains unchanged over a number of frames or dynamic that for instance takes account of motion of the sprite from frame to frame. The sprite includes information about background elements of a scene displayed in a number of frames. The standard allows a great freedom in that layers can be selected for display almost at will. The purpose of the sprite is to increase efficiency of coding of the video stream. The general idea of the invention is to take advantage of the information in the sprite in order to enable a change of aspect ratio for displays such that the entire display area can be used for showing all the information of the video stream without having to distort its content. More information about MPEG-4 can be found in the ISO/IEC standard 14496-2:2001, which is herein incorporated by reference.
The invention will now be described in more detail with reference being made to
As mentioned earlier the image processing device 12 receives a coded image data stream X having a format according to the MPEG-4 standard, step 36, which is provided to a demultiplexing unit 18. The demultiplexing unit 18 demultiplexes the stream according to the different object layers and frames and sends the separated coded layer information XO and XS to corresponding layer decoders 20, 22, step 38. In
If the image to be displayed has less satisfactory positions of the pixels, then the decoded image data is processed regarding mapping of these positions. If for instance the sprite does not have enough information in the cut-out areas, perhaps because some-pixels are missing, the sprite layer is processed by an image extending unit 27 (e.g. an ASIC or software running on a dedicated or general purpose processor) connected to the combining unit 26. This processing can then comprise the standard procedures described earlier, like stretching, estimation of object movement etc. One such procedure, which can be used, is the cutting out areas of previously shown images and using these in later images as described in WO 03/017649, which is herein incorporated by reference. In case there are only relatively small areas missing, this processing could also comprise more advanced texture gap filling methods like a geometrical image transformation of the sprite or the region of the sprite lacking pixel information. The image transformation can then include extrapolation of neighboring pixels, copying of border pixels, etc.
It is furthermore possible to shift some pixels of for instance the object layer in relation to the sprite layer in order to for instance provide an object, which is positioned at an edge of the frame in the first field of view also at the edge of the new field of view. In this way the image can be made to optimally correspond to a particular screen field of view. This is helpful if the object keeps moving in and out of consecutive images, which disturbs the image when displayed with the second aspect ratio. This thus allows the object layer in question to be reproduced in a geometrical position that is a mathematical function of the original geometrical position. This shifting is furthermore not limited to objects provided at an edge of a field of view, but can be applied for any object or any layer.
It should be realized that the control unit 24 can also control, in dependence of selections made for instance by a user, which objects are to be combined and thus also which decoders are to send objects to the combining unit 26. It should also be realized that it might be possible to select between different aspect ratios in the control unit and cut the sprite according to each desired aspect ratio. The control unit might have pre-knowledge of the aspect ratio of the display and control the cutting of the extra area automatically based on this. It is also possible to take account of objects moving out of the frame when performing the selection and cutting of extra areas, in which case the control unit has to have information also from the first decoder 20 in addition to the sprite decoder 22.
In order to further describe the present invention, reference will now be made to
If the scene that is to be shown next is shifted then the above-described method is continued with another sprite relevant to this scene, if one is at hand, to enlarge the area displayed. If no such sprite is at hand the image extending unit is arranged to use the above-mentioned processing to provide the remaining increase in field of view.
A second embodiment of the image processing device according to the present invention is shown in a block schematic in
The image processing device according to the present invention can be provided in the form of a one or more processors with corresponding program memory containing program code for performing the method according to the present invention. It is also possible to provide this functionality as a hardware unit, for instance as a suitably programmed ASIC circuit. It is furthermore possible to provide some units as processors accessing software code and others as hardware units
The invention can thus also be implemented as a computer program product. A computer program product should be understood to be any physical realization of a collection of commands enabling a processor—generic or special purpose—, after a series of loading steps to get the commands into the processor, to execute any of the characteristic functions of an invention. In particular the computer program product may be realized as data on a carrier such as e.g. a disk or tape, data present in a memory, data traveling over a network connection—wired or wireless—, or program code on paper. Apart from program code, characteristic data required for the program may also be embodied as a computer program product. One example of the computer program product according to the present invention is shown in
With the present invention there are several advantages obtained. The whole of the display is used without distorting any of the objects displayed. The change of aspect ratio is furthermore possible to obtain using information already available in the coded image data stream. This change can therefore be performed without additional complex estimation of scene changes in the image processing device. The invention is furthermore simple to implement with only slight variations of the decoder associated with the standardized coding format MPEG4.
There are several variations that can be made to the present invention. The invention is not limited to the sprite layer but can be applied for any layer having pixel regions extending beyond the field of view. Aspect ratios other than the ones described can for instance also be used. The opposite aspect ratio change is for instance also possible, i.e. from 16:9 to 4:3. The invention is furthermore not limited to a change of aspect ratio, but can be applied on any change of field of view. It is furthermore possible to have more objects and corresponding layers. It is furthermore not necessary to have separate decoders, but the decoding can in many instances be performed in one and the same decoder. The invention is furthermore not limited to MPEG4, but can be applied in other object-based compression applications as long as they have pixel regions that stretch outside the frame. The image extending unit need not be connected to the combiner but can just as well be connected to the relevant layer decoders. It might in fact not be needed at all if the sprite contains enough information that can be used for changing the field of view. The invention is furthermore not limited to television sets, but can be implemented in a video, DVD or any other type of image handling device. It can also be provided for connection to one or more screens, perhaps in the environment of a home network.
Number | Date | Country | Kind |
---|---|---|---|
04100376.5 | Feb 2004 | EP | regional |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/IB05/50350 | 1/27/2005 | WO | 00 | 7/31/2006 |