Patent Application
20060055826
The present invention relates to a video signal processing system, a video signal transmitter and a video signal receiver.
Usually, a sophisticated digital picture processing system consists of several distinct and separated modules like e.g. frame rate conversion, interlaced to progressive conversion, noise reduction and picture enhancement. Each of this blocks usually needs some analysis information to achieve highest quality. These analysis blocks are mostly integrated into the respective picture processing components
Therefore, the video processing system is typically considered as a sequence or concatenation of equally important modules where the output of one module is input to the next module.
On the other hand, several approaches propose hardware implementations of a motion estimator utilised for upconversion and noise reduction processes inside a pure analog video receiver. The principal idea of these implementations is to derive and utilise one set of motion vectors for these two processing stages. The use cases utilising this approach are restricted to uncompressed video signals.
Further, research has been performed in the area of motion estimation for MPEG2 coding using true-motion vectors. These approaches however are typically not implemented in state-of-the art designs. Systems which make use of inter-frame video compression generally perform block matching between pairs of fields or frames within the group of pictures (GOP) or video object planes (VOB).
Therefore, state of the art video signal processing systems employ up to three different motion estimators in the cascade of processing stages, noise reduction, encoding, and format conversion. As set out above, these motion estimators work independently and do not share any resources or information with each other. Furthermore, the motion estimators are typically based on different motion estimation algorithms resulting in picture quality degradations introduced by the different motion vector information generated by the motion estimators.
This is especially true when it comes to video coding. Traditional inter-frame video encoders estimate motion within a group of pictures or video object planes. This method however does not estimate the “true” motion of objects between fields or frames.
Furthermore, a state of the art system consisting of a noise reducer, a video encoder, and a format converter estimate motion at least twice in two different modules, namely once in the transmitter and once in the receiver.
It is the object underlying the present invention to provide a video processing system, a video signal transmitter, and a video signal receiver which allow a more efficient use of system resources.
This object is solved by a video signal processing system according to the present invention as defined in claim 1, by a video signal transmitter according to the present invention as defined in claim 10, and by a video signal receiver according to the present invention as defined in claim 17. Preferred embodiments thereof are respectively defined in the respective dependent claims.
The video signal processing system according to the present invention comprises a video signal or picture analysis module (VSAM) which is adapted to realize—in particular all—video signal or picture analysis processes with respect to received video data in concentrated and/or centralized form. Further a video signal or picture processing module (VSPM) is provided which is adapted to realize—in particular all—video signal or picture processing processes with respect to received video data in concentrated and/or centralized form.
It is a basic idea of the present invention to concentrate and/or centralized the respective video signal or picture analysis capabilities within a single video signal or picture analysis module (VSAM) and to concentrate and/or centralized the respective video signal or picture processing capabilities within a single video signal or picture processing module (VSPM).
According to a preferred embodiment of the present invention, said video signal or picture analysis module (VSAM) comprises—in particular all—video signal or picture analysis components in concentrated and/or centralized form, in particular motion estimation and/or segmentation, Additionally or alternatively, said video signal or picture processing module (VSPM) comprises—in particular all—video signal or picture processing components in concentrated and/or centralized form, in particular motion estimation and/or segmentation components, in particular frame rate conversion and/or picture improvement components.
According to an advantageous embodiment a video meta data transmission channel is provided which is adapted to connect said video signal or picture analysis module (VSAM) and said video signal or picture processing module (VSPM) so as to enable an exchange of video meta data.
In other words, an embodiment of the video signal processing system according to the present invention comprises a video signal transmitter and a video signal receiver which are connected by a video signal transmission channel, according to the present invention additionally comprises a video information transmission channel between the video signal transmitter and the video signal receiver.
The video signal transmitter according to the present invention comprises a video signal or picture analysis module (VSAM) as a global analysis module for analysing an incoming video signal and providing video information corresponding thereto to at least one video pre-processing stage within a video signal transmitter or a video signal or picture analysis module (VSAM) and/or via a video information or video meta data channel to at least one video signal processing stage within a video signal receiver or a video signal picture processing module (VSPM).
The video signal receiver according to the present invention comprises at least one video signal or picture processing module (VSPM) as a processing module for processing a video signal received from a video signal transmitter or a video signal or picture analysis module (VSAM) based on video information received from said video signal transmitter or said video signal or picture processing module (VSPM).
Therefore, according to the present invention, it is possible that video information, i.e. results of analysing the video signal, e.g. for noise reduction, encoding and format conversion, is not only handled inside respective modules and crosses the module boundaries as well as a passing of video information from the video signal transmitter to the video signal receiver.
According to the present invention a structure is provided to replace the picture analysis processing steps which are distributed among the respective components in current state of the art systems by a centralised, i.e. global, analysis. Therefore, e.g. a spatial matching process in an interlaced to progressive conversion filter, which determines the orientation of an edge before the actual interpolation is done, produces results, which are then also available to e.g. an noise reduction component or a frame rate conversion component.
Therefore, according to the present invention, the analysis information needed by the several modules of a sophisticated digital picture processing system can be used also by other modules from the processing chain, i.e. no analysis processing steps are done twice in a similar way (e.g. motion estimation and motion detection). Therefore, an effective utilisation of resources is achieved.
E.g. the computational overhead of the motion estimation approach in state of the art systems is evident. This becomes even more evident when the relative complexity of a motion estimator is considered compared to the other video processing modules in the system. The motion estimator is typically the most complex module and is largely responsible for the overall picture quality.
Another example is the computational overhead of noise reduction in state-of the art systems which is proportional to the quality of the noise reduction modules employed by the system. According to the present invention a high quality noise reduction might be employed distributed in the system, while the picture analysis for this noise reduction is performed centralised.
In the video signal processing system according the present invention preferably said video information transmission channel comprises a forward channel to transmit video meta data from the video signal transmitter or said video signal or picture analysis module (VSAM) to the video signal receiver or said video signal or picture processing module (VSPM).
In the video signal processing system according the present invention further preferably said video meta data comprises picture analysis information to process a video signal on the video signal receiver side or the side of said video signal or picture processing module (VSPM).
In the video signal processing system according the present invention alternatively or additionally preferably said video information transmission channel comprises a backward channel to transmit video quality data from the video signal receiver or video signal or picture processing module (VSPM) to the video signal transmitter or video signal or picture analysis module (VSAM).
In the video signal processing system according the present invention further preferably said video quality data provides a respective feedback to video meta data received by the video signal receiver or picture processing module (VSPM) from the video signal transmitter or video signal or picture analysis module (VSAM).
In the video signal processing system according the present invention alternatively or additionally further preferably said quality data comprises picture quality information to pre-process a video signal on the video signal transmitter side or the side of said video signal or picture analysis module (VSAM).
In the video signal transmitter according to the present invention preferably said global analysis module comprises a motion estimator, and/or an edge detector, and/or a noise measurement unit, and/or a film mode detector, and/or a histogram calculation unit, and/or a block detector, and/or a segmentation unit.
In the video signal transmitter according to the present invention alternatively or additionally preferably said global analysis module comprises a video meta data encoder to encode said video information into video meta data.
In the video signal transmitter according to the present invention further alternatively or additionally preferably said global analysis module adapts its parameter settings according to received picture quality information to improve its analysis functionality.
The video signal transmitter according to the present invention preferably additionally comprises at least one pre-processing stage adapted to receive video analysis information and to pre-process an incoming video signal according to said received video information before said video signal is transmitted to a video signal receiver.
In this case, in the video signal transmitter according to the present invention preferably said least one pre-processing stage comprises a noise reducer, and/or a video encoder.
Further this case, in the video signal transmitter according to the present invention alternatively or additionally preferably said least one pre-processing stage comprises a video meta data decoder.
In the video signal receiver according to the present invention preferably said at least one processing module comprises a video meta data decoder to decode said video information from received video meta data.
In the video signal receiver according to the present invention alternatively or additionally preferably said at least one processing module provides picture quality information as feedback to said video information to said video signal transmitter.
In the video signal receiver according to the present invention further alternatively or additionally preferably said at least one processing module comprises a video format converter, and/or a frame rate converter, and/or an interlaced to progressive converter, and/or a noise reducer, and/or a picture improvement unit, and/or a de-blocking unit.
Further features and advantages of the video signal processing system, the video signal transmitter, and the video signal receiver according to the present invention will become more apparent from the following description of exemplary embodiments thereof taken in conjunction with the accompanying figures.
First of all, basic principles of the present invention are described taking reference to the block diagram of
According to the present invention a digital picture processing system is categorised into analysis parts, preferably one, and into picture processing parts and the respective ones are centralised.
Incoming video data is passed to a central picture analysis block 1 and—via an optional forward channel 3—to a picture-processing block 2. A communication between the central picture analysis block 1 and the picture-processing block 2 to transmit results of the picture analysis is done across a Video Meta Data Stream, which contains all available picture analysis information, organized by a Video Meta Data Protocol (VMDP). The Video Meta Data is synchronous to the video data and the Video Meta Data Stream also passes through the forward channel 3.
As further shown in
The basic idea of the invention is further elucidated in connection with FIGS. 5 to 7 by way of a second embodiment according to the present invention showing the harmonisation of temporal prediction by utilisation of a common motion estimator for different video processing stages in a video system.
A corresponding general video system to which the second embodiment according to the present invention might be applied is shown in
The global analysis module according to the present invention, in this case the motion estimator which is preferably generating true-motion vectors, passes its analysis results, i.e. video information, here the (true) motion vectors to the different processing stages in a video processing system. The processing stages may consist of the noise reducer 19, the video encoder 20, and the format converter 27, as they are shown in
Noise reducer 19: Reduces noise from an analog input signal by applying a temporal filter and an optional spatial filter.
Video encoder 20: Compresses the video signal in spatial and temporal direction. Format converter 27: Converts the format of the video signal to match the desired output format. This includes the output to a progressive display.
The embodiment of
A full-blown video processing system employs all of these processing stages simultaneously.
According to the present invention a common set of motion vectors is employed in various video processing stages. High-quality video noise reducers and video format converters typically employ motion compensation as a technique to trace moving objects in a video sequence. Video coding algorithms are typically based on motion-compensation, too, in case they utilise inter-frame prediction. The proposed system according to the present invention makes use of a common motion estimator and feeds the derived motion vectors to the various modules applying motion estimation as part of their processing routine.
The motion vectors may be refined by an optional motion vector refinement unit 29 in case they are fed to the video encoder 20 to improve the vector quality. This step includes the refinement of the motion vectors to produce suitable motion vectors for the encoding stage, i.e. the video encoder 20. The video encoder 20 hence utilises motion vectors between successive fields and does not rely on motion vectors between pairs of fields or frames within a GOP/VOP. The refinement of the motion vectors can either be integrated in the motion estimator or it can be added as a separate module, i.e. the motion vector refinement unit 29, as shown in
The motion-compensated noise reducer 19 filters the analog signal. The temporal noise filter 19b makes direct use of the motion vectors generated by the motion estimator 28. The noise reducer 19 may apply a spatial filter 19a, too. One advantage of the proposed system is the use of a single noise reducer 19 for the analog video signal.
Since the video format converter 27 is applied after the video decoder 25 when the video baseband signal has been restored, the motion vectors for the format converter are either stored locally or—in case of a transmission channel 23—transmitted to a receiver. It is desirable to minimise the data size of these motion vectors in storage or transmission applications. Hence, the motion vectors may be compressed by a motion vector encoder 30 utilising a loss-less data compression technique such as variable length encoding (VLE) possibly in combination with run-length coding (RLC). Loss less compression techniques can typically decrease the data size by a factor of around 6. One of the advantages of the proposed system is the fact that many video codecs (encoder/decoder systems) contain a coprocessor for variable length encoding/decoding (VLX). An efficient implementation will utilise this coprocessor for video coding and motion estimation vector coding purposes. The coprocessor can run in parallel to the processing on the main processor. The VLX computation therefore utilises an idling coprocessor and is not increasing the computational load on the main processor.
Once the motion vectors are compressed, they are multiplexed into the transmitted/stored bit stream as private stream data together with the video elementary stream and an optional audio elementary stream. Since the maximum number of motion vectors per video frame is known, an upper limit for the number of allocated bits per frame can be derived. A 16×16 block size applied in the motion estimation process for a PAL signal for example will result in an effective data rate of approximately 100 kbps. The upper limit may be used to derive a constant bit rate for this private stream. The bit rate of the multiplexed stream is slightly increased due to the additional private stream in the transport layer. The multiplexed stream then may either be stored locally or transmitted to a receiver. The proposed system may be extended to handle multiple video streams simultaneously. The system may either output multiple transport/program streams or combine all streams into a single multiplexed stream.
Of course, a combination with the first embodiment of the present invention is also possible, in which case a VMDP encoder would be used to encode the compressed or uncompressed motion vectors.
As shown in
The proposed system is highly suitable for a PV R-style application. The block diagram of such an application exactly corresponds to
The advantage of the solution according to the present invention is that no processing steps are done twice. This reduces the computational load of the system or the computational resources can be concentrated. In this way, the system resources are used more effectively, e.g. the picture quality can be improved by the utilisation of a common motion estimator generating true-motion vectors. Also system cost can be reduced due to skipping redundant processing steps and the complexity of the overall system is reduced, since the resources might be shared, i.e. less components are required, e.g. a single common motion estimator for the encoding and the format conversion parts are used.
Further, the complete analysis information can be made available to all picture-processing components which might also lead to an improvement of the overall picture quality, e.g. by the harmonisation of motion vector information utilised by the different video processing stages. Generally, the picture-processing components can exploit all information or only parts of it, depending of their nature. This can increase the achievable picture quality, because analysis information, which was not accessible before can be utilised now.
Moreover, the system is scalable, because it is possible the leave out some of the analysis components without changing the picture processing module. In this way, it is straightforward to design and realize processing chains for devices, where only the mid-range or low-end picture quality is expected. This can reduce the time-to-market.
Still further, computation load can be shifted from the client side to the server side, e.g. the motion estimation necessary for the noise reduction and the format conversion processes.
An example is that high end 100 Hz TVs require motion compensated frame rate conversion whereas Mid Range or low end TVs can work with motion adaptive or static up-conversion. If the (expensive) motion estimator is left out, the VMDP encoder might set the motion vector part of the protocol to zero vectors. The frame rate converter performs then a static up-conversion. In this case, the motion estimator, which can be a hardware add-on component to picture improvement system, which is implemented in software on a DSP, can be left out without changing the software implementation.
The range of applications of the present invention covers analog video processing systems, mixed (analog/digital) video processing systems, and digital video processing systems. Systems comprising local encoders as well as remote encoders (e.g. broadcasters) as well as systems comprising integrated or separated encoder/decoder (codec) systems are supported.
The invention describes a new system approach picture processing. In this system, all picture analysis components like motion estimation or segmentation are concentrated in a single picture analysis module. All picture-processing components like frame rate conversion or picture improvement are preferably also concentrated in a central picture-processing module. The analysis information is transferred with a Video Meta Data Stream. The centralisation saves computational power, removes redundant processing steps and makes analysis information available to all picture-processing components. The system is scalable because parts of the analysis/picture processing components can be left out/changed, without changing the respective other module. Quality can be increased/decreased in this way.
| Number | Date | Country | Kind |
|---|---|---|---|
| 03001860.0 | Jan 2003 | EP | regional |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/EP04/00750 | 1/28/2004 | WO | 7/28/2005 |
