This invention relates to processing of a media signal in a media system.
The present invention also relates to a computer system for performing the method.
The present invention further relates to a computer program product for performing the method.
This invention further relates to a media system for processing a media signal.
EP 0,957,177 discloses a coder producing a signal. The coder comprises a quantizer for quantizing an input signal, an encoder for encoding the output of the quantizer and outputting coded data. It uses a scene change detector for detection of a scene change, and if the scene change is detected, a quantizing parameter adapted to the scene is supplied to the quantizer to a scene newly appearing after said scene change. The coder is used to produce a signal of quality at a low coding rate.
It is known that a system for video signal processing can be in an unpredictable state when a data dependent load of a software media processing algorithm becomes higher than its assigned or available resources after a scene change. In the worst case scenario the system crashes. During the system crash, and before the system recovers from the crash, there will a period of unreliable system behavior or at least a period of system behavior that unfortunately depends on and is affected by non-optimal data load(s). As a consequence, during said period, the non-optimal system behavior of the video signal processing system will degrade the visual output quality, which is annoying to an end-user.
It is therefore an object of the present invention to minimize the time of non-optimal data dependent system behavior, and its effects on the visual output quality.
The object is achieved by a method of the type mentioned in the opening paragraph, the method comprising the steps of:
In the first step, generally, the system control unit may have to monitor the resource usage and the progress to be able to recognize the current status of the media system media signal processing.
This is, in general, the reason, why the scene change detector or the structural load detector may help to protect the media system in that it has information about the progress and the resource usage in order to avoid or to prevent an eventually, subsequent overload or system crash.
In the second step, a first point in time is determined for a subsequent shot or scene change before one of these actually occurs; in the third step, a second point in time closely related to the first point in time is then determined; and in the fourth step—at the second point in time—an assigned quality level is decreased in order to actively release resources to avoid or prevent an overload of resources, i.e. to prevent that said data load of a software media processing algorithm becomes higher than assigned or available resources thus avoiding previously mentioned system crash. Finally, in the fifth step,—at or till said third point in time—the assigned quality level is adapted again in order to make the realized quality level to become stable.
The object of the invention—i.e. to minimize the time of non-optimal data dependent system behavior, and its effects on the visual output quality—is achieved since—in the time from said second point in time (from the fourth step) and until the realized quality level is once again stable (in the fifth step), the processing of the media signal in the media system is controlled by said decrease and subsequent adapting of the assigned quality, thus preventing unreliable non-optimal data load(s).
It is therefore an advantage of the invention to prevent overload by means of the above-mentioned steps.
The term of “scene change” known from the prior art may be expanded to also cover situation of a shot change as well. The shot change—as compared to the scene change—concerns change(s) within the scene, e.g. different views of people and/or different views in a room.
One of these changes was determined in the first step, i.e. “substantial visual change of a content”.
In a preferred embodiment of the invention, said step of determining a first point in time comprises the steps of:
In these two steps, the structural load control or indicator unit may derive information about at least one future visual change of the media signal, i.e. information about a scene change and/or a shot change, and then transfer it to the system control unit, where the point in time for said scene change or shot change is subsequently determined.
In a preferred embodiment of the invention said scalable algorithm is without error propagation.
In another preferred embodiment of the invention, the second point in time comes before the first point in time; alternatively, the second point in time coincides with the first point in time; or alternatively, the second point in time comes after the first point.
In a preferred embodiment of the invention said period for the adaptation time is in a time scale of fractions of a second, prefereably less than one second.
In a preferred embodiment of the invention media system is a PC, a Digital TV, a Settop-Box or a Display.
The object of the invention is further solved by a media system for processing a media signal comprising:
In a preferred embodiment of the invention, the media system's means for determining a first point in time comprises:
The media system provides the same advantages for the same reasons as described previously in relation to the method.
Hereby the present invention utilizes the capabilities of scalable video algorithms and exploits the fact that the human visual system (HVS) needs a short time to adapt to shot or scene changes. Generally, the scalable video algorithm(s) may reduce their quality level upon or before notification of the scene or load changes, freeing in this way resources, speeding up processing of the media signal and avoids previously mentioned overload or crash. The reduction (i.e. said decrease) in quality may be recovered within said time equivalent to the adaptation time of the HVS, and is therefore not annoying to a user of the media system.
For a background information, prior to a detailed discussion of the invention, the following paragraph should be considered as an introduction to various terms, which will be used in the following:
The video algorithms that are used currently in consumer applications are designed to have a high quality. Highquality is particularly eminent in case of single-function hardware components. Software video algorithms can be designed in several processing configurations to allow for different output quality levels in exchange of the required resources. Such software video algorithms can be named scalable video algorithms (SVAs). The scalability of the video algorithms allows a larger number of applications to run concurrently in a programmable component thereby improving the cost-effectiveness of the system. SVAs are controlled at run-time in their resource and quality behavior. The response to the run-time control may be realized by a control part of the SVA(s) that translates the assigned quality level to the SVA by the overall system control unit, to the appropriate processing configuration (realized quality level).
Instead of assigning quality levels it is also possible to assign budgets for resource usage, which anticipates a quality level. The really consumed budget may differ especially when the resource usage is data dependent.
The overall system control unit may allocate resources to processing algorithms (SVAs) based on average resource needs, in this way allowing more applications to run concurrently, and thus improving the cost-effectiveness of such as system. If the load of an SVA is higher than initially claimed, then the overall system control unit may react by reducing the quality level of this or some other less important algorithm, i.e. another SVA.
The invention along with a brief starting introduction to the prior art will be explained more fully below in connection with preferred embodiments and with reference to the drawings, in which:
Throughout the drawings, the same reference numerals indicate similar or corresponding features, functions, etc.
The term of scene change may be further expanded to cover also situations of a shot change. The shot change—as compared to the scene change—is about change(s) within the scene, e.g. different views of people and /or different views in a room.
Generally, there are - as discussed - three outcomes for the resource usage or load of an algorithm during the scene or shot change: an increase (a), no change (b) or a decrease (c). This is known from the prior art; in this document, the scenario that the load of an SVA increases after a scene or shot change is primarily considered, since this is the most critical case. The implication of no or decreased loads after scene or shot changes will be explained at the end of this document. If the load increases and the available resources are insufficient then the algorithms are riming behind, leading to a non-optimal system behavior, it may even cause time critical deadline(s) to be missed in the processing of the algorithm(s).
There may be algorithms whose load is dependent on statistical variations of certain parameters (e.g. motion and details). Statistical variations of the above-mentioned types of image contents are exhibited particularly after scene or shot changes.
The basic idea of the present invention is related to the observation that upon a scene or shot change, it takes a while before the human visual system (HVS) adapts to the new scene or shot. As a consequence, especially with data dependent video algorithms, it is possible to temporarily reduce the quality level of the output during a scene or shot change, exploiting this temporal lack of sensitivity of human perception. The goal is to stay within the assigned budget after sudden changes in resource needs with an implementation of such a system.
So far, scene or shot changes have only been discussed. According to the invention, the term of scene or shot change may be further expanded to cover the more general case of a structural load change. The structural load change—as compared to the scene change—is about change(sof the average load, which last for a longer time period, e.g. more than a few images in case of video processing.
The applied time slots before a shot or scene change and after a shot or scene change may be different. It is convenient to have a shorter time slot before the shot or scene change compared to the time slot applied after the shot or scene change. After the shot or scene change, an exponential increase of the quality, typical for recursive filters, may be appropriate. The time slot for unproblematic quality degradation is expected to be about 50- 100 ms before shot or scene changes, and 100-200 ms after shot or scene changes. With respect to the timing otherwise applied in signal processing of video signals, said time slots are quite long.
As was previously mentioned, after a shot or scene change the load of an SVA may increase, stay the same or even decrease. It may be appropriate to reduce the quality of a data dependent SVA upon notification of a shot or scene change without first checking whether the load increases; with the shown system the load of a data dependent SVA will always decrease temporally (compared to the non-adaptive behavior), and then increase to its assigned initial or new quality level. However, such an action has much less impact on the system's performance than if no action was taken.
For quality degradation during shot or scene changes, the SVAs may be divided into at least two categories:
In the first category, algorithms, which influence the output quality over time. If quality reduction occurs to these SVAS, then the quality increase over time may be affected due to error propagation.
In the second category algorithms that do not influence the output quality over time. With these, quality degradation may be non-critical during scene shot or channel changes, since quality degradation does not propagate over time.
In a preferred embodiment of the invention, it is preferred to perform quality degradation on SVAs (or functions of an SVA) of the above-mentioned second type.
Generally, the overall system control unit controls by means of the following steps:
Firstly, from an input stream, information about structural load changes from said input may be derived as early as possible within a video processing chain;
Secondly, said information, generally, may then be trnnsferred to the overall system control unit; and
Thirdly, the overall system control unn, on basis of said information, may currently optimize the overall output quality and the resource usage.
Similarily, the same steps may generally be applied when the point(s) in time for a structural load change is / are to be determined prior to the actual structural load change.
For cost-efficiency on a programmable platform, the overall system control unit may allocate budgets for algorithms to process data based on average or above average resource usage. The load of some algorithms may be data-dependent The load of the algorithms may increase after shot or scene changes as shown in figure Ia.
If the available resources are not sufficient, then the algorithms are runnng behind leading to non-optimal, undesired system behavior as shown in
The load of an algorithm may increase after a structural load change, e.g. shot or scene change. The notification about a structural load change may be performed by the structural load change detector or indicator that may pass this information to the following algorithms, before even the shot or scene change occurs to them due to a path latency as shown in
Upon shot or scene change notification, the shot or scene change detector, the overall system control unit, or the SVAs control parts may reduce their realized quality level by selecting the appropriate processing configuration as shown in
The decrease and increase in quality level of an SVA may be performed within a short time, taking the HVS adaptation time into account as shown in
The main advantage of the invention is that a short decrease and increase in the realized output quality may have no or little negative effects due the adaptation time of the human visual system, while freeing system resources to provide robustness and stability; and since the HVS immediately concentrates on the new image content, degradation of image quality may also be possible even shortly before shot or scene changes as shown in
It may be preferred to perform quality reduction to SVAs or functions of SVAs which have no influence on temporal quality degradation.
In step 90, the method is started. Variables, flags, buffers, etc., keeping track of points in time, the various stages of the media signal, assigned quality level(s), realized quality level(s), etc, on said media system are set to default values. When the method is started a second time, only corrupted variables, flags, buffers, types etc, are reset to default values.
In step 100, a system control unit may monitor a progress and a resource usage of the processing of the media signal. The progress - as previously discussed - may be reference numeral 21 A or 22A from
In step 200, a structural load control unit may determine a first point in time for a substantial load change of content Step 200 constitutes a generalisation of steps 201 and 202. Said substantial load change (of content) may be understood as consequence of substantial different load due to a scene change or a shot change. The scene change may be understood as the change from one scene (e.g. in a movie or film) to another scene, whereas the shot change may be a change within a scene, a change from the view of one person to a view, i.e. another shot, of another person. Said first point in time was indicated in
In step 201, information about at least one future-visual change of the media signal may be derived by the structural load change detector or indicator in the media system.
Of course, several visual changes of the media signal may be possible to determine, especially due to the nature of the media signal, when this is a live motion movie with scene changes and / or with different shots of various people. At least one of said changes may then be derived by said detector or indicator to an information, which is prepared to be transferred to the system control unit Corresponding to many visual changes of the media signal, correspondingly much information about point in times may be derived. It may further be the case that said visual change (with corresponding derivable point in time) is determined by means of various statistical computations of the media signal, e.g. wherein mean value, spread, spreading, variation, etc., may be considered.
In step 202 - as discussed in the foregoing step - said derived information may then be transferred to the system control unit.
In step 300, a second point in time based on the first point in time may be determined by the system control unit. As was discussed in
In step 400, the system control unit may decrease an assigned quality level of at least one scalable algorithm at the second point in time. As was previously discussed in
In step 500, the system control unit may adapt the assigned quality level of at least one scalable algorithm at or till a third point in time. As a consequence, a realized quality level will subsequently become stable shortly thereafter, wherein a realized quality level (17) is expected to become stable within a period of adaptation time. It may become stable preferably before the adaptation time of the human visual system has elapsed since said second point in time. In other words, as opposed to the foregoing, in this step said assigned quality level is adapted again at or till the third point in time, but the adapting has to take place at such point in time that said realized quality level is duly stable around the period for the HVS has elapsed since said second point in time. As an example, when time recursive filters are processed in one of the scalable algorithms in the media system, the realized quality level may increase exponentially after said adapting in the assigned quality level.
The quality level after said adapting of the assigned quality level, i.e. the realized quality level, when stable, may be higher or lower than the quality level before the decrease of the assigned quality level since the system control unit continuously performs scalable video processing in order to balance resources with quality. After a scene or shot change has been detected, the quality level of an SVA may drop immediately, but the amount of resource reduction may not be fixed or known. Therefore, there may be an aim for resource usage below the assigned budget Usually, the later assigned quality level - due to said adapt - may be higher and as a consequence the quality level for the SVA again may have to be increased again. However, also the opposite is possible, i.e. that still the resources used are above the budget. In the latter case, the assigned quality level may frther have to be decreased Usually, the method will start all over again as long as the media system are powered and still have a media signal to process. Otherwise, the method may terminate in step 600; however, when the media system is powered again, etc, the method may proceed from step 100.
Said media system may be a multimedia consumer terminal such as a PC, a Digital TV set, a Settop-Box or a Display capable of performing scalable video processing with video algorithms designed to balance resources to quality and further to take the mentioned structural load changes, e.g. scene or shot changes, into account when resources are allocated to obtain a corresponding quality level without overload or crash.
A computer readable medium may be magnetic tape, optical disc, digital video disk (DVD), compact disc (CD record-able or CD write-able), mini-disc, hard disk, floppy disk, smart card, PCMCIA card, etc.
Number | Date | Country | Kind |
---|---|---|---|
02077135.8 | May 2002 | EP | regional |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/IB03/02054 | 5/16/2003 | WO | 11/24/2004 |