The invention relates to a method of and device for generating an output signal from an input signal by applying a send effect processing to the input signal, wherein the input signal comprises a weighted sum of component signals, wherein dependencies between the weighted component signals are represented by parameters. The invention also relates to a binaural decoder for generating an improved binaural output signal, and a computer program product.
MPEG Surround is one of major advances in audio coding recently standardized by MPEG, see ISO/IEC 23003-1 MPEG Surround. MPEG Surround is a multi-channel audio coding tool that allows existing mono- and stereo-based coders to be extended to multi-channel. The MPEG Surround encoder typically creates a mono or stereo downmix from the multi-channel input signal, and derives spatial parameters from the multi-channel input signal. The downmix and spatial parameters are encoded in separate streams. However, the spatial parameters stream can be embedded in the downmix stream. The MPEG Surround decoder decodes the spatial parameters that are used to upmix the decoded downmix in order to obtain the multi-channel output signal. Since the spatial image of the multi-channel input signal is parameterized, MPEG Surround allows decoding the encoded stereo downmix onto other rendering devices, such as these comprising a reproduction on headphones. This particular mode of operation is referred to as the MPEG Surround binaural decoding process in which the spatial parameters are combined with the Head Related Transfer Function (HRTF) data (J. Breebaart, Analysis and Synthesis of Binaural Parameters for Efficient 3D Audio Rendering in MPEG Surround, ICME 07) to produce the so-called binaural output. In this mode a realistic surround experience can be provided using regular headphones.
Traditionally HRTF data is typically described as a set of pairs of impulse responses going from each speaker to both ears.
When the MPEG Surround binaural decoder is operated in a Low Power (LP) mode it can be implemented in mobile devices. In this mode in an offline process the raw HRTF data has been converted to a parametric domain allowing processing using low computational complexity. However, a disadvantage of the LP mode is that the parametric HRTF data represents typically only an anechoic portion of the raw HRTF data, i.e. it only covers a part of complete time domain responses which is primarily associated to directional cues. In practice, this means that the binaural decoder output signal will contain directional information, but will not sound very natural since there is hardly any externalization, which is primarily associated with the echoic part of the HRTF data. In order to compensate this lack of externalization, the MPEG Surround standard allows a use of a reverberation, as prescribed in ISO/IEC 23003-1 MPEG Surround Annex D. In such case, the MPEG Surround binaural decoder is extended with parallel reverberation. The input stereo downmix is fed to the reverberation process. The output of this process is directly added to the MPEG Surround binaural output. With such a parallel reverberation signal that is typically omni-directional, i.e. independent of direction, the echoic part is created and thus a more realistic surround experience is created.
However subjective tests with a reverberation, which is a type of a so-called send effect, added to the binaural output signal do not show satisfactory performance. One of the prominent artifacts in such binaural output is that when the original multi-channel encoder content is primarily present in the center channel, the binaural output signal sounds too reverberant.
A similar disadvantage holds for other send effects such as e.g. chorus, vocal doubler, fuzz, space expander, etc.
It is an object of the present invention to provide an improved method of generating an output signal from an input signal by applying a send effect processing to the input signal, which results in an improved output signal offering for some of the send effects an improved surround experience. The invention is defined by the independent claims. The dependent claims define advantageous embodiments.
This object is achieved according to the present invention in a method of generating the output signal as stated above and characterized in that the output signal is generated in dependence of the parameters to compensate for an unequal weighting of component signals comprised in the input signal.
The send effects are applied to the input signal as a whole and not to the individual component signals. Therefore, it is especially advantageous, to compensate for the unequal weighting of the component signals in the input signal while applying a send effect. Due to this compensation the strength of the send effect corresponding to the separate component signals is (nearly) proportional to the strength of each of the component signals, and thus resulting in more realistic surround experience. The invention is explained for a reverberation effect as an example of the send effect.
Reverberation is typically used to simulate acoustic reflections and can therefore be used in conjunction with (anechoic) HRTF data to place virtual sound sources out of the listener's head, i.e. in order to create a perception of a distance. The input signal is a downmix of component signals (e.g. the 6 channels of a multichannel representation) that are weighted before downmixing.
Typically, the component signals corresponding to surround channels comprised in a multichannel signal are attenuated before downmixing. When MPEG Surround encoding is used, the component signal corresponding to the center channel is effectively amplified in a stereo downmix (sqrt(0.5) per channel amounts to sqrt(2) when summing left and right downmix channel). This unequal weighting of the component signals comprised in the input signal results in the reverberation effect that is stronger for the component corresponding to the center channel and weaker for the components corresponding to the surround channels since a parallel reverberation employs the reverberation directly on the unequally weighted downmix. However, such unequal weighting does not match with the directional rendering of the 5.1 channels by using HRTF parameters, which (at least conceptually) map the restored component signals to the binaural signal. Therefore, when these signals, i.e. directional rendered signal based on restored component signals and the output signal obtained by applying reverberation to the input signal are mixed the externalization might not be natural in that the reverberation effect strength is dependent on the predominant direction of the original multichannel content. The adverse effect of the unequal weighting is reduced by modifying the generation of the output signal resulting from applying reverberation effect or any other send effect to the input signal such that it is adaptive to compensate the unequal weighting of component signals comprised in the input signal. This adaptation makes use of the parameters which comprise dependencies between the weighted component signals. The individually weighted components or combinations of the weighted components contributing to the input signal are not available anymore, as the component signals have been summed up (downmixed) after the weighting. However, the parameters allow for estimation of their contributions based on the dependencies between the weighted component signals represented by the parameters. There are various ways the adaptation of the generation of the output signal can be made, which are discussed in the following embodiments.
In an embodiment, the input signal is decomposed into a plurality of intermediate signals, wherein each of the intermediate signals is scaled with a respective gain to compensate for the unequal weighting of component signals comprised in the input signal. Generating intermediate signals (or at least using the intermediate signals conceptually) is beneficial when information from multiple component signals can be combined into the intermediate signals. For example left and right channel signals of the input signal both contain information from the center channel, when the MPEG Surround standard is used in a stereo compatible fashion. In such a case the intermediate signal corresponding to a center channel can be constructed using both left and right signals of the input signal. Furthermore, when the multichannel signal comprises five channel signals, i.e. the center channel signal, a left front channel signal, a left surround channel signal, a right front channel signal, and a right surround channel signal, the left front channel signal and the left surround channel signal can be combined in the intermediate signal, as well as the right front channel signal and the right surround channel signal can also be combined in the intermediate signal.
In a further embodiment, the respective gain corresponding to the respective intermediate signal is calculated as a weighted sum of predetermined further gains, wherein the predetermined further gains are derived from weights used to create the input signal, wherein the predetermined further gains are weighted with respective weights that are derived from relative contributions of the weighted component signals to the respective intermediate signal. One can approximate the component signals from the intermediate signal. MPEG Surround prescribes, for example, that OTT (one-to-two) processing block is used to create two signals from a single signal using the inter-channel intensity difference (IID) parameters, or TTT (two-to-three) processing block is used to create three signals from two signals, using channel prediction parameters and/or IID parameters. The gains can be applied on the signals created using the OTT and/or TTT processing blocks and the resulting signals can be downmixed again (a single channel is required for the send effect after all). However, the upmix step, i.e. creating multiple intermediate signals from the input signal, can be omitted because the energy distribution related to intermediate signals is known. Thus the current embodiment offers an efficient way to apply the gains to the intermediate signals, without actual restoring of the individual component signals contributing to these intermediate signals.
In a further embodiment, the relative contribution of the weighted component signals to the respective intermediate signal is derived from an intensity difference between the weighted component signals contributing to the intermediate signal, wherein the intensity difference is derived from the parameters. The energy distribution among the weighted component signals is comprised in the inter-channel intensity differences, which in turn are comprised in the parameters accompanying the input signal.
In a further embodiment, the input signal is scaled with a gain calculated as a weighted sum of further gains, wherein the further gains are derived from the parameters corresponding to the weighted component signals, wherein the further gains are weighted with weights that are derived from relative contributions of the weighted component signals or combinations of the weighted component signals to the input signal. This offers an efficient way to apply a gain to the input signal, without the actual need for restoring of the weighted component signals or combinations of the weighted component signals. For the mono input signal this means that a single gain is applied to the input signal. For the stereo input signal this means that two individual gains are applied, each for one of the two channels comprised in the input signal.
In a further embodiment, the relative contribution of the weighted component signals or the combinations of the weighted component signals are derived from intensity differences between weighted component signals contributing to the input signal, wherein the intensity differences are derived from the parameters. Conceptually, as in one of the previous embodiments, one can restore the weighted component signals from the input signal using e.g. several OTT processing blocks cascaded and in parallel. The OTT processing blocks are energy preserving, thus the energy distribution of the weighted component signals in the input signal is calculated based on the intensity differences comprised in the parameters. This distribution is relative to the energy of the input signal, thus an OTT processing block distributes the energy of its input signal over two output channels. Applying gains to the individual component signals can therefore be effectuated by applying a single gain to the input signal.
In a further embodiment, generating the output signal comprises adapting send effect processing applied to the input signal, based on the parameters. One could adjust the effect itself to compensate the weighing of the components but this is often a suboptimal solution in terms of efficiency.
In a further embodiment, generating the output signal comprises adapting the output signal itself, wherein the output signal is scaled with a gain that is adjusted in dependence of parameters. When adapting the output signal of send effect processing that is effected by e.g. a large time interval of the input signal (as it is often the case for reverberation filters), the parameters corresponding to certain time intervals may be mixed in a signal dependent manner due to the temporal smearing. In such a case it is advantageous to adapt the gain over time in dependence of the parameters, as well as the effect and signal properties.
In a further embodiment, the input signal and the parameters are the downmix signal and the spatial parameters, respectively, in accordance with the MPEG Surround standard. For MPEG Surround, the component signals are formed by the channels of a multichannel source (e.g. 5.1 audio from a DVD, multichannel recording with a multi-channel microphone), the spatial parameters describe relations between the channels or combinations (intermediate downmixes) of channels in a time- and frequency dependent manner.
According to another aspect of the invention there is provided a send effect device for generating an output signal from an input signal by applying a send effect processing to the input signal. It should be appreciated that the features, advantages, comments etc. described above are equally applicable to this aspect of the invention.
These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.
The invention proposes a method of generating an output signal 121 by applying a send effect processing to the input signal 101, which compensates for an unequal weighing of component signals in the input signal 101 in dependence of the parameters 102. The component signals contributing to the input signal 101 are often unequally weighted. The send effect device 100 generates the output signal 121 in such a manner that the unequal weighting is compensated for in dependence of the parameters 102. Parameters 102 comprise dependencies between the weighted component signals. In particular, parameters 102 comprise information about relative contributions of individual weighted component signals to the input signal 101. The parameters 102 allow estimating of the weighted component signals relative to the input signal. Since the weights used to weigh the component signals are known, since they are prescribed by the MPEG Surround bit-stream and decoder, the component signals themselves can be estimated. This leads to efficient processing in order to compensate the unequal weighting of the component signals in the input signal 101.
According to an embodiment, generating the output signal 121 comprises adapting the input signal 101. In this case the step of adapting the input signal precedes the step of applying a send effect processing.
with cij calculated from the MPEG Surround parameters and potentially HRTF data,
the output of the circuit 410 is a result of the matrix multiplication:
Due to dependence of Tumx matrix on the MPEG Surround parameters, the parameters 102 are also fed into the circuit 410. The resulting intermediate signals are fed into a gain compensation circuit 420, in which each of the intermediate signals is scaled with a respective gain to compensate the unequal weighting of the component signals comprised in the input signal. The circuit 420 implements a matrix multiplication of a vector comprising the three intermediate signals with a gain compensation matrix:
wherein Gl is a gain that corresponds to the left intermediate signal, Gr is a gain that corresponds to the right intermediate signal, and Gc is a gain corresponding to the center intermediate signal. The gains Gl and Gr are employed to compensate for any power loss due to surround gain gs. The gain Gc is employed to compensate for the power increase due to the center gain gc. This gain is independent of the MPEG Surround parameters and equal to Gc=1/(2·gc). The meaning of the surround gain and the center gain will be explained in more detail when
In an embodiment, the respective gain Gl, Gr, or Gc corresponding to the respective intermediate signal (the left intermediate signal, the right intermediate signal, or the center intermediate signal) is calculated as a weighted sum of predetermined further gains, wherein the predetermined further gains are derived from weights used to create the input signal 101. These predetermined further gains are weighted with respective weights that are derived from relative contributions of the weighted component signals to the respective intermediate signal.
The respective gains Gl and Gr are preferably calculated according to the following general expression:
wherein gf is the actual weight that has been used to scale the front channel signal pertaining to the input signal (typically gf=1, see the description of
The relative contribution of the weighted component signals to the respective intermediate signal is derived from an intensity difference IIDl, or IIDr (where the indices l and r stand for “left channel” and “right channel” respectively), between the weighted component signals contributing to the intermediate signal, wherein the intensity difference is derived from the parameters 102. These relative contributions are indicated by use of function f and (1−f). IIDl is the logarithmic inter-channel intensity difference (IID) between the weighted left front channel and the weighted left surround channel, and IIDr is logarithmic inter-channel intensity difference (IID) between the weighted right front channel and the weighted right surround channel. An example of f(IID) is:
Other functions are also possible, they should however map the logarithmic IID values to weights with the values between 0 and 1.
The scaled intermediate signals 421, 422, and 423 are fed into the circuit 430, which is the Three-To-Two (inverse-TTT) encoder module known from the MPEG Surround. The circuit 430 downmixes the three scaled intermediate signals into the signal 103 which subsequently is fed into the send effect processing circuit 110. For Tdmx, being the matrix representing the inverse-TTT module, the downmixing is implemented as matrix multiplication by:
Although the downmixing indicated above results in the stereo signal 103, the downmixing could also provide a mono signal.
For the example depicted in
Although circuits 410, 420, and 430 are depicted as separate circuits in
The circuit 110 depicts the send effect processing circuit, which comprises circuits 530, 520, and 510. In the circuit 530 the downmixing of the stereo signal 103, which resulted from adapting the input signal 101, is done resulting in a mono downmix 501. This downmix 501 is fed in parallel to the circuits 520 and 510 which create the reverberation output signal 121 from the downmix signal 501. For reverberation send effect the processing used in the circuits 510 and 520 can be as described in William G. Gardner, “Reverberation Algorithms” in “Applications of Digital Signal Processing to Audio and Acoustics”. Mark Kahrs and Karlheinz Brandenburg (Editors), Kluwer, March 1998, or Shreyas A. Paranjpe, Time-variant Orthogonal Matrix Feedback Delay Network Reverberator, Audio Engineering Society 110th Convention Paper 5381, Amsterdam, The Netherlands, 12-15 May 2001. Other send effect processing is described in DAFX: Digital Audio Effects, Udo Zolzer, Xavier Amatriain, Daniel Arfib, Jordi Bonada, Giovanni De Poli, Pierre Dutilleux, Gianpaolo Evangelista, Florian Keiler, Alex Loscos, Davide Rocchesso, Mark Sandler, Xavier Serra, Todor Todoroff, Contributor Udo Zolzer, Xavier Amatriain, Daniel Arfib, John Wiley and Sons, 2002.
Although the number of the intermediate signals is three, the number of intermediate signals is not restricted to three only and it could take any other value. However, the number of intermediate signals should preferably not exceed the number of the component signals. For MPEG Surround when the input signal is mono the preferable number of intermediate signals takes the following values: two, three, or five, which relates to specific configurations favoured by MPEG Surround.
The input signal 101 can also be a mono signal.
g=g
1
·c
0,1
·c
1,1
·c
3,1
+g
2
·c
0,1
·c
1,1
·c
3,2
+g
3
·c
0,1
·c
1,2
·c
4,1
+g
4
·c
0,1
·c
1,2
·C
4,2
+g
5
·c
0,2
·c
2,1
+g
6
·c
0,2
·c
2,2
where ci,j is defined by the IID of One-To-Two (OTT) box i as follows:
wherein the index i takes on values from 0 to 4 where index with a value 0 relates to the circuit 750, 1 to the circuit 740, 2 to the circuit 730, 3 to the circuit 710, and 4 to the circuit 720. Index j takes on values 1 or 2 and indicates the output channel of the corresponding OTT box i in the MPEG Surround decoder configuration (inverse of
In a further embodiment, the input signal 101 is scaled with a gain 120 calculated as a weighted sum of further gains, wherein the further gains are derived from the parameters 102 corresponding to the weighted component signals, wherein the further gains are weighted with weights that are derived from relative contributions of the weighted component signals or combinations of the weighted component signals to the input signal. The relative contribution of the weighted component signals or the combinations of the weighted component signals are derived from intensity differences between weighted component signals contributing to the input signal, wherein the intensity differences are derived from the parameters 102. As indicated above the signals 103a and 103b can thus be expressed as the result of the following matrix multiplication:
wherein the gains g1 and g2 are referred to as further gains.
For the embodiment of
g
m
=g
1
·f(IIDlr)a+g2·(1−f(IIDlr))a,
to both outputs of circuits 510 and 520, which perform the send effect processing. The gains may be delayed and/or adjusted to incorporate e.g. time-spreading effect which is relevant for the reverberation effect. In such a case the gains gm′ are modified such that:
where for example
g
m
′=α·g
m
[n]+(1−α)·gm[n−1],
with α a coefficient that weighs the gains of the current frame (n) and the previous frame (n−1) according to the temporal spreading of the signal intensity over subsequent frames by the reverberation.
In a further embodiment, the input signal and the parameters are the downmix signal and the parameters, respectively, in accordance with the MPEG Surround standard. The relation of the input signal to the downmix and the parameters to the spatial parameters of MPEG Surround should be clear based on the description of the figures.
Although the present invention has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Rather, the scope of the present invention is limited only by the accompanying claims. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in accordance with the invention. In the claims, the term comprising does not exclude the presence of other elements or steps.
Furthermore, although individually listed, a plurality of means, elements or method steps may be implemented by e.g. a single unit or processor. Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and the inclusion in different claims does not imply that a combination of features is not feasible and/or advantageous. Also the inclusion of a feature in one category of claims does not imply a limitation to this category but rather indicates that the feature is equally applicable to other claim categories as appropriate. In addition, singular references do not exclude a plurality. Thus references to “a”, “an”, “first”, “second” etc. do not preclude a plurality. Reference signs in the claims are provided merely as a clarifying example and shall not be construed as limiting the scope of the claims in any way. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer or other programmable device.
Number | Date | Country | Kind |
---|---|---|---|
08172499.9 | Dec 2008 | EP | regional |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/IB09/55779 | 12/16/2009 | WO | 00 | 6/22/2011 |