BRIEF DESCRIPTION OF THE DRAWINGS
Preferred embodiments of the present invention are subsequently explained with reference to the accompanying Figures, in which:
FIG. 1 is a block diagram of a preferred audio encoder;
FIG. 2 is a block diagram of a preferred audio decoder;
FIG. 3
a is a schematic representation of the encoded audio signal;
FIG. 3
b is a schematic representation of the side information for the first and/or the second time portion of FIG. 3a;
FIG. 4 is a representation of a prior art FIR pre-filter or post-filter, which is suitable for use in the present invention;
FIG. 5 illustrates the warping characteristic of a filter dependent on the warping factor;
FIG. 6 illustrates an inventive audio processor having a linear filter having a time-varying warping characteristic and a controller;
FIG. 7 illustrates a preferred embodiment of the inventive audio encoder;
FIG. 8 illustrates a preferred embodiment for an inventive audio decoder;
FIG. 9 illustrates a prior art filterbank-based coding algorithm having an encoder and a decoder;
FIG. 10 illustrates a prior art pre/post-filter based audio encoding algorithm having an encoder and a decoder; and
FIG. 11 illustrates a prior art LPC coding algorithm having an encoder and a decoder.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
Preferred embodiments of the present invention provide a uniform method that allows coding of both general audio signals and speech signals with a coding performance that—at least—matches the performance of the best known coding schemes for both types of signals. It is based on the following considerations:
- For coding of general audio signals, it is essential to shape the coding noise spectral envelope according to a masking threshold curve (according to the idea of “perceptual audio coding”), and thus a perceptually warped frequency scale is desirable. Nonetheless, there may be certain (e.g. harmonic) audio signals where a uniform frequency resolution would perform better that a perceptually warped one because the former can better resolve their individual spectral fine structure.
- For the coding of speech signals, the state of the art coding performance can be achieved by means of regular (non-warped) linear prediction. There may be certain speech signals for which some amount of warping improves the coding performance.
In accordance with the inventive idea, this dilemma is solved by a coding system that includes an encoder filter that can smoothly fade in its characteristics between a fully warped operation, as it is generally preferable for coding of music signals, and a non-warped operation, as it is generally preferable for coding of speech signals. Specifically, the proposed inventive approach includes a linear filter with a time-varying warping factor. This filter is controlled by an extra input that receives the desired warping factor and modifies the filter operation accordingly.
An operation of such a filter permits the filter to act both as a model of the masking curve (post-filter for coding of music, with warping on, λ=λ0), and as a model of the signal's spectral envelope (Inverse LPC filter for coding of speech, with warping off, λ=0), depending on the control input. If the inventive filter is equipped to handle also a continuum of intermediate warping factors 0≦λ≦λ0 then furthermore also soft in-between characteristics are possible.
Naturally, the inverse decoder filtering mechanism is similarly equipped, i.e. a linear decoder filter with a time-varying warping factor and can act as a perceptual pre-filter as well as an LPC filter.
In order to generate a well-behaved filtered signal to be coded subsequently, it is desirable to not switch instantaneously between two different values of the warping factor, but to apply a soft transition of the warping factor over time. As an example, a transition of 128 samples between unwarped and fully perceptually warped operation avoids undesirable discontinuities in the output signal.
Using such a filter with variable warping, it is possible to build a combined speech/audio coder which achieves both optimum speech and audio coding quality in the following way (see FIG. 7 or 8):
- The decision about the coding mode to be used (“Speech mode” or “Music mode”) is performed in a separate module by carrying out an analysis of the input signal and can be based on known techniques for discriminating speech signals from music. As a result, the decision module produces a decision about the coding mode/and an associated optimum warping factor for the filter. Furthermore, depending on the this decision, it determines a set of suitable filter coefficients which are appropriate for the input signal at the chosen coding mode, i.e. for coding of speech, an LPC analysis is performed (with no warping, or a low warping factor) whereas for coding of music, a masking curve is estimated and its inverse is converted into warped spectral coefficients.
- The filter with the time varying warping characteristics is used as a common encoder/decoder filter and is applied to the signal depending on the coding mode decision/warping factor and the set of filter coefficients produced by the decision module.
- The output signal of the filtering stage is coded by either a speech coding kernel (e.g. CELP coder) or a generic audio coder kernel (e.g. a filterbank/subband coder, or a predictive audio coder), or both, depending on the coding mode.
- The information to the transmitted/stored comprises the coding mode decision (or an indication of the warping factor), the filter coefficients in some coded form, and the information delivered by the speech/excitation and the generic audio coder.
The corresponding decoder works accordingly: It receives the transmitted information, decodes the speech and generic audio parts according to the coding mode information, combines them into a single intermediate signal (e.g. by adding them), and filters this intermediate signal using the coding mode/warping factor and filter coefficients to form the final output signal.
Subsequently, a preferred embodiment of the inventive audio encoder will be discussed in connection with FIG. 1. The FIG. 1 audio encoder is operative for encoding an audio signal input at line 10. The audio signal is input into a pre-filter 12 for generating a pre-filtered audio signal appearing at line 14. The pre-filter has a variable warping characteristic, the warping characteristic being controllable in response to a time-varying control signal on line 16. The control signal indicates a small or no warping characteristic or a comparatively high warping characteristic. Thus, the time-varying warp control signal can be a signal having two different states such as “1” for a strong warp or a “0” for no warping. The intended goal for applying warping is to obtain a frequency resolution of the pre-filter similar to the BARK scale. However, also different states of the signal/warping characteristic setting are possible.
Furthermore, the inventive audio encoder includes a controller 18 for providing the time-varying control signal, wherein the time varying control signal depends on the audio signal as shown by line 20 in FIG. 1. Furthermore, the inventive audio encoder includes a controllable encoding processor 22 for processing the pre-filtered audio signal to obtain an encoded audio signal output at line 24. Particularly, the encoding processor 22 is adapted to process the pre-filtered audio signal in accordance with a first coding algorithm adapted to a specific signal pattern, or in accordance with a second, different encoding algorithm suitable for encoding a general audio signal. Particularly, the encoding processor 22 is adapted to be controlled by the controller 18 preferably via a separate encoder control signal on line 26 so that an audio signal portion being filtered using the comparatively high warping factor is processed using the second encoding algorithm to obtain the encoded signal for this audio signal portion, so that an audio signal portion being filtered using no or only a small warping characteristic is processed using the first encoding algorithm.
Thus, as it is shown in the control table 28 for the signal on control line 26, in some situations when processing an audio signal, no or only a small warp is performed by the filter for a signal being filtered in accordance with the first coding algorithm, while, when a strong and preferably perceptually full-scale warp is applied by the pre-filter, the time portion is processed using the second coding algorithm for general audio signals, which is preferably based on hiding quantization noise below a psycho-acoustic masking threshold. Naturally, the invention also covers the case that for a further portion of the audio signal, which has the signal-specific pattern, a high warping characteristic is applied while for an even further portion riot having the specific signal pattern, a low or no warping characteristic is used. This can be for example determined by an analysis by synthesis encoder decision or by any other algorithms know in the art. However, the encoder module control can also be fixedly set depending on the transmitted warping factor or the warping factor can be derived from a transmitted coder module indication. Furthermore, both information items can be transmitted as side information, i.e., the coder module and the warping factor.
FIG. 2 illustrates an inventive decoder for decoding an encoded audio signal input at line 30. The encoded audio signal has a first portion encoded in accordance with a first coding algorithm adapted to a specific signal pattern, and has a second portion encoded in accordance with a different second coding algorithm suitable for encoding a general audio signal. Particularly, the inventive decoder comprises a detector 32 for detecting a coding algorithm underlying the first or the second portion. This detection can take place by extracting side information from the encoded audio signal as illustrated by broken line 34, and/or can take place by examining the bit-stream coming into a decoding processor 36 as illustrated by broken line 38. The decoding processor 36 is for decoding in response to the detector as illustrated by control line 40 so that for both the first and second portions the correct coding algorithm is selected.
Preferably, the decoding processor is operative to use the first coding algorithm for decoding the first time portion and to use the second coding algorithm for decoding the second time portion so that the first and the second decoded time portions are output on line 42. Line 42 carries the input into a post-filter 44 having a variable warping characteristic. Particularly, the post-filter 44 is controllable using a time-varying warp control signal on line 46 so that this post-filter has only small or no warping characteristic in a first state and has a high warping characteristic in a second state.
Preferably, the post-filter 44 is controlled such that the first time portion decoded using the first coding algorithm is filtered using the small or no warping characteristic and the second time portion of the decoded audio signal is filtered using the comparatively strong warping characteristic so that an audio decoder output signal is obtained at line 48.
When looking at FIG. 1 and FIG. 2, the first coding algorithm determines the encoder-related steps to be taken in the encoding processor 22 and the corresponding decoder-related steps to be implemented in decoding processor 36. Furthermore, the second coding algorithm determines the encoder-related second coding algorithm steps to be used in the encoding processor and corresponding second coding algorithm-related decoding steps to be used in decoding processor 36.
Furthermore, the pre-filter 12 and the post-filter 44 are, in general, inverse to each other. The warping characteristics of those filters are controlled such that the post-filter has the same warping characteristic as the pre-filter or at least a similar warping characteristic within a 10 percent tolerance range.
Naturally, when the pre-filter is not warped due to the fact that there is e.g. a signal having the specific signal pattern, then the post-filter also does not have to be a warped filter.
Nevertheless, the pre-filter 12 as well as the post-filter 44 can implement any other pre-filter or post-filter operations required in connection with the first coding algorithm or the second coding algorithm as will be outlined later on.
FIG. 3
a illustrates an example of an encoded audio signal as obtained on line 24 of FIG. 1 and as can be found on line 30 of FIG. 2. Particularly, the encoded audio signal includes a first time portion in encoded form, which has been generated by the first coding algorithm as outlined at 50 and corresponding side information 52 for the first portion. Furthermore, the bit-stream includes a second time portion in encoded form as shown at 54 and side information 56 for the second time portion. It is to be noted here that the order of the items in FIG. 3a may vary. Furthermore, the side information does not necessarily have to be multiplexed between the main information 50 and 54. Those signals can even come from separate sources as dictated by external requirements or implementations.
FIG. 3
b illustrates side information for the explicit signaling embodiment of the present invention for explicitly signaling the warping factor and encoder mode, which can be used in 52 and 56 of FIG. 3a. This is indicated below the FIG. 3b side information stream. Hence, the side information may include a coding mode indication explicitly signaling the first or the second coding algorithm underlying this portion to which the side information belongs to.
Furthermore, a warping factor can be signaled. Signaling of the warping factor is not necessary, when the whole system can only use two different warping characteristics, i.e., no warping characteristic as the first possibility and a perceptually full-scale warping characteristic as the second possibility. In this case, a warping factor can be fixed and does not necessarily have to be transmitted.
Nevertheless, in preferred embodiments, the warping factor can have more than these two extreme values so that an explicit signaling of the warping factor such as by absolute values or differentially coded values is used.
Furthermore, it is preferred that the pre-filter not only implements is warped but also implements tasks dictated by the first coding algorithm and the second coding algorithm, which leads to a more efficient functionality of the first and the second coding algorithms.
When the first coding algorithm is an LPC-based coding algorithm, then the pre-filter also performs the functionality of the LPC analysis filter and the post-filter on the decoder-side performs the functionality of an LPC synthesis filter.
When the second coding algorithm is a general audio encoder not having a specific noise shaping functionality, the pre-filter is preferably an LPC filter, which pre-filters the audio signal so that, after pre-filtering, psychoacoustically more important portions are amplified with respect to psychoacoustically less important portions. On the decoder-side, the post-filter is implemented as a filter for regenerating a situation similar to a situation before pre-filtering, i.e. an inverse filter which amplifies less important portions with respect to more important portions so that the signal after post-filtering is—apart from coding errors—similar to the original audio signal input into the encoder.
The filter coefficients for the above described pre-filter are preferably also transmitted via side information from the encoder to the decoder.
Typically, the pre-filter as well as the post-filter will be implemented as a warped FIR filter, a structure of which is illustrated in FIG. 4, or as a warped IIR digital filter. The FIG. 4 filter is described in detail in [KHL 97]. Examples for warped IIR filters are also shown in [KHL 97]. All those digital filters have in common that they have warped delay elements 60 and weighting coefficients or weighting elements indicated by β0, β1, β2, . . . . A filter structure is transformed to a warped filter, when a delay element in an unwarped filter structure (not shown here) is replaced by an all-pass filter, such as a first-order allpass filter D(z), as illustrated in on both sides of the filter structures in FIG. 4. A computationally efficient implementation of the left structure is shown in the right of FIG. 4, where the explicit usage of the warping factor λ and the implementation thereof is shown.
Thus, the filter structure to the right of FIG. 4 can easily be implemented within the pre-filter as well as within the post-filter, wherein the warping factor is controlled by the parameter λ, while the filter characteristic, i.e., the filter coefficients of the LPC analysis/synthesis or pre-filtering or post-filtering for amplifying/damping psycho-acoustically more important portions is controlled by setting the weighting parameters β0, β1, β2, . . . to appropriate values.
FIG. 5 illustrates the dependence of the frequency-warping characteristic on the warping factor λ for λs between −0.8 and +0.8. No warping at all will be obtained, when λ is set to 0.0. A psycho-acoustically full-scale warp is obtained by setting λ between 0.3 and 0.4. Generally, the optimum warping factor depends on the chosen sampling rate and has a value of between about 0.3 and 0.4 for sampling rates between 32 and 48 kHz. The then obtained non-uniform frequency resolution by using the warped filter is similar to the BARK or ERB scale. Substantially stronger warping characteristics can be implemented, but those are only useful in certain situations, which can happen when the controller determines that those higher warping factors are useful.
Thus, the pre-filter on the encoder-side will preferably have positive warping factors λ to increase the frequency resolution in the low frequency range and to decrease the frequency resolution in the high frequency range. Hence, the post-filter on the decoder-side will also have the positive warping factors. Thus, a preferred inventive time-varying warping filter is shown in FIG. 6 at 70 as a part of the audio processor. The inventive filter is, preferably, a linear filter, which is implemented as a pre-filter or a post-filter for filtering to amplify or damp psycho-acoustically more/less important portions or which is implemented as an LPC analysis/synthesis filter depending on the control signal of the system. It is to note at this point that the warped filter is a linear filter and does not change the frequency of a component such as a sine wave input into the filter. However, when it is assumed that the filter before warping is a low pass filter, the FIG. 5 diagram has to be interpreted as set out below.
When the example sine wave has a normalized original frequency of 0.6, then the filter would apply—for a warping factor equal to 0.0—the phase and amplitude weighting defined by the filter impulse response of this unwarped filter.
When a warping factor of 0.8 is set for this lowpass filter (now the filter becomes a warped filter), the sine wave having a normalized frequency of 0.6 will be filtered such that the output is weighted by the phase and amplitude weighting which the unwarped filter has for a normalized frequency of 0.97 in FIG. 5. Since this filter is a linear filter, the frequency of the sine wave is not changed.
Depending on the situation, when the filter 70 is only warped, then a warping factor or, generally, the warping control 16; or 46, has to be applied. The filter coefficients βi are derived from the masking threshold. These filter coefficients can be pre- or post-filter coefficients, or LPC analysis/synthesis filter coefficients, or any other filter coefficients useful in connection with any first or second coding algorithms.
Thus, an audio processor in accordance with the present invention includes, in addition to the filter having variable warping characteristics, the controller 18 of FIG. 1 or the controller implemented as the coding algorithm detector 32 of FIG. 2 or a general audio input signal analyzer looking for a specific signal pattern in the audio input 10/42 so that a certain warping characteristic can be set, which fits to the specific signal pattern so that a time-adapted variable warping of the audio input be it an encoded or a decoded audio input can be obtained. Preferably, the pre-filter coefficients and the post-filter coefficients are identical.
The output of the audio processor illustrated in FIG. 6 which consists of the filter 70 and the controller 74 can then be stored for any purposes or can be processed by encoding processor 22, or by an audio reproduction device when the audio processor is on the decoder-side, or can be processed by any other signal processing algorithms.
Subsequently, FIGS. 7 and 8 will be discussed, which show preferred embodiments of the inventive encoder (FIG. 7) and the inventive decoder (FIG. 8). The functionalities of the devices are similar to the FIG. 1, FIG. 2 devices. Particularly, FIG. 7 illustrates the embodiment, wherein the first coding algorithm is a speech-coder like coding algorithm, wherein the specific signal pattern is a speech pattern in the audio input 10. The second coding algorithm 22b is a generic audio coder such as the straightforward filterbank-based audio coder as illustrated and discussed in connection with FIG. 9, or the pre-filter/post-filter audio coding algorithm as illustrated in FIG. 10.
The first coding algorithm corresponds to the FIG. 11 speech coding system, which, in addition to an LPC analysis/synthesis filter 1100 and 1102 also includes a residual/excitation coder 1104 and a corresponding excitation decoder 1106. In this embodiment, the time-varying warped filter 12 in FIG. 7 has the same functionality as the LPC filter 1100, and the LPC analysis implemented in block 1108 in FIG. 11 is implemented in controller 18.
The residual/excitation coder 1104 corresponds to the residual/excitation coder kernel 22a in FIG. 7. Similarly, the excitation decoder 1106 corresponds to the residual/excitation decoder 36a in FIG. 8, and the time-varying warped filter 44 has the functionality of the inverse LPC filter 1102 for a first time portion being coded in accordance with the first coding algorithm.
The LPC filter coefficients generated by LPC analysis block 1108 correspond to the filter coefficients shown at 90 in FIG. 7 for the first time portion and the LPC filter coefficients input into block 1102 in FIG. 11 correspond to the filter coefficients on line 92 of FIG. 8. Furthermore, the FIG. 7 encoder includes an encoder output interface 94, which can be implemented as a bit-stream multiplexer, but which can also be implemented as any other device producing a data stream suitable for transmission and/or storage. Correspondingly, the FIG. 8 decoder includes an input interface 96, which can be implemented as a bit-stream demultiplexer for de-multiplexing the specific time portion information as discussed in connection with FIG. 3a and for also extracting the required side-information as illustrated in FIG. 3b.
In the FIG. 7 embodiment, both encoding kernels 22a, 22b, have a common input 96, and are controlled by the controller 18 via lines 97a and 97b. This control makes sure that, at a certain time instant, only one of both encoder kernels 22a, 22b outputs main and side information to the output interface. Alternatively, both encoding kernels could work fully parallel, and the encoder controller 18 would make sure that only the output of the encoding kernel is input into the bit-stream, which is indicated by the coding mode information while the output of the other encoder is discarded.
Again alternatively, both decoders can operate in parallel and outputs thereof can be added. In this situation, it is preferred to use a medium warping characteristic for the encoder-side pre-filter and for the decoder-side post-filter. Furthermore, this embodiment processes e.g. a speech portion of a signal such as a certain frequency range or—generally—signal portion by the first coding algorithm and the remainder of the signal by the second general coding algorithm. Then outputs of both coders are transmitted from the encoder to the decoder side. The decoder-side combination makes sure that the signal is rejoined before being post-filtered.
Any kind of specific controls can be implemented as long as they make sure that the output encoded audio signal 24 has a sequence of first and second portions as illustrated in FIG. 3 or a correct combination of signal portions such as a speech portion and a general audio portion.
On the decoder-side, the coding mode information is used for decoding the time portion using the correct decoding algorithm so that a time-staggered pattern of first portions and second portions obtain at the outputs of decoder kernels 36a, and 36b, which are, then, multiplexed into a single time domain signal, which is illustrated schematically using the adder symbol 36c. Then, at the output of element 36c, there is a time-domain audio signal, which only has to be post-filtered so that the decoded audio signal is obtained.
As discussed earlier in the summary after the Brief Description of the Drawings section, both the encoder in FIG. 7 as well as the decoder in FIG. 8 may include an interpolator 100 or 102 so that a smooth transition via a certain time portion, which at least includes two samples, but which preferably includes more than 50 samples and even more than 100 samples, is implementable. This makes sure that coding artifacts are avoided, which might be caused by rapid changes of the warping factor and the filter coefficients. Since, however, the post-filter as well as the pre-filter fully operate in the time domain, there are no problems related to block-based specific implementations. Thus, one can change, when FIG. 4 is again considered, the values for β0, β1, β2, . . . and λ from sample to sample so that a fade over from a, for example, fully warped state to another state having no warp at all is possible. Although one could transmit interpolated parameters, which would save the interpolator on the decoder-side, it is preferred to not transmit the interpolated values but to transmit the values before interpolation since less side-information bits are required for the latter option.
Furthermore, as already indicated above, the generic audio coder kernel 22b as illustrated in FIG. 7 may be identical to the coder 1000 in FIG. 10. In this context, the pre-filter 12 will also perform the functionality of the pre-filter 1002 in FIG. 10. The perceptual model 1004 in FIG. 10 will then be implemented within controller 18 of FIG. 7. The filter coefficients generated by the perceptual model 1004 correspond to the filter coefficients on line 90 in FIG. 7 for a time portion, for which the second coding algorithm is on.
Analogously, the decoder 100G in FIG. 10 is implemented by the generic audio decoder kernel 36b in FIG. 8, and the post-filter 1008 is implemented by the time-varying warped filter 44 in FIG. 8. The preferably coded filter coefficients generated by the perceptual model are received, on the decoder-side, on line 92, so that a line titled “filter coefficients” entering post-filter 1008 in FIG. 10 corresponds to line 92 in FIG. 8 for the second coding algorithm time portion.
However, compared to two parallel working encoders in accordance with FIGS. 10 and 11, which are both not perfect due to audio quality and bit rate, the inventive encoder devices and the inventive decoder devices only use a single, but controllable filter and perform a discrimination on the input audio signal to find out whether the time portion of the audio signal has the specific pattern or is just a general audio signal.
Regarding the audio analyzer within controller 18, a variety of different implementations can be used for determining, whether a portion of an audio signal is a portion having the specific signal pattern or whether this portion does not have this specific signal pattern, and, therefore, has to be processed using the general audio encoding algorithm. Although preferred embodiments have been discussed, wherein the specific signal pattern is a speech signal, other signal-specific patterns can be determined and can be encoded using such signal-specific first encoding algorithms such as encoding algorithm for harmonic signals, for noise signals, for tonal signals, for pulse-train-like Signals, etc.
Straightforward detectors are analysis by synthesis detectors, which, for example, try different encoding algorithms, together with different warping detectors to find out the best warping factor together with the best filter coefficients and the best coding algorithm. Such analysis by synthesis detectors are in some cases quite computationally expensive. This does not matter in a situation, wherein there is a small number of encoders and a high number of decoders, since the decoder can be very simple in that case. This is due to the fact that only the encoder performs this complex computational task, while the decoder can simply use the transmitted side-information.
Other signal detectors are based on straightforward pattern analyzing algorithms, which look for a specific signal pattern within the audio signal and signal a positive result, when a matching degree exceeds a certain threshold. More information on such detectors is given in [BLS05].
Moreover, depending on certain implementation requirements of the inventive methods, the inventive methods can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, in particular a disk or a CD having electronically readable control signals stored thereon, which can cooperate with a programmable computer system such that the inventive methods are performed. Generally, the present invention is, therefore, a computer program product with a program code stored on a machine-readable carrier, the program code being configured for performing at least one of the inventive methods, when the computer program products runs on a computer. In other words, the inventive methods are, there fore, a computer program having a program code for performing the inventive methods, when the computer program runs on a computer.
The above-described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the impending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.
Patent Application
20080004869
