Patent Grant
10854211
The present application is related to stereo processing or, generally, multi-channel processing, where a multi-channel signal has two channels such as a left channel and a right channel in the case of a stereo signal or more than two channels, such as three, four, five or any other number of channels.
Stereo speech and particularly conversational stereo speech has received much less scientific attention than storage and broadcasting of stereophonic music. Indeed in speech communications monophonic transmission is still nowadays mostly used. However with the increase of network bandwidth and capacity, it is envisioned that communications based on stereophonic technologies will become more popular and bring a better listening experience.
Efficient coding of stereophonic audio material has been for a long time studied in perceptual audio coding of music for efficient storage or broadcasting. At high bitrates, where waveform preserving is crucial, sum-difference stereo, known as mid/side (M/S) stereo, has been employed for a long time. For low bit-rates, intensity stereo and more recently parametric stereo coding has been introduced. The latest technique was adopted in different standards as HeAACv2 and Mpeg USAC. It generates a downmix of the two-channel signal and associates compact spatial side information.
Joint stereo coding are usually built over a high frequency resolution, i.e. low time resolution, time-frequency transformation of the signal and is then not compatible to low delay and time domain processing performed in most speech coders. Moreover the engendered bit-rate is usually high.
On the other hand, parametric stereo employs an extra filter-bank positioned in the front-end of the encoder as pre-processor and in the back-end of the decoder as post-processor. Therefore, parametric stereo can be used with conventional speech coders like ACELP as it is done in MPEG USAC. Moreover, the parametrization of the auditory scene can be achieved with minimum amount of side information, which is suitable for low bit-rates. However, parametric stereo is as for example in MPEG USAC not specifically designed for low delay and does not deliver consistent quality for different conversational scenarios. In conventional parametric representation of the spatial scene, the width of the stereo image is artificially reproduced by a decorrelator applied on the two synthesized channels and controlled by Inter-channel Coherence (ICs) parameters computed and transmitted by the encoder. For most stereo speech, this way of widening the stereo image is not appropriate for the recreating the natural ambience of speech which is a pretty direct sound since it is produced by a single source located at a specific position in the space (with sometimes some reverberation from the room). By contrast, music instruments have much more natural width than speech, which can be better imitated by decorrelating the channels.
Problems also occur when speech is recorded with non-coincident microphones, like in A-B configuration when microphones are distant from each other or for binaural recording or rendering. Those scenarios can be envisioned for capturing speech in teleconferences or for creating a virtually auditory scene with distant speakers in the multipoint control unit (MCU). The time of arrival of the signal is then different from one channel to the other unlike recordings done on coincident microphones like X-Y (intensity recording) or M-S (Mid-Side recording). The computation of the coherence of such non time-aligned two channels can then be wrongly estimated which makes fail the artificial ambience synthesis.
Conventional-technology references related to stereo processing are U.S. Pat. No. 5,434,948 or 8,811,621.
Document WO 2006/089570 A1 discloses a near-transparent or transparent multi-channel encoder/decoder scheme. A multi-channel encoder/decoder scheme additionally generates a waveform-type residual signal. This residual signal is transmitted together with one or more multi-channel parameters to a decoder. In contrast to a purely parametric multi-channel decoder, the enhanced decoder generates a multi-channel output signal having an improved output quality because of the additional residual signal. On the encoder-side, a left channel and a right channel are both filtered by an analysis filter-bank. Then, for each subband signal, an alignment value and a gain value are calculated for a subband. Such an alignment is then performed before further processing. On the decoder-side, a de-alignment and a gain processing is performed and the corresponding signals are then synthesized by a synthesis filter-bank in order to generate a decoded left signal and a decoded right signal.
On the other hand, parametric stereo employs an extra filter-bank positioned in the front-end of the encoder as pre-processor and in the back-end of the decoder as post-processor. Therefore, parametric stereo can be used with conventional speech coders like ACELP as it is done in MPEG USAC. Moreover, the parametrization of the auditory scene can be achieved with minimum amount of side information, which is suitable for low bit-rates. However, parametric stereo is as for example in MPEG USAC not specifically designed for low delay and the overall system shows a very high algorithmic delay.
According to an embodiment, an apparatus for encoding a multi-channel signal including at least two channels may have: a time-spectral converter for converting sequences of blocks of sampling values of the at least two channels into a frequency domain representation having sequences of blocks of spectral values for the at least two channels; a multi-channel processor for applying a joint multi-channel processing to the sequences of blocks of spectral values to obtain at least one result sequence of blocks of spectral values including information related to the at least two channels; a spectral-time converter for converting the result sequence of blocks of spectral values into a time domain representation including an output sequence of blocks of sampling values; and a core encoder for encoding the output sequence of blocks of sampling values to obtain an encoded multi-channel signal, wherein the core encoder is configured to operate in accordance with a first frame control to provide a sequence of frames, wherein a frame is bounded by a start frame border and an end frame border, and wherein the time-spectral converter or the spectral-time converter are configured to operate in accordance with a second frame control being synchronized to the first frame control, wherein the start frame border or the end frame border of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectral converter for each block of the sequence of blocks of sampling values or used by the spectral-time converter for each block of the output sequence of blocks of sampling values.
According to another embodiment, a method of encoding a multi-channel signal including at least two channels may have the steps of: converting sequences of blocks of sampling values of the at least two channels into a frequency domain representation having sequences of blocks of spectral values for the at least two channels; applying a joint multi-channel processing to the sequences of blocks of spectral values to obtain at least one result sequence of blocks of spectral values including information related to the at least two channels; converting the result sequence of blocks of spectral values into a time domain representation including an output sequence of blocks of sampling values; and core encoding the output sequence of blocks of sampling values to obtain an encoded multi-channel signal, wherein the core encoding operates in accordance with a first frame control to provide a sequence of frames, wherein a frame is bounded by a start frame border and an end frame border, and wherein the time-spectral converting or the spectral-time converting operates in accordance with a second frame control being synchronized to the first frame control, wherein the start frame border or the end frame border of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectral converting for each block of the sequence of blocks of sampling values or used by the spectral-time converting for each block of the output sequence of blocks of sampling values.
According to another embodiment, an apparatus for decoding an encoded multi-channel signal may have: a core decoder for generating a core decoded signal; a time-spectral converter for converting a sequence of blocks of sampling values of the core decoded signal into a frequency domain representation having a sequence of blocks of spectral values for the core decoded signal; a multi-channel processor for applying an inverse multi-channel processing to a sequence including the sequence of blocks to obtain at least two result sequences of blocks of spectral values; and a spectral-time converter for converting the at least two result sequences of blocks of spectral values into a time domain representation including at least two output sequences of blocks of sampling values, wherein the core decoder is configured to operate in accordance with a first frame control to provide a sequence of frames, wherein a frame is bounded by a start frame border and an end frame border, wherein the time-spectral converter or the spectral-time converter is configured to operate in accordance with a second frame control being synchronized to the first frame control, wherein the time-spectral converter or the spectral-time converter are configured to operate in accordance with a second frame control being synchronized to the first frame control, wherein the start frame border or the end frame border of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectral converter for each block of the sequence of blocks of sampling values or used by the spectral-time converter for each block of the at least two output sequences of blocks of sampling values.
According to another embodiment, a method of decoding an encoded multi-channel signal may have the steps of: generating a core decoded signal; converting a sequence of blocks of sampling values of the core decoded signal into a frequency domain representation having a sequence of blocks of spectral values for the core decoded signal; applying an inverse multi-channel processing to a sequence including the sequence of blocks to obtain at least two result sequences of blocks of spectral values; and converting the at least two result sequences of blocks of spectral values into a time domain representation including at least two output sequences of blocks of sampling values, wherein the generating the core decoded signal operates in accordance with a first frame control to provide a sequence of frames, wherein a frame is bounded by a start frame border and an end frame border, wherein the time spectral converting or the spectral time converting operates in accordance with a second frame control being synchronized to the first frame control, wherein the time-spectral converting or the spectral-time converting operate in accordance with a second frame control being synchronized to the first frame control, wherein the start frame border or the end frame border of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time spectral converting for each block of the sequence of blocks of sampling values or used by the spectral time converting for each block of the at least two output sequences of blocks of sampling values.
According to another embodiment, a non-transitory digital storage medium may have a computer program stored thereon to perform the inventive methods.
The present invention is based on the finding that at least a portion and advantageously all parts of the multi-channel processing, i.e., a joint multi-channel processing are performed in a spectral domain. Specifically, it is advantageous to perform the downmix operation of the joint multi-channel processing in the spectral domain and, additionally, temporal and phase alignment operations or even procedures for analyzing parameters for the joint stereo/joint multi-channel processing. Furthermore, a synchronization of the frame control for the core encoder and the stereo processing operating in the spectral domain is performed.
The core encoder is configured to operate in accordance with a first frame control to provide a sequence of frames, wherein a frame is bounded by a start frame border and an end frame border, and the time-spectral converter or the spectral-time converter are configured to operate in accordance with a second frame control being synchronized to the first frame control, wherein the start frame border or the end frame border of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectral converter (1000) for each block of the sequence of blocks of sampling values or used by the spectral-time converter for each block of the output sequence of blocks of sampling values.
In the invention, the core encoder of the multi-channel encoder is configured to operate in accordance with a framing control, and the time-spectral converter and the spectrum-time converter of the stereo post-processor and resampler are also configured to operate in accordance with a further framing control which is synchronized to the framing control of the core encoder. The synchronization is performed in such a way that a start frame border or an end frame border of each frame of a sequence of frames of the core encoder is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectral converter or the spectral time converter for each block of the sequence of blocks of sampling values or for each block of the resampled sequence of blocks of spectral values. Thus, it is assured that the subsequent framing operations operate in synchrony to each other.
In further embodiments, a look-ahead operation with a look-ahead portion is performed by the core encoder. In this embodiment, it is advantageous that the look-ahead portion is also used by an analysis window of the time-spectral converter where an overlap portion of the analysis window is used that has a length in time being lower than or equal to the length in time of the look-ahead portion.
Thus, by making the look-ahead portion of the core encoder and the overlap portion of the analysis window equal to each other or by making the overlap portion even smaller than the look-ahead portion of the core encoder, the time-spectral analysis of the stereo pre-processor can't be implemented without any additional algorithmic delay. In order to make sure that this windowed look-ahead portion does not influence the core encoder look-ahead functionality too much, it is advantageous to redress this portion using an inverse of the analysis window function.
In order to be sure that this is done with a good stability, a square root of sine window shape is used instead of a sine window shape as an analysis window and a sine to the power of 1.5 synthesis window is used for the purpose of synthesis windowing before performing the overlap operation at the output of the spectral-time converter. Thus, it is made sure that the redressing function assumes values that are reduced with respect to their magnitudes compared to a redressing function being the inverse of a sine-function.
Advantageously, a spectral domain resampling is performed either subsequent to the multi-channel processing or even before the multi-channel processing in order to provide an output signal from a further spectral-time converter that is already at an output sampling rate that may be used by a subsequently connected core encoder. But, the inventive procedure of synchronizing the frame control of the core encoder and the spectral time or time spectral converter can also be applied in a scenario where any spectral domain resampling is not executed.
On the decoder-side, it is advantageous to once again perform at least an operation for generating a first channel signal and a second channel signal from a downmix signal in the spectral domain and, advantageously, to perform even the whole inverse multi-channel processing in the spectral domain. Furthermore, the time-spectral converter is provided for converting the core decoded signal into a spectral domain representation and, within the frequency domain, the inverse multi-channel processing is performed.
The core decoder is configured to operate in accordance with a first frame control to provide a sequence of frames, wherein a frame is bounded by a start frame border and an end frame border. The time-spectral converter or the spectral-time converter is configured to operate in accordance with a second frame control being synchronized to the first frame control. Specifically, the time-spectral converter or the spectral-time converter are configured to operate in accordance with a second frame control being synchronized to the first frame control, wherein the start frame border or the end frame border of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectral converter for each block of the sequence of blocks of sampling values or used by the spectral-time converter for each block of the at least two output sequences of blocks of sampling values.
It is advantageous to use the same analysis and synthesis window shapes, since there is no redressing required, of course. On the other hand, it is advantageous to use a time gap on the decoder-side, where the time gap exists between an end of a leading overlapping portion of an analysis window of the time-spectral converter on the decoder-side and a time instant at the end of a frame output by the core decoder on the multi-channel decoder-side. Thus, the core decoder output samples within this time gap are not required for the purpose of analysis windowing by the stereo post-processor immediately, but may be used only for the processing/windowing of the next frame. Such a time gap can be, for example, implemented by using a non-overlapping portion typically in the middle of an analysis window which results in a shortening of the overlapping portion. However, other alternatives for implementing such a time gap can be used as well, but implementing the time gap by the non-overlapping portion in the middle is the advantageous way. Thus, this time gap can be used for other core decoder operations or smoothing operations between advantageously switching events when the core decoder switches from a frequency-domain to a time-domain frame or for any other smoothing operations that may be useful when the parameter changes or coding characteristic changes have occurred.
In an embodiment, a spectral domain resampling is either performed before the multi-channel inverse processing or is performed subsequent to the multi-channel inverse processing in such a way that, in the end, a spectral-time converter converts a spectrally resampled signal into the time domain at an output sampling rate that is intended for the time domain output signal.
Therefore, the embodiments allow to completely avoid any computational intensive time-domain resampling operations. Instead, the multi-channel processing is combined with the resampling. The spectral domain resampling is, in advantageous embodiments, either performed by truncating the spectrum in the case of downsampling or is performed by zero padding the spectrum in the case of upsampling. These easy operations, i.e., truncating the spectrum on the one hand or zero padding the spectrum on the other hand and advantageous additional scalings in order to account for certain normalization operations performed in spectral domain/time-domain conversion algorithms such as DFT or FFT algorithm complete the spectral domain resampling operation in a very efficient and low-delay manner.
Furthermore, it has been found that at least a portion or even the whole joint stereo processing/joint multi-channel processing on the encoder-side and the corresponding inverse multi-channel processing on the decoder-side is suitable for being executed in the frequency-domain. This is not only valid for the downmix operation as a minimum joint multi-channel processing on the encoder-side or an upmix processing as a minimum inverse multi-channel processing on the decoder-side. Instead, even a stereo scene analysis and time/phase alignments on the encoder-side or phase and time de-alignments on the decoder-side can be performed in the spectral domain as well. The same applies to the advantageously performed Side channel encoding on the encoder-side or Side channel synthesis and usage for the generation of the two decoded output channels on the decoder-side.
Therefore, an advantage of the present invention is to provide a new stereo coding scheme much more suitable for conversion of a stereo speech than the existing stereo coding schemes. Embodiments of the present invention provide a new framework for achieving a low-delay stereo codec and integrating a common stereo tool performed in frequency-domain for both a speech core coder and an MDCT-based core coder within a switched audio codec.
Embodiments of the present invention relate to a hybrid approach mixing elements from a conventional M/S stereo or parametric stereo. Embodiments use some aspects and tools from the joint stereo coding and others from the parametric stereo. More particularly, embodiments adopt the extra time-frequency analysis and synthesis done at the front end of the encoder and at the back-end of the decoder. The time-frequency decomposition and inverse transform is achieved by employing either a filter-bank or a block transform with complex values. From the two channels or multi-channel input, the stereo or multi-channel processing combines and modifies the input channels to output channels referred to as Mid and Side signals (MS).
Embodiments of the present invention provide a solution for reducing an algorithmic delay introduced by a stereo module and particularly from the framing and windowing of its filter-bank. It provides a multi-rate inverse transform for feeding a switched coder like 3GPP EVS or a coder switching between a speech coder like ACELP and a generic audio coder like TCX by producing the same stereo processing signal at different sampling rates. Moreover, it provides a windowing adapted for the different constraints of the low-delay and low-complex system as well as for the stereo processing. Furthermore, embodiments provide a method for combining and resampling different decoded synthesis results in the spectral domain, where the inverse stereo processing is applied as well.
Advantageous embodiments of the present invention comprise a multi-function in a spectral domain resampler not only generating a single spectral-domain resampled block of spectral values but, additionally, a further resampled sequence of blocks of spectral values corresponding to a different higher or lower sampling rate.
Furthermore, the multi-channel encoder is configured to additionally provide an output signal at the output of the spectral-time converter that has the same sampling rate as the original first and second channel signal input into the time-spectral converter on the encoder-side. Thus, the multi-channel encoder provides, in embodiments, at least one output signal at the original input sampling rate, that is advantageously used for an MDCT-based encoding. Additionally, at least one output signal is provided at an intermediate sampling rate that is specifically useful for ACELP coding and additionally provides a further output signal at a further output sampling rate that is also useful for ACELP encoding, but that is different from the other output sampling rate.
These procedures can be performed either for the Mid signal or for the Side signal or for both signals derived from the first and the second channel signal of a multi-channel signal where the first signal can also be a left signal and the second signal can be a right signal in the case of a stereo signal only having two channels (additionally two, for example, a low-frequency enhancement channel).
Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:
The multi-channel encoder of
The core encoder 1040 is configured to operate in accordance with a first frame control to provide a sequence of frames, wherein a frame is bounded by a start frame border 1901 and an end frame border 1902. The time-spectral converter 1000 or the spectral-time converter 1030 are configured to operate in accordance with a second frame control being synchronized to the first frame control, wherein the start frame border 1901 or the end frame border 1902 of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectral converter 1000 for each block of the sequence of blocks of sampling values or used by the spectral-time converter 1030 for each block of the output sequence of blocks of sampling values.
As illustrated in
In a further embodiment, the multi-channel processor 1010 is connected to a spectral domain resampler 1020, and an output of the spectral-domain resampler 1020 is input into the multi-channel processor. This is illustrated by the broken connection lines 1021, 1022. In this alternative embodiment, the multi-channel processor is configured for applying the joint multi-channel processing not to the sequences of blocks of spectral values as output by the time-spectral converter, but resampled sequences of blocks as available on connection lines 1022.
The spectral-domain resampler 1020 is configured for resampling of the result sequence generated by the multi-channel processor or to resample the sequences of blocks output by the time-spectral converter 1000 to obtain a resampled sequence of blocks of spectral values that may represent a Mid-signal as illustrated at line 1025. Advantageously, the spectral domain resampler additionally performs resampling to the Side signal generated by the multi-channel processor and, therefore, also outputs a resampled sequence corresponding to the Side signal as illustrated at 1026. However, the generation and resampling of the Side signal is optional and is not required for a low bit rate implementation. Advantageously, the spectral-domain resampler 1020 is configured for truncating blocks of spectral values for the purpose of downsampling or for zero padding the blocks of spectral values for the purpose of upsampling. The multi-channel encoder additionally comprises a spectral-time converter for converting the resampled sequence of blocks of spectral values into a time-domain representation comprising an output sequence of blocks of sampling values having associated an output sampling rate being different from the input sampling rate. In alternative embodiments, where the spectral domain resampling is performed before multi-channel processing, the multi-channel processor provides the result sequence via broken line 1023 directly to the spectral-time converter 1030. In this alternative embodiment, an optional feature is that, additionally, the Side signal is generated by the multi-channel processor already in the resampled representation and the Side signal is then also processed by the spectral-time converter.
In the end, the spectral-time converter advantageously provides a time-domain Mid signal 1031 and an optional time-domain Side signal 1032, that can both be core-encoded by the core encoder 1040. Generally, the core encoder is configured for a core encoding the output sequence of blocks of sampling values to obtain the encoded multi-channel signal.
The upper chart in
Contrary thereto, the lowest chart in
Typically, the sampling rate associated with a corresponding spectrum in
In the second chart of
Contrary thereto,
In order to keep the overall delay low, the present invention provides a method at the encoder-side for avoiding the need of a time-domain resampler and by replacing it by resampling the signals in the DFT domain. For example, in EVS it allows saving 0.9375 ms of delay coming from the time-domain resampler. The resampling in frequency domain is achieved by zero padding or truncating the spectrum and scaling it correctly.
Consider an input windowed signal x sampled at rate fx with a spectrum X of size Nx and a version y of the same signal re-sampled at rate fy with a spectrum of size Ny. The sampling factor is then equal to:
fy/fx=Ny/Nx
in case of downsampling Nx>Ny. The downsampling can be simply performed in frequency domain by directly scaling and truncating the original spectrum X:
Y[k]=X[k]·Ny/Nx for k=0 . . . Ny
in case of upsampling Nx<Ny. The up-sampling can be simply performed in frequency domain by directly scaling and zero padding the original spectrum X:
Y[k]=X[k]·Ny/Nx for k=0 . . . Nx
Y[k]=0 for k=Nx . . . Ny
Both re-sampling operations can be summarized by:
Y[k]=X[k]·Ny/Nx for all k=0 . . . min(Ny,Nx)
Y[k]=0 for all k=min(Ny,Nx) . . . Ny for if Ny>Nx
Once the new spectrum Y is obtained, the time-domain signal y can be obtained by applying the associated inverse transform iDFT of size Ny:
y=iDFT(Y)
For constructing the continuous time signal over different frames, the output frame y is then windowed and overlap-added to the previously obtained frame.
The window shape is for all sampling rates the same, but the window has different sizes in samples and is differently sampled depending of the sampling rate. The number of samples of the windows and their values can be easily derived since the shape is purely defined analytically. The different parts and sizes of the window can be found in
win_ovlp(k)=sin(pi*(k+0.5)/(2*ovlp_size)); for k=0 . . . ovlp_size−1
while the descending ovlp_size coefficients are given by:
win_ovlp(k)=sin(pi*(ovlp_size−1−k+0.5)/(2*ovlp_size)); for k=0 . . . ovlp_size−1
where ovlp_size is function of the sampling rate and given in
The new low-delay stereo coding is a joint Mid/Side (M/S) stereo coding exploiting some spatial cues, where the Mid-channel is coded by a primary mono core coder the mono core coder, and the Side-channel is coded in a secondary core coder. The encoder and decoder principles are depicted in
The stereo processing is performed mainly in Frequency Domain (FD). Optionally some stereo processing can be performed in Time Domain (TD) before the frequency analysis. It is the case for the ITD computation, which can be computed and applied before the frequency analysis for aligning the channels in time before pursuing the stereo analysis and processing. Alternatively, ITD processing can be done directly in frequency domain. Since usual speech coders like ACELP do not contain any internal time-frequency decomposition, the stereo coding adds an extra complex modulated filter-bank by means of an analysis and synthesis filter-bank before the core encoder and another stage of analysis-synthesis filter-bank after the core decoder. In the advantageous embodiment, an oversampled DFT with a low overlapping region is employed. However, in other embodiments, any complex valued time-frequency decomposition with similar temporal resolution can be used. In the following to the stereo filter-band either a filter-bank like QMF or a block transform like DFT is referred to.
The stereo processing consists of computing the spatial cues and/or stereo parameters like inter-channel Time Difference (ITD), the inter-channel Phase Differences (IPDs), inter-channel Level Differences (ILDs) and prediction gains for predicting Side signal (S) with the Mid signal (M). It is important to note that the stereo filter-bank at both encoder and decoder introduces an extra delay in the coding system.
Then, within the spectral domain, a further stereo processing 1010 is performed which incurs, at least, a downmix of left and right to the Mid signal M and, optionally, the calculation of a Side signal S and, although not explicitly illustrated in
Furthermore,
The encoded or core-encoded Mid signal, and the core-encoded Side signal are forwarded to a multiplexer 1500 that multiplexes these encoded signals together with side information. One kind of side information is the ID parameter output at 1421 to the multiplexer (and optionally to the stereo processing element 1010), and further parameters are in the channel level differences/prediction parameters, inter-channel phase differences (IPD parameters) or stereo filling parameters as illustrated at line 1422. Correspondingly, the
The stereo DFT can then provide different sampled versions of the signal which is further convey to the switched core encoder. The signal to code can be the Mid channel, the Side channel, or the left and right channels, or any signal resulting from a rotation or channel mapping of the two input channels. Since the different core encoders of switched system accept different sampling rates, it is an important feature that the stereo synthesis filter-bank can provides a multi-rated signal. The principle is given in
In
Furthermore,
Furthermore, the core decoder 1040 comprises an MDCT-based encoder branch 1430a and an ACELP encoding branch 1430b. Particularly, the mid coder for the Mid signals M and, the corresponding side coder for the Side signal s performs a switch coding between an MDCT-based encoding and an ACELP encoding where, typically, the core encoder additionally has a coding mode decider that typically operates on a certain look-ahead portion in order to determine whether a certain block or frame is to be encoded using MDCT-based procedures or ACELP-based procedures. Furthermore, or alternatively, the core encoder is configured to use the look-ahead portion in order to determine other characteristics such as LPC parameters, etc.
Furthermore, the core encoder additionally comprises preprocessing stages at different sampling rates such as a first preprocessing stage 1430c operating at 12.8 kHz and a further preprocessing stage 1430d operating at sampling rates of the group of sampling rates consisting of 16 kHz, 25.6 kHz or 32 kHz.
Therefore, generally, the embodiment illustrated in
Furthermore, the embodiment in
Furthermore, the encoder in
The core decoder 1600 is configured to operate in accordance with a first frame control to provide a sequence of frames, wherein a frame is bounded by a start frame border 1901 and an end frame border 1902. The time-spectral converter 1610 or the spectral-time converter 1640 is configured to operate in accordance with a second frame control being synchronized to the first frame control. The time-spectral converter 1610 or the spectral-time converter 1640 are configured to operate in accordance with a second frame control being synchronized to the first frame control, wherein the start frame border 1901 or the end frame border 1902 of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectral converter 1610 for each block of the sequence of blocks of sampling values or used by the spectral-time converter 1640 for each block of the at least two output sequences of blocks of sampling values.
Again, the invention with respect to the apparatus for decoding the encoded multi-channel signal 1601 can be implemented in several alternatives. One alternative is that the spectral domain resampler is not used at all. Another alternative is that a resampler is used and is configured to resample the core-decoded signal in the spectral domain before performing the multi-channel processing. This alternative is illustrated by the solid lines in
Particularly, in the first embodiment, i.e., where the spectral domain resampling is performed in the spectral domain before the multi-channel processing, the core decoded signal representing a sequence of blocks of sampling values is converted into a frequency domain representation having a sequence of blocks of spectral values for the core-decoded signal at line 1611.
Additionally, the core-decoded signal not only comprises the M signal at line 1602, but also a Side signal at line 1603, where a Side signal is illustrated at 1604 in a core-encoded representation.
Then, the time-spectral converter 1610 additionally generates a sequence of blocks of spectral values for the Side signal on line 1612.
Then, a spectral domain resampling is performed by block 1620, and the resampled sequence of blocks of spectral values with respect to the Mid signal or downmix channel or first channel is forwarded to the multi-channel processor at line 1621 and, optionally, also a resampled sequence of blocks of spectral values for the Side signal is also forwarded from the spectral domain resampler 1620 to the multi-channel processor 1630 via line 1622.
Then, the multi-channel processor 1630 performs an inverse multi-channel processing to a sequence comprising a sequence from the downmix signal and, optionally, from the Side signal illustrated at lines 1621 and 1622 in order to output at least two result sequences of blocks of spectral values illustrated at 1631 and 1632. These at least two sequences are then converted into the time-domain using the spectral-time converter in order to output time-domain channel signals 1641 and 1642. In the other alternative, illustrated at line 1615, the time-spectral converter is configured to feed the core-decoded signal such as the Mid signal to the multi-channel processor. Additionally, the time-spectral converter can also feed a decoded Side signal 1603 in its spectral-domain representation to the multi-channel processor 1630, although this option is not illustrated in
Thus, a little bit in analogy as to what has been discussed in the context of
Again,
All three core decoded signals are input into the time-spectral converter 1610 that generates three different sequences of blocks of spectral values 1613, 1611 and 1612.
The sequence of blocks of spectral values 1613 has frequency or spectral values up to the maximum output frequency and, therefore, is associated with the output sampling rate.
The sequence of blocks of spectral values 1611 has spectral values up to a different maximum frequency and, therefore, this signal does not correspond to the output sampling rate.
Furthermore, the signal 1612 spectral values up to the maximum input frequency that is also different from the maximum output frequency.
Thus, the sequences 1612 and 1611 are forwarded to the spectral domain resampler 1620 while the signal 1613 is not forwarded to the spectral domain resampler 1620, since this signal is already associated with the correct output sampling rate.
The spectral domain resampler 1620 forwards the resampled sequences of spectral values to a combiner 1700 that is configured to perform a block by block combination with spectral lines by spectral lines for signals that correspond in overlapping situations. Thus, there will typically be a cross-over region between a switch from an MDCT-based signal to an ACELP signal, and in this overlapping range, signal values exist and are combined with each other. When, however, this overlapping range is over, and a signal exists only in signal 1603 for example while signal 1602, for example, does not exist, then the combiner will not perform a block by block spectral line addition in this portion. When, however, a switch-over comes up later on, then a block by block, spectral line by spectral line addition will take place during this cross-over region.
Furthermore, a continuous addition can also be possible as is illustrated in
Similarly, the MDCT-based decoding stage 1600d and the time-domain bandwidth extension decoding stage 1600c can be coupled via a cross-fading block 1704 in order to obtain the core decoded signal 1603 that is then converted into the spectral domain representation at the output sampling rate so that, for this signal 1613, and spectral domain resampling is not necessary, but the signal can be forwarded directly to the combiner 1700. The stereo inverse processing or multi-channel processing 1603 then takes place subsequent to the combiner 1700.
Thus, in contrast to the embodiment illustrated in
As is illustrated in
Thus,
The choice of the stereo filter-bank is crucial for a low-delay system and the achievable trade-off is summarized in
The analysis and synthesis window of the filter-bank is another important aspect. In the advantageous embodiment the same window is used for the analysis and synthesis of the DFT.
It is also the same at encoder and decoder sides. It was paid special attention for fulfilling the following constraints:
Knowing these constraints the windows for the proposal 1 and 4 are described in
Furthermore, the first overlapping portion 1801 and the second overlapping portion 1802 additionally have zero padding portion of 1804 at the beginning and 1805 at the end thereof.
Furthermore,
Furthermore, the first overlapping portion 1811 of the second window starts at the end of the middle part 1803, i.e., the non-overlapping part of the first window, and the overlapping part of the second window, i.e., the non-overlapping part 1813 starts at the end of the second overlapping portion 1802 of the first window as illustrated.
When
In advantageous embodiments, the same analysis and synthesis windows are used only for the decoder illustrated in
However, in certain embodiments particularly with respect to the subsequent proposal/embodiment 1, an analysis window being generally in line with
Furthermore, it is to be noted that due to the overlap-add operation, the multiplication of sine to the power 0.5 multiplied by sine to the power of 1.5 once again results in a sine to the power of 2 result that may be used in order to have an energy conservation situation.
The proposal 1 has as main characteristics that the overlapping region of the DFT has the same size and is aligned with the ACELP look-ahead and the MDCT core overlapping region. The encoder delay is then the same as for the ACELP/MDCT cores and the stereo doesn't introduce any additional delay et the encoder. In case of EVS and in case the multi-rate synthesis filter-bank approach as described in
The encoder schematic framing is illustrated in
One major issue for proposal 1 is that the look-ahead at the encoder is windowed. It can be redressed for the subsequent processing, or it can be left windowed if the subsequent processing is adapted for taking into account a windowed look-ahead. It might be that if the stereo processing performed in the DFT modified the input channel, and especially when using non-linear operations, that the redressed or windowed signal doesn't allow to achieve a perfect reconstruction in case the core coding is bypassed.
It is worth noting that between the core decoder synthesis and the stereo decoder analysis windows there is a time gap of 1.25 ms which can be exploited by the core decoder post-processing, by the bandwidth extension (BWE), like Time Domain BWE used over ACELP, or by the some smoothing in case of transition between ACELP and MDCT cores.
Since this time gap of only 1.25 ms is lower than the 2.3125 ms that may be used by the standard EVS for such operations, the present invention provides a way to combine, resample and smooth the different synthesis parts of the switched decoder within the DFT domain of the stereo module.
As illustrated in
Thus, it becomes clear that second overlapping portion such as 1812 of
Thus, the core encoder 1040 is configured to use a look-ahead portion such as the look-ahead portion 1905 when core encoding the output block of the output sequence of blocks of sampling values, wherein the output look-ahead portion is located in time subsequent to the output block. The output block is corresponding to the frame bounded by the frame borders 1901, 1904 and the output look-ahead portion 1905 comes after this output block for the core encoder 1040.
Furthermore, as illustrated, the time-spectral converter is configured to use an analysis window, i.e., window 1904 having the overlap portion with a length in time being lower than or equal to the length in time of the look-ahead portion 1905, wherein this overlapping portion corresponding to overlapping 1812 of
Furthermore, the spectral-time converter 1030 is configured to process the output look-ahead portion corresponding to the windowed look-ahead portion advantageously using a redress function, wherein the redress function is configured so that an influence of the overlap portion of the analysis window is reduced or eliminated.
Thus, the spectral-time converter operating in between the core encoder 1040 and the downmix 1010/downsampling 1020 block in
Thus, it is made sure that the core encoder 1040, when applying its look-ahead functionality to the look-ahead portion 1095, performs the look-ahead function not portion but to a portion that is close to the original portion as far as possible.
However, due to low-delay constraints, and due to the synchronization between the framing of the stereo preprocessor and the core encoder, an original time domain signal for the look-ahead portion does not exist. However, the application of the redressing function makes sure that any artifacts incurred by this procedure are reduced as much as possible.
A sequence of procedures with respect to this technology is illustrated in
In step 1910, a DFT−1 of a zeroth block is performed to obtain a zeroth block in the time domain. The zeroth block would have been obtained a window used to the left of window 1903 in
Then, in step 1912, the zeroth block is windowed using a synthesis window, i.e., is windowed in the spectral-time converter 1030 illustrated in
Then, as illustrated in block 1911, a DFT−1 of the first block obtained by window 1903 is performed to obtain a first block in the time domain, and this first block is once again windowed using the synthesis window in block 1910.
Then, as indicated at 1918 in
Thus, if the window used for generating the second block was a sine window illustrated in
However, it is advantageous to use a square root of sine window for the analysis window and, therefore, the redressing function is a window function of 1/√{square root over (sin( ))}. This ensures that the redressed look-ahead portion obtained by block 1922 is as close as possible to the original signal within the look-ahead portion, but, of course, not the original left signal or the original right signal but the original signal that would have been obtained by adding left and right to obtain the Mid signal.
Then, in step 1924 in
Advantageously, the overlapping portion of the window 1904 extending into the look-ahead portion 1905 has the same length as the look-ahead portion, but it can also be shorter than the look-ahead portion but it is advantageous that it is not longer than the look-ahead portion so that the stereo preprocessor does not introduce any additional delay due to overlapping windows.
Then, the procedure goes on with the windowing of the second portion of the second block using the synthesis window as illustrated in block 1930. Thus, the second portion of the second block is, on the one hand, redressed by block 1922 and is, on the other hand, windowed by the synthesis window as illustrated in block 1930, since this portion may then be used for generating the next frame for the core encoder by overlap-add the windowed second portion of the second block, a windowed third block and a windowed first portion of the fourth block as illustrated in block 1932. Naturally, the fourth block and, particularly the second portion of the fourth block would once again be subjected to the redressing operation as discussed with respect to the second block in item 1922 of
Furthermore, as discussed before, it is to be noted that there is a time gap between the end of a window, i.e., the analysis window 1914 and the end frame border 1902 of the frame defined by the start frame border 1901 and the end frame border 1902 of
Particularly, the time gap is illustrated at 1920 with respect to the analysis windows applied by the time-spectrum converter 1610 of
Thus, the core decoder has additional time in order to core decode the samples in the time gap and/or to post-process the samples in the time gap as illustrated at block 1940. Thus, the time-spectrum converter 1610 already outputs a first block as the result of step 1938 there the core decoder can provide the remaining samples in the time gap or can post-process the samples in the time gap at step 1940.
Then, in step 1942, the time-spectrum converter 1610 is configured to window the samples in the time gap together with samples of the next frame using a next analysis window that would occur subsequent to window 1914 in
Thus, this time gap of, for example, 1.25 ms when the
Thus, once again, the core decoder 1600 is configured to operate in accordance with a first framing control to provide a sequence of frames, wherein the time-spectrum converter 1610 or the spectrum-time converter 1640 are configured to operate in accordance with a second framing control being synchronized with the first framing control, so that the start frame border or the end frame border of each frame of the sequence of frames is in a predetermined relation to a start instant or an end instant of an overlapping portion of a window used by the time-spectrum converter or the spectrum-time converter for each block of the sequence of blocks of sampling values or for each block of the resampled sequence of blocks of spectral values.
Furthermore, the time-spectrum converter 1610 is configured to use an analysis window for windowing the frame of the sequence of frames having an overlapping range ending before the end frame border 1902 leaving a time gap 1920 between the end of the overlap portion and the end frame border. The core decoder 1600 is, therefore, configured to perform the processing to the samples in the time gap 1920 in parallel to the windowing of the frame using the analysis window or wherein a further post-processing the time gap is performed in parallel to the windowing of the frame using the analysis window by the time-spectral converter.
Furthermore, and advantageously, the analysis window for a following block of the core decoded signal is located so that a middle non-overlapping portion of the window is located within the time gap as illustrated at 1920 of
In proposal 4 the overall system delay is enlarged compared to proposal 1. At the encoder an extra delay is coming from the stereo module. The issue of perfect reconstruction is no more pertinent in proposal 4 unlike proposal 1.
At decoder, the available delay between core decoder and first DFT analysis is of 2.5 ms which allows performing conventional resampling, combination and smoothing between the different core syntheses and the extended bandwidth signals as it is done for in the standard EVS.
The encoder schematic framing is illustrated in
In proposal 5, the time resolution of the DFT is decreased to 5 ms. The lookahead and overlapping region of core coder is not windowed, which is a shared advantage with proposal 4. On the other hand, the available delay between the coder decoding and the stereo analysis is small and a solution as proposed in Proposal 1 is needed (
The encoder schematic framing is illustrated in
In view of the above, advantageous embodiments relate, with respect to the encoder-side, to a multi-rate time-frequency synthesis which provides at least one stereo processed signal at different sampling rates to the subsequent processing modules. The module includes, for example, a speech encoder like ACELP, pre-processing tools, an MDCT-based audio encoder such as TCX or a bandwidth extension encoder such as a time-domain bandwidth extension encoder.
With respect to the decoder, the combination in resampling in the stereo frequency-domain with respect to different contributions of the decoder synthesis are performed. These synthesis signals can come from a speech decoder like an ACELP decoder, an MDCT-based decoder, a bandwidth extension module or an inter-harmonic error signal from a post-processing like a bass-post-filter.
Furthermore, regarding both the encoder and the decoder, it is useful to apply a window for the DFT or a complex value transformed with a zero padding, a low overlapping region and a hopsize which corresponds to an integer number of samples at different sampling rates such as 12.9 kHz, 16 kHz, 25.6 kHz, 32 kHz or 48 kHz.
Embodiments are able to achieve low bit-are coding of stereo audio at low delay. It was specifically designed to combine efficiently a low-delay switched audio coding scheme, like EVS, with the filter-banks of a stereo coding module.
Embodiments may find use in the distribution or broadcasting all types of stereo or multi-channel audio content (speech and music alike with constant perceptual quality at a given low bitrate) such as, for example with digital radio, Internet streaming and audio communication applications.
Advantageously, the signal aligner is configured to align the channels from the multi-channel signal using the broadband alignment parameter, before the parameter determiner 100 actually calculates the narrowband parameters. Therefore, in this embodiment, the signal aligner 200 sends the broadband aligned channels back to the parameter determiner 100 via a connection line 15. Then, the parameter determiner 100 determines the plurality of narrowband alignment parameters from an already with respect to the broadband characteristic aligned multi-channel signal. In other embodiments, however, the parameters are determined without this specific sequence of procedures.
Specifically, the multi-channel encoder further comprises a time-spectrum converter 150 for converting a time domain multi-channel signal into a spectral representation of the at least two channels within the frequency domain.
Furthermore, as illustrated at 152, the parameter determiner, the signal aligner and the signal processor illustrated at 100, 200 and 300 in
Furthermore, the multi-channel encoder and, specifically, the signal processor further comprises a spectrum-time converter 154 for generating a time domain representation of the mid-signal at least.
Advantageously, the spectrum time converter additionally converts a spectral representation of the side signal also determined by the procedures represented by block 152 into a time domain representation, and the signal encoder 400 of
Advantageously, the time-spectrum converter 150 of
In step 156, each channel is windowed using the analysis window with overlap ranges. Specifically, each channel is widowed using the analysis window in such a way that a first block of the channel is obtained. Subsequently, a second block of the same channel is obtained that has a certain overlap range with the first block and so on, such that subsequent to, for example, five windowing operations, five blocks of windowed samples of each channel are available that are then individually transformed into a spectral representation as illustrated at 157 in
In step 158, which is performed by the parameter determiner 100 of
Specifically, the operations of the steps 304 and 305 result in a kind of cross fading from one block of the mid-signal or the side signal in the next block of the mid signal and the side signal is performed so that, even when any parameter changes occur such as the inter-channel time difference parameter or the inter-channel phase difference parameter occur, this will nevertheless be not audible in the time domain mid/side signals obtained by step 305 in
In particular, the signal is received by an input interface 600. Connected to the input interface 600 are a signal decoder 700, and a signal de-aligner 900. Furthermore, a signal processor 800 is connected to a signal decoder 700 on the one hand and is connected to the signal de-aligner on the other hand.
In particular, the encoded multi-channel signal comprises an encoded mid-signal, an encoded side signal, information on the broadband alignment parameter and information on the plurality of narrowband parameters. Thus, the encoded multi-channel signal on line 50 can be exactly the same signal as output by the output interface of 500 of
However, importantly, it is to be noted here that, in contrast to what is illustrated in
Thus, the information on the alignment parameters can be the alignment parameters as used by the signal aligner 200 in
The input interface 600 of
The signal decoder is configured for decoding the encoded mid-signal and for decoding the encoded side signal to obtain a decoded mid-signal on line 701 and a decoded side signal on line 702. These signals are used by the signal processor 800 for calculating a decoded first channel signal or decoded left signal and for calculating a decoded second channel or a decoded right channel signal from the decoded mid signal and the decoded side signal, and the decoded first channel and the decoded second channel are output on lines 801, 802, respectively. The signal de-aligner 900 is configured for de-aligning the decoded first channel on line 801 and the decoded right channel 802 using the information on the broadband alignment parameter and additionally using the information on the plurality of narrowband alignment parameters to obtain a decoded multi-channel signal, i.e., a decoded signal having at least two decoded and de-aligned channels on lines 901 and 902.
In step 914, any further processing is performed that comprises using a windowing or any overlap-add operation or, generally, any cross-fade operation in order to obtain, at 915a or 915b, an artifact-reduced or artifact-free decoded signal, i.e., to decoded channels that do not have any artifacts although there have been, typically, time-varying de-alignment parameters for the broadband on the one hand and for the plurality of narrow bands on the other hand.
In particular, the signal processor 800 from
However, importantly, in order to calculate L and R by the mid/side-left/right conversion in block 820, the side signal S is not necessarily to be used. Instead, as discussed later on, the left/right signals are initially calculated only using a gain parameter derived from an inter-channel level difference parameter ILD. Therefore, in this implementation, the side signal S is only used in the channel updater 830 that operates in order to provide a better left/right signal using the transmitted side signal S as illustrated by bypass line 821.
Therefore, the converter 820 operates using a level parameter obtained via a level parameter input 822 and without actually using the side signal S but the channel updater 830 then operates using the side 821 and, depending on the specific implementation, using a stereo filling parameter received via line 831. The signal aligner 900 then comprises a phased-de-aligner and energy scaler 910. The energy scaling is controlled by a scaling factor derived by a scaling factor calculator 940. The scaling factor calculator 940 is fed by the output of the channel updater 830. Based on the narrowband alignment parameters received via input 911, the phase de-alignment is performed and, in block 920, based on the broadband alignment parameter received via line 921, the time-de-alignment is performed. Finally, a spectrum-time conversion 930 is performed in order to finally obtain the decoded signal.
Specifically, the narrowband de-aligned channels are input into the broadband de-alignment functionality corresponding to block 920 of
When
Furthermore, the DFT operations in blocks 810 correspond to element 810 in
Subsequently,
Additionally, the spectrum is also divided into different parameter bands. Each parameter band has at least one and advantageously more than one spectral lines. Additionally, the parameter bands increase from lower to higher frequencies. Typically, the broadband alignment parameter is a single broadband alignment parameter for the whole spectrum, i.e., for a spectrum comprising all the bands 1 to 6 in the exemplary embodiment in
Furthermore, the plurality of narrowband alignment parameters are provided so that there is a single alignment parameter for each parameter band. This means that the alignment parameter for a band applies to all the spectral values within the corresponding band.
Furthermore, in addition to the narrowband alignment parameters, level parameters are also provided for each parameter band.
In contrast to the level parameters that are provided for each and every parameter band from band 1 to band 6, it is advantageous to provide the plurality of narrowband alignment parameters only for a limited number of lower bands such as bands 1, 2, 3 and 4.
Additionally, stereo filling parameters are provided for a certain number of bands excluding the lower bands such as, in the exemplary embodiment, for bands 4, 5 and 6, while there are side signal spectral values for the lower parameter bands 1, 2 and 3 and, consequently, no stereo filling parameters exist for these lower bands where wave form matching is obtained using either the side signal itself or a prediction residual signal representing the side signal.
As already stated, there exist more spectral lines in higher bands such as, in the embodiment in
Nevertheless,
As illustrated, the level parameter ILD is provided for each of 12 bands and is quantized to a quantization accuracy represented by five bits per band.
Furthermore, the narrowband alignment parameters IPD are only provided for the lower bands up to a border frequency of 2.5 kHz. Additionally, the inter-channel time difference or broadband alignment parameter is only provided as a single parameter for the whole spectrum but with a very high quantization accuracy represented by eight bits for the whole band.
Furthermore, quite roughly quantized stereo filling parameters are provided represented by three bits per band and not for the lower bands below 1 kHz since, for the lower bands, actually encoded side signal or side signal residual spectral values are included.
Subsequently, an advantageous processing on the encoder side is summarized In a first step, a DFT analysis of the left and the right channel is performed. This procedure corresponds to steps 155 to 157 of
ILD parameters, i.e., level parameters and phase parameters (IPD parameters), are calculated for each parameter band on the shifted L and R representations. This step corresponds to step 160 of
In the final step, the time domain mid-signal m and, optionally, the residual signal are coded. This procedure corresponds to what is performed by the signal encoder 400 in
At the decoder in the inverse stereo processing, the Side signal is generated in the DFT domain and is first predicted from the Mid signal as:=g·Mid
where g is a gain computed for each parameter band and is function of the transmitted Inter-channel Level Difference (ILDs).
The residual of the prediction Side−g·Mid can be then refined in two different ways:
The two types of coding refinement can be mixed within the same DFT spectrum. In the advantageous embodiment, the residual coding is applied on the lower parameter bands, while residual prediction is applied on the remaining bands. The residual coding is in the advantageous embodiment as depict in
Subsequently a further embodiment of a joint stereo/multichannel encoder processing or an inverse stereo/multichannel processing is described.
1. Time-Frequency Analysis: DFT
It is important that the extra time-frequency decomposition from the stereo processing done by DFTs allows a good auditory scene analysis while not increasing significantly the overall delay of the coding system. By default, a time resolution of 10 ms (twice the 20 ms framing of the core coder) is used. The analysis and synthesis windows are the same and are symmetric. The window is represented at 16 kHz of sampling rate in
2. Stereo Parameters
Stereo parameters can be transmitted at maximum at the time resolution of the stereo DFT. At minimum it can be reduced to the framing resolution of the core coder, i.e. 20 ms. By default, when no transients is detected, parameters are computed every 2 ms over 2 DFT windows. The parameter bands constitute a non-uniform and non-overlapping decomposition of the spectrum following roughly 2 times or 4 times the Equivalent Rectangular Bandwidths (ERB). By default, a 4 times ERB scale is used for a total of 12 bands for a frequency bandwidth of 16 kHz (32 kbps sampling-rate, Super Wideband stereo).
3. Computation of ITD and Channel Time Alignment
The ITD are computed by estimating the Time Delay of Arrival (TDOA) using the Generalized Cross Correlation with Phase Transform (GCC-PHAT):
where L and R are the frequency spectra of the of the left and right channels respectively. The frequency analysis can be performed independently of the DFT used for the subsequent stereo processing or can be shared. The pseudo-code for computing the ITD is the following:
The ITD computation can also be summarized as follows. The cross-correlation is computed in frequency domain before being smoothed depending of the Spectral Flatness Measurement. SFM is bounded between 0 and 1. In case of noise-like signals, the SFM will be high (i.e. around 1) and the smoothing will be weak. In case of tone-like signal, SFM will be low and the smoothing will become stronger. The smoothed cross-correlation is then normalized by its amplitude before being transformed back to time domain. The normalization corresponds to the Phase-transform of the cross-correlation, and is known to show better performance than the normal cross-correlation in low noise and relatively high reverberation environments. The so-obtained time domain function is first filtered for achieving a more robust peak peaking. The index corresponding to the maximum amplitude corresponds to an estimate of the time difference between the Left and Right Channel (ITD). If the amplitude of the maximum is lower than a given threshold, then the estimated of ITD is not considered as reliable and is set to zero.
If the time alignment is applied in Time Domain, the ITD is computed in a separate DFT analysis. The shift is done as follows:
It may use an extra delay at encoder, which is equal at maximum to the maximum absolute ITD which can be handled. The variation of ITD over time is smoothed by the analysis windowing of DFT.
Alternatively the time alignment can be performed in frequency domain. In this case, the ITD computation and the circular shift are in the same DFT domain, domain shared with this other stereo processing. The circular shift is given by:
Zero padding of the DFT windows is needed for simulating a time shift with a circular shift. The size of the zero padding corresponds to the maximum absolute ITD which can be handled. In the advantageous embodiment, the zero padding is split uniformly on the both sides of the analysis windows, by adding 3.125 ms of zeros on both ends. The maximum absolute possible ITD is then 6.25 ms. In A-B microphones setup, it corresponds for the worst case to a maximum distance of about 2.15 meters between the two microphones. The variation in ITD over time is smoothed by synthesis windowing and overlap-add of the DFT.
It is important that the time shift is followed by a windowing of the shifted signal. It is a main distinction with the Binaural Cue Coding (BCC) of conventional technology, where the time shift is applied on a windowed signal but is not windowed further at the synthesis stage. As a consequence, any change in ITD over time produces an artificial transient/click in the decoded signal.
4. Computation of IPDs and Channel Rotation
The IPDs are computed after time aligning the two channels and this for each parameter band or at least up to a given ipd_max_band, dependent of the stereo configuration.
IPD[b]=angle(Σk=band
IPDs is then applied to the two channels for aligning their phases:
Where β=atan2(sin(IPDi[b]), cos(IPDi[b])+c), c=10ILD
5. Sum-Difference and Side Signal Coding
The sum difference transformation is performed on the time and phase aligned spectra of the two channels in a way that the energy is conserved in the Mid signal.
is bounded between 1/1.2 and 1.2, i.e. −1.58 and +1.58 dB. The limitation avoids aretefact when adjusting the energy of M and S. It is worth noting that this energy conservation is less important when time and phase were beforehand aligned. Alternatively the bounds can be increased or decreased.
The side signal S is further predicted with M:
where c=10ILD
The residual signal S′(f) can be modeled by two means: either by predicting it with the delayed spectrum of M or by coding it directly in the MDCT domain in the MDCT domain.
6. Stereo Decoding
The Mid signal X and Side signal S are first converted to the left and right channels L and R as follows:
Li[k]=Mi[k]+gMi[k], for band_limits[b]≤k<band_limits[b+1],
Ri[k]=Mi[k]−gMi[k], for band_limits[b]≤k<band_limits[b+1],
where the gain g per parameter band is derived from the ILD parameter:
For parameter bands below cod_max_band, the two channels are updated with the decoded Side signal:
Li[k]=Li[k]+cod_gaini·Si[k], for 0≤k<band_limits[cod_max_band],
Ri[k]=Ri[k]−cod_gaini·Si[k], for 0≤k<band_limits[cod_max_band],
For higher parameter bands, the side signal is predicted and the channels updated as:
Li[k]=Li[k]+cod_predi[b]·Mi−1[k], for band_limits[b]≤k<band_limits[b+1],
Ri[k]=Ri[k]−cod_predi[b]·Mi−1[k], for band_limits[b]≤k<band_limits[b+1],
Finally, the channels are multiplied by a complex value aiming to restore the original energy and the inter-channel phase of the stereo signal:
where a is defined and bounded as defined previously, and where β=atan2(sin(IPDi[b]), cos(IPDi[b])+c), and where atan2(x,y) is the four-quadrant inverse tangent of x over y.
Finally, the channels are time shifted either in time or in frequency domain depending of the transmitted ITDs. The time domain channels are synthesized by inverse DFTs and overlap-adding.
An inventively encoded audio signal can be stored on a digital storage medium or a non-transitory storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.
Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.
Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed.
Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.
Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium.
In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.
A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.
A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are advantageously performed by any hardware apparatus.
While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations and equivalents as fall within the true spirit and scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 16152450 | Jan 2016 | EP | regional |
| 16152453 | Jan 2016 | EP | regional |
This application is a continuation of U.S. patent application Ser. No. 16/035,471 filed Jul. 13, 2018, which is a continuation of International Application No. PCT/EP2017/051212, filed Jan. 20, 2017, which is incorporated herein by reference in its entirety, and additionally claims priority from European Applications Nos. EP 16152450.9, filed Jan. 22, 2016, and EP 16152453.3, filed Jan. 22, 2016, both of which are incorporated herein by reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5434948 | Holt et al. | Jul 1995 | A |
| 6073100 | Goodridge, Jr. | Jun 2000 | A |
| 6138089 | Guberman | Oct 2000 | A |
| 6549884 | Laroche et al. | Apr 2003 | B1 |
| 7089180 | Heikkinen | Aug 2006 | B2 |
| 7240001 | Chen et al. | Jul 2007 | B2 |
| 7630882 | Mehrotra et al. | Dec 2009 | B2 |
| 7831434 | Mehrotra et al. | Nov 2010 | B2 |
| 7917369 | Chen et al. | Mar 2011 | B2 |
| 7953604 | Mehrotra et al. | May 2011 | B2 |
| 8099292 | Thumpudi et al. | Jan 2012 | B2 |
| 8255228 | Hilpert et al. | Aug 2012 | B2 |
| 8255229 | Koishida et al. | Aug 2012 | B2 |
| 8255230 | Thumpudi et al. | Aug 2012 | B2 |
| 8315880 | Kovesi et al. | Nov 2012 | B2 |
| 8630861 | Chen et al. | Jan 2014 | B2 |
| 8700388 | Edler et al. | Apr 2014 | B2 |
| 8762159 | Geiger et al. | Jun 2014 | B2 |
| 8793125 | Breebaart et al. | Jul 2014 | B2 |
| 8811621 | Schuijers | Aug 2014 | B2 |
| 20050157883 | Herre et al. | Jul 2005 | A1 |
| 20060190247 | Lindblom | Aug 2006 | A1 |
| 20090222272 | Seefeldt | Sep 2009 | A1 |
| 20090313028 | Tammi et al. | Dec 2009 | A1 |
| 20110096932 | Schuijers | Apr 2011 | A1 |
| 20110106542 | Bayer et al. | May 2011 | A1 |
| 20110202355 | Grill et al. | Aug 2011 | A1 |
| 20120002818 | Heiko et al. | Jan 2012 | A1 |
| 20120033817 | Francois et al. | Feb 2012 | A1 |
| 20120045067 | Oshikiri | Feb 2012 | A1 |
| 20130121411 | Robillard et al. | May 2013 | A1 |
| 20130151262 | Lohwasser et al. | Jun 2013 | A1 |
| 20130226570 | Multrus et al. | Aug 2013 | A1 |
| 20130262130 | Ragot | Oct 2013 | A1 |
| 20130301835 | Briand et al. | Nov 2013 | A1 |
| 20130332148 | Ravelli et al. | Dec 2013 | A1 |
| 20140032226 | Raju et al. | Jan 2014 | A1 |
| 20140140516 | Taleb et al. | May 2014 | A1 |
| 20150010155 | Virette et al. | Jan 2015 | A1 |
| 20150049872 | Virette et al. | Feb 2015 | A1 |
| 20160247515 | Koishida et al. | Aug 2016 | A1 |
| 20170133023 | Disch et al. | May 2017 | A1 |
| 20180122385 | Chebiyyam | May 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 1953736 | Aug 2008 | EP |
| 2229677 | Sep 2015 | EP |
| 2947656 | Nov 2015 | EP |
| 2453117 | Apr 2009 | GB |
| 2008530616 | Aug 2008 | JP |
| 2010020333 | Jan 2010 | JP |
| 2011522472 | Jul 2011 | JP |
| 2012521012 | Sep 2012 | JP |
| 2013528824 | Jul 2013 | JP |
| 2013538367 | Oct 2013 | JP |
| 2013543600 | Dec 2013 | JP |
| 2015518176 | Jun 2015 | JP |
| 2391714 | Jun 2010 | RU |
| 2420816 | Jun 2011 | RU |
| 2491657 | Aug 2013 | RU |
| 2542668 | Feb 2015 | RU |
| 2562384 | Sep 2015 | RU |
| 201334580 | Aug 2013 | TW |
| 2006089570 | Aug 2006 | WO |
| 2007052612 | May 2007 | WO |
| 2010084756 | Jul 2010 | WO |
| 2012020090 | Feb 2012 | WO |
| 2012105886 | Aug 2012 | WO |
| 2012110473 | Aug 2012 | WO |
| 2014043476 | Mar 2014 | WO |
| 2014044812 | Mar 2014 | WO |
| 2014161992 | Oct 2014 | WO |
| 2016108655 | Jul 2016 | WO |
| 2016142337 | Sep 2016 | WO |
| Entry |
|---|
| Fuchs, Guillaume et al., “Low Delay LPC and MDCT-Based Audio Coding in the EVS Codec”, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, Apr. 19, 2015; pp. 5723-5727; XP033064796, Apr. 19, 2015, 5723-5727. |
| Helmrich, Christian R. et al., “Low-Delay Transform Coding Using the MPEG-H 3D Audio Codec”, AES Convention 139; Oct. 23, 2015; XP040672209, Oct. 23, 2015. |
| Herre, J et al., “Spatial Audio Coding: Next-Generation Efficient and Compatible Coding”, Convention Paper Presented at the 117th Convention. Audio Engineering Society Convention Paper, New York, NY, U.S.A. No. 6186., Oct. 28, 2004, 1-13. |
| Herre, J et al., “The Reference Model Architecture for MPEG Spatial Audio Coding”, Convention Paper Presented at the 118th Convention, Audio Engineering Society, New York, NY, US. No. 6447, May 28, 2005, 1-13. |
| Herre, J et al., “The Reference Model Architecture for MPEG Spatial Audio Coding”, Proc. 118th Convention of the Audio Engineering Society, ES, AES,, May 28, 2005, p. 1-13. |
| Herre, Jurgen , “From joint stereo to spatial audio coding—recent progress and standardization”, Proceedings of the International Conference on Digital Audioeffects; Oct. 5, 2004; pp. 157-162; XP002367849, Oct. 5, 2004, 157-162. |
| Jansson, Tomas , “UPTEC F11 034 Stereo Coding for ITU-T G.719 codec”, May 17, 2011; XP55114839; http://www.diva-portal.org/smash/get/diva2:417362/FULLTEXT01.pdf, May 17, 2011. |
| Martin, Rainer et al., “Low Delay Analysis/Synthesis Schemes for Joint Speech Enhancement and Low Bit Rate Speech Coding”, 6th European Conference on Speech Communication and Technology, Eurospeech '99. Budapest, Hungary, Sep. 5-9, 1999; pp. 1463-1466; XP001075956, Sep. 5, 1999, 1463-1466. |
| Valero, Maria L. et al., “A New Parametric Stereo and Multichannel Extension for MPEG-4 Enhanced Low Delay AAC (AAC-ELD)”, AES Convention 128; May 1, 2010; XP040509482, May 1, 2010. |
| Wada, Ted S. et al., “Decorrelation by resampling in frequency domain for multichannel acoustic echo cancellation based on residual echo enhancement”, Applications of Signal Processing to Audio and Acoustics (WASPAA); Oct. 16, 2011; pp. 289-292; XP032011497, Oct. 16, 2011, 289-292. |
| Vivette, David et al., “G.722 annex D and G.711.1 Annex F—New ITU-T stereo codecs”, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp. 528-532, May 26, 2013. |
| “Information technology—MPEG audio technologies—Part 3: Unified speech and audio coding”, ISO/IEC FDIS 23003-3:2011(E), ISO/IEC JTC 1/SC 29/WG 11, Sep. 20, 2011. |
| Bosi, Marina, et al., “ISO/IEC MPEG-2 advanced audio coding”, Journal of the Audio engineering society, 1997, vol. 45. No. 10, pp. 789-814., pp. 789-814. Uploaded in 2 parts. |
| Number | Date | Country | |
|---|---|---|---|
| 20190228786 A1 | Jul 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16035471 | Jul 2018 | US |
| Child | 16375437 | US | |
| Parent | PCT/EP2017/051212 | Jan 2017 | US |
| Child | 16035471 | US |
