The present invention relates to audio signal encoding, audio signal decoding and audio signal processing, and, in particular, to an encoder, a decoder and methods for backward compatible multi-resolution spatial audio object coding (SAOC).
In modern digital audio systems, it is a major trend to allow for audio-object related modifications of the transmitted content on the receiver side. These modifications include gain modifications of selected parts of the audio signal and/or spatial re-positioning of dedicated audio objects in case of multi-channel playback via spatially distributed speakers. This may be achieved by individually delivering different parts of the audio content to the different speakers.
In other words, in the art of audio processing, audio transmission, and audio storage, there is an increasing desire to allow for user interaction on object-oriented audio content playback and also a demand to utilize the extended possibilities of multi-channel playback to individually render audio contents or parts thereof in order to improve the hearing impression. By this, the usage of multi-channel audio content brings along significant improvements for the user. For example, a three-dimensional hearing impression can be obtained, which brings along an improved user satisfaction in entertainment applications. However, multi-channel audio content is also useful in professional environments, for example in telephone conferencing applications, because the talker intelligibility can be improved by using a multi-channel audio playback. Another possible application is to offer to a listener of a musical piece to individually adjust playback level and/or spatial position of different parts (also termed as “audio objects”) or tracks, such as a vocal part or different instruments. The user may perform such an adjustment for reasons of personal taste, for easier transcribing one or more part(s) from the musical piece, educational purposes, karaoke, rehearsal, etc.
The straightforward discrete transmission of all digital multi-channel or multi-object audio content, e.g., in the form of pulse code modulation (PCM) data or even compressed audio formats, demands very high bitrates. However, it is also desirable to transmit and store audio data in a bit rate efficient way. Therefore, one is willing to accept a reasonable tradeoff between audio quality and bit rate requirements in order to avoid an excessive resource load caused by multi-channel/multi-object applications.
Recently, in the field of audio coding, parametric techniques for the bit rate-efficient transmission/storage of multi-channel/multi-object audio signals have been introduced by, e.g., the Moving Picture Experts Group (MPEG) and others. One example is MPEG Surround (MPS) as a channel oriented approach [MPS, BCC], or MPEG Spatial Audio Object Coding (SAOC) as an object oriented approach [JSC, SAOC, SAOC1, SAOC2]. Another object—oriented approach is termed as “informed source separation” [ISS1, ISS2, ISS3, ISS4, ISS5, ISS6]. These techniques aim at reconstructing a desired output audio scene or a desired audio source object on the basis of a downmix of channels/objects and additional side information describing the transmitted/stored audio scene and/or the audio source objects in the audio scene.
The estimation and the application of channel/object related side information in such systems is done in a time-frequency selective manner. Therefore, such systems employ time-frequency transforms such as the Discrete Fourier Transform (DFT), the Short Time Fourier Transform (STFT) or filter banks like Quadrature Mirror Filter (QMF) banks, etc. The basic principle of such systems is depicted in
In case of the STFT, the temporal dimension is represented by the time-block number and the spectral dimension is captured by the spectral coefficient (“bin”) number. In case of QMF, the temporal dimension is represented by the time-slot number and the spectral dimension is captured by the sub-band number. If the spectral resolution of the QMF is improved by subsequent application of a second filter stage, the entire filter bank is termed hybrid QMF and the fine resolution sub-bands are termed hybrid sub-bands.
As already mentioned above, in SAOC the general processing is carried out in a time-frequency selective way and can be described as follows within each frequency band:
Time-frequency based systems may utilize a time-frequency (t/f) transform with static temporal and frequency resolution. Choosing a certain fixed t/f-resolution grid typically involves a trade-off between time and frequency resolution.
The effect of a fixed t/f-resolution can be demonstrated on the example of typical object signals in an audio signal mixture. For example, the spectra of tonal sounds exhibit a harmonically related structure with a fundamental frequency and several overtones. The energy of such signals is concentrated at certain frequency regions. For such signals, a high frequency resolution of the utilized t/f-representation is beneficial for separating the narrowband tonal spectral regions from a signal mixture. In the contrary, transient signals, like drum sounds, often have a distinct temporal structure: substantial energy is only present for short periods of time and is spread over a wide range of frequencies. For these signals, a high temporal resolution of the utilized t/f-representation is advantageous for separating the transient signal portion from the signal mixture.
The frequency resolution obtained from the standard SAOC representation is limited to the number of parametric bands, having the maximum value of 28 in standard SAOC. They are obtained from a hybrid QMF bank consisting of a 64-band QMF-analysis with an additional hybrid filtering stage on the lowest bands further dividing these into up to 4 complex sub-bands. The frequency bands obtained are grouped into parametric bands mimicking the critical band resolution of the human auditory system. The grouping allows for reducing the necessitated side information data rate to a size that can be efficiently handled in practical applications.
Current audio object coding schemes offer only a limited variability in the time-frequency selectivity of the SAOC processing. For instance, MPEG SAOC [SAOC] [SAOC1] [SAOC2] is limited to the time-frequency resolution that can be obtained by the use of the so-called Hybrid Quadrature Mirror Filter Bank (Hybrid-QMF) and its subsequent grouping into parametric bands. Therefore, object restoration in standard SAOC often suffers from the coarse frequency resolution of the Hybrid-QMF leading to audible modulated crosstalk from the other audio objects (e.g., double-talk artifacts in speech or auditory roughness artifacts in music).
The existing system produces a reasonable separation quality given the reasonably low data rate. The main problem is the insufficient frequency resolution for a clean separation of tonal sounds. This is exhibited as a “halo” of other objects surrounding the tonal components of an object. Perceptually this is observed as roughness or a vocoder-like artefact. The detrimental effect of this halo can be reduced by increasing the parametric frequency resolution. It was noted, that a resolution equal or higher than 512 bands (at 44.1 kHz sampling rate) is enough to produce perceptually significantly improved separation in the test signals. The problem with such a high parametric resolution is that the amount the side information needed increases considerably, into impractical amounts. Furthermore, the compatibility with the existing standard SAOC systems would be lost.
It is therefore highly appreciated, if concepts can be provided which teach how to overcome the above-described restrictions of the state of the art.
According to an embodiment, a decoder for generating an un-mixed audio signal including a plurality of un-mixed audio channels may have: an un-mixing-information determiner for determining un-mixing information by receiving first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information, and an un-mix module for applying the un-mixing information on a downmix signal, indicating a downmix of at least one audio object signal, to obtain an un-mixed audio signal including the plurality of un-mixed audio channels, wherein the un-mixing-information determiner is configured to determine the un-mixing information by modifying the first parametric information and the second parametric information to obtain modified parametric information, such that the modified parametric information has a frequency resolution which is higher than the first frequency resolution.
According to another embodiment, an encoder for encoding one or more input audio object signals may have: a downmix unit for downmixing the one or more input audio object signals to obtain one or more downmix signals, and a parametric-side-information generator for generating first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, such that the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information.
According to another embodiment, an encoded audio signal may have: a downmix portion indicating a downmix of one or more input audio object signals, a parametric side information portion including first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information.
According to another embodiment, a system may have: an inventive encoder for encoding one or more input audio object signals by obtaining one or more downmix signals indicating a downmix of one or more input audio object signals, by obtaining first parametric side information on the at least one audio object signal, and by obtaining second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information, and an inventive decoder for generating an un-mixed audio signal based on the one or more downmix signals, and based on the first parametric side information and the second parametric side information.
According to another embodiment, a method for generating an un-mixed audio signal including a plurality of un-mixed audio channels may have the steps of: determining un-mixing information by receiving first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information, and applying the un-mixing information on a downmix signal, indicating a downmix of at least one audio object signal, to obtain an un-mixed audio signal including the plurality of un-mixed audio channels, wherein determining the un-mixing information includes modifying the first parametric information and the second parametric information to obtain modified parametric information, such that the modified parametric information has a frequency resolution which is higher than the first frequency resolution.
According to another embodiment, a method for encoding one or more input audio object signals may have the steps of: downmixing the one or more input audio object signals to obtain one or more downmix signals, and generating first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, such that the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information.
Another embodiment may have a computer program for implementing the inventive methods when being executed on a computer or signal processor.
In contrast to state-of-the-art SAOC, embodiments of the present invention provide a spectral parameterization, such that
For the properties mentioned above, it is advantageous to have a parameterization which is understood by the standard SAOC decoder, but also allows for an efficient delivery of the information in the higher frequency resolution. The resolution of the underlying time-frequency representation determines the maximum performance of the enhancements. The invention here defines a method for delivering the enhanced high-frequency information in a way which is compact and allows a backwards compatible decoding.
An enhanced SAOC perceptual quality can be obtained, e.g., by dynamically adapting the time/frequency resolution of the filter bank or transform that is employed to estimate or used to synthesize the audio object cues to specific properties of the input audio object. For instance, if the audio object is quasi-stationary during a certain time span, parameter estimation and synthesis is beneficially performed on a coarse time resolution and a fine frequency resolution. If the audio object contains transients or non-stationaries during a certain time span, parameter estimation and synthesis is advantageously done using a fine time resolution and a coarse frequency resolution. Thereby, the dynamic adaptation of the filter bank or transform allows for
At the same time, traditional SAOC quality can be obtained by mapping standard SAOC data onto the time-frequency grid provided by the inventive backward compatible signal adaptive transform that depends on side information describing the object signal characteristics.
Being able to decode both standard and enhanced SAOC data, using one common transform, enables direct backward compatibility for applications that encompass mixing of standard and novel enhanced SAOC data. It also allows a time-frequency selective enhancement over the standard quality.
The provided embodiments are not limited to any specific time-frequency transform, but can be applied with any transform providing sufficiently high frequency resolution. The document describes the application to a Discrete Fourier Transform (DFT) based filter bank with switched time-frequency resolution. In this approach, the time domain signals are subdivided into shorter blocks, which also may overlap. The signal in each shorter block is weighted by a windowing function (normally having large values in the middle and at both ends tapered into zero). Finally the weighted signal is transformed into frequency domain by the selected transform, here, by application of the DFT.
A decoder for generating an un-mixed audio signal comprising a plurality of un-mixed audio channels is provided. The decoder comprises an un-mixing-information determiner for determining un-mixing information by receiving first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information. Moreover, the decoder comprises an un-mix module for applying the un-mixing information on a downmix signal, indicating a downmix of at least one audio object signal, to obtain an un-mixed audio signal comprising the plurality of un-mixed audio channels. The un-mixing-information determiner is configured to determine the un-mixing information by modifying the first parametric information and the second parametric information to obtain modified parametric information, such that the modified parametric information has a frequency resolution which is higher than the first frequency resolution.
Moreover, an encoder for encoding one or more input audio object signals is provided. The encoder comprises a downmix unit for downmixing the one or more input audio object signals to obtain one or more downmix signals. Furthermore, the encoder comprises a parametric-side-information generator for generating first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, such that the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information.
Furthermore, an encoded audio signal is provided. The encoded audio signal comprises a downmix portion, indicating a downmix of one or more input audio object signals, and a parametric side information portion comprising first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal. The frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information.
Moreover, a system is provided. The system comprises an encoder as described above and a decoder as described above. The encoder is configured to encode one or more input audio object signals by obtaining one or more downmix signals indicating a downmix of one or more input audio object signals, by obtaining first parametric side information on the at least one audio object signal, and by obtaining second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information. The decoder is configured to generate an un-mixed audio signal based on the one or more downmix signals, and based on the first parametric side information and the second parametric side information.
The encoder is configured to encode one or more input audio object signals by obtaining one or more downmix signals indicating a downmix of one or more input audio object signals, by obtaining first parametric side information on the at least one audio object signal, and by obtaining second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information. The decoder is configured to generate an audio output signal based on the one or more downmix signals, and based on the first parametric side information and the second parametric side information.
Furthermore, a method for generating an un-mixed audio signal comprising a plurality of un-mixed audio channels is provided. The method comprises:
Determining the un-mixing information comprises modifying the first parametric information and the second parametric information to obtain modified parametric information, such that the modified parametric information has a frequency resolution which is higher than the first frequency resolution.
Moreover, a method for encoding one or more input audio object signals is provided. The method comprises:
Moreover, a computer program for implementing one of the above-described methods when being executed on a computer or signal processor is provided.
Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:
Before describing embodiments of the present invention, more background on state-of-the-art-SAOC systems is provided.
In the case of a stereo downmix, the channels of the downmix signal 18 are denoted L0 and R0, in case of a mono downmix same is simply denoted L0. In order to enable the SAOC decoder 12 to recover the individual objects sj to sN, side-information estimator 17 provides the SAOC decoder 12 with side information including SAOC-parameters. For example, in case of a stereo downmix, the SAOC parameters comprise object level differences (OLD), inter-object correlations (IOC) (inter-object cross correlation parameters), downmix gain values (DMG) and downmix channel level differences (DCLD). The side information 20 including the SAOC-parameters, along with the downmix signal 18, forms the SAOC output data stream received by the SAOC decoder 12.
The SAOC decoder 12 comprises an upmixer which receives the downmix signal 18 as well as the side information 20 in order to recover and render the audio signals ŝ1 and ŝN onto any user-selected set of channels ŷ1 to ŷM, with the rendering being prescribed by rendering information 26 input into SAOC decoder 12.
The audio signals s1 to sN may be input into the encoder 10 in any coding domain, such as, in time or spectral domain. In case the audio signals s1 to sN are fed into the encoder 10 in the time domain, such as PCM coded, encoder 10 may use a filter bank, such as a hybrid QMF bank, in order to transfer the signals into a spectral domain, in which the audio signals are represented in several sub-bands associated with different spectral portions, at a specific filter bank resolution. If the audio signals s1 to sN are already in the representation expected by encoder 10, same does not have to perform the spectral decomposition.
As outlined above, side-information extractor 17 of
The side information extractor 17 depicted in
wherein the sums and the indices n and k, respectively, go through all temporal indices 34, and all spectral indices 30 which belong to a certain time/frequency tile 42, referenced by the indices l for the SAOC frame (or processing time slot) and m for the parameter band, and xin,k* is the complex conjugate of xin,k. Thereby, the energies of all sub-band values xi of an audio signal or object i are summed up and normalized to the highest energy value of that tile among all objects or audio signals.
Further, the SAOC side information extractor 17 is able to compute a similarity measure of the corresponding time/frequency tiles of pairs of different input objects s1 to sN. Although the SAOC side information extractor 17 may compute the similarity measure between all the pairs of input objects s1 to sN, SAOC side information extractor 17 may also suppress the signaling of the similarity measures or restrict the computation of the similarity measures to audio objects s1 to sN which form left or right channels of a common stereo channel. In any case, the similarity measure is called the inter-object cross-correlation parameter IOCi,jl,m. The computation is as follows
with again indices n and k going through all sub-band values belonging to a certain time/frequency tile 42, i and j denoting a certain pair of audio objects s1 to sN, and Re{ } denoting the operation of retaining only the real part (i.e., discarding the imaginary part) of the complex-valued argument.
The downmixer 16 of
This downmix prescription is signaled to the decoder side by means of down mix gains DMGi and, in case of a stereo downmix signal, downmix channel level differences DCLDi.
The downmix gains are calculated according to:
DMGi=20 log10(di+ε),(mono downmix),
DMGi=10 log10(d1,i2+d2,i2+ε),(stereo downmix),
where ε is a small number such as 10−9.
For the DCLDs the following formula applies:
In the normal mode, downmixer 16 generates the downmix signal according to:
for a mono downmix, or
for a stereo downmix, respectively.
Thus, in the abovementioned formulas, parameters OLD and IOC are a function of the audio signals, and parameters DMG and DCLD are functions of the downmix coefficients d. By the way, it is noted that d may be varying in time and frequency.
Thus, in the normal mode, downmixer 16 mixes all objects s1 to sN with no preferences, i.e., with handling all objects s1 to sN equally.
At the decoder side, the upmixer performs the inversion of the downmix procedure and the implementation of the “rendering information” 26 represented by a matrix R (in the literature sometimes also called A) in one computation step, namely, in case of a two-channel downmix
where matrix E is a function of the parameters OLD and IOC, and the matrix D contains the downmixing coefficients as
and wherein D* denotes the complex transpose of D. The matrix E is an estimated covariance matrix of the audio objects s1 to sN. In current SAOC implementations, the computation of the estimated covariance matrix E is typically performed in the spectral/temporal resolution of the SAOC parameters, i.e., for each (l,m), so that the estimated covariance matrix may be written as El,m. The estimated covariance matrix El,m is of size N×N with its coefficients being defined as
ei,jl,m=√{square root over (OLDil,mOLDjl,m)}IOCi,jl,m.
Thus, the matrix El,m with
has along its diagonal the object level differences, i.e., ei,jl,m=OLDil,m for i=j, since OLDil,m=OLDjl,m and IOCi,jl,m=1 for i=j. Outside its diagonal the estimated covariance matrix E has matrix coefficients representing the geometric mean of the object level differences of objects i and j, respectively, weighted with the inter-object cross correlation measure IOCi,jl,m.
In the following, embodiments of the present invention are described.
The decoder comprises an un-mixing-information determiner 112 for determining un-mixing information by receiving first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information.
Moreover, the decoder comprises an un-mix module 113 for applying the un-mixing information on a downmix signal, indicating a downmix of at least one audio object signal, to obtain an un-mixed audio signal comprising the plurality of un-mixed audio channels.
The un-mixing-information determiner 112 is configured to determine the un-mixing information by modifying the first parametric information and the second parametric information to obtain modified parametric information, such that the modified parametric information has a frequency resolution which is higher than the first frequency resolution.
The encoder comprises a downmix unit 91 for downmixing the one or more input audio object signals to obtain one or more downmix signals.
Furthermore, the encoder comprises a parametric-side-information generator 93 for generating first parametric side information on the at least one audio object signal and second parametric side information on the at least one audio object signal, such that the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information.
The encoder 61 is configured to encode one or more input audio object signals by obtaining one or more downmix signals indicating a downmix of one or more input audio object signals, by obtaining first parametric side information on the at least one audio object signal, and by obtaining second parametric side information on the at least one audio object signal, wherein the frequency resolution of the second parametric side information is higher than the frequency resolution of the first parametric side information.
The decoder 62 is configured to generate an un-mixed audio signal based on the one or more downmix signals, and based on the first parametric side information and the second parametric side information.
In the following, enhanced SAOC using backward compatible frequency resolution improvement is described.
It should be noted that adding the enhancement information to a single object in a mixture does not only improve the resulting quality of that specific object, but the quality of all objects sharing the approximate spatial location and having some spectral overlap.
In the following, backward compatible enhanced SAOC encoding with an enhanced encoder is described, in particular, an enhanced SAOC encoder which produces a bit stream containing a backward compatible side information portion and additional enhancements. The added information can be inserted into the standard SAOC bit stream in such a way that the old, standard-compliant decoders simply ignore the added data while the enhanced decoders make use of it. The existing standard SAOC decoders can decode the backward compatible portion of the parametric side information (PSI) and produce reconstructions of the objects, while the added information used by the enhanced SAOC decoder improves the perceptual quality of the reconstructions in most of the cases. Additionally, if the enhanced SAOC decoder is running on limited resources, the enhancements can be ignored and a basic quality reconstruction is still obtained. It should be noted that the reconstructions from standard SAOC and enhanced SAOC decoders using only the standard SAOC compatible PSI differ, but are judged to be perceptually very similar (the difference is of the similar nature as in decoding standard SAOC bit streams with an enhanced SAOC decoder).
The encoder comprises a downmix unit 91 for downmixing a plurality of audio object signals to obtain one or more downmix signals. For example, the audio object signals (e.g., the individual (audio) objects) are used by a downmix unit 91 to create a downmix signal. This may happen in time domain, frequency domain, or even an externally provided downmix can be used.
In the PSI-path, the (audio) object signals are transformed by a transform unit 92 from a time domain to a frequency domain, a time-frequency domain or a spectral domain (for example, by a transform unit 92 comprising one or more t/f-transform subunits 921, 922).
Moreover, the encoder comprises a parametric-side-information generator 93 for generating parametric side information. In the embodiment of
First, the signal is subdivided into analysis frames, which are then transformed into the frequency domain. Multiple analysis frames are grouped into a fixed-length parameter frame, e.g., in standard SAOC lengths of 16 and 32 analysis frames are common. It is assumed that the signal properties remain quasi-stationary during the parameter frame and can thus be characterized with only one set of parameters. If the signal characteristics change within the parameter frame, modeling error is suffered, and it would be beneficial to sub-divide the longer parameter frame into parts in which the assumption of quasi-stationarity is again fulfilled. For this purpose, transient detection is needed.
In an embodiment, the transform unit 92 is configured to transform one or more input audio object signals from the time domain to the time-frequency domain depending on a window length of a signal transform block comprising signal values of at least one of the one or more input audio object signals. The transform unit 92 comprises a transient-detection unit 101 for determining a transient detection result indicating whether a transient is present in one or more of the at least one audio object signals, wherein a transient indicates a signal change in one or more of the at least one audio object signals. Moreover, the transform unit 92 further comprises a window sequence unit 102 for determining the window length depending on the transient detection result.
For example, the transients may be detected by the transient-detection unit 101 from all input objects separately, and when there is a transient event in only one of the objects that location is declared as a global transient location. The information of the transient locations is used for constructing an appropriate windowing sequence. The construction can be based, for example, on the following logic:
The create-window-sequence unit 102 constructs the windowing sequence. At the same time, it also creates parameter sub-frames from one or more analysis windows. Each subset is analyzed as an entity and only one set of PSI-parameters are transmitted for each sub-block. To provide a standard SAOC compatible PSI, the defined parameter block length is used as the main parameter block length, and the possible located transients within that block define parameter subsets.
The constructed window sequence is outputted for time-frequency analysis of the input audio signals conducted by the t/f-analysis unit 103, and transmitted in the enhanced SAOC enhancement portion of the PSI.
The PSI consists of sets of object level differences (OLD), inter-object correlations (IOC), and information of the downmix matrix D used to create the downmix signal from the individual objects in the encoder. Each parameter set is associated with a parameter border which defines the temporal region to which the parameters are associated to.
The spectral data of each analysis window is used by the PSI-estimation unit 104 for estimating the PSI for standard SAOC part. This is done by grouping the spectral bins into parametric bands of standard SAOC and estimating the IOCs, OLDs and absolute objects energies (NRG) in the bands. Following loosely the notation of standard SAOC, the normalized product of two object spectra Si (f,n) and Sj(f,n) in a parameterization tile is defined as
where the matrix K(b,f,n):B×F
The spectral resolution can vary between the frames within a single parametric block, so the mapping matrix converts the data into a common resolution basis. The maximum object energy in this parameterization tile is defined to be the maximum object energy
Having this value, the OLDs are then defined to be the normalized object energies
And finally the IOC can be obtained from the cross-powers as
This concludes the estimation of the standard SAOC compatible parts of the bit stream.
A coarse-power-spectrum-reconstruction unit 105 is configured to use the OLDs and NRGs for reconstructing a rough estimate of the spectral envelope in the parameter analysis block. The envelope is constructed in the highest frequency resolution used in that block.
The original spectrum of each analysis window is used by a power-spectrum-estimation unit 106 for calculating the power spectrum in that window.
The obtained power spectra are transformed into a common high frequency resolution representation by a frequency-resolution-adaptation unit 107. This can be done, for example, by interpolating the power spectral values. Then the mean power spectral profile is calculated by averaging the spectra within the parameter block. This corresponds roughly to OLD-estimation omitting the parametric band aggregation. The obtained spectral profile is considered as the fine-resolution OLD.
The encoder further comprises a delta-estimation unit 108 for estimating a plurality of correction factors by dividing each of the plurality of OLDs of one of the at least one audio object signal by a value of a power spectrum reconstruction of said one of the at least one audio object signal to obtain the second parametric side information, wherein said plurality of OLDs has a higher frequency resolution than said power spectrum reconstruction.
In an embodiment, the delta-estimation unit 108 is configured to estimate a plurality of correction factors based on a plurality of parametric values depending on the at least one audio object signal to obtain the second parametric side information. E.g., the delta-estimation unit 108 may be configured to estimate a correction factor, “delta”, for example, by dividing the fine-resolution OLD by the rough power spectrum reconstruction. As a result, this provides for each frequency bin a (for example, multiplicative) correction factor that can be used for approximating the fine-resolution OLD given the rough spectra.
Finally, a delta-modeling unit 109 is configured to model the estimated correction factor in an efficient way for transmission. One possibility for modeling using Linear Prediction Coefficients (LPC) is described later below.
Effectively, the enhanced SAOC modifications consist of adding the windowing sequence information and the parameters for transmitting the “delta” to the bit stream.
In the following, an enhanced decoder is described.
The input downmix signal is transformed into frequency domain by a t/f-transform unit 111.
The estimated un-mixing matrix is applied on the transformed downmix signal by an un-mixing unit 110 to generate an un-mixing output.
Additionally, a decorrelation path is included to allow a better spatial control of the objects in the un-mixing. A decorrelation unit 119 conducts decorrelation on the transformed downmix signal and the result of the decorrelation is fed into the un-mixing unit 110. The un-mixing unit 110 uses the decorrelation result for generating the un-mixing output.
The un-mixing output is then transformed back into the time domain by a fit-transform unit 114.
The parametric processing path can take standard resolution PSI as the input, in which case the decoded PSI, which is generated by a standard-PSI-decoding unit 115, is adapted by a frequency-resolution-conversion unit 116 to the frequency resolution used in the t/f-transforms.
An alternative input combines the standard frequency resolution part of the PSI with the enhanced frequency resolution part and the calculations include the enhanced frequency resolution information. In more detail, an enhanced PSI-decoding unit 117 generates decoded PSI exhibiting enhanced frequency resolution.
An un-mixing-matrix generator 118 generates an un-mixing matrix based on the decoded PSI received from the frequency-resolution-conversion unit 116 or from the enhanced PSI-decoding unit 117. The un-mixing-matrix generator 118 may also generate the un-mixing matrix based on rendering information, for example, based on a rendering matrix. The un-mixing unit 110 is configured to generate the un-mixing output by applying this un-mixing matrix, being generated by the un-mixing-matrix generator 118, on the transformed downmix signal.
The first parametric information comprises a plurality of first parameter values, wherein the second parametric information comprises a plurality of second parameter values. The un-mixing-information determiner 112 comprises a frequency-resolution-conversion subunit 122 and a combiner 124. The frequency-resolution-conversion unit 112 is configured to generate additional parameter values, e.g., by replicating the first parameter values, wherein the first parameter values and the additional parameter values together form a plurality of first processed parameter values. The combiner 124 is configured to combine the first processed parameter values and the second parameter values to obtain a plurality of modified parameter values as the modified parametric information.
According to an embodiment, the standard frequency resolution part is decoded by a decoding subunit 121 and converted by a frequency-resolution-conversion subunit 122 into the frequency resolution used by the enhancement part. The decoded enhancement part, generated by an enhanced PSI-decoding subunit 123, is combined by a combiner 124 with the converted standard-resolution part.
In the following, the two decoding modes with possible implementations are described in more detail.
At first, decoding of standard SAOC bit streams with an enhanced decoder is described:
The enhanced SAOC decoder is designed so that it is capable decoding bit streams from standard SAOC encoders with a good quality. The decoding is limited to the parametric reconstruction only, and possible residual streams are ignored.
An un-mixing-matrix calculator 131, a temporal interpolator 132, and a window-frequency-resolution-adaptation unit 133 implement the functionality of the standard-PSI-decoding unit 115, of the frequency-resolution-conversion unit 116, and of the un-mixing-matrix generator 118 of
Normally, the frequency bins of the underlying time/frequency-representation are grouped into parametric bands. The spacing of the bands resembles that of the critical bands in the human auditory system. Furthermore, multiple t/f-representation frames can be grouped into a parameter frame. Both of these operations provide a reduction in the amount of necessitated side information with the cost of modeling inaccuracies.
As described in the SAOC standard, the OLDs and IOCs are used to calculate the un-mixing matrix G=ED*J, where the elements of E are defined as E(i,j)=IOCi,j√{square root over (OLDiOLDj)} approximates the object cross-correlation matrix, i and j are object indices, J≈(DED*)−1. The un-mixing-matrix calculator 131 may be conducted to calculate the un-mixing matrix.
The un-mixing matrix is then linearly interpolated by the temporal interpolator 132 from the un-mixing matrix of the preceding frame over the parameter frame up to the parameter border on which the estimated values are reached, as per standard SAOC. This results into un-mixing matrices for each time-/frequency-analysis window and parametric band.
The parametric band frequency resolution of the un-mixing matrices is expanded to the resolution of the time/frequency-representation in that analysis window by the window-frequency-resolution-adaptation unit 133. When the interpolated un-mixing matrix for parametric band b in a time-frame is defined as G(b), the same un-mixing coefficients are used for all the frequency bins inside that parametric band.
The window-sequence generator 134 is configured to use the parameter set range information from the PSI to determine an appropriate windowing sequence for analyzing the input downmix audio signal. The main requirement is that when there is a parameter set border in the PSI, the cross-over point between consecutive analysis windows should match it. The windowing determines also the frequency resolution of the data within each window (used in the un-mixing data expansion, as described earlier).
The windowed data is then transformed by the t/f-analysis module 135 into a frequency domain representation using an appropriate time-frequency transform, e.g., Discrete Fourier Transform (DFT), Complex Modified Discrete Cosine Transform (CMDCT), or Oddly stacked Discrete Fourier Transform (ODFT).
Finally, an un-mixing unit 136 applies the per-frame per-frequency bin un-mixing matrices on the spectral representation of the downmix signal X to obtain the parametric renderings Y. The output channel j is a linear combination of the downmix channels
The quality that can be obtained with this process is for most of the purposes perceptually indistinguishable from the result obtained with a standard SAOC decoder.
It should be noted that the above text describes reconstruction of individual objects, but in standard SAOC the rendering is included in the un-mixing matrix, i.e., it is included in parametric interpolation. As a linear operation, the order of the operations does not matter, but the difference is worth noting.
In the following, decoding of enhanced SAOC bit streams with an enhanced decoder is described.
The main functionality of the enhanced SAOC decoder is already described earlier in decoding of standard SAOC bit streams. This section will detail how the introduced enhanced SAOC enhancements in the PSI can be used for obtaining a better perceptual quality.
The decoder of
For example, at first, the value-expand-over-band unit 141 adapts the OLD and IOC values for each parametric band to the frequency resolution used in the enhancements, e.g., to 1024 bins. This is done by replicating the value over the frequency bins that correspond to the parametric band. This results into new OLDs OLDienh(f)=K(f,b)OLDi(b) and IOCs IOCi,jenh(f)=K(f,b)IOCi,j(b). K(f,b) is a kernel matrix defining the assignment of frequency bins f into parametric bands b.
Parallel to this, the delta-function-recovery unit 142 inverts the correction factor parameterization to obtain the delta function Cirec(f) of the same size as the expanded OLD and IOC.
Then, the delta-application unit 143 applies the delta on the expanded OLD values, and the obtained fine resolution OLD values are obtained by OLDifine(f)=Cirec(f)OLDienh(f).
In a particular embodiment, the calculation of un-mixing matrices, may, for example, be done by the un-mixing-matrix calculator 131 as with decoding standard SAOC bit stream: G(f)=E(f)D*(f)J(f), with Ei,j(f)=IOCi,jenh(f)√{square root over (OLDifine(f)OLDjfine(f))}, and J(f)≈(D(f)E(f)D*(f))−1. If wanted, the rendering matrix can be multiplied into the un-mixing matrix G(f). The temporal interpolation by the temporal interpolator 132 follows as per the standard SAOC.
As the frequency resolution in each window may be different (lower) from the nominal high frequency resolution, the window-frequency-resolution-adaptation unit 133 need to adapt the un-mixing matrices to match the resolution of the spectral data from audio to allow applying it. This can be made, e.g., by re-sampling the coefficients over the frequency axis to the correct resolution. Or if the resolutions are integer multiples, simply averaging from the high-resolution data the indices that correspond to one frequency bin in the lower resolution
The windowing sequence information from the bit stream can be used to obtain a fully complementary time-frequency analysis to the one used in the encoder, or the windowing sequence can be constructed based on the parameter borders, as is done in the standard SAOC bit stream decoding. For this, a window-sequence generator 134 may be employed.
The time-frequency analysis of the downmix audio is then conducted by a t/f-analysis module 135 using the given windows.
Finally, the temporally interpolated and spectrally (possibly) adapted un-mixing matrices are applied by an un-mixing unit 136 on the time-frequency representation of the input audio, and the output channel j can be obtained as a linear combination of the input channels
In the following, particular aspects of embodiments are described.
In an embodiment, the delta-modeling unit 109 of
Now, the estimation process of the correction factor, delta, and a possible modeling alternative using linear prediction coefficients (LPC) according to such an embodiment is described.
At first, delta estimation according to an embodiment is described.
The input to the estimation consists of the estimated fine-resolution power spectral profiles over the parameter block and from the coarse reconstruction of the power spectral profile based on the OLD and NRG parameters. The fine power spectrum profiles are calculated in the following manner. Si(f, n) is the complex spectrum of the i th object with f being the frequency bin index and 0≤n≤N−1 being the temporal window index in the modeling block of the length N. The fine-resolution power spectrum is then
The coarse reconstruction is calculated from the (de-quantized) OLDs and NRGs by Zi(f)=K(f,b)OLDi(b)NRGi(b),
where K(f, b) is the kernel matrix defining the assignment of frequency bins f into parametric bands b.
Two signals with differing spectral properties will be used as examples in this section: the first one is (pink) noise with practically flat spectrum (ignoring the spectral tilt), and the second is a tone from the instrument glockenspiel which has a highly tonal, i.e., peaky, spectrum.
It can be quickly noticed, the average difference between the fine and coarse value are rather small in the case of the noise signal, while the differences are very large in the tonal signal. These differences cause perceptual degradations in the parametric reconstruction of all objects.
The correction factor is obtained by dividing the fine-resolution curve by the coarse reconstruction curve:
Ci(f)=Pi(f)/Zi(f).
This allows recovering a multiplicative factor that can be applied on the rough reconstruction to obtain the fine-resolution curve:
Pirec(f)=Zi(f)Ci(f).
In the following, delta modeling is described.
The correction curve C is assigned into one or more modeling blocks over the frequency axis. A natural alternative is to use the same parameter band definitions as are used for the standard SAOC PSI. The modeling is then done for each block separately with the following steps:
It is possible to make a decision if the delta should be transmitted for each t-f tile (standard parametric band defining the frequency range and the parameter block the temporal range) independently. The decision can be made based on, for example,
Now, delta reconstruction and application is described.
The reconstruction of the correction curve follows the steps:
where i denotes the imaginary unit i=√{square root over (−1)}.
The inventive method and apparatus alleviate the aforementioned drawbacks of the state of the art SAOC processing using a filter bank or time-frequency transform with a high frequency resolution and providing an efficient parameterization of the additional information. Furthermore, it is possible to transmit this additional information in such a way that the standard SAOC-decoders can decode the backwards compatible portion of the information at a quality obtainable comparable to the one obtained using a standard-conformant SAOC encoder, and still allow the enhanced decoders to utilize the additional information for a better perceptual quality. Most importantly, the additional information can be represented in a very compact manner for efficient transmission or storage.
The presented inventive method can be applied on any SAOC scheme. It can be combined with any current and also future audio formats. The inventive method allows for enhanced perceptual audio quality in SAOC applications by a two-level representation of spectral side information.
The same idea can be used also in conjunction with MPEG Surround when replacing the concept of OLDs with channel-level differences (CLDs).
An audio encoder or method of audio encoding or related computer program as described above is provided. Moreover, an audio encoder or method of audio decoding or related computer program as described above is provided. Furthermore, an encoded audio signal or storage medium having stored the encoded audio signal as described above is provided.
Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.
The inventive decomposed signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.
Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed.
Some embodiments according to the invention comprise a non-transitory data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.
Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.
In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.
A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.
A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.
A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.
In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are performed by any hardware apparatus.
While this invention has been described in terms of several advantageous embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.
Number | Date | Country | Kind |
---|---|---|---|
13167485 | May 2013 | EP | regional |
This application is a continuation of copending International Application No. PCT/EP2013/070533, filed Oct. 2, 2013, which is incorporated herein by reference in its entirety, and additionally claims priority from U.S. Application No. 61/710,128, filed Oct. 5, 2012, and from European Application 13 167 485, filed May 13, 2013, which are all incorporated herein by reference in their entirety.
Number | Name | Date | Kind |
---|---|---|---|
20040122662 | Crockett et al. | Jun 2004 | A1 |
20070219808 | Herre et al. | Sep 2007 | A1 |
20090067634 | Oh et al. | Mar 2009 | A1 |
20090125313 | Hellmuth | May 2009 | A1 |
20090125314 | Hellmuth | May 2009 | A1 |
20100087938 | Oh | Apr 2010 | A1 |
20110040556 | Moon | Feb 2011 | A1 |
20110238425 | Neuendorf | Sep 2011 | A1 |
20120069898 | Valin et al. | Mar 2012 | A1 |
20120177204 | Hellmuth | Jul 2012 | A1 |
20120224702 | Den Brinker | Sep 2012 | A1 |
20120243690 | Engdegard | Sep 2012 | A1 |
20140074487 | Jang et al. | Mar 2014 | A1 |
20150287422 | Short | Oct 2015 | A1 |
20150332684 | Oh | Nov 2015 | A1 |
20160099002 | Ko et al. | Apr 2016 | A1 |
Number | Date | Country |
---|---|---|
101578654 | Nov 2009 | CN |
101855918 | Oct 2010 | CN |
102568487 | Jul 2012 | CN |
102696070 | Sep 2012 | CN |
WO 2009049895 | Apr 2009 | DE |
2010518460 | May 2010 | JP |
2010521703 | Jun 2010 | JP |
2010521866 | Jun 2010 | JP |
1020110082553 | Jul 2011 | KR |
2236046 | Sep 2004 | RU |
2011047887 | Apr 2011 | WO |
2011155170 | Dec 2011 | WO |
Entry |
---|
Bosi, et al., “ISO/IEC MPEG-2 Advanced Audio Coding1”, J. Audio Eng. Soc., vol. 45, No. 10, Oct. 1997, pp. 789-814. |
Engdegard, et al., “Spatial Audio Object Coding (SAOC)—The Upcoming MPEG Standard on Parametric Object Based Audio Coding”, Presented at the 124th Convention; May 17-20, 2008 Amsterdam, The Netherlands, 15 Pages. |
Faller, et al., “Binaural Cue Coding—Part II: Schemes and Applications”, IEEE Transactions on Speech and Audio Processing, vol. 11, No. 6, Nov. 2003, pp. 520-531. |
Faller, C , ““Parametric Joint-Coding of Audio Sources,””, 120th AES Convention, Paris, France, May 20, 2006, 12 pages. |
Girin, Laurent et al., “Informed Audio Source Separation from Compressed Linear Stereo Mixtures”. |
Herre, et al., “From SAC to SAOC—Recent Developments in Parametric Coding of Spatial Audio”, Fraunhofer Institute for Integrated Circuits, Erlangen, Germany. |
ISO/IEC 23003-2, “Information technology—MPEG audio technologies—Part 2: Spatial Audio Object Coding (SAOC)”. |
Liutkus, Antoine et al., “Informed source separation through spectrogram coding and data embedding”. |
Nesbit, Andrew et al., “Benchmarking Flexible Adaptive Time-Frequency Transforms for Underdetermined Audio Source Separation”, 1Electronic Engineering & Computer Science, Queen Mary, University of London, Mile End Road, London, E1 4NS, United Kingdom 2METISS Group, IRISA-INRIA, Campus de Beaulieu, 35042 Rennes Cedex, France. |
Ozerov, et al., “Informed source separation: source coding meets source separation”, IEEE Workshop on Applications of Signal Processing to Audio and Acoustics; Mohonk, NY, Oct. 2011, 5 pages. |
Parvaix, , “A Watermarking-Based Method for Informed Source Separation of Audio Signals With a Single Sensor”, IEEE Transactions 0:-.1 Audio. Speecii. And Language Processing. vol. 18, No. 6, Aug. 2010 Mathieu Parvaix, Student Member, IEEE, Laurent Girin, and Jean-Marc Brossier. |
Parvaix, Mathieu et al., “Informed Source Separation of Underdetermined Instantaneous Stereo Mixtures Using Source Index Embedding”, Grenoble Laboratory of Images, Speech, Signal and Automation (GIPSA-lab) CNRS UMR 5216 , Grenoble Institute of Technology, Grenoble, France. |
Zhang, et al., “An informed source separation system for speech signals”, HAL archives-ouverts.fr. |
Beack, Seungkwon et al., “An Efficient Time-Frequency Representation for Parametric-Based Audio Object Coding”, ETRI Journal, Nov. 30, 2011. |
Koo, Kyungryeol et al., “Variable Subband Analysis for High Quality Spatial Audio Object Coding”, Advanced Communication Technology, 2008.ICACT 2008. 10th International Conference on IEEE, Piscataway, NJ, USA, Feb. 17, 2008. |
Falch, Cornelia et al., “Spatial Audio Object Coding With Enhanced Audio Object Separation”, Proc. DAFx-10, Austria, IEM,, Sep. 6, 2010, pp. 1-7. |
Hellmuth, Oliver et al., “MPEG Spatial Audio Object Coding”, The ISO/MPEG Standard for Efficient Coding of Interactive Audio Scenes, Proceedings of the 129th Convention of the Audio Engineering Society, US, AES, Nov. 4, 2010, pp. 1-19. |
Herre, J et al., “The Reference Model Architecture for MPEG Spatial Audio Coding”, Proc. 118th Convention of the Audio Engineering Society, ES, AES,, May 28, 2005, p. 1-13. |
Number | Date | Country | |
---|---|---|---|
20150213806 A1 | Jul 2015 | US |
Number | Date | Country | |
---|---|---|---|
61710128 | Oct 2012 | US |
Number | Date | Country | |
---|---|---|---|
Parent | PCT/EP2013/070533 | Oct 2013 | US |
Child | 14678643 | US |