Patent Grant
11837242
Embodiments presented herein relate to a method, a transmitting node, a computer program, and a computer program product for supporting generation of comfort noise for at least two audio channels at a receiving node. Embodiments presented herein further relate to a method, a receiving node, a computer program, and a computer program product for generation of the comfort noise at the receiving node.
In communications networks, there may be a challenge to obtain good performance and capacity for a given communications protocol, its parameters and the physical environment in which the communications network is deployed.
For example, although the capacity in telecommunication networks is continuously increasing, it is still of interest to limit the required resource usage per user. In mobile telecommunication networks less required resource usage per call means that the mobile telecommunication network can service a larger number of users in parallel. Lowering the resource usage also yields lower power consumption in both devices at the user-side (such as in terminal devices) and devices at the network-side (such as in network nodes). This translates to energy and cost saving for the network operator, whilst enabling prolonged battery life and increased talk-time to be experienced in the terminal devices.
One mechanism for reducing the required resource usage for speech communication applications in mobile telecommunication networks is to exploit natural pauses in the speech. In more detail, in most conversations only one party is active at a time, and thus the speech pauses in one communication direction will typically occupy more than half of the signal. One way to utilize this property in order to decrease the required resource usage is to employ a Discontinuous Transmission (DTX) system, where the active signal encoding is discontinued during speech pauses.
During speech pauses it is common to transmit a very low bit rate encoding of the background noise to allow for a Comfort Noise Generator (CNG) system at the receiving end so as to fill the above-mentioned pauses with a background noise having similar characteristics as the original noise. The CNG makes the sound more natural compared to having silence in the speech pauses since the background noise is maintained and not switched on and off together with the speech. Complete silence in the speech pauses is commonly perceived as annoying and often leads to the misconception that the call has been disconnected.
A DTX system might further rely on a Voice Activity Detector (VAD), which indicates to the transmitting device whether to use active signal encoding or low rate background noise encoding. In this respect the transmitting device might be configured to discriminate between other source types by using a (Generic) Sound Activity Detector (GSAD or SAD), which not only discriminates speech from background noise but also might be configured to detect music or other signal types, which are deemed relevant.
Communication services may be further enhanced by supporting stereo or multichannel audio transmission. In these cases, the DTX/CNG system might also consider the spatial characteristics of the signal in order to provide a pleasant-sounding comfort noise.
A common mechanism to generate comfort noise is to transmit information about the energy and spectral shape of the background noise in the speech pauses. This can be done using significantly less number of bits than the regular coding of speech segments.
At the receiving device side the comfort noise is generated by creating a pseudo random signal and then shaping the spectrum of the signal with a filter based on information received from the transmitting device. The signal generation and spectral shaping can be performed in the time or the frequency domain.
An object of embodiments herein is to enable efficient generation of comfort noise for two or more channels.
According to a first aspect there is presented a method for supporting generation of comfort noise for at least two audio channels at a receiving node. The method is performed by a transmitting node. The method comprises determining spectral characteristics of audio signals on at least two input audio channels. The method comprises determining a spatial coherence between the audio signals on the respective input audio channels, wherein the spatial coherence is associated with perceptual importance measures. The method comprises dividing the spatial coherence into frequency bands, wherein a compressed representation of the spatial coherence is determined per frequency band by weighting the spatial coherence within each frequency band according to the perceptual importance measures. The method comprises signaling information about the spectral characteristics and information about the compressed representation of the spatial coherence per frequency band to the receiving node, for enabling the generation of the comfort noise for the at least two audio channels at the receiving node.
According to a second aspect there is presented transmitting node for supporting generation of comfort noise for at least two audio channels at a receiving node. The transmitting node comprises processing circuitry. The processing circuitry is configured to cause the transmitting node to determine spectral characteristics of audio signals on at least two input audio channels. The processing circuitry is configured to cause the transmitting node to determine a spatial coherence between the audio signals on the respective input audio channels, wherein the spatial coherence is associated with perceptual importance measures. The processing circuitry is configured to cause the transmitting node to divide the spatial coherence into frequency bands, wherein a compressed representation of the spatial coherence is determined per frequency band by weighting the spatial coherence within each frequency band according to the perceptual importance measures. The processing circuitry is configured to cause the transmitting node to signal information about the spectral characteristics and information about the compressed representation of the spatial coherence per frequency band to the receiving node, for enabling the generation of the comfort noise for the at least two audio channels at the receiving node.
According to a third aspect there is presented a computer program for supporting generation of comfort noise for at least two audio channels at a receiving node, the computer program comprising computer program code which, when run on a transmitting node, causes the transmitting node to perform a method according to at least the first aspect.
According to a fourth aspect there is presented a computer program product comprising a computer program according to the third aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium.
According to a fifth aspect there is presented a radio transceiver device. The radio transceiver device comprises a transmitting node according to the second aspect.
Advantageously these methods, these transmitting nodes, this computer program, this computer program product and this radio transceiver device enable efficient generation of comfort noise for two or more channels.
Advantageously these methods, these transmitting nodes, this computer program, this computer program product and this radio transceiver device enable comfort noise to be generated for two or more channels without suffering from the issues noted above.
Advantageously these methods, these transmitting nodes, this computer program, this computer program product and this radio transceiver device enable the amount of information that needs to be encoded in a stereo or multi-channel DTX system to be reduced, whilst keeping the ability to recreate a realistic stereo image at the receiving node.
Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the claims as well as from the drawings.
The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:
The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.
The spatial coherence constitutes spatial properties of the multi-channel audio representation and may also be called channel coherence as it is representing the coherence between the audio channels. In the following description, terms channel coherence and spatial coherence are interchangeably used.
When two mono encoders each having its own DTX system working separately on the signals in each of the two stereo channels are used, different energies and spectral shapes in the two different signals will be transmitted.
In most realistic cases the difference in energy and spectral shape between the signal in the left channel and the signal in the right channel will not be large but there could still be a big difference in how wide the stereo image of the signal is perceived.
If the random sequences used to generate the comfort noise is synchronized between the signal in the left channel and the signal in the right channel the result will be a stereo signal sounds with a very narrow stereo image and which gives the sensation of the sound originating from the center of the head of the listener. If instead the signal in the left channel and the signal in the right channel would not be synchronized it would give the opposite effect, i.e. a signal with a very wide stereo image.
In most cases the original background noise will have a stereo image that is somewhere in-between these two extremes which mean that, either with synchronized or not synchronized random sequences, there would be an annoying difference in the stereo image when the transmitting device switches between active speech encoding, with a good representation of the stereo width, and non-active noise encoding.
The perceived stereo image width of the original background noise might also change during a call, e.g. because the user of the transmitting device is moving around and/or because of things occurring in the background. A system with two mono encoders each having its own DTX system has no mechanism to follow these changes.
One additional issue with using a dual mono DTX system is that the VAD decision will not be synchronized between the two channels, which might lead to audible artifacts when e.g. the signal in the left channel is encoded with active encoding and the signal in the right channel is encoded with the low bit rate comfort noise encoding. It might also lead to that the random sequence will be synchronized in some time instances and unsynchronized in others, resulting in a stereo image that toggles between being extremely wide and extremely narrow over time.
Hence, there is still a need for an improved generation of comfort noise for two or more channels.
The following embodiments describe a DTX system for two channels (stereo audio), but the methods may in general be applied for DTX and CNG for multi-channel audio.
The transmitting node 200a might communicate with the receiving node 200b over a direct communication link 110 or over an indirect communication link 110 via one or more other devices, nodes, or entities, such as network nodes, etc. in the communication network 100.
In some aspects the transmitting node 200a is part of a radio transceiver device 200 and the receiving node 200b is part of another radio transceiver device 200. Additionally, in some aspects the radio transceiver device 200 comprises both the transmitting node 200a and the receiving node 200b. There could be different examples of radio transceiver devices. Examples include, but are not limited to, portable wireless devices, mobile stations, mobile phones, handsets, wireless local loop phones, user equipment (UE), smartphones, laptop computers, and tablet computers.
As disclosed above, a DTX system can be used in order to transmit encoded speech/audio only when needed.
The embodiments disclosed herein therefore relate to mechanisms for supporting generation of comfort noise for at least two audio channels at a receiving node 200b and for generation of comfort noise for at least two audio channels at a receiving node 200b. In order to obtain such mechanisms there is provided a transmitting node 200a, a method performed by the transmitting node 200a, a computer program product comprising code, for example in the form of a computer program, that when run on a transmitting node 200a, causes the transmitting node 200a to perform the method. In order to obtain such mechanisms there is further provided a receiving node 200b, a method performed by the receiving node 200b, and a computer program product comprising code, for example in the form of a computer program, that when run on processing circuitry of the receiving node 200b, causes the receiving node 200b to perform the method.
The spatial coherence is encoded in a very efficient way before transmission since the whole rationale behind using the DTX system 300 is to transmit as little information as needed in the pauses between the speech/audio.
According to an embodiment the perceptual importance measure is based on the spectral characteristics of the at least two input audio channels.
According to an embodiment the perceptual importance measure is determined based on the power spectra of the at least two input audio channels.
According to an embodiment the perceptual importance measure is determined based on the power spectrum of a weighed sum of the at least two input audio channels.
According to an embodiment the compressed representation of the spatial coherence is being one single value per frequency band.
The spatial coherence is encoded in a very efficient way before transmission since the whole rationale behind using the DTX system 300 is to transmit as little information as needed in the pauses between the speech/audio. One single value of the spatial coherence is therefore determined per frequency band.
The single value of the spatial coherence is determined by weighting the spatial coherence values within each frequency band. One purpose of the weighting function used for the weighting is to put a higher weight on spatial coherences that occur at frequencies being perceptually more important than others. Hence, the spatial coherence values within each frequency band are weighted according to the perceptual importance measures of the corresponding values of the spectral characteristics.
At the decoder at the receiving node 200b the coherence is reconstructed and a comfort noise signal with a stereo image similar to the original sound is created.
Embodiments relating to further details of supporting generation of comfort noise for at least two audio channels at a receiving node 200b as performed by the transmitting node 200a will now be disclosed.
The herein disclosed embodiments are applicable to a stereo encoder and decoder architecture as well as for a multi-channel encoder and decoder where the channel coherence is considered in channel pairs.
In some aspects the stereo encoder receives a channel pair [l(m, n) r(m, n)] as input, where l(m, n) and r(m, n) denote the input signals for the left and right channel, respectively, for sample index n of frame m. The signal is processed in frames of length N samples at a sampling frequency fs, where the length of the frame might include an overlap (such as a look-ahead and/or memory of past samples).
As in
[lwin(m,n)rwin(m,n)]=[l(m,n)win(n)r(m,n)win(n)],n=0,1,2, . . . ,N−1.
Hence, according to an embodiment the audio signals l(m, n), r(m, n), for frame index m and sample index n, of the at least two audio channels are windowed to form respective windowed signals lwin(m, n), rwin(m, n) before the spectral characteristics are determined. The choice of window might generally depend on various parameters, such as time and frequency resolution characteristics, algorithmic delay (overlap length), reconstruction properties, etc. The thus windowed channel pair [lwin(m, n) rwin(m, n)] is then transformed according to:
A general definition of the channel coherence Cgen(f) for frequency f is given by:
where Sxx(f) and Syy(f) represent the respective power spectrum of the two channels x and y, and Sxy (f) is the cross power spectrum of the two channels x and y. In a DFT based solution, the spectra may be represented by the DFT spectra. Particularly, according to an embodiment the spatial coherence C(m, k) for frame index m and frequency bin index k is determined as:
where L(m, k) is the spectrum of the windowed audio signal lwin(m, n), where R(m, k) is the spectrum of the windowed audio signal rwin(m, n), and where * denotes the complex conjugate.
The above expression for the coherence is commonly computed with a high frequency resolution. One reason for this is that the frequency resolution depends on the signal frame size, which typically would be the same for CNG encoding as for active speech/audio encoding where a high resolution is desired. Another reason is that a high frequency resolution allows for a perceptually motivated frequency band division. Yet another reason could be that elements of the coherence calculation, i.e. L(m, k), R(m, k), Sxx, Sxy, Syy may, in a typical audio encoder, be used for other purposes where a higher frequency resolution is desired. A typical value with a sampling frequency fs=48 kHz and frame length of 20 ms would be 960 frequency bins for the channel coherence.
For an application of DTX where it is crucial to keep the bit rate for encoding inactive (i.e. non-speech) segments low it is not feasible to transmit the channel coherence with high frequency resolution. To reduce the number of bits needed to represent the channel coherence, the spectrum can be divided into frequency bands, as shown in
All frequency bands might have equal frequency-wise width, but more common in audio coding applications is to match the width of each frequency band to the human perception of audio, thus resulting in comparatively narrow frequency bands for the low frequencies and increasing widths of the frequency bands for higher frequencies. Particularly, according to an embodiment the spatial coherence is divided into frequency bands of non-equal lengths. For example, the frequency bands can be created using the ERB-rate scale, where ERB is short for equivalent rectangular frequency bandwidth.
In one embodiment the compressed representation of the coherence is defined by the mean value of the coherence within each frequency band and this single value per frequency band is transmitted to the decoder at the receiving node 200b such that the decoder then can use this single value for all frequencies within the frequency band when generating the comfort noise, or possibly with some smoothing over signal frames and/or frequency bands in order to avoid abrupt changes in time and/or frequency.
However, as disclosed above in step S204, in another embodiment the different frequencies within a frequency band are in the determination of a single coherence value per frequency band given different weights depending on the perceptual importance measures.
There could be different examples of perceptual importance measures.
In some aspects the perceptual importance measure relates to the spectral characteristics.
Particularly, in one embodiment the perceptual importance measure relates to the magnitude or power spectra of the at least two input audio signals.
In another embodiment the perceptual importance measure relates to the magnitude or power spectrum of a weighted sum on the at least two input audio channels.
In some aspects high energy corresponds to high perceptual importance, and vice versa. Particularly, according to an embodiment the spatial coherence values within each frequency band are weighted such that the spatial coherence values corresponding to frequency coefficients with a higher power has more impact on this one single value of the spatial coherence compared to the spatial coherence values corresponding to frequency coefficients with lower energy.
According to an embodiment the different frequencies within a frequency band are given different weights depending on the power at each frequency. One rationale behind this embodiment is that a frequency with a higher energy should have more impact on the combined coherence value compared to another frequency with lower energy.
In some other aspects the perceptual importance measure relates to encoded spectral characteristics. The encoded spectral characteristics might closer (i.e. closer than spectral characteristics that are not encoded) reflect the signal as reconstructed at the receiving node 200b.
In some other aspects the perceptual importance measure relates to the spatial coherence. It may for example be more perceptually important to represent signal components with a higher spatial coherence more accurately than signal components with a lower spatial coherence. In another aspect the perceptual importance measure may be related to the spatial coherence over time, including actively encoded speech/audio segments. One reason for this is that it may be perceptually important to generate spatial coherence of similar characteristics as in actively encoded speech/audio segments.
Other perceptual importance measures are also envisioned.
According to an embodiment a weighted mean is used to represent the coherence in each frequency band where the transformed energy spectrum |LR(m, k)|2 for the mono signal lr(m, n)=w1l(m, n)+w2 r(m, n) defines the perceptual importance measures within frame m and is used as the weighting function. That is, in some aspects the energy spectrum |LR(m, k)|2 of lr(m, n)=w1l(m, n)+w2r(m, n) is used for weighting the spatial coherence values. The downmix weights w1 and w2 may be constant or variable over time or, if similar operation is performed in the frequency domain, constant or variable over frequency. In one embodiment, the weights on the channels are equal, e.g. w1=w2=0.5. Then, according to an embodiment each frequency band extends between a lower frequency bin and an upper frequency bin and the one single value of the spatial coherence Cw(m, b) for frame index m and frequency band b is determined as:
where m is the frame index, b is the index of the frequency band, Nband is the total number of frequency bands and where limit(b) denotes the lowest frequency bin of frequency band b. The parameter limit(b) thus describes the first coefficient in each frequency band and defines the borders between the frequency bands. In this embodiment limit(b) would also be defined for frequency band Nband to define the upper limit of frequency band Nband−1. There could be different ways to obtain limit(b). According to an embodiment limit(b) is provided as a function or lookup table.
Assuming that the energies are the same for all bins in a frequency band, then the weighted mean and the non-weighted mean will be equal. Further, assuming that the energy is zero for all bins in a frequency band except one bin, then the weighted mean will be equal to the coherence value of that one bin.
The spatial coherence values Cw(m, b) are then encoded to be stored or transmitted to a decoder at the receiving node 200b where comfort noise is generated using the decoded coherence in order to create a realistic stereo image.
Encoding of Spatial Coherence According to an Embodiment
The coherence representative values given per frequency band form a spatial coherence vector C,m=[C1,m C2,m . . . Cb,m . . . CN
In an embodiment, the coherence vector is encoded using a predictive scheme followed by a variable bit rate entropy coding. The coding scheme further improves the performance through an adaptive inter-frame prediction. The encoding of the coherence vector considers the following properties: (1) adaptable encoding to a varying per-frame bit budget Bm, (2) the coherence vector shows strong frame-to-frame similarity, and (3) error propagation should be kept low for lost frames.
To address the varying per-frame bit budget, a coarse-fine encoding strategy is implemented. More specifically, the coarse encoding is first achieved at a low bit rate and the subsequent fine encoding may be truncated when the bit limit is reached.
In some embodiments, the coarse encoding is performed utilizing a predictive scheme. In such embodiments, a predictor works along the coherence vector for increasing bands b and estimates each coefficient based on the previous values of the vector. That is, an intra-frame prediction of the coherence vector is performed and is given by:
Each predictor set p(q) consists of (Nbnd−1) predictors, each predictor comprising (b−1) predictor coefficients for each band b where q=1, 2, . . . Nq and Nq indicates a total number of predictor sets. As shown above, there are no previous values when b=1 and the intra-frame prediction of the coherence vector is zero. As an example, a predictor set number q when there are six coherence bands, Nbnd=6, is given by
P(q)={[p2,1(q)],[p3,1(q)p3,2(q)],[p4,1(q)p4,2(q)p4,3(q)],[p5,1(q)p5,2(q)p5,3(q)p5,4(q)],[p6,1(q)p6,2(q)p6,3(q)p6,4(q)p6,5(q)]}.
As another example, the total number of predictor sets may be four, i.e. Nq=4, which indicates that the selected predictor set may be signaled using 2 bits. In some embodiments, predictor coefficients for a predictor set q may be addressed sequentially and stored in a single vector of length ΣiN
In step 700, for each frame m, a bit variable (also referred to as a bit counter) to keep track of the bits spent for the encoding is initialized to zero (Bcurr,m=0). The encoding algorithm receives a coherence vector (Cb,m) to encode, a copy of the previous reconstructed coherence vector (Ĉb,m-1), and a bit budget Bm. In some embodiments, the bits spent in preceding encoding steps may be included in Bm and Bcurr,m. In such embodiments, the bit budget in the algorithm below can be given by Bm−Bcurr,m.
In step 710, a predictor set P(q*) which gives the smallest prediction error out of the available predictors P(q), q=1, 2, . . . , Nq is selected. The selected predictor set is given by
In some embodiments, b=1 is omitted from the predictor set because the prediction is zero and contribution to the error will be the same for all predictor sets. The selected predictor set index is stored and the bit counter (Bcurr,m) is increased with the required number of bits, e.g. Bcurr,m Bcurr,m+2 if two bits are required to encode the predictor set.
In step 720, a prediction weighting factor α is computed. The prediction weighting factor is used to create a weighted prediction as described in step 760 below. The weight factor α is determined based on a bit-budget Bm available for encoding the vector of spatial coherence values in each frame m.
In general terms, the weight factor α can take a value in the range from 0 to 1, i.e. from only using information from the current frame (α=1) to only using information from the previous frame (α=0) and anything in-between (0<α<1). It is in some aspects desirable to use an as high weight factor α as possible since a lower weight factor α might make the encoding more sensitive to lost frames. But selection of the weight factor α has to be balanced with the bit budget Bm per frame m since a lower value of the weight factor α commonly yields less encoded bits.
The value of the weight factor α used in the encoding has to, at least implicitly, be known in the decoder at the receiving node 200b. That is, in an embodiment, information about the weight factor α has to be encoded and transmitted (as in step S1016) to the decoder. In other embodiments, the decoder may derive the prediction weight factor based on other parameters already available in the decoder. Further aspects of how to provide the information about the weight factor α will be disclosed below.
It is further assumed that the bit budget Bm for frame m for encoding the spatial coherence is known in the decoder at the receiving node 200b without explicit signaling from the transmitting node 200a. In this respect the value of the bit budget Bm does not need to be explicitly signalled to the receiving node 200b. It comes as a side effect, since the decoder at the receiving node 200b knows how to interpret the bitstream it also knows how many bits have been decoded. The remaining bits are simply found at the decoder at the receiving node 200b by subtracting the decoded number of bits from the total bit budget (which is also known).
In some aspects, based on the bit-budget Bm a set of candidate weight factors is selected and a trial encoding (without performing the rate-truncation strategy as disclosed below) with the combined prediction and residual encoding scheme is performed for all these candidate weight factors in order to find the total number of encoded bits, given the candidate weight factor used. Particularly, according to an embodiment the weight factor α is determined by selecting a set of at least two candidate weight factors and performing trial encoding of the vector of spatial coherence values for each candidate weight factor.
In some aspects, which candidate weight factors to use during the trial encoding is based on the bit-budget Bm. In this respect, the candidate weight factors might be determined by means of performing a table lookup with the bit-budget Bm as input or by inputting the bit-budget Bm to a function. The table lookup might be performed on table values obtained through training on a set of background noise.
The trial encoding for each candidate weight factor yields a respective total number of encoded bits for the vector of spatial coherence values. The weight factor α might then be selected depending on whether the total number of encoded bits for the candidate weight factors fits within the bit-budget Bm or not. Particularly, according to an embodiment the weight factor α is selected as the largest candidate weight factor for which the total number of encoded bits fits within the bit-budget Bm. According to an embodiment the weight factor α is selected as the candidate weight factor yielding fewest total number of encoded bits when the total number of encoded bits does not fit within the bit-budget Bm for any of the candidate weight factors.
That is, if all candidate weight factors lead to a total number of encoded bits being within the bit-budget Bm, the highest candidate weight factor is selected as the weight factor α. Likewise, if only the lowest or none of the candidate weight factors lead to a total number of bits within the bit-budget Bm, the candidate weight factor that leads to the lowest number of bits is selected as the weight factor α. Which of the candidate weight factor is selected is then signaled to the decoder.
An illustrative example where the trial encoding is performed for two candidate weight factors αlow and αhigh, resulting in the number of bits Bcurrlow,m and Bcurrhigh,m, respectively, needed for the encoding of the vector of spatial coherence values will now be disclosed.
Using Bcurr,m as the input, two candidate weight factors αlow and αhigh are obtained, either by means of performing a table lookup with the bit-budget Bm as input or by inputting the bit-budget Bm to a function. Trial encoding is performed without the rate-truncation strategy described below for each candidate weight factor αlow and αhigh, yielding two values Bcurrlow,m and Bcurrhigh,m of the number of bits needed for the encoding. Based on this, one of the two candidate weight factors αlow and αhigh is selected according for the encoding as follows:
The selected weight factor α is encoded using one bit, e.g. “0” for αlow and “1” for αhigh. The third alternative in the expression above for the weight factor α should be interpreted as follows: If both candidate weight factors αlow and αhigh yield a resulting number of encoded bits that exceeds the bit budget Bm, then the candidate weight factor yielding the lowest number of encoded bits is selected.
For each of the bands b=1, 2, . . . Nbnd in step 730, the following steps are performed:
In step 740, an intra-frame prediction value, Ĉintra,b,m(q), is obtained. There are no preceding encoded coherence values for the first band (b=1). In some embodiments, the intra-frame prediction for the first band may be set to zero, Ĉintra,1,m(q)=0. In some embodiments, the intra-frame prediction for the first band may be set to an average value
In some alternative embodiments, the coherence value of the first band may be encoded separately. In such embodiments, the first value is encoded using a scalar quantizer to produce reconstructed value ĈSQ,1,m(q). Accordingly, the intra-frame prediction for the first band may be set to the reconstructed value, Ĉintra,1,m(q)=ĈSQ,1,m(q). The bit counter, Bcurr,m, is increased by the amount of bits required to encode the coefficient. For example, if 3 bits are used to encode the coefficient, 3 bits are added to the current amount of bits spent for the encoding, for example, Bcurr,m:=Bcurr,m+3.
For the remaining bands b=2, 3, . . . , Nbnd, the intra-frame prediction Ĉb,m(q) is based on previously encoded coherence values, i.e Ĉintra,b,m(q)=Σi=1b-1pb,i(q)Ĉi,m.
In step 750, an inter-frame prediction value, Ĉinter,b,m, is obtained based on previously reconstructed coherence vector elements from one or more preceding frames. In cases where the background noise is stable or changing slowly, the frame-to-frame variation in the coherence band values Cb,m will be small. Hence, an inter-frame prediction using the values from previous frame will often be a good approximation which yields a small prediction residual and a small residual coding bit rate. As an example, a last reconstructed value for band b may be used for an inter-frame prediction value, i.e. Ĉinter,b,m=Ĉb,m-1. An inter-frame linear predictor considering two or more preceding frames can be formulated as Ĉinter,m=Σn=1N
In step 760, a weighted prediction, Ĉpred,b,m(q), is formed based on the intra-frame prediction, Ĉintra,b,m(q), the inter-frame prediction, Ĉinter,b,m, and the prediction weighting factor α. In some embodiments, the weighted prediction is given by Ĉpred,b,m(q)=αĈintra,b,m(q)+(1−α)Ĉinter,b,m.
In step 770, a prediction residual is computed and encoded. In some embodiments, the prediction residual is computed based on the coherence vector and the weighted prediction, i.e. rb,m=Cb,m−Ĉpred,b,m(q). In some embodiments, a scalar quantizer is used to quantize the prediction residual to an index Ib,m. In such embodiments, the index is given by Ib,m=SQ(rb,m) where SQ(x) is a scalar quantizer function with a suitable range. An example of a scalar quantizer is shown in Table 1 below. Table 1 shows an example of reconstruction levels and quantizer indices for a prediction residual.
In some embodiments, the index Ib,m is encoded with a variable length codeword scheme that consumes fewer bits for smaller values. Some examples for encoding the prediction residual are Huffman coding, Golomb-Rice coding, and unary coding (the unary coding is the same as the Golomb-Rice coding with divisor 1). In the step of encoding the prediction residual, the remaining bit budget (Bm−Bcurr,m) needs to be considered. If the length of the codeword Lcode(Ib,m) corresponding to index Ib,m fits within the remaining bit budget, i.e. Lcode (Ib,m)≤Bm−Bcurr,m, the index Ib,m is selected as the final index Ib,m*. If the remaining bits are not sufficient to encode the index Ib,m, a bit rate truncation strategy is applied. In some embodiments, the bit rate truncation strategy includes encoding the largest possible residual value, assuming that smaller residual values cost fewer bits. Such a rate truncation strategy can be achieved by reordering a codebook as illustrated by table 800 in
If the length of the codeword determined by the upward search fits does not exceed bit budget, the final index is selected Ib,m*=Ib,mtrunc and Ib,m* is output to the bitstream and the reconstructed residual is formed based on the final index, i.e. {circumflex over (r)}b,m=R(Ib,m*).
If after the upward search, the length of the codeword still exceeds the bit budget, Lcode(Ib,mtrunc)>Bm−Bcurr,m, this means that the bit limit has been reached Bm=Bcurr,m. In such instances, the reconstructed residual is set to zero {circumflex over (r)}b,m=0 and an index is not added to the bitstream. Since the decoder keeps a synchronized bit counter, Bcurr,m, the decoder may detect this situation and use {circumflex over (r)}b,m=0 without explicit signaling.
In an alternative embodiment, if the length of the codeword associated with the initial index exceeds the bit budget, the residual value is immediately set to zero, thereby foregoing the upward search describe above. This could be beneficial if computational complexity is critical.
In step 780, a reconstructed coherence value Ĉb,m is formed based on the reconstructed prediction residual and the weighted prediction, i.e. Ĉb,m=Ĉpred,b,m(q)+{circumflex over (r)}b,m.
In step 790, the bit counter is incremented accordingly. As described above, the bit counter is increased throughout the encoding process 701.
In some embodiments, the frame-to-frame variations in the coherence vector are small. Hence, the inter-frame prediction using the previous frame value is often a good approximation which yields a small prediction residual and a small residual coding bit rate. Additionally, the prediction weighting factor α serves the purpose of balancing the bit rate versus the frame loss resilience.
In step 900, a bit counter, Bcurr,m, configured to keep track of the bits spent during the decoding process 901 is initialized to zero, i.e. Bcurr,m=0. For each frame m, the decoder obtains a copy of the last reconstructed coherence vector Ĉb,m-1 and a bit budget Bm.
In step 910, a selected predictor set P(q*) is decoded from a bitstream. The bit counter is increased by the amount of bits required to decode the selected predictor set. For example, if two bits are required to decode the selected predictor set, the bit counter, Bcurr,m, is increased by two, i.e. Bcurr,m:=Bcurr,m+2.
In step 920, the prediction weighting factor α corresponding to the weighting factor used in the encoder is derived.
For each of the bands b=1, 2, . . . Nbnd in step 930, the following steps are performed:
In step 940, an intra-prediction value, Ĉintra,b,m(q), is obtained. The intra-frame prediction for the first band is obtained similarly to step 740 of the encoding process 701. Accordingly, the intra-frame prediction for the first frame may be set to zero (Ĉintra,1,m(q)=0),
For the remaining bands b=2, 3, . . . , Nbnd, the intra-frame prediction Ĉintrab,m(q) is based on the previously decoded coherence values, i.e. Ĉintra,b,m(q)=Σi=1b-1pb,i(q)Ĉi,m.
In step 950, an inter-frame prediction value, Ĉinter,b,m, is obtained similarly to step 750 of the encoding process 701. As an example, a last reconstructed value for band b may be used for an inter-frame prediction value, i.e. Ĉinter,b,m=Ĉb,m-1.
In step 960, a weighted prediction, Ĉpred,b,m(q), is formed based on the intra-frame prediction, Ĉintra,b,m(q), the inter-frame prediction, Ĉinter,b,m, and the prediction weighting factor α. In some embodiments, the weighted prediction is given by Ĉpred,b,m(q)=αĈintra,b,m(q)+(1−α)Ĉinter,b,m.
In step 970, a reconstructed prediction residual, {circumflex over (r)}b,m, is decoded. If the bit counter, Bcurr,m, is below the bit limit, i.e. Bcurr,m<Bm, the reconstructed prediction residual is derived from an available quantizer index {circumflex over (r)}b,m=R(Ib,m*). If the bit counter equals or exceeds the bit limit, the reconstructed prediction residual is set to zero, i.e. {circumflex over (r)}b,m=0.
In step 980, a coherence value Ĉb,m is reconstructed based on the reconstructed prediction residual and the weighted prediction, i.e. Ĉb,m=Ĉpred,b,m(q)+{circumflex over (r)}b,m. In step 990, the bit counter is incremented.
In some embodiments, further enhancements of the CNG may be required in the encoder. In such embodiments, a local decoder will be run in the encoder where the reconstructed coherence values Ĉb,m are used.
In step S1008, the encoder combines the first prediction and the second prediction using the prediction weighting factor into a combined prediction.
In step S1010, the encoder forms a prediction residual using the vector element and the combined prediction. In step S1012, the encoder encodes the prediction residual with a variable bit rate scheme. In some embodiments, the prediction residual is quantized to form a first residual quantizer index, wherein the first residual quantizer index is associated with a first code word. In some embodiments, the step of encoding the prediction residual with the variable bit rate scheme includes encoding the first residual quantizer index as a result of determining that the length of the first code word does not exceed the amount of remaining bits. In some embodiments, the step of encoding the prediction residual with the variable bit rate scheme includes obtaining a second residual quantizer index as a result of determining that the length of the first code word exceeds the amount of remaining bits, wherein the second residual quantizer index is associated with a second code word, and wherein the length of the second code word is shorter than the length of the first code word. In such embodiments, the process 600 includes a further step in which the encoder determines whether the length of the second code word exceeds the determined amount of remaining bits.
In step S1014, the encoder reconstructs the vector element based on the combined prediction and the prediction residual. In step S1016, the encoder transmits the encoded prediction residual. In some embodiments, the encoder encodes also the prediction weighting factor and transmits the encoded prediction weighting factor.
In some embodiments, the process 1000 includes a further step in which the encoder receives a first signal on a first input channel, receives a second signal on a second input channel, determines spectral characteristics of the first signal and the second signal, determines a spatial coherence based on the determined spectral characteristics of the first signal and the second signal, and determines the vector based on the spatial coherence.
In step S1108, the decoder combines the first prediction and the second prediction using the prediction weighting factor into a combined prediction.
In step S110, the decoder decodes a received encoded prediction residual. In some embodiments, the step of decoding the encoded prediction residual includes determining an amount of remaining bits available for decoding and determining whether decoding the encoded prediction residual exceeds the amount of remaining bits. In some embodiments, the step of decoding the encoded prediction residual includes setting the prediction residual as zero as a result of determining that decoding the encoded prediction residual exceeds the amount of remaining bits. In some embodiments, the step of decoding the encoded prediction residual includes deriving the prediction residual based on a prediction index as a result of determining that decoding the encoded prediction residual does not exceed the amount of remaining bits, wherein the prediction index is a quantization of the prediction residual.
In step S1112, the decoder reconstructs the vector element based on the combined prediction and the prediction residual. In some embodiments, the vector is one of a sequence of vectors. In some embodiments, the process 1100 further includes a step in which the decoder generates signals for at least two output channels based on the reconstructed vector.
Particularly, the processing circuitry 210 is configured to cause the transmitting node 200a to perform a set of operations, or steps, as disclosed above. For example, the storage medium 230 may store the set of operations, and the processing circuitry 210 may be configured to retrieve the set of operations from the storage medium 230 to cause the transmitting node 200a to perform the set of operations. The set of operations may be provided as a set of executable instructions. Thus the processing circuitry 210 is thereby arranged to execute methods as herein disclosed.
In an embodiment the transmitting node 200a for supporting generation of comfort noise for at least two audio channels at a receiving node comprises a processing circuitry 210. The processing circuitry is configured to cause the transmitting node to determine spectral characteristics of audio signals on at least two input audio channels, and to determine a spatial coherence between the audio signals on the respective input audio channels, wherein the spatial coherence is associated with perceptual importance measures. The transmitting node is further caused to divide the spatial coherence into frequency bands, wherein a compressed representation of the spatial coherence is determined per frequency band by weighting the spatial coherence within each frequency band according to the perceptual importance measures. The transmitting node is further caused to signal information about the spectral characteristics and information about the compressed representation of the spatial coherence per frequency band to the receiving node, for enabling the generation of the comfort noise for the at least two audio channels at the receiving node.
The transmitting node 200a may further be caused to encode the spatial coherence vector by forming a first prediction of the vector, a second prediction of the vector, a prediction weighting factor, and a prediction residual using the vector and the combined prediction. The transmitting node may further be caused to encode the prediction residual with a variable bit rate scheme and to reconstruct the vector based on the combined prediction and the prediction residual. The transmitting node may further be caused to transmit the encoded prediction weighting factor and the encoded prediction residual to the receiving node 200b.
The storage medium 230 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The transmitting node 200a may further comprise a communications interface 220 at least configured for communications with a receiving node 200b. As such the communications interface 220 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 210 controls the general operation of the transmitting node 200a e.g. by sending data and control signals to the communications interface 220 and the storage medium 230, by receiving data and reports from the communications interface 220, and by retrieving data and instructions from the storage medium 230. Other components, as well as the related functionality, of the transmitting node 200a are omitted in order not to obscure the concepts presented herein.
In general terms, each functional module 210a-210d may in one embodiment be implemented only in hardware and in another embodiment with the help of software, i.e., the latter embodiment having computer program instructions stored on the storage medium 230 which when run on the processing circuitry makes the transmitting node 200a perform the corresponding steps mentioned above in conjunction with
The transmitting node 200a may be provided as a standalone device or as a part of at least one further device. For example, as in the example of
Alternatively, functionality of the transmitting node 200a may be distributed between at least two devices, or nodes. These at least two nodes, or devices, may either be part of the same network part or may be spread between at least two such network parts. Thus, a first portion of the instructions performed by the transmitting node 200a may be executed in a first device, and a second portion of the of the instructions performed by the transmitting node 200a may be executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the transmitting node 200a may be executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by a transmitting node 200a residing in a cloud computational environment. Therefore, although a single processing circuitry 210 is illustrated in
The receiving node 200b comprises a decoder for reconstructing the coherence and for creating a comfort noise signal with a stereo image similar to the original sound. The decoder may further be configured to form a first prediction of the vector and a second prediction of the vector, and to obtain a prediction weighting factor. The decoder may further be configured to combine the first prediction and the second prediction using the prediction weighting factor into a combined prediction. The decoder may further be configured to reconstruct the vector based on the combined prediction and the received and decoded prediction residual.
In the example of
The proposed solution disclosed herein applies to a stereo encoder and decoder architecture or a multi-channel encoder and decoder where the channel coherence is considered in channel pairs.
Here now follows a set of example embodiments to further describe the concepts presented herein.
where L(m, k) is the spectrum of the windowed audio signal lwin(m, n), where R(m, k) is the spectrum of the windowed audio signal rwin(m, n), and where * denotes complex conjugate.
where Nband denotes total number of frequency bands, and where limit(b) denotes the lower frequency bin of frequency band b.
Generally, all terms used in the example embodiments and appended claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to “a/an/the element, apparatus, component, means, module, step, etc.” are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.
The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended list of enumerated embodiments.
This application is a continuation of U.S. patent application Ser. No. 17/045,103, filed Oct. 2, 2020, now U.S. Pat. No. 11,404,069 B2, which is a 35 U.S.C. § 371 national phase filing of International Application No. PCT/EP2019/058650, filed Apr. 5, 2019, which claims priority to U.S. Provisional Application Nos. 62/652,941, 62/652,949, and 62/653,078, filed Apr. 5, 2018, the disclosures of which are incorporated herein by reference in their entireties.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9865274 | Vicinus | Jan 2018 | B1 |
| 20030219130 | Baumgarte et al. | Nov 2003 | A1 |
| 20040002856 | Bhaskar et al. | Jan 2004 | A1 |
| 20050154584 | Jelinek et al. | Jul 2005 | A1 |
| 20140211966 | Hetherington | Jul 2014 | A1 |
| 20150181018 | Mani et al. | Jun 2015 | A1 |
| 20150249892 | Kuhnel et al. | Sep 2015 | A1 |
| 20170047072 | Eriksson | Feb 2017 | A1 |
| 20180205830 | Wells-Rutherford et al. | Jul 2018 | A1 |
| 20180240471 | Markovich Golan et al. | Aug 2018 | A1 |
| 20210090582 | Norvell et al. | Mar 2021 | A1 |
| 20210151060 | Jansson et al. | May 2021 | A1 |
| 20210158825 | Norvell et al. | May 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 1639770 | Jul 2005 | CN |
| 101809657 | Aug 2010 | CN |
| 104050969 | Sep 2014 | CN |
| 3244404 | Nov 2017 | EP |
| 2005509926 | Apr 2005 | JP |
| 20160125481 | Oct 2016 | KR |
| 2015122809 | Aug 2015 | WO |
| Entry |
|---|
| Zhu, et al., “Audio Quality Assessment Improvement via Circular and Flexible Overlap,” 2011 IEEE International Symposium on Multimedia. (Year: 2011). |
| Zhu, et al., “Audio Quality Assessment Improvement via Circular and Flexible Overlap,” 2011 IEEE International Symposium on Multimedia.—see attached reference in previous Office action. (Year: 2011). |
| Author Unknown, “Technical Specification Group Services and System Aspects; Codec for Enhanced Voice Services (EVS); Detailed Algorithmic Description (Release 13),” Technical Specification 26.445, Version 13.4.1, Mar. 2017, 3GPP Organizational Partners, 659 pages. |
| Author Unknown, “Technical Specification Group Services and System Aspects; Codec for Enhanced Voice Services (EVS); Comfort Noise Generation (CNG) Aspects (Release 12),” Technical Specification 26.449, Version 12.0.0, Sep. 2014, 3GPP Organizational Partners, 8 pages. |
| Notice of Reasons for Refusal for Japanese Patent Application No. 2022-091269, dated Apr. 11, 2023, 8 pages. |
| Written Decision on Registration for Korean Patent Application No. 10-2020-7031954, dated Apr. 5, 2023, 8 pages. |
| Written Decision on Registration for Korean Patent Application No. 10-2020-7031390, dated Feb. 2, 2023, 6 pages. |
| Zhue, et al., “Audio Quality Assessment Improvement via Circular and Flexible Overlap,” International Symposium on Multimedia, 2011, IEEE, pp. 47-52. |
| Intention to Grant for European Patent Application No. 19717450.1, dated Feb. 23, 2021, 5 pages. |
| Extended European Search Report for European Patent Application No. 21185347.8, dated Aug. 3, 2021, 6 pages. |
| Notice of Reasons for Rejection for Japanese Patent Application No. 2020-554191, dated Jan. 11, 2022, 5 pages. |
| International Search Report and Written Opinion for International Patent Application No. PCT/EP2019/058650, dated Jul. 4, 2019, 11 pages. |
| Intention to Grant for European Patent Application No. 19716874.3, dated Mar. 16, 2021, 5 pages. |
| Intention to Grant for European Patent Application No. 19716874.3, dated Aug. 4, 2021, 5 pages. |
| International Search Report and Written Opinion for International Patent Application No. PCT/EP2019/058629, dated Jul. 8, 2019, 11 pages. |
| Non-Final Office Action for U.S. Appl. No. 17/045,103, dated Sep. 22, 2021, 9 pages. |
| Notice of Allowance for U.S. Appl. No. 17/045,103, dated Feb. 18, 2022, 7 pages. |
| Non-Final Office Action for U.S. Appl. No. 17/044,740, dated Nov. 19, 2021, 11 pages. |
| Final Office Action for U.S. Appl. No. 171044,740, dated Mar. 7, 2022, 12 pages. |
| Advisory Action for U.S. Appl. No. 17/044,740, dated May 26, 2022, 3 pages. |
| Notice of Allowance for U.S. Appl. No. 17/044,740, dated Jun. 29, 2022, 7 pages. |
| First Office Action for Chinese Patent Application No. 201980034376.5, dated Sep. 13, 2023, 6 pages. |
| Notice of Reasons for Refusal for Japanese Patent Application No. 2022-091269, dated Sep. 12, 2023, 4 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20220328055 A1 | Oct 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62652949 | Apr 2018 | US | |
| 62653078 | Apr 2018 | US | |
| 62652941 | Apr 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17045103 | US | |
| Child | 17844803 | US |
