Patent Grant
7047187
The present invention relates methods and apparatus for digitally encoding and decoding audio, and more particularly to methods and apparatus for embedding error concealment data in a digitally encoded audio signal with little or no perceptually noticeable distortion, and of utilizing the error concealment data to estimate corrupt portions of the audio signal.
It is well-known that media data is, to different degrees, vulnerable to channel errors when transmitted through an imperfect communication channel. For example, chunks of data may be lost due to transmission errors. One known method used to conceal the effects of data blocks transmission errors relies upon estimating or interpolating contents of lost blocks utilizing relationships between this content and the content of neighboring blocks. However, estimation and interpolation methods do not comprehend the actual content of lost data blocks, and the effectiveness of these methods decreases as the distance between a lost block and the available neighboring blocks increases. Thus, audible artifacts can often be detected after recovery.
Reliable transmission of digital audio over packet-switched networks such as the Internet that offer no quality of service (QoS) guarantee is a challenging task. Although channel coding can be used to protect the audio from packet loss, this type of protection increases the payload and thus requires extra bandwidth to transmit the audio stream. On the other hand, known methods of error concealment extract features from the received audio for use in the recovery of lost data. Error concealment methods are attractive because perceptual audio quality is improved without the need for additional payload.
By extracting audio features from an audio stream at an encoder and transmitting these features to a decoder along with the audio stream, both the computational complexity of receivers for error concealment and inaccuracies in the extraction of enhancement features by decoders can be reduced. Such transmission methods, however, suffer from many of the same disadvantages of channel coding and may not be useful at all because the feature transmission stream similarly increases the payload. Not only does the extra payload require increased bandwidth, but the extra payload also necessarily modifies the audio format if neither a common area nor a user data area is available. Because of the required format change, ordinary decoders can no longer decode the audio stream.
One configuration of the present invention therefore provides a method for concealing errors in an audio signal. This configuration includes digitally encoding the audio signal into a plurality of audio data packets representative of the audio signal; determining a perceptually tolerable distortion limit for the audio packets; and altering a value of at least one audio packet by an amount within the perceptually tolerable distortion limit utilizing information representative of a different audio data packet.
Another configuration of the present invention provides a method for concealing errors in an audio signal. This configuration includes decoding a digitally encoded audio signal, wherein the digitally encoded audio signal includes a plurality of audio data packets representative of the audio signal, and the plurality of audio data packets includes a plurality of altered audio data packets. Each altered audio data packet includes an alteration indicative of information representative of a different audio data packet, and each alteration is limited to a predetermined perceptually tolerable distortion limit. Also included in this configuration are determining that at least one audio data packet is missing or unavailable from the digitally encoded audio signal; extracting information representative of the missing or unavailable audio data packet from an alteration of at least one different, available audio data packet; and utilizing the extracted information to estimate the missing or unavailable audio data packet.
Yet another configuration of the present invention provides an apparatus for concealing errors in an audio signal. This apparatus is configured to digitally encode the audio signal into a plurality of audio data packets representative of the audio signal; and, utilizing a determined perceptually tolerable distortion limit for the audio packets, alter a value of at least one audio packet by an amount within the perceptually tolerable distortion limit utilizing information representative of a different audio data packet.
Still another configuration of the present invention provides an apparatus for concealing errors in an audio signal. This apparatus is configured to decode a digitally encoded audio signal. The digitally encoded audio signal includes a plurality of audio data packets representative of the audio signal, and the plurality of audio data packets includes a plurality of altered audio data packets. Each of the altered audio data packets includes an alteration indicative of information representative of a different audio data packet, and each the alteration is limited to a predetermined perceptually tolerable distortion limit. The apparatus is also configured to determine when an audio data packet is missing or unavailable from the digitally encoded audio signal; extract information representative of the missing or unavailable audio data packet from an alteration of at least one different, available audio data packet; and utilize the extracted information to estimate the missing or unavailable audio data packet.
Yet another configuration of the present invention provides a machine readable medium having recorded thereon instructions configured to instruct a computer to digitally encode the audio signal into a plurality of audio data packets representative of the audio signal; and, utilizing a determined perceptually tolerable distortion limit for the audio packets, alter a value of at least one audio packet by an amount within the perceptually tolerable distortion limit utilizing information representative of a different audio data packet.
Still another configuration of the present invention provides a machine readable medium having recorded thereon instructions configured to instruct a computer to decode a digitally encoded audio signal. The digitally encoded audio signal includes a plurality of audio data packets representative of the audio signal, and the plurality of audio data packets includes a plurality of altered audio data packets. Each altered audio data packet includes an alteration indicative of information representative of a different audio data packet, and each alteration is limited to a predetermined perceptually tolerable distortion limit. The recorded instructions also include instructions to determine when at least one audio data packet is missing or unavailable from the digitally encoded audio signal; extract information representative of the missing or unavailable audio data packet from an alteration of at least one different, available audio data packet; and utilize the extracted information to estimate the missing or unavailable audio data packet.
Configurations of the present invention provide error concealment in audio files or streams in which data is missing or otherwise unavailable. In addition, the concealed data in the audio files or streams provides little or no perceptual degradation relative to an audio file or stream not having concealed data, when the audio file or stream is decoded by a decoder that does not provide error concealment.
Further areas of applicability of the present invention will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples, while indicating the preferred embodiment of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
The present invention will become more fully understood from the detailed description and the accompanying drawings, wherein:
The following description of the preferred embodiment(s) is merely exemplary in nature and is in no way intended to limit the invention, its application, or uses.
As used herein, an audio data packet is “missing or unavailable” when it is sequentially required for decoding an encoded audio signal. For example, a packet may be missing or unavailable if it is dropped or lost during transmission, delayed in transmission beyond the time at which it is needed for decoding, or corrupted. Also as used herein, the recitation of a “first” element and a “second” element, etc., does not necessarily imply, by itself, an order of time or importance of the recited elements. However, neither is such recitation intended to exclude such ordering, if required by further context.
In one configuration of the present invention, data hiding is utilized to recover missing data chunks, such as a missing packet of an audio signal. Some audio content information for each audio packet is hidden in at least one other packet of an audio data stream. When data recovery is needed, the content of a lost packet is extracted from the hidden portion of non-corrupted packets of the audio data stream. Neighborhood interpolation and/or estimation is also used, in one embodiment, to further enhance the concealment effect.
For example, in one configuration of an audio encoder 10 and referring to
Coefficients produced by filter bank 12 inside a frequency band share similar perceptual behavior. Therefore, in one configuration of the present invention, coefficients are grouped together for estimation. In one configuration and referring to
Denote the (n,i)-band as the ith band at the nth time frame. Let us assume by way of example that coefficients b[n,k] in (n,i)-band are lost, where k∈Ki, and Ki is the index set of the ith band. In one embodiment, estimator 36 estimates coefficient b[n, k] as either {circumflex over (b)}0[n,k]=0, {circumflex over (b)}1[n,k]=b[n−1,k], {circumflex over (b)}2[n,k]=b[n+1,k], or {circumflex over (b)}3[n,k]=½(b[n−1,k]+b[n+1,k]).
In one configuration of the present invention in which it has been predetermined that embedding two bits of information in a band comprising the audio data packets is within a perceptually tolerable distortion limit for the packets, and referring again to
where argminc∈{0,1,2,3} denotes the value of the index c from the set {0, 1, 2, 3} that minimizes the value of the argument, written here as
Preferably, the selected c[n,i] is not embedded into the (n,i)-band itself, because when this information is needed, the band would be lost as would c[n,i]. Instead, in one configuration, the selected index c[n,i] for the ith band at the nth time frame is split into two bits and embedded separately into two neighboring bands. Thus,
which alters a value of at least one audio packet by an amount less than the predetermined perceptually tolerable distortion limit, utilizing information representative of a different audio packet. The process is repeated so that a plurality of audio packets are altered, each utilizing information representative of a different audio packet than the one being altered.
Estimator 36 in audio decoder 30 uses the higher and the lower bit of d[n,i] to determine whether the current band i is suitable for estimating the band in the next time frame ((n+1,i)-band) and in the previous time frame ((n−1,i)-band), respectively. For example, if the (n,i)-band were lost, from the lower bit of d[n+1,i] and the higher bit of d[n−1,i], estimator 36 determines whether the current band can be estimated from any of its neighboring time frames. When the current band is estimated from both neighboring time frames, it is scaled by ½. If one of its neighboring time frames is lost, the current band is estimated from the remaining neighbor. If both neighboring time frames are lost, then estimator 36 provides the default assumption that c[n,i]=0 and the coefficients are replaced by zeros.
Although not required for practicing this invention, it is advantageous for bitstream multiplexer 18 to utilize a packing rule that is most likely to increase the effectiveness of the estimates of lost coefficients. The most effective estimates of lost coefficients are those that utilize the nearest neighbors of the lost coefficient. Thus, in one configuration of the present invention, bitstream multiplexer 18 does not pack together adjacent coefficients along both time and frequency axes. By not packing together the adjacent coefficients, this configuration avoids the loss of estimation sources when a packet is dropped, thus providing greater assurance that estimator 36 will be able to utilize nearest neighbors for estimates of lost coefficients. Also in one configuration, estimation and/or interpolation of coefficients is used for additional error control.
Fragile digital watermarking (or hereinafter, “fragile watermarking”) is commonly defined as any watermarking method that is sensitive to any modifications to an encoded data stream. For purposes herein, any watermarking method that has an embedding rate sufficiently high (e.g., 1000 bits/sec for audio) will be sufficiently sensitive to modifications in an encoded data stream to be considered “fragile.” There are two bits for each d[n,i] and one d[n,i] per band in one configuration discussed above. Thus, for a dual channel audio clip with sampling rate 44100 Hz, the embedding rate is about 44100/1024×49×2×2≈8 kbits/sec.
One type of fragile watermarking method is least bit modulation (LBM). One example of LBM is the embedding of a bit into a host signal by replacing the least significant bit of a signal sample with a corresponding embedded bit. LBM has not been found suitable for copyright protection because it can easily be removed by simple truncation. However, deliberate attacks on error concealment coding are generally not likely. Embedding rates can also be quite high. For example, a bit can be embedded into each sample of a dual channel audio signal sampled at a rate of 44100 Hz, resulting in an embedding rate up to 44100×2≈80 kbit/sec.
It is desirable to adaptively select embedding locations for LBM because different signal samples may have different susceptibilities to distortion. However, in error concealment applications, side-information that could be used by decoder 30 to identify the embedding locations is usually not transmitted, nor are decoding keys generally made available. Therefore, in one configuration, both encoder 10 (more particularly, embedding block 28) and decoder 30 (more particularly, estimator 36) utilize predefined embedding locations.
In another configuration of the present invention, a fragile watermarking method is used that does not require decoder 30 to have knowledge of exact embedding locations. For an arbitrary host signal sequence x=x1,x2, . . . ,xN, embedding block 28 of encoder 10 embeds an integer k∈[0,K] selected so that:
LBM is a special case of this configuration in which N=1 and K=2.
There is more than one possible watermarked signal containing the same embedded information. Therefore, in one configuration, encoder 10 is configured to select locations of modifications so that the watermarked signal is perceptually closest to the original signal. Satisfactory results are obtained with this encoder 10 configuration even when used in conjunction with configurations of decoder 30 that lack knowledge of the locations at which modifications have been made.
Audio encoders that utilize fragile watermarking employ embedding blocks 28 that insert the watermark data after quantization, to prevent the watermark data from being destroyed. To make it easier to embed watermark information into an AAC coded signal or an otherwise compressed signal, one configuration of the present invention embeds watermark data into quantization indices that are obtained after partial decoding. After watermarking, the modified indices are Huffman encoded by encoder 16 without modification of the original codebook.
Perceptual modeling 20 of the original audio signal is used in one configuration of the present invention to determine which indices are to be modified and how much they are to be modified. For example, assume that a particular coefficient is known to survive a distortion level of 10 units without a significant adverse effect on perceived audio quality, and that the current quantization step size of the coefficient is 2 units. Where uniform quantization is used, the corresponding index can thus be varied by 5 steps without significantly affecting the perceived quality.
In one configuration, the audio file is compressed before information is embedded using modulo watermarking. Because of the compression, perceptual model 20 is not accessible. Although it is possible to estimate model parameters from the decompressed audio, one configuration of the present invention employs a heuristic method to achieve improved accuracy without the use of perceptual model 20.
More particularly, in this configuration, precomputation block 26 computes d[n,i] which is embedded by embedding block 28 into quantization indices q[n,k] of (n,i)-band, k∈Ki, where q[n,k] is a quantized version of b[n,k]. Let l≡Σk∈K
Because the enhancement features (i.e., the d's) are independently stored, they are useful even when only a fraction of them are retrieved correctly. Thus, embedding failures can be tolerated if and when they occur.
The imposition of a lower limit Imin restrains modification of small value indices, because small value indices are more likely to have high susceptibility to distortion. In particular, in one configuration of the present invention, no distortion is imposed on zero indices.
In one configuration of the present invention, satisfactory results were obtained with Imin set to 1, but in other embodiments, Imin is a design parameter that effects a trade-off between error free distortion and error concealment. For higher values of Imin, it is more likely that the embedding of d[n,i] will fail, leaving the indices with no distortion, at a cost of less effective error concealment.
Imax in another configuration is equal to the maximum possible value available in the Huffman table minus 1 to prevent indices from being out of bound after modification. Large indices are selected for modification because they can withstand larger distortion.
In another configuration and referring to
Referring to
Although at least one configuration of the present invention embeds hidden bits into an audio signal utilizing least significant bit modulation, other data hiding methods can also be utilized, provided the data hiding bit rate is equal to or larger than one bit per band per frame.
Testing has been performed at various error rates (i.e., dropped packet rates) on music ranging from classical to rock and roll. It has been observed that the slight drop in signal to noise ratio that results from watermark embedding LSB watermark embedding is between about 0.03 dB and 0.68 dB, and is offset by a signal to noise ratio gain at packet loss ratios as low as 0.01 (i.e., one packet out of 100 lost). The signal to noise ratio gain becomes more conspicuous as the packet loss ratio rises. Furthermore, the signal to noise ratio increase of the recovered audio has been found to be higher than for other types of error control, such as silence filling in the time domain, frame repetition in the time domain, frame repetition in the frequency domain, and noise filling in the frequency domain. Moreover, the format of the digitally encoded audio data need not be altered by configurations of the present invention that alter only the values of the encoded audio data. Thus, relative to unaltered encoded audio data, little or no perceptual degradation is experienced when altered encoded audio data is decoded by an audio decoder that does not provide error concealment. More particularly, in the tested configurations, there was no perceptual degradation in the laboratory and office testing environment after the watermark was embedded in the original data stream.
The Huffman codebook utilized by coding block 16 is optimized for the AAC encoder. Because configurations of the present invention modifies indices but retain this codebook, it is expected that the size of a compressed MPEG-2 AAC audio file will increase after watermark embedding. However, because relatively few indices are changed, the increase should be small. Tests with seven different audio clips resulted in size increases of less than 0.1% in each case. On the other hand, if an 8 kbits/sec rate were used to write explicit overhead to the audio rather than to embed watermarks, the total file size would increase 8/256=3% for audio encoded at 256 kbits/sec.
Configurations of each audio encoder and audio decoder of the present invention may comprise both hardware and software (or firmware), and it is a design choice as to whether some or all of the functional blocks represented in each figure represent separate hardware components. For example, encoder 10 and decoder 30 can be implemented as special purpose signal processors. Alternately, encoder 10 can be implemented as a server computer with suitable software and signal processing hardware (e.g., an analog-to-digital converter). Also, decoder 30 can be implemented as a suitably programmed general-purpose computer equipped with an audio output device. Software comprising instructions for the computers comprising encoder 10 and/or decoder 30 to perform one or more of the method configurations described herein may be supplied on a machine-readable medium or downloaded electronically from another computer or storage device.
In one configuration 144 and referring to
In yet another configuration 154 and referring to
In another configuration similar to that shown in
Configurations of the present invention will thus be seen to provide audio data recovery by data hiding in the presence of missing blocks resulting from transmission channel errors. Because some amount of knowledge about the actual content of lost blocks is concealed within neighboring portions of the data stream, a lost packet can be acceptably recovered using hidden data concealed in the non-corrupted received data packets. Configurations of the present invention can be overlaid with other error control methods to further enhance error concealment in MPEG-2 AAC audio streams. Although configurations of the present invention are described in detail for MPEG-2 AAC audio files and streams, other configurations of the present invention can be applied to other media formats. For example, in one configuration, watermarking is used for error concealment in an original, uncompressed data stream.
The description of the invention is merely exemplary in nature and, thus, variations that do not depart from the gist of the invention are intended to be within the scope of the invention. Such variations are not to be regarded as a departure from the spirit and scope of the invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 5233629 | Paik et al. | Aug 1993 | A |
| 5943347 | Shepard | Aug 1999 | A |
| 6714683 | Tian et al. | Mar 2004 | B1 |
| 6725192 | Araki | Apr 2004 | B1 |
| 6778953 | Edler et al. | Aug 2004 | B1 |
| Number | Date | Country | |
|---|---|---|---|
| 20030163305 A1 | Aug 2003 | US |
