Patent Application
20080114592
Embodiments of the present invention are directed transmission of signals over a packetized network and more particularly to reconstruction of lost frames.
In digitized speech transmission through a packetized network, one often needs to consider how to handle missing packets that may be lost due to erroneous deletion or overloaded network. Missing packets may cause discontinuities in the synthesized speech and under-run of the output speech buffer, which, in turn may cause a popping noise and/or distorted sound.
It is within this context that embodiments of the present invention arise.
The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:
Although the following detailed description contains many specific details for the purposes of illustration, anyone of ordinary skill in the art will appreciate that many variations and alterations to the following details are within the scope of the invention. Accordingly, examples of embodiments of the invention described below are set forth without any loss of generality to, and without imposing limitations upon, the claimed invention.
A method of low complexity and no delay reconstruction of missing packets is proposed for Linear Predictive Coding (LPC) based Speech decoder. An algorithm for implementing such a method may be adaptive to the number of consecutive lost frames. Embodiments of the method use mathematical extrapolation based on previous good or reconstructed frames to re-generate the base of the lost frames. The adaptation of different schemes in generating the missing frame may be based on the characteristics of the speech status at lost condition. This method differentiates from the prior art in a number of ways. First, this method can rely solely on a previous frame or frames, instead of both previous and future frames as in most prior art. Such implementations introduce no delay to the system. Second, by adapting the incoming order of the lost frame and the characteristics of LPC coder, the proposed method may reconstruct the lost frame(s) in a very low complexity, thus offering continuity and significant improvement of the synthesis speech quality when packet losses are encountered in the network.
Missing packets in real-time speech communication system may cause discontinuities or gaps in synthesized speech. If an audio frame is dropped during a relatively silent period, the ill effect is mostly likely unnoticeable by human ear. However, if the dropped frame is a voice frame, it may cause significant degradation of speech quality since a sharp edge in the resulting waveform may be created when an output audio buffer is exhausted due to deficiency of speech packets.
Linear predictive coding (LPC) is a tool used mostly in audio signal processing and speech processing for representing the spectral envelope of a digital signal of speech in compressed form, using the information of a linear predictive model. A speech encoder may receive an analog signal from a transducer such as a microphone. The analog signal may be converted to a digital signal. Alternatively, the encoder may generate the digital signal may be based on a software model of the speech to be synthesized. The digital signal may be encoded to compress it for storage and/or transmission. The encoding process may involve breaking down the signal in the time domain into a series of frames. Frames are sometimes referred to herein as packets, particularly in the context of data transmitted over a network. Each frame may last a few milliseconds, e.g., 10 to 15 milliseconds. Each frame may further divided up into a number of sub-frames, e.g., 4 to 10 sub-frames. Within each sub-frame may be several individual samples of the analog signal. There may be on the order of a hundred samples in a frame, e.g., 160 to 240 samples. To aid in compression, the digital signal may be encoded as an excitation value for each sample and a set of linear prediction coefficients. Each sub-frame may have its own set of linear prediction coefficients, e.g., about 4 to 10 LPC coefficients per sub-frame. The LPC coefficients are related to the peaks in the frequency domain signal for that particular sub-frame. The LPC coefficients may mathematically model or characterize a source of sound such as a vocal tract. The excitation values may model the sound generating impulse(s) applied to the sound source.
By way of example, some audio coding schemes, e.g., Code Excited Linear Prediction (CELP) and its variants, utilize Analysis-by-Synthesis (AbS), which means that the encoding (analysis) is performed by perceptually optimizing the decoded (synthesis) signal in a closed loop.
In order to achieve real-time encoding using limited computing resources, a CELP search for an optimum combination may be broken down into smaller, more manageable, sequential searches using a simple perceptual weighting function. Typically, the encoding may be performed in the following order:
LPC coefficients may be computed and quantized, e.g., as Line Spectral Pairs (LSPs). An adaptive (pitch) codebook is searched and its contribution removed. A fixed (innovation) codebook may then be searched and its contribution to the LPC coefficients may be determined. A decoder may produce the excitation from the encoded digital signal by summing contributions from the adaptive codebook and fixed codebook:
e[n]=e
a
[n]+e
f
[n]
where ea[n] is the adaptive (pitch) codebook contribution and ef[n] is the fixed (innovation) codebook contribution. The codebooks may be implemented in software, hardware or firmware.
In CELP decoding, the filter that shapes the excitation has an all-pole (infinite impulse-response) model of the form 1/A(z), where A(z) is called the prediction filter and is obtained using linear prediction (e.g., the Levinson-Durbin algorithm). An all-pole filter is used because it is a good representation of the human vocal tract and because it is easy to compute.
The process of decoding the compressed digital signal involves applying the excitation to the LPC coefficients to produce a digital signal representing the synthesized speech. This typically involves taking a weighted average that uses weights based on the LPC coefficients.
Synthesis of a final signal for conversion to analog and presentation by a transducer, e.g., a speaker, may involve a smoothing step. For example, a synthesized frame may be generated from the last half of one frame and the first half of the next frame. The LPC coefficients applied to each sub-frame of the synthesized frame may be determined based on weighted averages of the sub-frames that make up the synthesized frame. Generally, the LPC coefficients for a particular sub-frame are given greater weight. Weights LPC coefficients for the other sub-frames may decrease with distance in time from the particular sub-frame. It is noted that the same type of smoothing process may be applied by the encoder before the compressed digital signal is stored or transmitted.
According to an embodiment of the invention, a method 300 for lost frame reconstruction may proceed as illustrated in
In the analysis and categorization stage, one or more previous good frames are taken into account to categorize the current speech status as indicated at 302. According to one embodiment, among others, there may be four mutually exclusive categories of frames; namely, voice, unvoiced, high-to-low energy transition, low-to-high energy transition. Examples of waveforms corresponding to each of these categories are illustrated in
Once the previous good or reconstructed frame has been categorized a percentage factor may be associated with the lost frame based on the determined categorization. By way of example, and without loss of generality, percentage factors, P1, P2, P3, and P4, may be respectively assigned to the voice, unvoiced, high-to-low and low-to-high categories, as indicated at 304. By way of example, and without loss of generality, the percentage may increase when the subscript increases, which can be expressed mathematically as: P1<(P2, P3)<P4. Note that in this particular example P2 may be greater than P3 or vice versa. The percentage factors may be adaptively generated by a formula that takes into account sound characteristic statistics from previous frames, the incoming order of the missing packets and also subjective based on processed speech statistics. The formula used to generate the percentages may be adjusted based on a listener's experience with sound quality of speech synthesized with lost frame reconstruction using the algorithm.
Once a percentage has been associated with the lost frame, the frame reconstruction stage may proceed. By way of example, raw excitation samples may be generated based on the parameters of the last received frame (or last reconstructed frame) as indicated at 306. Based on the categorization determined for the lost frame, the raw excitation signal from the previous good frame or recovered frame may be manipulated to produce a reconstruction excitation signal as indicated at 308. For example, if the lost frame is classified as “voiced”, P1 percent of the raw excitation samples with highest magnitudes are zeroed out. By way of example, if there are 100 samples in a frame and P1=10%, the first though tenth highest magnitude excitation samples are set equal to zero (or some other suitable low value magnitude). Alternatively, if the classification is “unvoiced”, P2 percent of the raw excitation samples with highest magnitudes are zeroed out. Similarly, if the lost frame is classified as “high-to-low energy transition”, P3 percent of the raw excitation samples with highest magnitudes are zeroed out. Furthermore, if the lost frame is classified as “low-to-high energy transition”, P4 percent of the raw excitation samples with highest magnitudes are zeroed out.
The LPC coefficients for the previous received good frame (or previous reconstructed frame) are then applied to a LPC filter used to generate the reconstructed frame as indicated at 310. The reconstructed frame may be generated by applying the reconstruction excitation to the LPC filter. It is noted that samples in the reconstruction excitation that were set equal to zero during the reconstruction at 308 do not necessarily lead to zero-valued samples in the reconstructed frame due to the weighted averaging used to generate the reconstructed frame. If an adaptive codebook is being used, the adaptive codebook may be updated with the new excitation.
If two or more frames in a row were dropped the, the earliest dropped frame may be reconstructed from the immediately preceding good frame, as described above. The next dropped frame may then be reconstructed from the previous reconstructed frame using the algorithm described above. The percentages P1, P2, P3, P4 may be adaptively adjusted to avoid over-attenuating subsequent reconstructed frames. The percentages may decrease with each frame that must be recovered from a reconstructed frame.
It is noted that the algorithm may be implemented to recover lost frames on either the encoder side or the decoder side. In particular, the algorithm may be applied to audio frames lost after generation of a plurality of audio frames on an encoder side or to lost audio frames after receiving a plurality of audio frames on the decoder side.
The simplicity of the above algorithm demands a relatively small amount of computation power when implemented. On the other hand, since the reconstruction of a dropped frame depends only on previous frame, the algorithm does not introduce a delay associated with waiting for a future frame. Such extra delay might otherwise exaggerate the reduced quality associated with frame reconstruction since some amount of fidelity may be surrendered in the packet lost condition. Since the orientation and design of current linear prediction coefficient (LPC) decoders are relatively low in complexity and also low in decoder-introduced delay, the proposed algorithm reconstructs the missing speech frame with minimum effort and no extra delay introduced.
The frame reconstruction algorithm may be implemented in software or hardware or a combination of both. By way of example,
The memory 402 may be in the form of an integrated circuit, e.g., RAM, DRAM, ROM, and the like). The memory 402 may also be a main memory or a local store of a synergistic processor element of a cell processor. A computer program 403 that includes the frame reconstruction algorithm described above may be stored in the memory 402 in the form of processor readable instructions that can be executed on the processor module 401. The processor module 401 may include one or more registers 405 into which instructions from the program 403 and data 407, such as compressed audio signal input data may be loaded. The instructions of the program 403 may include the steps of the method of lost frame reconstruction, e.g., as described above with respect to
An algorithm in accordance with embodiments of the present invention has been implemented in several applications. Clear improvements of speech quality in the simulated packet lost network have been observed. At a packet loss rate of 10%, speech quality degradation is merely noticeable. When the loss rate increases to 20%, a comfortable speech is preserved without major artifacts, such as noise or popping/clicking sounds. By contrast, when the same speech passes through a simulated network without this algorithm, the speech is hardly tolerable at this loss rate.
While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications and equivalents. Therefore, the scope of the present invention should be determined not with reference to the above description but should, instead, be determined with reference to the appended claims, along with their full scope of equivalents. Any feature described herein, whether preferred or not, may be combined with any other feature described herein, whether preferred or not. In the claims that follow, the indefinite article “A” or “An” refers to a quantity of one or more of the item following the article, except where expressly stated otherwise. The appended claims are not to be interpreted as including means-plus-function limitations, unless such a limitation is explicitly recited in a given claim using the phrase “means for.”
This application claims the benefit of priority co-pending U.S. provisional application No. 60/865,111, to Eric H. Chen et al, entitled “LOW COMPLEXITY NO DELAY RECONSTRUCTION OF MISSING PACKETS FOR LPC DECODER” filed Nov. 9, 2006, the entire disclosures of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 60865111 | Nov 2006 | US |
