Patent Application
20110153335
1. Field of the Invention
The present invention relates to an audio signal processing method and apparatus that encode or decode an audio signal.
2. Discussion of the Related Art
In general, short term prediction, such as linear prediction coding (LPC), is performed on a time domain so as to compress a speech signal. Subsequently, a pitch is acquired with respect to a residual resulting from the short term prediction so as to perform long term prediction.
When long term prediction is performed with respect to a residual resulting from linear prediction coding, compressibility of a signal containing a speech component is high, but compressibility of a signal containing a non-speech component is low.
Accordingly, the present invention is directed to an audio signal processing method and apparatus that substantially obviate one or more problems due to limitations and disadvantages of the related art.
An object of the present invention is to provide an audio signal processing method and apparatus that are capable of performing long term prediction with respect to an audio signal containing a speech component and a non-speech component in a mixed state.
Another object of the present invention is to provide an audio signal processing method and apparatus that are capable of performing long term prediction with respect to an audio signal and coding a residual on a frequency domain.
Another object of the present invention is to provide an audio signal processing method and apparatus that are capable of obtaining prediction the most similar to a current frame using a preceding frame, i.e., a frame right before the current frame.
Another object of the present invention is to provide an audio signal processing method and apparatus that are capable of generating long term prediction information necessary for a decoder to perform long term synthesis using obtainable information (for example, a synthesized residual), not unobtainable information (for example, a source signal).
A further object of the present invention is to provide an audio signal processing method and apparatus that are capable of temporarily generating long term prediction information through long term prediction using a source signal and deciding final long term prediction information through long term synthesis in the vicinity thereof.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
To achieve these objects and other advantages and in accordance with the purpose of the invention, as embodied and broadly described herein, an audio signal processing method includes receiving a residual and long term prediction information, performing inverse frequency transforming with respect to the residual to generate a synthesized residual, and performing long term synthesis based on the synthesized residual and the long term prediction information to generate a synthesized audio signal of a current frame, wherein the long term prediction information comprises a final prediction gain and a final pitch lag, the final pitch lag has a range starting with 0, and the long term synthesis is performed based on a synthesized audio signal of a frame comprising a preceding frame.
In another aspect of the present invention, an audio signal processing apparatus includes an inverse transforming unit for performing inverse frequency transforming with respect to a residual to generate a synthesized residual and a long term synthesis unit for performing long term synthesis based on the synthesized residual and long term prediction information to generate a synthesized audio signal of a current frame, wherein the long term prediction information comprises a final prediction gain and a final pitch lag, the final pitch lag has a range starting with 0, and the long term synthesis is performed based on a synthesized audio signal of a frame comprising a preceding frame.
In another aspect of the present invention, an audio signal processing method includes performing long term prediction on a time domain using a source audio signal of a preceding frame to generate a temporary residual of a current frame, frequency transforming the temporary residual, inversely frequency transforming the temporary residual to generate a synthesized residual of the preceding frame, and deciding long term prediction information using the synthesized residual.
The step of deciding the long term prediction information may include performing long term synthesis using the synthesized residual to generate a synthesized audio signal of the preceding frame and deciding the long term prediction information using the synthesized audio signal.
The step of generating the temporary residual may include generating a temporary prediction gain and a temporary pitch lag, and the long term synthesis may be performed based on the temporary prediction gain and the temporary pitch lag.
The long term synthesis may be performed using one or more candidate prediction gains based on the temporary prediction gain and one or more candidate pitch lags based on the temporary pitch lag.
The long term prediction information may include a final prediction gain and a final pitch lag, and the long term prediction information may be decided based on the source audio signal.
In another aspect of the present invention, an audio signal processing apparatus includes a long term prediction unit for performing long term prediction on a time domain using a source audio signal of a preceding frame to generate a temporary residual of a current frame, a frequency transforming unit for frequency transforming the temporary residual, an inverse transforming unit for inversely frequency transforming the temporary residual to generate a synthesized residual of the preceding frame, and a prediction information decision unit for deciding long term prediction information using the synthesized residual.
The audio signal processing apparatus may further include a long term synthesis unit for performing long term synthesis using the synthesized residual to generate a synthesized audio signal of the preceding frame, wherein the prediction information decision unit may decide the long term prediction information using the synthesized audio signal.
The long term prediction unit may generate a temporary prediction gain and a temporary pitch lag, and the long term synthesis may be performed based on the temporary prediction gain and the temporary pitch lag.
The long term synthesis may be performed using one or more candidate prediction gains based on the temporary prediction gain and one or more candidate pitch lags based on the temporary pitch lag.
The long term prediction information may include a final prediction gain and a final pitch lag, and the long term prediction information may be decided based on the source audio signal.
In a further aspect of the present invention, there is provided a storage medium for storing digital audio data, the storage medium being configured to be read by a computer, wherein the digital audio data include long term flag information, a residual, and long term prediction information, the long term flag information indicates whether long term prediction has been applied to the digital audio data, the long term prediction information includes a final prediction gain and a final pitch lag generated through long term prediction and long term synthesis, and the final pitch lag has a range starting with 0.
The present invention has the following effects and advantages.
First, it is possible to perform long term prediction with respect to a speech signal and an audio signal containing a speech component and a non-speech component in a mixed state, thereby improving coding efficiency with respect to a signal that is repetitive, in particular, on a time domain.
Second, it is possible to refer to a preceding frame, i.e., a frame right before a current frame, so as to search prediction of the current frame, thereby obtaining the most similar prediction and thus reducing a bit rate of a residual.
Third, it is possible to perform long term synthesis through a decoder using obtainable information (for example, a quantized residual), not unobtainable information (for example, a source signal), thereby increasing a restoring rate of long term synthesis.
Fourth, it is possible to approximate long term prediction information (pitch lag and prediction gain) through relatively noncomplex processing and to more accurately decide prediction information within a searching range reduced based thereon, thereby reducing overall complexity.
It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principle of the invention. In the drawings:
Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. First of all, terminology used in this specification and claims must not be construed as limited to the general or dictionary meanings thereof and should be interpreted as having meanings and concepts matching the technical idea of the present invention based on the principle that an inventor is able to appropriately define the concepts of the terminologies to describe the invention in the best way possible. The embodiment disclosed herein and configurations shown in the accompanying drawings are only one preferred embodiment and do not represent the full technical scope of the present invention. Therefore, it is to be understood that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents when this application was filed.
According to the present invention, terminology used in this specification can be construed as the following meanings and concepts matching the technical idea of the present invention. Specifically, ‘coding’ can be construed as ‘encoding’ or ‘decoding’ selectively and ‘information’ as used herein includes values, parameters, coefficients, elements and the like, and meaning thereof can be construed as different occasionally, by which the present invention is not limited.
In this disclosure, in a broad sense, an audio signal is conceptionally discriminated from a video signal and designates all kinds of signals that can be perceived by a human. In a narrow sense, the audio signal means a signal having none or small quantity of speech characteristics. “Audio signal” as used herein should be construed in a broad sense. Yet, the audio signal of the present invention can be understood as an audio signal in a narrow sense in case of being used as discriminated from a speech signal.
Meanwhile, a frame indicates a unit used to encode or decode an audio signal, and is not limited in terms of sampling rate or time.
An audio signal processing method according to the present invention may be a long term encoding/decoding method, and an audio signal processing apparatus according to the present invention may be a the long term coding (encoding and/or decoding) device encoding/decoding apparatus. In addition, the audio signal processing method according to the present invention may be an audio signal encoding/decoding method to which the long term encoding/decoding method is applied, and the audio signal processing apparatus according to the present invention may be an audio signal encoding/decoding apparatus to which the long term encoding/decoding apparatus is applied. Hereinafter, a long term encoding/decoding apparatus will be described, and a long term encoding/decoding method performed by the long term encoding/decoding apparatus will be described. Subsequently, an audio signal encoding/decoding apparatus and method, to which the long term encoding/decoding apparatus and method are applied, will be described.
Referring first to
Referring to
Meanwhile, at Step S110, long term prediction is approximate to the multiplication of a signal preceding a signal at a given point of time n by a pitch lag and a prediction gain, which may be defined as represented by the following mathematical expression.
rt(n)=St(n)−b·St(n−d) [Mathematical expression 1]
Where, St(n) indicates a signal of the current frame, b indicates a prediction gain, d indicates a pitch lag, and rt(n) indicates a residual.
Since the prediction gain b and the pitch lag d at Step S110 are not final but are updated at a subsequent step, the prediction gain and the pitch lag at Step S110 may be referred to as a temporary prediction gain and a temporary pitch lag. On the other hand, a temporary residual rt(n) is not recalculated as a final prediction gain and a final pitch lag. However, a transformed (aliased) residual may be generated through frequency transforming, or a synthesized residual may be generated through long term synthesis.
At Step S110, a source signal, not a synthesized signal, is used so as to acquire a prediction similar to the current frame, and therefore, the preceding frame t−1 may be included in a search range of the prediction. This is because it is possible to use a source signal of the preceding frame t−1 without change. Also, Step S110 may be referred to as an open loop scheme or long term prediction.
Meanwhile, the following table shows an example of a mean square error (MSE), a pitch lag, a prediction gain, an output, and a search range when an open loop is performed.
The temporary prediction gain may be generated using the scheme indicated in Table 1, and the temporary pitch lag may be generated using the scheme indicated in Table 1. Also, Mathematical expression 1 is equal to the output indicated in Table 1.
Hereinafter, Step S110 will be described in terms of a buffer. Referring to
Referring back to
Hereinafter, Step S120 and Step S130 will be described in terms of a buffer. Referring to
Meanwhile, the psychoacoustic model 230 applies a masking effect to the input audio signal to generate a masking threshold. The masking effect is based on psychoacoustic theory. Auditory masking is explained by psychoacoustic theory. The masking effect uses properties of the psychoacoustic theory in that low volume signals adjacent to high volume signals are overwhelmed by the high volume signals, thereby preventing a listener from hearing the low volume signals.
The quantization unit 220 quantizes the frequency transformed residual signal based on the masking threshold (S120). The quantized residual signal {circumflex over (R)}t,t−1(ω) may be input to the inverse transforming unit 120 or may be an output of the long term encoding device. In the latter case, the quantized residual signal may be transmitted to a long term decoding device through a bit stream.
The inverse transforming unit 120 performs de-quantization and inverse frequency transforming (or frequency to time transforming) with respect to the frequency transformed residual to generate a synthesized residual {circumflex over (r)}t−1(n) of the preceding frame (S130). Here, the frequency to time transforming may be performed based on inverse quadrature mirror filterbank (IQMF) or inverse modified discrete Fourier transform (IMDCT), to which, however, the present invention is not limited.
Referring back to
The long term synthesis unit 130 decides a candidate prediction gain bc and a candidate pitch lag dc based on the temporary prediction gain b and the temporary pitch lag d generated by the long term prediction unit 110 (S140). For example, the candidate prediction gain and the candidate pitch lag may be decided within a range defined as represented by the following mathematical expression.
bc=b±α
dc=d±β [Mathematical expression 2]
Where, α and β indicate arbitrary constants.
The candidate prediction gain is a group consisting of one or more prediction gains, and the candidate pitch lag is a group consisting of one or more pitch lags. The search range is reduced based on the temporary prediction gain and the temporary pitch lag.
The long term synthesis unit 130 performs long term synthesis based on the candidate prediction gain be and the candidate pitch lag dc decided at Step S140 and the residual {circumflex over (r)}t−1(n) of the preceding frame generated at Step S130 to generate a synthesized audio signal Ŝt−1(n) of the preceding frame (S150).
Ŝt−1_(n)={circumflex over (r)}t−1(n)+bc·Ŝt−1(n−dc) [Mathematical expression 3]
Where, Ŝt−1(n) indicates a synthesized audio signal of the preceding frame with respect to the candidate prediction gain and the candidate pitch lag, {circumflex over (r)}t−1(n) indicates a synthesized residual of the preceding frame, and Ŝt−1(n) indicates a synthesized audio signal of the preceding frame.
Meanwhile, the following table shows an example of a mean square error (MSE), a pitch lag, a prediction gain, an output, and a search range when a closed loop is performed.
Also, Mathematical expression 4 may be equal to the output indicated in Table 2. Meanwhile, the search range is not decided as indicated in Table 2. The search range is decided according to the candidate prediction gain and the candidate pitch lag based on the temporary prediction gain and the temporary pitch lag at Step S110.
Meanwhile, the delay unit 150 delays a source signal St(n) with respect to the current frame to input a source signal St−1(n) of the preceding frame to the prediction information decision unit 140 upon processing the next frame.
The prediction information decision unit 140 compares the source signal St−1(n) of the preceding frame received from the delay unit 150 with the synthesized audio signal Ŝt−1_c(n) of the preceding frame generated at Step S150 to decide the most appropriate long term prediction information, i.e., the final prediction gain b0 and the final pitch lag d0 (S160). At this time, final prediction gain and the final pitch lag may be decided as represented by the following mathematical expression.
{bo, do}=are sum {St−1(n)−Ŝt−1—c(n)} [Mathematical expression 4]
Where, St−1(n) indicates a source signal of the preceding frame, Ŝt−1—c(n) indicates a synthesized audio signal of the preceding frame with respect to the candidate prediction gain and the candidate pitch lag, b0 indicates a final prediction gain, and d0 indicates a final pitch lag.
Mathematical expression 4 may be based on the mean square error (MSE) indicated in Table 2.
The final prediction gain and the final pitch lag generated at Step S160 result from searching of a signal the most similar to the current frame in frames including the (t−1)-th frame (i.e., the preceding frame) based on information that can be acquired by the decoder. Since the final pitch lag results from searching of a signal the most similar to the current frame in frames including the preceding frame in addition to the (t−1)-th frame, a final pitch lag range starts with 0, not N (frame length). If the final pitch lag range starts with N, only the remaining values excluding N can be transmitted. On the other hand, if the final pitch lag range starts with 0, all of the values can be transmitted.
Meanwhile, the prediction information decision unit may further generate long term flag information, which indicates whether long term prediction(or synthesis) has been applied, in addition to the final prediction gain and the final pitch lag.
First, the de-quantization unit 310 extracts a residual {circumflex over (R)}t,t−1(ω) from a bit stream and de-quantizes the extracted residual {circumflex over (R)}t,t−1(ω). Here, the residual may be a frequency transformed residual or an aliased residual as previously described.
Subsequently, the inverse transforming unit 320 performs inverse frequency transforming (or frequency to time transforming) with respect to the frequency transformed residual {circumflex over (R)}t,t−1(ω) to generate a residual {circumflex over (r)}t(n) of the current frame. Here, the frequency to time transforming may be performed based on inverse quadrature mirror interbank (IQMF) or inverse modified discrete Fourier transform (IMDCT), to which, however, the present invention is not limited.
The acquired residual {circumflex over (r)}t(n) may be the synthesized residual {circumflex over (r)}t(n) generated by the long term encoding device based on the aliased residuals. A de-quantization process and a long term synthesis process of the long term decoding device are shown in
Referring back to
Ŝt(n)={circumflex over (r)}t(n)+b0·Ŝt(n−d0) [Mathematical expression 6]
Where, {circumflex over (r)}t(n) indicates a (synthesized) residual, b0 indicates a final prediction gain, d0 indicates a final pitch lag, and Ŝt(n) indicates a synthesized audio signal of the current frame.
This long term synthesis process is similar to the process performed by the long term synthesis unit 130 of the long term encoding device previously described with reference to
The long term decoding device restores an audio signal of the current frame using the long term prediction information and the audio signal of the preceding frame through the above process.
The multi-channel encoder 410 receives a plurality of channel signals (two or more channel signals) (hereinafter, referred to as a multi-channel signal), performs downmixing to generate a mono downmixed signal or a stereo downmixed signal, and generates space information necessary to upmix the downmixed signal into a multi-channel signal. Here, space information may include channel level difference information, inter-channel correlation information, a channel prediction coefficient, downmix gain information, and the like. If the audio signal encoding device 400 receives a mono signal, the multi-channel encoder 410 may bypass the mono signal without downmixing the mono signal.
The band extension encoder 420 may generate band extension information to restore data of a downmixed signal excluding spectral data of a partial band (for example, a high frequency band) of the downmixed signal.
The long term encoding unit 430 performs long term prediction with respect to an input signal to generate long term prediction information b0 and d0. Meanwhile, the component 200 (the frequency transforming unit 210, the quantization unit 220, and the psychoacoustic model 230) previously described with reference to
The audio signal encoder 440 encodes a downmixed signal using an audio coding scheme when a specific frame or segment of the downmixed signal has a high audio property. Here, the audio coding scheme may be based on an advanced audio coding (AAC) standard or a high efficiency advanced audio coding (HE-AAC) standard, to which, however, the present invention is not limited. Meanwhile, the audio signal encoder 440 may be a modified discrete transform (MDCT) encoder.
Meanwhile, the audio signal encoder 440 may include the frequency transforming unit 210, the quantization unit 220, and the psychoacoustic model 230 previously described with reference to
The speech signal encoder 450 encodes a downmixed signal using a speech coding scheme when a specific frame or segment of the downmixed signal has a high speech property. Here, the speech coding scheme may be based on an adaptive multi-rate wide band (AMR-WB) standard, to which, however, the present invention is not limited. Meanwhile, the speech signal encoder 450 may also use a linear prediction coding (LPC) scheme. When a harmonic signal has high redundancy on the time axis, the harmonic signal may be modeled through linear prediction which predicts a current signal from a previous signal. In this case, the LPC scheme may be adopted to improve coding efficiency. Meanwhile, the speech signal encoder 450 may be a time domain encoder.
The multiplexer 460 multiplexes space information, band extension information, long term prediction information, and spectral data to generate an audio signal bit stream.
The demultiplexer 510 multiplexes spectral data, band extension information, long term prediction information, and space information from an audio signal bit stream.
The audio signal decoder 520 decodes spectral data corresponding to a downmixed signal using an audio coding scheme when the spectral data has a high audio property. Here, the audio coding scheme may be based on an AAC standard or an HE-AAC standard, as previously described. Meanwhile, the audio signal decoder 520 may include the de-quantization unit 310 and the inverse transforming unit 320 previously described with reference to
The speech signal decoder 530 decodes a downmixed signal using a speech coding scheme when the spectral data has a high speech property. Here, the speech coding scheme may be based on an AMR-WB standard, as previously described, to which, however, the present invention is not limited.
The long term decoding unit 540 performs long term synthesis using the long term prediction information and the (inversely transformed) residual signal to restore a synthesized audio signal. The long term decoding unit 540 may include the long term synthesis unit 330 previously described with reference to
The band extension decoder 550 decodes a bit stream of band extension information and generates an audio signal (or spectral data) of a different band (for example, a high frequency band) from some or all of the audio signal (or the spectral data) using this information.
When the decoded audio signal is downmixed, the multi-channel decoder 560 generates an output channel signal of a multi-channel signal (including a stereo channel signal) using space information.
The long term encoding device or the long term decoding device according to the present invention may be included in a variety of products, which may be divided into a standalone group and a portable group. The standalone group may include televisions (TV), monitors, and settop boxes, and the portable group may include portable media players (PMP), mobile phones, and navigation devices.
Referring first to
A user authentication unit 620 receives user information to authenticate a user. The user authentication unit 620 may include at least one selected from a group consisting of a fingerprint recognition unit 620A, an iris recognition unit 620B, a face recognition unit 620C, and a speech recognition unit 620D. The fingerprint recognition unit 620A, the iris recognition unit 620B, the face recognition unit 620C, and the speech recognition unit 620D receive fingerprint information, iris information, face profile information, and speech information, respectively, convert the received information into user information, and determine whether the user information coincides with registered user data to authenticate the user.
An input unit 630 allows a user to input various kinds of commands. The input unit 630 may include at least one selected from a group consisting of a keypad 630A, a touchpad 630B, and a remote control 630C, to which, however, the present invention is not limited. A signal coding unit 640 includes a long term coding device (a long term encoding device and/or a long term decoding device) 645. The long term encoding device 645 includes at least the long term prediction unit, the inverse transforming unit, the long term synthesis unit, and the prediction information decision unit of the long term encoding device previously described with reference to
The signal coding unit 640 encodes an input signal through quantization to generate a bit stream or decodes the signal using the received bit stream and spectral data to generate an output signal.
A controller 650 receives input signals from input devices and controls all processes of the signal coding unit 640 and an output unit 660. The output unit 660 outputs an output signal generated by the signal coding unit 640. The output unit 660 may include a speaker 660A and a display 660B. When an output signal is an audio signal, the output signal is output to the speaker. When an output signal is a video signal, the output signal is output to the display.
The audio signal processing method according to the present invention may be modified as a program which can be executed by a computer. The program may be stored in a recording medium which can be read by the computer. Also, multimedia data having a data structure according to the present invention may be stored in a recording medium which can be read by the computer. The recording medium which can be read by the computer includes all kinds of devices that store data which can be read by the computer. Examples of the recoding medium which can be read by the computer may include a read only memory (ROM), a random access memory (RAM), a compact disc ROM (CD-ROM), a magnetic tape, a floppy disc, and an optical data storage device. In addition, a recoding medium employing a carrier wave (for example, transmission over the Internet) format may be further included. Also, a bit stream generated by the encoding method as described above may be stored in a recording medium which can be read by a computer or may be transmitted using a wired or wireless communication network.
It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the inventions. Thus, it is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.
The present invention is applicable to encoding and decoding of an audio signal.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2009-00044623 | May 2009 | KR | national |
This application is the National Phase of PCT/KR2009/002743 filed on May 25, 2009, which claims priority under 35 U.S.C. to 119(e) to U.S. Provisional Application Nos. 61/055,465 filed on May 23, 2008 and 61/078,774 filed on Jul. 8, 2008 and under 35 U.S.C. 119(a) to Patent Application No. 10-2009-0044623 filed in the Republic of Korea on May 21, 2009, the entire contents of which are hereby expressly incorporated by reference into the present application.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/KR09/02743 | 5/25/2009 | WO | 00 | 11/23/2010 |
| Number | Date | Country | |
|---|---|---|---|
| 61055465 | May 2008 | US | |
| 61078774 | Jul 2008 | US |
