Coding of audio signals for data reduction is a ubiquitous technology. High-quality, low-bitrate coding is essential for enabling cost-effective media storage and for facilitating distribution over constrained channels (such as Internet streaming). The efficiency of the compression is vital to these applications since the capacity requirements for uncompressed audio may be prohibitive in many scenarios.
Several existing audio coding approaches are based on sliding-window time-frequency transforms. Such transforms convert a time-domain audio signal into a time-frequency representation which is amenable to leveraging psychoacoustic principles to achieve data reduction while limiting the introduction of audible artifacts. In particular, the modified discrete cosine transform (MDCT) is commonly used in audio coders since the sliding-window MDCT has properties that are inherently favorable for audio coding applications; in particularly, the sliding-window MDCT may be configured to achieve perfect reconstruction with overlapping nonrectangular windows without oversampling in the time-frequency representation.
While the time-frequency representation of an audio signal derived by a sliding-window MDCT provides an effective framework for audio coding, it is beneficial for coding performance to extend the framework such that the time-frequency resolution of the representation can be adapted based on the characteristics of the signal to be coded. For instance, such adaptation can be used to limit the audibility of coding artifacts. Several existing audio coders adapt to the signal to be coded by changing the window used in the sliding-window MDCT in response to the signal behavior. For tonal signal content, long windows are used to provide high frequency resolution; for transient signal content, short windows are used to provide high time resolution. This approach is commonly referred to as window switching.
Window switching approaches typically provide for short windows, long windows, and transition windows for switching from long to short and vice versa. It is common practice to switch to short windows based on a transient detection process. If a transient is detected in a portion of the audio signal to be coded, that portion of the audio signal is processed using short windows. It may be beneficial for coding performance, however, to allow for additional intermediate window sizes between short windows and long windows.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
In one aspect, an audio signal encoding method is provided. First and second audio signal frames are received in succession. A scaling windowing rotation is imparted to portions of the first and second audio signal frames to scale the second portion of the first audio signal frame and the first portion of the second audio signal frame based upon the second portion of the first window and the first portion of the second window. The result of the intermediate scaling windowing rotation is transformed to a first set of spectral coefficients that corresponds to the second portion of the first audio signal frame and a second set of spectral coefficients that corresponds to the first portion of the second audio signal frame.
In another aspect, an audio signal decoding method is provided. A set of spectral coefficients is received that corresponds to a second portion of a first audio signal frame and a first portion of a second audio signal frame. The set of spectral coefficients is transformed to a set of time-domain signal values. A scaling windowing rotation is imparted to the set of time-domain signal values, based upon a second portion of a first scaling window and a first portion of a second scaling window, to produce a second time-domain signal portion corresponding to a second portion of the first audio signal frame and a first time-domain signal portion corresponding to a first portion of the second audio signal frame.
In yet another aspect, an audio signal encoder is provided. A framer circuit is configured to receive an audio signal and to produce a corresponding succession of audio signal frames including a first frame followed next by a second frame. A windowing circuit is configured to impart a scaling windowing rotation to portions of the first and second audio signal frames to scale the second portion of the first audio signal frame and the first portion of the second audio signal frame based upon the second portion of the first window and the first portion of the second window, to produce an intermediate result. A transform circuit configured to transform the intermediate result to a first set of spectral coefficients that corresponds to the second portion of the first audio signal frame and a second set of spectral coefficients that corresponds to the first portion of the second audio signal frame.
In still another aspect, an audio decoder is provided.
A transform circuit is configured to transform the first and second sets of spectral coefficients to a set of time-domain signal values. A windowing circuit is configured to impart a scaling windowing rotation to the set of time-domain signal values, based upon a second portion of the first window and a first portion of the second window, to produce a second time-domain signal portion corresponding to a second portion of the first audio signal frame and a first time-domain signal portion corresponding to a first portion of the second audio signal frame.
Referring now to the drawings in which like reference numbers represent corresponding parts throughout:
In the following description of embodiments of an audio codec and method reference is made to the accompanying drawings. These drawings shown by way of illustration specific examples of how embodiments of the audio codec and method may be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the claimed subject matter.
Efficient audio coding requires the use of an appropriate signal-adaptive time-frequency resolution for representing the audio signal to be coded. Audio coders typically use long and short windows for tonal and transient content, respectively to achieve signal-adaptive time-frequency resolution. Efficient approaches to achieving signal-adaptive time-frequency resolution may use a multiplicity of different window lengths instead of only long and short windows.
An audio signal 101 denoted with time line 102 may represent an excerpt of a longer audio signal or stream, which may be a representation of time-varying physical sound features. The audio signal is segmented into frames 120-128 for processing as indicated by the frame boundaries 103-109. A windowing block (not shown) multiplicatively applies the sequence of windows 111, 113, and 115 to the audio signal to produce windowed signal segments for further processing. The windows are time-aligned with the audio signal in accordance with the frame boundaries. For example, window 113 is time-aligned with the audio signal 101 such that the window 113 is centered on the frame 124 having frame boundaries 105 and 107.
The audio signal 101 may be denoted as a sequence of discrete-time samples x[t] where t is an integer time index. A windowing block audio signal value scaling function, as for example depicted by 111, may be denoted as w[n] where n is an integer time index. The windowing block scaling function may be defined in one embodiment as
for 0≤n≤N−1 where N is an integer value representing the window time length. In another embodiment, a window may be defined as
Other embodiments may perform other windowing scaling functions provided that the windowing function satisfies the certain conditions explained in section 4C of the Appendix. A windowed segment may be defined as
xi[n]=wi[n]x[n+ti]
where i denotes an index for the windowed segment, wi[n] denotes the windowing function used for the segment, and ti denotes a starting time index in the audio signal for the segment. In some embodiments, the windowing scaling function may be different for different segments. In other words, different windowing time lengths and different windowing scaling functions may be used for different parts of the signal 101, for example for different frames of the signal or in some cases for different portions of the same frame.
The windowing block 407 applies windowing functions to the audio frames to produce windowed audio segments and provides the windowed audio segments to the transform block 409. Analysis results provided by the control block 405 to the transform block 409 may include transform sizes for the transform block 409 to use in processing the windowed audio segments. In some embodiments, the transform sizes provided by the control block 405 to the transform block 409 may correspond to the window sizes provided by the control block 405 to the windowing block 407. As will be understood by those of ordinary skill in the art, the output of the transform block 409 and results provided by the control block 405 may be processed by a data reduction and formatting block 411 to generate a data bitstream 413 which represents the input audio signal. In some embodiments, the data reduction and formatting may include the application of a psychoacoustic model and information coding principles as will be understood by those of ordinary skill in the art. The audio encoder 400 may provide the data bitstream 413 as an output for storage or transmission to a decoder (not shown) as explained below. The transform block 409 may be configured to carry out a MDCT, which may be defined mathematically as:
where
and where the values xi[n] are windowed time samples, i.e. time samples of a windowed audio segment. The values Xi[k] may be referred to generally as transform coefficients or specifically as modified discrete cosine transform (MDCT) coefficients. In accordance with the definition, the MDCT converts N time samples into
transform coefficients. For the purposes of this specification, the MDCT as defined above is considered to be of size N. Conversely, an inverse modified discrete cosine transform (IMDCT), which may be performed by a decoder (not shown), may be defined mathematically as:
where 0≤n≤0.5 N−1. As those of ordinary skill in the art will understand, a scale factor may be associated with one or both of the MDCT or the IMDCT. Additional mathematical details about the scale factor in the MDCT and IMDCT are provided in the Appendix in section 2F.
In typical embodiments, a transform operation such as an MDCT is carried out by transform block 409 for each windowed segment of the input signal 401. This sequence of transform operations converts the time-domain signal 401 into a time-frequency representation comprising MDCT coefficients corresponding to each windowed segment. The time and frequency resolution of the time-frequency representation are determined at least in part by the time length of the windowed segment and the size of the associated transform carried out on the windowed segment. In some embodiment, transform size corresponds to window length. For a windowed segment corresponding to a long time-length window, for example, the resulting time-frequency representation has low time resolution but high frequency resolution. For a windowed segment corresponding to a short time-length window, for example, the resulting time-frequency representation has relatively higher time resolution but lower frequency resolution than a time-frequency representation corresponding to a long-window segment. In some cases, a frame of the signal 401 may be associated with more than one windowed segment, as illustrated by the example frame 307 of
Each coder pipeline is aligned with a window center. A coder pipeline 560-1, for example, is configured to generate coding representing window 501, which entirely spans frame 532 and which spans portions of frames 530 and 532. A coder pipeline 560-3, for example, is configured to generate coding representing window 503, which entirely spans frame 534 and which spans portions of frames 532 and 536. A coder pipeline 560-5 is configured to generate coding representing window 505, which entirely spans frame 536 and which spans portions of frames 534 and 538. It will be appreciated that although the illustrative sequence of windows 501, 503, 505 and 507 shown in
It will be understood that the encoder 509 and the decoder 513 may operate independently. Transform coefficients produced by the encoder 509 may be transmitted over a communication system to the decoder 513. A signal processing circuit (such as the machine described with reference to
Consider for example the processing of the frame 534, which window 503 is centered upon. The encoder 509 includes windowing block 570 configured to apply the window 503 to audio samples within frame 534 and to audio samples within portions of the preceding frame 532 and portions of the subsequent frame 536, which are partially encompassed by window 503, to generate a windowed segment. The encoder includes an MDCT block 572 configured to produce transform coefficients 574-2 corresponding to time window 503 based upon the windowed segment produced by the windowing block 503. The decoder 513 includes an Inverse MDCT (IMDCT) block 576 to produce IMDCT output time-domain signal values based upon the transform coefficients 574-2 and includes a windowing block 578 configured to apply the window 503 to the IMDCT output time-domain signal values to produce a windowed result. In some embodiments, the window 578 applied after the IMDCT is the same as the window 572 applied before the MDCT; in other embodiments, they may be different. The decoder 513 includes an overlap-add block 580 configured to overlap and add the windowed result for frame 534 with a windowed result from a previous frame 532 associated with a previous window 501. The result of the overlap-add with the previous frame is that an indicated output segment 523 is constructed. Those of ordinary skill in the art will understand that overlap-add with the previous frame 532 (associated with window 501) obtains the benefit of time-domain aliasing cancellation in generating output samples for the segment 523 associated in part with the time window 503.
MDCT Coder with Windowing Operations Aligned to Window Transitions
Each frame time boundary is aligned with an instance of the coder pipeline. Each coder pipeline instance is configured to generate coding results based upon adjacent half-frames and corresponding adjacent half-windows. More specifically, a first transform block 672-1, which is part of a first coder pipeline 660-1, is aligned with frame 632; a second transform block 672-2 which is part of a second coder pipeline 660-3, is aligned with frame 634; and a third transform block 672-3 which is part of a third coder pipeline 660-5, is aligned with frame 636. coder pipeline 660-1 is centered on frame time boundary 644 and is configured to generate coding results based in part upon a second-half windowed segment generated using second-half of window 601, second half of frame 632, and first half of frame 634, and a first-half windowed segment generated using first-half of window 603, first half of frame 634, and second half of frame 632. A coder pipeline 660-3 is centered on frame time boundary 646 and is configured to generate coding results based in part upon a second-half windowed segment generated using second-half of window 603, second half of frame 634, and first half of frame 636, and a first-half windowed segment generated using first-half of window 605, first half of frame 636, and second half of frame 634. A coder pipeline 660-5 is centered on frame time boundary 648 and is configured to generate coding results based in part upon a second-half windowed segment generated using second-half of window 605, second half of frame 636, and first half of frame 638, and a first-half windowed segment generated using first-half of window 607, first half of frame 638, and second half of frame 636.
It will be understood that the encoder 609 and the decoder 613 may operate independently. Coefficients produced by the encoder 609 may be transmitted over a communication system to the decoder 613. A signal processing circuit (such as the machine described with reference to
Consider for example the first pipeline instance 660-1 processing of the second portion of window 601 and second portion of frame 632 together with the first-portion of window 603 and first portion of frame 634. The encoder 609 includes windowing block 670 configured to impart a windowing rotation operation on the second portion of frame 632 and the first portion of frame 634 based upon the second portion of window 601 and the first portion of window 603 so as to produce an windowing rotation result. More particularly, in some embodiments, a scaled portion of an input signal may be combined with a corresponding portion of an input signal scaled and folded about a frame time boundary to form an intermediate time-aliased time-domain signal. The frame time boundary acts as a fixed point about which the rotation occurs. In some embodiments, the scaling, folding, and combination may be realized via a windowing rotation operation. Moreover, in accordance with some embodiments, a vector of samples corresponding to a concatenation of the adjacent half-frames, a second half of frame 632 and first half of frame 634, is processed by a rotation matrix that corresponds in part to the windowing operations as explained in Appendix A. In particular, sections 3 and 4A of the Appendix provide a mathematical formulation of the windowing operation of the encoder 609 as a rotation matrix. In particular, Eq. (27) provides a mathematical formulation of the scaling, folding, and combination operations which may be realized via a windowing rotation operation. In particular, the submatrix in the box Eq. (26) is the “windowing rotation” of the
A transform block 672 accepts as input an intermediate encoding result corresponding to a concatenation of portions of the windowing rotation results from adjacent transitions. The transform block then transforms the concatenated windowing rotation result (the intermediate encoding result) by a discrete cosine transform (DCT) to produce DCT transform coefficients 674, also referred to as spectral coefficients. In some embodiments, a variation of a DCT known as a DCT-IV is used, which is known to those of ordinary skill in the art. The DCT transform coefficients likewise correspond to MDCT transform coefficients as explained mathematically in the Appendix. In the decoder section 613, an inverse DCT-IV transform may be performed in block 676 to produce an intermediate decoding result, which is subsequently provided in part to adjacent windowing rotation blocks 678. A windowing rotation block 678 may receive concatenated intermediate decoding results from adjacent inverse DCT blocks. The windowing rotation block 678 may processes the intermediate decoding results to perform windowing and overlap-add functions to generate output samples representing the second half of frame 632 and the first half of frame 634 for the output portion indicated by 621.
At operation 708, an intermediate encoding result may be formed by combining a portion of the windowing rotation for the transition from frame i-1 to frame i with a portion of the windowing rotation for the preceding window transition, for instance that from frame i-2 to i-1. In some cases, the preceding window transition may correspond to a window contained entirely within frame i-1. In some cases, finalizing the intermediate encoding results from frame i-1 in step 708 includes aggregating intermediate encoding results formed in step 717 for the previous iteration of the process. At operation 709, transform block 409/672 applies a DCT transform respectively to one or more intermediate encoding results for frame Fi-1. At operation 711, the data for frame i-1 may be finalized, which may include providing the data to data reduction and bitstream formatting block 411. At operation 713, windowing block 407/670 determines subsequent windowing rotations for frame Fi if the control block 405 has indicated that multiple windows should be applied for frame Fi. For example, if frame Fi is a long-window frame, operation 713 does not involve any further processing. If frame Fi is a short-window frame, for example, operation 713 determines windowing rotations for the short windows that are contained within the frame boundaries of frame Fi. At operation 715, the windowing rotations for any windows that are contained within the frame boundaries of frame Fi are applied. At operation 717, the results of the windowing rotations for windows within the frame boundaries of frame Fi are combined to form intermediated encoding results. The operation 717 may pertains to shorter windows. At operation 719, the frame counter is incremented and the process repeats for the next frame starting at operation 701.
It will be appreciated that the windowing block 670 and transform block 672 may start processing for a first half of a frame before the subsequent frame has been received. Moreover, part-way through processing of a frame, the control block 405 may determine that a different window shape should be used for a latter half of the frame than for the former half of the frame, and may instruct the windowing block 670 to associate a different window function with a second half of the frame. For example, the control block 405 may determine that audio data for a subsequent frame has a different characteristic requiring a different window and may in accordance with that determination, indicate that a latter half of a frame be processed with a different window shape in preparation for a transition to the window determined for the subsequent frame. For example, shorter windows are used for transients. In response to a determination by the control block 405 that a subsequent frame contains transients, a transition window may be applied to the latter half of the current frame. Thus, less lookahead for window planning is required since a window can be changed during the processing of a frame. As a result, lower signal processing latency can be achieved since processing of a frame can start before the subsequent frame is received. In contrast, in the first encoder 509 and first decoder 513 of
In the embodiment of
Similarly, in the embodiment of
When there are multiple windows within a frame, the first window straddles the beginning-of-frame boundary. Pipeline processing of successive windows may continue through the frame, from one window to the next, with a selected window and MDCT transform size. The windowing rotation of the pipeline processing is centered on the transitions between successive windows. The processing pipeline runs for each window transition within the frame. The last window in the frame straddles the end-of-frame boundary.
In some embodiments, all frames may be spanned by a number of windows that corresponds to a power of two. In some embodiments, each window may be processed by an MDCT that generates half as many spectral coefficients as the window length. Thus, each frame may be represented by the same number of spectral coefficients independent of the window size used to process the frame. In an example embodiment, the frame size N may be 1024. For a long-window frame, a single long window may span 2048 time-domain samples from the respective long-window frame and the adjacent frames; the corresponding MDCT may generate 1024 spectral coefficients. For a short-window frame, a short window may span 256 time-domain samples such that 8 overlapping short windows span the frame. An MDCT corresponding to a short window may generate 128 spectral coefficients. Considering that the short-window frame may consist of 8 windows, for each of which a corresponding MDCT may generate 128 spectral coefficients, a total of 1024 spectral coefficients may be generated for a short-window frame. In this example and in other examples that appropriately constructed, the number of spectral coefficients generated may be equivalent for a long-window frame and for frames consisting of multiple windows. In some embodiments, this regularity may be advantageous in that the bitstream formatting (such as that carried out in block 411) may be consistent for all frames independent of the window size used to process the frame. In prior approaches to window switching, the data rate may be significantly different for different window types. In some embodiments, the regularity may be advantageous in that the data reduction steps (such as those carried out in block 411) may employ commonly structured processes such as vector quantizers independent of the window size used to process the frame.
In some embodiments, using window sizes that are related by powers of two may improve the efficiency of the implementation of the associated transforms in that transforms with sizes related by powers of two may share common processing components. In some embodiments, a start window such as window 901 may be used to process the beginning of a signal. In some embodiments, a stop window such as 913 may be used to process the end of a signal. In some embodiments, a stop window and a start window may be used in succession at an intermediate point in a signal to form a sync point, such as that shown in
The window 1015 may be constructed as follows. Segment 1011 may consist of
zeros. Segment 1013 may be of length M0 and may comprise the left half (rising portion) of a length 2M0 window designed in accordance with the constraints discussed in Appendix A in section 4D. Segment 1015 may consist of
ones. Segment 1017 may be of length M2 and may comprise the right half (falling portion) of a length 2M2 window designed in accordance with the constraints discussed in section 4D of Appendix A. Segment 1019 may consist of
zeros.
The example described in conjunction with the depiction of
zeros, a second segment comprising the left half (rising portion) of a length 2m0 window designed in accordance with the constraints discussed in Appendix A, a third segment consisting of
ones, a fourth segment 917 comprising the right half (falling portion) of a length 2M1 window designed in accordance with the constraints discussed in Appendix A, and a fifth segment consisting of
zeros. In some embodiments, a last window for a frame associated with more than one window may be constructed in five segments in the following way, where m2 is defined as the minimum of M1 and M2 and M is defined as the maximum of m2 and M1: a first segment consisting of
zeros, a second segment comprising the left half (rising portion) of a length 2M1 window designed in accordance with the constraints discussed in section 4D of Appendix A, a third segment consisting of
ones, a fourth segment 917 comprising the right half (falling portion) of a length 2m2 window designed in accordance with the constraints discussed in Appendix A, and a fifth segment consisting of
zeros. In some embodiments, an intermediate window for a frame associated with more than one window may be constructed as a length 2M1 window designed in accordance with the constraints discussed in section 4D of Appendix A.
An MDCT embodiment such as that of
While embodiments of the invention have been discussed in the context of audio signals, those of ordinary skill in the art will understand that it is within the scope of the invention to operate on different classes of signals other than audio signals.
A mathematical framework for efficient implementation of
This section describes a general mathematical framework for implementation of the modified discrete cosine transform in audio coders or other signal processing applications. The implementation features:
2A. DCT-IV
The discrete cosine transform (DCT) of order N is the N×N matrix DAT defined by
with 0≤k, n<N, where k is the row index of the matrix and n is the column index. There are other variations of the DCT defined in the literature; this variation is referred to as the DCT-IV.
The DCT-IV as defined above satisfies the orthogonality condition
where IN denotes the N×N identity matrix and the superscript T denotes a matrix transpose. Note that a scale factor of
in the definition of the DCT-IV would lead to the condition DNTDN=IN. Such scaling issues will be addressed in a later section.
2B. MDCT
The modified discrete cosine transform (MDCT) of order N is the N×2N matrix CN defined by
with 0≤k<N and 0≤n<2N. This is also referred to as the forward MDCT.
2C. Relationship Between the MDCT and the DCT-IV
For N a power of 2, N>1, the MDCT and DCT-IV of order N are related by
where J denotes the anti-identity matrix. The sub-matrices in the block matrix on the right are each of size
the size subscripts have been omitted from the equation for simplicity.
2D. Inverse MDCT
The inverse MDCT is the 2N×N matrix given by the transpose of the MDCT. This can be written using the block matrix formulation as
noting that JNT=JN.
2E. Time-Domain Aliasing
The product of the inverse MDCT and the forward MDCT yields the 2N×2N matrix
The off-diagonal elements in the sub-matrices (the JN terms) correspond to time-domain aliasing incurred in the MDCT.
2F. Normalization
To normalize the result of applying the forward and inverse MDCT successively, the forward and inverse MDCT are each scaled by a factor
Furthermore, for the sake of computational efficiency, the forward MDCT and inverse MDCT may be realized in a codec as the negatives of the definitions given above. The MDCT as implemented in an encoder may be given by
and the inverse MDCT as implemented in the decoder may likewise be given by
with 0≤k<N and 0≤n<2N.
In matrix notation, the scaled forward and inverse transforms are
and the cascade of the forward and inverse transform yields
In typical processing scenarios, a transform is applied to a segment of a signal extracted and weighted by a window function:
where the window function wi[n] may vary as a function of the frame index i and where Hk,n are the kernel functions of the forward transform when viewed as functions of n.
where W1 is a 2N×2N diagonal matrix with the length-2N window w1[n] on the diagonal. As depicted in
sub-windows w1A, w1B, w1C and w1D corresponding to the four sub-frames spanned by the full window. Defining
diagonal matrices W1A, W1B, W1C and W1D similarly as W1 (with the corresponding sub-window on the diagonal), the windowing and MDCT operation from frame 1 can be further expressed as
Using the above formulation of the MDCT, this can be rewritten as
The expression in Eq. (22) refactors the MDCT into a sequence of matrix operations. The vector on the rightmost side of Eq. (22) is a vector of signal values. A first matrix operation applied to the vector of signal values in the sequence of matrix operations corresponds at least in part to a windowing operation. A second matrix operation, which is applied to the result of the first matrix operation, corresponds at least in part to a DCT transform. Scaling and permutation operations may be incorporated in either the first or the second matrix operation. The result of the second matrix operation corresponds to the MDCT of the vector of signal values. The formulation in Eq. (22) represents the computation of the MDCT of a single vector or sequence of signal values. In Eqs. (20)-(22), the sliding-window MDCT is constructed by considering the matrix processing of a succession of signal vectors. This construction indicates that the sliding-window MDCT can be formulated as windowing rotation operations centered at frame boundaries (for the case of long windows) followed by transform operations corresponding at least in part to a DCT transform. For the case of shorter windows, the windowing rotation operation may be centered more generally at window transition boundaries.
In signal processing applications such as audio coding, signals of arbitrary length are often processed via a sliding-window transform in which the window and transform operation is applied successively at regular intervals throughout the signal.
4A. Mathematical Formulation of the Forward MDCT
Using Eq. (18) as a basis, the sliding-window MDCT can be written in block diagonal form as
where D and W are defined as block-diagonal matrices; note that the block-diagonal sub-matrices operate on successive frames of the input signal:
The construction of W can be understood by considering the correspondence of the boxed elements above to the single-frame formulation of the MDCT in Eq. (22). The boxed sub-matrix is applied to the frame 1 signal [x02x11x12x22]T′ as part of the computation of the transform X1. Likewise, the subsequent two rows are applied to the frame 2 signal [x12 x21 x22 x31]T′ as part of the computation of the transform X2. Note that the non-zero elements in these two rows have the same form as the boxed sub-matrix, but are respectively shifted over by N columns; this corresponds to an inter-frame hop size of N for the sliding-window transform.
Note that the windowing block matrix in Eq. (25) has a block-diagonal structure. The diagonal of the matrix consists of windowing rotation submatrices that are centered on frame boundaries as indicated by the boxed element in Eq. (26) below:
The boxed element depicted in Eq. (22) corresponds to a windowing rotation operation carried out on the second half of frame 1 and the first half of frame 2 as expressed below:
where the matrix consists of zeros except for the diagonal (from the top left to the bottom right of the matrix) and the anti-diagonal (from the top right to the bottom left). The diagonal operates to scale the data in the vector; the anti-diagonal operates to negate the first half of the data in the vector, scale the vector, and flip the data around the center point to create a time-aliased version of the data. In full, the matrix thus operates to combine a scaled version of the data vector with a scaled and time-aliased version.
The structures of the matrices in Eqs. (24)-(27) gives rise to the pipeline in
4B. The Inverse MDCT and Perfect Reconstruction
The block-diagonal formulation of the sliding-window MDCT can be used to establish conditions to achieve perfect reconstruction via a frame-by-frame inverse MDCT. Constraining the overall forward block-diagonal transformation to be orthogonal, the cascade of the forward and inverse transforms can be expressed as
where the property DHD=I has been incorporated. Perfect reconstruction is thus achieved if WHW=I, which requires that the N×N sub-matrices on the diagonal satisfy an orthonormality condition. Consider the highlighted square submatrix in the windowing matrix Win Eq. (26). From this, the required orthogonality condition for the window is:
where the condition as stated is for the overlap of frames 1 and 2. If the corresponding condition holds for all of the sub-matrices on the diagonal (i.e., all of the window overlap regions), then perfect reconstruction is achieved for the full sliding-window transform.
4C. Window Conditions for Perfect Reconstruction
The orthogonality condition in Eq. (29) can be equivalently written as
from which four sub-matrix conditions can be derived:
Note that the latter two are related by a transpose and are thus equivalent conditions. Recalling that the W sub-matrices are diagonal, the three sub-matrix conditions for perfect reconstruction can be equivalently expressed as time-domain conditions for the window segments wA[n], wB[n],wC[n], and wD[n]:
where
Using Eqs. (35) and (36), Eq. (37) can be rewritten as follows:
From Eqs. (35), (36), (40) and (41), the following relationships can be derived:
Denoting the concatenation of the window segments wA[n] and wB[n] as wAB[n] (and similarly for wCD[n]), the requirements on the window can be simplified to
for 0≤n≤N−1. The first constraint means that the latter half of a frame's window must be power-complementary with its reverse (about its midpoint). The second constraint means that the reverse of the latter half of a frame's window must match the first half of the subsequent frame's window. Given those constraints, it follows that the first half of a frame's window must also be power-complementary with its reverse. Note however that all of the constraints apply to half of the window, meaning that the second half of a window can be designed independently from the first half, a property which will become important in the context of window switching.
If a symmetric window is assumed such that
for 0≤n≤N−1, meaning that the relationship
holds for 0≤n≤N−1 for the window halves wAB[n] and wCD[n], then Eq. (45) can be simplified to w1AB[n]2=w2AB[n]2 for 0≤n≤N−1 and Eq. (44) can be rewritten as
or, in terms of the full window w[n.]:
which is referred to as the Princen-Bradley condition [1].
In this section, two different approaches for implementing the sliding-window or running MDCT (and inverse MDCT) are reviewed.
5A. Realization Using Sliding Windows
A typical implementation of the running MDCT consists of the following steps
Running MDCTs are commonly implemented based on these steps or relatively minor variants. This series of operations is depicted in
5B. Realization Using Orthogonal Transforms
Referring again to
When the windows used in the sliding-window MDCT satisfy the perfect reconstruction conditions derived above, the N×N sub-matrices on the diagonal of W are orthogonal matrices. The sliding-window MDCT and inverse MDCT can thus be implemented in this way:
points of one frame and the initial
points of the next. This can be interpreted as the N-point overlap between successive length-2N windows applied to the respective frames.
This series of operations is depicted in
In audio coders, it is beneficial to adapt the window and transform based on the time-frequency behavior of the audio signal. For tonal signals, the use of long windows improves coding efficiency; on the other hand, the use of short windows for transient signals limits coding artifacts.
In order to preserve perfect reconstruction, an asymmetric transition window is needed to adjust the window size.
The machine 1400 can comprise, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system or system component, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, a headphone driver, or any machine capable of executing the instructions 1416, sequentially or otherwise, that specify actions to be taken by the machine 1400. Further, while only a single machine 1400 is illustrated, the term “machine” shall also be taken to include a collection of machines 1400 that individually or jointly execute the instructions 1416 to perform any one or more of the methodologies discussed herein.
The machine 1400 can include or use processors 1410, such as including an audio processor circuit, non-transitory memory/storage 1430, and I/O components 1450, which can be configured to communicate with each other such as via a bus 1402. In an example embodiment, the processors 1410 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a circuit such as a processor 1412 and a processor 1414 that may execute the instructions 1416. The term “processor” is intended to include a multi-core processor 1412, 1414 that can comprise two or more independent processors 1412, 1414 (sometimes referred to as “cores”) that may execute the instructions 1416 contemporaneously. Although
The memory/storage 1430 can include a memory 1432, such as a main memory circuit, or other memory storage circuit, and a storage unit 1136, both accessible to the processors 1410 such as via the bus 1402. The storage unit 1436 and memory 1432 store the instructions 1416 embodying any one or more of the methodologies or functions described herein. The instructions 1416 may also reside, completely or partially, within the memory 1432, within the storage unit 1436, within at least one of the processors 1410 (e.g., within the cache memory of processor 1412, 1414), or any suitable combination thereof, during execution thereof by the machine 1400. Accordingly, the memory 1432, the storage unit 1436, and the memory of the processors 1410 are examples of machine-readable media.
As used herein, “machine-readable medium” means a device able to store the instructions 1416 and data temporarily or permanently and may include, but not be limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., erasable programmable read-only memory (EEPROM)), and/or any suitable combination thereof. The term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store the instructions 1416. The term “machine-readable medium” shall also be taken to include any medium, or combination of multiple media, that is capable of storing instructions (e.g., instructions 1416) for execution by a machine (e.g., machine 1400), such that the instructions 1416, when executed by one or more processors of the machine 1400 (e.g., processors 1410), cause the machine 1400 to perform any one or more of the methodologies described herein. Accordingly, a “machine-readable medium” refers to a single storage apparatus or device, as well as “cloud-based” storage systems or storage networks that include multiple storage apparatus or devices. The term “machine-readable medium” excludes signals per se.
The I/O components 1450 may include a variety of components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I/O components 1450 that are included in a particular machine 1400 will depend on the type of machine 1100. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will likely not include such a touch input device. It will be appreciated that the I/O components 1450 may include many other components that are not shown in
In further example embodiments, the I/O components 1450 can include biometric components 1456, motion components 1458, environmental components 1460, or position components 1462, among a wide array of other components. For example, the biometric components 1456 can include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram based identification), and the like, such as can influence a inclusion, use, or selection of a listener-specific or environment-specific impulse response or HRTF, for example. In an example, the biometric components 1156 can include one or more sensors configured to sense or provide information about a detected location of the listener in an environment. The motion components 1458 can include acceleration sensor components (e.g., accelerometer), gravitation sensor components, rotation sensor components (e.g., gyroscope), and so forth, such as can be used to track changes in the location of the listener. The environmental components 1460 can include, for example, illumination sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect reverberation decay times, such as for one or more frequencies or frequency bands), proximity sensor or room volume sensing components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components 1462 can include location sensor components (e.g., a Global Position System (GPS) receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude may be derived), orientation sensor components (e.g., magnetometers), and the like.
Communication can be implemented using a wide variety of technologies. The I/O components 1450 can include communication components 1464 operable to couple the machine 1400 to a network 1480 or devices 1470 via a coupling 1482 and a coupling 1472 respectively. For example, the communication components 1464 can include a network interface component or other suitable device to interface with the network 1480. In further examples, the communication components 1464 can include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components to provide communication via other modalities. The devices 1470 can be another machine or any of a wide variety of peripheral devices (e.g., a peripheral device coupled via a USB).
Moreover, the communication components 1464 can detect identifiers or include components operable to detect identifiers. For example, the communication components 1464 can include radio frequency identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar code, multi-dimensional bar codes such as Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF49, Ultra Code, UCC RSS-2D bar code, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). In addition, a variety of information can be derived via the communication components 1064, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that may indicate a particular location, and so forth. Such identifiers can be used to determine information about one or more of a reference or local impulse response, reference or local environment characteristic, or a listener-specific characteristic.
In various example embodiments, one or more portions of the network 1480 can be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, the network 1480 or a portion of the network 1080 can include a wireless or cellular network and the coupling 1082 may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling 1482 can implement any of a variety of types of data transfer technology, such as Single Carrier Radio Transmission Technology (1×RTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, others defined by various standard-setting organizations, other long range protocols, or other data transfer technology. In an example, such a wireless communication protocol or network can be configured to transmit headphone audio signals from a centralized processor or machine to a headphone device in use by a listener.
The instructions 1416 can be transmitted or received over the network 1480 using a transmission medium via a network interface device (e.g., a network interface component included in the communication components 1064) and using any one of a number of well-known transfer protocols (e.g., hypertext transfer protocol (HTTP)). Similarly, the instructions 1416 can be transmitted or received using a transmission medium via the coupling 1472 (e.g., a peer-to-peer coupling) to the devices 1470. The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding, or carrying the instructions 1416 for execution by the machine 1400, and includes digital or analog communications signals or other intangible media to facilitate communication of such software.
While the above detailed description has shown, described, and pointed out novel features as applied to various embodiments, it will be understood that various omissions, substitutions, and changes in the form and details of the devices or algorithms illustrated can be made without departing from the scope of the disclosure. As will be recognized, certain embodiments of the inventions described herein can be embodied within a form that does not provide all of the features and benefits set forth herein, as some features can be used or practiced separately from others.
Moreover, although the subject matter has been described in language specific to structural features and methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.
This application is a Continuation of U.S. patent application Ser. No. 15/967,248, filed Apr. 30, 2018, which claims benefit of priority to U.S. Provisional Patent Application No. 62/492,011, filed on Apr. 28, 2017, which is incorporated by reference herein in its entirety.
Number | Name | Date | Kind |
---|---|---|---|
5179626 | Thomson | Jan 1993 | A |
6226608 | Fielder et al. | May 2001 | B1 |
7308401 | Tsushima et al. | Dec 2007 | B2 |
7512539 | Geiger et al. | Mar 2009 | B2 |
7876966 | Ojanpera | Jan 2011 | B2 |
7974837 | Tanaka | Jul 2011 | B2 |
8126951 | Huang et al. | Feb 2012 | B2 |
8548815 | Chivukula et al. | Oct 2013 | B2 |
8682645 | Taleb et al. | Mar 2014 | B2 |
8706508 | Suzuki et al. | Apr 2014 | B2 |
10847169 | Goodwin et al. | Nov 2020 | B2 |
20040184537 | Geiger et al. | Sep 2004 | A1 |
20040220805 | Geiger et al. | Nov 2004 | A1 |
20050083216 | Li | Apr 2005 | A1 |
20060115171 | Geiger et al. | Jun 2006 | A1 |
20060161427 | Ojala | Jul 2006 | A1 |
20060210180 | Geiger et al. | Sep 2006 | A1 |
20060253209 | Hersbach et al. | Nov 2006 | A1 |
20070174063 | Mehrotra et al. | Jul 2007 | A1 |
20070194952 | Breebaart et al. | Aug 2007 | A1 |
20080319739 | Mehrotra et al. | Dec 2008 | A1 |
20090048852 | Burns et al. | Feb 2009 | A1 |
20090083045 | Briand et al. | Mar 2009 | A1 |
20090094038 | Chivukula et al. | Apr 2009 | A1 |
20090240491 | Reznik | Sep 2009 | A1 |
20090299754 | Mehrotra | Dec 2009 | A1 |
20100309983 | Jax | Dec 2010 | A1 |
20110060433 | Dai et al. | Mar 2011 | A1 |
20110238426 | Fuchs et al. | Sep 2011 | A1 |
20110246139 | Kishi et al. | Oct 2011 | A1 |
20120232909 | Terriberry et al. | Sep 2012 | A1 |
20120278088 | Villemoes | Nov 2012 | A1 |
20150154971 | Boehm et al. | Jun 2015 | A1 |
20150170670 | Luyten | Jun 2015 | A1 |
20150215574 | Hinds | Jul 2015 | A1 |
20150332683 | Kim et al. | Nov 2015 | A1 |
20160019913 | Sugiyama et al. | Jan 2016 | A1 |
20160295128 | Schnittman et al. | Oct 2016 | A1 |
20160372128 | Baeckstroem et al. | Dec 2016 | A1 |
20180315435 | Goodwin et al. | Nov 2018 | A1 |
20190005969 | Dick et al. | Jan 2019 | A1 |
20200066287 | Davis | Feb 2020 | A1 |
20210005209 | Beack | Jan 2021 | A1 |
Number | Date | Country |
---|---|---|
WO-2018201113 | Nov 2018 | WO |
Entry |
---|
“U.S. Appl. No. 15/967,248, 312 Amendment filed Oct. 6, 2020”, 7 pgs. |
“U.S. Appl. No. 15/967,248, Non Final Office Action dated Mar. 18, 2020”, 14 pgs. |
“U.S. Appl. No. 15/967,248, Notice of Allowance dated Jul. 6, 2020”, 15 pgs. |
“U.S. Appl. No. 15/967,248, PTO Response to Rule 312 Communication dated Oct. 27, 2020”, 2 pgs. |
“U.S. Appl. No. 15/967,248, Response filed Jun. 18, 2020 to Non Final Office Action dated Mar. 18, 2020”, 8 pgs. |
“European Application Serial No. 18790505.4, Extended European Search Report dated Dec. 22, 2020”, 14 pgs. |
“International Application Serial No. PCT/US2018/030061, International Preliminary Report on Patentability dated Nov. 7, 2019”, 10 pgs. |
“International Application Serial No. PCT/US2018/030061, International Search Report dated Aug. 31, 2018”, 4 pgs. |
“International Application Serial No. PCT/US2018/030061, Invitation to Pay Additional Fees dated Jun. 15, 2018”, 4 pgs. |
“International Application Serial No. PCT/US2018/030061, Written Opinion dated Aug. 31, 2018”, 8 pgs. |
Britanak, V, et al., “A Survey of Efficient MDCT Implementations in MP3 audio coding standard: Retrospective and state-of-the-art”, Signal Processing 91(4), (2011), 624-672. |
Geiger, et al., “Audio Coding based on Integer Transforms”, AES Convention Paper No. 5471, [Online]. Retrieved from the Internet: <https//www.iis.fraunhofer.delcontentfdam/iis/de/doc/ame/conference/AES-111-Convention_Audio_CodingbasedonIntegerTransforms_AES5471.pdf>, (Nov. 30, 2001), 1-9. |
Geiger, Ralf, “Audio Coding Based on Integer Transforms (Dissertation)”, Technische Universitat I I menau, Germany, [Online]. Retrieved from the Internet: <URL: http://www.db-thueringen.de/servlets/DerivateSerlet/Derivate-13817/ilml-2007000278.pdf>, (2007), 132 pgs. |
Geiger, Ralf, “Audio Coding Based on Integer Transforms”, Dissertation zur Erlangung des akademischen Grades Doktor Technischen Universitat Ilmenau, pp. 1-99, [Online]. Retrieved from the Internet: <https//www.db-thueringeri.de/servlets/MCRFileNodeServlet/dbt_derivale_00013817/iln 1-2007000278.pdf>, (Nov. 2, 2007), 132 pgs. |
Helmrich, Christian, et al., “Low-Delay Transform Coding Using the MPEG-H 3D Audio Codec”, AES Convention 139. Paper 9355, (2015), 7 pgs. |
Princen, J P, et al., “Subband/Transform Coding Using Filter Bank Designs Based on Time Domain Aliasing Cancellation”, IEEE Proc. Intl. Conference on Acoustics, Speech, and Signal Processing (ICASSP), (1987), 2161-2164. |
Virette, David, “Low Delay Transform for High Quality Low Delay Audio Coding”, Signal and Image Processing. Universite de Rennes 1, [Online]. Retrieved from the Internet: <URL: https://hal.inria.fr/tel-01205574/document>, (2012), 197 pgs. |
Number | Date | Country | |
---|---|---|---|
20210134309 A1 | May 2021 | US |
Number | Date | Country | |
---|---|---|---|
62492011 | Apr 2017 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 15967248 | Apr 2018 | US |
Child | 17097690 | US |