This application, and the innovations and related subject matter disclosed herein, (collectively referred to as the “disclosure”) generally concern digital signal processing techniques and digital signal processors (DSPs) implementing such techniques. More particularly but not exclusively, approximated convolution techniques for simulating art impulse response, and DSPs implementing such techniques, are disclosed, with partitioned singular value decomposition (PTSVD) being but one example of such techniques. As an example, such techniques can provide a binaural rendering of a single audio channel. Other embodiments of disclosed techniques can provide a plurality of output renderings of a plurality of input channels, with an audio signal being but one particular example of such input signals. Disclosed convolution techniques can enjoy improved computational efficiency and can he more suited for real-time applications than prior convolution techniques. As well, such innovative convolution techniques can provide high-quality output renderings.
The innovations disclosed herein overcome many problems in the prior art and address the aforementioned as well as other needs. In some respects, the innovations disclosed herein are directed to methods for rendering one or more responses to one or more input signals using a selected convolution technique, and more particularly, hut not exclusively, to convolution techniques involving selected approximations to system response filters. Such techniques can allow a processing system to render one or more approximated system responses in real-time. As but one particular example, disclosed signal processing techniques can so render audio through a pair of headphones as to approximate a user's perception of audio output from a plurality of loudspeakers in any of a variety of environments (e.g., movie theaters, concert halls, night clubs, etc.). Disclosed signal processing techniques can also improve convergence rates, reduce computational overhead, and/or reduce memory requirements as compared to previously proposed techniques. Such improvements, in turn, can allow digital processors and other computing environments to have relatively lower power.
One or more filters for rendering a system response to an input signal can be defined. In some instances, such an impulse response filter has a filter length greater than 104 samples.
As an example, an impulse response filter corresponding to a system's response to an input signal can be provided. The impulse response filter can be approximated as combination of a plurality of selected M input filter components, where each input filter component has a corresponding component length N, and a plurality of selected M output filter components, where each output filter component has a corresponding plurality of P output filter coefficients. Each subsequent output filter coefficient can be delayed by N samples from a previous output filter coefficient. The selected M input filter components and the selected M output filter components can define a truncated approximation of the impulse response filter corresponding to a sub-plurality of M highest-energy partitions of a gross plurality of P partitions of the impulse response filter. Art infinite-impulse response approximation to the truncated approximation of the impulse response filter can be provided.
In some instances, the impulse response filter can be approximated by identifying relative energy content among partitions of the impulse response filter by partitioning the impulse response filter and factoring the partitioned impulse response filter. For example, singular value decomposition procedure on the partitioned impulse response filter can be performed to define an N×N singular vector corresponding to the input filter components and a P×P singular vector corresponding to the output filter components. The singular vectors can be truncated. For example, the N×N singular vector cart be truncated to be of size N×M and the P×P singular vector can be truncated to be of size P×M. Stated differently, the sub-plurality of M output filters can be selected by truncating the P×P singular vector to be of size P×M.
In some instances, the input signal comprises an audio signal and the system response comprises first and second binaural output signals. As an example, the impulse response filter can include a filter from one input channel to two output channels. The plurality of M output filter components can form a first plurality of M output filter components corresponding to one of the two output channels. The impulse response filter can be further approximated by defining a second plurality of M output filters corresponding to the other of the two output channels.
As noted above, an infinite impulse response can be introduced to the approximation of the impulse response filter. For example, an infinite impulse response approximation to the input filter components and an infinite impulse response approximation to the output filter components can be introduced. An order of the infinite impulse response approximation to the input filters can differ from an order of the infinite impulse response approximation to the output filters.
As another example, a first infinite impulse response approximation to the first plurality of output filter components can be introduced. As well, a second infinite impulse response approximation to the second plurality of output filter components can be introduced, together with a third infinite impulse response approximation to the input filter components. In some instances, an order of the first infinite impulse response approximation differs from an order of the second infinite impulse response and an order of the third infinite impulse response order.
An error function can correspond to a difference between the impulse response filter and the truncated approximation of the impulse response filter. A value of the error function can correspond in part to a number of filter components in the gross plurality P of filter components and a number of filter components M in the sub-plurality of components.
During design of the input and the output filter components, a combination of N, M, an order of the infinite impulse response approximation to the input filter components, and an order of the infinite impulse response approximation to the output filter components can be selected to correspond to a selected error-minimization criterion. In context of one input channel and two output channels, a combination of N, M, an order of the first infinite impulse response approximation, an order of the second infinite impulse response approximation, and an order of the third infinite impulse response approximation can be selected to correspond to a selected error-minimization criterion.
A digital signal processor can have an input channel and an output channel. A plurality of M input filter components can correspond to the input channel. Each in the plurality of input filter components can have a corresponding length N, and the plurality of input filter components can be approximated by an IIR approximation having a corresponding input IIR order. A plurality of M output filter components can correspond to the output channel, each output filter component having a corresponding plurality of P output filter coefficients. Each subsequent output filter coefficient can be delayed by N samples from a previous output filter coefficient. Each of the plurality of output filter components can be approximated by an IIR approximation having a corresponding output IIR order. Each of the input filter components can be associated with a corresponding output filter component such that the filter components are arranged to render a system response y(n) front an input signal x(n) according to
Some digital signal processors have one output channel.
In other instances, the output channel is a first output channel r the plurality of M output filter components constitute a first plurality of M output filter components, and the IIR approximation of the first plurality of output filter components constitutes a first IIR approximation having a corresponding first output IIR order. Such a digital signal processor can also include a second output channel, a second plurality of M output filter components corresponding to the second output channel, and an IIR approximation of the second plurality of output filter components having a corresponding second output IIR order. Each of the input filter components can be associated with a corresponding output filter component in the first plurality of output filter components to render a first system response y1(n) from an input signal x(n) according to
Each of the input filters can also be associated with a corresponding output filter in the second plurality of output filter components to render a second system response y2(n) from the input signal according to
In some digital signal processors, the input IIR order differs from the output IIR order. For example, in a digital signal processor having two output channels, the input IIR order can differ from either or both of the first output IIR order and the second output IIR order.
The foregoing and other features and advantages will become more apparent from the following detailed description, which proceeds with reference to the accompanying drawings.
Unless specified otherwise, the accompanying drawings illustrate aspects of the innovative subject matter described herein. Referring to the drawings, wherein like numerals refer to like parts throughout the several views and this specification, several embodiments of presently disclosed principles are illustrated by way of example, and not by way of limitation, wherein:
The following describes various innovative principles related to signal processing by way of reference to specific examples of techniques for rendering one or more system responses to one or more input signals, and more particularly but not exclusively, to techniques for rendering such response signals in real time. Nonetheless, one or more of the disclosed principles can be incorporated in various other filters to achieve any of a variety of corresponding system characteristics. Techniques and systems described in relation to particular configurations, applications, or uses, are merely examples of techniques and systems incorporating one or more of the innovative principles disclosed herein and are used to illustrate one or more innovative aspects of the disclosed principles.
Thus, filters and systems having attributes that are different from those specific examples discussed herein can embody one or more of the innovative principles, and can be used in applications not described herein detail, for example, in “hands-free” automobile communication systems, in aviation communication systems, in conference room speaker phones, in auditorium sound systems, etc. Accordingly, such alternative embodiments also fall within the scope of this disclosure.
The description that follows describes, illustrates and exemplifies one or more particular embodiments of the present invention in accordance with its principles. This description is not provided to limit the invention to the embodiments described herein, but rather to explain and teach the principles of the invention in such a way to enable one of ordinary skill in the art to understand these principles and, with that understanding, be able to apply them to practice not only the embodiments described herein, but also other embodiments that may come to mind in accordance with these principles. The scope of the present invention is intended to cover all such embodiments that may fall within the scope of the appended claims, either literally or under the doctrine of equivalents.
This document describes systems and methods for achieving approximate convolution using partitioned truncated singular value decomposition filtering for monaural rendering and/or binaural rendering. In many signal processing applications it is necessary to perform large convolutions in real-time. For systems where an exact convolution is too complex, Applicants show herein an approximation using a partitioned truncated singular value decomposition (PTSVD) filter. In this technique, the filter is first partitioned into P segments of length N, the singular value decomposition is performed on the N×P matrix, and only the largest M singular values and associated vectors are used to reconstruct the filter. Applicants show herein an efficient real-time implementation utilizing a filter bank and tapped delay line and then further simplify the structure utilizing an IIR model. Finally, Applicants show herein an application of the method in a simulated reverberation engine and compare complexity and memory load to state of the art methods.
Traditional audio signal processing problems both in telecommunications and multimedia often rely on FIR filter models, e.g. for the room impulse response, that can be very large and, consequently, difficult to implement in practice. State of the art techniques for implementing these filters in real-time systems use the overlap-add or overlap-save methods and partitioned frequency domain convolution to reduce complexity and delay. However, frequency domain techniques are inherently block based and introduce an amount of system latency. Alternative methods have been explored in certain application domains, such as using a perceptual model to remove certain time-frequency data from processing or subband decomposition of the impulse response. For short impulse responses (on the order of a few 100 FIR coefficients), IIR methods are attractive ways for reducing complexity, but these techniques fail for longer filters.
This document describes an alternative idea, partitioned truncated singular value decomposition (PTSVD) filtering, where the impulse response is partitioned in time, factorized using the singular value decomposition (SVD), and then reconstructed using only the M singular vectors corresponding to the M largest singular values. This filtering technique was initially explored for the purpose of creating efficient versions of linear phase bandpass and lowpass FIR prototypes. The image processing community also has used the truncated SVD for 2D filter design.
In this document, Applicants provide additional analysis of the filtering technique described above by extending it to systems that are not guaranteed to be linear phase and analyzing the tradeoff in complexity, memory usage, and approximation error. In Section 3, Applicants describe a filter structure that takes advantage of the truncated SVD matrices and leads to an efficient implementation. Applicants also describe a further approximation that both reduces the memory footprint and the computational complexity using an IIR input and output filter. This filtering technique not only has the benefit of reduced memory and complexity over traditional methods, it is also delay-less since it does not require a block-based processing structure.
Let h=[h(0)h(1) . . . h(L−1)] be an impulse response of length L. We can construct a N×P matrix H by partitioning h into P partitions of length N=L/P (and zero padding h if necessary so that it is of length P×N), H can then be factored using SVD as: H=USVH, where (·)H is the conjugate transpose, U and V are the N×N and P×P singular vectors which form a basis for the factorization, and S is a N×P matrix containing the singular values along its main diagonal. The singular values are assumed to be in descending order.
Mth order approximate filter, HM, can be created by using only the M largest singular values in its reconstruction. This is done by truncating U and V to be of size N×M and P×M, respectively, and taking the M×M portion of S corresponding to the largest singular values. The approximate filter is then: HM=UMSMVHM (hereinafter “Equation 1”). The error in our Mth order approximation is given as: e(M, N)=∥H−HM∥2, where ∥·∥2 is the entry-wise l2-norm. The use of the SVD guarantees that HM is the rank-M reconstruction of H with the lowest error, e(M, N).
At this point, there are two free parameters, M and N, that determine the error in the approximation filter.
Now, the discussion will focus on how the filter HM generated in the last section can be implemented efficiently for real-time applications. First, Equation 1 is rewritten in an expanded form as Equation 2:
In Equation 2, σm are the singular values from SM and um are the length N singular vectors of UM. Recognizing that the P columns of HM are the time-partitioned version of the filter, each delayed. N samples from the last, a filter implementation of H can be written as Equation 3:
In the above equation, x(n) is the vector of the last N samples of x at time step n.
The representation in
The PTSVD filter structure discussed above, in its initial form, requires M×(N+P) multiply-add instructions per input sample and memory of size M×(N+P+L)+N. A conventional time-domain FIR filter requires L operations per input sample and 2 L variables Implementing the FIR filter with partitioned convolution (partition size=N) greatly reduces complexity: 4αlog2(2 N)+4P+1 instructions per sample and 4 PN variables for overlap-add, where α is a platform specific FFT cost.
As mentioned in Section 3, the PTSVD structure becomes very efficient when the input and output filters are modeled with an IIR approximation. This results in a structure with 2.5 M(QU+QV) instructions and 3.5 M(OU+QV) variables, where QU and QV are the IIR approximation orders of the U and V filters (direct form H transpose using second order sections is assumed). The complexity and memory of the PTSVD-IIR are shown in
Note that the graphs in
Furthermore, the PTSVD filter structure also permits other models which may achieve better performance in certain contexts, such as using frequency domain processing for the um or vm filters or using an IIR model for only one section, or using varying IIR approximation orders for each um or vm, which are not analyzed in this work.
Since the N, M, QU, and QV are all integer valued, it is possible to calculate the finite set of (ni, mi, qui, qvi) that meet a given memory and complexity requirement on a particular platform. Although the choice of error metric should be application dependent (e.g., a spectro-temporal metric for audio applications), as a simple choice, the point that results in the lowest l2 approximation error can be chosen.
As an example, the impulse response from
The above discussion describes how a conventional convolution can be approximated with a lower complexity rank-M filter that is optimal in the l2 sense. Further, an efficient filter structure for the approximation and an IIR implementation that delivers low-error results with minimal complexity and memory usage for real-time systems have been described herein.
The technique described herein can he applied in other areas, including alternative low-rank approximations (as opposed to the SVD), joint spatio-temporal filter design for spatial audio rendering and beamforming, and adaptive implementation for applications such as echo cancelation. Adaptations of the PTSVD filtering structure, such as varying IIR approximation orders for each um and vm filter, frequency domain SVD analysis, and combination with subband methods, may also be accomplished.
In conventional binaural rendering, a pair of head-related impulse responses (HRIR), measured front source direction to left and right ears, is convolved with a source Signal to create the impression of a virtual 3D sound source when played on headphones. It is well known that using HRIRs measured in a real room, which includes a natural reverberant decay, increases the externalization and realism of the simulation. However, the HRIR filter length in even a small room can be many thousands of taps leading to computational complexity issues in real world implementations. Described herein is a new technique, partitioned truncated singular value decomposition (PTSVD) filtering, for approximating the convolution by partitioning the HRIR filters in time, performing a singular value decomposition on the matrix of filter partitions, and choosing the M singular-vectors corresponding to the M largest singular values to reconstruct the HRIR filters. The following disclosure will show how this can be implemented in an efficient filter-bank type structure with M tapped delay lines for real-time application. The following disclosure will also show how improvements to the technique, such as modeling the direct path HRIR separately can lead to improved rendering at minimal computational load.
The acoustic information of a sound source's location in three-dimensional space can be simulated over headphones through the use of the left and right binaural head-related impulse responses (HRIR). These HRIRs correspond to the impulse responses from the source to the subject's left and right ears as measured in an anechoic setting. To realistically recreate the sound of a specific room or increase the amount of externalization of a source, the binaural room impulse responses (BM) are used. These responses are typically in the range of 10000 to 100000 taps long at 44 kHz sampling rate for medium sized rooms and thus are complex to implement in real-time DSP systems.
Many techniques have been proposed to reduce the complexity of the convolution operation, the most notable being the overlap-save and overlap-add methods with variable block size. These techniques reduce complexity by partitioning the input data and filter into small blocks and use the FFT to perform the convolution in the frequency domain. Yet other techniques focus on perceptual metrics to reduce complexity. This disclosure shows how a technique developed for approximating the convolution, the PTSVD, can be used to implement binaural filters efficiently.
This section recaps some of the findings from the monaural rendering discussion above, Let h=[h(0)h(1) . . . h(L−1)] be an impulse response of length L. A N×P matrix H can be constructed by partitioning h into P partitions of length N=[L/P] (and zero padding h if necessary so that it is of length P×N). H can then be factored using the singular value decomposition (SVD) as H=USVH (Equation 1 from above), where (·)H is the conjugate transpose, U and V are the N×N and P×P singular vectors, respectively, and S is a N×P matrix containing the singular values along its main diagonal in descending order. An Mth order approximate filter. HM, can be created by using only the M largest singular values in its reconstruction, resulting in Equation 2 from above:
In Equation 2, σm are the singular values from SM and um are the length N singular vectors of UM, The use of the SVD guarantees that HM is the rank-M reconstruction of H with the lowest error, e(M,N)=∥H−HM∥F, where ∥·∥F is the Frobenius norm. This technique has been explored for creating linear phase bandpass and lowpass FIR filters that approximate a prototype filter response.
The P columns of HM are the time-partitioned version of the filter, each delayed N samples from the last, thus a filter implementation of H can be written as Equation 3 from above:
In Equation 3, x(n) is the vector of the last N samples of x at time step n.
The PTSVD filter proposed in the previous section and in the monaural discussion above can he made even more efficient when used in a binaural processing application where a single channel goes into the processing structure (the signal from a specific source direction) and a pair of left and right signals exits. In this case, the input, um, filters can be shared and separate output filters, vm, can be created for the left and right channels.
This can be easily done by cascading the left and right impulse responses, hl and hr, such that h=[hl(0)hl(1) . . . hl(L−1)hr(0)r(1) . . . hr(L−1)]. Then, after performing the SVD and truncation in Equations 1 and 2, the approximate filters will be in the matrix HM as
The filtered outputs for the binaural left and right channels can then be expressed as Equations 4 and 5:
The structure of the binaural PTSVD filter is shown in
Table 1 below shows the complexity, memory usage, and system latency for the proposed binaural method, partitioned convolution using overlap-add, FIR based convolution, and a FFT based convolution. It is assumed that the overlap-add structure uses a single FFT for the input and one IFFT for each output to minimize complexity. In the IIR version of the PTSVD technique, the output filters are modeled as Qv and Qu order filters.
A room with an impulse response of 200 ms (RT60) was measured with a KEMAR using 0.5 in microphones at the entrance to the simulated ear canals.
The binaural PTSVD described herein allows designers of binaural processing systems (e.g. for gaming, training, or multimedia applications) an additional level of DSP flexibility to deal with complexity issues that arise in real-time systems. This disclosure has shown how this technique can scale from an exact convolution to a low-complexity approximation of the convolution via choice of rank, M, and partition size, N. Modeling the PTSVD filters using DR filters both reduces the system latency. complexity. and memory usage significantly. Listening tests comparing the original and approximate filters along with an optimization technique using a genetic algorithm to find the best choose of N, M, Qu, and Qv for a given application may also he shown using the above techniques.
Directions and other relative references (e.g., up, down, top, bottom, left, right, rearward, forward, etc.) may be used to facilitate discussion of the drawings and principles herein, but are not intended to be limiting. For example, certain terms may be used such as “up,” “down,”, “upper,” “lower ” “horizontal,” “vertical,” “left,” “right,” and the like. Such terms are used, where applicable, to provide some clarity of description when dealing with relative relationships, particularly with respect to the illustrated embodiments. Such terms are not, however, intended to imply absolute relationships, positions, and/or orientations. For example, with respect to an object, an “upper” surface can become a “lower” surface simply by turning the object over. Nevertheless, it is still the same surface and the object remains the same. As used herein, “and/or” means “and” or “or”, as well as “and” arid “or.” Moreover, all patent and non-patent literature cited herein is hereby incorporated by references in its entirety for all purposes.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the disclosed innovations. Those of ordinary skill in the art will appreciate that the exemplary embodiments disclosed herein can be adapted to various configurations and/or uses without departing from the disclosed principles. For example, the principles described above in connection with any particular example can be combined with the principles described in connection with another example described herein. Various modifications to those embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of this disclosure. Accordingly, this detailed description shall not be construed in a limiting sense, and following a review of this disclosure, those of ordinary skill in the art will appreciate the wide variety of filtering and computational techniques can be devised using the various concepts described herein.
Similarly, the presently claimed inventions are not intended to be limited to the embodiments shown herein, but are to be accorded the full scope consistent with the language of the claims, wherein reference to an element in the singular, such as by use of the article “a” or “an” is not intended to mean “one and only one” unless specifically so stated, but rather “one or more”. All structural and functional equivalents to the elements of the various embodiments described throughout the disclosure that are known or later come to be known to those of ordinary skill in the art are intended to be encompassed by the features described and claimed herein. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed under the provisions of 35 USC 112, sixth paragraph, unless the element is expressly recited using the phrase “means for” or “step for”.
Thus, in view of the many possible embodiments to which the disclosed principles can be applied, we reserve to the right to claim any and all combinations of features described herein, including, for example, all that comes within the scope and spirit of the foregoing description and the combinations recited in the following claims.
This application claims the benefit of and priority to U.S. Patent Application No. 61/799,977, filed Mar. 15, 2013, the contents of which patent application are hereby incorporated by reference as if recited in full herein for all purposes.
Number | Date | Country | |
---|---|---|---|
61799977 | Mar 2013 | US |