Patent Application
20120101826
1. Field
This disclosure relates to audio signal processing.
2. Background
Many music applications on portable devices (e.g., smartphones, netbooks, laptops, tablet computers) or video game consoles are available for single-user cases. In these cases, the user of the device hums a melody, sings a song, or plays an instrument while the device records the resulting audio signal. The recorded signal may then be analyzed by the application for its pitch/note contour, and the user can select processing operations, such as correcting or otherwise altering the contour, upmixing the signal with different pitches or instrument timbres, etc. Examples of such applications include the QUSIC application (QUALCOMM Incorporated, San Diego, Calif.); video games such as Guitar Hero and Rock Band (Harmonix Music Systems, Cambridge, Mass.); and karaoke, one-man-band, and other recording applications.
Many video games (e.g., Guitar Hero, Rock Band) and concert music scenes may involve multiple instruments and vocalists playing at the same time. Current commercial game and music production systems require these scenarios to be played sequentially or with closely positioned microphones to be able to analyze, post-process and upmix them separately. These constraints may limit the ability to control interference and/or to record spatial effects in the case of music production and may result in a limited user experience in the case of video games.
A method of decomposing an audio signal according to a general configuration includes calculating, for each of a plurality of segments in time of the audio signal, a corresponding signal representation over a range of frequencies. This method also includes calculating a vector of activation coefficients, based on the plurality of calculated signal representations and on a plurality of basis functions. In this method, each activation coefficient of the vector corresponds to a different basis function of the plurality of basis functions, and each of the plurality of basis functions comprises a first corresponding signal representation over the range of frequencies and a second corresponding signal representation over the range of frequencies that is different than said first corresponding signal representation. Computer-readable storage media (e.g., non-transitory media) having tangible features that cause a machine reading the features to perform such a method are also disclosed.
An apparatus for decomposing an audio signal according to a general configuration includes means for calculating, for each of a plurality of segments in time of the audio signal, a corresponding signal representation over a range of frequencies; and means for calculating a vector of activation coefficients, based on the plurality of calculated signal representations and on a plurality of basis functions. In this apparatus, each activation coefficient of the vector corresponds to a different basis function of the plurality of basis functions, and each of the plurality of basis functions comprises a first corresponding signal representation over the range of frequencies and a second corresponding signal representation over the range of frequencies that is different than said first corresponding signal representation.
An apparatus for decomposing an audio signal according to another general configuration includes a transform module configured to calculate, for each of a plurality of segments in time of the audio signal, a corresponding signal representation over a range of frequencies; and a coefficient vector calculator configured to calculate a vector of activation coefficients, based on the plurality of calculated signal representations and on a plurality of basis functions. In this apparatus, each activation coefficient of the vector corresponds to a different basis function of the plurality of basis functions, and each of the plurality of basis functions comprises a first corresponding signal representation over the range of frequencies and a second corresponding signal representation over the range of frequencies that is different than said first corresponding signal representation.
Decomposition of an audio signal using a basis function inventory and a sparse recovery technique is disclosed, wherein the basis function inventory includes information relating to the changes in the spectrum of a musical note over the pendency of the note. Such decomposition may be used to support analysis, encoding, reproduction, and/or synthesis of the signal. Examples of quantitative analyses of audio signals that include mixtures of sounds from harmonic (i.e., non-percussive) and percussive instruments are shown herein.
Unless expressly limited by its context, the term “signal” is used herein to indicate any of its ordinary meanings, including a state of a memory location (or set of memory locations) as expressed on a wire, bus, or other transmission medium. Unless expressly limited by its context, the term “generating” is used herein to indicate any of its ordinary meanings, such as computing or otherwise producing. Unless expressly limited by its context, the term “calculating” is used herein to indicate any of its ordinary meanings, such as computing, evaluating, smoothing, and/or selecting from a plurality of values. Unless expressly limited by its context, the term “obtaining” is used to indicate any of its ordinary meanings, such as calculating, deriving, receiving (e.g., from an external device), and/or retrieving (e.g., from an array of storage elements). Unless expressly limited by its context, the term “selecting” is used to indicate any of its ordinary meanings, such as identifying, indicating, applying, and/or using at least one, and fewer than all, of a set of two or more. Where the term “comprising” is used in the present description and claims, it does not exclude other elements or operations. The term “based on” (as in “A is based on B”) is used to indicate any of its ordinary meanings, including the cases (i) “derived from” (e.g., “B is a precursor of A”), (ii) “based on at least” (e.g., “A is based on at least B”) and, if appropriate in the particular context, (iii) “equal to” (e.g., “A is equal to B”). Similarly, the term “in response to” is used to indicate any of its ordinary meanings, including “in response to at least.”
References to a “location” of a microphone of a multi-microphone audio sensing device indicate the location of the center of an acoustically sensitive face of the microphone, unless otherwise indicated by the context. The term “channel” is used at times to indicate a signal path and at other times to indicate a signal carried by such a path, according to the particular context. Unless otherwise indicated, the term “series” is used to indicate a sequence of two or more items. The term “logarithm” is used to indicate the base-ten logarithm, although extensions of such an operation to other bases (e.g., base two) are within the scope of this disclosure. The term “frequency component” is used to indicate one among a set of frequencies or frequency bands of a signal, such as a sample of a frequency domain representation of the signal (e.g., as produced by a fast Fourier transform) or a subband of the signal (e.g., a Bark scale or mel scale subband).
Unless indicated otherwise, any disclosure of an operation of an apparatus having a particular feature is also expressly intended to disclose a method having an analogous feature (and vice versa), and any disclosure of an operation of an apparatus according to a particular configuration is also expressly intended to disclose a method according to an analogous configuration (and vice versa). The term “configuration” may be used in reference to a method, apparatus, and/or system as indicated by its particular context. The terms “method,” “process,” “procedure,” and “technique” are used generically and interchangeably unless otherwise indicated by the particular context. The terms “apparatus” and “device” are also used generically and interchangeably unless otherwise indicated by the particular context. The terms “element” and “module” are typically used to indicate a portion of a greater configuration. Unless expressly limited by its context, the term “system” is used herein to indicate any of its ordinary meanings, including “a group of elements that interact to serve a common purpose.” Any incorporation by reference of a portion of a document shall also be understood to incorporate definitions of terms or variables that are referenced within the portion, where such definitions appear elsewhere in the document, as well as any figures referenced in the incorporated portion. Unless initially introduced by a definite article, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify a claim element does not by itself indicate any priority or order of the claim element with respect to another, but rather merely distinguishes the claim element from another claim element having a same name (but for use of the ordinal term). Unless expressly limited by its context, the term “plurality” is used herein to indicate an integer quantity that is greater than one.
A method as described herein may be configured to process the captured signal as a series of segments. Typical segment lengths range from about five or ten milliseconds to about forty or fifty milliseconds, and the segments may be overlapping (e.g., with adjacent segments overlapping by 25% or 50%) or nonoverlapping. In one particular example, the signal is divided into a series of nonoverlapping segments or “frames”, each having a length of ten milliseconds. A segment as processed by such a method may also be a segment (i.e., a “subframe”) of a larger segment as processed by a different operation, or vice versa.
It may be desirable to decompose music scenes to extract individual note/pitch profiles from a mixture of two or more instrument and/or vocal signals. Potential use cases include taping concert/video game scenes with multiple microphones, decomposing musical instruments and vocals with spatial/sparse recovery processing, extracting pitch/note profiles, partially or completely up-mixing individual sources with corrected pitch/note profiles. Such operations may be used to extend the capabilities of music applications (e.g., Qualcomm's QUSIC application, video games such as Rock Band or Guitar Hero) to multi-player/singer scenarios.
It may be desirable to enable a music application to process a scenario in which more than one vocalist is active and/or multiple instruments are played at the same time (e.g., as shown in
This disclosure describes methods that may be used to enable a use case for a music application in which multiple sources may be active at the same time. Such a method may be configured to analyze an audio mixture signal using basis-function inventory-based sparse recovery (e.g., sparse decomposition) techniques.
It may be desirable to decompose mixture signal spectra into source components by finding the sparsest vector of activation coefficients (e.g., using efficient sparse recovery algorithms) for a set of basis functions. The activation coefficient vector may be used (e.g., with the set of basis functions) to reconstruct the mixture signal or to reconstruct a selected part (e.g., from one or more selected instruments) of the mixture signal. It may also be desirable to post-process the sparse coefficient vector (e.g., according to magnitude and time support).
Task T100 may be implemented to calculate the signal representation as a frequency-domain vector. Each element of such a vector may indicate the energy of a corresponding one of a set of subbands, which may be obtained according to a mel or Bark scale. However, such a vector is typically calculated using a discrete Fourier transform (DFT), such as a fast Fourier transform (FFT), or a short-time Fourier transform (STFT). Such a vector may have a length of, for example, 64, 128, 256, 512, or 1024 bins. In one example, the audio signal has a sampling rate of eight kHz, and the 0-4 kHz band is represented by a frequency-domain vector of 256 bins for each frame of length 32 milliseconds. In another example, the signal representation is calculated using a modified discrete cosine transform (MDCT) over overlapping segments of the audio signal.
In a further example, task T100 is implemented to calculate the signal representation as a vector of cepstral coefficients (e.g., mel-frequency cepstral coefficients or MFCCs) that represents the short-term power spectrum of the frame. In this case, task T100 may be implemented to calculate such a vector by applying a mel-scale filter bank to the magnitude of a DFT frequency-domain vector of the frame, taking the logarithm of the filter outputs, and taking a DCT of the logarithmic values. Such a procedure is described, for example, in the Aurora standard described in ETSI document ES 201 108, entitled “STQ: DSR—Front-end feature extraction algorithm; compression algorithm” (European Telecommunications Standards Institute, 2000).
Musical instruments typically have well-defined timbres. The timbre of an instrument may be described by its spectral envelope (e.g., the distribution of energy over a range of frequencies), such that a range of timbres of different musical instruments may be modeled using an inventory of basis functions that encode the spectral envelopes of the individual instruments.
Each basis function comprises a corresponding signal representation over a range of frequencies. It may be desirable for each of these signal representations to have the same form as the signal representation calculated by task T100. For example, each basis function may be a frequency-domain vector of length 64, 128, 256, 512, or 1024 bins. Alternatively, each basis function may be a cepstral-domain vector, such as a vector of MFCCs. In a further example, each basis function is a wavelet-domain vector.
The basis function inventory A may include a set An of basis functions for each instrument n (e.g., piano, flute, guitar, drums, etc.). For example, the timbre of an instrument is generally pitch-dependent, such that the set An of basis functions for each instrument n will typically include at least one basis function for each pitch over some desired pitch range, which may vary from one instrument to another. A set of basis functions that corresponds to an instrument tuned to the chromatic scale, for example, may include a different basis function for each of the twelve pitches per octave. The set of basis functions for a piano may include a different basis function for each key of the piano, for a total of eighty-eight basis functions. In another example, the set of basis functions for each instrument includes a different basis function for each pitch in a desired pitch range, such as five octaves (e.g., 56 pitches) or six octaves (e.g., 67 pitches). These sets An of basis functions may be disjoint, or two or more sets may share one or more basis functions.
The inventory of basis functions may be based on a generic musical instrument pitch database, learned from an ad hoc recorded individual instrument recording, and/or based on separated streams of mixtures (e.g., using a separation scheme such as independent component analysis (ICA), expectation-maximization (EM), etc.).
Based on the signal representation calculated by task T100 and on a plurality B of basis functions from the inventory A, task T200 calculates a vector of activation coefficients. Each coefficient of this vector corresponds to a different one of the plurality B of basis functions. For example, task T200 may be configured to calculate the vector such that it indicates the most probable model for the signal representation, according to the plurality B of basis functions.
Task T200 may be configured to recover the activation coefficient vector for each frame of the audio signal by solving a linear programming problem. Examples of methods that may be used to solve such a problem include nonnegative matrix factorization (NNMF). A single-channel reference method that is based on NNMF may be configured to use expectation-maximization (EM) update rules (e.g., as described below) to compute basis functions and activation coefficients at the same time.
It may be desirable to decompose the audio mixture signal into individual instruments (which may include one or more human voices) by finding the sparsest activation coefficient vector in a known or partially known basis function space. For example, task T200 may be configured to use a set of known instrument basis functions to decompose an input signal representation into source components (e.g., one or more individual instruments) by finding the sparsest activation coefficient vector in the basis function inventory (e.g., using efficient sparse recovery algorithms)
It is known that the minimum L1-norm solution to an underdetermined system of linear equations (i.e., a system having more unknowns than equations) is often also the sparsest solution to that system. Sparse recovery via minimization of the L1-norm may be performed as follows.
We assume that our target vector f0 is a sparse vector of length N having K<N nonzero entries (i.e., is “K-sparse”) and that projection matrix (i.e., basis function matrix) A is incoherent (random-like) for a set of size˜K. We observe the signal y=Afo. Then solving minf∥f∥l
One approach is to use sparse recovery algorithms from compressive sensing. In one example of compressive sensing (also called “compressed sensing”) signal recovery Φx=y, y is an observed signal vector of length M, x is a sparse vector of length N having K<N nonzero entries (i.e., a “K-sparse model”) that is a condensed representation of y, and Φ is a random projection matrix of size M×N. The random projection Φ is not full rank, but it is invertible for sparse/compressible signal models with high probability (i.e., it solves an ill-posed inverse problem).
The activation coefficient vector f may be considered to include a subvector fn for each instrument n that includes the activation coefficients for the corresponding basis function set An. These instrument-specific activation subvectors may be processed independently (e.g., in a post-processing operation). For example, it may be desirable to enforce one or more sparsity constraints (e.g., at least half of the vector elements are zero, the number of nonzero elements in an instrument-specific subvector does not exceed a maximum value, etc.). Processing of the activation coefficient vector may include encoding the index number of each non-zero activation coefficient for each frame, encoding the index and value of each non-zero activation coefficient, or encoding the entire sparse vector. Such information may be used (e.g., at another time and/or location) to reproduce the mixture signal using the indicated active basis functions, or to reproduce only a particular part of the mixture signal (e.g., only the notes played by a particular instrument).
An audio signal produced by a musical instrument may be modeled as a series of events called notes. The sound of a harmonic instrument playing a note may be divided into different regions over time: for example, an onset stage (also called attack), a stationary stage (also called sustain), and an offset stage (also called release). Another description of the temporal envelope of a note (ADSR) includes an additional decay stage between attack and sustain. In this context, the duration of a note may be defined as the interval from the start of the attack stage to the end of the release stage (or to another event that terminates the note, such as the start of another note on the same string). A note is assumed to have a single pitch, although the inventory may also be implemented to model notes having a single attack and multiple pitches (e.g., as produced by a pitch-bending effect, such as vibrato or portamento). Some instruments (e.g., a piano, guitar, or harp) may produce more than one note at a time in an event called a chord.
Notes produced by different instruments may have similar timbres during the sustain stage, such that it may be difficult to identify which instrument is playing during such a period. The timbre of a note may be expected to vary from one stage to another, however. For example, identifying an active instrument may be easier during an attack or release stage than during a sustain stage.
To increase the likelihood that the activation coefficient vector will indicate an appropriate basis function, it may be desirable to maximize differences between the basis functions. For example, it may be desirable for a basis function to include information relating to changes in the spectrum of a note over time.
It may be desirable to select a basis function based on a change in timbre over time. Such an approach may include encoding information relating to such time-domain evolution of the timbre of a note into the basis function inventory. For example, the set An of basis functions for a particular instrument n may include two or more corresponding signal representations at each pitch, such that each of these signal representations corresponds to a different time in the evolution of the note (e.g., one for attack stage, one for sustain stage, and one for release stage). These basis functions may be extracted from corresponding frames of a recording of the instrument playing the note.
Method M200 includes multiple instances of task T100 (in this example, tasks T100A and T100B), wherein each instance calculates, based on information from a corresponding different frame of the audio signal, a corresponding signal representation over a range of frequencies. The various signal representations may be concatenated, and likewise each basis function may be a concatenation of multiple signal representations. In this example, task T200 matches the concatenation of mixture frames against the concatenations of the signal representations at each pitch.
The inventory may be constructed such that the multiple signal representations at each pitch are taken from consecutive frames of a training signal. In other implementations, it may be desirable for the multiple signal representations at each pitch to span a larger window in time (e.g., to include frames that are separated in time rather than consecutive). For example, it may be desirable for the multiple signal representations at each pitch to include signal representations from at least two among an attack stage, a sustain stage, and a release stage. By including more information regarding the time-domain evolution of the note, the difference between the sets of basis functions for different notes may be increased.
On the left,
The actual timbre of a flute contains more high-frequency energy than that of a piano, although the basis functions shown in the left plot of
A musical note may include coloration effects, such as vibrato and/or tremolo. Vibrato is a frequency modulation, with a modulation rate that is typically in a range of from four or five to seven, eight, ten, or twelve Hertz. A pitch change due to vibrato may vary between 0.6 to two semitones for singers, and is generally less than ±0.5 semitone for wind and string instruments (e.g., between 0.2 and 0.35 semitones for string instruments). Tremolo is an amplitude modulation typically having a similar modulation rate.
It may be difficult to model such effects in the basis function inventory. It may be desirable to detect the presence of such effects. For example, the presence of vibrato may be indicated by a frequency-domain peak in the range of 4-8 Hz. It may also be desirable to record a measure of the level of the detected effect (e.g., as the energy of this peak), as such a characteristic may be used to restore the effect during reproduction. Similar processing may be performed in the time domain for tremolo detection and quantification. Once the effect has been detected and possibly quantified, it may be desirable to remove the modulation by smoothing the frequency over time for vibrato or by smoothing the amplitude over time for tremolo.
This disclosure describes methods that may be used to enable a use case for a music application in which multiple sources may be active at the same time. In such case, it may be desirable to separate the sources, if possible, before calculating the activation coefficient vector. To achieve this goal, a combination of multi- and single-channel techniques is proposed.
Spatial separation methods alone may be insufficient to achieve a desired level of separation. For example, some sources may be too close or otherwise suboptimally arranged with respect to the microphone array (e.g. multiple violinists and/or harmonic instruments may be located in one corner; percussionists are usually located in the back). In a typical music-band scenario, sources may be located close together or even behind other sources (e.g., as shown in
For computational tractability, it may be desirable for the plurality B of basis functions to be considerably smaller than the inventory A of basis functions. It may be desirable to narrow down the inventory for a given separation task, starting from a large inventory. In one example, such a reduction may be performed by determining whether a segment includes sound from percussive instruments or sound from harmonic instruments, and selecting an appropriate plurality B of basis functions from the inventory for matching. Percussive instruments tend to have impulse-like spectrograms (e.g., vertical lines) as opposed to horizontal lines for harmonic sounds.
A harmonic instrument may typically be characterized in the spectrogram by a certain fundamental pitch and associated timbre, and a corresponding higher-frequency extension of this harmonic pattern. Consequently, in another example it may be desirable to reduce the computational task by only analyzing lower octaves of these spectra, as their higher frequency replica may be predicted based on the low-frequency ones. After matching, the active basis functions may be extrapolated to higher frequencies and subtracted from the mixture signal to obtain a residual signal that may be encoded and/or further decomposed.
Such a reduction may also be performed through user selection in a graphical user interface and/or by pre-classification of most likely instruments and/or pitches based on a first sparse recovery run or maximum likelihood fit. For example, a first run of the sparse recovery operation may be performed to obtain a first set of recovered sparse coefficients, and based on this first set, the applicable note basis functions may be narrowed down for another run of the sparse recovery operation.
One reduction approach includes detecting the presence of certain instrument notes by measuring sparsity scores in certain pitch intervals. Such an approach may include refining the spectral shape of one or more basis functions, based on initial pitch estimates, and using the refined basis functions as the plurality B in method M100.
A reduction approach may be configured to identify pitches by measuring sparsity scores of the music signal projected into corresponding basis functions. Given the best pitch scores, the amplitude shapes of basis functions may be optimized to identify instrument notes. The reduced set of active basis functions may then be used as the plurality B in method M100.
It may be desirable for method M100 to include onset detection (e.g., detecting the onset of a musical note) and post-processing to refine harmonic instrument sparse coefficients. The activation coefficient vector f may be considered to include a corresponding subvector fn for each instrument n that includes the activation coefficients for the instrument-specific basis function set Bn, and these subvectors may be processed independently.
A general onset detection method may be based on spectral magnitude (e.g., energy difference). For example, such a method may include finding peaks based on spectral energy and/or peak slope.
It may be desirable also to detect an onset of each individual instrument. For example, a method of onset detection among harmonic instruments may be based on corresponding coefficient difference in time. In one such example, onset detection of a harmonic instrument n is triggered if the index of the highest-magnitude element of the coefficient vector for instrument n (subvector fn) for the current frame is not equal to the index of the highest-magnitude element of the coefficient vector for instrument n for the previous frame. Such an operation may be iterated for each instrument.
It may be desirable to perform post-processing of the sparse coefficient vector of a harmonic instrument. For example, for harmonic instruments it may be desirable to keep a coefficient of the corresponding subvector that has a high magnitude and/or an attack profile that meets a specified criterion (e.g., is sufficiently sharp), and/or to remove (e.g., to zero out) residual coefficients.
For each harmonic instrument, it may be desirable to post-process the coefficient vector at each onset frame (e.g., when onset detection is indicated) such that the coefficient that has the dominant magnitude and an acceptable attack time is kept and residual coefficients are zeroed. The attack time may be evaluated according to a criterion such as average magnitude over time. In one such example, each coefficient for the instrument for the current frame t is zeroed out (i.e., the attack time is not acceptable) if the current average value of the coefficient is less than a past average value of the coefficient (e.g., if the sum of the values of the coefficient over a current window, such as from frame (t−5) to frame (t+4)) is less than the sum of the values of the coefficient over a past window, such as from frame (t−15) to frame (t−6)). Such post-processing of the coefficient vector for a harmonic instrument at each onset frame may also include keeping the coefficient with the largest magnitude and zeroing out the other coefficients. For each harmonic instrument at each non-onset frame, it may be desirable to post-process the coefficient vector to keep only the coefficient whose value in the previous frame was nonzero, and to zero out the other coefficients of the vector.
An EM algorithm may be used to generate an initial basis function matrix and/or to update the basis function matrix (e.g., based on the activation coefficient vectors). An example of update rules for an EM approach is now described. Given a spectrogram Vft, we wish to estimate spectral basis vectors P(f|z) and weight vectors Pt(z) for each time frame. These distributions give us a matrix decomposition.
We apply the EM algorithm as follows: First, randomly initialize weight vectors Pt(z) and spectral basis vectors P(f|z). Then iterate between the following steps until convergence: 1) Expectation (E) step—estimate the posterior distribution Pt(z|f), given the spectral basis vectors P(f|z) and the weight vectors Pt(z). This estimation may be expressed as follows:
2) Maximization (M) step—estimate the weight vectors Pt(z) and the spectral basis vectors P(f|z), given the posterior distribution Pt(z|f). Estimation of the weight vectors may be expressed as follows:
Estimation of the spectral basis vector may be expressed as follows:
It may be desirable to perform a method as described herein within a portable audio sensing device that has an array of two or more microphones configured to receive acoustic signals. Examples of a portable audio sensing device that may be implemented to include such an array and may be used for audio recording and/or voice communications applications include a telephone handset (e.g., a cellular telephone handset); a wired or wireless headset (e.g., a Bluetooth headset); a handheld audio and/or video recorder; a personal media player configured to record audio and/or video content; a personal digital assistant (PDA) or other handheld computing device; and a notebook computer, laptop computer, netbook computer, tablet computer, or other portable computing device. The class of portable computing devices currently includes devices having names such as laptop computers, notebook computers, netbook computers, ultra-portable computers, tablet computers, mobile Internet devices, smartbooks, and smartphones. Such a device may have a top panel that includes a display screen and a bottom panel that may include a keyboard, wherein the two panels may be connected in a clamshell or other hinged relationship. Such a device may be similarly implemented as a tablet computer that includes a touchscreen display on a top surface. Other examples of audio sensing devices that may be constructed to perform such a method and may be used for audio recording and/or voice communications applications include television displays, set-top boxes, and audio- and/or video-conferencing devices.
Chip/chipset CS10 includes a receiver which is configured to receive a radio-frequency (RF) communications signal (e.g., via antenna C40) and to decode and reproduce (e.g., via loudspeaker SP10) an audio signal encoded within the RF signal. Chip/chipset CS10 also includes a transmitter which is configured to encode an audio signal that is based on an output signal produced by apparatus A100 and to transmit an RF communications signal (e.g., via antenna C40) that describes the encoded audio signal. For example, one or more processors of chip/chipset CS10 may be configured to perform a decomposition operation as described above on one or more channels of the multichannel audio input signal such that the encoded audio signal is based on the decomposed signal. In this example, device D20 also includes a keypad C10 and display C20 to support user control and interaction.
The methods and apparatus disclosed herein may be applied generally in any transceiving and/or audio sensing application, including mobile or otherwise portable instances of such applications and/or sensing of signal components from far-field sources. For example, the range of configurations disclosed herein includes communications devices that reside in a wireless telephony communication system configured to employ a code-division multiple-access (CDMA) over-the-air interface. Nevertheless, it would be understood by those skilled in the art that a method and apparatus having features as described herein may reside in any of the various communication systems employing a wide range of technologies known to those of skill in the art, such as systems employing Voice over IP (VoIP) over wired and/or wireless (e.g., CDMA, TDMA, FDMA, and/or TD-SCDMA) transmission channels.
It is expressly contemplated and hereby disclosed that communications devices disclosed herein may be adapted for use in networks that are packet-switched (for example, wired and/or wireless networks arranged to carry audio transmissions according to protocols such as VoIP) and/or circuit-switched. It is also expressly contemplated and hereby disclosed that communications devices disclosed herein may be adapted for use in narrowband coding systems (e.g., systems that encode an audio frequency range of about four or five kilohertz) and/or for use in wideband coding systems (e.g., systems that encode audio frequencies greater than five kilohertz), including whole-band wideband coding systems and split-band wideband coding systems.
The foregoing presentation of the described configurations is provided to enable any person skilled in the art to make or use the methods and other structures disclosed herein. The flowcharts, block diagrams, and other structures shown and described herein are examples only, and other variants of these structures are also within the scope of the disclosure. Various modifications to these configurations are possible, and the generic principles presented herein may be applied to other configurations as well. Thus, the present disclosure is not intended to be limited to the configurations shown above but rather is to be accorded the widest scope consistent with the principles and novel features disclosed in any fashion herein, including in the attached claims as filed, which form a part of the original disclosure.
Those of skill in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, and symbols that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
Important design requirements for implementation of a configuration as disclosed herein may include minimizing processing delay and/or computational complexity (typically measured in millions of instructions per second or MIPS), especially for computation-intensive applications, such as playback of compressed audio or audiovisual information (e.g., a file or stream encoded according to a compression format, such as one of the examples identified herein) or applications for wideband communications (e.g., voice communications at sampling rates higher than eight kilohertz, such as 12, 16, 44.1, 48, or 192 kHz).
Goals of a multi-microphone processing system may include achieving ten to twelve dB in overall noise reduction, preserving voice level and color during movement of a desired speaker, obtaining a perception that the noise has been moved into the background instead of an aggressive noise removal, dereverberation of speech, and/or enabling the option of post-processing for more aggressive noise reduction.
An apparatus as disclosed herein (e.g., apparatus A100, A300, A310, A700, and MF100) may be implemented in any combination of hardware with software, and/or with firmware, that is deemed suitable for the intended application. For example, the elements of such an apparatus may be fabricated as electronic and/or optical devices residing, for example, on the same chip or among two or more chips in a chipset. One example of such a device is a fixed or programmable array of logic elements, such as transistors or logic gates, and any of these elements may be implemented as one or more such arrays. Any two or more, or even all, of the elements of the apparatus may be implemented within the same array or arrays. Such an array or arrays may be implemented within one or more chips (for example, within a chipset including two or more chips).
One or more elements of the various implementations of the apparatus disclosed herein may also be implemented in whole or in part as one or more sets of instructions arranged to execute on one or more fixed or programmable arrays of logic elements, such as microprocessors, embedded processors, IP cores, digital signal processors, FPGAs (field-programmable gate arrays), ASSPs (application-specific standard products), and ASICs (application-specific integrated circuits). Any of the various elements of an implementation of an apparatus as disclosed herein may also be embodied as one or more computers (e.g., machines including one or more arrays programmed to execute one or more sets or sequences of instructions, also called “processors”), and any two or more, or even all, of these elements may be implemented within the same such computer or computers.
A processor or other means for processing as disclosed herein may be fabricated as one or more electronic and/or optical devices residing, for example, on the same chip or among two or more chips in a chipset. One example of such a device is a fixed or programmable array of logic elements, such as transistors or logic gates, and any of these elements may be implemented as one or more such arrays. Such an array or arrays may be implemented within one or more chips (for example, within a chipset including two or more chips). Examples of such arrays include fixed or programmable arrays of logic elements, such as microprocessors, embedded processors, IP cores, DSPs, FPGAs, ASSPs, and ASICs. A processor or other means for processing as disclosed herein may also be embodied as one or more computers (e.g., machines including one or more arrays programmed to execute one or more sets or sequences of instructions) or other processors. It is possible for a processor as described herein to be used to perform tasks or execute other sets of instructions that are not directly related to a music decomposition procedure as described herein, such as a task relating to another operation of a device or system in which the processor is embedded (e.g., an audio sensing device). It is also possible for part of a method as disclosed herein to be performed by a processor of the audio sensing device and for another part of the method to be performed under the control of one or more other processors.
Those of skill will appreciate that the various illustrative modules, logical blocks, circuits, and tests and other operations described in connection with the configurations disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. Such modules, logical blocks, circuits, and operations may be implemented or performed with a general-purpose processor, a digital signal processor (DSP), an ASIC or ASSP, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to produce the configuration as disclosed herein. For example, such a configuration may be implemented at least in part as a hard-wired circuit, as a circuit configuration fabricated into an application-specific integrated circuit, or as a firmware program loaded into non-volatile storage or a software program loaded from or into a data storage medium as machine-readable code, such code being instructions executable by an array of logic elements such as a general purpose processor or other digital signal processing unit. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. A software module may reside in RAM (random-access memory), ROM (read-only memory), nonvolatile RAM (NVRAM) such as flash RAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An illustrative storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.
It is noted that the various methods disclosed herein (e.g., method M100 and other methods disclosed by way of description of the operation of the various apparatus described herein) may be performed by an array of logic elements such as a processor, and that the various elements of an apparatus as described herein may be implemented as modules designed to execute on such an array. As used herein, the term “module” or “sub-module” can refer to any method, apparatus, device, unit or computer-readable data storage medium that includes computer instructions (e.g., logical expressions) in software, hardware or firmware form. It is to be understood that multiple modules or systems can be combined into one module or system and one module or system can be separated into multiple modules or systems to perform the same functions. When implemented in software or other computer-executable instructions, the elements of a process are essentially the code segments to perform the related tasks, such as with routines, programs, objects, components, data structures, and the like. The term “software” should be understood to include source code, assembly language code, machine code, binary code, firmware, macrocode, microcode, any one or more sets or sequences of instructions executable by an array of logic elements, and any combination of such examples. The program or code segments can be stored in a processor-readable storage medium or transmitted by a computer data signal embodied in a carrier wave over a transmission medium or communication link.
The implementations of methods, schemes, and techniques disclosed herein may also be tangibly embodied (for example, in one or more computer-readable media as listed herein) as one or more sets of instructions readable and/or executable by a machine including an array of logic elements (e.g., a processor, microprocessor, microcontroller, or other finite state machine). The term “computer-readable medium” may include any medium that can store or transfer information, including volatile, nonvolatile, removable and non-removable media. Examples of a computer-readable medium include an electronic circuit, a semiconductor memory device, a ROM, a flash memory, an erasable ROM (EROM), a floppy diskette or other magnetic storage, a CD-ROM/DVD or other optical storage, a hard disk, a fiber optic medium, a radio frequency (RF) link, or any other medium which can be used to store the desired information and which can be accessed. The computer data signal may include any signal that can propagate over a transmission medium such as electronic network channels, optical fibers, air, electromagnetic, RF links, etc. The code segments may be downloaded via computer networks such as the Internet or an intranet. In any case, the scope of the present disclosure should not be construed as limited by such embodiments.
Each of the tasks of the methods described herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. In a typical application of an implementation of a method as disclosed herein, an array of logic elements (e.g., logic gates) is configured to perform one, more than one, or even all of the various tasks of the method. One or more (possibly all) of the tasks may also be implemented as code (e.g., one or more sets of instructions), embodied in a computer program product (e.g., one or more data storage media such as disks, flash or other nonvolatile memory cards, semiconductor memory chips, etc.), that is readable and/or executable by a machine (e.g., a computer) including an array of logic elements (e.g., a processor, microprocessor, microcontroller, or other finite state machine). The tasks of an implementation of a method as disclosed herein may also be performed by more than one such array or machine. In these or other implementations, the tasks may be performed within a device for wireless communications such as a cellular telephone or other device having such communications capability. Such a device may be configured to communicate with circuit-switched and/or packet-switched networks (e.g., using one or more protocols such as VoIP). For example, such a device may include RF circuitry configured to receive and/or transmit encoded frames.
It is expressly disclosed that the various methods disclosed herein may be performed by a portable communications device such as a handset, headset, or portable digital assistant (PDA), and that the various apparatus described herein may be included within such a device. A typical real-time (e.g., online) application is a telephone conversation conducted using such a mobile device.
In one or more exemplary embodiments, the operations described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, such operations may be stored on or transmitted over a computer-readable medium as one or more instructions or code. The term “computer-readable media” includes both computer-readable storage media and communication (e.g., transmission) media. By way of example, and not limitation, computer-readable storage media can comprise an array of storage elements, such as semiconductor memory (which may include without limitation dynamic or static RAM, ROM, EEPROM, and/or flash RAM), or ferroelectric, magnetoresistive, ovonic, polymeric, or phase-change memory; CD-ROM or other optical disk storage; and/or magnetic disk storage or other magnetic storage devices. Such storage media may store information in the form of instructions or data structures that can be accessed by a computer. Communication media can comprise any medium that can be used to carry desired program code in the form of instructions or data structures and that can be accessed by a computer, including any medium that facilitates transfer of a computer program from one place to another. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, and/or microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology such as infrared, radio, and/or microwave are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray Disc™ (Blu-Ray Disc Association, Universal City, Calif.), where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
An acoustic signal processing apparatus as described herein (e.g., apparatus A100 or MF100) may be incorporated into an electronic device that accepts speech input in order to control certain operations, or may otherwise benefit from separation of desired noises from background noises, such as communications devices. Many applications may benefit from enhancing or separating clear desired sound from background sounds originating from multiple directions. Such applications may include human-machine interfaces in electronic or computing devices which incorporate capabilities such as voice recognition and detection, speech enhancement and separation, voice-activated control, and the like. It may be desirable to implement such an acoustic signal processing apparatus to be suitable in devices that only provide limited processing capabilities.
The elements of the various implementations of the modules, elements, and devices described herein may be fabricated as electronic and/or optical devices residing, for example, on the same chip or among two or more chips in a chipset. One example of such a device is a fixed or programmable array of logic elements, such as transistors or gates. One or more elements of the various implementations of the apparatus described herein may also be implemented in whole or in part as one or more sets of instructions arranged to execute on one or more fixed or programmable arrays of logic elements such as microprocessors, embedded processors, IP cores, digital signal processors, FPGAs, ASSPs, and ASICs.
It is possible for one or more elements of an implementation of an apparatus as described herein to be used to perform tasks or execute other sets of instructions that are not directly related to an operation of the apparatus, such as a task relating to another operation of a device or system in which the apparatus is embedded. It is also possible for one or more elements of an implementation of such an apparatus to have structure in common (e.g., a processor used to execute portions of code corresponding to different elements at different times, a set of instructions executed to perform tasks corresponding to different elements at different times, or an arrangement of electronic and/or optical devices performing operations for different elements at different times).
The present Application for Patent claims priority to Provisional Application No. 61/406,376, entitled “CASA (COMPUTATIONAL AUDITORY SCENE ANALYSIS) FOR MUSIC APPLICATIONS: DECOMPOSITION OF MUSIC SIGNALS USING BASIS FUNCTION INVENTORY AND SPARSE RECOVERY,” filed Oct. 25, 2010, and assigned to the assignee hereof.
| Number | Date | Country | |
|---|---|---|---|
| 61406376 | Oct 2010 | US |
