Method and apparatus for speech feature recognition based on models of auditory signal processing

Description

FIELD OF THE INVENTION
This invention relates to the area of signal processing generally, and particularly to the processing of speech by methods designed to detect linguistically important features for the purpose of speech coding, recognition, categorization and compression and for speaker verification and identification.
BACKGROUND OF THE INVENTION
Almost all current technology for signal processing for applications in the areas of speech recognition and speaker verification or identification is based on a variant of the frequency spectrogram, which is a representation of the energy in a signal as a function of frequency and time. While spectrographic processing was originally implemented by means of analog electronic hardware, including analog filter banks (e.g. the voice print), currently, spectral analysis is primarily implemented using the techniques of digital signal processing. The methods of spectral analysis include fast Fourier transformation (FFT), power spectral density (PSD) analysis, extraction of linear predictive coding (LPC) and cepstral coefficients. Other methods include processing by digital filter banks comprising filters designed by standard methods and filters whose design is purportedly based on some feature of the response of the auditory system. Spectrographic processing is usually applied with the aim of extracting important linguistic features from the speech signal such as the frequencies and times of occurrence of the formants. These speech features are often obtained by comparing the spectrographic patterns to templates or rules. Other conventional signal processing techniques are used to detect speech features. For example, autocorrelation functions are used to extract the pitch of a voiced utterance; zero-crossing profiles are used to discriminate between voiced and unvoiced segments of speech (Schafer, R. W. and Rabiner, L. R. (1978): Digital Processing of Speech Signals. Englewood Cliffs (N.J.): Prentice-Hall).
In general, conventional methods of speech processing suffer from several well-known problems:
Susceptibility to noise. Because the profile of spectral energy that constitutes the spectrogram is sensitive to anything that changes the relative magnitude of in-band energies, spectrographic representations can be severely degraded in situations of practical interest, such as the presence of high background or line noise;
Sensitivity to spectral shaping or bandwidth reduction. The characteristics of the communication channel can affect the spectrum of the input signal, thereby altering the profile of spectral energy, and distorting the spectrogram;
Non-selectivity for speech. Spectrographic techniques measure the frequency profile of signal energy irrespective of the source of that energy. They are not inherently selective for speech signals. Sources of signal energy such as line and environmental noise or non-speech signals such as music or tones create spectrographic patterns that can result in the mis-identification of relevant speech parameters;
Difficulty in estimating formant information. Conventional speech processing methods often have difficulty in estimating the pitch and formant frequencies of a voiced utterance. Speech is a temporally and spectrally complex waveform. Voiced portions of speech comprise epochs of wide spectral bandwidth (corresponding to the glottal, or pitch, pulses) alternating with epochs characterized by a more discrete frequency spectrum (corresponding to the formant frequencies). For spectrographic schemes aimed at the extraction of formant frequencies, the energy in the glottal pulse represents a confounding element. Techniques well known to the art, such as cepstral analysis and pitch-synchronous spectral extraction, have been employed in an attempt to separate the pitch from formant information;
Difficulty in estimating pitch. Speech is non-stationary and non-periodic. In voiced segments of speech, pitch is rarely constant, and autocorrelation techniques for the extraction of pitch, which essentially measure periodicity, can be inaccurate;
Sensitivity to segmentation of input data. In spectrographic sound analysis methods, sound data are usually segmented or windowed into frames (generally 10 to 20 milliseconds long) for analysis. The onset and duration of the frame can affect the accurate localization of spectrographic features in the time and frequency domains. For small frame sizes, spectrographic methods can follow the dynamic character of the speech, but with reduced frequency resolution, whereas for larger frame sizes, the frequency resolution improves at the expense of the resolution of the dynamic time-domain characteristics. Accurate time and frequency localization of formants is difficult because the formant frequencies can vary between adjacent glottal pulses occurring less than 5 milliseconds apart.
Several schemes have been disclosed in the prior art to process speech using methods specifically derived from an analysis of signal processing in the human auditory system. U.S. Pat. No. 4,536,844 issued to Richard F. Lyon on Aug. 20, 1985 discloses a method and apparatus for simulating auditory response information where the input signals are analyzed by a filter-bank comprising an array of high-order filters, each created from a cascade of linear, time-invariant, second-order digital filter sections followed by a stage of rectification and nonlinear dynamic range compression (automatic gain control). While this process purports to produce a representation similar to the human neural response, the resulting response does not, in fact, correspond to the measured experimental data from auditory-nerve fibers (Pickles, J. O. (1988): Introduction to the Physiology of Hearing. 2nd edition. London: Academic Press). Similar processing schemes are also described in the literature (Seneff, S. (1985): A joint synchrony/mean-rate model of auditory speech processing. Journal of Phonetics 16, 55-76; Kates, J. M. (1991): A time-domain digital cochlear model. IEEE Transactions on Signal Processing 39, 2573-2592.) All of these approaches generate an essentially spectral representation of speech.
U.S. Pat. No. 4,905,285, issued to Jont B. Allen et al. on Feb. 27, 1990 also discloses a method based on a model that purports to represent the frequency distribution of human neural response. In this method, the speech signal is analyzed by a bank of filters whose frequency response is derived from a mathematical model of the motion of the basilar membrane. The time waveform which constitutes the output of each spectral band is passed through a series of threshold detectors. The times between successive threshold crossings of detectors are measured and accumulated into an interval histogram. Interval histograms for a plurality of spectral bands are then combined to produce an ensemble histogram. From this histogram, a profile of the dominant average frequency components of an input signal is generated by means of conventional signal processing techniques (inverse Fourier transformation and autocorrelation). U.S. Pat. No. 4,075,423 issued to M. J. Martin et al. on Feb. 21, 1978 discloses a similar scheme based on accumulating a histogram of frequency patterns of detected waveform peaks. Spectrographic processing schemes based on threshold crossings of detected waveform peaks are also well documented in the literature (Niederjohn, R. J. (1985): A zero-crossing consistency method for formant tracking of voiced speech in high noise levels. IEEE Transactions of Acoustics, Speech and Signal Processing, vol ASSP-33, 2; Ghitza, O. (1985): A measure of in-synchrony regions in the auditory nerve firing patterns as a basis of speech vocoding. Proceedings International Conference, Acoustics Speech and Signal Processing.)
There are several significant disadvantages and problems with the neural threshold-crossing methods of the prior art that limit their applicability. Chief among these are the issues of temporal granularity and computational intractability. Threshold-crossings of the neural response model occur only at discrete intervals, which can be spaced milliseconds apart in model fibers with low center frequencies; hence, spectral estimates obtained from the histograms of threshold crossings will be temporally coarse or granular. Computing the complete response of the neural model fibers requires the solution of the nonlinear cochlear model equations for a plurality of parallel channels. The computational load of performing these calculations in real-time or near real-time can be prohibitive. Finally, the neural threshold-crossing methods are not speech specific and thus do not result in the identification of unique speech features.
The present invention provides a novel signal processing system based on signal processing by the auditory system that overcomes these and other problems of the prior art.
SUMMARY OF THE INVENTION
This invention describes novel means for analyzing and parameterizing speech based on a model of signal processing by the human auditory system. The implementations of this invention extract information relevant to the efficient detection and recognition of important features of speech. Specifically, they provide robust, noise-immune, speech-specific detection of glottal pulses and formants with simultaneously high resolution in the time and frequency domains. The implementations of this invention differ fundamentally from conventional signal-processing methods of the prior art. For example, whereas conventional spectrographic representations are based on measuring the energy in a plurality of narrow frequency bands, the representations of the present invention are based on detecting temporal patterns of the response of a plurality of adjacent frequency channels over a restricted period of time.
Here we disclose three implementations of the invention. The first implementation, which is termed the neural-correlation implementation, is based on an analysis of a model of signal processing by the auditory system, whose implementation is described in Appendix A. In this implementation, glottal pulses and formants are determined from temporal patterns of firings of groups of adjacent auditory-nerve fibers of the model. Although the neural-correlation implementation is not the preferred implementation, it is described first because it most clearly illustrates the principal features of the signal processing approach that is used in this invention, and facilitates the subsequent description of the details of other implementations.
The second implementation of the invention, which is termed the phase-coherence implementation, is the preferred implementation. This implementation is based on the observation, derived from the model of signal processing by the auditory system, that essential temporal characteristics of the patterns of neural firing of each channel are preserved in temporal and spatial derivatives of the phase of motion of the basilar-membrane component of the model. In this implementation, glottal pulses and the time and frequency localization of formants are robustly determined from spatially and temporally local patterns of phase correlation in the output of groups of channels.
In the third implementation of the invention, termed the instantaneous-frequency implementation, speech features are detected from patterns present in histograms of the instantaneous frequency of response summed from a number of channels.
In general terms, the basic principal of this invention in all its three implementations is as follows:
Sound is analyzed using a model of the human cochlea which simulates the waveform propagation characteristics of the basilar membrane. Our preferred model is implemented as an array of filters, the frequency and phase response of each of these filters being chosen to substantially match waveform propagation characteristics at equally spaced haircell locations along the length of the basilar membrane of the cochlea.
The response sequences that are computed for the output of the filter channels depend on the implementation. In the phase-coherence implementation, the response quantity that is computed for each filter is the instantaneous phase of the output of the filter. In the instantaneous-frequency implementation, the response quantity is the instantaneous frequency of response. In the neural-correlation implementation, the response quantity is a non-linear and time-varying transformation of the instantaneous amplitude of the output of the filter; this transformation is designed to emulate the response of auditory-nerve fibers to the stimulus. The response sequences that are computed for the output of the filter channels depend on the implementation. In the phase-coherence implementation, the response quantity that is computed for each filter is the instantaneous phase of the output of the filter. In the instantaneous-frequency implementation, the response quantity is the instantaneous frequency of response. In the neural-correlation implementation, the response quantity is a non-linear and time-varying transformation of the instantaneous amplitude of the output of the filter; this transformation is designed to emulate the response of auditory-nerve fibers to the stimulus.
The response sequences computed from the array of filters are then processed by an array of primary feature detectors which are designed to emulate the signal processing characteristics of cells in the brainstem and auditory cortex. The essential attribute of these detectors is that they detect local spatial and temporal patterns of the response of the filters. For example, in the neural-correlation implementation, the primary feature detectors detect patterns in the response of an array of filters that correspond to patterns of the discharge of groups of auditory-nerve fibers. In the phase-coherence implementation, the primary feature detectors detect patterns in spatial and temporal derivatives of the instantaneous phase. In the instantaneous-frequency implementation, the primary feature detectors detect patterns of the instantaneous frequency from a group of channels. The primary feature detectors include local impulse detectors, which detect impulsive features in the stimulus, and local synchrony detectors, which detect synchronous regions of the response. These detectors respond to local spatial and temporal patterns of neural firing or spatio-temporal derivatives of basilar-membrane motion. In the context of these detectors, the term "local" means that each primary feature detector in the array detects only patterns of the response of filter channels over a restricted range of channels and over a restricted interval of time.
The outputs of the primary feature detectors are processed by an array of secondary feature detectors which detect patterns in the response of the array of primary feature detectors. Secondary feature detectors include the local formant detector, which detects the times of occurrence and the frequencies of the formants in speech, and the global pulse detector, which detects the times of occurrence of the glottal pulses. Whereas the term "local" indicates that the response of each channel of the secondary detector depends only upon a restricted range of channels of the primary feature detector stage, the term "global" means that the response of each channel of the secondary detector depends upon a large number of channels the primary feature detector stage.
This invention differs fundamentally from conventional signal processing methods of the prior art in that patterns of wave propagation characteristics in the human cochlea are used as the basis for extracting important features of the input stimulus waveform. Specifically, this invention utilizes combined spatial and time differentials from auditory model output waveforms to detect stereotypical patterns of human speech. Conventional filter bank methods for signal processing, such as those that use the frequency spectrogram, consider only energy or time correlation of individual channels without establishing relationships between channels. We show that the relationships between channels arising from wave propagation characteristics of the human cochlea provide unique new information for robustly detecting features of human speech.
Specifically, this invention improves upon prior-art methods in a number of important respects:
They are insensitive to additive noise;
They are insensitive to spectral shaping or bandwidth reduction of the input;
They are selective for the detection of speech features such as pitch and formant information;
They do not require critical data segmentation;
They simultaneously show high temporal and frequency resolution;
Results can be obtained in a computationally efficient manner.
Further objects and advantages of this invention will become apparent from the consideration of the drawings and ensuing descriptions.

BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a block diagram of the signal-processing stages that characterize the neural-correlation implementation of the invention.
FIG. 2A is a plot of response waveforms generated by the peripheral auditory filtering stage of the auditory model illustrated in FIG. 1 in response to a speech utterance, /a/.
FIG. 2B is a plot of the neural threshold crossings derived from the data of FIG. 2A produced in accordance with the method illustrated in FIG. 3.
FIG. 3 is a flow chart of an arrangement for determining the threshold crossings of the neural response used in conjunction with the neural-correlation implementation of the invention.
FIG. 4 is a flow chart illustrating a method for implementing the local impulse detector of the neural-correlation implementation of the invention.
FIG. 5A is a schematic representation of the arrangement of time-response windows used in the local impulse detector of the neural-correlation implementation.
FIG. 5B is a schematic representation of the arrangement of time-response windows used in the local synchrony detector of the neural-correlation implementation.
FIG. 6 is a block diagram of the signal-processing stages that characterize the phase-coherence implementation of the invention.
FIG. 7A is a flow chart of the signal processing operations performed by the cochlear signal-processing stage, useful for determining the magnitude and phase of basilar membrane response.
FIG. 7B is a schematic diagram of the waveforms that result from the processing operations depicted in FIG. 7A illustrating the operation of the cochlear signal-processing stage according to the invention.
FIG. 8 is a flow chart illustrating a method for implementing the local impulse detector of the phase-coherence implementation.
FIG. 9 is a plot of the response pattern to the utterance /a/ produced by the local impulse detector of the phase-coherence method in accordance with the method of FIG. 8, and the response pattern of the global pulse detector produced in accordance with the method of FIG. 15.
FIG. 10 is a flow chart illustrating a method for implementing the local synchrony detector of the phase-coherence implementation.
FIG. 11 is a plot of the response pattern to the utterance /a/ produced by the local synchrony detector of the phase-coherence implementation in accordance with the method of FIG. 10.
FIG. 12 is a flow chart illustrating a method for implementing the local formant detector.
FIG. 13 is a flow chart illustrating the operation of the spatial response filtering stage used in connection with the formant detector illustrated in FIG. 12.
FIG. 14 is a plot of the response pattern to the utterance /a/ produced by the local formant detector in accordance with the implementation of FIGS. 12-13.
FIG. 15 is a flow chart illustrating the operation of the global pulse detector.
FIG. 16 is a plot of the instantaneous frequency of the cochlear signal-processing stage to the utterance /a/.
FIG. 17 is a flow chart illustrating a method for implementing the local impulse detector of the instantaneous-frequency implementation.
FIG. 18 is a flow chart illustrating a method for implementing the local synchrony detector of the instantaneous-frequency implementation.
FIG. 19 is a block diagram of the peripheral auditory filtering stage based on a model of signal processing by the peripheral auditory system which motivates the principal implementations of the invention.
FIG. 20 is a plot of twelve representative cochlear frequency- and phase-response functions, useful in the implementation of the cochlear signal-processing stage, described in Appendix A.
FIG. 21A is a schematic equivalent circuit of the hair-cell filtering stage.
FIG. 21B is a block diagram of the synaptic filtering stage.

DETAILED DESCRIPTION OF THE EMBODIMENTS
We first describe the neural-correlation implementation because it illustrates the principal features of the signal processing approach that is used in this invention, and facilitates our subsequent description of the details of the phase-coherence and instantaneous-frequency implementations.
With reference to the drawings, FIG. 1 is a block diagram of the signal-processing stages that characterize the neural-correlation implementation of the invention. In this implementation, an input sound p(t) 102 is transformed by a signal converter 104 into a discrete-time input sound sequence, p[n] 106. In this embodiment, p(t) 102 is bandlimited to less than 4 kHz and is digitized at a sample rate of 8 kHz; however, it is understood that the bandwidth need not be limited to 4 kHz, and that other sampling rates can be used in the signal converter. Input sequence p[n] 106 is then passed through a peripheral auditory filtering stage 108 which may be implemented using a processor arrangement incorporating, for example, a type MC68040 microprocessor and a type DSP56001 digital signal processor. The programming instructions implemented on these processors are derived from a biophysical model of signal processing by the peripheral auditory system whose implementation is described in Appendix A. The output of peripheral auditory filtering stage 108 is an array of sequences, .lambda..sub.k [n] 110. These sequences represent the probability density functions of discharge of the primary afferent auditory-nerve fibers, where the channel number, k, represents the spatial dimension of the response. In this embodiment, there are 120 channels, 0.ltoreq.k.ltoreq.119, representing the response of locations linearly spaced along the basilar membrane, with characteristic frequencies ranging from 250 Hz (for k=0) to 3400 Hz (for k=119).
FIG. 2A is a plot of the response waveforms generated by the peripheral auditory filtering stage 108 of the auditory model illustrated in FIG. 1 in response to a speech utterance, /a/. The response to this utterance comprises an array of sequences .lambda..sub.k [n] 110, and consists of two distinct spatio-temporal patterns occurring in alternation, which we term an impulsive epoch 202 and a synchronous epoch 204. In the impulsive epoch 202, each channel tends to respond at the characteristic frequency of that channel. In synchronous epoch 204 which follows, there are several distinct groups of adjacent channels that respond in synchrony with each other at a frequency that corresponds to a proximal formant frequency. One of the principal features of this invention is that linguistically important speech features such as the location of the glottal pulses and formant frequencies can be determined by designing robust detectors to find these impulsive and synchronous epochs.
The output of all of the channels of the peripheral auditory filtering stage (i.e. .lambda..sub.k [n] 110) pass to a neural threshold-crossing stage 112 which produces output sequences DIFIT.sub.k [n] 114 indicating the times at which channel k is most likely to respond to sound. The output of neural threshold-crossing stage 112 goes to processing stages which represent feature detection by the central auditory system, the brainstem nuclei and the auditory cortex. A primary feature detection stage 116 comprises a local impulse detector 118 that detects patterns of nerve-fiber firings that correspond to the impulsive interval, and a local synchrony detector 120 that detects patterns of synchronous firings by groups of nerve fibers which correspond to the synchronous interval. Each local feature detector has a plurality of input sequences DIFIT.sub.k [n] 114 and a plurality of output sequences. Local impulse detector 118 generates an array of output sequences, IMPULSE.sub.k [n] 122, and local synchrony detector 120 generates an array of output sequences SYNCH.sub.k [n] 124. These detectors are termed local to indicate that the value of each output sequence at any time depends only upon a restricted subset of the channels of the input sequences over a small window of time. For example, the value of the output sequence for channel k.sub.0 at time n.sub.0 (i.e. IMPULSE.sub.k.sbsb.0 [n.sub.0 ]) may depend only on values of the input sequence, .lambda..sub.k [n], for a small range of channels near k.sub.0 and for a small range of time near n.sub.0. The output sequences of the primary detectors go to secondary feature detection stage 126 that comprises a plurality of feature detectors which perform the detection and identification of formants, vowels and glottal pulses. These secondary detectors are described in detail in connection with the phase-coherence implementation of the invention.
While it is possible to detect speech features directly from the correlation of the sequences representing the probability density functions of neural discharge, .lambda..sub.k [n] 110, the correlation process is simplified by processing .lambda..sub.k [n] 110 with neural threshold-crossing stage 112 to determine sequences DIFIT.sub.k [n] 114 that mark threshold crossings of the neural response: the times at which the probability of neural response of each channel increases rapidly following periods of comparative low response.
FIG. 3 is a flow of an arrangement for determining threshold crossings of the neural response. The neural input sequences, .lambda..sub.k [n] 110, are computed by peripheral auditory filtering stage 108 for every channel, k.sub.min .ltoreq.k.ltoreq.k.sub.max where k.sub.min and k.sub.max represent the low and high frequency channels of the filter bank, and for every time point in the utterance, n.sub.min .ltoreq.n.ltoreq.n.sub.max, where n.sub.min and n.sub.max represent the beginning and end of the input sequence. Each neural input sequence 110 is processed by a neural response filter stage 302 to produce a filtered neural response sequence, y.sub.k [n], which accentuates those portions of the neural response that correspond to sharp increases in the rate of neural response. In the preferred embodiment, for every channel k and time n, y.sub.k [n] is the smoothed derivative of .lambda..sub.k [n] computed by applying a discrete-time finite impulse response filter of order 2M which has the form: ##EQU1## The filtered neural response, y.sub.k [n], is processed by a neural response threshold comparator to produce a threshold sequence DIFIT.sub.k [n] that has a value of one at the onset of a sharp increase in the neural response and a value of zero otherwise. In the embodiment depicted in FIG. 3, the neural threshold comparator is implemented as a threshold comparator with hysteresis. With reference to FIG. 3, if the threshold comparator is enabled (i.e. if enable.sub.-- comp=TRUE 304) and the value of y.sub.k [n] exceeds thresh.sub.-- high 306, the value of the output sequence DIFIT.sub.k [n] is set to one 308 and the threshold comparator is disabled by setting enable.sub.-- comp=FALSE 310. Otherwise, the value of DIFIT.sub.k [n] is set to zero 312. The threshold comparator is re-enabled (enable.sub.-- comp=TRUE 314) when the value of y.sub.k [n] falls below thresh.sub.-- low 316. The method diagrammed in FIG. 3 results in output sequences, DIFIT.sub.k [n], that correspond to the times at which a fiber is likely to fire in response to a sound; however, it should be clear that the determination of peak times of neural firing can also be accomplished using any of a number of methods known to the art; for example, by finding the peak or threshold-crossings of .lambda..sub.k [n] or filtered versions of .lambda..sub.k [n]. The description of the embodiment in FIG. 3 is not meant to exclude these embodiments.
FIG. 2B shows DIFIT.sub.k [n] 114, the threshold-crossings of .lambda..sub.k [n] produced in accordance with the method described in conjunction with FIG. 3. In this figure one can clearly see that the response to a voiced utterance consists of two distinct spatio-temporal patterns occurring in alternation, an impulsive epoch 206 and a synchronous epoch 208. The pattern of threshold-crossings of the neural response in the impulsive epoch is characteristically splayed. During this epoch, each channel tends to respond at its own characteristic frequency. In the initial portion of this epoch it is generally true that, if an impulse occurs in sequence DIFIT.sub.k [n] 114 for some channel k=k.sub.0 at some time n=n.sub.0 (i.e. DIFIT.sub.k.sbsb.0 [n.sub.0 ]=1), then a response will occur earlier in time for a number of adjacent channels of lower characteristic frequency (that is, DIFIT.sub.k [n]=1 for k<k.sub.0 and n<n.sub.0), and later in time for a number of adjacent channels of higher characteristic frequency (that is, DIFIT.sub.k [n]= 1 for k>k.sub.0 and n>n.sub.0). In the final portion of this epoch, the above relationship of impulses is reversed. This response pattern is characteristic of the peripheral auditory filtering stage's response to impulsive sounds in general and to plosives and pitch pulses of voiced speech in particular.
FIG. 4 is a flow chart illustrating a method for implementing the local impulse detector 118 of the neural-correlation implementation of the invention. The input sequences, DIFIT.sub.k [n] 114, are computed in accordance with the method described in connection with FIG. 3 for every channel, k.sub.min .ltoreq.k.ltoreq.k.sub.max and for every time, n.sub.min .ltoreq.n.ltoreq.n.sub.max. For each channel, local impulse detector 118 detects a pattern of neural activity wherein a number of adjacent channels respond in sequence. This detection of a response pattern by local impulse detector 118 emulates the function of a "neural-coincidence detector" neuron. Such a neuron responds when a number of afferent fibers discharge simultaneously within a brief period of time. Central neurons which function as coincidence detectors are widely documented, and there is specific evidence in the auditory system of cells that function as coincidence detectors and that are sensitive to time differences of neural input at high frequencies.
With reference to FIG. 4, if a response occurs in channel k.sub.0 of the input sequence at time n.sub.0 (i.e. DIFIT.sub.k.sbsb.0 [n.sub.0 ]=1 402), a channel counter 404 is initialized to zero, chan.sub.-- count=0. This counter is used to count the number of channels adjacent to k.sub.0 that respond within a response-time window with respect to n=n.sub.0. For channel k.sub.0, the adjacent channels are defined to range from k.sub.0 -klow.sub.k.sbsb.0 to k.sub.0 +khigh.sub.k.sbsb.0. This formulation allows the number of adjacent channels to depend upon k.sub.0. In the embodiment depicted in FIG. 4, the values of klow.sub.k.sbsb.0 and khigh.sub.k.sbsb.0 are chosen to correspond to approximately one critical band centered around k.sub.0. A response-time window for each channel is defined to be a range of time values around n=n.sub.0, from n.sub.0 -nlow.sub.k.sbsb.0.sub.,k to n.sub.0 +nhigh.sub.k.sbsb.0.sub.,k. This formulation allows the size of the response-time window for the channel k, to depend upon both k.sub.0 and k. In the embodiment depicted in FIG. 4, nlow.sub.k.sbsb.0.sub.,k and nhigh.sub.k.sbsb.0.sub., k depend only upon k, and the response-time window is approximately equal to one characteristic period, that is, the reciprocal of the characteristic frequency of channel k. With reference to FIG. 4, for each channel between k=k.sub.0 -klow.sub.k.sbsb.0 406 and k=k.sub.0 +khigh.sub.k.sbsb.0 408, a neural response summing element 410 sums the responses, DIFIT.sub.k [n], within the time window n.sub.0 -nlow.sub.k.sbsb.0.sub.,k .ltoreq.n.ltoreq.n.sub.0 +nhigh.sub.k.sbsb.0.sub.,k to form response sum s. ##EQU2## If the value of the response sum s is greater than zero, at least one response has occurred in the response-time window, and the value of chan.sub.-- count is incremented by one 412. When the responses of all channels in the range k.sub.0 -klow.sub.k.sbsb.0 .ltoreq.k.ltoreq.k.sub.0 +khigh.sub.k.sbsb.0 have been surveyed, the value of chan.sub.-- count indicates the number of adjacent channels (including channel k.sub.0) that have at least one response in the appropriate response-time window. A normalized channel count norm.sub.-- chan 414 is determined by dividing the value of chan.sub.-- count by the total number of channels surveyed, khigh.sub.k.sbsb.0 -klow.sub.k.sbsb.0. If the value of norm.sub.-- chan is greater than a fractional acceptance criterion value, chan.sub.-- crit 416, then a local impulse is defined to have occurred in channel k.sub.0, and the value of the output sequence, IMPULSE.sub.k.sbsb.0 [n.sub.0 ] is set to one 418. Otherwise IMPULSE.sub.k.sbsb.0 [n.sub.0 ] is set to zero 420.
FIG. 5A is a schematic representation of the arrangement of time-response windows used in local impulse detector 118 of the neural-correlation implementation described with reference to FIG. 4. In this example, a response occurs in channel k=k.sub.0 at time n=n.sub.0 502. For this channel, khigh.sub.k0 =2 and klow.sub.k0 =2. A set of shaded boxes 504 indicates the extent of the response-time window for each channel. Here, nlow.sub.k.sbsb.0.sub.,k and nhigh.sub.k.sbsb.0.sub.,k depend upon k. Responses occur within the response-time windows for four of the five channels surveyed, hence the value of channel counter 412 is chan.sub.-- count=4 and the value the normalized channel count 414 is norm.sub.-- chan=0.8. Thus, if the value of the fractional acceptance criterion, chan.sub.-- crit 416 is no more than 0.8, the criterion for detection of a local impulse will have been met for channel k=k.sub.0 at time n=n.sub.0 and the value of the output sequence, IMPULSE.sub.k.sbsb.0 [n.sub.0 ], will be set to one.
In the synchronous epoch 208 of FIG. 2B, there are several distinct groups of responding channels. Each group comprises a plurality of adjacent channels that respond in synchrony with each other at a frequency that corresponds to a proximal formant frequency. In general, a synchronous interval can be defined as that range of k and n for which a number of adjacent channels all respond at approximately the same time. The synchronous interval is detected by a primary feature detector, the local synchrony detector 120. This detector produces a response in output channel k.sub.0 at time n.sub.0 if a response occurs in sequence DIFIT.sub.k [n] 114 for channel k=k.sub.0 at time n=n.sub.0 (i.e. DIFIT.sub.k.sbsb.0 [n.sub.0 ]=1), and responses occur simultaneously in a number of adjacent channels of lower and higher characteristic frequencies (i.e., DIFIT.sub.k [n]=1 for klow.sub.k.sbsb.0 .ltoreq.k.ltoreq.khigh.sub.k.sbsb.0 and nlow.sub.k.sbsb.0.sub.,k .ltoreq.n.ltoreq.nhigh.sub.k.sbsb.0.sub.,k, where klow.sub.k.sbsb.0 and khigh.sub.k.sbsb.0 represent, respectively, the range of channels that are in synchrony and nlow.sub.k.sbsb.0.sub.,k and nhigh.sub.k.sbsb.0.sub.,k represent the times over which synchrony is assessed.
The local synchrony detector 120 in the neural-correlation implementation is identical to the local impulse detector 118 described in conjunction with FIG. 4, except that, whereas local impulse detector 118 detects the sequential pattern of response, local synchrony detector 120 detects a simultaneous pattern of response of a number of adjacent channels. The block diagram of the local synchrony detector 120 is therefore identical to that of local impulse detector 118 depicted in FIG. 4. The only differences are the values of the time-response windows for each channel, nlow.sub.k.sbsb.0.sub.,k and nhigh.sub.k.sbsb.0,.sub.k, and possibly the range of adjacent channels surveyed for channel k: klow.sub.k.sbsb.0 and khigh.sub.k.sbsb.0. The output sequence for local synchrony detector 120 is denoted SYNCH.sub.k [n] 124 instead of IMPULSE.sub.k [n] 122.
FIG. 5B shows an example of the disposition of response-time windows for use in connection with local synchrony detector 120. In this example, a response occurs in channel k=k.sub.0 at time n=n.sub.0 506. For this channel, khigh.sub.k0 =2 and klow.sub.k0 =2. A set of shaded boxes 508 indicate the extent of the response-time windows for each adjacent channel. Here, nlow.sub.k.sbsb.0.sub.,k =nhigh.sub.k.sbsb.0.sub.,k= 1 for all k.sub.0 and k, so the response-time windows for all adjacent channels are cotemporaneous. Responses occur within the response-time windows for four of the five channels surveyed, hence the value of channel counter 412 is chan.sub.-- count=4 and the value of normalized channel count 414 is norm.sub.-- chan=0.8. Thus, if the value of acceptance criterion chan.sub.-- crit 416 is no more than 0.8, the criterion for detection of a local synchrony will have been met for channel k=k.sub.0 at time n=n.sub.0 and the value of the output sequence, SYNCH.sub.k.sbsb.0 [n.sub. 0 ], will be set to one.
The local impulse and synchrony detectors of the neural-correlation implementation represent a substantial improvement over prior art spectrographic methods; in the neural-correlation implementation, detection of features is temporally and spatially local and relatively unaffected by additive noise or spectral shaping of the input signal. However, there are two deficits with the neural-correlation implementation that limit its applicability: temporal granularity and computational intractability. These same deficits were discussed in connection with the prior-art method of Allen et al. (U.S. Pat. No. 4,905,285). Temporal granularity means that threshold-crossings of the neural response of each channel (i.e. DIFIT.sub.k [n] 114) occur only at discrete intervals, which can be spaced milliseconds apart in model channels with low center frequencies; hence, the response of local impulse and synchrony detectors, which require the input from several primary channels will be temporally coarse or granular. Computational intractability occurs because computing the complete response of the neural model channels (i.e. .lambda..sub.k [n] 110) requires the solution of linear and nonlinear cochlear model equations (as described in Appendix A) for 120 parallel channels in the preferred embodiment. The computational load of performing these calculations in real-time or near real-time is currently in excess of that permitted by even relatively large arrays of conventional general purpose CISC (e.g. type MC68040) RISC (e.g. type MC88100) or DSP (e.g. type MC56001) processors.
We now disclose details of the phase-coherence implementation of our invention. This implementation is preferred because it maintains the advantages of local detection of features such as local impulses and local synchrony, but improves upon the neural-correlation implementation by offering the additional advantages of fine temporal granularity and computational tractability. The significant improvements of the phase-coherence implementation result from the fact, which can be demonstrated from analysis of the peripheral auditory filtering stage 108 described in Appendix A, that the essential temporal information necessary for the detection of speech features such as the glottal pulses and formant frequencies is contained in correlations of .PHI..sub.k [n], the phase of the output of the cochlear-hydrodynamic stage of the auditory model 1910. Because .PHI..sub.k [n] can be computed with fine time resolution (equivalent to the sampling interval), the temporal granularity of the method can be arbitrarily fine. Because the cochlear-hydrodynamic stage 1910 of the auditory model is linear it is considerably simpler to implement than the complete model of the peripheral auditory system which includes substantial non-linear components. Hence, the computational efficiency of the phase-coherence implementation is also much improved.
FIG. 6 is a block diagram of the signal-processing stages that characterize the phase-coherence implementation of the invention. In this implementation, input sound p(t) 102 is transformed by signal converter 104 into discrete-time input sound sequence, p[n] 106 as described in conjunction with FIG. 1. Then, sequence p[n] 106 is processed by a cochlear signal processing stage 602, which is a filter bank comprising a plurality of channels, each channel, k, with a cochlear filter response function H.sub.k [l]. The cochlear signal processing stage 602, as well as the primary feature detection stage 606 and the secondary feature detection stage 616 to be described below, can be implemented using a processor arrangement incorporating, for example, a type MC68040 microprocessor and a type DSP56001 digital signal processor. The cochlear filter response functions of cochlear signal processing stage 602 summarize the signal processing operations characteristic of the outer 1904 and middle 1906 ear and cochlear hydrodynamics of the inner ear 1910 of the peripheral auditory filtering stage 108 which is described in Appendix A. The output of channel k of cochlear signal processing stage 602 is a pair of sequences representing the instantaneous magnitude, A.sub.k [n] 716, and phase, .PHI..sub.k [n] 718 of the response of one place on the basilar membrane of the cochlea to sound.
The instantaneous magnitude 716 and phase 718 sequences pass to a primary feature detection stage 606 which comprises a local impulse detector 608 of the phase-coherence implementation that detects patterns of spatial and temporal derivatives of the phase of basilar-membrane motion that correspond to the impulsive interval, and a local synchrony detector 610 of the phase-coherence implementation that detects patterns of spatial and temporal derivatives of the phase of the basilar-membrane motion that correspond to the synchronous interval. These local detectors are analogous to local impulse detector 118 and local synchrony detector 120 of the neural-correlation implementation described in connection with FIGS. 4 and 5. Each local feature detector of the phase-coherence implementation has a plurality of input sequences (i.e. A.sub.k [n] 716 and .PHI..sub.k [n] 718) and a plurality of output sequences. Local impulse detector 608 generates output sequences IMPULSE.sub.k [n] 612, and local synchrony detector 610 generates sequences SYNCH.sub.k [n] and AVG.sub.-- SYNCH.sub.k [n] 614. As discussed in connection with the neural-correlation implementation, these detectors are termed local to indicate that the value of each output sequence at any given time depends only upon a restricted subset of the channels of the input sequences over a small window of time.
The output sequences of primary feature detector stage 606 go to a secondary feature detector stage 616 which comprises a plurality of detectors. The secondary feature detectors include a local formant detector 618, which uses input from both local impulse detector 608 and local synchrony detector 610 to determine the spatio-temporal location of formants, producing an array of output sequences FORM.sub.k [n] 620. A spatial response filtering stage 622 sharpens the frequency extent of the formants detected by local formant detector 616 and produces an array of output sequences LFORM.sub.k [n] 624. The sequences LFORM.sub.k [n] represent the output of a signal processing method which responds preferentially to voiced speech at the time and frequency of occurrence of the formants.
The secondary feature detection stage 616 includes global feature detectors for glottal pulses and vowels. These detectors are termed global to indicate that the value of each output sequence at any given time depends upon most or all of the channels of the input sequences over some period of time. A global glottal-pulse detector 626 uses information from local impulse detector 608 to determine the temporal location of the glottal pulses, and generates an output sequence PPULSE[n] 628. A global vowel detector 630 operates on the input from the formant spatial response filtering stage 622 to produce an array of outputs that denote the presence of vowels, VOWEL.sub.k [n] 632.
FIG. 7A is a flow chart of the signal processing operations performed by cochlear signal processing stage 602 to obtain sequences of instantaneous magnitude, A.sub.k [n] 716, and instantaneous phase, .PHI..sub.k [n] 718. The operations diagrammed in this flow chart are particularly suited to implementation using a processor arrangement incorporating a type DSP56001 digital signal processor, though implementations incorporating a general purpose microprocessor, for example, a type MC68040 microprocessor, are not excluded. FIG. 7B is a schematic diagram of the waveforms that result from the processing operations depicted in FIG. 7A. In the preferred embodiment, the input sound p[n] 106 is segmented into frames N points in duration to form a framed input sequence, p.sub.i [n] 704, where the index i indicates the number of the frame and subsequent frames overlap by N/2 points. In the preferred embodiment, the sampling rate is approximately 8 kHz and N is 256 points, so each frame is approximately 32 milliseconds in duration. The discrete Fourier transformation of each frame of p.sub.i [n] 704 is taken using well-known fast Fourier transformation techniques to obtain a framed Fourier-transformed input sequence, P.sub.i [l] 706, where 0.ltoreq.l.ltoreq.N is the frequency index of each transform frame (only the magnitude of P.sub.i [l] is shown in FIG. 7B). For each channel, k, the framed Fourier-transformed input sequence P.sub.i [l] 706 is multiplied by the cochlear filter frequency-response function for that channel, H.sub.k [l], 0.ltoreq.l.ltoreq.N, to form an array of framed Fourier-transformed cochlear output sequences, X.sub.ki [l] 708. Because the cochlear filter frequency-response functions are sharply bandpass, the filtered output sequence, X.sub.ki [n] can be described as the real part of a framed complex cochlear-filtered output sequence Z.sub.ki [n]:
X.sub.ki [n]=Re{Z.sub.ki [n]}=Re{A.sub.ki [n]e.sup..PHI..sbsp.ki.sup.[n] }.
where A.sub.ki [n] is defined to be the instantaneous magnitude and .PHI..sub.ki [n] is defined to be the instantaneous phase. To derive A.sub.ki [n] and .PHI..sub.ki [n] for any channel, we use the fact that Z.sub.ki [l], the Fourier transform of Z.sub.ki [n], is the Hilbert transform of X.sub.ki [n] (Oppenheim, A. V. and Schafer, R. W. (1989): Discrete-time Signal Processing, Section 10.4. Englewood Cliffs (N.J.): Prentice-Hall). Z.sub.ki [l] is constructed from X.sub.ki [l], by setting the highest N/2 complex components of each frame of X.sub.ki [l] to zero and multiplying the remaining coefficients by two. ##EQU3## Thus, the array of N-point framed complex sequences Z.sub.ki [n] 712 is obtained by taking the N-point inverse Fourier transform of all the frames of Z.sub.ki [l]. In FIG. 7B, the real part of Z.sub.ki [n] is shown with solid lines and the imaginary part is shown with dashed lines. The frames of Z.sub.ki [n] are assembled using the well-known overlap-save technique with an overlap of N/2 to form a complex cochlear-filtered response, Z.sub.k [n] 714. Finally, the instantaneous magnitude, A.sub.k [n] 716 and phase, .PHI..sub.k [n] 718 are computed from Z.sub.k [n] 712 by rectangular-to-polar conversion: ##EQU4## In the preferred embodiment, the operations of Fourier transformation filtering are implemented on a high-speed, special purpose digital signal processor (e.g. type MC56001).
The instantaneous magnitude, A.sub.k [n] 716, and phase .PHI..sub.k [n] 718, of channel k can also be computed by other methods. For example, the complex output sequence, Z.sub.k [n] can be obtained by convolving the input sequence, p[n] 106 with the time-domain complex impulse response of the cochlear filter, h.sub.k [n]. For each channel, h.sub.k [n] can be obtained by setting the highest N/2 complex points of the cochlear filter frequency-response function, H.sub.k [l] to zero, and computing the inverse Fourier transform.
FIG. 8 is a flow chart illustrating a method for implementing local impulse detector 608 of the phase-coherence implementation. The principle of local impulse detector 608 is to detect a stereotyped pattern of spatial and time derivatives of basilar-membrane phase velocity that correspond to the sequential pattern of neural activity evoked by an impulse such as a glottal pulse. In the phase-coherence implementation, information on the relative timing of adjacent groups of nerve fibers is derived from d.PHI..sub.k [n]/dx, the spatial derivative of phase of basilar-membrane motion. Because adjacent channels in the preferred embodiment of cochlear signal processing stage 602 correspond to positions linearly spaced on the basilar membrane, dx is constant and d.PHI..sub.k [n]/dx is proportional to d.PHI..sub.k [n]/dk. In the preferred embodiment, for computational simplicity we compute the spatial phase derivative from the forward or backward difference of two adjacent channels (e.g. d.PHI..sub.k [n]/dx=.PHI..sub.k [n]-.PHI..sub.k-1 [n]); however, other means of computing the spatial and temporal derivatives of response phase are not excluded. Since the phase of the motion of the linear cochlear-hydrodynamic stage 1910 for each channel reflects the phase of the non-linear cochlear model of neural firing, .PHI..sub.k [n] can be computed efficiently (with high computational tractability) and with high temporal resolution (low temporal granularity).
A local impulse is defined to occur in the channel k=k.sub.0 at time n=n.sub.0 if the phase velocity d.PHI..sub.k [n]/dx is negative for n<n.sub.0 and becomes positive for n>n.sub.0 while the spatio-temporal phase velocity, that is the derivative of the spatial phase velocity with respect to time, d.sup.2 .PHI..sub.k [n]/dxdt, is greater than zero. The inputs to the local impulse detector 608 are the sequences of instantaneous magnitude, A.sub.k [n] 716 and instantaneous phase, .PHI..sub.k [n] 718, whose computation is discussed in connection with FIG. 7. For each channel, k, at each time, n, a magnitude threshold comparator 802 determines whether the magnitude A.sub.k [n] 716 exceeds a magnitude threshold, mag.sub.-- thresh. If A.sub.k [n]>mag.sub.-- thresh, then a phase comparator 804 tests whether the phase velocity of channel k is negative or positive. If the phase velocity is negative, a lag counter 806, lag.sub.-- count is incremented by one. This lag counter determines the length of time during which phase velocity is negative. When the phase velocity becomes zero or positive, an interval comparator 808 determines whether the cumulative time during which phase velocity was negative (i.e. lag.sub.-- count), exceeds a fixed value, lag.sub.-- max. If so, an impulse is said to be detected on channel k and the value of an output impulse sequence, IMPULSE.sub.k [n] is set to one 810, otherwise IMPULSE.sub.k [n] is set to zero 812. Then lag.sub.-- count is reset to zero 814, and the process is repeated for all times and channels.
In the preferred embodiment, phase comparator 804 tests only for positive or negative phase velocity and not for the condition of uniformly positive spatio-temporal phase velocity, d.sup.2 .PHI..sub.k [n]/dxdt.
In a ramification of local impulse detector 608 of the phase-coherence implementation which is not shown, a more general phase comparator is implemented which requires not only that the phase velocity be negative (i.e. d.PHI..sub.k [n]/dx<0) for a period exceeding lag.sub.-- max, but also that the spatio-temporal phase velocity be uniformly positive (i.e. d.sup.2 .PHI..sub.k [n]/dxdt>0). Furthermore, in the general phase comparator, the input to the phase comparator for each channel k.sub.0 is the phase .PHI..sub.k [n] of a plurality of channels adjacent to k.sub.0, not just the two channels of the preferred embodiment. The types of processing by the general phase comparator comprise linear and non-linear transformation of the input phase, as well as temporal and spatial derivatives of the input. That is, for channel k.sub.0, the output of the general phase comparator, PHASE.sub.-- COMP.sub.k.sbsb.0 [n], has the form: ##EQU5## where k.sub.low and k.sub.high represent the range of channels over which the phase comparison is made (which may be functions of k.sub.0) and f(.PHI..sub.k [n]) represents functions of phase that may include spatial derivatives of phase (e.g. d.PHI..sub.k [n]/dx), temporal derivatives (e.g. d.PHI..sub.k [n]/dt) and spatio-temporal derivatives (e.g. d.sup.2 .PHI..sub.k [n]/dxdt).
FIG. 9 is a plot of the response pattern to the utterance /a/ produced by 120 local impulse detectors of the phase-coherence method 608 in accordance with the method of FIG. 8. The responses of the local impulse detectors tend to occur at preferred times, forming a series of wavy lines 902 at intervals of the pitch periods.
An important attribute of local impulse detector 608 in the phase-coherence implementation is that the determination of output sequences, IMPULSE.sub.k [n] 608, is spatially and temporally local. This means that an impulse in channel k at time n is determined by examining only derivatives of spatial phase, d.PHI./dx, over a small number of channels around k, and only for a brief time interval around n. In the preferred embodiment, the spatial extent of the computation is less than a critical band and the temporal extent is about one millisecond, which is a small fraction of a pitch period. Because the computation of sequences IMPULSE.sub.k [n] 612 is based a local history of spatial derivatives of response phase, the output of local impulse detector 608 is highly insensitive to additive noise or spectral shaping of the input.
FIG. 10 is a flow chart illustrating a method for implementing local synchrony detector 610 of the phase-coherence implementation. The principle of local synchrony detector 610 is to detect a pattern of constant phase velocity on the basilar membrane that corresponds to the pattern of synchronous neural activity evoked by the formants. Local synchrony is defined to occur in channel k at time n if the phase velocity of a plurality of channels adjacent to k is similar, that is, the spatio-temporal phase velocity is nearly constant for a period of time. The inputs to local synchrony detector 610 are sequences of instantaneous magnitude, A.sub.k [n] 716 and phase, .PHI..sub.k [n] 718. For each channel, k, at each time, n, a magnitude threshold comparator 1002 determines whether magnitude A.sub.k [n] 716 exceeds a magnitude threshold, mag.sub.-- thresh. If A.sub.k [n]>mag.sub.-- thresh, then a synchrony comparator 1004 determines whether the spatial phase velocity is nearly zero. In the preferred embodiment, synchrony comparator 1004 is realized by an implementation which tests whether the phase velocities of two adjacent channels, k and k+ 1, are within a tolerance value, synch.sub.-- tol, of each other. If .vertline.d.PHI..sub.k [n]/dx.vertline.<synch.sub.-- tol, synchrony is said to be detected on channel k and the value of an output synchrony sequence, SYNCH.sub.k [n] is set to A.sub.k [n] 1006, otherwise SYNCH.sub.k [n] is set to zero 1008. A synchrony filtering stage 1010 filters sequence SYNCH.sub.k [n] to produce a filtered synchrony sequence, AVG.sub.-- SYNCH.sub.k [n] 614. In the preferred embodiment, a exponentially averaging filter is used:
AVG.sub.-- SYNCH.sub.k [n]=AVG.sub.-- SYNCH.sub.k [1]*decay.sub.-- fac+SYNCH.sub.k [n],
where the exponential averager time constant, decay.sub.-- fac, is a constant less than one. The synchrony filtering stage emulates the physiology of a cell acting as a leaky integrator, or averager. Cells with this property are well known and documented in the neurophysiological literature.
FIG. 11 is a plot of the response pattern to the utterance /a/ produced by the local synchrony detector 610 of the phase-coherence implementation. The sequences SYNCH.sub.k [n] are generated using the method described in conjunction with FIG. 10 using a value of synch.sub.-- tol that corresponds to 10 .mu.sec. The darkly shaded areas indicate spatio-temporal regions in which there is strong synchronous response.
FIG. 12 is a flow chart illustrating a method for implementing local formant detector 618. The formant detector receives input from both local impulse detector 608 (i.e. sequences IMPULSE.sub.k [n] 612) and local synchrony detector 610 (i.e. sequences AVG.sub.-- SYNCH.sub.k [n] 614). Each channel of local formant detector 618 produces an output when a synchronous response occurs on that channel within a given time window after a local impulse. In the preferred embodiment, formant detection is enabled by setting a formant time counter form.sub.-- enable to a positive value, form.sub.-- max 1204 when a local impulse occurs on channel k at time n (i.e. the sequence IMPULSE.sub.k [n]=1 1202). While form.sub.-- enable is greater than zero 1206, formant detection is enabled and a synchrony threshold comparator 1208 detects whether the magnitude of the averaged synchronized response, AVG.sub.-- SYNCH.sub.k [n] 614, is greater than a threshold value, synch.sub.-- thresh. If AVG.sub.-- SYNCH.sub.k [n]>synch.sub.-- thresh, a formant is defined to occur on the channel k at time n and the value of the output sequence FORM.sub.k [n] is set to a value of AVG.sub.-- SYNCH.sub.k [n] 1210. Otherwise FORM.sub.k [n] is set to zero 1212 and form.sub.-- enable is reset 1214.
The neurophysiological process of lateral inhibition is commonly used in cells throughout the nervous system to sharpen the spatial extent of responses.
FIG. 13 is a flow chart illustrating the operation of spatial response filtering stage 622 that can be used in connection with the formant detector illustrated in FIG. 12 to sharpen the spatial extent of formant detection. The spatial response filtering stage 622 detects the condition that the amplitude of the formant sequence, FORM.sub.k [n] 620 is maximum at some channel k=k.sub.0, and decreases for surrounding channels, both at higher values of k (i.e. values of k.sub.0 <k.ltoreq.k.sub.0 +khigh.sub.k.sbsb.0) and lower values of k (i.e. k.sub.0 -klow.sub.k.sbsb.0 <k.sub.0). The spatial filtering bandwidth, defined as the range of channels over which effective spatial response filtering takes place for channel k.sub.0, is between k.sub.0 +khigh.sub.k.sbsb.0 and k.sub.0 -klow.sub.k.sbsb.0, where the values of the constants khigh and klow depend on channel number, k.sub.0. In the preferred embodiment, the values of khigh.sub.k.sbsb.0 and klow.sub.k.sbsb.0 are chosen as a function of k so that the spatial filtering bandwidth is approximately two critical bands centered around k.sub.0. With reference to FIG. 13, for each time n, and for each channel k=k.sub.0, spatial response filtering stage 622 determines whether, for a range of channels from k=k.sub.0 1302 to k=k.sub.0 -klow.sub.k.sbsb.0 1304 the amplitude of the formant at k is less than the formant at the next lower channel, i.e. whether FORM.sub.k [n].gtoreq.FORM.sub.k-1 [n] 1306. If so, a counter, lsum 1308, is incremented by one. Similarly, for a range of channels from k=k.sub.0 1310 to k=k.sub.0 +khigh.sub.k.sbsb.0 1312, if the amplitude of the formant at k is less than the formant at the next higher channel, (i.e. if FORM.sub.k [n].gtoreq.FORM.sub.k+1 [n] 1314. counter lsum is incremented by one 1316. At the completion of this process, a normalized count, norm.sub.-- sum 1318, is computed by dividing lsum by spatial filtering bandwidth, norm.sub.-- lsum=lsum/(khigh.sub.k.sbsb.0 -klow.sub.k.sbsb.0). If the value of the normalized count exceeds a criterion value, (i.e. norm.sub.-- lsum>lat.sub.-- inhib.sub.-- crit 1320), a spatially filtered formant is said to be detected at k=k.sub.0 and n, and the value of an output sequence LFORM.sub.k.sbsb.0 [n] is set to FORM.sub.k.sbsb.0 [n] 1322. Otherwise, the value of LFORM.sub.k.sbsb.0 [n] is set to zero 1324.
FIG. 14 is a plot of the response pattern, LFORM.sub.k [n] 624, to the utterance /a/ produced by the local formant detector 618 in accordance with the implementation of FIGS. 12-13. The spatial and temporal localization of information shown in this representation represents a dramatic improvement over prior-art techniques for formant detection, most of which are based on spectrographic representations. Unlike the spectrographic representations of voiced speech that characterize the prior art, the representation of formant information in FIG. 14 is highly localized simultaneously in both the frequency and time domains. Because the method of formant detection described in our invention is based on spatial gradients of phase of response, the method is highly insensitive to additive noise or spectral shaping of the input. The detection of formants by the methods of this invention uses the fact that voiced portions of speech are characterized by the alteration of impulsive and synchronous epochs of response, hence the representation of formant information produced by this method is highly selective for the formants of speech and resistant to non-speech signals.
FIG. 15 is a flow chart illustrating the operation of the global pulse detector 626. This detector can be used to detect accurately the times of occurrence of glottal pulses in voiced utterances. The global pulse detector 626 produces an output when a given fraction of the outputs of local impulse detectors 608 have responded within a narrow time window. At each time, n.sub.0, a local impulse summation stage 1502 adds the response of the local impulse detectors for all channels, IMPULSE.sub.k [n] 612 over a small time window around n=n.sub.0 from n.sub.0 -nlow.ltoreq.n.ltoreq.n.sub.0 +nhigh to form an impulse summation, impulse.sub.-- count. The impulse summation is normalized by the total number of channels to form a normalized impulse count 1504:
norm.sub.-- count=impulse.sub.-- count/(k.sub.max -k.sub.min +1).
The normalized impulse count 1504 is compared to a criterion value, impulse.sub.-- crit 1506. If norm.sub.-- count>impulse.sub.-- crit, then a global pulse is defined to have occurred and the value of sequence PPULSE[n] is set to one 1508. Otherwise, PPULSE[n] is set to zero 1510. The top trace of FIG. 9 904 shows the response of the global pulse detector to the utterance /a/.
The method of locating global pulses from the output of an aggregate of local impulse detectors represents a novel improvement over the prior art. A significant feature of global pulse detector 626 described in conjunction with FIG. 15 that distinguishes it from the prior-art glottal-pulse detectors is that the temporal location of each glottal pulse is individually determined from spatial patterns of correlation of response phase from local ensembles of channels occurring in small periods of time. Hence, the output of global pulse detector 626 is highly insensitive to the magnitude of the input sound as well as to additive noise and spectral shaping of the input. Furthermore, each glottal pulse is individually localized within a fraction of a millisecond based on temporally local patterns of phase correlation. This contrasts with pitch detectors disclosed in the prior art, which are generally based on the temporal correlations of speech signals over long periods of time, usually greater than one pitch period. Temporal correlation methods are best suited to signals which are periodic or near-periodic. However, in speech, pitch is rarely constant and therefore the location of glottal pulses by correlation methods can produce highly variable and aperiodic results, compromising the performance of temporal correlation methods. The performance of prior-art methods of pitch detection also degrade rapidly in noise.
In a ramification of global pulse detector 626 of the phase-coherence implementation which is not shown, the global pulses are determined from the output of the local formant detector, 618. In this ramification, the output of all the channels of the local formant detector, 618 is summed at every time n to form a sequence SUM.sub.-- LFORM[n]: ##EQU6## Then, for each value of n, the value of SUM.sub.-- LFORM[n] is compared to a threshold value, sum.sub.-- lform.sub.-- high. If the value of SUM.sub.-- LFORM[n] exceeds the threshold value, then a global pulse is said to be detected and the value of sequence PPULSE[n] is set to one. Otherwise, PPULSE[n] is set to zero. The threshold comparator in this ramification has hysteresis. This means that after a global pulse is detected, the value of SUM.sub.-- LFORM[n] must drop below a threshold value sum.sub.-- lform.sub.-- low before another pulse can be detected.
The priorly disclosed invention of Allen et al. (U.S. Pat. No. 4,905,285) has suggested a method of determining the spectral frequencies of formants by analyzing inverse interval histograms accumulated from the response of model cochlear-nerve fibers. This approach suffers from a number of deficiencies, including the problems of temporal granularity and computational intractability we have discussed in conjunction with the neural threshold-crossing implementation of FIG. 1. Temporal granularity results from the attempt to estimate the frequency of neural discharge from each model nerve fiber. Because frequency is estimated from the time interval between pairs of neural pulses, estimates of frequency can only be made at discrete times, corresponding to the threshold crossings of the neutral responses, and are therefore temporally granular. Furthermore, because the variance of the time between neural firings increases in model nerve fibers with lower characteristic frequency, the response of these low-frequency fibers is not only more temporally granular than that of high-frequency fibers, but the estimated frequency has a higher variance as well. The method of Allen et al. (U.S. Pat. No. 4,905,285) is computationally intractable as well. It requires the solution of a nonlinear cochlear model with a number of fibers (85 fibers in the preferred embodiment) to obtain the times of neural firing. Finally, the method of Allen et al. (U.S. Pat. No. 4,905,285) is not inherently speech specific: it does not distinguish speech from non-speech sounds.
In contrast to the prior art, the present invention provides an implementation, which we term the instantaneous-frequency implementation, for deriving information on speech features such as formant frequencies and glottal pulses from the frequency of response of channels of the cochlear signal processing stage 602 described in conjunction with FIG. 6 which improves upon the method of Allen et al. (U.S. Pat. No. 4,905,285). The instantaneous-frequency implementation is temporally fine-grain and computationally tractable and inherently specific for the detection of speech features. The basis of the implementation is the finding, demonstrated by analysis of the peripheral auditory filtering stage 108 described in Appendix A, that the temporal relationships between threshold-crossings in the output of the cochlear model of neural output are preserved in the instantaneous frequency of each channel of the underlying basilar membrane motion. Because the cochlear signal processing stage 602 of the model of the peripheral auditory system is linear, the computational efficiency of the frequency calculations is improved. In this implementation, input sound p(t) 102 is transformed by signal converter 104 into discrete-time input sound, sequence, p[n] 106. Input sequence 106 is processed by cochlear signal processing stage 602 to produce sequences of instantaneous magnitude, A.sub.k [n] 716 and phase, .PHI..sub.k [n] 718, as described in connection with FIG. 7 and Appendix A. The instantaneous frequency, f.sub.k [n], of channel k is represented by the change in instantaneous phase as a function of time: ##EQU7## where dn/dt is the sampling rate of the sequences. Because the phase, .PHI..sub.k [n], of each channel of the peripheral model can be accurately estimated with high temporal resolution, it is possible to obtain an estimate of the time derivative, d.PHI..sub.k [n]/dt, with high temporal resolution. In the embodiment depicted in FIG. 17, for computational simplicity we compute the discrete-time derivative, d.PHI..sub.k [n]/dn, from the forward or backward difference of two time points (i.e. d.PHI..sub.k [n]/dn=.PHI..sub.k [n]-.PHI..sub.k [n-1]); however, other means of computing the time derivative are not excluded.
FIG. 16 is a plot of the response pattern of f.sub.k [n] produced by the instantaneous-frequency implementation to a voiced utterance, /a/. The response consists of two distinct spatio-temporal patterns, an impulsive epoch 1602 and a synchronous epoch 1604 occurring in alternation. These epochs are equivalent to the impulsive 202 and synchronous 204 epochs previously described in connection with FIG. 2A. In the impulsive epoch 1602, elicited by each glottal pulse of the utterance, the instantaneous frequency response shows a characteristic `ladder-like` pattern, where each `rung` of the ladder represents the instantaneous frequency of one channel responding at or near that channel's center frequency, CF.sub.k, for a brief period of time (approximately two milliseconds). In the synchronous epoch 1604, the responses of most of the model channels become entrained into a plurality of distinct bands of frequency corresponding to the resonances of the vocal tract, the formants, for a brief period of time (2-5 milliseconds). In this epoch, the instantaneous frequencies of the responses of groups of channels are approximately the same, and correspond to the frequency of the proximal formant. The local impulse detector and local synchrony detector of the instantaneous-frequency implementation are similar to those described for the phase-coherence implementation in conjunction with FIGS. 8 and 10, but are implemented using information from histograms of instantaneous frequency (which is proportional to d.PHI..sub.k [n]/dn) computed for a plurality of channels rather than from information on the spatial derivatives of response phase (which is proportional to d.PHI..sub.k [n]/dk).
FIG. 17 is a flow chart illustrating a method for implementing the local impulse detector of the instantaneous-frequency implementation. The purpose of the local impulse detector is to detect the `ladder-like` pattern 1602 produced in the impulsive interval of the instantaneous frequency response. With reference to FIG. 17, the instantaneous frequency, f.sub.k [n] 1701, is computed for every channel, k, and every time n for which the magnitude of the input sequence exceeds a threshold value, A.sub.k [n]>mag.sub.-- thresh 1702. A center-frequency comparator 1704 is used to determine whether the instantaneous frequency of channel k is equivalent to that channel's center frequency, denoted CF.sub.k. In the embodiment depicted in FIG. 17, the center-frequency comparator 1704 determines whether the average deviation of f.sub.k [n] from CF.sub.k is less than a constant value, freq.sub.-- tol:
.vertline.f.sub.k [n]/CF.sub.k -1.vertline.<freq.sub.-- tol
If so, a counter, int.sub.-- count 1706, is incremented. This counter measures the length of the impulsive interval for channel k, the period during which the instantaneous frequency of channel k is within a given tolerance of the channel's center frequency. If the instantaneous frequency of channel k differs by more than the tolerance value from the channel's center frequency, the impulsive interval ends, and an interval comparator 1708 determines whether the length of the impulsive interval exceeds a fixed value, int.sub.-- max. If so, an impulse is said to be detected on channel k and the value of an output impulse sequence, IMPULSE.sub.k [n] is set to one 1710, otherwise IMPULSE.sub.k [n] is set to zero 1712. Then int.sub.-- count is reset to zero 1714, and the process is repeated for all times and channels.
FIG. 18 is a flow chart illustrating a method for implementing the local synchrony detector of the instantaneous-frequency implementation. In this implementation, a frequency histogram of M+1 bins is formed with bins B.sub.i,0.ltoreq.i.ltoreq.M. In the embodiment depicted in FIG. 18, the bins of the frequency histogram span a frequency range from f.sub.min to f.sub.max in equal logarithmic increments, where f.sub.min and f.sub.max represent, respectively, the center frequencies corresponding to the low and high channels of cochlear signal processing stage 602, k.sub.min and k.sub.max. For each channel, k, and time, n, a magnitude threshold comparator 1802 determines whether the magnitude of the response, A.sub.k [n] 716 exceeds a magnitude threshold, A.sub.k [n]>mag.sub.-- thresh. If so, the instantaneous frequency, f.sub.k [n], is computed 1804, and the bin of frequency histogram, j, corresponding to this frequency is determined 1806 as
j=int(.alpha. log f/f.sub.min),
where int() denotes the integer part operator and .alpha.=M/log (f.sub.max /f.sub.min) is a constant. Bin j of the frequency histogram is then incremented by one 1808. This procedure is repeated for all channels, k, to construct the frequency histogram B.sub.i [n] at a given time, n. Peaks in this frequency histogram indicate values of frequency at which a plurality of channels respond in synchrony. The frequency histogram, B.sub.i [n] gives a similar representation to SYNCH.sub.k [n] 1006.
A histogram filtering stage 1810 filters histogram B.sub.i [n] to produce a filtered frequency histogram, AVG.sub.-- B.sub.i [n]. In the embodiment depicted in FIG. 18, an exponentially averaging filter is used:
AVG.sub.-- B.sub.i [n]=AVG.sub.-- B.sub.i [n-1]*decay.sub.-- fac+B.sub.i [n],
where the exponential averager time constant, decay.sub.-- fac, is less than one. The filtered frequency histogram, AVG.sub.-- B.sub.i [n] gives a similar representation to AVG.sub.-- SYNCH.sub.k [n] 614. A secondary feature detection stage of the instantaneous-frequency implementation, which detects formants, glottal pulses and vowels is identical to the secondary feature detection stage of the phase-correlation implementation 616 described in conjunction with FIGS. 6, 12, 13 and 15.
In summary, we have described several methods for analyzing speech to detect and categorize linguistically important features, such as the frequencies and times of formants and glottal pulses. Representations of the formants and glottal pulses are obtained from local and global temporal and spatial correlations of the response computed from a plurality of channels of a cochlear signal processing stage. The features detected by this method are highly unaffected by additive noise and insensitive to spectral shaping or bandwidth reduction of the input. They are selective for the detection of speech features such as glottal and formant information. They do not require critical data segmentation. They simultaneously show high temporal and frequency resolution. Results can be obtained in a computationally efficient manner.
The scope of the invention should be determined by the appended claims and their legal equivalents, rather than by the examples given.
Appendix A--Description of an implementation of a model of the peripheral auditory system
FIG. 19 shows a block diagram of the peripheral auditory filtering stage 108 based on a model of signal processing by the peripheral auditory system which motivates the principal implementations of the invention. An incident sound p(t) 102 is processed by linear filters which emulate the filtering characteristics of the outer 1904 and middle 1906 ears to produce an output U(t) 1908 which describes the volume velocity of the stapes. The volume velocity of the stapes is input into a cochlear hydrodynamic filtering stage 1910, a filter bank which comprises a plurality of sharply frequency-selective bandpass filters whose frequency responses are derived from an analysis of a three-dimensional hydromechanical model of the fluid wave processes of the cochlea which comprises part of the peripheral auditory filtering stage 108. In the preferred embodiment, the cochlear hydrodynamic filtering stage 1910 has 120 channels corresponding to the response at positions linearly spaced along the basilar membrane. Each channel, k, has a different frequency of maximum response, the center frequency, CF.sub.k. The center frequency as well as other characteristics of each filter channel vary as a function of k, which represents the spatial dimension of the response. The linear spacing of channels of the model of the basilar membrane results in values of CF.sub.k which are approximately logarithmically spaced in frequency over the range CF.sub.0 250 Hz (for k=0) to CF.sub.119 =3.4 kHz (for k=119).
FIG. 20 is a plot of representative cochlear frequency-response functions, H.sub.k [l] 2002, for 12 channels selected to cover the range 250 Hz to 3.4 kHz. While we have chosen to derive these filter functions from consideration of models and data of the auditory system, we do not exclude other filter response functions for H.sub.k [l] derived by other means. For each channel of the cochlear hydrodynamic filtering stage 1910, the output is a waveform x(t) 1912 that represents the motion of the basilar membrane of the cochlea at a location characteristic of the frequency, CF.sub.k. In the preferred embodiment, x(t) is determined at a temporal resolution of approximately 15 .mu.sec by the process described in connection with FIG. 7.
Each channel of cochlear hydrodynamic filtering stage 1910 is followed by an instantaneous, non-linear hair-cell filtering stage 1914 which represents the mechanical-to-electrical transduction process in the hair cells in the organ of Corti. This hair-cell filtering stage can be described by a lumped-element circuit model shown in FIG. 21A. The hair cell's nonlinear transduction conductance, G.sub.T, is a function of the displacement of the hair-cell cilia x(t): ##EQU8## The cell's membrane potential, V.sub.M (t) can be obtained by solving the circuit of FIG. 21A: ##EQU9## where
G.sub..SIGMA. (t)=G.sub.T (x(t))+G.sub.P +G.sub.L +G.sub.K (3)
and
V.sub..SIGMA. (t)=(V.sub.E G.sub.T (x(t))+V.sub.K G.sub.K +V.sub.L G.sub.L)/G.sub..SIGMA. (t). (4)
The circuit parameters are given as:
______________________________________Parameter Description Value______________________________________V.sub.E Endolymphatic potential 100 mVV.sub.K Potassium equilibrium potential -80 mVV.sub.L Nonspecific leakage potential -30 mVG.sub.L Nonspecific leakage conductance 1 nSG.sub.K Potassium conductance 26.1 nSG x-independent free-energy 1.26 of transductionZ x-dependent free-energy 7.1 of transductionG.sub.T.sup.MAX Maximum transduction conductance 7.5 nSG.sub.P Electrode leakage conductance 28.7 nSC Hair cell capacitance 15 pF______________________________________
The output of hair-cell filtering stage 1914 is a hair-cell membrane potential, V.sub.M (t) 1916 which is input into a calcium-channel filtering stage 1918 derived from a biophysically based model of the hair cell's calcium channel (Lewis, R. S. (1985): The ionic basis of frequency selectivity in hair cells of the bullfrog's sacculus. Ph.D. Thesis, California Institute of Technology). The calcium-channel filtering stage 1918 relates the calcium current in a hair cell, I.sub.ca (t) 1920, to the cell's membrane potential:
I.sub.Ca (t)=G.sub.Ca m.sup.3 (t)(V.sub.M (t)-V.sub.Ca), (5)
where the kinetics of the calcium channel are represented by the activation parameter, m, which is controlled by first-order kinetics, ##EQU10## and the kinetic rate constants, .alpha..sub.M and .beta..sub.M are functions of the membrane potential:
.alpha..sub.M (V.sub.M)=.alpha..sub.0 e.sup.-(V.sbsp.M.sup.+V.sbsp.0.sup.)/V.sbsp.A.sup.)+k.sbsp.A,
.beta..sub.M (V.sub.M)=.beta..sub.0 e.sup.+(V.sbsp.M.sup.+V.sbsp.0.sup.)/V.sbsp.B.sup.)+k.sbsp.B.(7)
The parameters of calcium channel filtering stage 1918 at 37.degree. C. are given as follows:
______________________________________Parameter Description Value______________________________________G.sub.Ca Calcium channel conductance 6.14 nSV.sub.Ca Calcium equilibrium potential 100 mV.alpha..sub.O 118472 sec.sup.-1.beta..sub.O 5.04 sec.sup.-1k.sub.A Closing kinetic constant 2650 sec.sup.-1k.sub.B Opening kinetic constant 4884 sec.sup.-1V.sub.A Closing voltage sensitivity 8.01 mVV.sub.B Opening voltage sensitivity 6.17 6.17 mVV.sub.O Voltage offset 70 mV______________________________________
The hair-cell 1914 and calcium-channel 1918 filtering stages are followed by a synaptic filtering stage 1922. This is a time-varying non-linear filtering stage that represents the neuro-chemistry of the synapse between the hair-cell and the afferent fibers of the cochlear nerve. FIG. 21B shows a block diagram of the synaptic filtering stage 1922. This filtering stage describes the relation between the calcium current, I.sub.Ca (t) 1920 and the (normalized) concentration of neuro-transmitter in the synaptic cleft, c(t) 1924. Synaptic filtering stage 1922 is described by three coupled differential equations with non-constant coefficients, as given by Meddis (Meddis, R. (1986): Simulation of mechanical to neural transduction in the auditory receptor. J. Acoust. Soc. Am. 79: 702-711; Meddis, R. (1988): Simulation of auditory-neural transduction: further studies. J. Acoust. Soc. Am. 83: 1056-1063). ##EQU11## where q(t) and w(t) represent the normalized concentrations of neuro-transmitter in a reprocessing store 2102 and a free transmitter pool 2104 respectively, and x, y, l and r are rate constants describing the turnover of transmitter among the stores of the model. The presynaptic permeability, k(t) is taken to be proportional to the hair cell's calcium current, I.sub.Ca (t) 1920 in the speech frequency range:
k(t)=k.sub.0 +.xi.k.sub..DELTA..sup.MAX I.sub.Ca (t). (9)
The parameters of the synaptic filtering stage 1922 are:
______________________________________Parameter Description Value______________________________________.chi. Reprocessing store turn-over rate 85.9y Neurotransmitter production rate 15.1l Synapse loss rate 1994.7r Synapse reabsorbsion rate 8007.6k.sub.O Resting presynaptic permeability 16.1k.sub..DELTA..sup.MAX Maximum permeability increase 297.3.xi. Calcium coupling factor -1.67______________________________________
A neural firing stage 1926 relates the concentration of neurotransmitter in the cleft 1924 to the probability density function of discharge of action potentials in the primary cochlearnerve fibers that carry information to the brain, .lambda.(t) 1928. In one implementation of this model, .lambda.(t) is taken to be directly proportional to c(t).
The peripheral auditory filtering stage 108 is realized by implementing the computations of Equations 1-9 using a processor arrangement incorporating, for example, a type MC68040 microprocessor or type DSP56001 digital signal processor. In this implementation, the incident sound p(t) 102 is transformed by a signal converter 104 into a discrete-time sound sequence, p[n] 104. Then, discrete-time sequences with index n analogous to sampled versions of functions of a continuous variable, t, are computed within the processor as follows:
The equivalent discrete-time output of the implementation of the cochlear hydrodynamic filtering stage, 1910 is an array of sequences x.sub.k [n], whose computation is implemented either by means of the Fourier transformation techniques similar to those described in conjunction with FIG. 7, or by direct convolution of p[n] with h.sub.k [n], the impulse response obtained by inverse Fourier transformation of the cochlear frequency-response functions, H.sub.k [l] 2002 by methods well known to the art. In the preferred embodiment of peripheral auditory filtering stage 108, there are 120 channels of H.sub.k [l],0.ltoreq.k.ltoreq.119, which span the frequency range of approximately 250 Hz to 3.4 kHz.
Discrete-time sequences for G.sub.T [n] (Equation 1), G.sub..SIGMA. [n] (Equation 3), V.sub..SIGMA. [n] (Equation 4), are directly computed by substituting x[n] into the equations in place of x(t). The membrane potential sequence, V.sub.M [n], is obtained by solving Equation 2 with a standard numerical integration method, such as the Runge-Kutta technique (Press, W. H., Flannery, B. P., Teukolsky, S. A. and Vetterling, W. T. (1988): Numerical Recipes in C., Chapter 15. Cambridge: Cambridge University Press). The discrete-time sequence for the calcium current, I.sub.Ca [n] (Equation 5) is computed from the membrane potential sequence, V.sub.M [n], and the activation parameter, m[n], where m[n] is obtained by solving Equation 6 with a standard numerical integration method. Sequences for the kinetic rate constants, .alpha..sub.M and .beta..sub.M are obtained from Equations 8 by substitution of V.sub.M [n]. The presynaptic permeability sequence k[n] is obtained from Equation 9 by substitution of I.sub.Ca [n]. The discrete-time sequence for concentration of neuro-transmitter in the synaptic cleft, c[n] is obtained from Equation 8 by solution of the coupled equations by standard numerical methods. The sequence representing the probability density function, .lambda.[n] is obtained from Equation 9 by substitution of the sequence c[n]. The array of sequences .lambda..sub.k [n] 110 is created by solving the response of the model equations for all channels, 0.ltoreq.k.ltoreq.119.
FIG. 2 shows the response of the model of the peripheral auditory system to the utterance /a/. This response pattern accurately simulates the response recorded from auditory-nerve fibers in mammalian ears to speech utterances (Shamma, S. (1985): Speech processing in the auditory nervous system, I: The representation of speech sounds in the responses of the auditory nerve. J. Acoust. Soc. Am. 78, 1612-1621).

Claims

1. A method for analyzing a stimulus waveform comprising:
a) providing a signal processing means which simulates the response properties of the human cochlea by supplying a plurality of output waveforms each of which corresponds to the response of said simulated cochlea at a selected location along its length;
b) applying said stimulus waveform to the input of said simulated cochlea;
c) processing said stimulus waveform by said signal processing means to produce a plurality of output waveforms at selected locations along the length of said simulated cochlea;
d) detecting primary features of said stimulus waveform by comparing the spatial and time relationships of said output waveforms including their spatial and time derivatives with stereotypical output waveforms and their spatial and time derivatives;
e) producing a plurality of sequences of said primary features at selected locations along said simulated cochlea;
f) detecting secondary features of said stimulus waveform by comparing said sequences of primary features with stereotypical patterns of said primary features;
g) producing a plurality of sequences of said secondary features; and
h) producing an output representation comprising one or more sequences each of which results from detecting combined spatial and temporal relationships of said primary and secondary feature sequences.
2. The method for analyzing the stimulus waveform of claim 1 wherein said stimulus waveform is an acoustic stimulus such as sound or speech.
3. The method for analyzing the stimulus waveform of claim 1 wherein said simulated cochlea comprises a plurality of band-pass filter channels, each of said filters substantially matching the frequency response characteristic at a selected location along said simulated cochlea, with the timing and phase characteristics of the output of the ensemble of said filters substantially matching waveform propagation characteristics of the human cochlea.
4. The method for analyzing the stimulus waveform of claim 3 wherein said selected locations are equally spaced along the length of said simulated cochlea, approximately corresponding to logarithmic spacing of the center frequencies of said band-pass filter channels.
5. The method for analyzing the stimulus waveform of claim 1 wherein the step of detecting primary features comprises the comparison of said output waveforms and their spatial and time derivatives with stereotypical output waveforms and their spatial and time derivatives resulting from an impulsive stimulus waveform.
6. The method of claim 5 wherein the step of detecting primary features resulting from an impulsive stimulus waveform comprises:
a) producing from said simulated cochlea a plurality of output sequences comprising an instantaneous magnitude sequence, A.sub.k [n], and an instantaneous phase sequence, .PHI..sub.k [n], where channel index k corresponds to location along said simulated cochlea and n corresponds to discrete-time index;
b) computing a spatial phase derivative sequence, d.PHI..sub.k [n]/dx, from the difference of said .PHI..sub.k [n] sequences of two adjacent channels;
c) computing a spatio-temporal phase derivative sequence, d.sup.2 .PHI..sub.k [n]/dx dt, from the difference of d.PHI..sub.k [n]/dx sequences for two sequential values of n;
d) determining for each selected channel index k.sub.o and selected time index n.sub.o whether all of the following conditions are met for a predetermined interval of time near n.sub.o, representing stereotypical response patterns for an impulsive stimulus waveform:
i) whether said instantaneous magnitude exceeds a predetermined magnitude threshold;
ii) whether said spatial phase derivative is negative for n<n.sub.o and becomes positive for n>n.sub.o ;
iii) whether said spatio-temporal phase derivative is generally greater than zero;
e) producing a plurality of primary feature sequences, IMPULSE.sub.k [n], which is set to one when said conditions are met and zero when they are not met.
7. The method of claim 5 wherein the step of detecting primary features resulting from and impulsive stimulus waveform comprises:
a) producing from said simulated cochlea a plurality of output sequences comprising an instantaneous magnitude sequence, A.sub.k [n], and an instantaneous frequency sequence, f.sub.k [n], where channel index k corresponds to location along said simulated cochlea and n corresponds to discrete-time index;
b) determining for each selected channel index k.sub.o and selected time index n.sub.o whether all of the following conditions are met, representing stereotypical response patterns for an impulsive stimulus waveform:
i) whether said instantaneous magnitude exceeds a predetermined magnitude threshold;
ii) whether said instantaneous frequency is approximately equal to the center frequency of said selected channel for a predetermined interval of time;
c) producing a plurality of primary feature sequences, IMPULSE.sub.k [n], which is set to one when said conditions are met and zero when they are not met.
8. The method for analyzing the stimulus waveform of claim 1 wherein the step of detecting primary features comprises the comparison of said output waveforms and their spatial and time derivatives with stereotypical output waveforms and their spatial and time derivatives resulting from a sinusoidal stimulus waveform.
9. The method of claim 8 wherein the step of detecting primary features resulting from a sinusoidal stimulus waveform comprises:
a) producing from said simulated cochlea a plurality of output sequences comprising an instantaneous magnitude sequence, A.sub.k [n], and an instantaneous phase sequence, .PHI..sub.k [n], where a channel index k corresponds to location along said simulated cochlea and n corresponds to discrete time index;
b) computing a spatial phase derivative sequence, d.PHI..sub.k [n]/dx, from the difference of said .PHI..sub.k [n] sequences of two adjacent channels;
c) determining for each selected channel index k.sub.o and selected time index n.sub.o whether all of the following conditions are met for a predetermined interval of time near n.sub.o, representing stereotypical response patterns for a sinusoidal stimulus waveform:
i) whether said instantaneous magnitude exceeds a predetermined magnitude threshold;
ii) whether said spatial phase derivative is approximately constant;
d) producing a plurality of primary feature sequences, SYNCH.sub.k [n], which are set to A.sub.k [n] when said conditions are met and zero when they are not met.
10. The method of claim 8 wherein the step of detecting primary features resulting from a sinusoidal stimulus waveform comprises:
a) producing from said simulated cochlea a plurality of output sequences comprising an instantaneous magnitude sequence, A.sub.k [n], and an instantaneous frequency sequence, f.sub.k [n], where channel index k corresponds to location along said simulated cochlea and n corresponds to discrete-time index;
b) determining for each selected channel index k.sub.o and selected time index n.sub.o whether all of the following conditions are met, representing stereotypical response patterns for a sinusoidal stimulus waveform:
i) whether said instantaneous magnitude exceeds a predetermined magnitude threshold;
ii) whether a preponderance of substantially adjacent channels have approximately the same instantaneous frequency;
c) producing a plurality of primary feature sequences, SYNCH.sub.k [n], which is set to A.sub.k [n] when said conditions are met and zero when they are not met.
11. The method for analyzing the stimulus waveform of claim 1 wherein the step of detecting secondary features comprises the comparison of patterns of said sequences of primary features with a stereotypical pattern of said sequences of primary features resulting from a human glottal pulse stimulus waveform.
12. The method of claim 11 wherein the step of detecting secondary features resulting from a glottal pulse stimulus waveform comprises:
a) producing from said simulated cochlea a plurality of primary feature sequences, IMPULSE.sub.k [n], where channel index k corresponds to location along said simulated cochlea and n corresponds to discrete-time index;
b) forming an impulse summation by adding the values of IMPULSE.sub.k [n] for all channels over a predetermined interval of time around a selected time index n.sub.o ;
c) determining for said selected time index n.sub.o whether the following condition is met, representing the stereotypical response pattern for a glottal pulse stimulus waveform:
i) whether said impulse summation exceeds a predetermined threshold;
d) producing a secondary feature sequence, PPULSE[n], which is set to one when said condition is met and zero when it is not met.
13. The method for analyzing the stimulus waveform of claim 1 wherein the step of detecting secondary features comprises the comparison of said sequences of primary features with a stereotypical pattern of said sequences of primary features resulting from a human vowel stimulus waveform.
14. The method of claim 13 wherein the step of detecting secondary features resulting from a human vowel stimulus waveform comprises:
a) producing from said simulated cochlea a plurality of primary feature sequences, IMPULSE.sub.k [n] and SYNCH.sub.k [n], where channel index k corresponds to location along said simulated cochlea and n corresponds to discrete-time index;
b) determining for each selected time index n.sub.o whether the following conditions are met, representing the stereotypical response pattern for a human vowel stimulus waveform:
i) whether in a predetermined interval of time preceding n.sub.o the value of IMPULSE.sub.k [n] is non-zero;
ii) whether the value of SYNCH.sub.k [n.sub.o ] is non-zero;
c) producing a plurality of secondary feature sequences, FORM.sub.k [n], which is set to SYNCH.sub.k [n] when said conditions are met and zero when they are not met.
15. The method of claim 14 wherein the plurality of primary feature sequences SYNCH.sub.k [n] is replaced by a plurality of primary feature sequences AVG.sub.-- SYNCH.sub.k [n] produced by an averaging filter.
16. The method for analyzing the stimulus waveform of claim 1 wherein the step of producing an output representation comprises the comparison of said sequences of secondary features with a stereotypical pattern of said sequences of secondary features resulting from a human vowel stimulus waveform.
17. The method of claim 16 wherein the step of producing an output representation resulting from a human vowel stimulus waveform comprises:
a) producing from said simulated cochlea a plurality of secondary feature sequences, FORM.sub.k [n], where channel index k corresponds to location along said simulated cochlea and n corresponds to discrete-time index;
b) determining for each selected channel index k.sub.o and each selected time index n.sub.o whether the following condition is met, representing the stereotypical response pattern for a human vowel stimulus waveform:
i) whether the value of the formant sequence FORM.sub.k [n] is maximum at k.sub.o and decreases for substantially adjacent channels;
c) producing a plurality of output representation sequences, LFORM.sub.k [n], which is set to FORM.sub.k [n] when said condition is met and zero when it is not met.
18. The method for analyzing the stimulus waveform of claim 1 wherein the step of processing said stimulus waveform by said signal-processing means comprises:
a) converting the stimulus waveform by means of a signal converter to form a discrete-time input sequence p[n], where n is the discrete-time index;
b) segmenting said input sequence into frames p.sub.i [n] of predetermined length N;
c) computing a discrete Fourier transform, P.sub.i [l], of each frame of said input sequence, with index l corresponding to discrete Fourier frequency;
d) providing H.sub.k [l], the filter response function of said simulated cochlea, with channel index k corresponding to location along said simulated cochlea;
e) multiplying P.sub.i [l] by H.sub.k [l] to form X.sub.ki [l];
f) constructing a Hilbert transform sequence, Z.sub.ki [l], by setting to zero the last N/2 points of X.sub.ki [l] and multiplying the first N/2 points by two;
g) computing an inverse discrete Fourier transform of Z.sub.ki [l] to form frames of a complex filter response sequence z.sub.ki [n];
h) assembling frames of z.sub.ki [n] to form a complex cochlear-filtered response sequence z.sub.k [n];
i) computing the instantaneous magnitude sequence, A.sub.k [n], and the instantaneous phase sequence, .PHI..sub.k [n], from the complex sequence z.sub.k [n]; and
j) providing a plurality of output sequences comprising said instantaneous magnitude sequence and said instantaneous phase sequence for each said channel index k.
19. The method for analyzing the stimulus waveform of claim 1 wherein the step of processing said stimulus waveform by said auditory model means comprises:
a) producing from said simulated cochlea a plurality of output sequences comprising an instantaneous magnitude sequence, A.sub.k [n], and an instantaneous phase sequence, .PHI..sub.k [n], where channel index k corresponds to location along said simulated cochlea and n corresponds to discrete-time index;
b) computing an instantaneous frequency sequence, f.sub.k [n], from the scaled difference of said .PHI..sub.k [n] sequences for two sequential values of n;
c) providing a plurality of output sequences comprising said instantaneous magnitude sequence and said instantaneous frequency sequence for each said channel index k.
20. An apparatus for analyzing a stimulus waveform comprising:
a) a signal processing means which simulates the response properties of the human cochlea by supplying a plurality of output waveforms each of which corresponds to the response of said simulated cochlea at a selected location along its length;
b) means for applying said stimulus waveform to the input of said simulated cochlea;
c) filtering means in said signal processing means for producing a plurality of output waveforms and their spatial and time derivatives at selected locations along the length of said simulated cochlea;
d) first pattern matching means for comparing said output waveforms and their spatial and time derivatives with stereotypical output waveforms and their spatial and time derivatives to provide sequences of primary features of said stimulus waveform;
e) means for producing a plurality of said sequences of said primary features at selected locations as said stimulus waveform is propagated along said simulated cochlea;
f) second pattern matching means for comparing said sequences of primary features with stereotypical patterns of said primary features to provide a sequence of secondary features of said stimulus waveform;
g) means for producing a plurality of said sequences of said secondary features; and
h) third pattern matching means for detecting combined spatial and temporal relationships of said primary and secondary feature sequences to produce an output representation comprising one or more sequences.
21. The apparatus as described in claim 20 wherein the input stimulus waveform is an acoustic stimulus such as sound or speech.
22. The apparatus as described in claim 20 wherein said filtering means comprises a plurality of band-pass filters, each of said filters substantially matching the frequency response characteristic at a selected location along said simulated cochlea, with the timing and phase characteristics of the output of the ensemble of said filters substantially matching waveform propagation characteristics of the human cochlea.
23. The apparatus as described in claim 20 wherein said selected locations are equally spaced along the length of said simulated cochlea, approximately corresponding to logarithmic spacing of the center frequencies of said band-pass filter channels.
24. The apparatus of claim 20 wherein said first pattern matching means includes means for comparing said output waveforms and their spatial and time derivatives with stereotypical output waveforms and their spatial and time derivatives resulting from an impulsive stimulus waveform.
25. The apparatus of claim 20 wherein said first pattern matching means includes means for comparing said output waveforms and their spatial and time derivatives with stereotypical output waveforms and their spatial and time derivatives resulting from a sinusoidal stimulus waveform.
26. The apparatus of claim 20 wherein said second pattern matching means includes means for comparing patterns of said sequences of primary features with a stereotypical pattern of said sequences of primary features resulting from a human glottal pulse stimulus waveform.
27. The apparatus of claim 20 wherein said second pattern matching means includes means for comparing said sequences of primary features with a stereotypical pattern of said sequences of primary features resulting from a human vowel stimulus waveform.
28. The apparatus of claim 20 wherein said third pattern matching means includes means for comparing said sequences of secondary features with a stereotypical pattern of said sequences of secondary features resulting from a human vowel stimulus waveform.

US Referenced Citations (4)

Number	Name	Date
2938079	Flanagan	May 1960
4932062	Hamilton	Jun 1990
5095904	Seligman et al.	Mar 1992
5155802	Mueller et al.	Oct 1992

Non-Patent Literature Citations (12)

Entry
Seneff, "A Joint Synchrony/Mean-Rate Model of Auditory Speech Processing", Academic Press Limited, 1988, pp. 101-111.
Ghitza, "Auditory Neural Feedback as a Basis for Speech Processing", IEEE ICASSP, 1988, pp. 91-94.
Kates, "A Time-Domain Digital Cochlear Model", IEEE Trans. on Signal Processing, vol. 39, No. 12, Dec. 1991, pp. 2573-2592.
Cheng et al., "Automatic and Reliable Estimation of Glottal Closure Instant and Period", IEEE Trans. on ASSP, vol. 37, No. 12, Dec. 1989, pp. 1805-1815.
Kishi, "The Properties and Configuration of the Short Time DFT Hilbest Transformers", IEEE 1989, pp. 1019-1022.
Pei et al., "Relationship Among Digital One/Half Band Filters, Low/High Order Differentiators and Discrete/Differentiating Hilbest Transformers", IEEE 1991, pp. 1657-1660.
Liu et al., "An Analog Integrated Speech Front-End Based on the Auditory Periphery", IEEE 1991, pp. II-861 to II-864.
Nguyen et al., "A Physiology Motivated Front-End for Speech Recognition", IEEE 1991, pp. II-503.fwdarw.II-508.
Kates, "An Adaptive Cochlear Model", IEEE 1991, pp. 3621-3624.
Wu et al., "Specialized Physiology-Based Channels for the Detection of Articulatory-Acoustic Events: A Preliminary Scheme & Its Performance", IEEE 1989, pp. 2013-2016.
Friedman, "Detection and Frequency Estimation of Narrow-Band Signals by Means of the Instantaneous-Frequency Distribution (IFD)", IEEE 1988, pp. 71-76.
Beet et al., "Improved Speech Recognition Using a Reduced Auditory Representation", BCC 1988, pp. 75-78.

Method and apparatus for speech feature recognition based on models of auditory signal processing

Information

Patent Number

Date Filed

Date Issued

Inventors

Original Assignees

Examiners

Agents

CPC

US Classifications

Field of Search

US

International Classifications

Abstract

Description

Claims

US Referenced Citations (4)

Non-Patent Literature Citations (12)