The present invention relates to a technique for analyzing human voice quality and, more specifically, to a vocal fry (hereinafter referred to as “VF”) detecting apparatus for detecting a segment of a specific voice quality referred to as vocal fry, in speech signals.
In human-machine communication scenario, it is necessary to automatically extract information other than text-based information (hereinafter referred to as “paralinguistic information”) in speech. Conventionally, prosodic features such as pitch, power and duration have been used as acoustic features for extracting paralinguistic information. Recent studies, however, have reported that voice quality information due to modality in the laryngeal voice source, for example, breathiness, creakiness and harshness also takes an important role in the perception of paralinguistic information.
VF, creak, creaky voice, glottal fry, pulse register and laryngealization are terminologies conventionally found in the literature for a voice quality characterized by a train of relatively discrete laryngeal (or glottal) excitations (or pulses of brief duration), with almost complete damping of the vocal tract between successive glottal pulses, usually accompanied by extremely low fundamental frequencies, and irregular durations of glottal cycles. The auditory perception of VF is of “rapid series of taps like a stick being run along a railing” or the “imitated sound of motor boat engine” or similar to “food cooking in a hot frying pan.”
VF carries important linguistic and paralinguistic information depending on the language. In German, VF often occurs near morpheme boundaries. In Japanese, besides the VF appearing in low tension voices, it also appears in expressive emphasizing utterances as a pressed voice. Such pressed voice carries paralinguistic information primarily associated with feelings or attitudes of surprise, admiration and suffering. VF utterance portions (hereinafter referred to as “VF segments”) in such pressed voices are often observed to have very low fundamental frequencies.
Further, VF segments have characteristic irregularities, possibly causing severe errors in pitch determination algorithms, which are important for prosodic information extraction. Thus, knowledge about the location of VF could be useful in extracting paralinguistic information as well as in improvement of pitch determination performance.
There are many studies reporting physiological, perceptual and acoustic properties of VF in several research areas. Many of them report qualitative or descriptive analyses of acoustic features that are related with different voice qualities. However, only a few evaluate their performance for automatic detection purposes.
Non-Patent Document 1: Ishi, C. T., “Analysis of Autocorrelation-based parameters for Creaky Voice Detection,” Proc. of The 2nd International Conference on Speech Prosody: 643-646, 2004.
The fundamental frequency ranges for VF are reported as being consistently lower than 100 Hz, with averages around 24 to 52 Hz. The glottal pulses in VF can be associated with two or even three pulses in a rapid succession followed by a period of significant vocal tract damping.
Many acoustic analyses of VF have been conducted in temporal, spectral and cepstral domains. Usual methods evaluate periodicity (or harmonicity) properties using a short-term analysis frame with fixed length.
A problem of using fixed length frame arises when VF segments have very low fundamental frequencies (that is, very large inter-pulse time intervals). In a standard (commonly used) analysis frame length around 25 to 32 milliseconds, it is often the case that only one glottal pulse lies within the analysis frame in VF segments, and sometimes, no glottal pulse lies within the frame. The presence of at least two glottal pulses within the analysis frame would be necessary for some harmonic structure in the spectrum to appear, or for autocorrelation peaks reflecting some short-term periodicity between glottal pulses to appear.
A simple approach to this problem could be taken by increasing the analysis frame length. In Non-Patent Document 1, autocorrelation-based periodicity analysis was conducted using an adaptively variable frame length. However, such solution solves only part of the problem, since more than two glottal pulses with different inter-pulse intervals may be present within a large analysis frame. This would disturb the harmonic structure in the spectrum, or reduce the magnitude of autocorrelation (or cepstral) peaks.
Therefore, an object of the present invention is to provide a VF detecting apparatus capable of highly accurate VF detection while avoiding the problems of disturbance of harmonic structure in the spectrum or reduced peaks of autocorrelation.
Another object of the present invention is to provide a VF detecting apparatus capable of highly accurate VF detection with a method in synchronization with glottal pulses, while avoiding the problems of disturbance of harmonic structure in the spectrum or reduced peaks of autocorrelation.
A further object of the present invention is to provide a VF detecting apparatus capable of highly accurate VF detection with a method in synchronization with glottal pulses, while avoiding the problems of disturbance of harmonic structure in the spectrum or reduced peaks of autocorrelation, by using an appropriate analysis frame.
According to a first aspect, the present invention provides a VF detecting apparatus for detecting a VF section in a speech signal, including: first framing means for framing the speech signal with a first frame having a first frame length and a first frame shift amount; power peak detecting means for detecting power peak in each of a series of first frames output from the first framing means; second framing means for framing the speech signal with a second frame having a second frame length longer than the first frame length and a second frame shift amount larger than the first frame shift amount; periodicity determining means for determining presence or absence of periodicity in each of a series of second frames output from the second framing means; power peak selecting means for selecting, from among the power peaks detected by the power peak detecting means, a power peak in the second frame determined by the periodicity determining means to have no periodicity; and means for searching, for each of the power peaks selected by the power peak selecting means, for a power peak having cross-correlation with another power peak in a prescribed section including the power peak, larger than a prescribed threshold, and detecting the prescribed section including the power peak in the speech signal as the VF section.
In the speech signal framed with the first frame, the power peak is detected. In the speech signal framed with the second frame signal, presence/absence of periodicity is determined. The first frame has shorter frame length and smaller amount of frame shift than the second frame. Therefore, in the speech signal framed with the first frame, even the waveform having low fundamental frequency can be detected with higher accuracy than in the speech signal framed with the second frame. On the other hand, the frame length of the second frame is longer than the first frame and, therefore, presence of periodicity therein can more accurately be determined. Of the detected power peaks, one existing at a portion of no periodicity is highly likely the VF pulse. Further, if such a VF pulse candidate has high correlation with another, neighboring pulse in a prescribed section, it is more likely that the candidate is a VF pulse. As the section including a power peak corresponding to the VF pulse as such is detected as the VF section, the VF section can be detected with high accuracy. As the first and second frames are used for processing, frames appropriate for signal processing can be utilized, allowing VF detection with high accuracy.
Preferably, the power peak detecting means includes: a power peak candidate detecting means for detecting, from a series of first frames, one having larger power than any other frames in a prescribed section including the frame and the difference is larger than a predetermined first threshold value, as the power peak candidate; and means for detecting, from the power peak candidates detected by the power peak candidate detecting means, one having larger power than each frame in a section wider than the prescribed section and the maximum value of difference is larger than a predetermined second threshold value, as the power peak.
More preferably, the section wider than the prescribed section refers to a section corresponding to 10 milliseconds of the speech signal.
More preferably, the periodicity determining means includes: means for calculating, in each of the series of second frames, in-frame periodicity measure of the maximum power peak in the frame, as a function of auto-correlation in a prescribed lag range in the frame, and for determining presence or absence of periodicity, depending on whether auto-correlation peak is larger than a prescribed threshold function or not.
The determining means may calculate the measure for periodicity by multiplying an autocorrelation value related to the maximum power peak by a function as a monotonically decreasing function of a lag from the maximum power peak in the frame of interest.
Preferably, the prescribed threshold function is obtained by multiplying a predetermined constant larger than 0 and smaller than 1 by the monotonously decreasing function.
More preferably, the periodicity determining means further includes periodicity correcting means for correcting a value of periodicity measure of the second frames at portions other than portions where frames having periodicity measures larger than a predetermined constant continue by a prescribed number, among the second frames determined to have periodicity by the determining means, to a value that is to be determined to have no periodicity.
Further preferably, the apparatus further includes filtering means for filtering out frequency components outside a prescribed frequency band of the speech signal, before applying the speech signal to the first and second framing means.
According to a second aspect, the present invention provides a storage medium storing a computer program that causes, when executed by a computer, the computer to operate as any of the VF detecting apparatuses described above.
100 an automatic communication system
102, 174 a speech signal
120 a speech recognition apparatus
122 a VF detecting apparatus
124 a response forming apparatus
126 a knowledge base
128 a speech synthesizing apparatus
132 VF section information
162 a very-short-term peak detection processing unit
164 a short-term periodicity detecting unit
166 a periodicity checking unit
168 a similarity checking unit
170 peak position information
172 short-term periodicity information
176 VF candidate information
190, 250 a framing unit
192 a very-short-term power calculating unit
196 a peak comparing unit
254 an IFP calculating unit
258 a periodicity determining unit
260 a continuity checking unit
310 an IPS calculating unit
312 an IPS comparing unit
314 a threshold value storing unit
316 a VF segment determining unit
To solve the frame length problem, the inventors of the present invention decided to realize a glottal pulse-synchronized processing, when no periodicity can be found within the fixed length analysis frame. For this purpose, in the present embodiment, candidates for glottal pulses are detected based on the damping and low fundamental frequency properties of VF. This is based on the phenomenon that damping in large inter-pulse intervals is characterized by an up and down movement in the amplitude envelope, or in a local power contour, of the speech signal.
Another problem regarding automatic VF detection is that most acoustic analyses evaluate temporal or spectral features of pre-segmented voiced parts of the speech signal. In a real problem of automatic VF detection from the whole speech utterance including consonants and non-speech segments, many insertion errors might occur since such segments also usually have a periodic characteristics. Thus, the problem is how to discriminate between the aperiodicity caused by VF and reverberations caused by consonants and background non-speech signals.
In order to solve this problem, the present invention introduces evaluation of similarity measure between successive (or close) glottal pulses. The measure is based on an assumption that the vocal tract configuration does not change much between generations of two glottal pulses and thus, the vocal tract responses are expected to be similar.
Automatic communication system 100 further includes: a response forming apparatus 124 receiving the speech recognition result 130 from speech recognition apparatus 120 and VF section information 132 from VF detecting apparatus 122, integrating paralinguistic information processing using VF section information 132 with the speech recognition result 130 to understand speaker intentions, and outputting text information and voice quality information to provide appropriate response; a knowledge base 126 referred to by response forming apparatus 124 when forming the response, storing knowledge enabling formation of appropriate response for the combination of text information and paralinguistic information of the speech; and a speech synthesizing apparatus 128 synthesizing speech from the text information of the response output from response forming apparatus 124 with voice quality instructed by response forming apparatus 124 and outputting as a speech signal 104. The speech signal 104 is converted to an analog signal by a circuit, not shown, amplified and supplied to a speaker.
VF detecting apparatus 122 further includes: a very-short-term peak detection processing unit 162 detecting a local power peak in the output of band-pass filter 160 as a VF pulse candidate using a frame having the frame length of 5 milliseconds and frame interval of 2.5 milliseconds (in the present specification, referred to as a “very-short-term frame”) and outputting peak position information 170; and a short-term periodicity detecting unit 164 detecting a portion not having short-term periodicity indicating possible presence of VF in the output of band-pass filter 160 discriminating from other portions, using a commonly used frame having the frame length of 25 to 32 milliseconds and frame length of 10 or 5 milliseconds (in the present specification, referred to as a “short-term frame”), and outputting short-term periodicity information 172.
VF detecting apparatus 122 further includes: a periodicity checking unit 166 for receiving peak position information 170 from very-short-term peak detection processing unit 162 and short-term periodicity information 172 from short-term periodicity detecting unit 164, respectively, for selecting, as a VF frame candidate, frames including respective peaks at portions where no short-term periodicity exists from among peaks indicated by peak position information 170, and for outputting as VF candidate information 176; and a similarity checking unit 168 for identifying only the VF candidate having a similar pulse within prescribed preceding and succeeding ranges as the VF, for using VF candidate information 176 output from periodicity checking unit 166 and speech signal 174 having frequency components of 100 to 1500 Hz output from band-pass filter 160, and for outputting a VF section information 132 indicating the section where VF exists.
Referring to
It is not necessarily clear from these figures what threshold value (power threshold value) is to be set for discriminating between VF and NF. The threshold value is selected based on a result of experiment as will be described later and, by way of example, the value of 7 dB is used as the threshold value.
Short-term periodicity detecting unit 164 shown in
Referring to
The IFP value of autocorrelation analysis by IFP calculating unit 254 is defined as the correlation value of the maximum peak, normalized by “frame length/(frame length−lag).” This normalization is for compensating the property of autocorrelation function as monotonous decreasing function that autocorrelation decreases as the lag increases.
Only autocorrelation peaks whose lags are smaller than 15 milliseconds (corresponding to fundamental frequency larger than about 66.7 Hz) are considered for periodicity analysis in IFP calculating unit 254. This means that at least two glottal cycles are present in the analysis frame.
Periodicity determining unit 258 performs the following process on the autocorrelation peaks corresponding to fundamental frequencies larger than 200 Hz. Specifically, the periodicity of all sub-harmonics above 66.7 Hz is checked. This process prevents misdetection of periodicity due to strong harmonics around the first formant, rather than a periodicity due to repetition of glottal cycles.
Referring to
In contrast, referring to
Periodicity checking unit 166 shown in
The IPS value calculated by IPS calculating unit 310 is calculated as cross-correlation function between the waveform around the power peak as the object of processing and the waveforms around the previous power peaks, as already described. The frame length for cross-correlation calculation is limited to 15 milliseconds, in order to avoid the interference of irregularly spaced glottal pulses in the similarity calculation.
Cross-correlation is estimated in a range of 5 milliseconds around the power peak position, and the maximum value is taken as the IPS value. High IPS values indicate high probability of the detected power peaks representing VF pulses. For calculation of the IPS value, the search range of power peaks is limited to 100 milliseconds before the object power peak, and cross-correlation with the power peak is calculated. The value of 100 milliseconds corresponds to the maximum time interval allowed between two glottal excitation pulses. The maximum value of excitation pulse corresponds to an extremely low fundamental frequency of 10 Hz.
Referring to
Automatic communication system 100 having the above-described configuration, particularly the VF detecting apparatus 122 operates as follows. Referring to
Response forming apparatus 124 accesses knowledge base 126 using the plurality of candidates included in speech recognition result 130 applied from speech recognition apparatus 120 and VF section information applied from VF detecting apparatus 122, and thereby forms a response that would be most relevant from the combination of the candidates of speech recognition result and the VF segment. The response consists of response text information and information designating voice quality of the response speech, and it is applied to speech synthesizing apparatus 128. Speech synthesizing apparatus 128 synthesizes speech signal 104 for reproducing the designated text information with the designated voice quality, and applies the signal to the speaker.
In the following, the operation of VF detecting apparatus 122 will be described. Referring to
Very-short-term peak detection processing unit 162 detects a power peak in a very-short-term frame through the following process, and applies as peak position information to periodicity checking unit 166. Specifically, referring to
Very-short-term power calculating unit 192 calculates the very-short-term power for each frame, and applies the result to memory 194 for storage. Memory 194 stores values of the very-short-term powers for a prescribed number of latest frames.
Peak comparing unit 196 compares each frame with a preceding frame and a succeeding frame. If the power differences of the frames are larger than the power threshold value PwTH, the frame is regarded as a power peak candidate, and peak comparing unit 196 outputs peak position information 170 indicating the frame position, to periodicity checking unit 166.
Short-term periodicity detecting unit 164 shown in
IFP calculating unit 254 calculates the IFP value for each frame stored in memory 252, and applies the value to periodicity determining unit 258. Periodicity determining unit 258 corrects the IFP value of each frame applied from IFP calculating unit 254 by comparison with the threshold function. Specifically, if any sub-harmonic IFP value of each frame is smaller than the threshold value, periodicity determining unit 258 sets the IFP value of the frame to null. Periodicity determining unit 258 applies the IFP values of respective frames to continuity checking unit 260.
Regarding the IFP values of respective frames applied from periodicity determining unit 258, continuity checking unit 260 corrects, unless at least three continuous frames have non-null IFP values, the IFP values of the frames to null. The IFP value of each frame after the continuity check by continuity checking unit 260 is applied as short-term periodicity information 172 to periodicity checking unit 166 shown in
Periodicity checking unit 166 takes only the portion corresponding to frames having null IFP values as the VF segment candidate, based on the short-term periodicity information 172 applied from short-term periodicity detecting unit 164, from peak position information 170 applied from very-short-term peak detection processing unit 162, and applies the same as VF candidate information 176 to similarity checking unit 168.
Referring to
Automatic detection of VF by VF detecting apparatus 122 in accordance with the above-described embodiment was evaluated through comparison between the duration (VFdur) of the automatically detected VF segment with a period manually determined to be VF and labeled as such (VFdur_human). In the following, the ratio between VFdur and VFdur_human will be referred to as VF ratio. The segment labeled as VF is considered as correctly detected only if VF ratio is ⅔ or higher. By counting the number of segments not labeled VF but determined by automatic detection to be VF (VFdur_ins), insertion error was checked. The detection result and insertion error result are grouped into two, that is, “detection” and “detection?,” depending on detection performance or severity of the insertion error. The group “detection?” includes segments detected as “VF” with the VF ratios between ⅓ to ⅔, and insertions whose “VFdur_ins” values are shorter than 30 milliseconds.
Several combinations of parameter values involved in the embodiment above were tested, in order to reduce insertion errors without degrading detection performance. First, power peak thresholds were reset by adjusting the IPS value to 0.0 and IFP value to 1.0. This corresponds to using only power information.
Next, power threshold value was fixed at 7 dB and IPS threshold value was set to 0.0.
Finally, several IPS threshold values were tested by setting power threshold value to 7 dB and IFP threshold value to 0.6, respectively. Referring to
Regarding the group “R” (segments of which VF features were not perceived by humans), most of the samples were not detected as VF in automatic detection. In “VF?” group, however, part of samples was detected as “VF.” The results indicate that the VF automatic detecting apparatus in accordance with the present embodiment attained results fairly consistent with the results of human perception.
A global detection rate is calculated as the summation of VFdur divided by the summation of VFdur_human. A global insertion error is calculated as the summation of VFdur_ins divided by the summation of VFdur_human. For the parameter combination of “power=7 db, IFP=0.6 and IPS=0.6,” the global detection rate of 73.3% and global insertion error rate of 3.9% are obtained. The detection rate of 73.3% can still be improved by post-processing the detection results. By way of example, by merging close VF segments or by other methods, the detection rate may be improved. For applications allowing slightly higher insertion error rate without causing any problem, the detection rate may be improved by further adjusting the parameters.
As described above, according to the present embodiment, vocal fry can automatically be detected by using a combination of IFP and IPS parameters.
VF detecting apparatus 122 and automatic communication system 100 in accordance with the present embodiment may be implemented by computer hardware, a program executed by the computer hardware and data stored in the computer hardware.
Referring to
Referring to
Though not shown, computer 340 may further include a network adaptor board providing connection to local area network (LAN).
The computer program causing computer system 330 to operate as the automatic communication system 100 and VF detecting apparatus 122 in accordance with the present embodiment may be stored on a DVD disk 362 or semiconductor memory 364 loaded to DVD drive 350 or semiconductor memory drive 352, and further transferred to hard disk 354 Alternatively, the program may be transmitted to computer 340 through a network, not shown, and stored in hard disk 354. The program is loaded to RAM 360 when executed. The program may be directly loaded to RAM 350 from DVD disk 362, semiconductor memory 364 or through the network.
The program includes a plurality of instructions causing computer 340 to operate as automatic communication system 100 and VF detecting apparatus 122 in accordance with the present embodiment. Some of the basic functions to execute the processes in accordance with these instructions are provided by the operating system (OS) operating on computer 340, a third party program or various tool kit modules installed in computer 340. Therefore, the program may not necessarily include all the functions to realize the operation of automatic communication system 100 and VF detecting apparatus 122 in accordance with the present embodiment. The program may include only the instructions to execute the operation of automatic communication system 100 and VF detecting apparatus 122 described above, by calling appropriate functions or “tools” in a controlled manner to attain desired results. The operation of computer system 330 is well known and, therefore, detailed description will not be given here.
Power threshold storage unit 198 shown in
The embodiments as have been described here are mere examples and should not be interpreted as restrictive. The scope of the present. invention is determined by each of the claims with appropriate consideration of the written description of the embodiments and embraces modifications within the meaning of, and equivalent to, the languages in the claims.
The present invention is applicable to a system for detecting VF segments from a speech signal and obtaining paralinguistic information from the speech signal based on the detected VF segments, as well as to a man-machine interface enabling appropriate response based on the paralinguistic information.
Number | Date | Country | Kind |
---|---|---|---|
2005-250454 | Aug 2005 | JP | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/JP2005/023365 | 12/20/2005 | WO | 00 | 2/13/2008 |