Patent Grant
7366659
Sometimes it is desirable to control the speed at which a sound recording is played, such as messages played back using an answering machine or service; messages received using a network device (e.g., Internet based audio streaming); in speech learning tools for the hard of hearing and hearing aids; and in tape recorders and the like.
Conventional methods for processing sound signals whose speed has been altered are based on either time-domain or frequency-domain techniques. In general, time-domain techniques are used to process sounds generated from conversations or speech while frequency-domain techniques are used to process sounds generated from music. Efforts to use time-domain techniques on music have resulted in less than satisfactory results because music is “polyphonic” and, therefore, cannot be modeled using a single pitch, which is the underlining basis for time-domain techniques. Likewise, efforts to use frequency-domain techniques to process speech have also been less than satisfactory because they add a reverberant quality, among other things, to speech-based signals.
Attempts have been made to minimize the side-effects of frequency-domain techniques but they have resulted in limited improvements in sound quality. See for example, J. Laroche, “Improved phase vocoder time-scale modification of audio,” IEEE Trans. on Speech and Audio Proc., Vol. 7, no. 3, pp. 323-332, May 1999.
Other advances, mainly in time-domain based, time-scaling techniques have used the fact that speech signals can be separated into various types of signal “portions” those being “non-stationary” (sounds such as ‘p’, ‘t’, and ‘k’) and “stationary” portions (vowels such as ‘a’,‘u’,‘e’ and sounds such as ‘s’, ‘sh’). Conventional time-domain systems process each of these portions in a different manner (e.g., no time-scaling for short non-stationary portions). See for example E. Moulines, J. Laroche, “Non-parametric techniques for pitch-scale and time-scale modification of Speech”, Speech Commun., vol 16, pp. 175-205, February 1995. However, similar alterations of the time-scaling process based on the stationary features of a sound signal have not yet found their way into frequency-domain systems. As in time domain systems, frequency-domain systems should process non-stationary signal portions in a different manner than stationary portions in order to achieve improvements in sound quality.
For example, time-domain systems process non-stationary portions in small increments (i.e., the entire portion is broken up into smaller amounts so it can be analyzed and processed) while stationary portions are processed using large increments. The phrase “frame-size” is used to describe the number of signal samples that are processed together at a given time.
Conventional frequency-domain techniques use a fixed frame-size and do not alter the frame-size based on signal characteristics. By failing to alter the frame size or to otherwise vary the type of time-scaling used to process non-stationary signal portions, sound quality is sacrificed.
Accordingly, it is desirable to provide methods and devices for selectively generating time-scaled sound signals in order to provide improvements in sound quality.
It is a further desire of the present invention to provide methods and devices for selectively generating sound signals which combine the advantages of both time and frequency-domain processed signals.
It is yet an additional desire of the present invention to provide methods and devices for removing unwanted reverberant sound qualities in frequency-domain processing.
Further desires of the present invention will be apparent from the drawings, detailed description of the invention and claims which follow.
In accordance with the present invention there are provided techniques for selectively generating speed adjusted, sound signals (i.e., time-scaled signals) using both time and frequency-domain processed, time-domain, time-scaled signals one of which comprises: a control unit adapted to generate first and second weights from an input sound signal (e.g., music or speech); a time-domain processor adapted to generate a time-domain processed, time-domain, time-scaled signal (“first signal”); a frequency-domain processor adapted to generate a frequency-domain processed, time-domain, time-scaled signal (“second signal”); and a mixer adapted to adjust the first signal using the first weight, adjust the second signal using the second weight, combine the so adjusted signals and for outputting a time-scaled, sound signal. In a further embodiment of the present invention, the control unit can be adapted to adjust the first and second weights based on a scaling factor. By so adapting the weights the correct contribution from each processed signal (i.e., correct balance between time-domain and frequency-domain processed signals) is used depending on the type of sound signal input.
In addition, the present invention provides for selectively applying time-scaling to only the stationary portions of an input sound signal and for making use of a frame-size which is adapted to the portion (i.e., stationary or non-stationary) of a signal being processed (referred to as an “adaptive frame-size”, for short) in order to further improve the sound quality of a speed-adjusted signal.
Referring to
The input sound signal is also input into the processors 2,3. The time-domain processor 3 is adapted to generate and output a time-domain processed, time-scaled signal (“first signal”) via pathway 300 to mixer 5. Frequency-domain processor 2 is adapted to: transform a time-domain signal into a frequency domain signal; process the signal; and then convert the signal back into a time-domain, time-scaled signal. Thereafter, processor 2 is adapted to output this frequency-domain processed, time-domain, time-scaled signal (“second signal”) via pathway 200 to the mixer 5. Upon receiving such signals from the processors 2,3 the mixer 5 is adapted to apply the first weight a to the first signal and the second weight b to the second signal in order to adjust such signals. Mixer 5 is further adapted to combine the so adjusted signals and then to generate and output a time-scaled, sound signal via pathway 500.
In this way, the present invention envisions combining both time-domain and frequency-domain processed signals in order to process both speech and music-based, input sound signals. By so doing, the limitations described previously above are minimized.
Operation of the control unit 4 and processors 2,3 will now be described in more detail. As shown, the control unit 4 comprises a sound discriminator 42, signal statistics unit 43 and weighting generator 41. Upon input of a sound signal via pathway 100 the discriminator 42 and signal statistics unit 43 are adapted to determine whether the input signal is a speech or music-based signal. Thereafter, the weighting generator 41 is adapted to generate weights a and b. As envisioned by the present invention, if the signal is a speech signal the value of the weight a will be larger than the value of the weight b. Conversely, if the input signal is a music signal the value of the weight b will be larger than the value of the weight a. In effect, the weights a and b determine which of the signals 200,300 will have a bigger influence on the ultimate output signal 500 heard by a user or listener. In this manner, the control unit 4 balances the use of a combination of the first signal 300 and second signal 200 depending on the type of sound signal input into device 1.
Continuing, suppose a user (i.e., listener) of device 1 wishes to vary the speed of the speech or music signal he or she is listening to. Enter the scaling factor. It is the scaling factor which acts to adjust the speed at which the signal is heard. As envisioned by the present invention, the control unit 4 is adapted to adjust the first and second weights a and b based on the scaling factor input via pathway 101.
Before continuing, it should be noted that the scaling factor input via pathway 101 may be manually input by a user or otherwise generated by a scaling factor generator (not shown).
According to one embodiment of the present invention, as the value of the scaling factor increases the control unit 4 is adapted to increase the second weight b and decrease the first weight a. Conversely, as the value of the scaling factor decreases the control unit 4 is further adapted to decrease the second weight b and increase the first weight a. This adjustment of weights a and b based on a scaling factor is done in order to select the proper “mixing” of signals 200,300 generated by processors 2,3. In other words, if the value of weight a is large then the ultimate signal 500 output by mixer 5 will be heavily influenced by the signal originating from time-domain processor 3; if the value associated with weight b is large then the output 500 generated by mixer 5 will be heavily influenced by the signal generated by frequency-domain processor 2. This mixing of both signal types allows techniques envisioned by the present invention to take advantage of the benefits offered by both as the scaling factor changes.
In a further example, suppose a user of device 1 wishes to slow down the speed of a sound signal. To do so, she would normally increase the scaling factor. According to the present invention, such an increase in the scaling factor affects the weights a and b. More particular, such an increase results in an increase in weight b and a decrease in weight a. This leads to an output sound signal 500 which is influenced more by a signal generated by the frequency-domain processor 2 than one generated by the time-domain processor 3.
In one simplified embodiment of the concepts just discussed, device 1 is adapted to adjust weights a and b only when an input sound signal transitions from a speech to a music signal or vice-versa. For example, if a speech signal is detected, a “full” weight is assigned to the first signal (e.g., a=1; b=0); while if music is detected, the full weight is assigned to the second signal (e.g., a=0, b=1). In these special cases, when one of the weights is equal to zero, no processing by the respective processor occurs (e.g., when a=0, b=1 no time-domain processing occurs, only frequency domain processing). This may occur when the input signal comprises substantially speech or music. In sum, the mixer 5 substantially acts as a switch either outputting the time-domain processed or the frequency-domain processed signal (i.e., first or second signal). It should be noted that although the discussion above and below focuses on speech and music-like sound signals, devices envisioned by the present invention will also process other sound signals as well. In such a case the input signal is classified as either a speech or music signal (i.e., if the signal is more speech-like, then it is classified as speech; otherwise, it is classified as a music signal).
The special case described above requires only a limited amount of synchronization (i.e., delay matching) between the time and frequency-domain processed signals, namely, at the transitions from speech to music and vice-versa. It should be understood, however, that in other embodiments of the present invention (i.e., where a and b are both non-zero) synchronization has to be performed almost constantly.
In addition to utilizing both time and frequency-domain processed signals, the present invention envisions further improvement of a time-scaled (i.e., speed adjusted) output sound signal by treating stationary and non-stationary signal portions differently and by using an adaptive frame-size.
In one embodiment of the present invention, processors 2,3 are adapted to detect whether an instantaneous input sound signal comprises a stationary or non-stationary signal. If a non-stationary signal is detected, then time-scaling sections 22,32 within processors 2,3 are adapted to selectively withhold time-scaling (i.e., these signal portions are not time-scaled). In other words, only stationary portions are selected to be time-scaled.
By selecting stationary signal portions for time-scaling and not non-stationary portions, the original characteristics of “impulsive” sounds and “onset” sounds (both of which are non-stationary) are maintained. This is important in order to generate time-scaled speech which sounds original in nature to a listener.
Though sections 22,32 do not apply time-scaling to non-stationary signal portions they are nonetheless adapted to process non-stationary signal portions using alternative processes such that the signals generated comprise characteristics which are substantially similar to an input sound signal.
As briefly mentioned above, devices envisioned by the present invention also make use of an adaptive frame size. In general, the frame-size determines how much of the input signal will be processed over a given period of time. The frame-size is typically set to a range of a few milliseconds to some tens of milliseconds. It is desirable to change the frame-size depending on the stationary nature of the signal.
Referring back to
At substantially the same time that the adaptive frame-size is being generated by section 21, the input signal is being processed by a frequency-domain, time-scaled section 22. This section 22 is adapted to generate the time-scaled second signal using techniques known in the art. In addition, however, according to the present invention, section 22 is influenced by a scaling factor input via pathway 101. The resulting signal is sent to a delay section 23 which is adapted to add a delay to the second signal and to process such a signal using the adaptive frame-size generated by section 21. It is this processed signal that becomes the second signal which is eventually adjusted by weight b.
As mentioned before, delays are necessary to synchronize the outputs of the time-domain and frequency-domain processors 2,3. Without synchronization, the two signals (time-domain and frequency domain processed signals) would not be aligned in time resulting in an output sound signal 500 which contains an echo. Both time-domain and frequency-domain processors may produce delays that vary over time. For time-domain processing, the delay may vary due to slight, short-term changes in the scaling factor. Although a user may set a target scaling factor, the actual scaling factor at a given moment in time may differ from such a target. To offset such an effect and still achieve a target scaling factor set by a user, sections 22,32 are adapted to time-scale stationary signal portions by an amount slightly greater than a user's target scaling factor. Besides slight short-term variations in the scaling factor, significant short-term variations may also occur during time-domain and frequency-domain processing. For example, sounds such as ‘t’,‘k’,‘p’ may not be scaled at all, while short-term stationary “phonemes”, such as ‘a’,‘e’, ‘s’ may be scaled more to achieve an average scaling factor that equals a target scaling factor.
On the other hand, for frequency-domain processing, the delay period is determined by the frame-size. A short frame-size introduces less delay than a large frame-size. If the outputs of the frequency-domain and time-domain processors 2,3 are mixed using weights a and b that are non-zero, these delays have to match (although a variation of a few milliseconds maybe tolerated, for example, when short-term stationary phonemes are being processed; but note that such variations introduce spectral changes and tend to degrade sound quality).
Referring again back to
In an alternative embodiment of the present invention, one of the delay units 23,33 is adapted to control the other via pathway 320 or the like to ensure the appropriate delays are utilized within each unit to prevent echoing and the like.
Time-scaled, speed-adjusted signals generated by using an adaptive frame size have lower amounts of reverberation as compared with signals generated using conventional techniques.
Features of the present invention have been illustrated by the examples discussed above. Modifications may be made to these examples without departing from the spirit and scope of the present invention, the scope of which is determined by the claims which follow:
| Number | Name | Date | Kind |
|---|---|---|---|
| 4246617 | Portnoff | Jan 1981 | A |
| 4864620 | Bialick | Sep 1989 | A |
| 5630013 | Suzuki et al. | May 1997 | A |
| 5699404 | Satyamurti et al. | Dec 1997 | A |
| 5828994 | Covell et al. | Oct 1998 | A |
| 5828995 | Satyamurti et al. | Oct 1998 | A |
| 6049766 | Laroche | Apr 2000 | A |
| 6519567 | Fujii | Feb 2003 | B1 |
| Number | Date | Country |
|---|---|---|
| WO 0013172 | Mar 2000 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20030229490 A1 | Dec 2003 | US |
