Patent Application
20050267739
The present invention relates generally to systems and methods for quality improvement in an electrically reproduced speech signal. More particularly, the present invention relates to systems and methods for enhanced artificial bandwidth expansion for signal quality improvement.
Speech signals are usually transmitted on a conventional telephone bandwidth in telecommunication systems, such as a GSM (Global System for Mobile Communications) network. The traditional bandwidth for speech signals in such systems is less than 4 kHz (0.3-3.4 kHz) although speech contains frequency components up to 10 kHz. This limited bandwidth can result in poor performance in both quality and intelligibility of the speech signals. In other words, the limited bandwidth can greatly degrade the naturalness of the transmitted voice signal.
Humans perceive better quality and intelligibility if the frequency band of a speech signal is wideband, i.e. up to 8 kHz. Thus, in order to improve the naturalness of a transmitted speech signal, one approach is to use wideband speech coders such as a wideband adaptive multi-rate (WB-AMR) coder.
Existing methods for improving the quality of narrowband speech by artificial bandwidth expansion can be divided into two groups. In the first group a codebook can be used to generate missing frequency components of the upper band of speech (e.g. between 4.0 kHz and 8 kHz). A codebook can comprise frequency vectors of different spectral characteristics, all of which cover the same upper band. The frequency range can be expanded by, on a frame-by-frame basis, selecting the optimal vector and adding into it the received spectral components of the lower band (e.g. 0 kHz to 4 kHz). In a second group, the original narrowband speech can be up-sampled in order to create aliased frequency components and the levels of these new frequencies can be adjusted to create the high frequency components of a speech signal.
However, these existing artificial bandwidth expansion methods for improving a narrowband speech signal can suffer from problems and inefficiencies. For example, some of these methods are based on classifying the incoming speech frames by their phonetic content. For example, phonemes like /s/, /k/, and /a/ are classified in different classes. Based on the classification, an upper band envelope can be formed. The envelope can be used to shape the upper band spectrum that was originally obtained from the lower band spectrum by aliasing. However, upper bands generated using this approach are not always very natural. For example, because transitions between different phones in speech can be very smooth, artificial decision boundaries in the classification scheme can create unnecessary discontinuities to the expansion process. Furthermore, misclassification can cause noticeable artifacts. In addition, bandwidth expansion methods that use Linear Prediction (LP) analysis to estimate the behavior of the spectral envelope to attenuate the aliased frequency components can suffer from insufficient attenuation of the aliased frequency components, which in turn, deteriorates the speech quality.
Artificial bandwidth expansion methods based on codebooks require storage of the frequency vectors in order to expand the bandwidth of the received speech sound. Storage of codebooks increases the amount of memory resources needed in order to perform the bandwidth expansion. In addition, because mobile phones are required to be capable of operating in a variety of environments, such as different noise conditions or to transfer speech signals of various languages, it is difficult to configure a codebook that is capable of producing quality bandwidth expansion for the many different environments.
As such, there is a need for an improved system, method, device, and computer code product for artificially expanding the bandwidth of a narrowband speech signal to improve the quality and naturalness of the speech signal.
One embodiment of the invention relates to a method for artificially expanding a narrowband signal by expanding the narrowband signal to produce an unshaped wideband signal, forming a magnitude shaping function using a neural network, and amplifying/attenuating the unshaped wideband signal using the magnitude shaping function to form an artificially expanded wideband signal. The unshaped wideband signal can be expanded in any number of ways include by aliasing the narrowband signal.
Another embodiment of the invention relates to a device for artificially expanding a narrowband signal. One embodiment of the device can include a lowband to highband transfer filter configured for expanding the narrowband signal into an unshaped wideband signal, a neural network configured for forming a magnitude shaping function, and a magnitude shaping module for amplifying/attenuating the unshaped wideband signal according to the magnitude shaping function to form an artificially expanded wideband signal. Embodiments of the device can also include a feature evaluation module configured for evaluating, selecting, and passing features of the narrowband signal on to the neural network, so that the neural network can form the magnitude shaping function based on the features passed by the feature evaluation module. One or more genomes can be used set weights in the neural network. The genomes can be produced by an evolution module based on a simulation environment configured to simulate an environment in which the device is used. The lowband to highband transfer filter can be configured to form the unshaped wideband signal any number of ways including alias the narrowband signal.
Still another embodiment of the invention includes a mobile communication device having a receiver, a lowband to highband transfer filter, a neural network, and a magnitude shaping module. The receiver can be capable of receiving a narrowband speech signal. The lowband to highband transfer filter can be capable of expanding the narrowband signal into an unshaped wideband signal. The neural network can be capable of forming a magnitude shaping function based on features of the narrowband speech signal. The magnitude shaping module can be capable of amplifying/attenuating,the unshaped wideband signal according to the magnitude shaping function to form an artificially expanded wideband speech signal.
A further embodiment of the invention can include a transcoder device configured for operating in a communication network. The transcoder can include a receiver capable of receiving a narrowband speech signal, a lowband to highband transfer filter capable of expanding the narrowband signal into an unshaped wideband signal, a neural network capable of forming a magnitude shaping function based on features of the narrowband speech signal, a magnitude shaping module for amplifying/attenuating the unshaped wideband signal according to the magnitude shaping function to form an artificially expanded wideband speech signal, and a transmitter capable of transmitting the artificially expanded wideband speech signal.
Another embodiment of the invention can comprise a system for artificially expanding the bandwidth of a narrowband speech signal. The system can include an evolution subsystem capable of producing one or more genomes based on a simulation environment configured to simulate an environment in which a communication device is used and an online processing subsystem capable of artificially expanding the bandwidth of a narrowband speech signal. The online processing subsystem may include a lowband to highband transfer filter capable of expanding the narrowband speech signal into an unshaped wideband signal, a neural network capable of forming a magnitude shaping function based on features of the narrowband speech signal, and a magnitude shaping module for amplifying/attenuating the unshaped wideband signal according to the magnitude shaping function to form an artificially expanded wideband speech signal. The genomes produced by the evolution subsystem may be used to set weights in the neural network.
Other embodiments of the invention can include computer code products for artificially expanding a narrowband speech signal. One embodiment of a computer code product according to the present invention can include computer code configured to expand the narrowband speech signal to produce an unshaped wideband signal, form a magnitude shaping function using a neural network, and amplify/attenuate the unshaped wideband signal using the magnitude shaping function to form an artificially expanded wideband signal.
Still further embodiments of the invention can include neuroevolution training systems for creating genomes for use by an online processing system capable of expanding narrowband speech signals into an artificially expanded wideband speech signals. One embodiment of a neuroevolution training system according to the present invention can include a learning sample management module configured to manage speech samples that can be used to train the system, a fitness evaluation module configured to evaluate the quality of the artificially expanded wideband speech signals, and an evolution module configured to perform an artificial evolution by mutating and recombining the genomes based on the evaluation of the fitness evaluation modules. The fitness evaluation module may be configured to compare the artificially expanded wideband speech signal to a corresponding speech sample in the learning sample management module to determine if the artificially expanded wideband speech signal is similar to the original wideband sample of speech. The fitness evaluation module may also be configured to produce an objective fitness value of the artificially expanded wideband speech signal. The evolution module may be configured to use the object fitness value to create a fitness ranking for the genomes. The evolution module can be configured to select genomes for reproduction based fitness rankings for the genomes. The evolution modules may also act as a process controller for directing operation of the learning sample management module and the fitness evaluation module.
Other principle features and advantages of the invention will become apparent to those skilled in the art upon review of the following drawings, the detailed description, and the appended claims.
a is graphical representation of one embodiment of an original narrowband signal in accordance with the present invention.
b, c, and d are graphical representations of various embodiments of unshaped wideband signals generated from the narrowband signal of
a is a graphical representation of one embodiment of an unshaped wideband signal in accordance with the present invention.
b is graphical representation of one embodiment of magnitude shaping curve for the unshaped wideband signal of
c is a graphical representation of one embodiment of an expanded wideband signal shaped by the magnitude shaping curve of
Embodiments of the current invention relate to improving quality (naturalness, richness, etc.) of an electrically reproduced speech signal by artificially expanding the bandwidth of the sound. For example, the quality of narrowband speech transmitted in a telecommunications network can be improved by inserting into it new frequency components that may not have been transmitted. In one embodiment, the naturalness of telephone speech received by a mobile terminal or network can be improved by artificially doubling the bandwidth of the sound. Hence, it is possible to convert narrowband speech to a wideband form without explicitly using wideband speech coding methods. One particular situation in which embodiments of the invention can be particularly useful is in communication systems which handle both narrowband and wideband encoded transmitted speech. In this situation, the difference in quality between the signals is decreased by embodiments of the invention by artificially converting narrowband signals into wideband signals.
One embodiment of the invention uses control points generated by a neural network, fuzzy logic controller, or other device or method from features of the original narrowband signal to shape the upper band spectral envelope of an unshaped wideband signal transformed from the original narrowband signal. The neural network can be trained with variable data to evolve networks capable of performing well in different environments (e.g. different noise types, noise levels, languages, speech codecs, etc.). A neuroevolution method (the process of evolving neural network controllers for different control tasks) based on genetic algorithms can be used to evolve the artificial neural network. An upper frequency band can be generated as a mirror image by aliasing the narrowband spectral information. The neural network can be configured for analyzing the narrowband speech frames and producing control parameters for spline curves that can be used to amplify/attenuate the spectral components at the upper frequency band. The evolved networks can be recurrent, meaning that they can internally collect and use “historical information” about the process and are thus not limited to narrowband information from the current processed frame.
One embodiment of a system according to the present invention can include two modes: a learning mode and a processing mode. The learning mode can be configured to evolve new networks capable of performing artificial bandwidth expansion in various environments. The processing mode can be configured to use one of the evolved networks to expand the bandwidth of a received narrowband signal. In order to increase efficiency, the system can be configured so that the learning mode is executed off line to produce a good neural network that can be used in the processing mode online. In this manner, the processing mode remains computationally effective by moving the relatively computationally expensive learning mode off line.
In the learning mode, a population of neural networks can be evolved using neuroevolution methods. The population can be tested against training samples and the best performing networks can be recombined and mutated to produce a population of next generation networks. The learning mode can be terminated after a certain number of generations have passed or some specific criterion is met (e.g. the best network produces results that are within a certain range of the original wideband frames). This data oriented approach can make it easier to adapt a network for different operating environments. The system can be trained for the specific operating conditions it will be operating in, thus enabling it to perform better than a generic algorithm.
Referring to
The evolution subsystem 12 can be configured to generate a random population of genomes 16 and, using the online processing subsystem 14, expand a predefined set learning samples with each of them. The evolution subsystem 12 can calculate a fitness value for each of the individual genomes 16 by evaluating an objective function, which can, for example, measure the quality of the expansion result using some metric appropriate for the problem. The evolution subsystem 12 can then evolve the population of genomes 16, recombining and mutating the individuals genomes 16. This evaluation-evolution cycle can be continued until a specified end condition is reached.
After an acceptable solution has been found, its genome 16 can be merged with the online processing subsystem 14 to form the final expansion algorithm. Should the processing environment change, a new genome 16 can be evolved later using a similar evolution process in a properly altered simulation environment 18, and installed into the online processing subsystem 14 replacing the old genome 16. This leads to considerable adaptability.
In a sample embodiment of the invention, the LSMM 20 can be configured for handling the preprocessing of samples in a training simulation. For example, the LSMM 20 can simulate the processes a telephone speech signal goes through when it is transmitted from the speaker to the receiver. The LSMM 20 can be responsible for providing a narrowband signal to the online processing subsystem 14 during the training simulation and providing the corresponding wideband reference signal to the FiEM 22.
During preprocessing, the samples can be transformed from wideband signals to narrowband signals. Preferably, the system should avoid introducing processing delay or the delay should be countered during the teaching process since signals should be as synchronized as possible in order to maximize effectiveness of the fitness function. Various processing paths can be included in the system.
In order to accommodate an online processing subsystem 14 that uses frame based processing, the narrowband signal can be split into frames of speech, for example 10 ms frames. Some overlap between frames can be used to reduce the effect of FFT windowing and to enable linear averaging between the frames to avoid sudden jumps at frame edges. As such, actual processed frames can be slightly longer than a typical frame length (such as 12.25 ms as opposed to 10 ms).
As described above, the EM 24 can be configured for evolving new genomes that can act as parameters for the online processing subsystem 14. The EM 24 can also act as a process controller for the learning process, directing other modules of the evolution subsystem 12. Multiple possible embodiments of evolution methods can be used to implement various embodiments of the invention.
The EM 24 can be responsible for generating the initial population. In one embodiment, a completely random set of genes can be generated, utilizing a random number generator. Alternatively, a method for initializing the weights of a neural network can be used.
The EM 24 can be optimized to select only certain learning samples in order to decrease the computational load. For example, the population may include multiple samples from the same speaker (person). If so, the EM 24 can be configured to select only one sample per speaker. In addition, the EM 24 can be configured to draw only a prescribed number of random samples for each generation. It may be advantageous to not use all training samples for a predetermined number of initial generations (e.g. the first 150 generations) so that initial evolution can be done quickly.
The FiEM 22 can be used to evaluate how well a given sample was expanded by comparing the expanded signal received from the online processing subsystem 14 to a wideband signal from the LSMM 20. The comparison metric can measure the difference between the two signals as psychoacoustically accurately as possible. A fitness assignment can be used and a ranking based on fitness assignment can be produced for this purpose. By using a ranking system, the actual fitness values are not critical and thus the actual values produced by the metric can be highly nonlinear as long as it introduces an approximate subjective speech quality based on ordering. Alternatively, the FiEM 22 could simply measure whether the expansion improves the speech quality of the narrowband signal and select the algorithm parameters that most improve speech quality. Because spectral envelope of the expanded speech signal can be the principal key for a high subjective quality speech signal produced by a bandwidth expansion system, in one embodiment the spectral metric can be used as the quality measure for the expanded signal.
One embodiment of the invention can use a frame based processing scheme to compute the spectral quality metric between two signals. Frame-based processing is particularly useful in Fast Fourier Transform (FFT) based spectral estimation. One problem with FFT is that it assumes that signals under investigation are Wide-Sense Stationary (WSS). While speech is not a WSS process (since articulators are moving and shaping the vocal tract) selecting a time frame in which the vocal tract movement is negligible (e.g. on the order of 15-20 ms for most vowels) enables use of FFT.
Preferably, the time duration of the analysis frame can be long enough to span at least one pitch period, but also short enough that articulators do not move considerably during the frame. If the frame length is too short, the results can fluctuate very rapidly depending on the exact positioning of the frame. If the frame length is too long, the speech process during may not be stationary enough and the results of the spectral estimation may not be accurate. When selecting the frame length, may be useful to avoid the length used by the expansion process of the online processing subsystem 14, because using the same length might let some frame synchronization error get through as the frame pre-FFT windowing reduces the importance of the samples on the frame edges. Allowing a temporal overlap in frames reduces this risk, but as speech is not stationary and, especially during glides, the exact frame time interval can affect the FFT analysis results considerably, using a different framing scheme for the fitness evaluation may improve the system robustness. The frames may be Hamming windowed prior to using FFT in the spectral quality metric to improve the spectrum estimate.
Many different spectral distance metrics may be used as a quality measure according to embodiments of the present invention. For example the following spectral distortion metric may be used to measure spectral distortion between two envelope shapes in the extension band:
and Ak(ω) is the original and Âk(ω) the predicted envelope of the k'th (temporally aligned) frame of wideband speech and ωs is the wideband sampling frequency. The compensating gain factor GC can have the effect of removing the mean difference between the two envelopes, in which case DHC measures only the spectral distortion between the envelope shapes.
In another embodiment, log spectral distortion (LSD) can be used as a distance metric. For example, for an artificial bandwidth expansion, the following may be used:
where Amb(ejω) and σrel denote the modeled frequency spectrum and relative gain of the missing frequency band of the original wideband signal and Âmb(ejω) and {circumflex over (σ)}rel denote the corresponding parameters of the artificially expanded band.
The LSD can also be expressed in the cepstral domain. For a sequence of speech frames, the root mean square average of the LSD may be given by:
where E{ } can denote the expectation operation and cepstral coefficients c0, c1, . . . can be calculated from the AR coefficients and the relative gain. Other spectral distance metrics, such as log spectral mean-square-error (MSE) or cepstral MSE, can also be used as well without departing from the spirit and scope of the invention.
Bandpass filtering in the narrowband signal in a system that expands telephone band speech can present problems. For example, when the base expansion band is generated directly from the narrowband signal the expansion band may have some gaps. Preferably, these gaps should not be included into the spectral distance calculation, as their inclusion can cause a genome to benefit from considerable amplification at the gaps which could add noise to the system. As such, preferably, MSE distance is measured only from the spectral bands which are not generated from the attenuated bands of the narrowband signal. A Lowband to Highband Transform Filter (LHTF) transforms the narrowband input frame into a wideband frame. The LHTF determines which bands are not generated from the attenuated band of the narrowband signal. Similarly, cepstral calculations can filter out the gap bands by using a raised cosine bandpass filter (as shown in
The evolution process can sometimes, especially in the beginning of an evolution, generate genomes that cause extreme attenuations on some spectral bands. These attenuations may, in some cases, be so extreme that due to the limited accuracy of a digital computer, zeros occur in the magnitude spectrum of the frame. In one embodiment, log10(1+x) instead of logo (x) is used as the logarithm function in order to make the distance metric more robust against the pathological signal frames which cause zeros to appear in the magnitude spectrum. This also de-emphasized errors that can occur near the spectral zeros (as shown in
where SMSEf is the frame log spectral mean square error, V1 is the beginning of the valid expansion band, V2 is the end of the valid expansion band, X(ejω) is the spectrum of the reference signal frame and {circumflex over (X)}(ejω) is the spectrum of the expanded signal; and CMSEf is the frame cepstral mean square error, Nc is the number of cepstral coefficients to include in the evaluation, Hcf is the spectral domain raised cosine bandpass filter and X(ejω) and {circumflex over (X)}(ejω) are as before.
The frame objective values can be combined to produce a signal objective value for each genome. In one embodiment, this can be done in two operations. First, quality values for frames of each training signal can be combined to produce an objective value for the signal and then the signal objective values can be combined by simply calculating an average. Ideally, the training data contains an equal number of frames for each phoneme, however, it is also desirable to have the data contain frames in a natural order so that the system can learn to exploit information in the frame order, if recurrent feedback is used.
Preferably, the objective value combiner emphasizes large errors. In one embodiment, this can be done by applying an extra cost factor for errors that are in some sense much larger than the average error level. By doing so, is may be possible to reduce the amount of artifacts in the produced speech.
In another embodiment, the frames are combined to produce a sample objective value by averaging them together. The final genome objective value can be characterized as follows:
where S is the set of all samples, F(s) is the set of frames in the sample s, MSEf(f) is either the SMSE or the CMSE of frame f in sample s, |F(s)| is the number of frames in the sample s and |S| is the number of samples in the learning sample set.
The online processing subsystem 14 is responsible for the actual bandwidth expansion. In one embodiment, the online processing subsystem 14 can be integrated to the target system. As such, preferably, the online processing subsystem 14 is computationally efficient, straightforward to implement, and as robust as possible. Also, preferably, the online processing subsystem 14 is easy to customize and change, and has the ability to adapt to different operating environments by simple retraining.
In one embodiment, as shown in
In one embodiment, the samples to be expanded can be processed frame by frame. The actual framing may be done in other parts of the system. For example, during evolution the ESS LSMM may do the actual framing or in a live system, the framing may be done by the surrounding telephone system. The online processing subsystem 14 operation can be customized to be used with various different framing methods.
The FeEM is responsible for evaluating the frame features passed to the NNM. In effect, the features are the sensors of the expansion process. The expansion quality can be directly related to the quality of features selected by the FeEM. If the features do not contain information that is important for deciding how a frame should be expanded, a low quality expansion can result. If irrelevant information is sent to the NNM, the NNM can be overloaded making it difficult for the NNM to make good decisions. As such, preferably, the selected features should be accurate, and give all relevant information, while discarding the information irrelevant for the expansion process.
Preferably, the FeEM should select as few features as possible to minimize the dimensionality of the solution space the neuroevolution process must search. Each additional search space dimension can slow the learning process and can also add to the risk of not reaching a solution at all. In addition, the number of training samples needed to prevent overtraining grows with the input space dimension. For fully connected neural networks, the number of free parameters can grow quite aggressively when input are added. The FeEM should be configured to weigh the tradeoff between providing the NNM will all the possible features that could be useful for the process so that the NNM has all the information it needs and minimizing the number of features provided to keep the size of the search space feasible.
Various methods can be used in connection with embodiments of the invention for solving the information tradeoff. In one embodiment, the FeEM can FFT transform the signal frames so both time and spectral domain features can be used without extra cost and thus the selected features can be implemented in the domain in which it is easiest to implement.
In one embodiment, one of the features selected by the FeEM is the gradient index. The gradient index is based on the sum of magnitudes of the gradient of the speech signal at each change of direction. It can be defined as:
where ψ(κ)=½|ψ(κ)−ψ(κ)|, in which ψ(κ) denotes the sign of the gradient snb(κ)−snb(κ−1). The gradient index can have low values during voiced sounds and high values during unvoiced sounds.
Another feature which can be used by embodiments of the invention is the differential energy ratio. The differential energy ratio can be defined as the ratio of energy of the second derivative of the signal and the energy of the signal. The second derivative can be approximated with an FIR filter with impulse response h(n)=δ(n)−2δ(n−1)+δ(n−2). The differential energy ratio can be expressed as:
Another feature which can be used with various embodiment of the invention is the ratio of energies feature. This feature is capable of detecting temporal changes in relative signal energies and can be defined as the logarithm of the ratio of current and last frame energies. It can be expressed as:
Because the neural network can have a control task, it can be helpful to know something about the power at the different spectral bands of the original narrowband frame to be able to deduce the desired amplification levels and thus the control values that should be output to the MSM. One feature which can be used for this purpose is the average sub-band magnitude features. The average sub-band magnitude can be defined as the average magnitude of some or all of the spectral sub-bands. The average sub-band magnitudes can be transformed into a logarithmic (decibel) domain to make it easier for the neuroevolution to extract the relevant information and to make the feature more human readable. The average sub-band magnitude feature can be expressed as:
where k0 is the FFT index corresponding to the starting frequency and k1 is the FFT index corresponding to the ending frequency of the processed spectral band and Snb(k) is the FFT coefficient with index k.
The NNM can be responsible for transforming the input features passed from the FeEM into parameters used by the MSM to produce a magnitude shaping curve for the expansion band. The NNM weights are passed from the ESS genome. The NNM can be used as a function approximator to estimate the mapping from features to MSM parameters. In one embodiment, the ESS can be configured to do all the required learning so that no learning algorithm is required in the NNM.
The neural network shown in
In one embodiment, the inputs 68 for the neural network 64 can be normalized during the teaching process by scaling them using the estimated means and standard deviances for the features. For N available data of the kth feature the following calculations can be made:
The resulting normalized features can have zero mean and unit variance. The online processing subsystem 14 deployed into the final operating environment can use the estimates found during training.
The LHTF can be configured to transform the narrowband input frame into a wideband frame, creating a basic highband which can be shaped by the MSM to form the final expansion band. In one embodiment, the wideband frame can be generated using spectral folding by controlled aliasing to produce the highband. The narrowband signal can be upsampled by two, by inserting zeros between the samples in the narrowband frame. This approach is similar to mirroring the lowband into the highband in the frequency domain.
In an alternative embodiment, transforming the lowband into a highband comprised a simple translation in the spectral domain. This creates an exact copy of the lowband as a highband.
In still another alternative embodiment, nonlinear distortion of the upsampled and lowpass filtered signal to generate the highband information and combining it with the original narrowband lowband in the spectral domain can be used. In this embodiment, the nonlinearity used is a simple quadratic, suing the function g(s(n))=(s(n))2, which can be configured to produce harmonic distortions only therefore ensuring that the tonal components of the generated wideband signal match the harmonic structure of the bandlimited signal during voiced sounds. One embodiment of the results of the nonlinear distortion is illustrated in
The MSM can be responsible for attenuating and amplifying the highband produced by the LHTF to produce the final, natural sounding expansion band for the speech frame. It can use information extracted from frame features by the neural network to create a modulation curve which can be used to adjust the spectral envelope of the LHTF generated highband to better resemble the original wideband signal.
a illustrates one embodiment of an LHTF expanded frame.
The shaping curve can be selected independent of the other modules by the MSM. The neuroevolution process can strive to optimize the module input parameters for the selected curve. Preferably, the magnitude shaping curve should not be too flexible, otherwise overfitting may occur. It is desirable that the magnitude shaping curve is coarse to guarantee that the harmonic structure of the speech signal remains continuous while the spectral envelope is adapted. In addition, the curve should be adapted to balance the number of free parameters to make learning more efficient, while still including enough parameters to be able to adapt well to the high frequency range of the original wideband signal during training.
Preferably, the magnitude shaping curve should be smooth, since abrupt, discontinuous changes in the spectral envelope generally rare and could cause the curve to have a long impulse response. This could lead to quality degradation caused by impulse response clipping induced by short frame length. A smooth curve is also intuitively more pleasing as one role of the MSM can be a filter in the source-filter model and with the LHTF performing the role of the source. Therefore, preferably the magnitude shaping curve should be continuous. In practice, to this end, the LHTF can be configured so that it does not introduce any spectral envelope discontinuities.
In addition to other features of the magnitude shaping curve, computational efficiency can be considered as well as real-time curve generation. In one embodiment of the invention, cubic spline can be used as the curve type. They are smooth, local (a change in some part of the curve changes only a finite number of control points surrounding it), interpolative curves with feasible computational requirements. Interpolativity can make it easy to achieve a continuity in the spectral domain between the low and high bands by simply setting the first control point of the curve to move the beginning of the highband spectrum to coincide with the lowband spectrum endpoint.
The magnitude shaping control for splines can be done using fixed frequency control points. The selection of the control points can affect the flexibility and level of control the curve has on different sub-bands of the signal. As such, the number of control points should be such that adequate frequency resolution for efficient control of the spectral envelope is reached, but the module does not alter the harmonic structure of the signal or adapt to possible noise in the teaching samples.
In addition to the number of control points, their relative locations can be assigned. In one embodiment of the invention, instead of setting the control points to a fixed distance from each other they are set to fixed frequency warped locations to give the control system a frequency resolution similar to that of the human auditory system. In embodiments using telephone speech samples, care should be taken when setting control points in spectral gaps. In one embodiment, fixed amplification can be used to prevent excessive amplifications or attenuations for control points situated in spectral gaps.
The narrowband signal entering the online processing subsystem 76 is feed into the FeEM 84 as well as the LHTF 86. The FeEM 84 evaluates features of the frames of the narrowband signal and passes selected features to the NN 82 in the form of parameters. The NN 82 transforms the feature parameters into magnitude shaping information. In this embodiment, the magnitude shaping information is in the form of spline control points 88. The spline control points 92 are fed into the MSM 94. The MSM 94 uses the spline control points 92 to produce a magnitude shaping frame 96.
Some of the neural network outputs can be feed back to the FeEM 90 to be used as feedback for the next processed frame. These feedback outputs can be arbitrary output information selected by the evolution which can be used to help in the expansion of the next frame.
Simultaneously, the LHTF 86 transforms the inputted narrowband signal into a unshaped wideband signal 98. The unshaped wideband signal 98 is combined with the magnitude shaping frame 96 by a combiner thus producing an artificially expanded wideband signal from the inputted narrowband signal.
It should be understood that the invention is not confined to the particular embodiments set forth herein as illustrative, but embraces all such modifications, combinations, and permutations as come within the scope of the appended claims. The present invention in not limited to a particular operating environment. Those skilled in the art will recognize that the systems, methods, devices and computer code products of the present invention may be advantageously operated on different platforms. Thus, the description of the exemplary embodiments is for purposes of illustration and not limitation.
