Patent Application
20080147385
The present invention relates generally to speech processing. More particularly, the present invention relates to the implementation of voice conversion in speech processing.
This section is intended to provide a background or context to the invention that is recited in the claims. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.
Voice conversion is a technique that is used to effectively shield a speaker's identity, i.e., to modify the speech of a source speaker, such that it sounds as if the speech were spoken by a different, “target” speaker.
A variety of different voice conversion systems are currently under development, and such systems may be used in a variety of applications. For example, voice conversion can be utilized for extending the language portfolio of high-end text-to-speech (TTS), also referred to as high-quality or HQ TTS systems for branded voices in a cost efficient manner. In this context, voice conversion can be used to make a branded synthetic voice speak in languages that the original individual cannot speak. In addition, new TTS voices can be created using voice conversion, and the same techniques can be used in several types of entertainment applications and games. There are also several new features that could be implemented using the voice conversion technology, such as text message reading with the voice of the sender.
One technique that can be used in voice conversion involves utilizing a codebook-based approach. A codebook is a collection acoustic units of speech sounds that a person utters. Codebooks are structured to provide a one-to-one mapping between unit entries in a source codebook and the unit entries in the target codebook. The codebook is sometimes implemented by incorporating all of the available training data into the codebook, and sometimes a smaller codebook is generated. Codebook-based voice conversion is discussed in M. Abe, S. Nakamura, K. Shikano, H. Kuwabara, “Voice Conversion through Vector Quantization”, in Proceedings of ICASSP, April 1988, the content of which is incorporated herein by reference in its entirety.
Although promising, codebook-based techniques have traditionally suffered from a number of drawbacks. For example, when codebooks are used, the output often contains a number of discontinuities. Additionally, the memory requirements and the computational complexity can become large using a codebook-based approach if the objective is to achieve accurate conversion results. One attempt to improve the continuity issue in voicebook-based voice conversion is discussed in L. M Arslan, David Talkin, “Voice Conversion by Codebook Mapping of Line Spectral Frequencies and Excitation Spectrum”, in Proceedings of Eurospeech, September 1997, the content of which is incorporated herein by reference in its entirety. However, it would be desirable to still further alleviate the issues discussed above, while also improve the conversion accuracy when codebook-based approaches are used.
Various embodiments of the present invention provide an improved system method for codebook-based voice conversion that both significantly reduces the memory footprint and improves the continuity of the output. The various embodiments may also serve to reduce the computational complexity and enhance the conversion accuracy. The footprint reduction is achieved by implementing the paired source-target codebook as a multi-stage vector quantizer (MSVQ). During the conversion, N best candidates in a tree search are taken as the output from the quantizer. The N candidates for each vector to be converted are used in a dynamic programming-based approach that finds a smooth but accurate output sequence. The method is flexible and can be used in different voice conversion systems. In addition to the above, the various embodiments can be used to avoid over-fitting training data; they can be adjusted to different use cases; and they are scalable to different memory footprints and complexity levels. Still further, the system and method comprise a fully data-driven technique; there is no requirement to gather any language-specific knowledge.
The various embodiments of the present invention can be used in conjunction with the voice conversion framework described in U.S. patent application Ser. No. 11/107,334, filed Apr. 15, 2005 and incorporated herein by reference in its entirety.
These and other advantages and features of the invention, together with the organization and manner of operation thereof, will become apparent from the following detailed description when taken in conjunction with the accompanying drawings, wherein like elements have like numerals throughout the several drawings described below.
Various embodiments of the present invention provide an improved system method for codebook-based voice conversion that both significantly reduces the memory footprint and improves the continuity of the output. The various embodiments may also serve to reduce the computational complexity and enhance the conversion accuracy. The method is flexible and can be used in different voice conversion systems. In addition to the above, the various embodiments can be used to avoid over-fitting training data; they can be adjusted to different use cases; and they are scalable to different memory footprints and complexity levels. Still further, the system and method comprise a fully data-driven technique; there is no requirement to gather any language-specific knowledge.
The footprint reduction is achieved in the various embodiments of the present invention by implementing the paired source-target codebook as a MSVQ. During the conversion, N best candidates in a tree search are taken as the output from the quantizer. The N candidates for each vector to be converted are used in a dynamic programming-based approach that finds a smooth but accurate output sequence.
The training of the paired source-target quantizer is performed in a joint source-target space, using a distortion measure operating in the source-target space. All of the individual stages can be trained simultaneously using a multistage vector quantizer simultaneous joint design algorithm. One such algorithm is described in detail in LeBlanc, W. P., Bhattacharya, B., Mahmoud, S. A. & Cuperman, V., “Efficient Search and Design Procedures for Robust Multi-Stage VQ of LPC Parameters for 4 kb/s Speech Coding”, IEEE Transactions on Speech and Audio Processing 1, 4 (1993). p. 373-385, the contents of which are incorporated herein by reference in its entirety. Once training has been completed, a search is performed using only the source side of the space, while the output is produced using only the target portions of the joint vectors.
For the MSVQ, the number of stages and the sizes of the stages can be adjusted depending on design goals, including goals relating to target accuracy, memory consumption, computational complexity, etc. The search procedure can be implemented, for example, using a M-L tree search procedure. This procedure is depicted in
After the N best candidates are available for a given number of vectors to be converted, the optimized output sequence is obtained using dynamic programming. For each candidate, the corresponding source-space distance is stored during the search procedure. In addition, a transition distance is computed between each neighboring candidate pair. These distances together are used in the dynamic programming-based approach for finding an “optimal output sequence,” i.e. the path that results in the smallest overall distance. The relative importance between the accuracy and the smoothness can be set using user-defined or predetermined weighting factors.
In the depiction shown in
The following compares the use of one embodiment of the present invention with a pair of conventional conversion systems. These method were tested in a practical voice conversion environment in the conversion of the line spectral frequencies (LSFs). The 10-dimensional LSF parameters were estimated from 90 sentences at 10 millisecond intervals. 14,942 vectors were selected for training, and a distinct set of another 14,942 vectors were used for testing. As mentioned above, this test included three models. The first model followed an embodiment of the present invention, using three stages with 16 vectors in each stage. The second model included a full codebook containing all of the training vectors. The third model contained a small codebook having the same footprint as the embodiment of the present invention described in the first model (with real source-target vectors). The dynamic programming process was omitted to obtain comparable results.
The three models were evaluated from three different viewpoints: performance/accuracy, memory requirements, and computational load. The accuracy was measured using the average mean squared error, while the memory requirements were computed as the number of vector elements that have to be stored in the memory. The computational load was estimated as the number of vector comparisons required during the search procedure. The results of the evaluation, computed using the testing data, are summarized in Table 1 below.
The results outlined in Table 1 show that the selected embodiment of the present invention performed strongly from all aspects: it clearly provided the best accuracy and the lowest memory usage. While the third model offered similar memory and complexity levels, the conversion accuracy was significantly lower that the selected embodiment of the present invention.
The present invention is described in the general context of method steps, which may be implemented in one embodiment by a program product including computer-executable instructions, such as program code, executed by computers in networked environments. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.
Software and web implementations of the present invention could be accomplished with standard programming techniques with rule based logic and other logic to accomplish the various database searching steps, correlation steps, comparison steps and decision steps. It should also be noted that the words “component” and “module,” as used herein and in the claims, is intended to encompass implementations using one or more lines of software code, and/or hardware implementations, and/or equipment for receiving manual inputs.
The foregoing description of embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the present invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the present invention. The embodiments were chosen and described in order to explain the principles of the present invention and its practical application to enable one skilled in the art to utilize the present invention in various embodiments and with various modifications as are suited to the particular use contemplated.
