Patent Application
20120323585
The present invention relates to the field of conferencing systems, and in particular to a technique for reducing audio artifacts caused by time compression of audio playout.
Traditionally, IP-based voice and video conferencing systems have communicated over reliable enterprise networks that control for quality of service. In such networks, the most significant timing impairment comes from relative clock drifts in the end points. As users increasingly set up remote and home offices, however, conferencing systems are now connected over less reliable networks such as wireless and the public Internet. In such networks, timing impairments such as jitter and out of order packets are likely to occur with greater frequency and increased severity.
Consider the impairment of jitter. During a period of network congestion, packets may arrive at the conference system in large bursts. For audio, the passive solution is a deep buffer, which the system can fill from the network at a bursty rate. Meanwhile, the system plays the audio out of the buffer to the listener at a consistent smooth rate. This rate is equal to some desired play-out frame rate. While this solution is simple, the large buffer required has the downside of adding significant audio latency.
Conferencing systems have attempted to avoid the latency problem by using a time-compression algorithm to modify the speed of audio play out. Such algorithms use signal processing to shorten the duration of an audio signal without affecting pitch. When used to combat network jitter, a burst of many frames from the network is time compressed to reduce the number of frames to be played out to the listener.
Ideally, time-compression algorithms would create very natural sounding audio while handling the significant compression needed for network jitter. In fact, however, at high compression rates, these algorithms often result in audio artifacts. The two dominant artifacts that can be found in systems using existing algorithms can be described as sounding rough and sounding ghostly.
In some systems frequency domain techniques such as phase vocoders have been used. These techniques tend to have artifacts that could be described as having a ghostly sound. Time compression techniques have frequently generated rough sounding artifacts. Reducing these artifacts would improve the user's experience with conferencing systems.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate an implementation of apparatus and methods consistent with the present invention and, together with the detailed description, serve to explain advantages and principles consistent with the invention. In the drawings,
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the invention. It will be apparent, however, to one skilled in the art that the invention may be practiced without these specific details. In other instances, structure and devices are shown in block diagram form in order to avoid obscuring the invention. References to numbers without subscripts or suffixes are understood to reference all instance of subscripts and suffixes corresponding to the referenced number. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment.
Various techniques are disclosed for improving time compression to reduce artifacts by adapting the time compression based on the state of the received audio. One feature of embodiments disclosed herein involves bounding the amount of time compression based on audio characteristics. Another feature provides a way of determining the most correlated portions of segments of audio. Another feature provides a way of distinguishing between voiced speech and unvoiced speech. Another feature provides a way of distinguishing between silence, voiced speech, and unvoiced speech. Another features provides adapts time compression during periods of lengthy silence. Another feature allows for reducing time compression during sensitive portions of the received audio. One or more of these features may be present in different embodiments.
In the following, the terms “packet” and “frame” are used interchangeably. A “sample” is a single scalar number representing an instantaneous moment of audio. A frame or packet is a sequence of samples representing a span of time in the audio, typically 10 msec. A “pitch period” of an audio signal is a measurement of the smallest repeating unit of the signal, and may also be referred to as a “pitch length” of the audio.
Embodiments described below make time compression techniques more adaptive to audio conditions. Although the description below is set forth in terms of speech-based audio, many of the techniques can be used with non-speech based audio. Other techniques may also be employed for the reduction of artifacts, such as the techniques described in U.S. patent application Ser. No. 12/911,314, “Artifact Reduction in Packet Loss Concealment,” filed Oct. 25, 2010, which is incorporated by reference in its entirety for all purposes.
Although the techniques described below relate to time compression of audio, in various embodiments these techniques may be used where both audio and video signals are available and should be kept synchronized, such as in audio-video conferencing systems, where lip-synch errors between video of a speaker and the corresponding audio can lead to subconscious viewer stress and dislike of the conferencing experience.
In speech-based audio, there are three dominant states: silence, voiced-speech, and unvoiced-speech. In one embodiment, to meet artifact and compression rate goals, each speech state may be processed differently. For example, sections of silence may be removed, voiced-speech may be shortened by increments synchronized to a pitch period of the audio, and white unvoiced-speech may be shortened more aggressively without synchronization.
To differentiate between these three speech states, measures such as signal-to-noise ratio (SNR), correlation, and whiteness may be employed, as described in more detail below.
One conventional time compression technique uses what is known as an Overlap-And-Add (OLA) approach. This is a time domain method of compression. The amount to be shortened is determined by the length of overlap of two portions of audio. The audio signal is cut into two segments, the segments are overlapped, and the segments are “added” together to produce a time-compressed audio signal. A common technique is to take two consecutive 10 ms frames of audio, and completely overlap them, shortening the audio play out by the 10 ms overlap length.
A better approach known to the art is Synchronized OLA (SOLA), in which the segments are synchronized to some measured factor, such as the pitch of a speaker's voice or another sound, and the frames are slid across each other until a maximum correlation is achieved at the overlap point. SOLA techniques typically overlay by an integer number of pitch periods, to avoid phase jump artifacts.
A third approach known to the art is known as silence removal. While both OLA and SOLA techniques may generate artifacts in the overlapped and added audio, silence removal avoids artifacts by simply removing or shortening periods of silence.
A desired approach to time compression generally balances the amount of compression achieved with a number of artifacts generated in the resulting audio.
Time compression using OLA and SOLA techniques cuts an audio signal into two segments. For example, let x be an array of N samples of an audio signal, stored in memory, where the oldest sample available in memory is designated x[1] and the newest sample in memory is designated x[N]. In such embodiments, N represents small time periods from around 10 ms to as much as a few hundred milliseconds.
Now cut the signal into two segments by defining a cut point c. Let x[1] through x[c] be the older segment and let x[c+1] through x[N] be the newer segment for a given cut value c. OLA techniques shift the newer segment back in time so that the two segments overlap by a length of L. In an OLA technique, L is always ½ N; in a SOLA technique, L may be other values, depending on the synchronization. The embodiments below generally provide a way to optimize the value of L so that artifacts are avoided.
In the overlap region, the two segments are merged or added together, typically using a weighted addition, such as is described below. The result is an output signal sample array y that is L−1 samples shorter than x[N].
To get time compression of more than L−1 samples, the technique may be performed iteratively, so that some samples of the array y from a prior iteration can be used as part of the array x for a new iteration. From iteration to iteration, the size of N may be a function of the number of the samples fed-back from the array y, the number of new audio samples received from the input source, and the play-out frame rate. In this iterative embodiment, the array x may contain both new samples and samples from previous iterations of the overlapping technique.
If the audio signal is received at a rate of 1600 samples per second, then a 10 ms frame has 160 samples, thus generally leading to a value of N of 160, since the system keeps at least one frame of audio in memory. However, if the signal is bursty, so that the 10 ms frame has 3200 or 4800 samples per second, for example, then N may be 320 or 480. Consider a situation where N=320. If the system is able to compress out only 100 samples of the 320 original samples, leaving 220 samples, and plays out 160 samples during one iteration, then 60 samples may be left over for the next iteration, so that even if 160 newly received samples are buffered in memory, the total number of buffered samples in memory may be 220.
In one embodiment, instead of cutting the buffered samples into two pieces and overlapping them, the buffered samples may be cut into a larger number of pieces, each of which is then overlapped with the preceding piece. In such an embodiment, N may not be the all of the samples stored in memory, but is simply the number of samples used for determining the overlap of any given iteration. For example, if the buffer is divided into three equal pieces, then N may take the value of the number of samples in the first ⅔ of the buffer.
As described in more detail below, the disclosed techniques attempt to optimize the values of c and L before performing OLA. Various embodiments may trade-off competing aspects, such as minimizing computational complexity, maximizing the time compression, minimizing overall algorithmic delay, and minimizing audio artifacts. Various embodiments described below define some threshold values that may be tuned for best audio quality, including a maximum overlap length Lmax, a low SNR Threshold, a high SNR Threshold, and a white Threshold.
In one embodiment, the technique is run iteratively. That is, the actions performed by the technique are performed once per output frame. One or more frames of audio may be used as input to the technique, producing a single frame of output audio. The frame rate period may vary, but is typically in the range of 10 ms to 30 ms. At the beginning of the technique illustrated in
In block 310, a value for c is computed. A simple computation for c would be
c=L=floor(N/2)
However, such a simple application for c would result in artifacts when the time-compression technique is used iteratively. For example, assume that a subset of the samples x[1] through x[D] have already been in the overlap region of a prior time-compression iteration. Compressing those samples again is likely to introduce undesirable artifacts. To minimize artifacts on the current iteration, the overlap should only be done on samples x[D+1] through x[N]. Therefore a better version of c would be
c=D+floor((N−D)/2)
In addition, experience shows that the overlap regions tend to yield artifacts. Therefore, in one embodiment an additional constraint may be placed on c by using a tuned maximum overlap length Lmax, as follows:
c=Min(D+floor((N−2)/2), D+Lmax)
In one embodiment, the Lmax value may be tuned to a maximum pitch expected in the audio. For speech audio, the Lmax value may be tuned based on a maximum pitch of a speaker's voice, and may be experimentally chosen, by an implementer sampling audio to determine a maximum pitch of the speaking voice of any person.
In block 320, an array of values SNR[n] may be computed as a measure of a signal-to-noise ratio across the samples in the array x.
In block 330, the technique may compute a value for L_MostCorr. The newer of the two OLA segments is to be slid back by some distance L, creating an overlap region with the older segment. The technique searches all possible overlap lengths, and L_MostCorr should be chosen the one that creates the highest correlation between the older segment and the newer segment. Although SOLA techniques use a similar approach to determining a most synchronized overlap of the segments, in SOLA the maximum testable L_MostCorr is c. In one embodiment, correlations may be tested beyond c, meaning that the newer segment will overhang beyond x[1] in the correlation computation. Such a technique helps prevent false pitch detection.
Preferably, the correlation computation determines an actual dominant pitch in the room, and L_MostCorr should be tested beyond the overlap range. For example, if the testing for maximum correlation only went back as far as a maximum human pitch period and found a peak, deciding that the peak represented a dominant pitch in a room with the speaker, such a testing might overlook external sound sources, such as thunder going on outside the room. Thus, the computation might be selecting correlations based on something not really dominant in the room.
Although in the thunder example the low frequency of the thunder itself may be out of range, harmonic frequencies may be within the range that are considered in this computation.
In block 340, an array of correlations may be calculated across the entire set of samples in memory, for use in computing a whiteness value W. If the ratio between the highest correlation and the lowest correlation is not large, then the samples may be considered relatively white and the largest correlation may not be considered very meaningful. Thus in one embodiment, the value for W may be computed as a ratio between the correlation at the L_MostCorr sample and the lowest correlation at all other tested L values.
In block 350, a number of quiet samples #Q may be computed. In one embodiment, this value may be computed by computing a number of quiet samples in each of two portions of the array x as follows. A first number of quiet samples may be calculated as the maximum value of #Q1 such that
SNR[c+#Q1+1] through SNR[c]<LT
A second number of quiet samples may be calculated as the maximum value of #Q2 such that
SNR[c+1] through SNR[c+#Q2]<LT
The number of quiet samples #Q may then be calculated as
Max(#Q1, #Q2)
In one embodiment, the low SNR Threshold value LT and a corresponding high SNR Threshold value HT may be experimentally determined using listening tests. These thresholds are defined so that samples below the low SNR Threshold value are probably quiet, samples above the high SNR Threshold value are definitely not quiet, and samples in between those two thresholds are uncertain. In one embodiment, the low SNR Threshold value LT and the high SNR Threshold value HT are tuned based on what produces the least artifacts in the time-compressed audio.
Blocks 415 through 430 illustrate one embodiment of adjusting the value L with a random number. In block 415, if this block is reached frequently, such that periodicity might be introduced into the audio, then in block 420 the value R may be computed as a random number between 0 and the number of quiet samples #Q. otherwise, the value are may be set to 0 in block 425. In block 430, the overlap value L is calculated as follows:
L=#Q−R
The second portion of the technique 400 then completes, allowing the calculated value of L to be used for the actual compression of block 240 of
If the number of quiet samples #Q is less than the predetermined threshold as determined in block 410, then the audio may be considered to contain speech or other non-quiet sound. For such audio, in block 435 a determination is made of whether the maximum value of signal-to-noise ratio data stored in the SNR array is less than the high SNR Threshold value HT.
If so, then in block 440 the whiteness of the audio data is compared to the whiteness threshold value WT. An implementer may determine the whiteness threshold WT by listening tests.
If the signal-to-noise ratio is above the high SNR Threshold value HT or the audio data has a whiteness less than the whiteness threshold value WT, then the L_MostCorr value may be used to determine the overlap amount L in blocks 445 through 455. Otherwise, the correlation information may be considered of too little value to be used, and the value L may be computed in block 460 through 475 from the values C and D that were determined as described in block 310 above. Thus, unlike previous techniques, midrange correlated audio data that is considered white (e.g., unvoiced speech) may be treated similar to silent periods such as the techniques described in blocks 415 through 430 above. The use of the whiteness computations allows distinguishing unvoiced speech from voiced speech, and treating unvoiced speech similar to silence.
If the L_MostCorr value is to be used, then in block 445, the L_MostCorr value is checked to ensure that it is not too large or too small for quality overlap and add time compression. Thus if the L_MostCorr value is greater than (c−D), the value is too large, because it would compress audio data that was compressed in the previous iteration. If the L_MostCorr value is less than a predetermined minimum most correlated overlap length threshold value, then the overlap region may be considered too small to have a smooth overlap and add without artifacts. The minimum L_MostCorr value threshold may be selected as desired; in one embodiment, the threshold value is experimentally determined by listening tests. If the L_MostCorr value is usable, then the overlap amount L is set to the L_MostCorr value in block 450; otherwise, no overlap is feasible and the overlap amount L is set to 0 in block 455.
If instead of using the L_MostCorr value, because the relatively white audio is to be treated as if it were silent, then in blocks 460 through 475 a frequency of the event approach is used similar to the whiteness randomization described above in blocks 415 through 430.
In block 460, if the audio is considered white in too many iterations of the technique, then a random value may be used to avoid artifacts that would otherwise be generated from overlapping to frame boundaries. As in the randomization of blocks 415 through 430, in various embodiments a threshold value for “too many iterations” may be predetermined as a number of consecutive iterations, for example 10 consecutive iterations, or may be predetermined as a number of recent iterations, regardless of consecutiveness, such as 10 of the last 15 iterations. The threshold value may be considered as a threshold number of audio frames in a recent period that contain unvoiced speech. If a randomization is to be performed, then in block 465 a random number R is calculated as a random number between 0 and c−D; otherwise, in block 470 the value R is set to 0. Then in block 475, the overlap amount L is calculated as
L=(c−D)−R
Once the overlap amount L is calculated, then the overlap and add compression of block 240 may be performed.
In block 510, and output array y is created from the input audio array x for the portion prior to the overlap. Thus,
y[k]=x[k]
for values of k between 1 and (c-L).
In block 530, the portion beyond the overlap region is also copied into the array y:
y[k]=x[k+L]
for values of k between c+1 and N.
In block 520, the actual overlap is performed, for values of k between (c-L+1) and c. In one embodiment, functions w1 and w2 are used to weight the values of the corresponding x array values. In one embodiment, the function w1 is implemented as an audio fade out and the function w2 is implemented as an audio fade in. Thus, the output audio array y starts out the same as the input audio array x and in the same as the input audio array x, but during the middle samples, a listener would hear the first segment of the input array x fade out as the second segment fades in. The result is that the audio sequence of N samples is compressed to a sequence of (N-(L−1) samples.
The use of fade in and fade out functions w1 and w2 is illustrative and by way of example only. In other embodiments, other types of functions w1 and w2 may be used, including ones that simply switch the audio output from the older segment to the newer segment without any type of fade in/out.
The output audio array y may then be processed further or sent to the playback mechanism of the apparatus 100.
Unlike the conventional techniques, that attempt to mitigate audio artifacts by lowering the rate of compression, resulting in a loss of efficiency, the techniques described herein mitigate audio artifacts by adapting the rate of compression where necessary, allowing higher compression rates while preserving a low level of artifacts. Thus, the techniques described above, may allow more efficient time compression while maintaining or improving audio quality.
Some embodiments of the disclosed techniques ensure that true pitch phase is maintained, avoiding synchronization to a false pitch frequency that may result in rough artifacts. By tracking the status of the audio and adapting compression parameters, various embodiments may minimize artifacts while maximizing compression. By using a randomized approach to eliminating long period of silence or unvoiced speech, some embodiment may reduce unnatural artifacts that can be generated by removing blocks of audio from such long silent periods. This may be particularly valuable in a conferencing situation, where one direction of the audio may have long periods of silence.
By allowing for compression where feasible, but skipping compression where compression would lead to an artifact, compression may be concentrated in sections of audio where the processing can be masked. Thus, various embodiments of the apparatus and techniques described above provide a better listening experience for the user, particularly in conferencing environments.
It is to be understood that the above description is intended to be illustrative, and not restrictive. For example, the above-described embodiments may be used in combination with each other. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the invention therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.”
