Patent Grant
8825186
The invention concerns digital audio processing and in particular the detection of periods where samples can be deleted or repeated unobtrusively so as to change the average sample-rate or to provide time delay modification.
It is often necessary to modify the sample rate of a digital audio signal or to change the number of samples that represent a particular time segment of an audio signal in a file or digital store. Such modifications are used to synchronise the sample rates of independently-sourced digital signals, or to modify the size of a file or data set. A temporary increase or decrease in the sampling frequency can also be used as a means of delaying or advancing the timing of audio information.
Where there are no limitations on processing time or complexity, interpolation of new samples having the required sampling frequency and phase by taking weighted sums of existing samples is often the preferred method. However, a known alternative method, which has the advantage of not introducing significant delay in real-time processing applications, is the deletion or repetition of samples at points chosen to minimise the audibility of the deletion or repetition. An effective method for identifying such points is thus an important feature of audio sample rate converters operating by sample deletion and repetition.
The inventors have appreciated that points for unobtrusive sample repetition or deletion can be determined from evaluation of the differences between the values of succeeding audio samples.
The invention consists, in one aspect, of a method and apparatus for changing the number of samples representing a time segment of an audio signal by deleting or repeating existing samples at positions in the said time segment dependant on the values of said audio samples characterised in that
In some embodiments stereo audio is processed and respective left and right differences between succeeding sample values are evaluated and samples are repeated or deleted where both the left difference and the right difference are below the said threshold at two or more consecutive sample points.
Suitably, a sample is deleted or repeated in response to a defined number of consecutive inter-sample differences being less than the threshold value.
Advantageously, the said threshold value is reduced when samples are deleted or repeated and the said threshold value is increased when a difference between succeeding samples exceeds the said threshold value.
In one embodiment sample deletion or repetition is inhibited during a set number of sample periods following a point of sample deletion or repetition.
Suitably, the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence on a cumulative measure of inter-sample difference values.
Advantageously, the said set number of sample periods during which sample deletion or repetition is inhibited is reduced in response to an increase in the said threshold value.
In an alternative embodiment the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence upon a measure of audio frequency.
The said measure of audio frequency may be a count of zero crossings of the audio signal.
In a further embodiment the said set number of sample periods during which sample deletion or repetition is inhibited is adjusted in dependence upon a measure of absolute audio amplitude.
An example of the invention will now be described with reference to the drawings in which:
A system for identifying candidate digital audio samples for nearly-inaudible deletion or repetition according to an exemplary embodiment will now be described with reference to
An input stream of audio samples (1) is applied to a subtractor (2) and a single-sample store (3). The subtractor (2) determines the magnitude of the value-difference between the current sample and the previous sample from the store (3). This value-difference magnitude between a pair of consecutive samples (separated in time by one period of the sampling clock) is input to a comparator (4) which compares it with a threshold value (5). The threshold value (5), which is greater than zero, is determined by a silence-threshold block (6) which will be described in detail below.
If the audio is silent, successive samples will have the same value and so the value-difference magnitude will be zero, and the comparator (4) will determine that it is below the threshold value (5). If the audio is nearly-silent the difference is also likely to be below the threshold value, but the outcome will depend on the frequency spectrum and the amplitude of the audio; lower-frequency, lower-amplitude audio will be more likely to give a difference-value below a given threshold value.
The output from the comparator (4) is passed to a control logic block (7) which generates commands (8) to delete (drop) or repeat the current sample in dependence upon a sample-rate control input (9). The control input (9) is derived from an external process (not illustrated) which determines whether the number of samples needs to be increased or decreased. If the external process determines that fewer samples are required, the control signal (9) causes the control logic block (7) to issue sample deletion commands (8) at points in the audio where the deletion is unlikely to be audible. Similarly, if the external process determines that additional samples are required, the control signal (9) causes the control logic block (7) to issue sample repetition commands (8) at points of low audibility. If no change to the number of samples is required, the control signal (9) causes the control logic (7) to prevent the output of sample repetition or deletion commands (8).
In some embodiments the control logic block (7) only outputs a sample deletion or sample repetition command if a fixed, small number (say 4) consecutive value-difference magnitudes have been less than the threshold (5); and/or, after a command has been issued, no further commands are issued until a further fixed number (say 1,020) of succeeding input samples have been received. These numbers are chosen to achieve the necessary maximum range of adjustment of the sample rate and to minimise the audibility of the process. The commands (8) are passed to an external process (not illustrated) which can either delete or repeat the current sample so as to obtain a modified sample stream or set of samples.
The silence-threshold block (6) determines the threshold value (5) so as to ensure that samples resulting in a value-difference less than the threshold value can be deleted or repeated with minimal subjective distortion to the audio. This is done by a process of continuous adjustment of the threshold in response to the sequence of audio samples to be processed. The threshold is lowered by a fixed decrement value whenever a sample deletion or repetition command is issued; and, the threshold is raised by a fixed increment value whenever the current value-difference magnitude is greater than the current threshold.
The level at which the threshold stabilises for a given audio signal of constant amplitude will depend on the ratio of the rate of threshold increment to the rate of threshold decrement. The fixed decrement value and the fixed increment value can be chosen so as to achieve automatically a threshold value for which sample deletion or repetition is substantially inaudible for a wide range of audio material. Suitable maximum and minimum limits to the variation of the threshold value can also be applied and the increment and decrement functions inhibited when necessary to keep the threshold value within these limits. The maximum and minimum limits can be set at the levels at which the audibility of sample deletion or repetition is just acceptable for the loudest and quietest signals respectively.
The process of threshold increment and decrement can continue during periods where, according to the control signal (9), no sample modification is required. This will ensure that sample modification can be re-started quickly without waiting for the threshold level to adapt to the characteristics of the audio signal.
It is possible to extend the technique to a two channel stereo audio system having left and right audio channels where samples are simultaneously deleted or repeated in both channels. Respective inter-sample difference magnitudes are determined for both audio channels and compared with a single threshold. In the system of
Either the single-channel or the stereo system can be improved by providing for the rate of sample modification (i.e. deletion or repetition) to be adjusted in dependence on the dynamics of the audio. If the amplitude of the audio is decaying after a loud peak, there is a short time period after the peak (of the order of a few tens of milliseconds) during which distortion of the audio caused by sample deletion or repetition will be subjectively masked. If the amplitude continues to decay, sample modification becomes more noticeable and can be perceived as a pitch change.
In the system of
Referring to
If the current audio sample was selected for modification in step (21), decision step (22) passes control to step (26) in which the current threshold value is stored. A further decision step (27) determines whether the current threshold value is above the threshold value stored on the previous occasion that a sample was modified. If the threshold is higher, the audio level is likely to be reducing after a peak (during which no sample modifications have occurred) and a faster rate of sample modification is likely to be acceptable. In this case, step (28) causes the delay between sample modifications to be reduced by changing the mode of operation of step (21) to a “Fast Modification” mode. This mode is maintained until a fixed number of samples have been processed in step (21).
After step (28), or a decision that the threshold is lower than at the time of the previous sample modification at step (27), the threshold value is decremented in step (29), as described with reference to FIG. (1), and control is passed to step (25) to commence the processing of the next audio sample.
In this way the Fast Modification mode is entered as the audio level begins to fall. In the Fast Modification mode the minimum permitted interval between sample modifications is set at a low value for a fixed number of samples. For example, the normal delay between modifications could be 1,024 sample periods and in the Fast Modification Mode the minimum permitted interval could be set to 32 sample periods until about 20 ms of audio has been processed; the minimum period can then be doubled after each succeeding 20 ms period until the “normal” delay of 1,024 samples is reached, or the Fast Modification mode is re-started at step (28).
When the audio is very quiet the process of
The system of
Dropping or repeating samples has the effect of changing the pitch of tones. If tones can be recognised and the pitch measured, it is then possible to determine a permitted rate of sample modification that causes a barely perceptible pitch change for that tone and to set the minimum permitted delay to that value. If the absolute amplitude of the audio can also be measured then the permitted rate of sample modification can be adjusted knowing the effect of loudness on the perceptibility of pitch changes. The system of the invention can thus be modified by the addition of frequency or amplitude measuring processes which control the time between sample deletion or repetition. Simple frequency measurement by detecting and counting zero crossings of the audio signal, or signals for stereo, can be used.
The application of the invention to stereo audio has been described; however, it may also be extended to other multi-channel audio formats such as “surround sound” or “5.1 channel” systems. In one method a multi-channel audio programme is “mixed down” to a mono or stereo version, which is analysed as described above, and samples are deleted or repeated simultaneously in all the channels of the multi-channel programme in response to the analysis of the mixed-down audio.
Another method is to compare inter-sample difference values from more than two channels with the threshold; and, to adjust the threshold and simultaneously delete or repeat samples in all channels in dependence on the results of the comparisons.
The invention has been described by way of examples and other variations are possible within the scope of the following claims. It should be noted that this invention can be to hardware and/or software arrangements as will be apparent to a skilled person.
| Number | Date | Country | Kind |
|---|---|---|---|
| 0715758.9 | Aug 2007 | GB | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 3812423 | Cronson et al. | May 1974 | A |
| 4434498 | Mathieu | Feb 1984 | A |
| 4899146 | Podolak et al. | Feb 1990 | A |
| 4953214 | Takeguchi et al. | Aug 1990 | A |
| 5136529 | Makie et al. | Aug 1992 | A |
| 5298899 | Oh et al. | Mar 1994 | A |
| 5692017 | Shiokawa | Nov 1997 | A |
| 5834930 | Hung et al. | Nov 1998 | A |
| 6144413 | Zatsman | Nov 2000 | A |
| 6304297 | Swan | Oct 2001 | B1 |
| 6697688 | Kuwaoka | Feb 2004 | B1 |
| 6724834 | Garrett et al. | Apr 2004 | B2 |
| 6757296 | Larsson et al. | Jun 2004 | B1 |
| 6836456 | Tonami | Dec 2004 | B2 |
| 6961698 | Gao et al. | Nov 2005 | B1 |
| 7088774 | Moni et al. | Aug 2006 | B1 |
| 7194379 | Chivers | Mar 2007 | B1 |
| 7672742 | Ng et al. | Mar 2010 | B2 |
| 7894489 | Shenoi | Feb 2011 | B2 |
| 8254404 | Rabenko et al. | Aug 2012 | B2 |
| 20030043856 | Lakaniemi et al. | Mar 2003 | A1 |
| 20050021327 | Gilbert | Jan 2005 | A1 |
| 20050053130 | Jabri et al. | Mar 2005 | A1 |
| 20060029142 | Arad | Feb 2006 | A1 |
| 20060039471 | Dane et al. | Feb 2006 | A1 |
| 20060146185 | Chen | Jul 2006 | A1 |
| 20060184261 | Ng et al. | Aug 2006 | A1 |
| 20060197692 | Gong et al. | Sep 2006 | A1 |
| 20060222130 | Dominique et al. | Oct 2006 | A1 |
| 20080007653 | Mori et al. | Jan 2008 | A1 |
| 20080082189 | Kuwaoka | Apr 2008 | A1 |
| 20090103573 | LeBlanc et al. | Apr 2009 | A1 |
| Number | Date | Country |
|---|---|---|
| 1 684 516 | Jul 2006 | EP |
| 2039448 | Aug 1980 | GB |
| 2039448 | Aug 1980 | GB |
| 56080938 | Jul 1981 | JP |
| WO 0232054 | Apr 2002 | WO |
| Entry |
|---|
| European Search Report for European Application No. EP 08 16 2264 dated Oct. 16, 2008. |
| Number | Date | Country | |
|---|---|---|---|
| 20090048696 A1 | Feb 2009 | US |
