Patent Application
20230162743
The present disclosure relates to audio processing, and in particular, to using audio watermarks to indicate audio processing.
Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
Media players are becoming more configurable, including the media players implemented in mobile telephones, The media player may include a variety of decoders, pre-processors, post-processors, etc. that perform various types of audio processing (e.g., according to user selection, preferences, machine learning, etc.).
A selected type of audio processing may be performed by components at more than one point in the audio processing chain. This results in the possibility that more than one component may perform the processing, resulting in double processing of the audio. One problem of double processing is that it consumes extra resources (electricity, processor cycles, battery life, etc.), which is especially undesirable in a mobile device. Another problem of double processing is that the double-processed audio may have a perceptible difference from (single) processed audio, resulting in a negative user experience.
One way to avoid double processing is to communicate between components that the processing has been performed. This communication may be via control signals, control messages, metadata, etc.
One issue with using control signals, control messages, metadata, etc. to communicate between components is that these communications must conform to the inter-component communication requirements of the mobile device operating system. For example, for security purposes, the operating system may not allow the communications to be passed directly between components, but instead may require the communications to be intermediated by a security component. This involves extra effort in many ways. instead of concentrating on the audio processing aspects, the audio component developer also needs to maintain expertise in the security aspects of the operating system. Second, if the operating system modifies its security system, the audio component developer is required to update the audio processing component to conform, even if there is otherwise no effect on the operational details of the audio processing. As a specific example, in the Android™ operating system, audio metadata cannot go through the Android™ audio chain directly due to the design of the Android™ architecture.
Given the above, there is a need to communicate information regarding double processing in ways other than using control signals, control messages, metadata, etc. between audio processing components. Described herein are techniques related to detecting double processing using audio watermarking.
According to an embodiment, a method of audio processing comprises detecting, by a processing component, a transient in first audio data. The method further comprises transforming a portion of the first audio data related to the transient into frequency domain data. The method further comprises comparing a first band of the frequency domain data and a second band of the frequency domain data. When the first band is uncorrelated with the second band, the method further comprises performing processing by the processing component on the first audio data to generate second audio data. When the first band is correlated with the second band, the method further comprises using the first audio data as the second audio data without performing processing by the processing component. In this manner, the method uses the detected audio watermark to determine whether or not the processing is performed.
The audio watermark may be inserted as per the following method. Prior to detecting the transient in the first audio data (see above), the method further comprises decoding, by a decoder component, third audio data to generate fourth audio data. The method further comprises detecting a transient in the fourth audio data, wherein the transient in the fourth audio data corresponds to the transient in the first audio data. The method further comprises transforming a first portion of the fourth audio data related to the transient in the fourth audio data into first frequency domain data. The method further comprises duplicating a first band of the first frequency domain data into a second band of the first frequency domain data to generate second frequency domain data. The method further comprises transforming the second frequency domain data to generate a second portion. The method further comprises generating fifth audio data, wherein the fifth audio data corresponds to the fourth audio data having the first portion replaced with the second portion. (The fifth audio data corresponds to the first audio data discussed above.)
According to another embodiment, an apparatus for audio processing includes a. processor and a memory. The processor is configured to control the apparatus to perform one or more of the method steps discussed above. The apparatus may additionally include similar details to those of one or more of the methods described herein.
According to another embodiment, a non-transitory computer readable medium stores a computer program that, when executed by a processor, controls an apparatus to execute processing including one or more of the methods described herein.
The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations.
Described herein are techniques related to audio watermarking. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.
In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.
In this document, the terms “and”, “or” and “and/or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and/or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e. “either A or B”, “at most one of A and B”).
The processor 102 generally controls the operation of the mobile device 100, The processor 102 may be one or more processors. The processor 102 may execute one or more computer programs, such as the operating system (e,g., the Android™ operating system, the iOS™ operating system, etc.), various application programs (e.g., a media player program, an audio effects program, etc,), etc, The processor 102 may include a digital signal processor (DSP), or may execute computer programs that implement DSP functionality.
The memory 104 generally stores the instructions executed by, and the data operated on, by the processor 102. These instructions and data may include the various computer programs (e.g., the operating system, the applications, etc.), media data (audio data, video data, audiovisual data, etc.), configuration data (e.g., user settings and preferences, etc.), etc. The memory 104 may include volatile components (e.g., random access memory (RAM), etc.), non-volatile components (e.g., read-only memory (ROM), flash memory, etc.), etc.
The radio 106 generally controls wireless data exchange between the mobile device 100 and other wireless devices and networks. The radio 106 may be one or more radios of various types, such as a cellular radio, an IEEE 802.11 standard radio (e.g., WiFi™ radio), an IEEE 802.15.1 standard radio (e.g., Bluetooth™ radio), etc. The radio 106 may function to obtain media content, for example streaming content (e.g., for processing by the processor 102), downloaded content (e.g., for storage by the memory 104), etc. The radio 106 may be omitted in certain embodiments of the mobile device 100 (e.g., when the mobile device 100 is a voice recorder with media player functionality).
The speaker 108 generally outputs sound corresponding to audio data. For example, the speaker 108 may output streamed audio data received by the mobile device 100, stored audio data stored by the mobile device 100, etc. The speaker 108 may be omitted in certain embodiments of the mobile device 100 (e.g., when the mobile device 100 connects to an external speaker via a wired or wireless connection). For example, the mobile device 100 may connect to wireless earbuds via the radio 106.
The microphone 110 generally receives sound that the mobile device 100 may use for various purposes. For example, the microphone 110 may receive background noise that the mobile device 100 may use to adjust how it processes audio data. As another example, the microphone 110 may receive voice commands that the mobile device 100 may use to control the media player functionality or to set user configuration preferences. The microphone 110 may be omitted in certain embodiments of the mobile device 100 (e.g., when the mobile device 100 connects to an external microphone via a wired or wireless connection).
The bus 112 generally connects the other components of the mobile device 100. The bus 112 may include one or more buses having one or more types, such as an inter-integrated circuit (I2C) bus, an inter-integrated circuit sound (I2S) bus, a serial peripheral interface (SPI) bus, etc.
In general, the mobile device 100 implements media player functionality to output media data, including audio data. The mobile device 100 may also implement audio post-processing, such as audio effects, on the audio data. As discussed in more detail below, the mobile device 100 implements an audio watermark to avoid double-processing of audio signals. The mobile device 100 may also implement additional functionality (e.g., telephone functionality, web browser functionality, camera functionality, two-factor authentication functionality, etc.) that for brevity is not described in detail.
The applications layer 202 generally includes applications that the mobile device 100 executes to implement various functions. For example, the applications in the application layer 202 may interact with operating system components in the framework layer 204 to implement the media player functionality. This arrangement enables the mobile device 100 to work with multiple applications, each having different functionality, that may be selected by the user according to their preferences. The applications layer 202 includes a media player application 210 and a user interface application 212.
The media player application 210 generally implements the media player functionality for the mobile device 100. The media player application 210 may be one of multiple media player applications on the mobile device 100, each having different functionality. Example functions implemented by the media player application 210 include media file organization (playlists, shuffle play, etc.), media playback (play, pause, skip, etc.), etc. The media player application 210 is generally built as a collection of lower level operating system functions provided by the framework layer 204.
The user interface application 212 generally implements user interface functionality related to the media player functionality of the mobile device 100. In particular to the audio processing framework 200, the user interface application 212 may be used to select various post-processing and audio effects that may be implemented outside of the media player application 210. (The media player application 210 itself may also implement the audio effects, depending upon the implementation.) These audio effects are discussed in more detail below.
The framework layer 204 generally includes framework components, operating system components, services and programming interfaces that the applications in the applications layer 202 use to implement the applications. For example, a particular media player application 210 in the applications layer 202 may be built using specific components in the framework layer 204 to implement the media file organization functionality, the media playback functionality, etc. Specific to the audio processing framework 200, the framework layer 204 includes a media player service 220 and an effects service 222.
The media player service 220 generally includes the framework components that implement the media player functionality. The media player service 220 interacts with the file system of the mobile device 100 to access an audio file 230 (e.g., stored audio data, streaming audio data, etc.). The media player service 220 interacts with various components in the vendor layer 206 (e.g., to perfoi in decoding, etc.), as further discussed below. The media player service 220 processes the audio file 230 and outputs an audio signal 232 to the effects service 222.
The effects service 222 generally includes components that implement post-processing functionality, including audio effects. The effects service 222 interacts with various components in the vendor layer 206 (e.g., to apply various effects), as further discussed below. The effects service 222 applies the audio effects to the audio signal 232 output from the media player service 220 and generates an audio signal 234.
The effects service 222 may also interact with a mixer component (not shown) in the framework layer 204. The mixer component generally mixes in system audio (e.g., alerts, notifications, etc) with the other audio signals. For example, when the user is listening to audio, the mixer may mix in a ring sound to indicate that the mobile device 100 is receiving a telephone call. The mixer component may mix the system audio prior to the effects service 222 (e.g., mixing with the audio signal 232), after the effects service 222 (e.g., mixing with the audio signal 234), etc.
As mentioned above, the components of the framework layer 204 may themselves interact with components in the vendor layer 206.
The vendor layer 206 generally includes components that are developed by entities other than the entity that developed the components in the framework layer 204. For example, the framework layer 204 may implement the Android™ operating system from Google LLC or the iOS™ operating system from Apple Inc.; the vendor layer 206 may implement components from Dolby Laboratories, Inc., Apple Inc., Sony Corp., the Fraunhofer Society, etc. This arrangement allows the components in the vendor layer 206 to extend the functionality of the mobile device 100 beyond the base functionality provided by the framework layer 204 yet remain within the control of the framework layer 204. Specific to the audio processing framework 200, the vendor layer 206 includes a decoder component 240 and a processing component 242.
The decoder component 240 generally performs decoding of the audio file 230. For example, the media player service 220 may invoke a particular decoder component 240 to decode a particular type of audio file 230. The decoder component 240 may also be referred to as a codec component, where “codec” stands for the combination of a coder and a decoder (although generally the term codec may be used even when the component does not perform coding). As mentioned above, the decoder component 240 may be one or more decoder components that implement one or more different decoding processes. For example, when the audio file 230 is an MP3 file, the media player service 220 may interact with an MP3 decoder as the decoder component 240. As another example, when the audio file 230 is a Dolby Digital Plus™ file, the media player service 220 may interact with a Dolby Digital Plus™ decoder as the decoder component 240. Other example decoders include Advanced Audio Coding (AAC) decoders, Apple™ Lossless Audio Codec (ALAC) decoders, etc. A particular decoder component 240 may also apply audio effects, as further discussed below.
The processing component 242 generally performs post-processing on the audio signal 232 to generate the audio signal 234, for example to apply audio effects. For example, the effects service 222 may invoke a particular processing component 242 to apply a particular audio effect to the audio signal 232. The processing component 242 may also be referred to as an effects processing component or a post-processing component. The processing component 242 may be one or more processing components that implement one or more audio effects. Audio effects include volume leveling, volume modeling, dialogue enhancement, and intelligent equalization. Audio effects are discussed in more detail below.
As discussed above, audio effects may be applied by multiple components in the framework layer 204. For example, the media player service 220 may generate the audio signal 232 with an audio effect by processing the audio file 230 using a selected decoder component 240. As another example, the effects service 222 may generate the audio signal 234 with an audio effect by processing the audio signal 232 using a selected processing component 242. When the audio signal 232 has the audio effect applied by the media player service 220, it would be desirable for the effects service 222 to refrain from applying the audio effect in order to avoid double processing.
Unfortunately, in the audio processing framework 200, the audio path is limited to audio signals. The term “audio path” generally refers to the audio input to and output from the media player service 220 and the effects service 222. For example, the audio path may only accept two channels of pulse-code modulation (PCM) samples represented by 16 bit integers. The audio path does not by itself allow additional control signals, metadata, etc. for the media player service 220 to indicate that it has applied an audio effect.
To overcome this limitation, the decoder component 240 inserts an audio watermark into the audio signal 232 to indicate that it has applied an audio effect. When the processing component 242 detects the audio watermark, it does not itself apply the audio effect; otherwise it applies the audio effect. In this manner, using the audio watermark avoids double processing, without requiring additional control signals, metadata, etc. to be passed outside of the audio chain.
The decoder component 302 receives the audio file 230 (see
The decoder component 302 may implement one or more decoding processes. In general, the decoding process performed will depend upon the format of the audio file 230.
example, the media player service 220 (in the framework layer 204, see
The transient detector 304 detects a transient in the audio signal 320. In general, a transient is a high amplitude, short-duration sound at the beginning of a waveform that occurs in phenomena such as musical sounds, noises or speech. A transient may also be described as a short-duration signal that represents a non-harmonic segment of a sound source. Generally, a transient occurs in the attack portion of the sound, but it also may occur in the release portion. A transient may contain a high degree of nonperiodic components and a greater magnitude of high frequencies than the harmonic content of that sound. A transient need not directly depend on the frequency of the tone it initiates (or terminates).
The transient detector 304 may use one or more processes to detect a transient. For example, the audio signal 320 may be a time domain signal composed of samples, with the samples grouped into units such as blocks, sub-blocks, frames, etc. The transient detector 304 may detect the transient in a particular block of the audio signal 320. The transient detector 304 may examine each block of samples for an increase in energy (above a defined threshold) from one block to the next. The block size and threshold may be adjusted as desired. For example, block sizes of 256 samples, 128 samples, 64 samples, etc. may be used. The threshold may be based on the relative peak levels of adjacent blocks; thresholds between 1.5 and 2.5 may be used, with 2.0 providing good results. The threshold may be lowered in order to detect more transients (e.g., so that a detection becomes more likely as more time passes), or increased to detect fewer (e.g., so that once a detection has occurred, there is less need to detect a subsequent transient in the near term). The transient detector 304 may dynamically adjust the threshold in order to achieve a target rate of transients detected in a given time period (e.g., 1 transient detected per 1 second). As a specific example, the transient detector 304 may implement transient detection as described in “Digital Audio Compression Standard (AC-3, E-AC-3) Revision B”, ATSC Document A/52B. When the transient detector 304 does not detect a transient, the decoder component 300 continues processing the audio file 230. (In such a case, the output of the decoder component 300 may be considered to be the audio signal 320.) When the transient detector 304 detects a transient, the flow continues with the components 306-312. When the transient detector 304 does not detect a transient, the flow may skip to the selection component 314.
The transform component 306 transforms a portion 328 of the audio signal 320 related to the transient into frequency domain data 330. For example, when the transient detector 304 detects a transient in a particular block of the audio signal 320, that particular block then corresponds to the portion 328.
The transform component 306 may use one or more transform processes to transform the portion 328. As an example, when the portion 328 is a block of 256 samples, the transform component may perform a fast Fourier transform (FFT) using a block size of 512 points and 256 points of overlap (referred to as a window); alternatively, block sizes of 1024 points or 2048 points may be used.
The transform component 306 may use a Hann (also referred to as a Hanning) window. Other window types may be used as desired. For example, a Hamming window may be used, with parameters a0=0.54 and a1=0.46. As another example, a Blackman window may be used, with parameter α=0.16. As another example, a Gaussian window may be used, with parameter Δ=0.1.
The duplication component 308 receives the frequency domain data 330 and duplicates one band (the source band) into another band (the target band) to generate frequency domain data 332. The process of duplication may also be referred to as replication or copying. In the frequency domain data 330, the target band may be referred to as the original target band; and in the frequency domain data 332, the target band may be referred to as the duplicated target band. This replacement of one band by another serves as the audio watermark. Because the replication (duplication) is performed in relation to a detected transient (e.g., after the transient), the perceptual masking may result in improved fidelity.
The duplication component 308 may also perform scaling of the energy in the target band so that the energy level of the duplicated target band matches the energy level of the original target band, instead of the energy level of the source band. For example, the spectral shape of the source band is duplicated, but the energy level of the target band is maintained. The energy level may be represented in decibels (dB).
The duplication component 308 may operate on a variety of spectral bands and ranges. For example, the frequency domain data 330 may range from 0 to 12 kHz, and the source and target bands may have a bandwidth of between 500 and 1500 Hz. This bandwidth may be increased (in order to make detection of the audio watermark easier) or decreased (in order to decrease the likelihood that the watermark affects the listener experience) as desired. Experiments show that a bandwidth of 1000 Hz provides a good balance between detectability (by the processing component 242 of
The center frequencies of the bands used as the source and target bands may be adjusted as desired. Experiments show that one reasonable option is the source band includes 3500 Hz (e.g., the center frequency is 3500 Hz) and the target band includes 5500 Hz (e.g., the center frequency is 5500 Hz). Another reasonable option is the source band includes 4500 Hz and the target band includes 6500 Hz.
Combining these bandwidths and center frequencies, one reasonable option is the source band is 3-4 kHz and the target band is 5-6 kHz. Another reasonable option is the source band is 4-5 kHz and the target band is 6-7 kHz.
Although the duplication occurs in the perceptible audio range (e.g., between 3 and 12 kHz), because the duplication is associated with a transient, the audio watermark may be imperceptible to the average listener. This duplication thus serves as a watermark to indicate that the audio effects has been applied. The watermark is referred to as an audio watermark since it occurs within the perceptible audio range, as opposed to being communicated out-of-band using metadata, control signals, etc.
A specific example illustrating the duplication of the source band to the target band is discussed below with reference to
The inverse transfoim component 310 performs an inverse transform on the frequency domain data 332 to generate a portion 338. The portion 338 thus corresponds to the portion 328, but with the audio watermark (e.g., the source band duplicated into the target band). In general, the inverse transform component 310 performs an inverse of the transform performed by the transform component 306. For example, the inverse transform component may perform an inverse FFT with 512 points using a 256-point window, to generate a block of 256 time-domain samples.
The recombiner component 312 receives the audio signal 320 and the portion 338, and generates an audio signal 340. The audio signal 340 corresponds to the audio signal 320, but with the portion 328 replaced by the portion 338. For example, when the portion corresponds to a block of samples, the recombiner 312 replaces the block containing the transient (the portion 328) with the portion 338.
The selection component 314 receives the audio signal 340 and the audio signal 320, selects one according to whether a transient was detected, and outputs the selection as the audio signal 232 (see also FIG, 2). When the transient detector 304 has not detected a transient, the selection component selects the audio signal 320 (that is, without the audio watermark) as the audio signal 232, When the transient detector 304 has detected the transient, the selection component 314 selects the audio signal 340 (that is, with the audio watermark) as the audio signal 232.
In summary, because the audio watermark is inserted in the audio signal 320 in association with a transient, the presence of the transient serves to diminish the perception of a listener that the audio signal 232 has been modified to contain the audio watermark.
The transient detector 402 detects a transient in the audio signal 232 (see also
The transform component 404 transforms a portion 428 of the audio signal 232 related to the transient into frequency domain data 430. For example, when the transient detector 402 detects a transient in a particular block of the audio signal 232, that particular block then corresponds to the portion 428. In general, the transform component 404 performs a similar transform process as performed by the transform component 306 (see
The comparison component 406 receives the frequency domain data 430 and compares the two bands (potentially) duplicated by the decoder component 300 (see
The processing component 408 selectively processes the audio signal 232, based on the transient detector 402 not detecting a transient or the result 432 indicating the bands are uncorrelated, to generate an audio signal 434. This processing generally corresponds to applying an audio effect to the audio signal 232, as discussed in more detail below. The processing component 408 operates in three modes.
In the first mode, when the transient detector 402 does not detect a transient, the processing component 408 processes the audio signal 232 to generate the audio signal 434. In this mode, with no transient present to provide an audio watermark, the processing component 408 assumes that the decoder component 300 did not apply the audio effect, and so the processing component 408 applies the audio effect to generate the audio signal 434.
In the second mode, when the transient detector 402 detects a transient and the results 432 are uncorrelated, the processing component 408 processes the audio signal 232 to generate the audio signal 434. In this mode, the uncorrelated bands indicate that the decoder component 300 did not apply the audio watermark, and hence did not apply the audio effect; so the processing component 408 applies the audio effect to generate the audio signal 434.
In the third mode, when the transient detector 402 detects a transient and the results 432 are correlated, the processing component 408 does not process the audio signal 232. In this mode, the correlated bands indicate that the decoder component 300 applied the audio watei mark, and hence applied the audio effect; so to avoid double processing, the processing component 408 may refrain from operation on the audio signal 232.
In summary, detecting the audio watermark enables the processing component 408 to selectively apply the audio effect, in order to avoid double processing.
The selection component 410 selects the audio signal 434 or the audio signal 232, based on the transient detector 402 not detecting a transient or the result 432 indicating the bands are correlated, to generate the audio signal 234 (see also
In the first mode, when the transient detector 402 does not detect a transient, the selection component 410 selects the audio signal 434 to be the audio signal 234. In this mode, with no transient present, the processing module 408 applies the audio effect to the audio signal 232 to generate the audio signal 434. Because this mode may result in double processing, the transient detector 402 (and the transient detector 304 of
In the second mode, when the transient detector 402 detects a transient and the results 432 are correlated, the selection component 410 selects the audio signal 232 to be the audio signal 234. In this mode, the correlated results indicate that the decoder component 300 (see FIG, 3) applied the audio effect to the audio signal 232, so it may be used. In this manner, double processing of the audio signal is avoided.
In the third mode, when the transient detector 402 detects a transient and the results 432 are uncorrelated, the selection component 410 selects the audio signal 434 to be the audio signal 234. In this mode, the uncorrelated results indicate that the decoder component 300 (see FIG, 3) did not apply the audio effect to the audio signal 232, so the audio signal 434 (with the audio effect applied by the processing component 408) may be used. In this manner, the audio effect may be reliably applied while avoiding double processing, without requiring metadata or other out-of-band control signals between components.
Audio Effects
As discussed above, audio effects may be applied by various components of the audio processing framework 200 (see
Volume leveling describes an effect that maintains consistent playback levels regardless of the source selection and content. For example, when the user switches between different songs in a playlist or switches from listening to music to watching a movie, the volume stays the same. This feature may continuously analyze the audio based on a psychoacoustic model of loudness perception to assess how loud a listener perceives the audio to be. This information is then used to automatically adjust the perceived loudness to a consistent playback level. The volume leveling may be performed using auditory scene analysis, a cognitive model of audio perception developed through analyzing data about audio sources. This ensures that the loudness of the audio is not adjusted in the audio signal at inappropriate moments, such as during a naturally decaying note in a song. The volume leveler may adjust individual channels of the audio and individual frequency bands within a channel to prevent unwanted compression-based “pumping” and “breathing” artifacts. The result is consistently leveled audio, free from the artifacts associated with traditional volume-leveling solutions.
Volume modeling describes an effect that compensates for the reference level used for audio mixing. In the recording studio, audio is mixed at what audio professionals refer to as the reference level, typically around 85 decibels. Although this is generally considered loud, it's the volume level at which most people can perceive the entire spectrum of audio in a mix and hear the intended tonal balance. This is important because of how we actually hear. Typically, the lower the volume, the less well we can hear the high and low audio frequencies—the treble and bass. Traditional volume controls, however, treat all frequencies alike. So when you turn down the volume, you lose the perception of the high and low audio frequencies and the tonal balance suffers. To compensate for this, the volume modeler analyzes the incoming audio, groups similar frequencies into critical bands, and applies appropriate amounts of gain to each.
Dialogue enhancement describes an effect that dynamically applies processing to improve the intelligibility of the spoken portion of audio. This postprocessing feature is designed to improve dialogue perception and understanding for listeners. This involves monitoring the audio track to detect the presence of dialogue. The dialogue enhancer analyzes features from the audio signal and applies pattern recognition to detect the presence of dialogue from moment to moment. When dialogue is detected, the Dialogue Enhancer may perform two types of dynamic audio processing: dynamic spectral rebalancing of dialogue, and dynamic suppression of intrusive signals (although other techniques may also be used).
The dynamic spectral rebalancing of dialogue enhances the middle to high frequencies, which are most important to intelligibility. In simple terms, the speech spectrum is altered where necessary to accentuate the dialogue content in a way that allows the listener to more clearly distinguish the content.
Dynamic suppression of intrusive signals lowers the level of middle to high frequencies of sounds in the audio mix that are not related to dialogue. These are sounds that are determined to be interfering with the intelligibility of the dialogue.
Intelligent equalization describes an effect directed toward providing consistency of spectral balance, also known as timbre. This is accomplished by continuously monitoring the spectral balance of the audio and comparing it to a specified spectral profile (or timbre), known as the reference spectral profile. An equalization filter dynamically transforms the original audio tone to the specified reference spectral profile. This process is different from existing equalization presets found on many audio systems (such as presets for jazz, rock, or voice), where the presets apply the same change across a frequency, regardless of the content. Typically, when a user sets a bass boost level in a traditional equalizer, the setting may not be appropriate as the bass content in the source audio increases; too much bass may cause distortion. The intelligent equalizer does not adjust the bass if sufficient bass is evident in the signal. When the source audio does not have enough bass, the intelligent equalizer boosts the bass appropriately. The result is the desired sound without over-processing or distortion.
At 602, encoded audio data is decoded to generate decoded audio data. For example, the decoder component 302 (see
At 604, a transient is detected in the decoded audio data. For example, the transient detector 304 (see
At 606, a first portion of the decoded audio data related to the transient in the decoded audio data is transformed into first frequency domain data. The first portion may correspond to a block of samples. For example, the transfoi m component 306 (see
At 608, a first band of the first frequency domain data is duplicated into a second band of the first frequency domain data to generate second frequency domain data. This duplication may also include scaling the energy in the duplicated target band to match that of the original target band. For example, the duplication component 308 (see
At 610, the second frequency domain data is transformed to generate a second portion. For example, the inverse transform component 310 (see FIG, 3) may transform the frequency domain data 332 to generate the portion 338.
At 612, watermarked audio data is generated, where the watermarked audio data corresponds to the decoded audio data having the first portion replaced with the second portion. For example, the recombiner component 312 (see
At 614, a transient is detected in first audio data. (In general, the first audio data corresponds to the watermarked audio data of 612; however, at the time of 614 the presence of the wateimark is unknown, so the label “first audio data” is used.) For example, the transient detector 402 (see
At 616, a portion of the first audio data related to the transient is transformed into frequency domain data. For example, the transform component 404 (see
At 618, a first band of the frequency domain data and a second band of the frequency domain data are compared. For example, the comparison component 406 (see
At 620, when the first band is uncorrelated with the second band, processing is performed on the first audio data to generate second audio data. The uncorrelated bands indicate that the audio watermark is not present. In this situation, the first audio data does not have the audio effect, so it needs to be applied. For example, the processing component 408 (see
At 622, when the first band is correlated with the second band, the first audio data is used as the second audio data without performing processing. The correlated bands indicate the presence of the audio watermark. In this situation, the first audio data has the audio effect, so (to avoid double processing) the first audio data is used as the second audio data, without applying an audio effect. For example, the selection component 410 (see
Variations and Options
In
In such an embodiment, the watermarking (when the server applies the effect) enables the mobile device to avoid double processing the audio.
Implementation Details
An embodiment may be implemented in hardware, executable modules stored on a computer readable medium, or a combination of both (e.g., programmable logic arrays). Unless otherwise specified, the steps executed by embodiments need not inherently be related to any particular computer or other apparatus, although they may be in certain embodiments. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices, in known fashion.
Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein. The inventive system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein. (Software per se and intangible or transitory signals are excluded to the extent that they are unpatentable subject matter.)
The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the present disclosure may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the disclosure as defined by the claims.
Various aspects of the present invention may be appreciated from the following enumerated example embodiments (EEEs):
EEE 1. A method of audio processing, the method comprising:
detecting, by a processing component, a transient in first audio data;
transforming a portion of the first audio data related to the transient into frequency domain data;
comparing a first band of the frequency domain data and a second band of the frequency domain data;
when the first band is uncorrelated with the second band, performing processing by the processing component on the first audio data to generate second audio data; and
when the first band is correlated with the second band, using the first audio data as the second audio data without performing processing by the processing component.
EEE 2. The method of EEE 1, wherein prior to detecting the transient in the first audio data, the method further comprises:
decoding, by a decoder component, third audio data to generate fourth audio data;
detecting a transient in the fourth audio data, wherein the transient in the fourth audio data corresponds to the transient in the first audio data;
transforming a first portion of the fourth audio data related to the transient in the fourth audio data into first frequency domain data;
duplicating a first band of the first frequency domain data into a second band of the first frequency domain data to generate second frequency domain data;
transforming the second frequency domain data to generate a second portion; and
generating fifth audio data, wherein the fifth audio data corresponds to the fourth audio data having the first portion replaced with the second portion,
wherein the fifth audio data corresponds to the first audio data.
EEE 3. The method of EEE 2, wherein the third audio data includes an audio signal and metadata, wherein decoding the third audio data comprises decoding the audio signal and the metadata to generate the fourth audio data.
EEE 4. The method of any one of EEEs 2-3, wherein decoding the third audio data further comprises:
applying an audio effect to generate the fourth audio data.
EEE 5. The method of any one of EEEs 2-4, wherein prior to copying the first band into the second band, the first hand has a first energy level and the second band has a second energy level,
wherein copying the first band into the second band includes scaling the first energy level to the second energy level.
EEE 6. The method of any one of EEEs 1-5, wherein performing processing by the processing component on the first audio data to generate the second audio data comprises:
applying an audio effect to the first audio data.
EEE 7. The method of EEE 6, wherein the audio effect is at least one of a volume leveler effect, a volume modeler effect, a dialogue enhancer effect, and an intelligent equalizer effect.
EEE 8. The method of any one of EEEs 1-7, wherein the first portion comprises a plurality of samples of the first audio data that includes the transient.
EEE 9. The method of any one of EEEs 1-8, wherein the first band is a band that includes 3500 Hz and the second band is a band that includes 5500 Hz.
EEE 10. The method of any one of EEEs 1-8, wherein the first band is a band that includes 4500 Hz and the second band is a band that includes 6500 Hz.
EEE 11. The method of any one of EEEs 1-10, wherein the first band and the second band each have a bandwidth of between 500 and 1500 Hz.
EEE 12. The method of any one of EEEs 1-10, wherein the first band and the second band each have a bandwidth of 1000 Hz.
EEE 13. The method of any one of EEEs 1-12, wherein the frequency domain data includes a third band, wherein the third band is between the first band and the second band.
EEE 14. The method of any one of EEEs 1-13, wherein the frequency domain data is within a perceptible audio range.
EEE 15. The method of any one of EEEs 1-14, wherein the frequency domain data is between 3 and 12 kHz.
EEE 16. A non-transitory computer readable medium storing a computer program that, when executed by a processor, controls an apparatus to execute processing including the method of any one of EEEs 1-15.
EEE 17. An apparatus for audio processing, the apparatus comprising:
a processor; and
a memory,
wherein the processor is configured to control the apparatus to detect, by a processing component, a transient in first audio data;
wherein the processor is configured to control the apparatus to transform a portion of the first audio data related to the transient into frequency domain data;
wherein the processor is configured to control the apparatus to compare a first band of the frequency domain data and a second band of the frequency domain data;
wherein, when the first band is uncorrelated with the second band, the processor is configured to control the apparatus to perform processing by the processing component on the first audio data to generate second audio data; and
wherein, when the first band is correlated with the second band, the processor is configured to control the apparatus to use the first audio data as the second audio data without performing processing by the processing component.
EEE 18. The apparatus of EEE 17, wherein prior to detecting the transient in the first audio data:
the processor is configured to control the apparatus to decode, by a decoder component, third audio data to generate fourth audio data;
the processor is configured to control the apparatus to detect a transient in the fourth audio data, wherein the transient in the fourth audio data corresponds to the transient in the first audio data;
the processor is configured to control the apparatus to transform a first portion of the fourth audio data related to the transient in the fourth audio data into first frequency domain data;
the processor is configured to control the apparatus to duplicate a first band of the first frequency domain data into a second band of the first frequency domain data to generate second frequency domain data;
the processor is configured to control the apparatus to transform the second frequency domain data to generate a second portion; and
the processor is configured to control the apparatus to generate fifth audio data, wherein the fifth audio data corresponds to the fourth audio data having the first portion replaced with the second portion,
wherein the fifth audio data corresponds to the first audio data.
EEE 19. The apparatus of any one of EEEs 17-18, wherein prior to copying the first band into the second band, the first band has a first energy level and the second band has a second energy level,
wherein copying the first band into the second band includes scaling the first energy level to the second energy level.
EEE 20. The apparatus of any one of EEEs 17-19, wherein the frequency domain data is within a perceptible audio range.
| Number | Date | Country | Kind |
|---|---|---|---|
| PCT/CN2020/088816 | May 2020 | CN | international |
| 20177393.4 | May 2020 | EP | regional |
This application claims priority of European Patent Application No. 20177393.4 filed May 29, 2020, U.S. Provisional Patent Application No. 63/027,286 filed May 19, 2020, and PCT Application No. PCT/CN2020/088816 filed May 6, 2020, all of which are incorporated herein by reference in their entirety.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2021/031103 | 5/6/2021 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 63027286 | May 2020 | US |
