The techniques relates to audio data and, more specifically, coding of audio data.
A higher order ambisonics (HOA) signal (often represented by a plurality of spherical harmonic coefficients (SHC) or other hierarchical elements) is a three-dimensional representation of a sound field. This HOA or SHC representation may represent this sound field in a manner that is independent of the local speaker geometry used to playback a multi-channel audio signal rendered from this SHC signal. This SHC signal may also facilitate backwards compatibility as this SHC signal may be rendered to well-known and highly adopted multi-channel formats, such as a 5.1 audio channel format or a 7.1 audio channel format. The SHC representation may therefore enable a better representation of a sound field that also accommodates backward compatibility.
In general, techniques are described for performing spatial masking with respect to the spherical harmonic coefficients (which may also be referred to as higher-order ambisonic (HOA) coefficients). Spatial masking may leverage the inability of the human auditory system in detecting a quieter sound when a relatively louder sound occurs in a spatially proximate location to the quieter sound. The techniques described in this disclosure may enable an audio coding device to evaluating a soundfield expressed by the spherical harmonic coefficients to identify these quieter (or less energetic) sounds that may be masked by relatively louder (or more energetic) sounds. The audio coding device may then assign more bits for coding the quieter sounds while assigning more bits (or maintaining a number of bits) for coding the louder sounds. In this respect, the techniques described in this disclosure may facilitate coding of the spherical harmonic coefficients.
In one aspect, a method comprises decoding a bitstream to generate first multi-channel audio data having a plurality of channels corresponding to speakers arranged in a defined speaker geometry, performing an inverse rendering process with respect to the generated multi-channel audio data to generate a plurality of spherical harmonic coefficients, and rendering second multi-channel audio data having a plurality of channels corresponding to speakers arranged in a local speaker geometry based on the plurality of spherical harmonic coefficients.
In another aspect, an audio decoding device comprises one or more processors configured to decode a bitstream to generate first multi-channel audio data having a plurality of channels corresponding to speakers arranged in a first speaker geometry, perform an inverse rendering process with respect to the generated multi-channel audio data to generate a plurality of spherical harmonic coefficients, and render second multi-channel audio data having a plurality of channels corresponding to speakers arranged in a local speaker geometry based on the plurality of spherical harmonic coefficients.
In another aspect, an audio decoding device comprises means for decoding a bitstream to generate first multi-channel audio data having a plurality of channels corresponding to speakers arranged in a first speaker geometry, means for performing an inverse rendering process with respect to the generated multi-channel audio data to generate a plurality of spherical harmonic coefficients, and means for rendering second multi-channel audio data having a plurality of channels corresponding to speakers arranged in a local speaker geometry based on the plurality of spherical harmonic coefficients.
In another aspect, a non-transitory computer-readable storage medium has stored thereon instructions that, when executed, cause one or more processors of an audio decoding device to decode a bitstream to generate first multi-channel audio data having a plurality of channels corresponding to speakers arranged in a first speaker geometry, perform an inverse rendering process with respect to the generated multi-channel audio data to generate a plurality of spherical harmonic coefficients, and render second multi-channel audio data having a plurality of channels corresponding to speakers arranged in a local speaker geometry based on the plurality of spherical harmonic coefficients.
In another aspect, a method of compressing audio data comprises performing spatial analysis based on a plurality of spherical harmonic coefficients that describe a three-dimensional sound field to identify a spatial masking threshold, and compressing the audio data based on the identified spatial masking thresholds to generate a bitstream.
In another aspect, a device comprises one or more processors configured to perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a three-dimensional sound field to identify a spatial masking threshold and compress the audio data based on the identified spatial masking thresholds to generate a bitstream.
In another aspect, a device comprises means for performing spatial analysis based on a plurality of spherical harmonic coefficients that describe a three-dimensional sound field to identify a spatial masking threshold, and means for compressing the audio data based on the identified spatial masking thresholds to generate a bitstream.
In another aspect, a non-transitory computer-readable storage medium has stored thereon instructions that, when executed, cause one or more processors to perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a three-dimensional sound field to identify a spatial masking threshold, and compress the audio data based on the identified spatial masking thresholds to generate a bitstream.
In another aspect, a method of compressing audio comprises rendering a plurality of spherical harmonic coefficients that describe a sound field of the audio in three dimensions to generate multi-channel audio data, performing spatial analysis with respect to the multi-channel audio data to identify a spatial masking threshold, and compressing the multi-channel audio data based on the identified spatial masking threshold to generate a bitstream.
In another aspect, a device comprises one or more processors configured to render a plurality of spherical harmonic coefficients that describe a sound field of the audio in three dimensions to generate multi-channel audio data, perform spatial analysis with respect to the multi-channel audio data to identify a spatial masking threshold, and compress the multi-channel audio data based on the identified spatial masking threshold to generate a bitstream.
In another aspect, a device comprises means for rendering a plurality of spherical harmonic coefficients that describe a sound field of the audio in three dimensions to generate multi-channel audio data, means for performing spatial analysis with respect to the multi-channel audio data to identify a spatial masking threshold, and means for compressing the multi-channel audio data based on the identified spatial masking threshold to generate a bitstream.
In another aspect, a non-transitory computer-readable storage medium has stored thereon instructions that, when executed, cause one or more processors to render a plurality of spherical harmonic coefficients that describe a sound field of the audio in three dimensions to generate multi-channel audio data, perform spatial analysis with respect to the multi-channel audio data to identify a spatial masking threshold, and compress the multi-channel audio data based on the identified spatial masking threshold to generate a bitstream.
In another aspect, a method of compressing audio data comprises determining a target bitrate for a bitstream representative of the compressed audio data, performing spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, and performing, based on the target bitrate, either i) parametric inter-channel audio encoding and spatial masking using the spatial masking threshold or ii) the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding to generate a bitstream representative of the compressed audio data.
In another aspect, a device comprises one or more processors configured to determine a target bitrate for a bitstream representative of the compressed audio data, perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, and perform, based on the target bitrate, either i) parametric inter-channel audio encoding and spatial masking using the spatial masking threshold or ii) the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding to generate a bitstream representative of the compressed audio data.
In another aspect, a device comprises means for determining a target bitrate for a bitstream representative of the compressed audio data, means for performing spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, and means for performing, based on the target bitrate, either i) parametric inter-channel audio encoding and spatial masking using the spatial masking threshold or ii) the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding to generate a bitstream representative of the compressed audio data.
In another aspect, a non-transitory computer-readable storage medium has stored thereon instructions that, when executed, cause one or more processors to determine a target bitrate for a bitstream representative of the compressed audio data, perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, and perform, based on the target bitrate, either i) parametric inter-channel audio encoding and spatial masking using the spatial masking threshold or ii) the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding to generate a bitstream representative of the compressed audio data.
In another aspect, a method of compressing multi-channel audio data, the method comprises performing spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the multi-channel audio data in three dimensions to identify a spatial masking threshold, rendering the spherical harmonic coefficients to generate the multi-channel audio data, performing spatial masking with respect to one or more base channels of the multi-channel audio data using the spatial masking threshold, and performing parametric inter-channel audio encoding with respect to the multi-channel audio data, including the spatially masked one or more base channels of the multi-channel audio data, to generate a bitstream.
In another aspect, a device comprises one or more processors to perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the multi-channel audio data in three dimensions to identify a spatial masking threshold, render the spherical harmonic coefficients to generate the multi-channel audio data, perform spatial masking with respect to one or more base channels of the multi-channel audio data using the spatial masking threshold, and perform parametric inter-channel audio encoding with respect to the multi-channel audio data, including the spatially masked one or more base channels of the multi-channel audio data, to generate a bitstream.
In another aspect, a device comprises means for performing spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the multi-channel audio data in three dimensions to identify a spatial masking threshold, means for rendering the spherical harmonic coefficients to generate the multi-channel audio data, means for performing spatial masking with respect to one or more base channels of the multi-channel audio data using the spatial masking threshold, and means for performing parametric inter-channel audio encoding with respect to the multi-channel audio data, including the spatially masked one or more base channels of the multi-channel audio data, to generate a bitstream.
In another aspect, a non-transitory computer-readable storage medium has stored thereon instructions that, when executed, cause one or more processors to perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the multi-channel audio data in three dimensions to identify a spatial masking threshold, render the spherical harmonic coefficients to generate the multi-channel audio data, perform spatial masking with respect to one or more base channels of the multi-channel audio data using the spatial masking threshold, and perform parametric inter-channel audio encoding with respect to the multi-channel audio data, including the spatially masked one or more base channels of the multi-channel audio data, to generate a bitstream.
In another aspect, a method of compressing audio data, the method comprises performing spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, performing spatial masking with respect to the plurality of spherical harmonic coefficients using the spatial masking threshold, and generating a bitstream that includes the plurality of spatially masked spherical harmonic coefficients.
In another aspect, a device comprises one or more processors to perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, perform spatial masking with respect to the plurality of spherical harmonic coefficients using the spatial masking threshold, and generate a bitstream that includes the plurality of spatially masked spherical harmonic coefficients.
In another aspect, a device comprises means for performing spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, means for performing spatial masking with respect to the plurality of spherical harmonic coefficients using the spatial masking threshold, and means for generating a bitstream that includes the plurality of spatially masked spherical harmonic coefficients.
In another aspect, a non-transitory computer-readable storage medium has stored thereon instructions that, when executed, cause one or more processors to perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, perform spatial masking with respect to the plurality of spherical harmonic coefficients using the spatial masking threshold, and generate a bitstream that includes the plurality of spatially masked spherical harmonic coefficients.
The details of one or more aspects of the techniques are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of these techniques will be apparent from the description and drawings, and from the claims.
The evolution of surround sound has made available many output formats for entertainment nowadays. Examples of such surround sound formats include the popular 5.1 format (which includes the following six channels: front left (FL), front right (FR), center or front center, back left or surround left, back right or surround right, and low frequency effects (LFE)), the growing 7.1 format, and the upcoming 22.2 format (e.g., for use with the Ultra High Definition Television standard). Further examples include formats for a spherical harmonic array.
The input to the future MPEG encoder is optionally one of three possible formats: (i) traditional channel-based audio, which is meant to be played through loudspeakers at pre-specified positions; (ii) object-based audio, which involves discrete pulse-code-modulation (PCM) data for single audio objects with associated metadata containing their location coordinates (amongst other information); and (iii) scene-based audio, which involves representing the sound field using coefficients of spherical harmonic basis functions (also called “spherical harmonic coefficients” or SHC).
There are various ‘surround-sound’ formats in the market. They range, for example, from the 5.1 home theatre system (which has been the most successful in terms of making inroads into living rooms beyond stereo) to the 22.2 system developed by NHK (Nippon Hoso Kyokai or Japan Broadcasting Corporation). Content creators (e.g., Hollywood studios) would like to produce the soundtrack for a movie once, and not spend the efforts to remix it for each speaker configuration. Recently, standard committees have been considering ways in which to provide an encoding into a standardized bitstream and a subsequent decoding that is adaptable and agnostic to the speaker geometry and acoustic conditions at the location of the renderer.
To provide such flexibility for content creators, a hierarchical set of elements may be used to represent a sound field. The hierarchical set of elements may refer to a set of elements in which the elements are ordered such that a basic set of lower-ordered elements provides a full representation of the modeled sound field. As the set is extended to include higher-order elements, the representation becomes more detailed.
One example of a hierarchical set of elements is a set of SHC. The following expression demonstrates a description or representation of a sound field using SHC:
This expression shows that the pressure pi at any point {rr, θr, φr} of the sound field can be represented uniquely by the SHC Anm(k). Here,
c is the speed of sound (˜343 m/s), {rr, θr, φr} is a point of reference (or observation point), jn(•) is the spherical Bessel function of order n, and Ynm (θr, φr) are the spherical harmonic basis functions of order n and suborder m. It can be recognized that the term in square brackets is a frequency-domain representation of the signal (i.e., S(ω, rr, θr, φr)) which can be approximated by various time-frequency transformations, such as the discrete Fourier transform (DFT), the discrete cosine transform (DCT), or a wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.
In any event, the SHC Anm(k) can either be physically acquired (e.g., recorded) by various microphone array configurations or, alternatively, they can be derived from channel-based or object-based descriptions of the sound field. The former represents scene-based audio input to an encoder. For example, a fourth-order representation involving 1+24 (25, and hence fourth order) coefficients may be used. To illustrate how these SHCs may be derived from an object-based description, consider the following equation. The coefficients Anm(k) for the sound field corresponding to an individual audio object may be expressed as
A
n
m(k)=g(ω)(−4πik)hn(2)(krs)Ynm*(θs,φs),
where i is √{square root over (−1)}, hn(2)(•) is the spherical Hankel function (of the second kind) of order n, and {rs, θs, φs} is the location of the object. Knowing the source energy g(ω) as a function of frequency (e.g., using time-frequency analysis techniques, such as performing a fast Fourier transform on the PCM stream) allows us to convert each PCM object and its location into the SHC Anm(k). Further, it can be shown (since the above is a linear and orthogonal decomposition) that the Anm(k) coefficients for each object are additive. In this manner, a multitude of PCM objects can be represented by the Anm(k) coefficients (e.g., as a sum of the coefficient vectors for the individual objects). Essentially, these coefficients contain information about the sound field (the pressure as a function of 3D coordinates), and the above represents the transformation from individual objects to a representation of the overall sound field, in the vicinity of the observation point {rr, θr, φr}. The remaining figures are described below in the context of object-based and SHC-based audio coding.
While shown as a single device, i.e., the device 10 in the example of
As shown in the example of
The SHC 20A may refer to coefficients associated with one or more spherical harmonics. These spherical harmonics may be analogous to the trigonometric basis functions of a Fourier series. That is, spherical harmonics may represent the fundamental modes of vibration of a sphere around a microphone similar to how the trigonometric functions of the Fourier series may represent the fundamental modes of vibration of a string. These coefficients may be derived by solving a wave equation in spherical coordinates that involves the use of these spherical harmonics. In this sense, the SHC 20A may represent a 3D sound field surrounding a microphone as a series of spherical harmonics with the coefficients denoting the volume multiplier of the corresponding spherical harmonic.
Lower-order ambisonics (which may also be referred to as first-order ambisonics) may encode sound information into four channels denoted W, X, Y and Z. This encoding format is often referred to as a “B-format.” The W channel refers to a non-directional mono component of the captured sound signal corresponding to an output of an omnidirectional microphone. The X, Y and Z channels are the directional components in three dimensions. The X, Y and Z channels typically correspond to the outputs of three figure-of-eight microphones, one of which faces forward, another of which faces to the left and the third of which faces upward, respectively. These B-format signals are commonly based on a spherical harmonic decomposition of the soundfield and correspond to the pressure (W) and the three component pressure gradients (X, Y and Z) at a point in space. Together, these four B-format signals (i.e., W, X, Y and Z) approximate the sound field around the microphone. Formally, these B-format signals may express the first-order truncation of the multipole expansion.
Higher-order ambisonics refers to a form of representing a sound field that uses more channels, representing finer modal components, than the original first-order B-format. As a result, higher-order ambisonics may capture significantly more spatial information. The “higher order” in the term “higher order ambisonics” refers to further terms of the multimodal expansion of the function on the sphere in terms of spherical harmonics. Increasing the spatial information by way of higher-order ambisonics may result in a better expression of the captured sound as pressure over a sphere. Using higher order ambisonics to produce the SHC 20A may enable better reproduction of the captured sound by speakers present at the audio decoder.
The audio rendering unit 14 represents a unit configured to render the SHC 20B to one or more channels 22A-22N (“channels 22,” which may also be referred to as “speaker feeds 22A-22N”). Alternatively, when not transforming the SHC 20A to the SHC 20B, the audio rendering unit 14 may represent a unit configured to render the one or more channels 22A-22N from the SHC 20A. In some instances, the audio rendering unit 14 may render the SHC 20B to 32 channels (shown as channels 22 in the example of
The audio encoding unit 16 may represent a unit configured to perform some form of audio encoding to compress the channels 22 into a bitstream 24. In some examples, the audio encoding unit 16 may include modified versions of audio encoders that conform to known spatial audio encoding standards, such as a Moving Picture Experts Group (MPEG) Surround defined in International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) 23003-1 or MPEG-D Part 1 (which may also be referred to as “Spatial Audio Coding” or “SAC”) or MPEG Advanced Audio Coding (AAC) defined in both Part 7 of the MPEG-2 standard (which is also known as ISO/IEC 13818-7:1997) and Subpart 4 in Part 3 of the MPEG-4 standard (which is also known as ISO/IEC 14496-3:1999).
The spatial analysis unit 18 may represent a unit configured to perform spatial analysis of the SHC 20A. The spatial analysis unit 18 may perform this spatial analysis to identify areas of relative high and low pressure density (often expressed as a function of one or more of azimuth, angle, elevation angle and radius (or equivalent Cartesian coordinates)) in the sound field, analyzing the SHC 20A to identify spatial properties 26. These spatial properties 26 may specify one or more of an azimuth, angle, elevation angle and radius of various portions of the SHC 20A that have certain characteristics. The spatial analysis unit 18 may identify the spatial properties 26 to facilitate audio encoding by the audio encoding unit 16. That is, the spatial analysis unit 18 may provide the spatial properties 26 to the audio encoding unit 16, which may be modified to take advantage of psychoacoustic spatial or positional masking and other spatial characteristics of the sound field represented by the SHC 20A.
Spatial masking may leverage tendencies of the human auditory system to mask neighboring spatial portions (or 3D segments) of the sound field when a high energy acoustic energy are present in the sound field. That is, high energy portions of the sound field may overwhelm the human auditory system such that portions of energy (often, adjacent areas of low energy) are unable to be detected (or discerned) by the human auditory system. As a result, the audio encoding unit 18 may allow lower number of bits (or equivalently higher quantization noise) to represent the sound field in these so-called “masked” segments of space, where the human auditory systems may be unable to detect (or discern) sounds when high energy portions are detected in neighboring areas of the sound field defined by the SHC 20A. This is similar to representing the sound field in those “masked” spatial regions with lower precision (meaning possibly higher noise).
In operation, the audio encoding device 10 may implement various aspects of the techniques described in this disclosure by first invoking the spatial analysis unit 18 to performing spatial analysis with respect to the SHC 20A that describe a three-dimensional sound field to identify the spatial properties 26 of the sound field. The audio encoding device 10 may then invoke the audio rendering unit 14 to render the channels 22 (which may also be referred to as the “multi-channel audio data 22”) from either the SHC 20A (when, as noted above, the time-frequency analysis is not performed) or the SHC 20B (when the time-frequency analysis is performed). After or concurrent to the rendering this multi-channel audio data 22, the audio encoding device 10 may invoke the audio encoding unit 16 to encode the multi-channel audio data 22 based on the identified spatial properties 26 to generate the bitstream 24. As noted above, the audio encoding unit 16 may perform a standards-compliant form of audio encoding that has been modified in various ways to leverage the spatial properties 26 (e.g., to perform the above described spatial masking).
In this way, the techniques may effectively encode the SHC 20A such that, as described in more detail below, an audio decoding device, such as the audio decoding device 30 shown in the example of
In some instances, this spatial masking may be performed in conjunction with other types of masking, such as simultaneous masking. Simultaneous masking, much like spatial masking, involves the phenomena of the human auditory system, where sounds produced concurrent (and often at least partially simultaneously) to other sounds mask the other sounds. Typically, the masking sound is produced at a higher volume than the other sounds. The masking sound may also be similar to close in frequency to the masked sound. Thus, while described in this disclosure as being performed alone, the spatial masking techniques may be performed in conjunction with or concurrent to other forms of masking, such as the above noted simultaneous masking.
In this manner, the techniques may enable the audio encoding device 11 to render a plurality of spherical harmonic coefficients 20B that describe a sound field of the audio in three dimensions to generate multi-channel audio data (which is shown as channels 22 in the example of
In some instances, when compressing the audio data, the audio encoding device 11 may allocate bits in the bitstream for either a time-based representation of the multi-channel audio data or a frequency-based representation of the multi-channel audio data based on the spatial masking threshold.
In some instances, when compressing the audio data, the audio encoding device 11 may allocate bits in the bitstream for either a time-based representation of the multi-channel audio data or a frequency-based representation of the multi-channel audio data based on the spatial masking threshold and a temporal masking threshold.
In some instances, when compressing the audio data, the audio encoding device 11 may perform a parametric inter-channel audio encoding (such as an MPEG Surround audio encoding) with respect to the multi-channel audio data to generate the bitstream.
In some instances, when compressing the audio data, the audio encoding device 11 may allocating bits for representing the multi-channel audio data based on the spatial masking threshold to generate the bitstream.
In some instances, the audio encoding device 11 may transform the multi-channel audio data from the spatial domain to the time domain. When compressing the audio data, the audio encoding device 11 may then allocate bits for representing various frequency bins of the transformed multi-channel audio data based on the spatial masking threshold to generate the bitstream.
Generally, the audio decoding device 30 performs an audio decoding process that is reciprocal to the audio encoding process performed by the audio encoding device 10 with the exception of performing spatial analysis, which is typically used by the audio encoding device 10 to facilitate the removal of extraneous irrelevant data (e.g., data that would be masked or incapable of being perceived by the human auditory system). In other words, the audio encoding device 10 may lower the precision of the audio data representation as the typical human auditory system may be unable to discern the lack of precision in these areas (e.g., the “masked” areas, both in time and, as noted above, in space). Given that this audio data is irrelevant, the audio decoding device 30 need not perform spatial analysis to reinsert such extraneous audio data.
While shown as a single device, i.e., the device 30 in the example of
As shown in the example of
The inverse audio rendering unit 34 may represent a unit configured to perform an rendering process inverse to the rendering process performed by the audio rendering unit 14 of the audio encoding device 10 to recover the SHC 20B. The inverse audio rendering unit 34 may apply the inverse transform matrix, K−1, described above. Alternatively, when the SHC 20A was not transformed to generate the SHC 20B, the inverse audio rendering unit 34 may represent a unit configured to render the SHC 20A from the channels 22 through application of the inverse matrix K−1. In some instances, the inverse audio rendering unit 34 may render the SHC 20B from 32 channels corresponding to 32 speakers arranged in a dense T-design for the reasons described above.
The inverse time-frequency analysis unit 36 may represent a unit configured to perform an inverse time-frequency analysis of the spherical harmonic coefficients (SHC) 20B in order to transform the SHC 20B from the frequency domain to the time domain. The inverse time-frequency analysis unit 36 may output the SHC 20A, which may denote the SHC 20B as expressed in the time domain. Although described with respect to the inverse time-frequency analysis unit 36, the techniques may be performed with respect to the SHC 20A in the time domain rather than performed with respect to the SHC 20B in the frequency domain.
The audio rendering unit 38 represents a unit configured to render the channels 40A-40N (the “channels 40,” which may also be generally referred to as the “multi-channel audio data 40” or as the “loudspeaker feeds 40”). The audio rendering unit 38 may apply a transform (often expressed in the form of a matrix) to the SHC 20A. Because the SHC 20A describe the sound field in three dimensions, the SHC 20A represent an audio format that facilitates rendering of the multichannel audio data 40 in a manner that is capable of accommodating most decoder-local speaker geometries (which may refer to the geometry of the speakers that will playback multi-channel audio data 40). Moreover, by rendering the SHC 20A to channels for 32 speakers arranged in a dense T-design at the audio encoding device 10, the techniques provide sufficient audio information (in the form of the SHC 20A) at the decoder to enable the audio rendering unit 38 to reproduce the captured audio data with sufficient fidelity and accuracy using the decoder-local speaker geometry. More information regarding the rendering of the multi-channel audio data 40 is described below with respect to
In operation, the audio decoding device 30 may invoke the audio decoding unit 32 to decode the bitstream 24 to generate the first multi-channel audio data 22 having a plurality of channels corresponding to speakers arranged in a first speaker geometry. This first speaker geometry may comprise the above noted dense T-design, where the number of speakers may be, as one example, 32. While described in this disclosure as including 32 speakers, the dense T-design speaker geometry may include 64 or 128 speakers to provide a few alternative examples. The audio decoding device 30 may then invoke the inverse audio rendering unit 34 to perform an inverse rendering process with respect to generated the first multi-channel audio data 22 to generate the SHC 20B (when the time-frequency transforms is performed) or the SHC 20A (when the time-frequency analysis is not performed). The audio decoding device 30 may also invoke the inverse time-frequency analysis unit 36 to transform, when the time frequency analysis was performed by the audio encoding device 10, the SHC 20B from the frequency domain back to the time domain, generating the SHC 20A. In any event, the audio decoding device 30 may then invoke the audio rendering unit 38, based on the encoded-decoded SHC 20A, to render the second multi-channel audio data 40 having a plurality of channels corresponding to speakers arranged in a local speaker geometry.
Each of the surround encoders 50 may perform a modified version of the above referenced MPEG Surround to encode the multi-channel audio data. This modified version may represent a version of MPEG Surround that encodes the multi-channel audio data 22 based on the spatial properties 26 determined by the spatial analysis module 18 (shown in the example of
In some instances, the spatial parameter estimation units 54 may each be modified to determine these spatial parameters based at least in part on the spatial properties 26 determined by the spatial analysis unit 18. To illustrate, each of the spatial parameter estimation units 54 may calculate the delta or difference between the channels and thereby determining the spatial parameters (which may include inter-channel level, inter-channel time and inter-channel correlation) based on the spatial properties 26. For example, based on the spatial properties 26, the spatial parameter estimation units 54 may determine an accuracy with which to specify the spatial parameters (or, in other words, how coarsely to quantize the parameters when not a lot of energy is present).
In any event, each of the surround encoders 50 output the one of the corresponding subset of the channels 22 to a corresponding one of the audio encoders 52, which encodes this one of the corresponding subset of the channels 22 as a mono-audio signal. That is, each of the audio encoders 52 represents a mono aural audio encoder 52. The audio encoders 52 may include a corresponding one of the entropy encoders 56A-56N (“entropy encoders 56”). Each of the entropy encoders 56 may perform a form of lossless statistical coding (which is commonly referred to by the misnomer “entropy coding”), such as Huffman coding, to encode the one of the corresponding subset of the channels 22. In some instances, the entropy encoders 56 may each perform this entropy coding based on the spatial properties 26. Each of the entropy encoders 56 may output an encoded version of multi-channel audio data, which may be multiplexed with other encoded versions of multi-channel audio data and the spatial parameters 24B to form the bitstream 24.
In the example of
In the example of
The surround decoder 72A may represent a unit configured to resynthesize the remaining ones of the corresponding subset of the channels 22 based on spatial parameters denoted as the bitstream portions 24B. The surround decoders 72 may each include a corresponding one of sound synthesis units 76A-76N (“sound synthesis units 76”) that receives the decoded one of the corresponding subsets of the channels 22 and these spatial parameters. Based on the spatial parameters, each of the sound synthesis units 76 may resynthesize the remaining ones of the corresponding subsets of the channels 22. In this manner, the audio decoding unit 32 may decode the bitstream 24 to generate the multi-channel audio data 22.
To illustrate, the above equation for determining the loudspeaker feeds in terms of the SHC may be modified as follows:
In the above equation, the VBAP matrix is of size M rows by N columns, where M denotes the number of speakers (and would be equal to five in the equation above) and N denotes the number of virtual speakers. The VBAP matrix may be computed as a function of the vectors from the defined location of the listener to each of the positions of the speakers and the vectors from the defined location of the listener to each of the positions of the virtual speakers. The D matrix in the above equation may be of size N rows by (order+1)2 columns, where the order may refer to the order of the SH functions. The D matrix may represent the following matrix:
The g matrix (or vector, given that there is only a single column) may represent the gain for speaker feeds for the speakers arranged in the decoder-local geometry. In the equation, the g matrix is of size M. The A matrix (or vector, given that there is only a single column) may denote the SHC 20A, and is of size (Order+1)(Order+1), which may also be denoted as (Order+1)2.
In effect, the VBAP matrix is an M×N matrix providing what may be referred to as a “gain adjustment” that factors in the location of the speakers and the position of the virtual speakers. Introducing panning in this manner may result in better reproduction of the multi-channel audio that results in a better quality image when reproduced by the local speaker geometry. Moreover, by incorporating VBAP into this equation, the techniques may overcome poor speaker geometries that do not align with those specified in various standards.
In practice, the equation may be inverted and employed to transform the SHC 20A back to the multi-channel feeds 40 for a particular geometry or configuration of loudspeakers, which again may be referred to as the decoder-local geometry in this disclosure. That is, the equation may be inverted to solve for the g matrix. The inverted equation may be as follows:
The g matrix may represent speaker gain for, in this example, each of the five loudspeakers in a 5.1 speaker configuration. The virtual speakers locations used in this configuration may correspond to the locations defined in a 5.1 multichannel format specification or standard. The location of the loudspeakers that may support each of these virtual speakers may be determined using any number of known audio localization techniques, many of which involve playing a tone having a particular frequency to determine a location of each loudspeaker with respect to a headend unit (such as an audio/video receiver (A/V receiver), television, gaming system, digital video disc system, or other types of headend systems). Alternatively, a user of the headend unit may manually specify the location of each of the loudspeakers. In any event, given these known locations and possible angles, the headend unit may solve for the gains, assuming an ideal configuration of virtual loudspeakers by way of VBAP.
In this respect, the techniques may enable a device or apparatus to perform a vector base amplitude panning or other form of panning on the plurality of virtual channels to produce a plurality of channels that drive speakers in a decoder-local geometry to emit sounds that appear to originate form virtual speakers configured in a different local geometry. The techniques may therefore enable the audio decoding unit 32 to perform a transform on the plurality of spherical harmonic coefficients, such as the SHC 20A, to produce a plurality of channels. Each of the plurality of channels may be associated with a corresponding different region of space. Moreover, each of the plurality of channels may comprise a plurality of virtual channels, where the plurality of virtual channels may be associated with the corresponding different region of space. The techniques may, in some instances, enable a device to perform vector base amplitude panning on the virtual channels to produce the plurality of channel of the multi-channel audio data 40.
The audio decoding device 30 may then invoke the inverse audio rendering unit 34 to perform an inverse rendering process with respect to generated the first multi-channel audio data 22 to generate the SHC 20B (when the time-frequency transforms is performed) or the SHC 20A (when the time-frequency analysis is not performed) (102). The audio decoding device 30 may also invoke the inverse time-frequency analysis unit 36 to transform, when the time frequency analysis was performed by the audio encoding device 10, the SHC 20B from the frequency domain back to the time domain, generating the SHC 20A. In any event, the audio decoding device 10 may then invoke the audio rendering unit 38 to render the second multi-channel audio data 40 having a plurality of channels corresponding to speakers arranged in a local speaker geometry based on the SHC 20A (104).
In this way, the techniques may use existing audio coders (and modify various aspects of it to accommodate spatial information from the SHC). To do that, the techniques may take the SH coefficients and render them (using renderer R1) to an arbitrary—but dense set of loudspeakers. The geometry of these loudspeakers may be such that an inverse renderer (R1_inv) can regenerate the SH signals. In some examples, the renderer may be just a single matrix (frequency independent) and one which has an inverse counter-part matrix such that the R1×R1_inv=Identity matrix. These renderers exist for geometries described by T-Design or Platonic Solids. The loudspeaker feeds generated by the renderer (R1) may be coded using ‘off-the-shelf’ audio coders that will be modified by spatial information gleaned/analyzed from the SHC. In some instances, the techniques may take usual audio-coding approaches whereby, one or more of inter-channel level/time/correlation between the speaker feeds are maintained. Compression is used to pack more channels into the bits allocated for a single channel, etc.
At the decoder, the techniques may enable the decoder to recover the speaker feeds and put them through the INVERSE-RENDERER (R1_inv) to retrieve the original SHC. These SHC may be fed into another renderer (R2) meant to cater for the local speaker geometry. Typically, the techniques provide that the number of speaker feeds generated at the output of R1 is dense relative to the number of speakers ever likely to be at the output of Renderer R2. In other words, a much higher number of speakers than the actual number of speakers ever likely to be at the output of the R2 renderer is assumed when rendering the first multi-channel audio data.
It is to be recognized that depending on the example, certain acts or events of any of the techniques described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.
where Ep
The spatial analysis unit 18 shown in the example of
In some examples, the spatial masking threshold may be used with a temporal (or, in other words, simultaneous) masking threshold. Often, the spatial masking threshold may be added to the temporal masking threshold to generate an overall masking threshold. In some instances, weights are applied to the spatial and temporal masking thresholds when generating the overall masking threshold. These threshold may be expressed as a function of ratios (such as a signal-to-noise ratio (SNR)). The overall threshold may be used by a bit allocator when allocating bits to each frequency bin. The audio encoding unit 16 of
The audio encoding device 10′, however, also includes a mode selector unit 150, which represents a unit that determines whether to render the SHC 20B prior to encoding the channels 22 or transmit the SHC 20B directly to the audio encoding unit 16 without first rendering the SHC20B to the channels 22. Mode selector unit 150 may receive a target bitrate 152 as an input from a user, another device or via any other way by which the target bitrate 152 may be input. The target bitrate 152 may represent data defining a bitrate or level of compression for the bitstream 24.
In one example, for higher bitrates specified by the bitrate 152, the mode selector unit 150 may determine that the SHC 20B are to be audio encoded directly by audio encoding unit 16 using the spatial masking aspects of the techniques described in this disclosure. One example of higher bitrates may be bitrates equal to or above 256 Kilobits per second (Kbps). Thus, for bitrates such as 256 Kbps, 512 Kbps and/or 1.2 megabits per second (Mbps) (where 256 Kbps may, in this example represent a threshold bitrate used to determine the higher bitrates from the lower bitrates), the audio encoding unit 16 may operate directly on the SHC 20B and the SHC 20B are not rendered to the channels 22 by audio rendering unit 14.
For lower bitrates specified by the bitrate 152, the mode selector unit 150 may determine that the SHC 20B are to be first rendered by the audio rendering unit 14 to generate the channels 22 and then subsequently encoded by the audio encoding unit 16. In this instance, the audio encoding unit 16 may perform the spatial masking techniques with respect to the first channel, while the remaining channels undergo parametric encoding, such as that performed in accordance with MPEG surround and other parametric inter-channel encoding schemes.
The audio encoding unit 16 may specify (either in encoded or non-encoded form) the mode selected by mode selector unit 150 in the bitstream so that the decoding device may determine whether parametric inter-channel encoding was performed when generating the bitstream 24. While not shown in detail, the audio decoding device 30 may be modified in a similar manner to that of the audio encoding device 10′ (where such audio decoding device 30 may be referred to as the audio decoding device 30′). This audio decoding device 30′ may likewise include a mode selector unit similar to mode selector unit 150 that determines whether to output either the channels 22 to the inverse audio rendering unit 34 or the SHC 20B to the inverse time-frequency analysis unit 36. In some instances, this mode may be inferred from the target bitrate 152 to which the bitstream 24 corresponds (where this target bitrate 152 may be specified in the bitstream 24 and effectively represents the mode given that the audio decoding device 30′ may infer this mode from the target bitrate 152).
In this respect, the techniques described in this disclosure may enable the audio encoding device 10′ to perform a method of compressing audio data. In performing this method, the audio encoding device 10′ may determine a target bitrate for a bitstream representative of the compressed audio data and perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold. Based on the target bitrate, the audio encoding device 10′ may perform either i) parametric inter-channel audio encoding and spatial masking using the spatial masking threshold or ii) the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding to generate a bitstream representative of the compressed audio data.
In some instances, when performing either i) the parametric inter-channel audio encoding and the spatial masking using the spatial masking threshold or ii) the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding, the audio encoding device 10′ may determine that the target bitrate is below a threshold bitrate, and in response to determining that the target bitrate is below the threshold bitrate, perform the parametric inter-channel audio encoding and the spatial masking using the spatial masking threshold to generate the bitstream. The threshold bitrate, may for example, be equal to 256 Kilobits per second (Kbps).
In some instances, when performing either i) the parametric inter-channel audio encoding and the spatial masking using the spatial masking threshold or ii) the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding, the audio encoding device 10′ may determine that the target bitrate is equal to or exceeds a threshold bitrate, and in response to determining that the target bitrate is equal to or exceeds the threshold bitrate, performing the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding to generate the bitstream.
In some instances, the audio encoding device 10′ may further render the plurality of spherical harmonic coefficients to multi-channel audio data. When performing either i) the parametric inter-channel audio encoding and the spatial masking using the spatial masking threshold or ii) the spatial masking using the spatial masking threshold without performing the parametric inter-channel audio encoding, the audio encoding device 10′ may determine that the target bitrate is below a threshold bitrate, and in response to determining that the target bitrate is below the threshold bitrate, performing the spatial masking using the spatial masking threshold with respect to one or more base channels of the multi-channel audio data and performing the parametric inter-channel audio encoding with respect to the multi-channel audio data to generate the bitstream. Again, the threshold bitrate may be equal to 256 Kilobits per second (Kbps).
In some instances, the audio encoding device 10′ may also allocate bits in the bitstream for either a time-based representation of the audio data or a frequency-based representation of the audio data based on the spatial masking threshold.
In some instances, the parametric inter-channel audio encoding comprises a moving picture experts group (MPEG) Surround.
Moreover, the techniques described in this disclosure may enable the audio encoding device 10′ to perform a method of compressing multi-channel audio data. In performing this method, the audio encoding device 10′ may perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the multi-channel audio data in three dimensions to identify a spatial masking threshold, and render the spherical harmonic coefficients to generate the multi-channel audio data. The audio encoding device 10′ may also perform spatial masking with respect to one or more base channels of the multi-channel audio data using the spatial masking threshold, and perform parametric inter-channel audio encoding with respect to the multi-channel audio data, including the spatially masked one or more base channels of the multi-channel audio data, to generate a bitstream.
In some instances, the audio encoding device 10′ may determine a target bitrate at which to encode the multi-channel audio data as the bitstream. In this context, when performing the spatial masking and the parametric inter-channel audio encoding, the audio encoding device 10′, when the target bitrate is less than a threshold bitrate, performs the spatial masking with respect to the one or more base channels of the multi-channel audio data and performing the parametric inter-channel audio encoding with respect to the multi-channel audio data, including the spatially masked one or more base channels of the multi-channel audio data, to generate the bitstream.
In some instances, the threshold bitrate is equal to 256 Kilobits per second (Kbps). In some instances, this threshold bitrate is specified by a user or application. That is, this threshold bitrate may be configurable or may be statically set. In some instances, the target bitrate is equal to 128 Kilobits per second (Kbps). In some instances, the parametric inter-channel audio encoding comprises a moving picture experts group (MPEG) Surround.
In some instances, the audio encoding device 10′ also performs temporal masking with respect to the multi-channel audio data using a temporal masking threshold.
Additionally, various aspects of the techniques may further (or alternatively) enable the audio encoding device 10′ to perform a method of compressing audio data. In performing this method, the audio encoding device 10′ may perform spatial analysis based on a plurality of spherical harmonic coefficients that describe a sound field of the audio data in three dimensions to identify a spatial masking threshold, perform spatial masking with respect to the plurality of spherical harmonic coefficients using the spatial masking threshold, and generate a bitstream that includes the plurality of spatially masked spherical harmonic coefficients.
The audio encoding device 10′ may, in some instances, determine a target bitrate at which to encode the multi-channel audio data as the bitstream. When performing the spatial masking, the audio encoding device 10′ may, when the target bitrate is equal to or greater than a threshold bitrate, perform the spatial masking with respect to the plurality of spherical harmonic coefficients. In some instances, the threshold bitrate is equal to 256 Kilobits per second (Kbps). The target bitrate is equal or greater than 256 Kilobits per second (Kbps) in these instances.
In some instances, the audio encoding device 10′ may further perform temporal masking with respect to plurality of spherical harmonic coefficients using a temporal masking threshold.
While described above as performing spatial masking analysis with respect to the spherical harmonic coefficients, the techniques described above with respect to the example of
The simultaneous masking unit 164 represents a unit that performs a simultaneous analysis (which may also be referred to as a “temporal analysis”) of the SHC 170B to determine one or more simultaneous masking thresholds 172. The simultaneous masking unit 164 may evaluate the sound field described by the SHC 170B to identify, as one example, concurrent but separate sounds. When there is a large difference in gain between two concurrent sounds, typically only the loudest sound (which may represent the sound with the largest energy) need be accurately represented while the comparably quieter sound may be less accurately represented (which is typically done by allocating less bits to the comparably quite sound). In any event, the simultaneous making unit 164 may output one or more simultaneous masking thresholds 172 (often specified on a frequency bin by frequency bin basis).
The spatial masking unit 166 may represent a unit that performs spatial analysis with respect to the SHC 170B and in accordance with various aspects of the techniques described above to determine one or more spatial masking thresholds 174 (which likewise may be specified on a frequency bin by frequency bin basis). The spatial masking unit 166 may output the spatial masking thresholds 174, which are combined by a combiner 176 with the temporal masking thresholds 172 to form overall masking thresholds 178. The combiner 176 may add or perform any other form of mathematical operation to combine the temporal masking thresholds 172 with the spatial masking thresholds 174 to generate the overall masking thresholds 178.
The bit allocation unit 168 represents any unit capable of allocating bits in a bitstream 180 representative of audio data based on a threshold, such as the overall masking thresholds 178. The bit allocation unit 168 may allocate bits using the various thresholds 178 to identify when to allocate more or less bits. Commonly, the bit allocation unit 168 operates in multiple so-called “passes,” where the bit allocation unit 168 allocates bits for representing the SHC 170B in the bitstream 180 during a first initial bit allocation pass. The bit allocation unit 168 may allocate bits conservatively during this first pass so that a bit budget (which may correspond to the target bitrate) is not exceeded. During second and possibly subsequent bit allocation passes, the bit allocation unit 168 may allocate any bits remaining in a bit budget (which may correspond to a target bitrate) to further refine how various frequency bins of the SHC 170B are represented in the bitstream 180. While described as allocating bits based on the overall masking thresholds 178, the bit allocation unit 168 may allocate bits based on any one or more of the spatial masking thresholds 174, the temporal masking thresholds 172 and the overall masking thresholds 178.
The simultaneous masking unit 164 of the audio encoding device 160 may then perform a simultaneous analysis (which may also be referred to as a “temporal analysis”) of the SHC 170B to determine one or more simultaneous masking thresholds 172 (204). The simultaneous making unit 164 may output one or more simultaneous masking thresholds 172 (often specified on a frequency bin by frequency bin basis).
The spatial masking unit 166 of the audio encoding device 160 may perform a spatial analysis with respect to the SHC 170B and in accordance with various aspects of the techniques described above to determine one or more spatial masking thresholds 174 (which likewise may be specified on a frequency bin by frequency bin basis) (206). The spatial masking unit 166 may output the spatial masking thresholds 174, which are combined by a combiner 176 with the simultaneous masking thresholds 172 (which may also be referred to as “temporal masking thresholds 172”) to form overall masking thresholds 178 (208). The combiner 176 may add or perform any other form of mathematical operation to combine the temporal masking thresholds 172 with the spatial masking thresholds 174 to generate the overall masking thresholds 178.
The bit allocation unit 168 represents any unit capable of allocating bits in a bitstream 180 representative of audio data based on a threshold, such as the overall masking thresholds 178. The bit allocation unit 168 may allocate bits using the various thresholds 178 to identify when to allocate more or less bits (210) in the manner described above. Again, while described as allocating bits based on the overall masking thresholds 178, the bit allocation unit 168 may allocate bits based on any one or more of the spatial masking thresholds 174, the temporal masking thresholds 172 and the overall masking thresholds 178.
In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.
By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.
The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.
Various embodiments of the techniques have been described. These and other aspects of the techniques are within the scope of the following claims.
This application claims the benefit of U.S. Provisional Application No. 61/828,132, filed May 28, 2013.
Number | Date | Country | |
---|---|---|---|
61828132 | May 2013 | US |