TARGET MID-SIDE SIGNALS FOR AUDIO APPLICATIONS

Abstract

The present disclosure relates to a method and audio processing arrangement for extracting a target mid (and optionally a target side) audio signal from a stereo audio signal. The method comprises obtaining (S1) a plurality of consecutive time segments of the stereo audio signal and obtaining (S2), for each of a plurality of frequency bands of each time segment of the stereo audio signal, at least one of a target panning parameter (Θ) and a target phase difference parameter (Φ). The method further comprises extracting (S3), for each time segment and each frequency band, a partial mid signal representation (211, 212) based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ) of each frequency band and forming (S4) the target mid audio signal (M) by combining the partial mid signal representations (211, 212) for each frequency band and time segment.

Description

TECHNICAL FIELD OF THE INVENTION

The present invention relates to source separation, signal enhancement, and signal processing. The present invention further relates to a method for extracting target mid and side audio signals from a stereo audio signal. The present invention further relates to a processing device employing the aforementioned method.

BACKGROUND OF THE INVENTION

In the field of audio processing, audio source separation is an important in several applications. In one simple application, a separated speech audio signal is kept or provided with additional gain whereas the background audio signal is omitted or attenuated relative to the speech audio signal. This can enhance the intelligibility of the speech audio signal.

In audio source separation, the signal of interest which is targeted for estimation or extraction is termed the target signal. The target signal is not limited to speech and may be any general audio signal (such as a musical instrument) or even a plurality of audio signals which are to be separated from an audio mix comprising noise and/or additional audio signals which are not of interest.

For stereo audio signals which comprise two audio signals, associated with a left and right audio channel respectively, one method of processing, which also implicitly performs some source separation, is to extract mid and side audio signals from the left and right audio signals, wherein the mid and side audio signals are proportional to the sum and difference of the left and right audio signals respectively. The mid signal will emphasize audio components that are equal in magnitude and in-phase between the left and right audio signals, whereas the side audio signal will attenuate or eliminate such signals. Accordingly, calculation of mid and side signals from the left and right audio signals constitutes an efficient method for respectively boosting or attenuating in-phase center-panned sources. Mid and side signals can further be converted back to left and right (conventional stereo) audio signals.

To the extent that a center-panned source is of interest, it is noted that source separation has been implicitly performed: the mid signal will contain this source while the side signal will not. The side audio signal will comprise mainly potentially uninteresting background audio signals which could be attenuated or omitted to enhance the intelligibility of the center panned in-phase audio source.

A drawback of conventional mid and side signal calculation is that when the desired audio source is not center-panned, the implicit source separation based on mid/side audio signal separation fails. To address this, specific mid and side signal calculation techniques have been developed for stereo audio signals with specific panning properties, techniques which are capable of successfully targeting stationary non-center panned audio sources. See e.g. Stereo Music Source Separation via Bayesian Modeling, by Master, Aaron, Ph.D. dissertation, Stanford University, 2006.

While these techniques alleviate some of the shortcomings of basic mid and side signal extraction, such panning-specific extraction techniques are still incapable of separating more general audio sources present in a stereo audio signal, such as moving audio sources, sources with reverberation, or sources which are spatially dominant at various points in time-frequency space.

It is therefore a purpose of this disclosure to provide such an improved method and an audio processing system for performing enhanced audio source separation for stereo audio signals.

SUMMARY OF THE INVENTION

According to a first aspect of the invention there is provided a method for extracting a target mid audio signal from a stereo audio signal, the stereo audio signal comprising a left audio signal and a right audio signal. The method comprises the steps of obtaining a plurality of consecutive time segments of the stereo audio signal, wherein each time segment comprises a representation of a portion of the stereo audio signal, and obtaining, for each frequency band of a plurality of frequency bands of each time segment of the stereo audio signal, at least one of a target panning parameter and a target phase difference parameter. The target panning parameter represents a distribution over the time segment of a magnitude ratio between the left and right audio signals in the frequency band and the target phase difference parameter represents a distribution over the time segment of the phase difference between the left and right audio signals of the stereo audio signal.

The method further comprises extracting, for each time segment and each frequency band, a partial mid signal representation, wherein the partial mid signal representation is based on a weighted sum of the left and right audio signal, wherein a weight of each of the left and right audio signals is based on at least one of the target panning parameter (Θ) and the target phase difference parameter of each frequency band and time segment, and forming the target mid audio signal by combining the partial mid signal representations for each frequency band and time segment.

Obtaining at least one of the target panning parameter and target phase difference parameter may comprise receiving, accessing or determining the target panning parameter and/or target phase difference parameter. At least one of the target panning parameter and/or target phase difference parameter may be replaced with a default value for at least one time segment and frequency band.

With consecutive time segments it is meant segments that describe a time portion of the audio signal wherein later time segments describe later time portions of the audio signal and earlier time segments describe earlier time portions of the audio signal. The consecutive time segments may be overlapping in time or non-overlapping in time.

The representation of a portion of the stereo audio signal may be any time domain representation or any frequency domain representation. The frequency domain representation may be any linear time-frequency domain such as a Short-Time Fourier Transform, STFT, representation or a Quadrature Mirror Filter, QMF, representation.

The invention is at least partially based on the understanding that by obtaining a target panning parameter and/or target phase difference parameter for each time segment and frequency band, a target mid audio signal may be extracted which at all times targets a time and/or frequency variant audio source in the stereo audio signal. Moreover, the method allows extraction of a target mid audio signal targeting two or more audio sources simultaneously which in the stereo audio signal are separated in frequency, as the target mid audio signal is created using individual target panning parameters for each time segment and frequency band.

In some implementations, the weight of the left and right audio signals is based on the target panning parameter such that the left or right audio signal with a greater magnitude is provided with a greater weight.

That is, the left or right audio signal which is associated with a greater magnitude or power (which is indicated by the target panning parameter) will be provided with a greater weight and contribute more to the formation of the target mid audio signal.

In some implementations, the method further comprises extracting, for each time segment and frequency band, a partial side signal representation, wherein the partial side signal representation is based on a weighted difference between the left and right audio signals, wherein a weight of each of the left and right audio signals is based on at least one of the target panning parameter and the target phase difference parameter of each frequency band and time segment and forming a target side audio signal by combining each partial side signal representation for each frequency band and time segment.

In other words, a target side audio signal may be formed in parallel with the target mid audio signal and the target mid and side audio signals form a complete representation of the stereo audio signal. It is understood that any implementation relating to the formation or processing of the target mid audio signal may be performed analogously with the forming and processing of the target side audio signal as will be described in the below.

If the stereo audio signal comprises e.g. an ideal center panned audio source, or a single audio source panned under the constant power law, the target mid audio signal will ideally capture all signal energy of the audio source and the target side audio signal will capture no such energy. For a general audio source(s) however, the target mid audio signal will capture the target audio source(s) and, likely, some other non-target sounds while the target side audio signal will capture non-target sounds while likely eliminating, or nearly eliminating, the target source(s). Inclusion of a target side audio signal in addition to the target mid audio signal ensures that all audio signals of the left and right audio signal pair are present in the target mid and side audio signal pair. For instance, this allows for lossless reconstruction of the left and right audio signals.

Any functions described in relation to a method may have corresponding features in a system or device and vice versa.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be described in more detail with reference to the appended drawings, showing currently preferred embodiments of the invention.

FIG. 1 is a block diagram depicting an analysis arrangement according to some implementations.

FIG. 2 is a flow chart illustrating a method according to one implementation.

FIG. 3 is a block diagram depicting an alternative analysis arrangement according to some implementations.

FIG. 4a illustrates a set of target panning parameters and target phase difference parameters distributed over a plurality of time segments and frequency bands according to some implementations.

FIG. 4b illustrates a set of partial mid signal representations forming a target mid audio signal according to some implementations.

FIG. 4c illustrates a set of partial side signal representations forming a target side audio signal according to some implementations.

FIG. 5a is a block diagram of an analysis arrangement according to some implementations.

FIG. 5b is a block diagram of an analysis arrangement using alternative panning and phase difference parameters according to some implementations.

FIG. 6a is a block diagram of an audio processing system with a mono source separator and a mid/side processor according to some implementations.

FIG. 6b is a block diagram of another audio processing system with a stereo processor operating on alternative left and right audio signals according to some implementations.

FIG. 6c is a block diagram of yet another audio processing system which applies an estimated stereo softmask to the target mid audio signal reconstructs processed left and right audio signals according to some implementations.

DETAILED DESCRIPTION OF CURRENTLY PREFERRED EMBODIMENTS

Systems and methods disclosed in the present application may be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of tasks does not necessarily correspond to the division into physical units; to the contrary, one physical component may have multiple functionalities, and one task may be carried out by several physical components in cooperation.

The computer hardware may for example be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that computer hardware. Further, the present disclosure shall relate to any collection of computer hardware that individually or jointly execute instructions to perform any one or more of the concepts discussed herein.

Certain or all components may be implemented by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that when executed by one or more of the processors carry out at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken are included. Thus, one example is a typical processing system (i.e. a computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system further may include a memory subsystem including a hard drive, SSD, RAM and/or ROM. A bus subsystem may be included for communicating between the components. The software may reside in the memory subsystem and/or within the processor during execution thereof by the computer system.

The one or more processors may operate as a standalone device or may be connected, e.g., networked to other processor(s). Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

The software may be distributed on computer readable media, which may comprise computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to a person skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, physical (non-transitory) storage media in various forms, such as EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information, and which can be accessed by a computer. Further, it is well known to the skilled person that communication media (transitory) typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

A detailed description of currently preferred embodiments is included in the below together with an introductory section of extraction of basic mid and side audio signals.

Extraction of Basic Mid and Side Audio Signals

Stereo audio signals comprising a left L and a right R audio signal are alternatively represented as a mid audio signal M and a side audio signal S. Basic mid and side audio signals M, S are constructed from the left and right audio signals L, R using the analysis equations

$\begin{array}{cc}M=0.5L+0\text{.5}R& \mathrm{Eq}.1\end{array}$ $\begin{array}{cc}S=0.5L-0.5R& \mathrm{Eq}.2\end{array}$

and the original left and right audio signals L, R may be reconstructed from the mid and side audio signals M, S using the following synthesis equations

$\begin{array}{cc}L=M+S& \mathrm{Eq}.3\end{array}$ $\begin{array}{cc}R=M-S& \mathrm{Eq}.4\end{array}$

The mid audio signal M boosts audio signal features that are in-phase and centered in the stereo mix whereas the side audio signal S attenuates in-phase audio signal features. For instance, if the stereo audio signal contains a center-panned left and right audio signal pair L, R (equal magnitude, in-phase components in each of the left and right audio signals), the mid audio signal M will contain the stereo audio signal while the side audio signal S will eliminate it; a basic mid/side audio signal targets a center-panned audio signal for inclusion in the mid audio signal M and for exclusion in the side audio signal S.

Stereo Signals in Time Domain and Time-Frequency Domain

The left, right, mid and side audio signals L, R, M, S may be represented in the time domain or frequency domain. The frequency domain representation may for example be a Short-Time Fourier Transform, STFT, representation or a Quadrature Modulated Filter, QMF representation. For example, the left and right audio signals L, R can be represented in the frequency domain as

$\begin{array}{cc}L=❘L❘e^{i\angle L}& \mathrm{Eq}.5\end{array}$ $\begin{array}{cc}R=❘R❘e^{i\angle R}& \mathrm{Eq}.6\end{array}$

wherein |L| and |R| denotes the signal magnitude for the left and right audio signals L, R respectively, and wherein ∠L and ∠R denotes the phase of the left and right audio signals L, R respectively. In general |L|, |R|, ∠L and ∠R are all functions of frequency, w, and time, t, however the time and frequency dependence ω, t will be excluded from all equations for compactness. There are, therefore, many values represented by each of the symbols |L|, |R|, ∠L and ∠R. Each such value corresponds to the audio content of the left or right signals, and does not necessarily characterize a single target source. The values may generally be described as “detected” because they are considered to be detected from a stereo input signal in many contexts.

The left and right audio signals L, R may alternatively be represented with a combination of a detected magnitude parameter U and detected panning parameter θ as

$\begin{array}{cc}U=\sqrt{{❘L❘}^2+{❘R❘}^2}& \mathrm{Eq}.7\end{array}$ $\begin{array}{cc}\theta =\mathrm{arc}\tan \left(\frac{❘R❘}{❘L❘}\right)& \mathrm{Eq}.8\end{array}$

meaning that the left and right audio signals L, R from equations 5 and 6 may be represented in terms of U, θ instead of in terms of |L|, |R|. For instance, the left and right audio signals L, R may be expressed as

$\begin{array}{cc}L=U\cos \left(\theta \right)e^{i\angle L}& \mathrm{Eq}.9\end{array}$ $\begin{array}{cc}R=U\sin \left(\theta \right)e^{i\angle R}.& \mathrm{Eq}.10\end{array}$

The detected magnitude parameter U and detected panning parameter θ forms a representation of the stereo audio signal which is referred to as a Stereo-Polar Magnitude (SPM) representation. While the SPM representation may be replaced with any equivalent representation, the SPM representation of the stereo audio signal will be utilized in the following description.

The detected panning parameter θ ranges from 0 to

$\frac{\pi }{2}$

wherein 0 indicates a stereo audio signal in which only the left audio signal L is non-zero and

$\frac{\pi }{2}$

indicates a stereo audio signal in which only the right audio signal R is non-zero. A detected panning parameter value of

$\theta =\frac{\pi }{4}$

indicates a centered stereo audio signal with equal magnitudes for the left and right audio signals L, R. While the detected panning parameter is derived from a stereo input signal, it also agrees mathematically with a model for source mixing. For an audio source S_x with magnitude |S_x|, phase ψ_x, and a known panning parameter Θ_x, the left and right audio signals L, R may be expressed as

$\begin{array}{cc}L=❘S_x❘\cos \left({\theta }_x\right)e^{i{\psi }_x}& \mathrm{Eq}.11\end{array}$ $\begin{array}{cc}R=❘S_x❘\sin \left({\theta }_x\right)e^{i{\psi }_x}& \mathrm{Eq}.12\end{array}$

It can be shown that the detected panning parameter θ for time-frequency points in a signal created with equation 11 and 12 will equal Θ_x of the audio source S_x, as:

$\begin{array}{cc}\theta =\mathrm{arc}\tan \left(\frac{❘R❘}{❘L❘}\right)=\mathrm{arc}\tan \left(\frac{S_x❘\sin \left({\Theta }_x\right)❘}{S_x❘\cos \left({\Theta }_x\right)❘}\right)={\Theta }_x.& \mathrm{Eq}.13\end{array}$

Mid and Side Audio Signals with Fixed Panning

The basic mid and side audio signals M, S are extracted by weighting each of the left and right audio signals L, R equally to target an audio source assumed to exist in equal proportions in the left and right audio signals L, R (a centered audio source). For non-centered audio sources, the weighting of the left and right audio signals L, R in equations 1 and 2 should be adjusted.

For instance, the coefficients of the left and right audio signals L, R in equation 1 and 2 may be changed from 0.5 under the constraint that the sum of the coefficients should equal 1. This enables the coefficients for the left and right audio signals L, R in equation 1 to be e.g. 0.7 and 0.3 (and equivalently for the side signal construction in equation 2) for an audio source which appears mainly in the left audio signal. However, this leads to an undesirable situation wherein the magnitude of the mid and side audio signals M, S varies considerably when the audio source moves between the left and right audio signals L, R (assuming constant power law mixing).

To circumvent this issue the mid and side audio signals M, S may be extracted from the left and right audio signals L, R in accordance with

$\begin{array}{cc}M=\cos \left({\Theta }_x\right)L+\sin \left({\Theta }_x\right)R& \mathrm{Eq}.14\end{array}$ $\begin{array}{cc}S=\sin \left({\Theta }_x\right)L-\cos \left({\Theta }_x\right)R& \mathrm{Eq}.15\end{array}$

for an audio source S_x with a known or estimated “target” panning parameter Θ_x wherein the left and right audio signals L, R can be reconstructed as

$\begin{array}{cc}L=\cos \left({\Theta }_x\right)M+\sin \left({\Theta }_x\right)S& \mathrm{Eq}.16\end{array}$ $\begin{array}{cc}R=\sin \left({\Theta }_x\right)M-\cos \left({\Theta }_x\right)S.& \mathrm{Eq}.17\end{array}$

For example, if Θ_x=0, indicating an audio source appearing only in the left audio signal, the mid audio signal M becomes equal to the left audio signal L and the side audio signal S is equal to the right audio signal R, and vice versa for a

${\Theta }_x=\frac{\pi }{2}.$

Similarly, a mid audio signal M for an audio source appearing with equal magnitudes in the left and right audio signals L, R

$\left(i.e.{\Theta }_x=\frac{\pi }{4}\right)$

weights the left and right audio signals L, R equally. Furthermore, an audio source located center to left and center to right will result in a weighting of the left and right audio signals L, R with Θ_x in the range of 0 to

$\frac{\pi }{4}$

and

$\frac{\pi }{4}$

to

$\frac{\pi }{2}$

respectively. The coefficients used to weight the left and right audio signals L, R to construct the mid and side audio signals M, S and vice versa are thus based on a fixed target panning parameter θ_x,

Extraction of Target Mid and Side Audio Signals

An aspect of the present invention relates to creation of a target mid and/or side audio signal M, S based on at least one of a target panning parameter Θ and a target phase difference parameter Φ obtained for each time segment and frequency band of a stereo audio signal. The target panning parameter Θ represents an estimate or indication of the magnitude ratio between the left and right audio signals L, R in each time segment and frequency band, for sounds corresponding to the target source or sources. The target phase difference parameter Φ represents an estimate or indication of the phase difference between the left and right audio signals L, R in each time segment and frequency band, for sounds corresponding to the target source or sources.

In some implementations, the panning parameter Θ and/or phase difference parameter Φ of each time segment and frequency band 111, 112 is the median, mean, mode, numbered percentile, maximum or minimum panning parameter Θ and/or phase difference parameter Φ of the time segment. In general, the detected panning parameter θ and/or detected phase difference parameter ϕ exists for each sample of the stereo audio signal. However, the target panning parameter Θ and/or phase difference parameter Φ used to create the target mid and side audio signals M, S are not necessarily of such fine granularity and may represent a statistical feature (e.g. the average) of multiple samples, such as an average of all samples within a predetermined time segment. For example, whereas the detected panning parameter θ and/or detected phase difference parameter ϕ may assume different values more than one thousand times per second the target panning parameter Θ and/or phase difference parameter Φ only change a few times per second.

FIG. 1 illustrates an analysis arrangement 10 extracting a target mid M and target side S audio signal from a stereo signal comprising a left and right audio signals L, R. The extraction of the mid and side audio signals M, S is performed in the extraction unit 12 of the analysis arrangement 10 which obtains the left and right audio signals L, R together with a target panning parameter Θ and/or target phase difference parameter Φ for a plurality of time segments 100 and frequency of bands 111, 112.

The extraction unit 12 obtains the left and right audio signals L, R (stereo audio signal) as a plurality of consecutive time segments, each time segment comprising a representation of a portion of the left and right audio signals R, L. Alternatively, the extraction unit 12 is configured to divide the left and right audio signals R, L into the plurality of consecutive time segments. Each time segment comprises two or more frequency bands of the stereo audio signal wherein each frequency band in each time segment is associated with at least one of an obtained target panning parameter θ and an obtained target phase difference parameter Φ.

In the embodiment shown in FIG. 1 the analysis arrangement 10 receives the left and right audio signals L, R and a target panning parameter Θ and/or target phase difference parameter Φ for each frequency band 111, 112 of each time segment wherein the number of frequency bands ranges from band 1 to band B. Θ_(t,1) and Φ_(t,1) represents the target panning parameter Θ and target phase difference parameter Φ for a first frequency band 111 of time segment t while Θ_(t,2) and Φ_(t,2) represents the target panning parameter and target phase difference parameter for a second frequency band 112 of the time segment t.

With the left and right audio signals L, R and a target panning parameter Θ and/or target phase difference parameter Φ the extraction unit 12 extracts the target mid audio signal M and the target side audio signal S. In some implementations, the extraction unit 12 extracts only one of the target mid M and side S audio signal, such as only the mid audio signal M.

To allow modeling of the interchannel phase difference between the left and right audio signals L, R the genuine source phase of a target audio source is defined as being proportionally closer to the left or right audio signal which exhibits the greater magnitude. That is, for a stereo audio signal which is dominated in power or magnitude by the left audio signal L the phase of left audio signal L will be closer to the genuine source phase of the audio source compared to the phase of the right audio signal R, and vice versa for a stereo audio signal dominated by the right audio signal R. An audio source S_x is modeled to appear in the left and right audio signals L, R according to the following mixing equations

$\begin{array}{cc}L=❘S_x❘\cos \left({\Theta }_x\right)e^{i\left({\psi }_x+{\Phi }_x{\sin }^2{\Theta }_x\right)}& \mathrm{Eq}.18\end{array}$ $\begin{array}{cc}R=❘S_x❘\sin \left({\Theta }_x\right)e^{i\left({\psi }_x-{\Phi }_x{\cos }^2{\Theta }_x\right)}& \mathrm{Eq}.19\end{array}$

wherein ψ_x is the phase of the audio source S_x and Φ_x is the phase difference parameter for the target source.

Based on the mixing model described in equations 17 and 18 above, the extraction device 12 may employ the following analysis equations to target a source with target panning parameter Θ and target phase difference parameter Φ.

$\begin{array}{cc}M=\cos \left(\Theta \right)e^{i\left(-\Phi {\sin }^2\Theta \right)}L+\sin \left(\Theta \right)e^{i\left(\Phi {\cos }^2\Theta \right)}R& \mathrm{Eq}.20\end{array}$ $\begin{array}{cc}S=\sin \left(\Theta \right)e^{i\left(-\Phi {\sin }^2\Theta \right)}L-\cos \left(\Theta \right)e^{i\left(\Phi {\cos }^2\Theta \right)}R& \mathrm{Eq}.21\end{array}$

It is understood that Θ and Φ can vary with time and frequency, corresponding with the characteristics of a targeted source. The mid and side signals of equations 20 and 21 may be termed “target mid and side signals.” Therefore M and S may be understood as signals comprised of separate signal components corresponding to the various values of Θ and Φ for each time segment and frequency band 111, 112. The weighting of the phase difference parameter Φ for the left and right audio signals L, R with sin² Θ and cos² Θ is merely exemplary and other weighting functions based on O are possible. For instance, sin² Θ_x may be replaced with

$\frac{2}{\mathrm{\pi \Theta }}$

and cos² Θ may be replaced with

$1-\frac{2}{\mathrm{\pi \Theta }}.$

Similarly, the model of equations 18 and 19 which uses weighting of the phase difference parameter Φ for the left and right audio signals L, R with sin² Θ_x and cos² Θ_x is also merely exemplary.

In equation 20 the target mid audio signal M is based on a weighted sum of the left and right audio signals L, R wherein the weight of the left audio signal is cos(Θ)e^{i(−Φ sin2Θ)} and the weight of the right audio signal is sin(Θ)e^{i(Φ cos2Θ)} and wherein the panning and/or phase difference parameter Θ, Φ is obtained for each time segment and frequency band 111, 112.

Similarly, in equation 20 the target side audio signal S is based on weighted difference between the left and right audio signals L, R wherein the weight of the left audio signal is sin(Θ)e^{i(−Φ sin2Θ)} and the weight of the right audio signal is cos(Θ)e^{i(−Φ cos2Θ)} and wherein the panning and/or phase difference parameter Θ, Φ is obtained for each time segment and frequency band 111, 112.

The weights of the left and right audio signals L, R in the weighted sum and weighted difference of equations 20 and 21 used to extract the mid and side audio signals M, S are complex valued and comprises a real valued magnitude factor, e.g. cos(Θ), and an complex valued phase factor, e.g. e^{i(−Φ sin2Θ)}. The obtained target panning and/or target phase difference parameters Θ, Φ will influence the weights to enable extraction of the target mid and/or side audio signal M, S.

For instance, the real valued magnitude factor may be based on Θ such that the left or right audio signal with a greater magnitude is provided with a greater weight in the weighted sum. Moreover, the complex valued factor may be based on Φ such that a greater phase difference Φ means that there is a greater difference between the respective phases of the weights in the weighted sum. In some implementations, the complex valued factor of each weight in the weighted sum is based on both Φ and Θ, e.g. one of the weights is e^{i(−Φ sin2Θ)}, such that the weight of the left or right audio signal L, R with a greater magnitude is provided with a smaller phase.

Analogously, the real valued magnitude factor in the weighted difference (for the target side signal extraction) may be based on the target panning parameter Θ such that the left or right audio signal with a greater magnitude is provided with a smaller weight in the weighted difference. Moreover, the complex valued factor may be based on Φ such that a greater phase difference Φ means that there is a greater difference between the respective phases of the weights in the weighted difference. In some implementations, the complex valued factor of each weight in the weighted difference is based on both Φ and Θ, e.g. one of the weights is sin(Θ)e^{i(−Φ sin2Θ)}, such that the weight of the left or right audio signal with a greater magnitude is provided with a smaller phase.

In some implementations, only one of the target panning and phase difference parameter Θ, Φ is obtained for at least one time segment and frequency band 111, 112 wherein the other one is assigned a default value. For instance, if only the target panning parameter Θ is obtained the target phase difference parameter Φ is assumed to be Φ=0 and if only the target phase difference parameter Φ is obtained the target panning parameter Θ is assumed to be

$\Theta =\frac{\pi }{4}.$

These default values assume an audio source centered in the stereo audio signal with no interchannel phase difference, however other default values are possible.

The extraction device 12 obtains the target mid audio signal M by combining partial mid signal components or representations M_{(t, n)} of each frequency band in each time segment. Similarly, extraction device 12 obtains the target side audio signal S by combining partial side signal representations S_{(t, n)} of each frequency band in each time segment. A partial mid signal component or representation M_{(t, n)} is created for each frequency band 1 . . . . B of a time segment t and by combining all the partial mid signal representations M_{(t, n)} for time segment, t, a target mid audio signal time segment M_(t) is created wherein a sequence of target mid audio signal time segments M_(t) forms the target mid audio signal M. The target side audio signal S is formed in an analogous manner. Combining the partial mid signal representations M_{(t, n)} for frequency bands 1 . . . . B may comprise processing the partial target mid audio signals M_{(t, n)} with a synthesis filterbank.

With further reference to the flow chart in FIG. 2 the method performed by the analysis arrangement 10 is described in detail. The stereo audio signal is obtained by the analysis arrangement 10 at S1 and the target panning and/or phase difference parameter Θ, Φ is obtained by the analysis arrangement 10 for a plurality of time segments 100 and frequency bands 111, 112 at step S2.

At step S3 a partial mid signal representation M_{(t, n)}, is extracted by the mid/side extractor 12 (e.g. in accordance with equation 19 in the above) wherein the partial mid signal representation is based on a weighted sum of the left and right audio signals L, R. The weight of each of the left and right audio signals L, R is based on at least one of the target panning parameter Θ and the target phase difference parameter Φ of each frequency band 111, 112 and time segment. Analogously, a partial side signal representation S_{(t, n)} is extracted at step S31 (e.g. in accordance with equation 20 in the above) wherein the partial side signal representation is based on a weighted difference of the left and right audio signals L, R.

In some implementations, the target side audio signal S is not computed and the extractor unit 12 is e.g. an extractor unit 12 configured to extract exclusively the target mid audio signal M. Alternatively, the target side audio signal S is computed but scaled down at step S51 by a small value, or zero, prior to reconstruction of the left and right audio signals L, R at step S6 (e.g. using the reconstruction equations 21 and 22 in the below). As the target mid audio signal M in some cases is expected to capture substantially all energy of the target audio source the target side audio signal S may comprise mainly undesirable noise or background audio which can be omitted or attenuated.

Following step S3 and step S31 the method goes to step S4 and S41 comprising forming target mid and side audio signals M, S by combining the partial mid and side signal representations M_{(t, n)}, S_{(t, n)} into target mid and side audio signals M, S respectively. The method continues with step S5 and S51 comprising performing processing of the target mid and/or side audio signal M, S and examples of processing involves attenuating the target side audio signal S or providing the target mid audio signal M to a mono source separator as will be described in connection to FIGS. 6a, 6b and 6c in the below.

Lastly, the method goes to step S6 comprising reconstructing the left and right audio signals L, R from the target mid and side audio signals M, S. The reconstruction of the left and right audio signals L, R from the target mid and side audio signals M, S is performed by a synthesis arrangement as will be described more in detail in connection to FIGS. 5a and 5b in the below.

As an example of the operation of the analysis arrangement 10, an audio source of the left and right audio signals L, R with time varying panning is considered. The audio source is panned, under the constant power law, at a constant rate from fully right to fully left as time t advances from 0 to 10 seconds. For basic mid and side audio signals extracted for this audio source, e.g. in accordance with equation 1 and 2 in the above, the mid audio signal M will capture substantially all energy of the audio source at t=5 and substantially no energy at t=0 and t=10 whereas the side audio signal S will capture substantially all the energy of the audio source at t=0 and t=10 and substantially no energy at t=5. The target mid audio signal M, extracted with varying panning parameters and/or phase difference parameters Θ, Φ for each time segment and frequency band will contain substantially all of the audio source energy for any t∈[0, 10] whereas the target side audio signal S will contain substantially no energy of the of the time varying audio source.

The constant power law ensures that the audio signal power (which is proportional to the sum of the square of the amplitudes) of the left and right audio signals L, R remains constant for all panning angles. For instance, the audio signal amplitudes of the left and right audio signal are scaled with a scaling factor which depends on the panning angle to ensure that the square of the sum of the left and right audio signal is equal for all panning angles. Different from the linear panning law, which ensures that the sum of the signal amplitudes is constant which allows the total signal power to vary with the panning angle, the constant power panning law will ensure that the perceived audio signal power is constant for all panning angles. The constant power law is e.g. described in “Loudness Concepts & Pan Laws” by Anders Øland and Roger Dannenberg, Introduction to Computer Music, Carnegie Mellon University.

This demonstrates how the target mid and side audio signals M, S is capable of targeting and separating audio source(s) that are varying in time and frequency in a stereo audio signal. Additionally, if a second audio source of a separate frequency is moved between left and right audio signals L, R at a rate different from the first audio source (or is stationary e.g. at a center panning), the target mid audio signal M targets the first and second audio source in individual frequency bands meaning that substantially all energy from both audio sources will be present in the target mid audio signal M despite the audio sources being of different frequency while also being shifted between the left and right audio signals L, R at different rates.

In FIG. 3 a combined analysis arrangement 10′ is depicted that is configured to extract a target mid and side audio signals M, S in accordance with the above while also extracting or determining the target panning parameter Θ and/or phase difference parameter Φ for each frequency band 111, 112 and time segment of the left and right audio signals L, R using a parameter extractor unit 14. Accordingly, the target panning parameter θ and/or phase difference parameter Φ are obtained alongside the left and right audio signals L, R or determined from the left and right audio signals L, R. An example method of doing so will now be described.

The target phase difference parameter Φ may be calculated, e.g. by the parameter extractor 14, as the typical or dominant phase difference between the left and right audio signals L, R in each time segment and frequency band 111, 112. For instance, in the time-frequency domain (e.g. in the STFT domain), detected phase differences may be calculated for each STFT tile as ϕ=Arg(R/L), and by analyzing the distribution on ϕ, a value of Φ may be estimated to be a dominant or typical value for the time segment and frequency band. Similarly, detected panning parameters θ may be calculated as θ=arctan (|R|/|Z|) for each STFT tile, and by analyzing the distribution thereupon, a values of Θ may be estimated to be a dominant or typical value for the time segment and frequency band, wherein L and R represent the left and right audio signals L, R in the time segment and frequency band 111, 112. Typical or dominant values estimated from a distribution may be the median, mean, mode, numbered percentile, minimum or maximum values.

In “Dialog Enhancement via Spatio-Level Filtering and Classification” (Convention Paper 10427 of the 149th Convention of the Audio Engineering Society by Master, A et al.) a method for obtaining indicators of the average value and spread of the panning and interchannel phase difference labelled thetaMid, thetaWidth, phiMid and phiWidth is proposed. The panning parameter Θ used by the analysis arrangement 10 could be based on the thetaMiddle proposed in this convention paper and, similarly, the target phase difference parameter Φ used by the analysis arrangement 10 could be based on phiMiddle of the so called “Shift and Squeeze” or “S&S” parameters proposed in this paper.

For each time segment and frequency band within it, the audio processing system may create a 51-bin histogram on θ weighted by the magnitude U squared, i.e. U². The system does the same for the detected panning parameter ϕ, as well as a version of ϕ ranging from 0 to 2*pi, which is called ϕ2. However, these histograms each use 102 bins. The histograms are each smoothed over their given dimensions and vs time segments. For the smoothed θ histograms, the system detects the target panning parameter as the highest peak, which is called thetaMiddle, and also the width around this peak necessary to capture 40% of energy in the histogram, which is called theta Width. It does the same for ϕ and ϕ2, recording phiMiddle, phi2Middle, phiWidth and phi2Width, but requiring 80% energy capture for widths. The system records final values for phiMiddle and phiWidth based on which had a higher concentration in phi space as indicated by a smaller phiWidth value

For instance, the parameter extractor 14 in FIG. 3 creates, for each time segment and frequency band within it, a histogram of the detected panning parameter θ weighted with the total signal power U² from equation 7 and does the same for the detected phase difference parameter ϕ. The histograms may have an arbitrary number of bins such as ten or more bins, or 50 or more bins. In one implementation, each histogram has 51 bins. The histograms are each smoothed with a suitable smoothing function with respect to at least the time dimension and/or with respect to the time and θ or ϕ dimension. For the smoothed θ histograms, the system detects the θ value of the highest peak, which is called thetaMiddle, and also the width around this peak necessary to capture 40% of energy in the histogram, which is called theta Width. The same is done for ϕ, recording phiMiddle and phiWidth, but requiring 80% energy capture for widths. It is understood that the parameter extractor 14 may be configured to either obtain the detected the detected panning parameter θ and/or detected phase difference parameter ϕ and/or detected magnitude U or determine the detected panning parameter θ and/or detected phase difference parameter ϕ and/or detected magnitude U (using e.g. equation 7) from the left and right audio signal L, R.

Details of Time Segments and Frequency Bands

With reference to FIG. 4a a plurality of time segments 110, 120 ranging from a first time segment to time segment t are depicted wherein each time segment 110, 120 comprises a plurality of frequency bands 111, 112, 121, 122 ranging from a first frequency band to frequency band B. The time segments 110, 120 and frequency bands 111, 112, 121, 122 forms a time and frequency tile representation wherein each frequency band of each time segment forms a respective time/frequency tile.

As indicated in FIG. 4a, each frequency band 111, 112, 121, 122 of each time segment 110, 120 is associated with a respective target panning parameter Θ and/or target phase difference parameter Φ. At least two frequency bands 111, 112, 121, 122 (or sub-bands) are defined for each time segment 110, 120, such as a high frequency band and a low frequency band comprising frequencies above and below a threshold frequency. That is, each time segment 110, 120 comprises frequency bands 1 . . . . B wherein B is at least two. Experiments have shown that octave or quasi-octave frequency bands are well suited for targeting of dialogue audio sources such as frequency bands with edges at 0 Hz, 400 Hz, 800 Hz, 1600 Hz, 3200 Hz, 6400 Hz, 13200 Hz and 24000 Hz, although different band distributions are possible.

The duration of each time segment may correspond to 10 ms to 1 s of the stereo audio signal. In some implementations each time segment corresponds to a plurality of frames of the stereo audio signal, such as 10 frames, wherein each frame e.g. represents 10 to 200 ms of the stereo audio signal, such as 50 to 100 ms of the stereo audio signal. Experiments have shown that using 10 overlapping frames to represent one time segment, wherein each frame is 50 to 100 ms long with 75% overlap, is a good trade-off between rapid response, parameter stability, parameter reliability and computational expense for typical dialog sources. However, time segments of longer or shorter duration are also considered, specifically for audio sources other than dialog, such as music.

FIG. 4b illustrates partial mid audio signal representations M_{(t, b)} extracted for each time segment 210, 220 and frequency band 211, 212, 221, 222. As seen, a first time segment 210 comprises a partial mid audio signal representation M_(1,1) for the first frequency band 211, a second partial mid audio signal representation M_(1,1) for the second frequency band 212 and so forth until the last mid audio signal representation M_(1,B) of frequency band B. By combining the partial mid audio signal representations M_(1,1) . . . . M_(1,B) of the first time segment 210 a first time segment M₍₁₎ of the target mid audio signal M is created. Similarly, subsequent time segments M₍₂₎ . . . . M_(t) of the target mid audio signal M are created by combining the partial mid audio signal representations M_(1,1) . . . . M_(1,B) of subsequent time segments 220. Finally, the target mid audio signal M is created by combining each time segment M₍₁₎ . . . . M_(t) in sequence.

With further reference to FIG. 4c there is illustrated partial side audio signal representations S_{(t, b)} extracted for each time segment 310, 320 and frequency band 311, 312, 321, 322. The partial side audio signals representations S_{(t, b)} are combined to form the target side audio signal S in a manner analogous to the combination of the partial mid audio signal representations M_{(t, b)} described in the above. It is understood that the extraction and combination of the partial mid audio signal representations M_{(t, b)} and/or partial side audio signal representations S_{(t, b)} may be performed in a single step by a single processing unit or as two or more steps performed by two or more processing units.

Reconstruction of Left and Right Audio Signals from Target Mid and Side Audio Signals

FIG. 5a illustrates a block chart illustrating a synthesis arrangement 20 for performing reconstruction of the left and right audio signals L, R from the target mid and side audio signals M, S. The synthesis unit 20 comprises a reconstructor unit 22 which reconstructs the left and right audio signals L, R from a target mid and side audio signals M, S using the panning parameter Θ and/or phase difference parameter Φ. For instance, the reconstructor unit 22 implements the following synthesis equations

$\begin{array}{cc}L=\left(\cos \left(\Theta \right)M+\sin \left(\Theta \right)S\right)e^{i\left(\Phi {\sin }^2\Theta \right)}& \mathrm{Eq}.22\end{array}$ $\begin{array}{cc}R=\left(\sin \left(\Theta \right)M-\cos \left(\Theta \right)S\right)e^{i\left(\Phi {\cos }^2\Theta \right)}& \mathrm{Eq}.23\end{array}$

for the target mid and side audio signals M, S extracted with equations 20 and 21. The target panning parameter Θ and/or target phase difference parameter Φ obtained by the synthesis arrangement is here equal to the target panning parameter Θ and/or phase difference parameter Φ determined or obtained by the analysis arrangement. Alternatively, the target panning parameter Θ and/or phase difference parameter Φ used by the synthesis arrangement are alternative parameters Θ_ALT, Φ_ALT and/or at least one of Θ and Φ is unspecified and replaced with a default value for at least one time segment and frequency band.

As seen in equation 22 and 23, the reconstruction of the left and right audio signals L, R is based on a weighted sum and weighted difference of the target mid and side audio signals M, S. However, these equations may be modified to downweight or eliminate the contribution from the side signal. For instance, the left and right audio signals L, R may be reconstructed from only the target mid audio signal M with a mid weighting factor of cos(÷)e^{i(−Φ sin2Θ)} for the left signal and a mid weighting factor of sin(Θ)e^{i(−Φ sin2Θ)} for the right signal respectively, wherein the weighting factors are based on at least one of the target panning parameter Θ and target phase difference parameter ϕ. In this example, the contribution from the side signal is zero. Other contributions between zero and the values shown in equations 22 and 23 are possible.

Similarly, the left and right audio signals L, R may be reconstructed by taking the target side audio signal S into account. In such cases the left audio signal L is based on a weighted sum of the target mid audio signal M and target side audio signal S wherein the weights, cos(Θ)e^{i(−Φ sin2Θ)} and sin(Θ))e^{i(−Φ sin2Θ)} are based on at least one of the panning parameter and the phase difference parameter. While the right audio signal R is based on a weighted difference between the target mid audio signal M and the target side audio signal S wherein the weights, sin(Θ)e^{i(−Φ cos2Θ)} and cos(Θ)e^{i(−Φ cos2Θ)}, are based on at least one of the panning parameter Θ and the phase difference parameter Φ. The contribution from the mid signal may also be downweighted or reduced to zero.

FIG. 5b is a block chart illustrating a synthesis arrangement 20 for performing reconstruction using equations 22 and 23 with an alternative panning parameter Φ_ALT and/or alternative phase difference parameter Φ_ALT which results in alternative left and right audio signals L_ALT, R_ALT. In the implementation shown,

${\Theta }_{ALT}={\Theta }_{CEN}=\frac{\pi }{4}$

and Φ_ALT=Φ_CEN=0 for all time segments and frequency bands to create a center-panned stereo audio signal comprising a centered left and right audio signals L_CEN, R_CEN with no inter-channel phase difference.

The alternative mid and side weighing factors may be defined as the mid and side weighting factors described in equation 22 and 23, with Θ, Φ replaced with the alternative counterparts Θ_ALT, Φ_ALT obtained for each time segment and frequency band. In some implementations, the alternative left and right audio signals L_ALT, R_ALT are created using only the target mid audio signal M and the alternative mid weighting factors (being e.g. the coefficients of the target mid audio signal M in equation 22 and 23 with Φ_ALT, Φ_ALT replacing Θ, Φ). Alternatively, the alternative left and right audio signals L_ALT, R_ALT are created using both the target mid and target side audio signal M, S and the alternative mid and side weighting factors (being e.g. the coefficients of the target mid audio signal M and target side audio signal S in equation 22 and 23 respectively, with Θ_ALT, Φ_ALT replacing Θ, Φ).

Processing Arrangements Implementing Target Mid and Side Audio Signals

In the following, FIGS. 6a, 6b and 6c depicts block diagrams illustrating an audio processing system comprising the analysis arrangement 10 and synthesis arrangement 20 in combination with additional audio processing units such as a mono source separator 30, a mid/side processor 40, a stereo processor 50 and a stereo softmask estimator 60 for performing audio processing of the target mid and side audio signals M, S or alternative left and right audio signals L_CEN, R_CEN.

FIG. 6a illustrates an audio processing system for processing a stereo audio signal via the target mid and side audio signal M, S extracted by the analysis arrangement 10. As seen, the target mid audio signal M is used as input to a mono source separator 30. Additionally, the target mid and side M, S audio signals are provided to a mid/side processor 40 for applying a boost and/or attenuation to the audio signals.

The target mid audio signal M is provided to a mono source separation system 30 configured to separate an audio source in the target mid audio signal M and produce a source separated mid audio signal M_sep. For instance, the mono source separation system 30 produces a source separated mid audio signal M_sep with enhanced intelligibility of at least one audio source present in the target mid audio signal M.

The source separated mid audio signal M_sep is provided together with the target side audio signal S to a mid/side processor 40 to produce processed mid and side audio signals M′, S′. The mid/side processor 40 may apply a gain to the source separated mid audio signal M_sep and/or attenuate the target side signal S. In some cases, the mid/side processor 40 sets the target side audio signal S to zero. The processed mid and side audio signals M′, S′ are then provided to a synthesis arrangement 20 comprising a left and right audio signals reconstruction unit 22 which reconstructs a processed left and right audio signals L′, R′ based on the processed mid and side audio signals M′, S′ and the panning and/or phase difference parameters Θ, Φ obtained by the analysis arrangement 10.

In some implementations, the source separated mid audio signal M_sep is provided directly to the synthesis arrangement 20 which reconstructs a processed left and right audio signals L′, R′ based on at least the source separated mid audio signal M_sep (and optionally the target side audio signal S) and the target panning and/or phase difference parameters Θ, Φ.

Alternatively, the mono source separation system 30 is omitted and the target mid audio signal M and target side audio signal S is provided directly to the mid and side audio processing unit 40 which extracts processed mid and side audio signals M′, S′ from the target mid and side audio signals M, S.

FIG. 6b illustrates how the target mid audio signal M and the target side audio signal S are used to extract alternative centered left and right audio signals L_CEN, R_CEN to enable use of a stereo processing system 50 for center panned stereo audio signals.

The analysis arrangement 10a obtains an arbitrary left and right audio signals L, R and the target panning and/or phase difference parameters Θ, Φ for each time segment and frequency band and extracts a target mid audio signal M and a target side audio signal S. The target mid and side audio signal M, S is provided to the synthesis arrangement 20a which also obtains a set of alternative centered panning and/or phase difference parameters Θ_CEN, Φ_CEN indicating center panned stereo audio signal. Accordingly, the analysis arrangement 10a and synthesis arrangement 20a creates a center panned stereo signal comprising a centered left and right audio signals L_CEN, R_CEN from an arbitrary original stereo signal.

The center panned alternative left and right audio signals L_CEN, R_CEN are provided to a stereo processing system 50 configured to perform stereo source separation on center panned stereo sources to output processed centered left and right audio signals L′_CEN, R′_CEN. The processed centered left and right audio signals L′_CEN, R′_CEN features e.g. enhanced intelligibility of at least one audio source present in the centered left and right audio signals L_CEN, R_CEN.

The processed centered left and right audio signals L′_CEN, R′_CEN are then provided to a second analysis arrangement 10b which extracts a processed mid and side audio signals M′, S′ using the processed centered left and right audio signals L′_CEN, R′_CEN and the centered panning and/or phase difference parameters Θ_CEN, Φ_CEN. Lastly, a second synthesis arrangement 20b utilizes the original panning and/or phase difference parameters Θ, Φ obtained or determined by the first analysis arrangement 10a to reconstruct processed left and right audio signals L′, R′.

It is noted that the centered panning and/or phase difference parameters Θ_CEN, Φ_CEN indicating center panning with no interchannel phase difference is one alternative of among many possible values of alternative panning and/or phase difference parameters Θ_ALT, Φ_ALT. For instance, the centered alternative parameters Θ_CEN, Φ_CEN may be replaced with arbitrary alternative parameters Θ_ALT, Φ_ALT which e.g. indicates a strictly right or left stereo audio signal with or without a non-zero phase difference.

It is further noted that the target mid and side audio signals M, S fed between the first analysis arrangement 10a and first synthesis arrangement 20a may be subject to mono signal separation processing and/or mid and side processing as described in connection to FIG. 6a in the above. The same applies to the processed target mid and side audio signals M′, S′ fed between the second analysis arrangement 10b and the second synthesis arrangement 20b. That is, a mono source separator 30 and/or a mid/side processor 40 from FIG. 6a may be used to process the mid and side audio signals passed between at least one of: the first analysis arrangement 10a and first synthesis arrangement 20a and between the second analysis arrangement 10b and the second synthesis arrangement 20b. Additionally, while the first analysis and synthesis arrangements 10a, 20a are depicted as separate units they may be combined into a single unit, with reference to equations 20, 21, 22, 23 it is clear that the operations for creating the alternative left and right audio signals L_CEN, R_CEN may be performed as a single step. The same applies to the second analysis and synthesis arrangements 10b, 20b.

In some implementations, the alternative (e.g. centered) left and right audio signals L_CEN, R_CEN are created by the first synthesis arrangement 20a using only the target mid audio signal M. Analogously, the second analysis arrangement 10b may be configured to extract only the processed target mid audio signal M′.

FIG. 6c illustrates an analysis arrangement 10 operating in conjunction with a filter estimator configured to receive the left and right audio signals L, R and estimate a filter for source separation. The filter estimator is e.g. a stereo softmask estimator 60 configured to receive the left and right audio signals L, R and estimate a softmask F. The stereo softmask estimator 60 may be any softmask estimator and e.g. configured to output a softmask F intended for stereo source separation. Traditionally, the softmask F is applied to the left and right audio signals L, R of a stereo audio signal.

In the implementation shown in FIG. 6c, the estimated stereo softmask F is applied to the target mid audio signal M to form a source separated mid audio signal M_sep. The source separated mid audio signal M_sep is provided together with a target side audio signal S to a mid/side processor 40 which outputs a processed target mid audio signal M′ and a processed target side audio signal S′ wherein the processed side audio signal S′ may be downweighted compared with S, including a downweight down to zero. The processed target mid audio signal M′ and the processed target side audio signal S′ are provided to a synthesis unit 20 which reconstructs processed left and right audio signals L′, R′. Optionally, the mid/side processor is omitted whereby the source separated mid audio signal M_sep, and optionally the target side audio signal S are provided directly to the synthesis unit 20 for reconstruction.

The stereo softmask estimator 60 is configured to determine or obtain the panning and/or phase difference parameter Θ, Φ and provide these parameters to the analysis arrangement 10. Alternatively, the panning and/or phase difference parameter Θ, Φ are obtained from elsewhere by the analysis arrangement 10 or determined by the analysis arrangement 10 based on the left and right audio signals L, R.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the disclosure discussions utilizing terms such as “processing”, “computing”, “calculating”, “determining”, “analyzing” or the like, refer to the action and/or processes of a computer hardware or computing system, or similar electronic computing devices, that manipulate and/or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

It should be appreciated that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention. Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

Furthermore, some of the embodiments are described herein as a method or combination of elements of a method that can be implemented by a processor of a computer system or by other means of carrying out the function. Thus, a processor with the necessary instructions for carrying out such a method or element of a method forms a means for carrying out the method or element of a method. Note that when the method includes several elements, e.g., several steps, no ordering of such elements is implied, unless specifically stated. Furthermore, an element described herein of an apparatus embodiment is an example of a means for carrying out the function performed by the element for the purpose of carrying out the invention. In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description. The person skilled in the art realizes that the present invention by no means is limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, the stereo processing system in FIG. 6c may further comprise a mono source separator 30 operating on the mid audio signal M or the source separated mid audio signal M_sep. As a further example it is envisaged that all components or processes relating to forming a target mid and side audio signal and processing a mid and side audio signal may be configured to form or process only the target mid audio signal or, alternatively, the target mid audio signal and the target side audio signal. Similarly, reconstructing a left and right audio signal (e.g. an alternative or centered left and right audio signal) based on the target mid and side audio signals may comprise reconstructing a left and right audio signal using only the target mid audio signal or, alternatively, using both the target mid and side audio signal.

Various features and aspects will be appreciated from the following enumerated exemplary embodiments (“EEEs”):

• • EEE 1. A method for extracting a target mid audio signal (M) from a stereo audio signal, the stereo audio signal comprising a left audio signal (L) and a right audio signal (R), said method comprises: obtaining (S1) a plurality of consecutive time segments of the stereo audio signal, wherein each time segment comprises a representation of a portion of the stereo audio signal; obtaining (S2), for each frequency band of a plurality of frequency bands of each time segment of the stereo audio signal, at least one of: a target panning parameter (Θ), representing a distribution over the time segment of a magnitude ratio between the left and right audio signals (L, R) in the frequency band, and a target phase difference parameter (Φ), representing a distribution over the time segment of the phase difference between the left and right audio signals (L, R) of the stereo audio signal; extracting (S3), for each time segment and each frequency band, a partial mid signal representation (211, 212), wherein the partial mid signal representation (211, 212) is based on a weighted sum of the left and right audio signal (L, R), wherein a weight of each of the left and right audio signals (L, R) is based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ) of each frequency band; and forming (S4) the target mid audio signal (M) by combining the partial mid signal representations (211, 212) for each frequency band and time segment.
  • EEE 2. The method according to EEE 1, wherein the weight of the left and right audio signals (L, R) is based on the target panning parameter (Θ) such that the left or right audio signal with a greater magnitude is provided with a greater weight.
  • EEE 3. The method according to EEE 1 or EEE 2, wherein the weight of the left and right audio signals (L, R) is complex valued weight, and wherein a phase of the complex valued weight is based on the target phase difference parameter (Φ).
  • EEE 4. The method according to EEE 3, wherein the complex valued weight is based on the target phase difference parameter (Φ) and the target panning parameter (Θ) such that the phase difference created by application of the complex weight is lesser for the left or right audio signal with a greater magnitude.
  • EEE 5. The method according to any of the preceding EEEs, further comprising: providing the stereo audio signal to a stereo source separation system (60), configured to output a filter for stereo source separation; and applying the filter to the target mid audio signal to form a processed target mid audio signal (M_sep).
  • EEE 6. The method according to any of the preceding EEEs, further comprising: providing the target mid audio signal (M) to a mono source separation system (30) configured to perform audio source separation on a mono audio signal and output a processed mono audio signal; and using the processed mono audio signal as a processed target mid audio signal (M_sep).
  • EEE 7. The method according to any of EEEs 5-6, further comprising reconstructing processed left and right audio signals (L′, R′) forming a processed stereo audio signal; wherein each of the processed left and right audio signals (L′, R′) is based on the processed target mid audio signal (M_sep) weighted with a respective left and right weighting factor, wherein the left and right weighting factor is based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ) for each time segment and frequency band.
  • EEE 8. The method according to any of the preceding EEEs, further comprising:
  • reconstructing an alternative left and right audio signal (L_cen, R_cen) forming an alternative stereo audio signal; and wherein each of the alternative left and right audio signal (L_cen, R_cen) is based on weighted contributions of at least the target mid audio signal, each contribution being based on a respective alternative mid weighting factor, wherein each mid weighting factor is based on at least one of an alternative panning parameter (Θ_cen) and an alternative phase difference parameter (Φ_cen), for each time segment and each frequency band.
  • EEE 9. The method according to EEE 8, further comprising: performing stereo signal processing on the alternative stereo audio signal forming a processed alternative stereo audio signal comprising a processed alternative left and right audio signal (L′_cen, R′_cen); and reconstructing a processed target mid audio signals (M″); wherein the processed target mid audio signal (M′) is based on a sum of the processed alternative left and right audio signals (L′_cen, R′_cen) weighted with weighting factors, wherein the weighting factors are based on at least one of the alternative panning parameter (Θ_cen) and the alternative phase difference parameter (Φ_cen) of each frequency band and time segment.
  • EEE 10. The method according to EEE 8 or 9, wherein the alternative panning parameter (Θ_cen) and/or phase difference parameter (Φ_cen) indicates center panning.
  • EEE 11. The method according to any of the preceding EEEs, further comprising: extracting (S31), for each time segment and frequency band, a partial side signal representation (311, 312), wherein the partial side signal representation (311, 312) is based on a weighted difference between the left and right audio signal (L, R), wherein a weight of each of the left and right audio signal (L, R) is based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ) of each frequency band and time segment; and forming (S41) a target side audio signal by combining each partial side signal representation for each frequency band and time segment.
  • EEE 12. The method according to EEE 11, further comprising: performing processing (S5, S51) of at least one of the target mid and target side audio signal (M, S) to form a processed target mid and target side audio signal (M′, S′); reconstructing (S6) a processed left and right audio signal (L′, R′) forming a processed stereo audio signal; wherein the processed left audio signal (L′) is based on a weighted sum of the processed target mid audio signal and target side audio signal (M′, S′), wherein the weights are based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ), and wherein the processed right audio signal (R′) is based on a weighted difference between the processed target mid audio signal and target side audio signal (M′, S′), wherein the weights are based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Θ).
  • EEE 13. The method according to EEE 12, wherein performing processing of at least one of the target mid audio signal and target side audio signal (M, S) comprises at least one of: attenuating the target side audio signal (S); applying a gain to the target mid audio signal (M); performing mono signal source separation on the target mid audio signal (M); and applying stereo source separation filter to the target mid audio signal (M).
  • EEE 14. The method according to any of the preceding EEEs, wherein obtaining at least one of: the target panning parameter (Θ) and the target phase difference parameter (Φ), comprises: obtaining at least one of a plurality of detected panning parameters (θ) and a plurality of detected phase difference parameters (φ); obtaining, for each detected panning parameter (θ) and detected phase difference parameter (φ) a detected magnitude parameter (U); and determining at least one of the target panning parameter (θ) and the target phase difference parameter (φ) by calculating an average of at least one of the plurality of detected panning parameters (θ) and the plurality of detected phase difference parameters (φ), said average being a weighted average weighted with the detected magnitude parameter (U).
  • EEE 15. The method according to EEE 14, further comprising: sorting each of said at least one of plurality of detected panning parameters (θ) and a plurality of detected phase difference parameters (φ) into a predetermined number of bins; wherein each bin is associated with at least one of a detected panning parameter value and a detected phase difference parameter value, and wherein said average of at least one of the plurality of detected panning parameters and the plurality of detected phase difference parameter (φ) is based on the detected panning parameter value and a detected phase difference parameter value.
  • EEE 16. An audio processing system configured to extract a target mid audio signal (M) from a stereo audio signal, the stereo audio signal comprising a left audio signal (L) and a right audio signal (R), said audio processing system comprising an extractor unit (12) configured to: obtain a plurality of consecutive time segments of the stereo audio signal, wherein each time segment comprises a representation of a portion of the stereo audio signal, obtain, for each frequency band of a plurality of frequency bands of each time segment of the stereo audio signal, at least one of: a target panning parameter (Θ), representing a distribution over the time segment of a magnitude ratio between the left and right audio signals (L, R) in the frequency band, and a target phase difference parameter (Φ), representing a distribution over the time segment of the phase difference between the left and right audio signals (L, R) in the frequency band, extract, for each time segment and each frequency band, a partial mid signal representation (211, 212), wherein the partial mid signal representation (211, 212) is based on a weighted sum of the left and right audio signal (L, R), wherein a weight of each of the left and right audio signals (L, R) is based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ) of each frequency band, and form the target mid audio signal (M) by combining the partial mid signal representations for each frequency band and time segment.
  • EEE 17. The audio processing system according to EEE 16, wherein the weight of the left and right audio signals (L, R) is based on the target panning parameter (Θ) such that the left or right audio signal with a greater magnitude is provided with a greater weight.
  • EEE 18. The audio processing system according to EEE 16 or 17, wherein the weight of the left and right audio signals (L, R) is complex valued weight, and wherein a phase of the complex valued weight is based on the target phase difference parameter (Θ).
  • EEE 19. The audio processing system according to EEE 18, wherein the complex valued weight is based on the target phase difference parameter (Φ) and the target panning parameter (Θ) such that the phase difference created by application of the complex weight is lesser for the left or right audio signal with a greater magnitude.
  • EEE 20. The audio processing system according to any of EEEs 16-19, further comprising: a stereo source separation system (60), configured to output a filter for stereo source separation, wherein the stereo audio signal is provided to the stereo source separation system (60) and the filter for stereo source separation is applied to the target mid audio signal to form a processed target mid audio signal (Msep).
  • EEE 21. The audio processing system according to any of EEEs 16-20, further comprising a mono source separation system (30) configured to perform audio source separation on a mono audio signal and output a processed mono audio signal, wherein the target mid audio signal (M) is provided to the mono source separation system (30) and the processed mono audio signal as a processed target mid audio signal (M_sep).
  • EEE 22. The audio processing system according to any of EEEs 20-21, further comprising a reconstructor unit (20) configured to reconstruct a processed left and right audio signals (L′, R′) forming a processed stereo audio signal, wherein each of the processed left and right audio signals (L′, R′) is based on the processed target mid audio signal (M_sep) weighted with a respective left and right weighting factor, wherein the left and right weighting factor is based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ) for each time segment and frequency band.
  • EEE 23. The audio processing system according to any of EEEs 16-22, further comprising a reconstructor unit (20) configured to reconstruct an alternative left and right audio signal (L_cen, R_cen) forming an alternative stereo audio signal, wherein each of the alternative left and right audio signal (L_cen, R_cen) is based on weighted contributions of at least the target mid audio signal, each contribution being based on a respective alternative mid weighting factor, wherein each mid weighting factor is based on at least one of an alternative panning parameter (Θ_cen) and an alternative phase difference parameter (Φ_cen), for each time segment and each frequency band.
  • EEE 24. The audio processing system according to any of EEEs 16-23, further comprising a stereo processor (50) configured to perform stereo signal processing on the alternative stereo audio signal forming a processed alternative stereo audio signal comprising a processed alternative left and right audio signal (L′_cen, R′_cen), and a second extractor unit configured to extract a processed target mid audio signal (M″), wherein the processed target mid audio signal (M′) is based on a sum of the processed alternative left and right audio signals (L′_cen, R′_cen) weighted with weighting factors, and wherein the weighting factors are based on at least one of the alternative panning parameter (Θ_cen) and the alternative phase difference parameter (Φ_cen) of each frequency band and time segment.
  • EEE 25. The audio processing system according to any of EEEs 23-24, wherein the alternative panning parameter (Θ_cen) and/or phase difference parameter (Φ_cen) indicates center panning.
  • EEE 26. The audio processing system according to any of EEEs 16-25, wherein the extractor unit 12 is further configured to extract, for each time segment and frequency band, a partial side signal representation (311, 312), wherein the partial side signal representation (311, 312) is based on a weighted difference between the left and right audio signal (L, R), wherein a weight of each of the left and right audio signal (L, R) is based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ) of each frequency band and time segment, and form a target side audio signal by combining each partial side signal representation for each frequency band and time segment.
  • EEE 27. The audio processing system according to any of EEEs 16-26, further comprising a mid/side processor (40) configured to process at least one of the target mid and target side audio signal (M, S) to form a processed target mid and target side audio signal (M′, S′), and a reconstructor unit (20), configured to reconstruct a processed left and right audio signal (L′, R′) forming a processed stereo audio signal; wherein the processed left audio signal (L′) is based on a weighted sum of the processed target mid audio signal and target side audio signal (M′, S′), wherein the weights are based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Φ), and wherein the processed right audio signal (R′) is based on a weighted difference between the processed target mid audio signal and target side audio signal (M′, S′), wherein the weights are based on at least one of the target panning parameter (Θ) and the target phase difference parameter (Θ).
  • EEE 28. The audio processing system according to EEE 27, wherein the processing wherein the processing performed by the mid/side processor (40) comprises at least one of: attenuation of the target side audio signal (S), application of a gain to the target mid audio signal (M), mono signal source separation performed on the target mid audio signal (M), and application of a stereo source separation filter to the target mid audio signal (M).
  • EEE 29. The audio processing system according to any of EEEs 16-28, further comprising a parameter extractor unit (14) configured to: obtain at least one of a plurality of detected panning parameters (θ) and a plurality of detected phase difference parameters (φ), obtain for each detected panning parameter (θ) and detected phase difference parameter (φ) a detected magnitude parameter (U), and determine at least one of the target panning parameter (θ) and the target phase difference parameter (φ) by calculating an average of at least one of the plurality of detected panning parameters (θ) and the plurality of detected phase difference parameters (Φ), said average being a weighted average weighted with the detected magnitude parameter (U).
  • EEE 30. The audio processing system according to EEE 29, wherein the parameter extractor unit (14) is further configured to: sort each of said at least one of plurality of detected panning parameters (θ) and a plurality of detected phase difference parameters (φ) into a predetermined number of bins, wherein each bin is associated with at least one of a detected panning parameter value and a detected phase difference parameter value, and wherein said average of at least one of the plurality of detected panning parameters and the plurality of detected phase difference parameter (φ) is based on the detected panning parameter value and a detected phase difference parameter value.
  • EEE 31. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any of EEEs 1-15.
  • EEE 32. The method according to EEE 2 or any one of EEEs 3-15 when dependent on EEE 2, or the audio processing device according to EEE 17 or any one of EEEs 18-30 when dependent on EEE 17, wherein a magnitude of each of the left and right audio signal is determined and in response to determining that the magnitude of the left audio signal is larger than the magnitude of the right audio signal, determining the weight of the left audio signal to be greater than the weight of the right audio signal, and in response to determining that the magnitude of the right audio signal is larger than the magnitude of the left audio signal, determining the weight of the right audio signal to be greater than the weight of the left audio signal.
  • EEE 33. The method according to EEE 4 or any one of EEEs 5-15 when dependent on EEE 4, or the audio processing device according to EEE 19 or any one of EEEs 20-30 when dependent on EEE 19, wherein a magnitude of each of the left and right audio signal is determined, and wherein and in response to determining that the magnitude of the left audio signal is larger than the magnitude of the right audio signal, determining the weight of the left audio signal to cause a first phase shift when applied to the left audio signal determining the weight of the right audio signal to cause a second phase shift when applied to the right audio signal, and in response to determining that the magnitude of the right audio signal is larger than the magnitude of the left audio signal, determining the weight of the right audio signal to cause a first phase shift when applied to the right audio signal determining the weight of the left audio signal to cause a second phase shift when applied to the left audio signal, and wherein the first phase shift is less than the second phase shift.
  • EEE. 34. The method according to EEE 8 or any one of EEEs 9-15 when dependent on EEE 8, or the audio processing device according to EEE 23 or any one of EEEs 24-30 when dependent on EEE 23, wherein the alternative left and right audio signals are second left and right audio signals, respectively, and/or where the alternative stereo signal is a second stereo signal.
  • EEE. 35. The method according to EEE 9 or 10 or any one of EEEs 11-15 when dependent on EEE 9 or 10, or, wherein performing stereo signal processing on the alternative stereo audio signal (L_cen, R_cen) comprises applying source separation on the alternative stereo audio signal (L_cen, R_cen) to output the processed alternative left and right audio signals (L′_cen, R′_cen).

Claims

1-16. (canceled)

17. A computer-implemented method for extracting a target mid audio signal from a stereo audio signal, the stereo audio signal comprising a left audio signal and a right audio signal, said method comprising: obtaining a plurality of consecutive time segments of the stereo audio signal, wherein each time segment comprises a representation of a portion of the stereo audio signal;obtaining, for each frequency band of a plurality of frequency bands of each time segment of the stereo audio signal, at least one of:a target panning parameter, representing a distribution over the time segment of a magnitude ratio between the left and right audio signals in the frequency band, wherein the target panning parameter represents a statistical feature of multiple samples within the time segment, the statistical feature being the median, mean, mode, numbered percentile, maximum or minimum panning parameter, anda target phase difference parameter representing a distribution over the time segment of the phase difference between the left and right audio signals of the stereo audio signal, wherein the target phase difference parameter represents a statistical feature of multiple samples within the time segment, the statistical feature being the median, mean, mode, numbered percentile, maximum or minimum phase difference parameter;extracting, for each time segment and each frequency band, a partial mid signal representation, wherein the partial mid signal representation is based on a weighted sum of the left and right audio signal, wherein a weight of each of the left and right audio signals is based on at least one of the target panning parameter and the target phase difference parameter of each frequency band; andforming the target mid audio signal by combining the partial mid signal representations for each frequency band and time segment.

18. The method according to claim 17, wherein the weight of the left and right audio signals is based on the target panning parameter such that the left or right audio signal with a greater magnitude is provided with a greater weight.

19. The method according to claim 17, wherein the weight of the left and right audio signals is complex valued weight, and wherein a phase of the complex valued weight is based on the target phase difference parameter.

20. The method according to claim 19, wherein the complex valued weight is based on the target phase difference parameter and the target panning parameter such that the phase difference created by application of the complex weight is lesser for the left or right audio signal with a greater magnitude.

21. The method according to claim 17, further comprising: providing the stereo audio signal to a stereo source separation system, configured to output a filter for stereo source separation; andapplying the filter to the target mid audio signal to form a processed target mid audio signal.

22. The method according to claim 17, further comprising: providing the target mid audio signal to a mono source separation system configured to perform audio source separation on a mono audio signal and output a processed mono audio signal; andusing the processed mono audio signal as a processed target mid audio signal.

23. The method according to claim 21, further comprising reconstructing processed left and right audio signals forming a processed stereo audio signal;wherein each of the processed left and right audio signals is based on the processed target mid audio signal weighted with a respective left and right weighting factor, wherein the left and right weighting factor is based on at least one of the target panning parameter and the target phase difference parameter for each time segment and frequency band.

24. The method according to claim 17, further comprising: reconstructing an alternative left and right audio signal forming an alternative stereo audio signal; andwherein each of the alternative left and right audio signal is based on weighted contributions of at least the target mid audio signal, each contribution being based on a respective alternative mid weighting factor, wherein each mid weighting factor is based on at least one of an alternative panning parameter and an alternative phase difference parameter, for each time segment and each frequency band.

25. The method according to claim 24, further comprising: performing stereo signal processing on the alternative stereo audio signal forming a processed alternative stereo audio signal comprising a processed alternative left and right audio signal; andreconstructing a processed target mid audio signals;wherein the processed target mid audio signal is based on a sum of the processed alternative left and right audio signals weighted with weighting factors,wherein the weighting factors are based on at least one of the alternative panning parameter and the alternative phase difference parameter of each frequency band and time segment.

26. The method according to claim 24, wherein the alternative panning parameter and/or phase difference parameter indicates center panning.

27. The method according to claim 25, wherein performing stereo signal processing on the alternative stereo audio signal comprises applying source separation on the alternative stereo audio signal (to output the processed alternative left and right audio signals (L′cen, R′cen).

28. The method according to claim 17, further comprising: extracting, for each time segment and frequency band, a partial side signal representation, wherein the partial side signal representation is based on a weighted difference between the left and right audio signal, wherein a weight of each of the left and right audio signal is based on at least one of the target panning parameter and the target phase difference parameter of each frequency band and time segment; andforming a target side audio signal by combining each partial side signal representation for each frequency band and time segment.

29. The method according to claim 28, further comprising: performing processing of at least one of the target mid and target side audio signal to form a processed target mid and target side audio signal;reconstructing a processed left and right audio signal forming a processed stereo audio signal;wherein the processed left audio signal is based on a weighted sum of the processed target mid audio signal and target side audio signal, wherein the weights are based on at least one of the target panning parameter and the target phase difference parameter, andwherein the processed right audio signal is based on a weighted difference between the processed target mid audio signal and target side audio signal, wherein the weights are based on at least one of the target panning parameter and the target phase difference parameter.

30. The method according to claim 29, wherein performing processing of at least one of the target mid audio signal and target side audio signal comprises at least one of: attenuating the target side audio signal;applying a gain to the target mid audio signal;performing mono signal source separation on the target mid audio signal; andapplying stereo source separation filter to the target mid audio signal.

31. The method according to claim 17, wherein obtaining at least one of: the target panning parameter and the target phase difference parameter, comprises: obtaining at least one of a plurality of detected panning parameters and a plurality of detected phase difference parameters;obtaining, for each detected panning parameter and detected phase difference parameter a detected magnitude parameter; anddetermining at least one of the target panning parameter and the target phase difference parameter by calculating an average of at least one of the plurality of detected panning parameters and the plurality of detected phase difference parameters, said average being a weighted average weighted with the detected magnitude parameter.

32. The method according to claim 31, further comprising: sorting each of said at least one of plurality of detected panning parameters and a plurality of detected phase difference parameters into a predetermined number of bins;wherein each bin is associated with at least one of a detected panning parameter value and a detected phase difference parameter value, and wherein said average of at least one of the plurality of detected panning parameters and the plurality of detected phase difference parameter is based on the detected panning parameter value and a detected phase difference parameter value.