Patent Application
20230395079
An aim to process an input signal which is a mixture of multiple sources and to create an output mixture in which the relative level of the sources is modified. One example is to make the speech in a movie audio track clearer, louder, and more intelligible.
The proposed method may apply source separation to estimate the sources and remix these estimates by applying automatically generated time-varying, signal-adaptive gains. The remixing aims to fulfill a time-varying criterion concerning the separated sources and their relationship in the output mix. The output mixture has to be smooth and esthetically pleasing. For this purpose, a temporal context is taken into consideration during the generation of the remixing gains so to avoid abrupt and unaesthetic changes.
An envisioned application is to enable object-based audio personalization, e.g., based on MPEG-H Audio [1, 2]. Based on MPEG Unified Speech and Audio Coding, the MPEG-H Audio standard offers many extensions for use in the context of immersive 3D audio, such as coding and rendering of multi-channel and object signals, transmission of object metadata, the compressed transmission of (speaker layout agnostic) object positions and trajectories, and it allows for personalization and user interactivity on the decoder side that is enabled and controlled by object metadata. The underlying main ideas of the new codec are to provide suitable means for an immersive experience, for universal delivery, and for personal interactivity.
Personal interactivity is a particularly demanded use case, for example, for personalizing the audio track in movies and TV programs. In fact, it has been shown that the balance between the speech and the background signals is extremely personal [3, 4]. However, often, only a mono, stereo, or multi-channel mix of all sources is available instead of sub-mixes of the sources. Ways to automatically generate alternative mixes with different relative levels starting from the available mix are desired. The resulting mix has to be of high sound quality and esthetically pleasing. The system proposed in this report and shown in
Alternative application scenarios might involve traditional broadcasting and streaming services. In these, full personalization is usually not available (or needed), but an alternative audio track (generated as described in this report) can be generated offline and offered by the broadcasting/streaming provider. In a further envisioned application, the alternative audio track could be generated directly by the end-device. In other words, all modules are placed in the end-device.
Typically, constant gains are applied on the estimated target source and/or on the residual sources, e.g., in order to modify the SNR (signal to noise ratio) during the remixing. The SNR may be the ratio of the target signal to the at least one residual signal. These constant (over time) gains can be set by the final user, or they can be pre-defined and fixed, or they aim to optimize a global criterion. However constant gains and a global criterion have several problems:
An alternative to constant gains, well-known among audio engineers, could be side-chain ducking, i.e., controlling the time-varying level of one (ducked) signal based on the absolute level of another ducking signal. The ducked and the ducking signals could be the outputs from the separation. This approach is also suboptimal because the amount of ducking is only based on the level of one signal and not on properties relative to all signals involved. Moreover, side-chain ducking is not robust against the unavoidable errors (e.g., leaking components) in the source separation module. Furthermore, a traditional side-chain ducking applies a stronger attenuation on the ducked signal, when the level of the ducking signal is higher. This may have a benefit in keeping the overall level of the resulting mixture approximately constant, but is not useful for, e.g., guaranteeing a level of intelligibility of a speech signal when mixed on top of a background signal. For the intelligibility, the attenuation needs to be stronger when the ducking signal is softer, so that it becomes better audible in the mixture.
According to an embodiment, a system for processing audio signals may have: a source separation block configured to estimate, from an input signal evolving in time along a discrete succession of time instants or time slots, a target signal and at least one residual signal to be subsequently remixed according to at least one remixing gain variable along the discrete succession; a control block configured to determine, for a determined current time instant or time slot, a first, relative metrics on the target signal, in the determined current time instant or time slot, wherein the first, relative metrics compares a level of the target signal with a level of the at least one residual signal or the input signal, in the determined current time instant or time slot; and a temporal context block configured to determine temporal context information based on a second, relative metrics in at least one future and/or past time instant or time slot, the second, relative metrics comparing a level of the target signal with a level of the input signal or the at least one residual signal, in the at least one future and/or past time instant or time slot, the at least one future time instant or time slot being, in the discrete succession, after the determined current time instant or time slot, and the past time instant or time slot being, in the discrete succession, before the determined current time instant or time slot, wherein the control block is configured to generate at least one remixing gain associated to the determined current time instant or time slot based on: the first, relative metrics in the determined current time instant or time slot; and the temporal context information.
According to another embodiment, a method for processing audio signals may have: a source separation step obtaining, from an input signal evolving in time along a discrete succession of time instants or time slots, a target signal and at least one residual signal to be subsequently remixed according to at least one remixing gain variable along the discrete succession; a control step determining, for a determined current time instant or time slot, a first, relative metrics in the determined current time instant or time slot, wherein the first, relative metrics compares a level of the target signal with a level of the input signal, or the at least one residual signal, in the determined current time instant or time slot; and a temporal context step determining temporal context information based on a second, relative metrics in at least one future and/or past time instant or time slot, the second, relative metrics comparing a level of the target signal with a level of the input signal or the at least one residual signal in the at least one future and/or past time instant or time slot, the at least one future time instant or time slot being, in the discrete succession, after the determined current time instant or time slot, and the past time instant or time slot being, in the discrete succession, before the determined current time instant or time slot, the method including generating at least one remixing gain based on: the first, relative metrics in the determined current time instant or time slot; and the temporal context information.
Another embodiment may have a non-transitory digital storage medium having a computer program stored thereon to perform the inventive method when said computer program is run by a computer.
According to an aspect, there is provided a system for processing audio signals, comprising:
The system may be such that the temporal context information includes at least one metrics on the target signal, or a processed version thereof, in the at least one determined future and/or past time instant or time slot.
The system may be such that the temporal context information includes information on at least one previously obtained remixing gain.
The system may be such that the temporal context information includes information on at least one rough remixing gain obtained for the at least one determined future and/or past time instant or time slot.
The system may be such that there are defined at least one first remixing criterion and one second remixing criterion for generating at least one rough remixing gain. At least one criterion condition may perform a discrimination between using the first remixing criterion and using the second remixing criterion at each time instant or time slot, so that, based on the at least one criterion condition, each time instant or time slot is associated to one of the at least one first remixing criterion and second remixing criterion. The at least one criterion condition may be a condition on the at least one metrics on at least the target signal, or a processed version thereof, at the determined current time instant or time slot, or on information obtained from the at least one metrics on the at least the target signal or a processed version thereof, so that the determined current time instant or time slot is associated to one of the at least one first remixing criterion and one second remixing criterion based on the metrics on the target signal, or a processed version thereof, in the determined current time instant or time slot, wherein the system is further configured to obtain the at least one remixing gain for the determined current time slot or time instant by considering temporal context information so as to deviate, from the at least one rough remixing gain, based on a deviation obtained from the temporal context information.
The at least one criterion condition may include a condition (e.g. clearance condition) on e.g. at least one relative metrics (e.g. SNRi) on the current determined time instant or time slot being compared to a threshold (e.g. target clearance), so that the first remixing criterion (e.g. implying high rough gain and/or unitary gain in some examples) is assigned to the current determined time instant or time slot when the at least one relative metrics is over the threshold (e.g. target clearance), and the second criterion (e.g. implying lower rough gain than the first criterion and/or a rough gain which permits to achieve the target clearance between the target signal and the background signal) is assigned to the current determined time instant or time slot when the at least one relative metrics is below the threshold (e.g. target clearance).
The at least one criterion condition may include a condition (e.g. gating condition, also known as intensity condition) on e.g. at least one absolute metrics (e.g. intensity) on the current determined time instant or time slot being compared to a threshold (e.g. intensity threshold), so that the first remixing criterion (e.g. implying high rough gain or unitary gain in some examples) is assigned to the current determined time instant or time slot when the at least one absolute metrics is below the threshold (e.g. intensity threshold), and the second remixing criterion (e.g. implying lower rough gain than the second criterion and/or a rough gain which permits to achieve the target clearance between the target signal and the background signal) is assigned to the current determined time instant or time slot when the at least one absolute metrics is over the threshold (e.g. intensity threshold).
According to an aspect, the criterion condition may be based on an “OR” condition of multiple conditions, e.g. an “OR” condition of the intensity condition (e.g. gating condition) and of the clearance condition, so that the first criterion is assigned to the current determined time instant or time slot when at least one of the intensity condition (intensity lower than the intensity threshold) and the clearance condition (intensity higher than the target clearance) is verified. Otherwise (i.e. if neither of the intensity condition and the clearance condition is verified), the second criterion is assigned to the current determined time instant or time slot.
For example, we may have (from the so-called formula (5) below):
with C being the target clearance and G being the intensity threshold, g(t) being the rough gain, t being the current determined time instant or time slot, SNRin(t) being the signal to noise ratio in input (or more in general a relative metrics) between the target signal (or processed version thereof) and the residual signal (background signal) or processed version thereof, or the input signal (or processed version thereof), and is being the intensity (or more in general an absolute metrics) of the target signal (or processed version thereof).
The system may be configured to deviate from the at least one rough remixing gain by correcting the at least one rough remixing gain by an amount associated to a previously obtained at least one remixing gain for a time instant or time slot preceding the determined current time instant or time slot.
The system may be configured to deviate from the at least one rough remixing gain by correcting the at least one rough remixing gain for a gain amount associated to a previously obtained at least one remixing gain for a time instant or time slot preceding the determined current time instant or time slot subjected to the fulfilment of a deviation condition based on the temporal context information, wherein the temporal context information includes information on rough remixing gains already obtained for time instants or time slots following the determined time instant or time slot. The deviation condition may be fulfilled, in some aspects, when a predetermined number of rough remixing gains already obtained for time instants or time slots following the determined time instant or time slot are associated to a remixing criterion which is different from the remixing criterion of the current determined time instant or time slot or of the time instant or time slot preceding the current determined time instant or time slot. If the deviation condition is not fulfilled, the at least one remixing gain for the determined current time instant or time slot may be maintained the same of the at least one remixing gain for the current determined time instant or time slot or the time instant or time slot preceding the determined current time instant or time slot.
The system may be further configured to correct the at least one rough remixing gain through a linear combination of the at least one rough remixing gain (g(t)) and the previously obtained at least one remixing gain (gsmooth(t)).
The system may be such that the linear combination is based on a predefined parameter τ comprised between 0 and 1, wherein the first predefined parameter (e.g. τ) scales the at least one rough remixing gain and a second predefined parameter (e.g. 1−τ) between 0 and 1 scales the previously obtained at least one remixing gain, wherein the sum between the first predefined parameter and the second predefined parameter is 1.
The system may be such that the parameter T is a first predefined parameter for a deviation from the first remixing criterion to the second remixing criterion and a second predefined parameter, different from the first predefined parameter, for a deviation from the second remixing criterion to the first remixing criterion.
The system may be such that the at least one criterion condition includes a condition on at least the relative metrics at the determined current time instant or time slot, so that if the relative metrics between the target signal, or a processed version thereof, and the at least one residual signal, or a processed version thereof, or input signal, or a processed version thereof, at the determined current time instant or time slot is greater than a predetermined relative threshold, then the determined current time slot or time instant is associated to the first remixing criterion; and if the relative metrics between the target signal, or a processed version thereof, and the at least one residual signal at the determined current time instant or time slot is smaller than the predetermined relative threshold, then the determined current time slot or time instant is associated to the second remixing criterion. The first remixing criterion may adopt a first ratio between: the rough remixing gain associated to the target signal, or a processed version thereof; and the rough remixing gain associated to the input signal, or a processed version thereof, or the at least one residual signal, or a processed version thereof; the second remixing criterion adopts a second ratio between: the rough remixing gain associated to the target signal, or a processed version thereof; the rough remixing gain associated to the input signal, or a processed version thereof, or the at least one residual signal, or a processed version thereof. The second ratio may be higher than the first ratio. The deviation may include gradually moving the ratio between the remixing gain associated to the target signal, or a processed version thereof, and the remixing gain associated to the at least one residual signal, or a processed version thereof, or the input signal or a processed version thereof, from the first ratio to the second ratio, or vice versa.
The system may be such that the at least one criterion condition includes a condition on at least the absolute metrics at the determined current time instant or time slot, so that if the absolute metrics on the target signal, or a processed version thereof, at the determined current time instant or time slot is smaller than a predetermined absolute threshold, then the determined current time slot or time instant is associated to the first remixing criterion; and if the absolute metrics on the target signal, or a processed version thereof, at the determined current time instant or time slot is greater than the predetermined absolute threshold, then the determined current time slot or time instant is associated to the second remixing criterion. The first remixing criterion may adopt a first ratio between: the rough remixing gain associated to the target signal, or a processed version thereof; and the rough remixing gain associated to the input signal, or a processed version thereof, or the at least one residual signal or a processed version thereof. The second remixing criterion adopts a second ratio between: the rough remixing gain associated to the target signal, or a processed version thereof; the rough remixing gain associated to the input signal, or a processed version thereof, or the at least one residual signal, or a processed version thereof. The second ratio may be higher than the first ratio. The deviation includes gradually moving the ratio between the remixing gain associated to the target signal, or a processed version thereof, and the remixing gain associated to the at least one residual signal, or a processed version thereof, or the input signal or a processed version thereof, from the first ratio to the second ratio, or vice versa.
The system may be such that the deviation condition is fulfilled when a predetermined number of rough remixing gains already obtained for time instants or time slots in a time window following the determined time instant or time slot is associated to a remixing criterion which is different from the remixing criterion associated to the current determined time instant or time slot or to the time instant or time slot preceding the current determined time instant or time slot. If the deviation condition is not fulfilled, the at least one remixing gain for the determined current time instant or time slot is maintained the same of the at least one remixing gain for a time instant or time slot preceding the determined current time instant or time slot.
The system may be such that the deviation condition is fulfilled when a predetermined number of rough remixing gains already obtained for time instants or time slots in a time window following the determined time instant or time slot is associated to a remixing criterion which is different from the remixing criterion associated to a time instant or time slot preceding the time instants or time slots in the time window following the determined time instant or time slot, such as one of the two time instants or time slots immediately preceding the time instants or time slots in the time window following the determined time instant or time slot (e.g. one of the current determined time instant or time slot and the time instant or time slot immediately preceding the current determined time instant or time slot). If the deviation condition is not fulfilled, the at least one remixing gain for the determined current time instant or time slot is maintained the same of the at least one remixing gain for a time instant or time slot preceding the determined current time instant or time slot.
The system may be such that the deviation condition is not fulfilled at least when the rough remixing gain associated to the determined current time instant or slot is associated to a remixing criterion different from the remixing criterion associated to the current determined time instant or time slot or time instant or time slot preceding the current determined time instant or time slot. In that case the at least one remixing gain for the determined current time instant or time slot is maintained the same of the at least one remixing gain for the current determined time instant or time slot or the time instant or time slot preceding the determined current time instant or time slot.
The system may be such that the deviation condition is not fulfilled at least when the rough remixing gain associated to the determined current time instant or slot is associated to a remixing criterion different from the remixing criterion associated to a time instant or time slot preceding the time instants or time slots in the time window following the determined time instant or time slot, such as one of the two time instants or time slots immediately preceding the time instants or time slots in the time window following the determined time instant or time slot (e.g. one of the current determined time instant or time slot and the time instant or time slot preceding the current determined time instant or time slot). In that case the at least one remixing gain for the determined current time instant or time slot is maintained the same of the at least one remixing gain for time instant or time slot preceding the time instants or time slots in the time window following the determined time instant or time slot, such as one of the two time instants or time slots immediately preceding the time instants or time slots in the time window following the determined time instant or time slot (i.e. one of the current determined time instant or time slot and the time instant or time slot preceding the current determined time instant or time slot).
The system may be such that the second remixing criterion is dominant over the first remixing criterion, and the deviation condition is evaluated when the time instant or time slot preceding the time instants or time slots in the time window following the determined time instant or time slot (such as one of the two time instants or time slots immediately preceding the time instants or time slots in the time window following the determined time instant or time slot, e.g. one of the current determined time instant or time slot and the time instant or time slot preceding the current determined time instant or time slot) is associated to the second remixing criterion, while the evaluation of the deviation condition is deactivated when the time instant or time slot preceding the time instants or time slots in the time window following the determined time instant or time slot (such as one of the two time instants or time slots immediately preceding the time instants or time slots in the time window following the determined time instant or time slot, e.g. one of the current determined time instant or time slot and the time instant or time slot preceding the current determined time instant or time slot) is associated to the first remixing criterion.
The system may be such that the second remixing criterion is dominant over the first remixing criterion, and the deviation condition is evaluated when the current determined time instant or time slot or the time instant or time slot preceding the current determined time instant or time slot is associated to the second remixing criterion, while the evaluation of the deviation condition is deactivated when the current determined time instant or time slot or the time instant or time slot preceding the current determined time instant or time slot is associated to the first remixing criterion.
The system may be configured to distinguish, based on the at least one metrics on the target signal or a processed version thereof, in the at least one determined current time instant, and on the temporal context information, between transitory time interval and non-transitory time intervals, so as to in the non-transitory time interval, assign the value of the at least one rough remixing gain according to the current remixing criterion to the at least one remixing gain; and to deviate from the at least one rough remixing gain according to the current remixing criterion in the transitory time intervals.
The system may be configured to associate, to the target signal or a processed version thereof, an activity information for each time instant or time slot which acknowledges whether, for each time instant or time slot, target signal, or the processed version thereof, is active or non-active based on the at least one metrics in each time instant or time slot, wherein the at least one criterion condition keeps into account the activity information.
The system may be such that the at least one future and/or past time instant or time slot is in a time window of predetermined time length.
The system may be such that the activity information is active for time instants or time slots for which the at least one absolute metrics, associated to a level or loudness of the target signal, or a processed version thereof, as being greater than an absolute predefined threshold and/or at least one relative metrics, comparing the target signal, or a processed version thereof, with the at least one residual signal, or a processed version thereof, or input signal, or a processed version thereof, is greater than a relative predefined threshold.
The system may be such that the activity information is additionally active for time instants or time slots within a time window in which the time instants or time slots have the at least one absolute metrics, associated to a level or loudness of the target signal, or a processed version thereof, smaller than the absolute predefined threshold and/or the at least one relative metrics, comparing the target signal, or a processed version thereof, with the at least one residual signal, or a processed version thereof, or input signal, or a processed version thereof, is smaller than the relative predefined threshold, but the time window has length smaller than a predetermined time threshold.
The system may be such that the activity information is negative for time instants or time slots within a time window in which the time instants or time slots have the at least one absolute metrics, associated to a level or loudness of the target signal, or a processed version thereof, smaller than the absolute predefined threshold and/or the at least one relative metrics, comparing the target signal, or a processed version thereof, with the at least one residual signal, or a processed version thereof, or input signal, or a processed version thereof, is smaller than the relative predefined threshold, and the time window has length greater than the predetermined time threshold.
The system may be configured to define the at least one gain for a plurality of consecutive time instants or time samples to gradually deviating from a first remixing criterion towards a second remixing criterion.
The system may be configured to perform, for the determined current time instant or time slot, a time averaging on a plurality of time instants or time slots which precede and/or follow the determined time instant, so as to obtain an average of the at least one metrics, e.g. along the plurality of time instants or time slots.
The system may be configured to reassign the same value of the target signal or a processed version thereof, and/or the input signal, or a processed version thereof, or the at least one residual signal, or a processed version thereof, as the obtained average.
The system may be configured to shift the at least one gain as obtained for each time instant or time step of the discrete succession of time instants or time slots by a predetermined number of time instants or time steps towards the past.
The system may further include a remixing block configured to apply, for the determined current time instant or time slot, the at least one gain and the at least one residual signal.
The system may be such that the signals (e.g. at least one of target signal, input signal, residual signal, etc.) is in the time domain.
The input signal, the target signal and the at least one residual signal may be in the frequency domain ins some aspects.
The at least one remixing gain may include different remixing gains for different frequency bands in some aspects.
The system may be such that the at least one metrics in the determined current time instant or time slot and the at least one metrics in the at least one determined future and/or past time instant or time slot is subdivided onto metrics for different frequency bands, so as to obtain the different remixing gains for different frequency bands.
The system may be such that the at least one metrics, for the determined current time instant or time slot and for the at least one future and/or past time instant or time slot, is weighted according to weighting coefficients which vary according to the frequency.
The system may further comprise a remixing block providing a remixed output signal in which the target signal, or a processed version thereof, and the at least one residual signal, or a processed version thereof, are mixed together according to the at least one gain.
The system may be configured to encode a bitstream encoding the target signal, or a processed version thereof, and the at least one residual signal, or a processed version thereof, or input signal, or a processed version thereof, and the at least one gain. The system may be configured to transmit the encoded bitstream.
The system may be such that
The system may be such that the at least one metrics on the target signal, or a processed version thereof, in the determined current time instant or time slot further includes:
The system may be such that the at least one metrics on the target signal, or a processed version thereof, in the determined current time instant or time slot further includes:
In examples, the at least one gain may be obtained as (see also below in formula (12))
where chold(t) is a variable indicating the length of the still remaining time to keep the current gain value and it can be chold(t)=0 if gsmooth(t−1)>g(t), otherwise chold(t)=max{chold(t−1)−1, kin(t)}, where kmin(t) indicates the location of the minimum gain value within a window of tholdahead future values if this value is smaller than the current smoothed value, e.g., kmin(t)=argmin{g(t:t+tholdahead)} if min{g(t:t+tholdahead)}<gsmooth(t−1), otherwise kmin(t)=0.
According to an aspect, there is provided a method for processing audio signals, comprising:
According to an aspect, there is a provided non-transitory storage unit storing instructions which, when executed by a processor, cause the processor to perform the method above.
According to an aspect, there is provided a system for processing audio signals, comprising at least one processor to:
Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:
The modules in the figure can be distributed in different devices, i.e., signal encoding, transmission, and decoding can take place before or after the remixing module.
For the following explanation we can categorize all signal components in an input mixture x(t) such that they belong to one of two source signals: a target source signal s(t) (e.g., the speech recordings of all speakers in a movie soundtrack or all lead instruments in a musical recording) and a background signal b(t) comprising all residual audio sources not belonging to the target source:
x(t)=s(t)+b(t). (1)
Source separation of audio signals aims to estimate s(t), given the mixture signal x(t) (input signal 102). The output of the separation is an estimate of the target source ŝ(t). Optionally more secondary sources can be estimated and output by the source separation module, e.g., an estimate of the residual sources {circumflex over (b)}(t). It has to be noted that there are separation systems where ŝ(t) and {circumflex over (b)}(t) do not sum up to x(t), e.g., [5], but an estimate for {circumflex over (b)}(t) can also be obtained as {circumflex over (b)}(t)=x(t)−ŝ(t). In examples below, even if either ŝ(t) (or, respectively, x(t)) is processed, it is also possible to obtain an estimation of x(t)(or, respectively, ŝ(t)), simply by adopting the formula {circumflex over (b)}(t)=x(t)−ŝ(t) (x(t)=ŝ(t)−{circumflex over (b)}(t), respectively).
A post-filtering can be applied to ŝ(t) and {circumflex over (b)}(t), e.g., an equalizer for enhancing and/or attenuating certain frequency regions or a post-processing for removing musical noise.
In many application scenarios, the estimated source signals s(t) and b(t) are not intended to be listened to separately, but they are remixed with a partial modification of the relative levels [2, 6]. The notion of Signal-to-Noise Ratio (SNR) can be used here, referring the level difference between s(t) and b(t) or their estimates.
There are many solutions for source separation, e.g., [2, 7, 8, 5] and references therein. The solutions may rely on hand-designed audio signal processing algorithms, e.g., [2], also referred to as “classical signal processing”, or the solutions may be based on deep learning, see e.g., [8, 5]. The technique proposed in this report is not limited to any specific source separation system. The estimates of the sources are in real world likely not perfect. Various imperfections, such as cross-leaking components, artifacts, distortions, and colorations can be introduced by the source separation. It is important to consider this fact while remixing.
A control block 120 may be provided, which may have, in input, the target signal 114 and, for example, either the input signal 102 and/or the one or more residual signals 112 (the input signal 102 or the target signal 114 is also called “signal 302” or “first signal 302”). (
On the basis of the temporal context information 132, the control block 120 may appropriately define, time instant by time instant or time slot by time slot, the at least one remixing gain (e.g. remixing gains) 125 to be provided to the remixing block 150.
Accordingly, the obtained output signal (output mix) 104 will be remixed by keeping into consideration not only the target signal 114 at a particular time instant or time slot, but also on the target signal in the temporal context (e.g., future or past time instants or slots).
The input signal 102 and the separated signals 114, 112 (302), and/or the processed versions of those signals, are signals evolving in time along a discrete succession of time instants or time slots. Each time instant may be, for example, associated to a particular sample (e.g., signal in the time domain), e.g. present in the input signal (e.g. ambience, effects, and music). Otherwise, time may be understood as being subdivided (e.g. partitioned) into a plurality of time slots, and each time slot may be associated to a signal description in the frequency domain (e.g., digital Fourier transforms DFT, short-time Fourier transform STFT, fast Fourier transform FFT, and so on). In the frequency domain, a plurality of values may be associated to the particular time slot, each value being, for example, a coefficient to be associated to a particular frequency. There is no particular difference in this case between whether the signal(s) is(are) in the time domain or in the frequency domain. Hence, most of the following explanations are common for both the time domain case and the frequency domain case.
As seen in particular in
The time instant or time slot 403 (subsequently indicated to as “future time instant or time slot 403”) happens to be, in the time evolution (according to the discrete succession) of the target signal 114 and of the signal 302, subsequent to the current time instant or time slot 401. Accordingly, the time instant or time slot 403 is understood to be “in the future” with respect to the time instant or time slot 401. The values of the target signal 104 and of the signal 302 (112, 102), or processed versions thereof, are respectively indicated to with 1143 and 1123. It will be shown that it is possible to have different remixing gains 124 for different time slots or time instants. Moreover, it is possible to adapt the gain 124 associated to the time instant or time slot 401 as being obtained by also considering the value of the time instant or time slot 403.
The same may be performed for other time instants or time slots with respect to the time instant or time slot 401.
However, when the time instant 401 is processed, the values 1143 and 1123 at the future time instant or time slot 403 may be already known (e.g., stored in buffers). Below, where it is explained that the time instant or time slot 401 is the current time instant or time slot, it is meant that the current time instant or time slot 401 is currently processed, even though the future time instants or time slots (e.g., 403) are already known and/or some form of preprocessing is already performed to the future time instants or time slots, e.g., 403. Accordingly, the fact that some time instants or time slots are in the future shall not be understood as obtaining some features which are unknown, but it is more than the current time instant or time slot 401 is adapted to the future time instant or time slot 403, which is already known.
For example, it is possible to first obtain rough remixing gains (e.g. according to determined remixing criteria) for a plurality of time instants or time slots (e.g. for all the temporal evolution of the input signal), including any of 401, 403, 407, 417, 425, 406, 416, 426. After having obtained the rough remixing gains 125, it is subsequently possible to obtain the remixing gains 124, e.g. by performing deviations from the rough remixing gains 125 (in particular in transitory intervals or transition intervals) so as to obtain the remixing gains 124, e.g. by making use of temporal context information. This process may be performed iteratively, e.g. by first obtaining the remixing gains 124 for time instants in the past, then for a present time instant, and subsequently for the time instants in the future.
Moreover, it is intended that, after having processed the current time instant or time slot 401 (e.g. t1), subsequently the current time instant or time slot 401 is updated to another time instant or time slot (e.g. the time instant or time slot immediately subsequent t2=t1+1).
The at least one metrics which are obtained may include, for example, absolute metrics 4141 and/or 4143 associated to absolute magnitudes (e.g., loudness, level, power, energy, etc., of the particular signal 114 or 302) at the current time instant or time slot 401 or 403, as can be obtained, for example, from the value 1141 and/or 1143.
The at least one metrics which are obtained may include, for example, relative metrics 4145 and 4146. The relative metrics 4145 and/or 4146 may be the at least one metrics. For example, a relative metrics 4145 may be obtained by comparing, for example, an absolute metrics of the target signal 114 (e.g., as obtained from value 1141) with an absolute metrics of signal 302 (112, 102) at the current time instant slot 401 (e.g., as obtained from value 1121). Another relative metrics 4146 may take into account values 1143 and 1123 of the target signal 114 and the signal 302 (112 or 102) in the at least one future and/or past time instant or time slot 403. The metrics 4145 and 4146 are shown as being obtained at comparing blocks 425′ and 426′, respectively. An example of relative metrics 4145, 4146 the (possibly frequency-weighted) may be the relative intensity of the signals, e.g., SNR(t) (also indicated as SNRin(t) in formulas (5) and (6)), which may imply, for example, a ratio between absolute metrics such as those above. Multiple relative metrics may form a composite relative metrics. A metrics may imply, for example, a norm on the instant value of the signal. For example, a 1-norm, a 2-norm, etc. may be used. The metrics may be a norm, such as 1-norm, a 2-norm, etc. A norm may provide a non-negative real number which keeps into consideration the channels of the signal (e.g., the sum of their absolute values, the square root of the sum of their squared values, etc.). Further, multiple metrics (absolute metrics, relative metrics, or both) may be combined with each other to obtain a metrics which is a composite metrics (and partially relative metrics and partially absolute metrics).
An example of absolute metrics 4141, 4143 is the absolute intensity of the signals, possibly frequency-weighted, e.g. absolute metrics such as the intensity of the target signal 114 and/or the intensity of the signal 302 (e.g., 102, 112), respectively. Another example of absolute metrics 4141, 4143 may be an estimate of the perceived time-varying loudness and/or loudness difference. Another example of absolute metrics is a time-dependent quality or intelligibility metric or a speech activity probability. Another example of absolute metrics is a combination of these or other time-dependent features of the signals (multiple absolute metrics may form a composite absolute metrics).
Particular functions may be obtained with the present examples. For example, it is possible to apply the most appropriate remixing gains at each time instant or time slot (401, 403, etc.) and, for example, smoothing some gains (e.g., when transitioning from a first remixing gain to a second remixing gain, as will be explained below).
The generation of the at least one remixing gain 124 may be subjected to the definition of one or more remixing criteria. A remixing criterion may be, for example, a criterion for obtaining a particular goal (e.g., attenuating a background signal or boosting a particular target signal). The choice of a particular criterion may generally be associated to the metrics 4141 and/or 4145 (or respectively 4143 and 4146) in a particular time slot 401 (or respectively 403). A remixing criterion may therefore be associated to the value of a particular time instant or time slot 401 or 403. It may be seen that, in some cases, the current time instant or time slot 401 and the past and/or future time instant or time slot 403 are two time instants or time slots for which different remixing criteria are chosen (e.g. due to different results of an activity detection operation). It may be that, for the determined current time instant or time slot 401, the control block 120 chooses not to completely follow the remixing criterion as would be defined based on the metrics 4141 and 4145 of the target signal 114 at the current time instant or time slot 401: the control block 120 may therefore keep into account the temporal context 132. For example, while different remixing criteria may be defined for the current time instant or time slots 401 and 403 on the basis of the metrics 4141 and 4145 associated to the same time instant or time slot, the remixing criteria can also be not completely respected, by virtue of using the temporal context 132 and in particular, the metrics 4146 and/or 4143 associated to future and/or past time instants or time slots, thereby operating a deviation.
At instant tB, noise 112 (residual signal, background signal), which was previously at level LH1, is to be subsequently played back at level LH2<LH1, so as to increase the quality of the output signal 104 by reducing the noise 112 (by a quantity indicated by 38 in
In theory, for the time instants or time slots before time instant tB, a unitary remixing gain (e.g. 0 dB) could be applied to the noise 112, while a remixing gain less than unitary (negative in decibel) could be chosen for time instants or time slots after time instant tB (in particular in the interval tDA). Hence, the level of the noise 112 would be modified from level LH1 to a level LH2 which causes the difference between the speech 114 and the noise 112 to be the quantity indicated with 42 (clearance). This is a behavior which is subdivided in two remixing criteria:
An example in formula (4) (see below, and see also formula (5)).
The first remixing criterion may be based, for each time instant or time slot before tB, on relative and/or absolute metrics 4145, 4141 associated to exactly that time instant or time slot. On the other side, the second remixing criterion may be based, for each time instant or time slot in the interval tDS (but which is in the future with respect to the time instants or time slots before tB), on relative and/or absolute metrics 4146, 4143 associated to exactly that future time instant or time slot. At time slot or time instant tB, an abrupt change of criterion (and of gate, accordingly) would occur, and the noise 112 would jump from level LH2 to level LH1.
However, it has been understood that this abrupt change would not be pleasant for a listener, and could cause an unwanted pumping effect.
A more smoothed transition (e.g. identified by ramp 2112 in
Notably, we obtain a subdivision into three regions:
In other terms, we note that:
In the second, intermediate region 200G (interval tDS), the determined current time instant or time slot 401 will have a remixing gain 124 which is intermediate between those associated to the current time instant or time slot before tA and after tB.
The same applies in the interval tDR (in which ramp 2113 is experienced), at which the remixing gain 124 also changes gradually again causing the noise 112 to change from level LH2 to a higher level LH1. Even in this case (ramp 2113), at any determined current time instant or time slot before tE (e.g. in interval tOR), the remixing criterion provides a rough value of the gain that would cause the level LH2, while a time instants in the time interval tDR after tE should have a remixing gain causing the level LH1. However, it is possible to take into account the gain as it would be at in the time instants after tER according to the criterion, and accordingly, choose a remixing gain value intermediate between the gain value that causes the level LH2 and the gain value which causes the gain LH1. This is dual to the above-mentioned case of ramp 2112, where at any determined current time instant or time slot before tB (e.g. in interval tOA), the remixing criterion provides a rough value of the gain that would cause the level LH1, while a time instants in the time interval tDS after tB should have a remixing gain which is the gain that causes the level LH2. However, it is possible to take into account the gain as it would be at in the time instants after tEA according to the criterion, and accordingly, choose a remixing gain value intermediate between the gain value that causes the level LH2 and the gain value which causes the gain LH1. The duality can be easily seen in intervals tDS and tDR, and is obtained, for example, by applying formulas (10), (11), and (12) (see below). To achieve this goal, in one example it is possible to apply the shifting as discussed, for example, in 4.9. Other techniques are notwithstanding possible.
In time interval 46, there is no gradual modification between two different remixing criteria, but instead it is remained in the remixing gain as would be defined by the second remixing criterion instead of moving towards the gain defined by the first remixing criterion. It is possible to make use, in some cases, of an utterance integration 330, which permits to recognize that the time interval 46 between the two time intervals (e.g. encompassing tDA and tOR) at which the speech is obtained is still an interval in which the target signal 114 is active. It is noted that some remixing criteria may be dominant over other remixing criteria. For example, the second remixing criterion adopted in the remixing region 200H2 is dominant over the first remixing criterion adopted in the remixing region 200H1: we want to maintain the gain 124 for the residual signal 112 low for coping with situations in which the absence of the target signal is only due to a pause within two words, without increasing the loudness of the noise 112. To the contrary, the first remixing criterion is non-dominant: in the intermediate region 200G, the ramp 2112 is immediately generated, without waiting too much. Hence, before moving from one dominant remixing criterion towards a non-dominant remixing criterion, there may be inspected, in the target information in a time window immediately after the determined current time instant 401, whether the totality (or at least a great number, greater than a first predetermined threshold) of future time instants 406 (or 416) are associated to the non-dominant remixing criterion; while before moving from one non-dominant remixing criterion towards a dominant remixing criterion, there may be no such inspection, or in alternative there may be a less strong condition than that for transitioning from the dominant criterion towards the non-dominant criterion: for example, when transitioning from the non-dominant remixing criterion to the dominant remixing criterion there may be inspected whether a little number of future time instants (e.g. over a second predetermined threshold) or time slots is associated to the dominant criterion, wherein the second predetermined threshold is lower than the first predetermined threshold.
By virtue of the above, it is possible to see that a first remixing criterion and a second remixing criterion may be, in general, used for generating at least one rough remixing gain (e.g. in non-transitory phases). The rough remixing gain 125 may subsequently be corrected by applying a deviation (see also below), e.g. in transitory phases.
The different remixing criteria apply different gains (e.g. different rough gains) and, therefore, will cause different remixings. The discrimination between the remixing criteria is generally made based on a criterion condition. The criterion condition may take into account the metrics 4145 (absolute metrics for the determined current time instant 401) and/or 4141 (relative metrics determined current time instant 401) (see
The criterion condition may take into account the metrics 4141 (absolute metrics for the determined current time instant 401) and/or 4145 (relative metrics determined current time instant 401) on the target signal 114 or a processed version thereof (such as version 314, 335 and, e.g. in case of relative metrics 4145, also versions 312 and 332 of the input mix 102 or the residual signal 112).
In non-transitory conditions (such as in high gain region 200H1 and in the low gain region 200H2), the first and second criteria may be easily respected. For example, in the high gain region 200H1, the first remixing criterion is respected: the gain for the time instants or time slots of the background signal 112 is maintained unitary. The second remixing criterion may provide a reduction of the gain for the residual signal 112 with respect to the first remixing criterion (or in particular, an increase of the ratio between the remixing gain associated the remixing gain associated to the target signal over to the residual signal 112 from the first remixing criterion to the second remixing criterion).
Notwithstanding, in some cases, as explained above, it is possible to deviate from the first and second criteria (e.g., in case of transitory; see intermediate region 200G in
It is also to be noted that each of the first and second remixing criterion is associated to a rough remixing gain (the rough remixing gain based on the first remixing criterion being in principle different from the rough remixing gain based on the second remixing criterion), which can be, notwithstanding, modified (e.g. corrected, deviated). The deviation may be based, for example, on the temporal context information 132. The deviation is evident in
In particular, the deviation may take into consideration the time slots or time instants which are immediately subsequent to the determined current time instant or time slot 401 (e.g. window 406 or 416 in
Some transient variation of the target signal 114 and/or the residual signal 112 or input signal 102 may cause a time instant or slot to be associated to have an incorrect value, so that its metrics 4141 (absolute metrics) or 4145 (relative metrics) may be incorrect, which could drive to be associated to a wrong remixing criterion. In addition or alternatively, there may be the possibility of having a transient disturbance, noise. It is also possible to experience a pause between two words: in
It is possible, as explained above, to verify whether a deviation condition is fulfilled or not. The deviation condition may be at least partially based on the temporal context information 132 (e.g., a window 406 or 416 of time instants or time slots, which are in the future with respect to the determined time instant or time slot 401). If all the future instants are associated to the second criterion (provided that they are in a time window 406 or 416 of a predetermined length, also indicated with tHOLDAHEAD), then the deviation is performed by correcting the rough gain 125. Accordingly, the gain 124 for the residual signal 112 may gradually increase (time interval tDR).
An example valid for example for the transitory in time interval tDR and in time interval 46 is provided by method 500 of
Subsequently, at step 506, it is intended to see whether the rough remixing gain 125 is to be corrected. Therefore, temporal context information may be obtained from the temporal context block 130. At step 508, the deviation condition is evaluated. A condition on the immediately subsequent time instants or time slots 406 or 416 immediately subsequent to the determined current time instant or time slot 401 may be evaluated. For example, if, within a predetermined time window of a predetermined length, all the subsequent time instants or time slots are associated to the different criteria, then it is transitioned to step 510, in which the deviation is performed by correcting the rough gain, e.g. using the techniques discussed with respect to formulas (10) and/or (12). This may be obtained, for example, by defining the at least one gain 124 as a linear combination which keeps into account both the rough gain (g(t)) as obtained from the metrics 4141 (absolute metrics) and 4145 (relative metrics) on the target signal 114 at the determined time instant or time slot 401 and by also taking into account the preceding version (e.g. gsmooth(t)) of the at least one gain 124 immediately preceding the determined current time instant or time slot 401. Accordingly, it may be gradually transitioned from a particular criterion to another criterion.
In case the evaluation of the deviation condition at step 508 determines that not all the future instants in the future time window 406 or 416 will be associated to another criterion, but some of them will also be associated to the current criterion as determined at step 502, then it is transitioned towards step 512 and the gain 124 is maintained constant with respect to the previous one (i.e., the gain 124 as already obtained for the immediately previous current time instant or time slot 425 immediately preceding the determined time instant or time slot 401). More in general, at step 508 it is possible to take into account a time instant or time slot preceding the time instants or time slots (e.g. 406 or 416) following the determined time instant or time slot, such as one of the two time instants or time slots (t and t−1) immediately preceding the time instants or time slots (e.g. 406 or 416) in the time window following the determined time instant or time slot, so as to compare whether the criterion associated to the future time window 406 or 416 is the same of the criterion associated to the one time instant or time slot (t, t−1) immediately preceding the time instants or time slots (e.g. 406 or 416) following the determined time instant or time slot, while at step 502 there may be, in addition or in alternative, determined the remixing criterion of the time instant or time slot t−1, as well.
In the present example, one remixing criterion is dominant (prevailing) with respect to another remixing criterion. For example, in
In some examples, method(s) 500 and/or 600 may include, e.g. at the end, shifting the least one gain (124, gsmooth) as obtained for each time instant or time step of the discrete succession of time instants or time slots by a predetermined number of time instants or time steps towards the past.
Notably, before the determined current time instant (t), the gain(s) 124 (including the immediately preceding time instant or time slot 425 or t−1) is(are) the gain(s) as already obtained (e.g., correct gains in previous iterations for preceding time instants, e.g. gsmooth(t)). On the other side, the remaining instants (instant or slot 401 and the subsequent ones) may have only the rough gain(s) 125, previously obtained based on the metrics (absolute metrics and/or relative metrics) on the target signal 114 that are at those time instants. Therefore, during the process, the final gain(s) 124 of each (all) time instant(s) are subsequently and iteratively updated.
In order to take into consideration the temporal context (e.g. at step 508), an evaluation may be performed on the window 406 or 416 (tholdahead) of the immediately subsequent time instants or slots. Here, the rough gains 125 (g) are evaluated. It is looked (determined) whether they are associated to the first criterion or the second criterion, and/or it is looked (determine) whether they have the same remixing criterion of one of the time instants or slots immediately preceding the window 406 or 416, e.g. the determined time instant or slot 401. This may be the evaluation which is carried out in step 508 of
It is to be noted that it is not strictly necessary to evaluate the obtained gains in the window of rough gains. It is also simply possible to evaluate whether the first or second evaluation criterion are chosen (e.g., roughly chosen). After that, the correction will be performed as explained above.
Both the target signal 114 and the first signal 302 (112, 102) may be processed to obtain a short-term level estimation 314 and 312, respectively. The operations of the short-term level estimations will be explained below in subsection 4.2, but it is already explained that they are associated to a first order IIR filter. A smoothing time constant α may be used for both blocks 306 and 308. It is also possible to transfer into a logarithmic domain to better reflect the magnitude response of the human audio.
On the signals 114 and 302 (102, 112) (or on their processed versions 314 and 312) it is possible to perform a first target activity detection at TAD block 318. The operations of the TAD block 318 are also discussed below in detail in block 338 and in formula (5). In an example, the TAD block 318 may compare the target signal 114 (or a processed version 314 thereof) with an absolute threshold 315 (“absolute gate”) and/or can compare the target signal 114 (or a processed version 314 thereof) with a relative threshold 316 (“relative gate”) (e.g., in comparison with the first signal 302, i.e. the input signal 102 or one of the residual signals 112 or a processed version 312 thereof). If the target signal 114 is not big enough in comparison with the first signal 302 (input signal 102 or the residual signal(s) 112), then it is imagined that in the particular time instant or time slot, the target signal 114 is inactive. Accordingly, in short term activity information 320 may be generated indicating that the target signal is active. If, on the other side, the target signal 114 is not big enough (e.g., either in absolute terms or in relative terms with respect to the input signal or one of the residual signals) then the short-term activity information 320 indicates that the target signal 114 is supposed to be inactive (non-active). Here, the short-term activity information 320 is considered to be a gate signal, which may be understood as a binary information, which indicate that the target signal 114 is considered to be active or non-active. It is to be noted that the short term activity detection information 320 is not definitive in at least some examples. In fact, downstream, this information may be filtered and changed by also taking into account the behavior of the target signal 114 for the time instants and/or time slots closely consecutive to the determined current time instant.
It is to be noted that the short term activity detection information 320 may in general take into account uniquely the evolution of the signals 114 and 302 (e.g., 102 or 112) of the processed versions thereof 314 and 312, but in general does not take into consideration the signal (e.g. 114 and/or 302) at samples and/or instants around the considered time instant. As it will be shown in the following, this can give some issues, since it is possible that a pause is performed between two different words in a speech and this could cause (if the speech is the target signal 114) that the short-term activity information 320 is different between the samples and/or slots carrying the words and the sample and/or slot carrying the pause between the words. In some cases, this can be unacceptable, since this could cause the modification of the remixing parameters between the time instants and/or time slots carrying the word and the time instant and/or time slots carrying the pause between the words. Said in other terms, even if we may want that the speech has a gain which is relatively higher than the gains gained for the background, it is possible that we do not want to modify it instantaneously, since an instantaneous modification is understood as unpleasant by a human listening.
However, it has been understood that, by making use of context information (e.g., 370 and/or 372), it is possible to address at least some of these inconveniences. A context based integration block 330 is provided.
Block 330 may permit to perform an utterance integration (see also section 4.4 below). Block 330 may in some examples be described as follows: a cumulative sum of the target signal 114 (or one of its processed version 314, 334) and a cumulative sum of first signal 302 (112 or 102, or one of its processed versions 312 or 332) may be obtained depending on whether activity is detected (based on the activity information 320) for time instants or time slots for which activity is detected. In some examples, all the time instants or all the time slots of an interval in of time instants associated to the same criterion are assigned the same value (e.g. the average of the cumulative sum), and they may be assigned to have the same value. Notably, in case some scattered time instants or time slots are associated to a different criterion (e.g., to the dominant criterion), they may be reassigned to the dominant criterion. In addition or in alternative, the block 330 may wait up to a minimum threshold of consecutive time instants or time slots associated to the non-dominant criterion before giving the same value for all the preceding time instants and time slots. This may therefore be an averaging which makes use of temporal context information from the future and/or from the past. Further information is provided in section 4.4.
The output of the block 330 may be an averaged version of the target signal 114 (314) and the first signal 302 (102, 112). A gain computation block 340 may be provided. The gain computation block 340 may operate according to a constraint (such as a target clearance in the example of the attenuation as shown in
It is noted that the gain computation block 340, as such, defines a remixing criterion which only takes into account the metrics 4141 (absolute metrics) and/or 4145 (relative metrics) of the current time instant 401, but does not take into account future or past time instants or slots 403 and their metrics 4146 (relative metrics) and/or 4143 (absolute metrics). The output 343 of the gain computation block 340 may therefore be, in some examples, an output which does not provide a variable remixing gain (e.g. it is not smoothed). It is possible to understand the output gain 343 as a rough remixing gain which has to be subsequently refined by taking into account metrics (e.g. relative metrics 4146 and/or absolute metrics 4143) on future time instants and/or past time instants. Notably, the gain computation block 340 basically embodies the second remixing criterion which is verified, for example, in the third, low gain region of
The TAD refinement block 338 may be seen as identifying the time intervals in which the second remixing criterion is not to be used. This can be, for example, the high gain region 200H1 of
The activity information 342 operates like a gate in gain gating block 350. The activity information 342 may discriminate between choosing the first remixing criterion and the second remixing criterion. Notably, the output 352 of the block 350 (gated gain) is still a rough gain. In the example of
With reference to formula (5) (see below), it may be that the rough gain g(t) is defined as:
In this case, the rough gain (gated gain) 352 (125) may be g(t). The determination between the different gains may be made by taking into account the absolute gate 315, which may be the value G with which the intensity Îs (absolute metrics) is compared, so as to obtain the activity information (which e.g. provides information whether the speech is active). The determination between the different gains may be made by taking into account the target clearance so that, if SNRin(t)>C, then the first criterion (e.g. g(t)=1) is chosen, otherwise the second criterion
Different ways of defining the rough gains (and/or of determining which remixing criterion each time instant pertain) may be implemented.
It is to be noted that, in examples, the elements 308, 306, 318, 330, 52, 54, 56 shown in
At block 360 (smoothing block), it is possible to smoothly modify the remixing gain for the noise 112 (e.g. actuating a deviation from the remixing criteria). Here, the ramp 2112, for example, may be generated. In the time instants and/or time slots in the intermediate region (interval tDS), neither the first remixing criterion nor the second remixing criterion is used. To the contrary, temporal context information (as explained above) permits to take into account past time instant(s) and/or future time instant(s). Therefore, the gain can be gradually reduced in the ramp 2112. The same would apply in the interval tDR, where an ascending ramp 2113 is obtained analogously. Reference can also be made to sections 4.7 and 4.8 below. It is also noted that at least one remixing gain 124 may be seen as being obtained by refining a rough remixing gain 343 or 352 by adding an additive component (modifying component) which corrects the rough remixing gain 343 or 352, smoothening the obtained remixing gain 124.
In some examples, also the start of the ramp 2112 or 2113 (at time instant tA or tR) is based on the knowledge of the future temporal context information: knowing that there will be a change in remixing criterion soon (e.g. within a temporal window 406 or 416 immediately subsequent to the determined time instant or time slot 401), the deviation may start.
Reference can be made, for example, to formulas (10) and (12). Hence, the modifying (correcting) can be based on the immediately preceding and/or subsequent time gain 124 (gsmooth(t−1)) as previously provided. By taking into account the gain as output for the immediately preceding time instant or time slot it is possible to obtain a gradual descending or ascending effect for the gain. This is shown in formula (10) below is for the descending gains (e.g., ramp 2112 in the intermediate region in the interval tDS) and formula (12) is for ramp 2113 in the interval tDR. It may be stated, therefore, that the rough gain 343, 352 is refined by taking into account future and/or past time instants or time slots 403. Block 360 may have inputs 357 (associated to τatt(t); 358 (τrel(t)) and tholdahead (359) as explained in sections 5.7 and 5.8. It is noted that τatt is greater than τrel.
As explained above, the control block 120 may provide temporal information 122 on the current time instant or time slot which will be subsequently used as temporal context information 132 (e.g., for subsequent time instants or time slots, and/or for refining a previously obtained rough gain 125, so as to deviate from the rough gain 124 to obtain the remixing gain 125). As it will be shown later, the temporal information 122 on the current time instant or time slot may include at least one of the output of the utterance integration block 330 (e.g., 332, 334) or information associated thereto; rough gain 343 and/or activity information (e.g. gate information) 342; a gated gain (e.g., rough gain 352); and/or the at least one remixing gain 124 (e.g., gsmooth(t−1)). Some of these information will be explained in greater detail below.
As explained above, we propose, inter alia, to generate the output mix y(t) (output signal 104) by remixing the estimated sources with a time-varying linear combination:
y(t)=h(t)ŝ(t)+g(t){circumflex over (b)}(t), (2)
where t is the time-index (time slot or time instant) and h(t) and g(t) are the signal-adaptive remixing gains to be determined by the control module (control block 120), for which additional details are given in Sec. 3.3. This is equivalent to combining ŝ(t) and x(t), i.e., y(t)=k(t)ŝ(t)+z(t)x(t), where k(t) and z(t) are the remixing gains in this case. The remixing gains h(t), g(t), k(t), z(t) can be frequency-dependent or broadband (equal for all frequencies). The following discussion uses broadband gains for illustrating the operations.
On the output mix y(t) (104) further post-processing can be applied, such as loudness normalization, dynamic range compression, or applying equalization.
The signals are here discussed as they were real-valued time signals, but the same problem could be formulated in the time-frequency domain, e.g., Short-time Fourier Transform (STFT) domain.
The remixing gains 124 are in general computed based on features (metrics) of the input signals ŝ(t) (112) and x(t) 102 (and/or potentially of {circumflex over (b)}(t) 112) along with a criterion, and parameters that define the desired features of the output mixture y(t). These parameters can be user-defined or fixed by one or more presets.
A prominent feature (metrics) is (or is associated to) the intensity of the signals (which may be the absolute metrics 4141 and 4143). Different ways of quantifying the intensity of a signal can be used here, with different computational requirements. These are for example:
Another important feature (metrics) is the intensity difference between signals (relative metrics 4145 and/or 4146). Different ways of quantifying the intensity difference exist and are applicable for the proposed method. These may be for example:
From our experience, it is particularly useful to set condition on the minimum intensity difference (also referred as clearance), leaving the input mix 102 unchanged if the estimated intensity difference in it is already big enough.
As an example for the control criterion for computing the remixing gains, let us set a specific value C (target clearance 339 in
Considering Eq. (2) (formula (2)), the output SNR between the target source signal and the residual signal in the output mixture after applying the remixing gains can be estimated as:
where w(·) is an optional frequency weighting, e.g., k-weighting [9]. For the sake of clarity, we can set h(t)=1 and ignore w(·):
from which it is clear that SNRout(t) can be controlled by g(t).
Our example control criteria needs to find g(t) such that SNRout(t)>C (clearance condition), together with the condition that that g(t)=1 if the intensity of ŝ(t) is below a certain threshold G, i.e., Îs(t)<G (gating condition or intensity condition). A time-varying, signal-adaptive, broadband solution can be:
(The solution according to formula (5) is substantially a solution which takes into account, for each time instant 401, only metrics 4141 and 4145 on values of that time instant, without taking into account different (future or past) values. The gating condition and/or the clearance condition may form or be comprised in the criterion condition).
The input SNR (also indicated with SNRi or SNRin) can be estimated as:
If the temporal context would be ignored, the intensities Ix and Îs could be computed as Ix=w(x(t)2) and Îs=w(ŝ(t)2), however the temporal context 132 can be essential for the esthetical pleasantness of the final result. We may use the temporal context as detailed in Sec. 4. In fact, limitation, time integration, and smoothing may be applied on the remixing gains g(t) and/or on the involved signals (e.g., Îs and SNRin(t) also indicated with SNRi(t)) so to avoid abrupt transitions and pumping, and to generally obtain a smooth and esthetically pleasing output mix.
The smooth gains generated by taking into account the temporal context 132 and the final esthetical pleasantness could be referred to as gsmooth(t):
g
smooth(t)=smooth(g(t)) (7)
It is possible that gsmooth(t) do not strictly fulfill the criterion used for computing the first gains (rough gain) g(t), e.g., by not fulfilling the instantaneous SNR criterion (e.g. criterion condition) at locations in which large gain changes are smoothed over time (e.g. the above discussed second, intermediate region in
Instead of SNRout(t)>C and Îs (which would be a criterion which does not take into account metrics 4146, 4143 on future or past time instants or time slots 403), estimates of the perceived momentary or short-term loudness (e.g., [9]) can be used as intensity measures for the control criteria. Preferences for loudness differences are investigated in [3, 4]. Other criteria can be based on a partial loudness model [10] or on time-dependent intelligibility or quality metrics, similarly to [11]. Also a voice activity detection could be usefully integrated, e.g., by replacing the gating condition Îs<G with a condition based on speech presence probability.
Finally, it is possible to extend the solution of Eq. (5) to provide gains that are not only time-varying and signal-adaptive, but also frequency-varying.
Also, the control module could take {circumflex over (b)}(t) instead of x(t) as input and similar results could be achieved. In other words, in addition to ŝ(t), only one signal between x(t) and {circumflex over (b)}(t) is needed for the Control module. Our preference is having access to x(t) (as in
Control module in detail: Operational block diagram of an example of the usage of temporal context for producing gsmooth(t).
A non-essential part of the proposed method contains the automatic adjustment of one or more of the operational parameters of the method, e.g., “Target clearance”, “Attack”, or “Release”. This can be based on the classification of the non-speech parts of the input mix x(t), e.g., if these are dominated by music content or by ambient noise and effects. This information can be used to adjust the “Target clearance 339” accordingly, e.g., to a different value as suggested by the findings in [3, 4].
Another option is to adjust the remixing parameters based on a quality estimate of the separation. Such an estimate can be done based on ŝ(t) (114) and x(t) (102), as presented in [12] or based on deep neural networks (DNNs), similarly to [11]. E.g., if the separation quality is low (e.g., because of challenging input mix 102), the smoothing parameters can be set to be more conservative and a smaller clearance can be selected.
The Content classification and Parameter adjustment functionalities are not required for the basic operation of the proposed method, but the parameters can be adjusted also manually or fixed by constant presets. However, a classifier 52 may classify a content of the signals 114 and/or 102 and/or 112. For this purpose, the classifier 52 may have a class determiner 54 which, for example, distinguishes a first class from a second class, for example speech from non-speech, music or other tonal noises from transient events, whereby both a class of the noises and a number of differentiated classes can be arbitrary. The class determiner 54 may provide the determined class to a parameter adjuster 58 by means of a class determination signal 56. The classifier 52 may be configured to set at least one parameter of the combining and/or the signal attenuation based on a result of the classification. The parameters set by means of the parameter adjuster 58 can thus relate to any further operation of the device 40.
In a first stage, the temporal context 132 may be used for smoothing the intensities of the inputs ŝ(t) (114) and x(t) (102, or more in general the first signal 302). Let us consider the input intensity of x(t) (same operations hold for ŝ(t)). As already mentioned, one way to quantify the intensity of x(t) is to compute the power of the signal filtered so to mimic the frequency response of the human ear: Ix(t)=w(∥x(t)∥2). This is smoothed, e.g., with a first-order infinite impulse response, IIR, filter:
I
x,smooth(t)=αIx,smooth(t−1)+(1−α)Ix(t), (8)
where α is a feedback coefficient, e.g., computed from a smoothing time-constant. The smoothed estimate 314, 312 can be further transformed into a logarithmic domain to better reflect the magnitude response of the human auditory system. This is referred to as Ex(t) for the input signal 102 (or more in general the first signal 302) and as Ês(t) for the target source signal 114.
The smoothed intensity estimates are used for a simple level-based activity detection. A gate signal 320 is produced, signaling if Ês(t) is big enough in absolute terms, i.e., it is bigger than an absolute threshold and in relative terms, i.e., compared to Ex(t) with a relative threshold.
More in general, the gate signal 320 may represent a short-term activity detection, which indicates the activity of the target signal 114 but which may be modified by taking into account the temporal context, for example.
The parameters 315 and 316 may be an absolute threshold (e.g. so-called “absolute gate”, and also indicated with G) and/or a relative threshold (e.g. so-called “relative gate”, which is optional).
4.4 Utterance Integration (UI) (e.g. Block 330)
If the target source 114 is speech, it has to be observed that people tend to talk louder during the first syllables of an utterance. This means that Ês(t) is higher in the utterance beginning compared to the rest of the utterance. Assuming a constant level or the background sources, the effect on the gain is that in the beginning of the utterance less background attenuation is needed than later on and the attenuation changes gradually over time to more attenuation. This “creeping” background attenuation is perceived esthetically rather unpleasing.
UI (e.g. at block 330) takes as the input the TAD output gate signal 320 and the two initial signal level estimates Ês(t) and Ex(t) (314 and 312).
UI implements a sliding window mean computation applied on the linear-domain level estimates before transforming them back in the logarithmic domain. The computation has two main modes of operation: start of utterance and sliding. The more interesting is the first one:
A benefit of this processing is that the level estimate remains constant during the start of an utterance and also later on it changes more slowly. The constant level estimate results into a more consistent gain value and avoid the “creeping gain” problem, making the output esthetically much more pleasant. The output of UI may be refined level estimate Ês(t) and Êb(t). The later may be used, for example, to obtain at least one of the metrics 4141, 4143, 4145, 4146.
The window is also called “filtering window” and may make use of values of any of the signals 114, 302 (112, 102) or their processed versions (314, 312) to obtain filtered versions 334 and 332 of those signals (334 is the filtered version of 114 or 314; 342 is the filtered version of 302, e.g. 102, 112, or the processed version 312. A filtering window for the determined current time instant or time slot 401 could be, for example, represented by the union of the pluralities of future and past samples 406 and 407.
Since the intensity estimates are now temporally more stable, it is beneficial to refine the TAD processing, similarly to Sec. 4.3. A long-term activity detection 342 (here considered a gate signal, e.g. a binary signal) is therefore obtained.
The parameters 315 and 316 may be an absolute threshold (e.g. G, “absolute gate”, which may be the G of formula (5)) and/or a relative threshold (relative gate, optional).
The core of the gain computation can be now carried out as explained in Sec. 3.3 (see in particular Eq. 5) and by using the stable and smooth intensity estimates and the gate signal obtained so far. The output is g(t), which undergoes a temporal smoothing as explained in the following.
The temporal smoothing can be implemented in various ways, but we may use a simple first-order IIR-filtering approach as an example (other techniques may be implemented). The control inputs to the smoothing method are attack time (357) tatt (e.g. corresponding to the ramp 2112 and to the transition from the first remixing criterion to the second remixing criterion), release time (358) trel (e.g. corresponding to the ramp 2113 and to the transition from the second remixing criterion to the first remixing criterion), and hold look-ahead time tholdahead (359). The first two time constants define feedback coefficient values through
τatt=1−e−1/t
for the attack, and similarly for the release. Other translation formulas may also be used and these are only exemplary. The basic attack/release smoothing produces the smoothed gains:
g
smooth(t)=(1−τ(t))gsmooth(t−1)+τ(t)g(t), (10)
where
A problem with this smoothing is that if there is a short pause in the target source signal 114, e.g., between words, sentences, or talkers, the attenuation gain starts the release phase, the background signal comes (partly) back up before being attenuated again when the speech continues. An attempt to solve this pumping problem in the earlier works is to use a constant hold time which delays the release phase with a constant amount. A drawback of this is that the release is delayed, regardless if the need for background attenuation continues or not. This can cause unpleasant gaps after the target activity (i.e., speech) has ended. We propose a signal-adaptive mechanism of hold look-ahead for solving this problem: the smoothing uses a look-ahead buffer into the future and detects if the gain applies the same amount or more attenuation within the window of length tholdahead. If this is the case, operation similar to normal hold is activated and the current gain value is kept, otherwise attack and release smoothing is performed normally. This process can be exemplified by surrounding Eq. 10 (formula (10)) with some additional logic:
where chold(t) is a variable indicating the length of the still remaining time to keep the current gain value and it can be
c
hold(t)=0 if gsmooth(t−1)>g(t),
otherwise
c
hold(t)=max{chold(t−1)−1,kmin(t)},
where kmin(t) indicates the location of the minimum gain value within a window of tholdahead future values if this value is smaller than the current smoothed value, e.g.,
kmin(t)=argmin{g(t:t+tholdahead)} if min{g(t:t+tholdahead)}<gsmooth−1), otherwise
k
min(t)=0.
See also
The description so far is sample-synchronized in the sense that the potential background attenuation induced by applying the produced gains would start exactly at the same sample as the target becomes active. When this is combined with the attack/release-smoothing, the result is that the background attenuation may be perceived to start in a delayed fashion, i.e., too late. Additionally sometimes an earlier attack start of the attenuation is desired for esthetical reasons. A solution is to implement a temporal shift between the gain and the audio signals by shifting the gains by some small time, look-ahead or shift. This operation may conclude the generation of gsmooth(t).
In the case of shifting being used,
In addition or alternatively, it is possible to start the ramp directly based on the temporal context information (e.g., by knowing that in the future there is a change of criterion, it is possible to start the deviation).
A (possibly incomplete) list of related works is reported in the following, pointing out commonalities and differences with the approach proposed in this report.
5.1 Remixing Separated Sources without Signal-Adaptive Control
Some advantageous aspect of the present examples are here below briefly resumed:
Present examples mainly refer to a system (e.g. 100) for processing audio signals. The system (e.g. 100) may comprise a source separation block (e.g. 110) estimating, from an input signal (e.g. 102) which evolves in time along a discrete succession of time instants or time slots (e.g. 401, 403), a target signal (e.g. 114) and at least one residual signal (e.g. 112) to be subsequently remixed (e.g. at remixing block 150, which is part or not part of the system 100) according to at least one remixing gain (e.g. 124) variable along the discrete succession.
The system 100 may comprise a control block (e.g. 120) determining, for a determined current time instant or time slot (e.g. 401), at least one metrics (e.g. one of an absolute metrics 4141 and a relative metrics 4145) on the target signal (e.g. 114, 1141), or a processed version (e.g. 314, 334) of the target signal (e.g. 114, 1141), in the determined current time instant or time slot (e.g. 401). The at least one metrics (e.g. one of an absolute metrics 4141 and a relative metrics 4145) may e.g., be, or be based on at least one relative metrics (e.g. 4145) between the target signal (e.g. 114, 1141), or a processed version (e.g. 314, 334) of the target signal (e.g. 114, 1141), and the input signal (102, 1121), or a processed version (e.g. 312, 332) of the input signal, or the at least one residual signal (112, 1121), or a processed version (312, 332) thereof, in the determined current time instant or time slot (401). The at least one metrics may be a relative metrics (e.g. 4145). For example the at least one relative metrics may be, or be based on, the SNRin (e.g. signal-to-noise ratio) of the input signal (e.g. 102) or of the processed version thereof (e.g. 314 and/or 334). The SNRin (e.g. signal-to-noise ratio) may be, or be associated to, a relative intensity between the target signal (e.g. 114) and the input signal (e.g. 102, 1121), or a processed version (e.g. 312, 332) of the input signal, or the at least one residual signal (e.g. 112, 1121), or a processed version (e.g. 312, 332) of the at least one residual signal. Examples are provided in formulas (5) and (6). For example, according to formula (6) (see also above)
where the Ix and Îs are intensities (or weighted versions of intensities) of the input signal 102 (or or a processed version (e.g. 312, 332) of the input signal) and of the target signal 114 (or processed version thereof). In some examples, numerals 334 and 332 of
The system 100 may comprise a temporal context block (e.g. 130). The temporal context block (e.g. 130) may, for example, perform at least one of the operations:
Therefore, at least one future time instant and at least one past time instant (or at least one of them) may be determined at the temporal context block (e.g. 130). The at least one future time instant or time slot (e.g. 403, 406, 416 or one in a window, such as a window 417, 407, 416, 426, etc.) may be, in the discrete succession, after the determined current time instant or time slot (e.g. 401). The past time instant or time slot (e.g. 425, or in a window 407, 417) may be, in the discrete succession, before the determined current time instant or time slot. The temporal context information (e.g. 132) may, for example, be or be based on or at least include at least one metrics on the target signal in at least one future time instant and/or at least one past time instant (e.g. at least one relative metrics 4146, at least one absolute metrics 4143, or both). The temporal context information (e.g. 132) may, for example, be or be based on or at least include a previously obtained remixing gain (in some examples it may be at least one previously obtained rough remixing gain g(t) e.g. in the future time instants; in some examples it may be at least one previously obtained smoothed, final remixing gain gsmooth(t), and in some other examples it may comprise both, or be, at least one previously obtained rough remixing gain, e.g. in the future time instants, and at least one previously obtained smoothed, final remixing gain gsmooth(t−1), e.g. for a preceding time instant or time slot).
The control block (e.g. 120) may be configured to generate at least one remixing gain associated to the determined current time instant or time slot by (e.g. 401, t) considering:
The at least one remixing gain may for example be obtained after having compared the relative metrics (e.g. SNRiin) with a threshold (e.g. C, 339). In some examples (e.g. in the example of formula (5)), if the relative metrics (e.g. 4145) is below the threshold (e.g. SNRiin<C), then there is defined a gain g(t) (e.g. rough gain) such that the distance between the level of the target signal (or processed version thereof) and the level of the level of the input signal (e.g. 102, 1121), or a processed version (e.g. 312, 332) of the input signal, or of the at least one residual signal (e.g. 112, 1121), or a processed version (e.g. 312, 332) of the at least one residual signal, is increased (e.g. up to C or at least C, e.g. reaching the target clearance 42), e.g. by attenuating the at least one residual signal (e.g. 112), or processed version of the at least one residual signal, and/or by boosting the target signal, or the processed version thereof. If the relative metrics (e.g. SNRiin) is over the threshold (e.g. SNRin>C), then the rough remixing gain g(t) may be maintained as the input gain (e.g. g(t)=1), since the minimum distance C (e.g. target clearance) is already obtained. Once the rough gain g(t) is obtained, it is possible to modify it by taking into account the temporal context information 132. For example, a smoothed version of the remixing gain gsmooth(t) may be obtained when, from the temporal context information 132, variations of the (e.g. rough) gain in subsequent time instants are determined. For example, if the subsequent time instants are all (or at least prevalently) associated to a different (e.g. rough) gain (e.g. to the attenuating gain), then the rough gain may be modified so as to slightly fade towards the different gain.
In some examples, there are defined at least one first remixing criterion (e.g. implying g(t)=1) and one second remixing criterion (e.g. implying
or in any case implying a g(t) which is less than the g(t) at the first remixing criterion) for generating the rough remixing gain (e.g. at the particular determined current time instant t). At least one criterion condition (e.g. a comparison between a relative metrics, e.g. SNRin(t), and a predetermined threshold, e.g. C) may therefore be defined to perform a discrimination between using the first remixing criterion and using the second remixing criterion at each time instant or time slot. In some examples there may be, in addition or alternative, also a comparison between an absolute metrics, such as an intensity Îs(t) of the target signal s(t) with another threshold G, so that if Îs(t)<G, then the rough remixing gain is chosen to be unitary g(t)=1, otherwise
if Îs(t)>G (e.g. attenuated background); in some examples (like in formula (5)), both the criterion conditions may form one OR-condition based on both a first condition (comparison of SNRin with C, or another relative metrics) with a first threshold (C, 339) and another second condition (comparison of intensity Îs(t), or another absolute metrics 4141, with a second threshold, e. g. G, e.g. 315). Therefore on the at least one criterion condition, each time instant or time slot is associated to one of the at least one first remixing criterion and second remixing criterion (e.g. for a first time instant t1 it may be that g(t1)=1 and for a second time instant t2 it may be that
this being decided through the evaluation of the criterion condition on the relative metrics and/or the absolute metrics). Hence, at least one criterion condition may be a condition on the at least one (relative and/or absolute) metrics on at least the target signal, or a processed version thereof, at the determined current time instant or time slot, or on information obtained from the at least one metrics on the at least the target signal or a processed version thereof. The determined current time instant or time slot is associated to one of the at least one first remixing criterion and one second remixing criterion based on the metrics on the target signal, or a processed version of the target signal, in the determined current time instant or time slot.
The system may also obtain (e.g. determine) the at least one remixing gain (e.g. in smoothed version in some examples, which is also indicated with gsmooth(t)) for the determined current time slot or time instant (t) by considering temporal context information 132 so as to deviate, from the at least one rough remixing gain, based on a deviation obtained from the temporal context information 132. In some examples, by being known that the next future time instants or time slots (totally or partially in a subsequent time window) the rough remixing criterion g(t+Δt) will be different, then some deviations may be possible. It is possible to understand that the deviations permit to obtain a graceful transition from a remixing gain implied by a remixing criterion to another remixing gain implied by another remixing criterion. Examples of deviations are proposed in formulas (10) and (12). It is possible to correct the rough remixing gain (125) by an amount associated to a previously obtained remixing gain for a time instant or time slot preceding the determined current time instant or time slot; this means that the already obtained remixing gain. E.g., in one example at the preceding time instant or time slot t−1 a remixing gain gsmooth(t−1) has been obtained, and at time instant or time slot t the remixing gain gsmooth(t) may be obtained by correcting the at least one rough remixing gain (125) by an amount associated to a previously obtained at least one remixing gain (e.g. gsmooth(t−1) for a time instant or time slot (e.g. 425) preceding the determined current time instant or time slot, like in formulas (10) and (12). It is possible, in addition or alternative, to correct the at least one rough remixing gain g(t) through a linear combination of the through remixing gain obtained (e.g. through the evaluation of the criterion condition applied to the relative and/or absolute metrics) for the present current time slot or time instant t and the remixing gain (gsmooth(t−1)) obtained for the preceding time slot or time instant t−1. Therefore, by taking into account the temporal context information 132 (comprising e.g. information such as gsmooth(t−1), which is information on the past, and/or information such as the rough gain for subsequent time slots or time instants, which is information on the future) it is possible to properly deviate from the remixing criterion defined by evaluating the criterion condition.
In some examples, the deviation from the rough remixing gain (e.g. g(t)) by correcting the at least one rough remixing gain (e.g. g(t)) for a gain amount associated to a previously obtained remixing gain (e.g. gsmooth(t−1)) for a time instant or time slot (e.g. t−1) preceding the determined current time instant or time slot (e.g. t) may be subjected to the fulfilment of a deviation condition. The deviation condition may also be based on the temporal context information 132. In this case, the temporal context information 132 may include information on rough remixing gains already obtained for time instants or time slots following the determined time instant or time slot (e.g. in a time window from t, or t+1, to t+tholdahead, or in some examples another window which is not immediately subsequent to the current time instant or time slot t). The deviation condition may be fulfilled e.g. when a predetermined number (e.g., according to examples, the a predetermined number, or the majority, or all) of rough remixing gains already obtained for time instants or time slots (e.g. in the time window from t, or t+1, to t+tholdahead) following the determined time instant or time slot (e.g. t) are associated to a remixing criterion which is different from the remixing criterion of the time instant or time slot preceding the current determined time instant or time slot (or a time instant or time slot preceding the time instants or time slots in the time window following the determined time instant or time slot, such as one of the two time instants or time slots, like t−1 and t, immediately preceding the time instants or time slots in the time window following the determined time instant or time slot, e.g. one of the current determined time instant or time slot and the time instant or time slot preceding the current determined time instant or time slot), and otherwise the deviation condition is not fulfilled. If the deviation condition is satisfied, then the deviation is carried out. Otherwise the remixing gain (e.g. g(t)) for the determined current time instant or time slot (e.g. t) may be maintained the same of the at least one remixing gain for a time instant or time slot (e.g. t−1) preceding the determined current time instant or time slot (e.g. t). For example, if only a low number of subsequent time instants (e.g. in the window from t, or t+1, to t+tholdahead) is assigned to a different remixing criterion, then the deviation is not performed, but if a great number (e.g. all in some examples) of subsequent time instants (e.g. in the window from t, or t+1, to t+tholdahead) is assigned to a different remixing criterion, then the deviation is performed. Therefore, disturbance may be tolerated.
In some examples, one remixing criterion may be dominant over another criterion. In some examples the remixing criterion according to which the residual signal 112 is attenuated (e.g. when
may be dominant over the remixing criterion according to which the residual signal 112 is not attenuated (e.g. when g(t)=1). This because it has been understood that this is advantageous, e.g. when the target signal 114 is speech and the residual signal 112 is noise, so as to avoid an abrupt increase of noise e.g. between two words.
It is to be noted that the system 100 may have or may not have the remixing block 150, according to the examples. The remixing block 150 may simply receive the target signal 114, and residual signal 114 (or the input signal 102) together with the remixing gain (e.g. gsmooth(t)), and the remixing block 150 will apply the remixing gain (e.g. gsmooth(t)) to the signal (e.g. to the residual signal).
However, in some examples the remixing gain (e.g. gsmooth(t)) is not necessarily to be applied to the signal (e.g., input signal 102, target signal 114, or residual signal 112) at the same time t for which it has been obtained. Indeed, the system 100 may shift the at least one remixing gain (gsmooth(t)) as obtained for each time instant or time step of the discrete succession of time instants or time slots by a predetermined number of time instants or time steps towards the past. For example, the remixing gain gsmooth(t) may be assigned to gsmooth(t−D), where D is a predetermined number of time slots or time instants. Hence, a better smoothing may be obtained.
Some additional variants and/or additional or alternative aspects and/or examples are discussed here below.
The gain computation block 340 provides at least one gain according to a second criterion (e.g., in the low gain region 200H2 in
While there is no ramp 2112 and 2113 obtained, notwithstanding, the utterance integration block 330 may permit to maintain the low gain for the background level 112. This is because the utterance integration block 330 has the possibility of looking in the future with the temporal context information 372 (132), which provides metrics 4146 and/or 4143 regarding future time instants or time slots 403 (or more in detail, a window 406 or 416 of future time instants or time slots). It is also possible to take into consideration past time instants or time slots, such as those in the window 407 or 417 immediately preceding the determined current time instant or time slot 401. The utterance integration, therefore, permits to maintain the level at the criterion established for the dominant second remixing criteria at the expense of the non-dominant first remixing criterion. A possibility is provided when transitioning from the second criterion to the first criterion. Other examples may also completely avoid the utterance integration.
Another example is provided by avoiding the utterance integration 330 and the short term TAD block 318, but maintaining the blocks 340, 348, 350, and 360, for example. Also in this case, it is possible to obtain a soft transitioning between the two remixing criteria. Information from the future (part of the temporal context information) may also indicate the start of the ramp 2112 and 2113 at time instants tA and tR.
In some cases, it is possible (e.g., when the information from the future does not provide the time in the instant in which the ramp shall be started) that the gates 124 as provided could, for example, be shifted by a predetermined amount towards the past. However, in some examples, this could be post-processing operation down streamed to block 360 (but up streamed to the remixing block 150).
Upstream to the remixing block 150, it is possible to encode a bitstream encoding the target signal (114), or a processed version (314, 334) thereof, and the at least one residual signal (112), or a processed version (312, 332) thereof, or input signal (102), or a processed version (312, 332) thereof, and the at least one gain (124). The bitstream may be stored and/or transmitted (e.g., through electric or wireless transmissions media) and may be subsequently received, read and decoded upstream to the remixing block 150.
Additionally or alternatively upstream to the control block 120, it is possible to encode a bitstream encoding the target signal (114), or a processed version (314, 334) thereof, and the at least one residual signal (112), or a processed version (312, 332) thereof, or input signal (102), or a processed version (312, 332) thereof. The bitstream may be stored and/or transmitted (e.g., through electric or wireless transmissions media) and may be subsequently received, read and decoded upstream to the control block 120.
Basically, any of blocks 110, 120, 130, 150, may be separated from the other ones or may be in the same device of at least one of the other ones.
Here above reference is often made to the at least one remixing gain mostly using examples in which the gain g(t) (in its rough version) or gsmooth(t) (in its corrected, deviated version) is the remixing gain to be applied to the background noise 112 (b(t)). Notwithstanding, it is also possible to apply a gain h(t) (in its rough version) or hsmooth(t) to the target signal 114 (s(t)). The at least one gain (either in its rough version or in its smoothed version) may also comprise both the remixing gain to be applied to the background noise 112 (b(t)) and the gain h(t) (in its rough version) or hsmooth(t) to the target signal 114 (s(t)) and may therefore be formed e.g. by a 2-elements vector.
In some examples, we will have that a second ratio (which may be 1/gsmooth(t), e.g. obtained at the second remixing criterion, when the background signal 112 is attenuated) between the rough remixing gain associated to the target signal (which may be 1) and the rough remixing gain (which may be gsmooth(t)<1) associated to the input signal (or processed version thereof) or the target signal (or processed version thereof) may be higher than a first ratio (which may be 1, e.g. obtained at the first remixing criterion, e.g. non-attenuating the background signal) between the rough remixing gain (which may be 1) associated to the target signal and the rough remixing gain (which may be 1) associated to the input signal (or processed version thereof) or the target signal (or processed version thereof). During the transitional periods, the ratio may be moved from the first ratio to the second ratio, or vice versa.
The examples above also refer to a method for processing audio signals, comprising:
The examples above also refer to a non-transitory storage unit storing instructions which, when executed by a processor, cause the processer to process audio signals, according to:
The implementation in hardware or in software may be performed using a digital storage medium, for example cloud storage, a floppy disk, a DVD, a Blue-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.
Some examples according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.
Generally, examples of the present invention may be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine-readable carrier.
Other examples comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier. In other words, an examples of the method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.
A further examples of the methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. A further example is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet. A further examples comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. A further examples comprises a computer having installed thereon the computer program for performing one of the methods described herein.
In some examples, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some examples, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are performed by any hardware apparatus.
While this invention has been described in terms of several advantageous embodiments, there are alterations, permutations, and equivalents, which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 102021201668.5 | Feb 2021 | DE | national |
This application is a continuation of copending International Application No. PCT/EP2022/054432, filed Feb. 21, 2022, which is incorporated herein by reference in its entirety, and additionally claims priority from German Application No 102021201668.5, filed Feb. 22, 2021, which is also incorporated herein by reference in its entirety. There are provided techniques for audio signal processing, such as for signal-adaptive remixing of separated audio sources and for providing gains therefor.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/EP2022/054432 | Feb 2022 | US |
| Child | 18452084 | US |
