This application is based upon and claims the benefit of priority from Japanese Patent Application No. 2012-277999, filed on Dec. 20, 2012 and Japanese Patent Application No. 2013-235396, filed on Nov. 13, 2013; the entire contents of which are incorporated herein by reference.
Embodiments described herein relate generally to a signal processing device, a signal processing method, and a computer program product.
A technology for removing a speech signal (human voice or the like) from an acoustic signal may be used to make background sound that is lost in speech and that is hard to make out easily audible, or to play a piece of music karaoke style by removing the voice of the singer from music content. For example, a technology for removing a speech signal from acoustic signals of two channels, a right signal and a left signal, is known.
Now, there are various relationships between signals regarding acoustic signals of two channels. When the signals of two channels are given as a left signal L and a right signal R, respectively, these are modeled in the following manner.
L=B
L
+C
L
+e
L
R=B
R
+C
R
+e
R
Now, BL and BR are background sound signals included in the left signal and the right signal, respectively. Also, CL and CR are speech signals included in the left signal and the right signal, respectively. Moreover, eL and eR are noises included in the left signal and the right signal, respectively. The noise includes a microphone noise, and an encoding noise. Many contents are created such that the speech signals are equally included in the left signal and the right signal. Thus, as conditions regarding the left signal and the right signal, there are four conditions as follows depending on the combinations of whether the background sounds are equal and whether the noises are equal.
Condition 1: BL≠BR, eL=eR
Condition 2: BL≠BR, eL≠eR
Condition 3: BL=BR, eL=eR
Condition 4: BL=BR, eL≠eR
Conditions 1 and 2 are cases where the background sounds are different for the left signal and the right signal. For example, a stereo signal corresponds to Conditions 1 and 2. Conditions 3 and 4 are cases where the background sounds are equal between the left signal and the right signal. For example, a case where a monaural signal is input as a two-channel signal corresponds to Conditions 3 and 4.
Acoustic signals of TV broadcasting correspond, in many cases, to Condition 1. Acoustic signals recorded in some DVDs correspond to Condition 3. Other acoustic signals such as the acoustic signals of videos on the Internet include signals of various conditions, and it is not possible to grasp in advance to which condition an acoustic signal corresponds. Also, according to Condition 3, the left signal and the right signal perfectly match each other, and thus, recognition is easy. However, because of the influence of noises, it is generally difficult to distinguish Condition 4 from Conditions 1 and 2 based on input acoustic signals.
As described above, acoustic signals include signals of various conditions. However, the conventional technology of removing a speech signal from acoustic signals of two channels is effective only for the acoustic signals of Conditions 1 and 2, and is not capable of appropriately removing speech from the acoustic signals of Conditions 3 and 4. For example, speech cannot be removed from a monaural signal.
According to an embodiment, a signal processing device includes an acquirer, a first background sound calculator, a first signal generator, an extractor, a similarity calculator, and a mixer. The acquirer is configured to acquire a first acoustic signal and a second acoustic signal. The first background sound calculator is configured to calculate a first background sound signal in which a speech signal is removed, based on the first acoustic signal and the second acoustic signal. The first signal generator is configured to generate a first reference signal from at least one of the first acoustic signal and the second acoustic signal. The extractor is configured to extract a second background sound signal by removing a speech signal from the first reference signal. The similarity calculator is configured to calculate a first similarity indicating a degree of similarity between feature data of the first background sound signal and feature data of the second background sound signal. The mixer is configured to calculate a weighted sum of the first background sound signal and the second background sound signal in such a way that a greater weight is given to the first background sound signal as the first similarity is higher and a greater weight is given to the second background sound signal as the first similarity is lower.
Hereinafter, preferred embodiments of a signal processing device according the invention will be described in detail with reference to the drawings.
A signal processing device according to a first embodiment first calculates a background sound signal (for example, a difference signal) obtained by removing a speech signal from acoustic signals of two channels. Next, a reference signal in which the speech signal is removed is generated from the acoustic signals. Then, the similarity between the background sound signal and the reference signal is calculated, and a weighted sum of the background sound signal and the reference signal is calculated according to the weight according to the similarity. A background sound signal obtained by removing a speech signal from the acoustic signals is thereby generated also under a condition where the same background sound signal is included in the acoustic signals of two channels.
The acquirer 101, the first background sound calculator 102, the first signal generator 103, the extractor 104, the similarity calculator 105, and the mixer 106 may be realized by a processing device such as a CPU (Central Processing Unit) executing programs, that is, by software, or may be realized by hardware such as an IC (Integrated Circuit), or may be realized by a combination of software and hardware.
The acquirer 101 acquires acoustic signals of two channels, a first acoustic signal and a second acoustic signal.
The first background sound calculator 102 calculates a first background sound signal in which the speech signal is removed, from the first acoustic signal and the second acoustic signal. For example, the first background sound calculator 102 calculates a difference signal which is the difference between the first acoustic signal and the second acoustic signal as the first background sound signal. In the following, a case where the difference signal is used as the first background sound signal will be described as an example. Additionally, the calculation method of the first background sound signal is not restricted to the above, and any method that is conventionally used may be applied as long as the method allows calculation of the background sound signal with the first acoustic signal and the second acoustic signal as stereo signals. For example, it is possible to apply a method of calculating a similarity between left and right signals for each of frequency bands which have been divided, and suppressing a signal in a frequency band to a greater degree as the similarity is higher, to thereby calculate a background sound signal in which a signal localized at the center including speech is suppressed.
The first signal generator 103 generates a first reference signal from at least one of the first acoustic signal and the second acoustic signal. The extractor 104 extracts a second background sound signal by removing the speech signal from the first reference signal. The similarity calculator 105 calculates a first similarity indicating the degree of similarity between the difference signal and the second background sound signal. The mixer 106 calculates a weighted sum of the difference signal and the second background sound signal according to a weight determined by the first similarity.
Next, an operation of the signal processing device 100 will be described with reference to
First, the acquirer 101 acquires a first acoustic signal and a second acoustic signal (step S11). The acquirer 101 may acquire a first acoustic signal and a second acoustic signal which are acoustic signals of two channels, or may extract (acquire) a first acoustic signal and a second acoustic signal from video data including acoustic signals. Furthermore, the acquirer 101 may acquire a first acoustic signal and a second acoustic signal by selecting signals of two channels from acoustic signals of a larger number of channels, such as acoustic signals of 5.1 channels, for example, or by down-mixing acoustic signals of a large number of channels by a predetermined factor. In the present embodiment, the first acoustic signal is the left signal of acoustic signals of two channels, and the second acoustic signal is the right signal.
Next, the first background sound calculator 102 calculates a difference signal which is the difference between the first acoustic signal and the second acoustic signal (step S12). The difference signal is calculated by the following Equation (1) with the first acoustic signal as L and the second acoustic signal as R.
S=(L−R)/2 (1)
Then, the first signal generator 103 generates a first reference signal by one of the first acoustic signal, the second acoustic signal, and a weighted sum of the first acoustic signal and the second acoustic signal (step S13). In the following, the weighted sum of the first acoustic signal and the second acoustic signal is taken as the first reference signal. The first reference signal is calculated by the following Equation (2), for example. Additionally, the weight is not restricted to the example (½) of Equation (2).
M=(L+R)/2 (2)
Next, the extractor 104 extracts a second background sound signal by removing the speech signal from the first reference signal (step S14). For example, the extractor 104 extracts a second background sound signal from the first reference signal by sound source separation using nonnegative matrix factorization (NMF). An example of an extraction method that uses the nonnegative matrix factorization will be described below.
First, the extractor 104 Fourier-transforms a first reference signal from time t to time t+N−1, and obtains an amplitude spectrum and a phase spectrum of the first reference signal. Here, N is the number of samples that are the targets of Fourier transform, and is 2048, for example. Then, the extractor 104 reads a set of bases for representing the amplitude spectrum of the speech signal, and a set of bases for representing the amplitude spectrum of the background sound signal. These bases may be learned and prepared in advance by using the speech signal and the background sound signal. For example, the extractor 104 uses twenty bases. A matrix representation of the set of bases for representing the amplitude spectrum of the speech signal is given as Ev. Also, a matrix representation of the set of bases for representing the amplitude spectrum of the background sound signal is given as EB. Then, the extractor 104 factorizes, using the nonnegative matrix factorization, the amplitude spectrum of the first reference signal into the format of a factor and the bases which have been read, and obtains the value of the factor. This calculation is calculation of w that minimizes the value of the following Equation (3) where a vector indicating the amplitude spectrum of the first reference signal is given as p, a vector of a factor to be obtained is given as w, and a matrix in which Ev and EB are arrayed is given as E (=[Ev EB]).
∥p−Ew∥2 (3)
Specifically, the extractor 104 performs calculation of the following Equation (4).
Here, “•x” indicates an x-th component of the vector, and “•x,y” indicates a component at row x and column y of the matrix. Also, wk(n) is the value at the n-th repetition of calculation of wk. The extractor 104 repeatedly performs calculation of Equation (3) until the variation in the value of wk is at a predetermined value or less due to the repetition, or the repetition is performed a predetermined number of times. Additionally, as an initial value of repetition of wk(n), any value other than zero may be used. For example, a random number other than zero is used as the initial value.
Moreover, a factor regarding Ev is given as wv, and a factor regarding EB is given as WB. That is, the relationship of the following Equation (5) is established.
Next, the extractor 104 calculates the amplitude spectrum of the second background sound signal by using the factors obtained. The amplitude spectrum of the second background sound signal is calculated based on EBwB. The extractor 104 may calculate the amplitude spectrum of the speech signal and subtract the amplitude spectrum of the speech signal from the amplitude of the first reference signal to thereby calculate the amplitude spectrum of the second background sound signal. That is, the extractor 104 may calculate the amplitude spectrum of the second background sound signal by p−EvWv.
Lastly, the extractor 104 obtains the second background sound signal by performing inverse-Fourier transform using the calculated amplitude spectrum of the second background sound signal and the phase spectrum of the first reference signal.
Additionally, the extraction method of the second background sound signal is not restricted to the method described above. It is also possible to extract the second background sound signal from the first reference signal by using a band-pass filter that attenuates the speech.
When the processing for time t to time t+N−1 is over, extraction of the second background sound signal is repeatedly performed while changing the processing target time.
Next, the similarity calculator 105 calculates a first similarity which is the degree of similarity between feature data of the difference signal and feature data of the second background sound signal (step S15). An operation of the similarity calculator 105 will be described with reference to
As illustrated in
As illustrated in
Next, the level calculator 1201 calculates a first reference signal level which is the amplitude of a signal within a unit time for the first reference signal in the same manner as in S131 (step S132). Then, the level calculator 1201 calculates a second background sound signal level which is the amplitude of a signal within the unit time for the second background sound signal in the same manner as in S131 (step S133).
Next, the generator 1202 calculates the first similarity from the difference signal level and the second background sound signal level (step S134). For example, the first similarity takes a value between zero and one. The generator 1202 first calculates a ratio Rate between the difference signal level Lev(S) and the second background sound signal level Lev(A) by the following Equation (6).
Rate=Lev(S)/Lev(A) (6)
Then, the generator 1202 calculates the first similarity by using the Rate. The generator 1202 simply calculates the first similarity such that the value is greater as the value of the Rate is closer to 1. The generator 1202 calculates a first similarity Sim by the following Equation (7), for example. Here, β is a parameter of a positive number, and 0.5 is used, for example.
In the case the value of the Rate is smaller than a predetermined standard, the difference signal may be assumed to be a noise. On the other hand, in the case the value of the Rate exceeds one, it may be assumed that the difference signal level has become higher than the second background sound signal level because the second background sound signal has become smaller than the actual background sound due to the influence of insufficient extraction accuracy for the second background sound signal or the like. Accordingly, the value of the first similarity may be made one when the Rate exceeds one. That is, the first similarity is calculated by the following Equation (8).
Here, a case of using the amplitude of a signal as the feature data of the difference signal and of the second background sound signal is described. The first similarity may also be calculated by using a combination of pieces of feature data other than the amplitude of a signal and a calculation method of a distance Z between the pieces of feature data. For example, the generator 1202 may directly use signal values as the pieces of feature data, calculate the distance between the signal values of respective signals as Z, and calculate the first similarity based on the distance Z. For example, the generator 1202 calculates Z by the following Equation (9), and calculates Sim by the following Equation (10) using Z which has been calculated.
Here, A is the second background sound signal, “•(i)” is a signal value at time i, and Σ is a sum for the time i within a unit time. Also, the generator 1202 may calculate Sim based on the similarity regarding the pattern of the signal values. For example, the generator 1202 calculates the correlation between S and A, takes its inverse number as Z, and calculates Sim. Also, Sim may be calculated using, instead of the signal values, the similarity regarding the pattern of factors obtained by Fourier-transforming the signal values. For example, the generator 1202 may calculate the correlation between a plurality of factors obtained by Fourier-transforming the difference signal and the second background sound signal, and take its inverse number as Z. Also, the generator 1202 may calculate the correlation between the amplitude spectrum of the difference signal and the amplitude spectrum of the second background sound signal, and take its inverse number as Z.
According to the method described above, the pieces of feature data are scalar values, and the first similarity is calculated based on the similarity thereof. Vectors including two or more scalar values indicating the features of signals may be taken as the feature data, and the first similarity may be calculated based on the similarity thereof. For example, the generator 1202 may take vectors having two scalar values of Equations (6) and (9) as the feature data, and calculate the first similarity based on the weighted sum of Equations (8) and (10).
Next, the second similarity is calculated in the same manner as in step S134 by using the difference signal level and the first reference signal level (step S135). The second similarity is given as Sim2.
We will return to
Next, the similarity acquirer 1003 acquires the already calculated similarity which is the first similarity that is already calculated by the operation at a previous time (step S113). The already calculated similarity may be substituted by prior information obtained by using metadata such as metadata assigned to an acoustic signal in advance or metadata included in video content. For example, if information that video content is for stereo broadcasting is assigned, operation is possible with the already calculated similarity being one.
Next, the corrector 1004 corrects the first similarity based on the second similarity and the non-reliability (step S114). When the second similarity and the non-reliability are low, this is a case where the difference signal is assumed likely to be a noise, and the difference signal is assumed unlikely to be similar to the second background sound signal. On the other hand, when the second similarity and the non-reliability are high, the difference signal is not a noise, and thus, the difference signal is assumed likely to be similar to the second background sound signal. Thus, the first similarity is corrected based on the levels of the second similarity and the non-reliability. For example, the corrector 1004 gives parameters for adjusting the amounts of correction by the second similarity and the non-reliability as a and b, and corrects and replaces the first similarity by the value of the following Equation (11).
Sim+a(Sim2−0.5)+b(Bel−0.5) (11)
Additionally, the corrector 1004 may correct the first similarity by at least one of the second similarity and the non-reliability. In this case, for example, one of a and b is made zero, and the first similarity is calculated by Equation (11). Also, the corrector 1004 may replace the first similarity by the weighted sum of the first similarity, the second similarity and the non-reliability given by the following Expression (12). Here, d1, d2 and d3 are weight coefficients whose total sum is one.
d
1Sim+d2Sim2+d3Bel (12)
Furthermore, the parameters (a, b) for adjusting the amount of correction, and the weight coefficients (d1, d2, d3) may be controlled by the already calculated similarity. In the case the already calculated similarity is low (that is, the proportion of noise in the difference signal is high), and the noise is in proportion to the amplitude of the first reference signal, the amount of correction by the second similarity is preferably made greater. That is, a and d2 are made greater as the already calculated similarity is lower, and a and d2 are made smaller as the already calculated similarity is higher.
The first similarity of time t to time t+N−1 may be calculated by the method described above. The similarity calculator 105 calculates the first similarity for each time while shifting the time by s. For example, after performing calculation for time t to time t+N−1, the similarity calculator 105 calculates the first similarity for time t+s to time t+N−1+s (where s<N).
Since s is smaller than N, the ranges of time where the first similarity is calculated overlap each other. With respect to such overlapping ranges of time, the similarity calculator 105 may calculate the average value of the already calculated first similarity and the currently calculated first similarity as the first similarity of the time.
Furthermore, the first similarity may be smoothed in the time direction. That is, for example, the similarity calculator 105 calculates the first similarity of time t+s to time t+N−1+s by alpha-blending the same with the first similarity of time t to time t+N−1. The temporal variation in the first similarity is thereby smoothed, and an effect of preventing occurrence of a noise in a first output signal and a second output signal output in the present embodiment, or of suppressing shaky sound is achieved.
An example modification (a similarity calculator 105-2) of the similarity calculator will be described with reference to
The second signal generator 301 generates a third reference signal from the first reference signal and the second background sound signal. The level calculator 302 calculates a difference signal level and a third reference signal level as pieces of feature data of the difference signal and the third reference signal. The similarity generator 303 generates the first similarity from the difference signal level and the third reference signal level.
The flow chart of
Also, the weight may be controlled by the already calculated similarity which is the first similarity already calculated at previous time.
It is desirable that, when the already calculated similarity is high, the weight to be given to the second background sound signal is increased, and when the already calculated similarity is low, the weight to be given to the first reference signal is increased. When the already calculated similarity is low, the proportion of noise in the difference signal is expected to be high. Accordingly, the likelihood of a difference signal being a noise may be determined by comparing the feature data of the first reference signal and the feature data of the difference signal, and the calculation accuracy for the first similarity may be expected to be improved.
We will return to
Next, the similarity generator 303 calculates the first similarity from the difference signal level and the third reference signal level in the same manner as in step S134 (step S24).
Additionally, also in the case of determining the first similarity from the difference signal and the third reference signal, the calculation method of the pieces of feature data and the first similarity is not restricted to the method described above. The patterns of signal values, factors obtained by Fourier-transforming the signal values, and the scalar values or vector values formed from the patterns of the factors may be used as pieces of feature data, and the first similarity may be calculated by the similarity of the pieces of feature data.
We will return to
L
OUT
=αS+(1−α)B (13)
R
OUT
=αS+(1−α)B (14)
The weight α to be given to the difference signal is controlled to be greater as the first similarity is higher. For example, the value of the first similarity may be used as α as it is. That is, α is generated by the following Equation (15).
α=Sim (15)
The following Equation (16) may be used for calculation such that α is greater when the first similarity is closer to one. Here, γ is a parameter of a positive number. Also, the values of α corresponding to Sim may be held in a table.
The range of values of α is desirably between zero and one. Also, the upper limit value of α corresponding to Sim may be set to one or less. For example, α may take a value between zero and 0.5 according to the value of Sim.
Additionally, besides the calculation methods of the first output signal and the second output signal expressed by Equations (13) and (14), a difference signal of reversed phase may be added to one of the first output signal and the second output signal. That is, the first output signal and the second output signal may be calculated by the following Equations (17) and (18). An effect of increased stereo feeling of sound may thereby be achieved.
L
OUT
=αS+(1−α)B (17)
R
OUT=α(−S)+(1−α)B (18)
The mixer 106 outputs the first output signal and the second output signal to an external device, a storage device or the like. That mixer 106 may output both the first output signal and the second output signal, or may output one of the first output signal and the second output signal.
In this manner, according to the signal processing device of the first embodiment, a weighted sum of a difference signal and a second background sound signal is calculated according to the similarity between the feature data of the difference signal and the feature data of the second background sound signal. Then, the background sound may be appropriately output with respect to various input signals.
Additionally, the speech signal is human voice, for example, but is not restricted thereto, and it may be any signal as long as it may be separated from a background sound signal. For example, in the case of applying nonnegative matrix factorization or the like, an arbitrary signal may be separated as the speech signal by appropriately changing the speech signal and the background sound signal to be used in learning.
The functions of the extractor 604 and the mixer 606 of the second embodiment are different from those according to the first embodiment. Other configurations and functions are the same as those in
The extractor 604 extracts, from a first reference signal, a second background sound signal in which the speech signal is removed and the speech signal. The mixer 606 calculates a weighted sum of a difference signal, the second background sound signal and the speech signal according to a weight determined based on a first similarity.
Next, an operation of the signal processing device 200 of the second embodiment will be described with reference to
In step S75, the extractor 604 extracts the speech signal from the first reference signal (step S75). The speech signal is obtained by subtracting the second background sound signal from the first reference signal. The extractor 604 may also calculate the speech signal by calculating Evwv in the same manner as in step S14.
In step S77, the mixer 606 calculates a weighted sum of the difference signal, the second background sound signal and the speech signal, and generates the first output signal and the second output signal (step S77). First, the mixer 606 calculates a factor α for determining the ratio of weights of the difference signal and the second background sound signal based on the first similarity by the method described in step S16. Next, the mixer 606 acquires a factor λ for determining the amplitude of the background sound signal, and a factor μ for determining the amplitude of the speech signal. The values of λ and μ are zero or more, and may be determined in advance in such a way as to achieve a predetermined effect. For example, to make the speech signal easily audible, the value of μ is set to be greater than the value of λ. Also, in order to enable one to enjoy the ambience of a venue in a sports show or the like, the value of μ is made smaller than the value of λ such that the voice of the commentator is reduced and the background sound is increased.
Also, the values of λ and μ may be acquired by providing a factor acquirer for receiving a set value specified by a user, for example. Moreover, the values of λ and μ may be directly specified, or may be specified according to the ratio and the average levels of λ and μ.
The mixer 606 calculates the first output signal and the second output signal by the following Equations (19) and (20). Here, the speech signal is given as V.
L
OUT=λ(αS+(1−α)B)+μV (19)
R
OUT=λ(αS+(1−α)B)+μV (20)
The image display window 803 is a window for displaying an image of a video. The play button 804 is a button for starting playback of a video. The stop button 805 is a button for stopping playback of a video. The display bar 806 is a display bar for displaying the mixing ratio of the speech signal. The display bar 807 is a display bar for displaying the mixing rate of the background sound signal.
The display bar 806 includes a specification button 806-a for displaying the currently specified mixing ratio of the speech signal. The display bar 807 includes a specification button 807-a for displaying the currently specified mixing ratio of the background sound signal.
A user may specify the mixing ratio of the speech signal by touching the specification button 806-a and sliding the same in the lateral direction along the display bar 806. Likewise, a user may specify the mixing ratio of the background sound signal by the specification button 807-a. The mixing ratio of the speech signal and the mixing ratio of the background sound signal correspond to μ and λ in step S77, respectively. Thus, a user may set the factor λ and the factor μ to be used by the mixer 606 through a screen as in
The specification button 806-a indicates μMIN, which is the minimum value of μ determined in advance, when at the left end of the display bar 806, and indicates μMAX, which is the maximum value of μ determined in advance, when at the right end, and indicates an intermediate value when at the middle position. Like the specification button 806-a, the specification button 807-a corresponds to values from a minimum value λMIN and a maximum value λMAX of λ.
A user may freely set the mixing amounts of the speech signal and the background sound signal by moving the specification button 806-a and the specification button 807-a while watching a video. A desired acoustic signal may thereby be enjoyed according to the scene or content of a video.
As described above, the signal processing device 200 of the second embodiment calculates a weighted sum of the speech signal and a signal of the weighted sum of the difference signal and the second background sound signal calculated according to the weight according to the similarity of the feature data of the difference signal and the feature data of the second background sound signal. Accordingly, a signal in which the background sound and the speech are mixed at a predetermined ratio may be output with respect to various input signals.
As described above, according to the first and second embodiments, not only in the case of a stereo signal, but also in the case of a monaural signal or the like in which there is an equal background sound signal in acoustic signals, a background sound signal which is obtained by removing a speech signal from acoustic signals may be appropriately generated.
The third embodiment differs from the second embodiment in the function of the mixer 706 and in that the third background sound generator 707 is additionally provided. Other configurations and functions are the same as those in
Many contents are created such that speech signals are equally included in left signals and right signals. However, in a case where speakers speak from the left and from the right such as a homemade video taken by an amateur or recording using a stereo microphone, speech signals may be included in difference signals. Thus, the third background sound generator 707 removes a speech signal included in a difference signal.
The third background sound generator 707 generates a third background sound signal by further removing the speech signal from a first sound signal (such as a different signal). The generation of a third background sound signal can be performed similarly to extraction of a second background sound signal from a first reference signal by the extractor 104, for example.
Next, an operation of the signal processing device 300 of the third embodiment will be described with reference to
In step S87, the third background sound generator 707 generates the third background sound signal from the first background sound signal (step S87).
In step S88, the mixer 706 calculates a weighted sum of the third background sound signal, the second background sound signal and the speech signal, and generates the first output signal and the second output signal (step S88).
First, the mixer 706 calculates a factor α for determining the ratio of weights of the third background signal and the second background sound signal based on the first similarity by the method described in step S16. Next, the mixer 706 acquires a factor λ for determining the amplitude of the background sound signal, and a factor μ for determining the amplitude of the speech signal.
The mixer 706 calculates the first output signal and the second output signal by the following Equations (21) and (22) by using the third background sound signal. Here, the third background sound signal is given as B′.
L
OUT=λ(αB′+(1−α)B)+μV (21)
R
OUT=λ(αB′+(1−α)B)+μV (22)
As described above, the signal processing device 300 of the third embodiment uses the third background sound signal by further removing the speech signal from the difference signal, which allows speech to be removed in more contents.
The fourth embodiment differs from the third embodiment in the functions of the extractor 904, the similarity calculator 905, the mixer 906, and the third background sound generator 907 and in that the setter 908 is additionally provided. Other configurations and functions are the same as those in
The third embodiment in which the third background sound generator 707 is additionally provided effective when importance is placed on the background sound signal in generating the output signal, but cannot be utilized and increases the cost when importance is placed on the speed signal. Thus, in the fourth embodiment, whether or not to simplify the processing of the extractor 904 and whether or not to simplify the processing of the third background sound generator 907 are controlled depending on a sound source on which importance is placed in generating an output signal to reduce the calculation cost while maintaining the quality of the output signal.
Whether or not to simplify the processing of the extractor 904 and the third background sound generator 907 may be controlled according to the conditions of
Referring back to
When the factor μ is set to 0 or when λ−μ is equal to or larger than a threshold λTH, the setter 908 determines that importance is placed on the background signal in the generation of the output signal and determines the output sound source to be the background sound signal. Here, the threshold λTH can be set to any positive value such as half the maximum value λMAX. When the factor μ is not 0 and when λ−μ is smaller than the threshold λTH, the setter 908 determines the output source to be the speech signal. In addition, the setter 908 may set the output sound source information to be a one dimensional value expressing the distance to the background sound signal. In this case, the value of the sound source information is set to be proportional to λ−μ or λ/μ with a certain maximum value.
Next, an operation of the signal processing device 400 of the fourth embodiment will be described with reference to
In step S94, the similarity calculator 905 initializes the first similarity. The initial value may be set to 0, for example (step S94).
Next, in step S95, the setter 908 sets the output sound source by using the values of the factor λ and the factor μ used for generation of the output signal (step S95).
In step S96, the extractor 904 extracts the second background sound signal from the first reference signal based on whether or not the output sound source is a background sound signal and the magnitude of the first similarity, or based on the value representing the distance to the background sound signal and the magnitude of the first similarity (step S96). For example, the extractor 904 simplifies the processing as the weighted linear sum of the magnitude of the first similarity and the distance to the background sound of the output sound source is larger. The extractor 904 simplifies the processing by reducing the number of times of repetition of Equation (3), for example. Alternatively, the extractor 904 may simplify the processing by using a band-pass filter that reduces the speech.
Note that extractor 904 controls whether or not to simplify the processing by using the first similarity (calculated similarity, etc.) calculated at time before the processing target time.
Next, in step S97, the extractor 904 extracts the speech signal from the first reference signal (step S97). The extractor 904 may extract the speech signal by the same method as that of the extractor 604.
In step S98, the similarity calculator 905 calculates the first similarity by using the feature data of the difference signal, the feature data of the second background sound signal, and the feature data of the first reference signal (step S98). The similarity calculator 905 may calculate the similarity by the same method as that of the similarity calculator 105. The extractor 904, the mixer 906, and the third background sound generator 907 refer to the latest similarity calculated by the similarity calculator 905 to perform the respective processes.
In step S99, the third background sound generator 907 generates the third background signal from the first background signal based on whether or not the output sound source is a background sound signal and the magnitude of the first similarity, or based on the value representing the distance to the background sound signal and the magnitude of the first similarity (step S99). For example, the third background sound generator 907 simplifies the processing as the weighted linear sum of the magnitude of the first similarity and the distance to the background sound of the output sound source is smaller. The third background sound generator 907 performs the same processing as the extraction of the second background sound signal, and simplifies the processing by reducing the number of times of repetition of Equation (3), for example. Alternatively, the third background sound generator 907 may simplify the processing by using a band-pass filter that reduces the speech. The third background sound generator 907 may also simplify the processing by outputting the difference signal as the third background sound signal without any change.
Lastly, in step S100, the mixer 906 calculates a weighted sum of the third background sound signal and the second background sound signal, and generates the first output signal and the second output signal (step S100). The mixer 906 calculates the first output signal and the second output signal by the Equations (21) and (22) by using the third background sound signal similarly to the mixer 706 by using the factor λ for determining the amplitude of the background sound signal and the factor μ for determining the amplitude of the speech signal used by the setter 908.
As described above, the signal processing device 400 of the fourth embodiment gives priority to processing relating to generation or extraction of a signal with the largest weight of the third background sound signal, the second background sound signal and the speech signal relating to the output signal, which can reduce the calculation cost while maintaining the quality.
Next, a hardware configuration of the signal processing device according to the first to fourth embodiments will be described with reference to
The signal processing device according to the first to fourth embodiments includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 or a RAM (Random Access Memory) 53, a communication I/F 54 for connecting to a network and performing communication, and a bus 61 connecting each unit.
Programs to be executed by the signal processing device according to the first to fourth embodiments are provided being embedded in the ROM 52 or the like in advance.
The programs to be executed by the signal processing device according to the first to fourth embodiments may also be provided a computer program product by being recorded, as a file in an installable or executable format, in a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), a DVD (Digital Versatile Disk) or the like.
Furthermore, the programs to be executed by the signal processing device according to the first to fourth embodiments may be provided by being stored on a computer connected to a network such as the Internet, and being downloaded via the network. Also, the programs to be executed by the signal processing device according to the first to fourth embodiments may be provided or distributed via a network such as the Internet.
The programs to be executed by the signal processing device according to the first to fourth embodiments may cause a computer to function as each unit of the signal processing device described above. This computer may be realized by the CPU 51 reading the programs from a computer-readable recording medium onto a main memory device.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
Number | Date | Country | Kind |
---|---|---|---|
2012-277999 | Dec 2012 | JP | national |
2013-235396 | Nov 2013 | JP | national |