The present invention relates to a technique capable of enhancing sounds in a desired narrow range (sound enhancement technique).
When a movie shooting device (video camera or camcorder), for example, equipped with a microphone is zoomed in on a subject to shoot the subject, it is preferable for video recording that only sounds from around the subject should be enhanced in synchronization with the zoom-in shooting. Techniques (sharp directive sound enhancement techniques) to enhance sounds in a narrow range including a desired direction (a target direction) have been studied and developed. The sensitivity of a microphone pertinent to directions around the microphone is called directivity. When the directivity in a particular direction is sharp, sounds arriving from a narrow range including the particular direction are enhanced and sounds outside the range are suppressed. Three conventional techniques relating to the sharp directive sound enhancement technique will be described here first. The term “sound(s)” as used herein is not limited to human voice but refers to “sound(s)” in general such as music and ambient noise as well as calls of animals and human voice.
[1] Sharp Directive Sound Enhancement Technique Using Physical Properties
Typical examples of this category include shotgun microphones and parabolic microphones. The principle of an acoustic tube microphone 900 will be described first with reference to
The principle of a parabolic microphone 910 will be described next with reference to
[2] Sharp Directive Sound Enhancement Technique Using Signal Processing
Typical examples of this category include phased microphone arrays (see non-patent literature 1).
[3] Sharp Directive Sound Enhancement Technique by Selective Pickup of Reflected Sounds
Typical examples of this category include multi-beam forming (see non-patent literature 2). The multi-beam forming is a sharp directive sound enhancement technique that collects individual sounds, including direct sounds and reflected sounds, together to pick up sounds arriving from a target direction with a high signal-to-noise ratio and has been studied more intensively in the field of wireless rather than acoustics.
Processing of the multi-beam forming in a frequency domain will be described below. Symbols will be defined prior to the description. The index of a frequency is denoted by a and the index of a frame-time number is denoted by k. Frequency domain representations of analog signals received at M microphones are denoted by X→ (ω, k)=[X1(ω, k), . . . , XM(ω, k)]T, the direction from which a direct sound from a sound source located in a direction θs to be enhanced is denoted by θs1, the directions from which reflected sounds arrive is denoted by θs2, . . . , θsR. Here, T represents transpose and R−1 is the total number of reflected sounds. A filter that enhances a sound from a direction θsr is denoted by W→(ω, θsr). Here, r is an integer that satisfies 1≦r≦R.
A precondition for the multi-beam forming is that the directions from which direct and reflected sounds arrive and their arrival times are known. That is, the number of objects, such as walls, floors, reflectors, that are obviously expected to reflect sounds is equal to R−1. The number of reflected sounds, R−1, is often set at a relatively small value such as 3 or 4. This is based on the fact that there is a high correlation between a direct sound and a low-order reflected sound. Since the multi-beam forming enhances individually sounds and synchronously adds the enhanced signals, an output signal Y(ω, k, θs) can be given by equation (1). Here, H represents Hermitian transpose.
Delay-and-sum beam forming will be described as a method for designing a filter W→(ω, θsr). Assuming that direct and reflected sounds arrive as plane waves, then filter W→(ω, θsr) can be given by equation (2).
where, h→(ω, θsr)=[h1(ω, θsr), . . . , hM(ω, θsr)]T is a propagation vector of a sound arriving from a direction θsr.
Assuming that plane waves arrive at a linear microphone array (a microphone array in which M microphones are linearly arranged), then the elements hm(ω, θsr) that make up h→(ω, θsr) can be given by equation (3).
where m is an integer that satisfies 1≦m≦M, c is the speed of sound, u represents the distance between adjacent microphones, j is an imaginary unit, and τ(θsr) represents a time delay between a direct sound and a reflected sound arriving from the direction θsr.
Lastly, an output signal Y(ω, k, θs) is transformed to a time domain to obtain a signal in which a sound from the sound source located in the target direction θs is enhanced.
Step 1
An AD converter 110 converts analog signals output from M microphones 100-1, . . . , 100-M to digital signals x→(t)=[x1(t), . . . , xM(t)]T. Here, t represents the index of a discrete time.
Step 2
A frequency-domain transform section 120 transforms the digital signal of each channel to a frequency-domain signal by a method such as fast discrete Fourier transform. For example, for the m-th (1≦m≦M) microphone, signals xm((k−1) N+1), . . . , xm(kN) at N sampling points are stored in a buffer. Here, N is approximately 512 in the case of sampling at 16 KHz. Fast discrete Fourier transform of the analog signals of M channels stored in the buffer is performed to obtain frequency-domain signals X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T.
Step 3
Each of enhancement filtering sections 130-r (1≦r≦R) applies a filter W→H(ω, θsr) for a direction θsr to the frequency-domain signals X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T and outputs a signal Zr(ω, k) in which a sound from the direction θsr is enhanced. That is, each enhancement filtering section 130-r (1≦r≦R) performs processing given by equation (4):
Zr(ω, k)={right arrow over (W)}H(ω, θsr){right arrow over (X)}(ω,k) (4)
An adder 140 takes inputs of the signals Z1(ω, k), . . . , ZR(ω, k) and outputs a sum signal Y(ω, k). The addition can be given by equation (5):
Step 5
A time-domain transform section 150 transforms the sum signal Y(ω, k) to a time domain and outputs a time-domain signal y(t) in which the sound from the direction θs is enhanced.
In some situations, for example in a situation where there are multiple sound sources in about the same direction at different distances from a microphone, it may be desired that sounds arriving from the sound sources be selectively enhanced by the sharp directive sound enhancement technique. Consider a situation where a movie shooting device equipped with microphone is zoomed in on a subject to shoot the subject as in the example described earlier. If there is a sound source (referred to as the “rear sound source) in the rear of the focused subject (referred to as the “focused sound source”) in the range of the directivity of the microphone, a sound from the focused sound source and a sound from the rear sound source are mixed and enhanced, giving viewers an unnatural listening experience. Therefore, a technique capable of enhancing sounds in a narrow range including a desired direction according to distances from a microphone (a sound spot enhancement technique) is desired. Three conventional techniques relating to the sound spot enhancement technique will be described by way of illustration.
According to the sharp directive sound enhancement technique described in category [1], a sound arriving from a target direction cannot be enhanced unless the microphone itself is pointed to the target direction, as can be seen from the examples of the acoustic tube microphones and the parabolic microphones. That is, when the target direction can vary, driving and control means for changing the orientation of the acoustic tube microphone or the parabolic microphone itself is needed unless a human physical action is used. Furthermore, while the parabolic microphone excels in high-SN ratio sound pickup because the parabolic microphone can focus the energy of sounds reflected by the parabolic reflector on the focal point, it is difficult for the parabolic microphone as well as the acoustic tube microphone to achieve a high directivity, for example a visual angle of approximately 5° to 10° (a sharp directivity of an angle of approximately ±5° to ±10° with respect to a target direction).
According to the sharp directive sound enhancement technique described in category [2], in order to achieve a higher directivity, more microphones and a larger array size (a larger full length of array) are required. It is not realistic to increase the array size unlimitedly, because of a restricted space where the phased microphone array is placed, costs, and the number of microphones capable of performing real-time processing. For example, microphones available on the market are capable of real-time processing of up to approximately 100 signals. The directivity that can be achieved with a phased microphone array with about 100 microphones is approximately ±30° with respect to a target direction and therefore it is difficult for a phased microphone array to enhance a sound from a target direction with a sharp directivity of approximately ±5° to ±10°, for example. Furthermore, it is difficult for the conventional technique in category [2] to pick up a sound from a target direction with a high SN ratio so that the sound is not buried in sounds from other directions than the target direction.
According to the sharp directive sound enhancement technique described in category [3], while a sound from a target direction can be picked up with a high SN ratio so that the sound is not buried in sounds from directions other than the target direction and sounds from any directions can be enhanced without needing the driving and control means mentioned above, it is difficult for the technique to achieve a high directivity. In particular, human voice includes a high proportion of frequency components in a range from approximately 100 Hz to approximately 2 kHz. However, it is difficult for the conventional technique in category [3] to achieve a sharp directivity of approximately ±5° to ±10° in a target direction in such a low frequency band.
The sound spot enhancement technique described in (1) does not take any measures for protecting against interference sources because the technique uses the delay-and-sum array method. The sound spot enhancement technique described in (2) requires a plurality of microphone arrays and therefore can be disadvantageous because of the increased size of and cost of the system. The increased size of the microphone arrays restricts the installation and conveyance of the arrays. Information concerning reverberation varies with environmental changes and it is difficult for the sound spot enhancement technique described in (3) to robustly respond to such environmental changes.
In light of these circumstances, a first object of the present invention is to provide a sound enhancement technique (a sound spot enhancement technique) that can pick up a sound with a sufficiently high SN ratio and follow a sound from any direction without needing physically moving a microphone, and yet has a sharper directivity in a desired direction than the conventional techniques and can enhance sounds according to the distances from the microphone array. A second object of the present invention is to provide a sound enhancement technique (a sharp directive sound enhancement technique) that can pick up a sound with a sufficiently high SN ratio, can follow a sound from any direction without needing physically moving a microphone, and yet has a sharper directivity in a desired direction than the conventional techniques.
(Sound Spot Enhancement Technique)
A transmission characteristic ai,g of a sound that comes from each of one or more positions that are assumed to be sound sources (where i denotes the direction and g denotes the distance for identifying each position) and arrives at microphones (the number of microphones M≧2) is used to obtain a filter for a position that is a target of sound enhancement [a filter design process]. Each transmission characteristic ai,g is represented by the sum of transfer functions of a direct sound that comes from a position determined by a direction i and a distance g and directly arrives at the M microphones and transfer functions of one or more reflected sounds that is produced by reflection of the direct sound off an reflective object and arrives at the M microphones. The filter is designed to be applied, for each frequency, to a frequency-domain signal transformed from each of M picked-up signals obtained by picking up sounds with the M microphones. The filter obtained as a result of the filter design process is applied to a frequency-domain signal for each frequency to obtain an output signal [a filter application process]. The output signal is a frequency-domain signal in which the sound from the position that is the target of sound enhancement is enhanced.
Each transmission characteristic ai,g may be, for example, the sum of a steering vector of a direct sound and a steering vector(s) of one or more reflected sounds whose decays due to reflection and arrival time differences from the direct sound have been corrected or may be obtained by measurements in a real environment.
In the filter design process, a filter may be obtained for each frequency such that the power of sounds from positions other than the position that is the target of sound enhancement is minimized. Alternatively, a filter may be obtained for each frequency such that the SN ratio of a sound from the position that is the target of sound enhancement is maximized. Alternatively, a filter may be obtained for each frequency such that the power of sounds from positions other than one or more positions that are assumed to be sound sources is minimized while a filter coefficient for one of the M microphones is maintained at a constant value.
Alternatively, the filter may be obtained for each frequency in the filter design process such that the power of sounds from positions other than the position that is the target of sound enhancement and suppression points is minimized on conditions that (1) the filter passes sounds in all frequency bands from the position that is the target of sound enhancement and that (2) the filter suppresses sounds in all frequency bands from one or more suppression points. Alternatively, the filter may be obtained for each frequency by normalizing a transmission characteristic as,h of a sound from the position at i=s, g=h that is the target of sound enhancement. Alternatively, a filter may be obtained for each frequency by using a spatial correlation matrix represented by transfer functions ai,g corresponding to positions other than the position that is the target of sound enhancement. Alternatively, the filter may be obtained for each frequency such that the power of sounds from positions other than the position that is the target of sound enhancement is minimized on condition that the filter reduces the amount of decay of a sound from the position that is the target of sound enhancement to a predetermined value or less. Alternatively, a filter may be obtained for each frequency by using a spatial correlation matrix represented by frequency-domain signals obtained by transforming signals obtained by observation with a microphone array. Alternatively, a filter may be obtained for each frequency by using a spatial correlation matrix represented by transfer functions ai,g corresponding to each of one or more positions that are assumed to be sound sources.
(Sharp Directive Sound Enhancement Technique)
A transmission characteristic aθ of a sound that comes from each of one or more directions from which sounds assumed to come and arrives at microphones (the number of microphones M≧2) is used to obtain a filter for a position that is a target of sound enhancement [a filter design process]. Each transmission characteristic aθ is represented by the sum of transfer functions of a direct sound that comes from a direction θ and directly arrives at the M microphones and transfer functions of one or more reflected sounds that is produced by reflection of the direct sound off an reflective object and arrives at the M microphones. The filter is designed to be applied, for each frequency, to a frequency-domain signal transformed from each of M picked-up signals obtained by picking up sounds with the M microphones. The filter obtained as a result of the filter design process is applied to a frequency-domain signal for each frequency to obtain an output signal [a filter application process]. The output signal is a frequency-domain signal in which the sound from the position that is the target of sound enhancement is enhanced.
Each transmission characteristic aθ may be, for example, the sum of a steering vector of a direct sound and a steering vector(s) of one or more reflected sounds whose decays due to reflection and arrival time differences from the direct sound have been corrected or may be obtained by measurements in a real environment.
In the filter design process, a filter may be obtained for each frequency such that the power of sounds from directions other than the direction that is the target of sound enhancement is minimized. Alternatively, a filter may be obtained for each frequency such that the SN ratio of a sound from the direction that is the target of sound enhancement is maximized. Alternatively, a filter may be obtained for each frequency such that the power of sounds from directions from which sounds are likely to arrive is minimized while a filter coefficient for one of the M microphones is maintained at a constant value.
Alternatively, the filter may be obtained for each frequency in the filter design process such that the power of sounds from directions other than the direction that is the target of sound enhancement and null directions is minimized on conditions that (1) the filter passes sounds in all frequency bands from the direction that is the target of sound enhancement and that (2) the filter suppresses sounds in all frequency bands from one or more null directions. Alternatively, the filter may be obtained for each frequency by normalizing a transmission characteristic as of a sound from the direction θ=s that is the target of sound enhancement. Alternatively, a filter may be obtained for each frequency by using a spatial correlation matrix represented by transfer functions aφ corresponding to directions other than the direction that is the target of sound enhancement. Alternatively, the filter may be obtained for each frequency such that the power of sounds from directions other than the direction that is the target of sound enhancement is minimized on condition that the filter reduces the amount of decay of a sound from the direction that is the target of sound enhancement to a predetermined value or less. Alternatively, a filter may be obtained for each frequency by using a spatial correlation matrix represented by frequency-domain signals obtained by transforming signals obtained by observation with a microphone array.
(Sound Spot Enhancement Technique)
Since the sound spot enhancement technique of the present invention uses not only a direct sound from a desired direction but also reflected sounds, the sound spot enhancement technique is capable of picking up sounds with a sufficiently high SN ratio from the direction. Furthermore, the sound spot enhancement technique of the present invention is capable of following a sound in any direction without needing to physically move the microphone because sound enhancement is accomplished by signal processing. Moreover, since each transmission characteristic ai,g is represented by the sum of the transmission characteristic of a direct sound that comes from the position determined by a direction i and a distance g and directly arrives at M microphones and the transmission characteristic(s) of one or more reflected sounds that are produced by reflection of the sound off an reflective object and arrive at the M microphones, a filter that increases the degree of suppression of coherence which determines the degree of directivity in a desired direction can be designed to typical filter design criteria, as will be described later in further detail in the <<Principle of Sound Spot Enhancement Technique>> section. That is, a sharper directivity in a desired direction can be achieved than was previously possible. Since reflected sounds are used as will be described later in further detail in the <<Principle of Sound Spot Enhancement Technique>> section, there are significant differences in transmission characteristic among sounds from different positions at different distances in about the same direction as viewed from the microphone array. By extracting the differences among transfer functions by beam forming, sounds in a narrow range including a desired direction can be enhanced according to distances from the microphone array.
(Sharp Directive Sound Enhancement Technique)
Since the sharp directive sound enhancement technique of the present invention uses not only a direct sound from a desired direction but also reflected sounds, the sharp directive sound enhancement technique is capable of picking up sounds with a sufficiently high SN ratio from the direction. Furthermore, the sharp directive sound enhancement technique of the present invention is capable of following a sound in any direction without needing to physically move the microphone because sound enhancement is accomplished by signal processing. Moreover, since each transmission characteristic aφ is represented by the sum of the transmission characteristic of a direct sound that comes from a direction φ and directly arrives at M microphones and the transmission characteristic(s) of one or more reflected sounds that are produced by reflection of the sound off an reflective object and arrive at the M microphones, a filter that increases the degree of suppression of coherence which determines the degree of directivity in a desired direction can be designed to typical filter design criteria, as will be described later in further detail in the <<Principle of Sharp Directive Sound Enhancement>> section. That is, a sharper directivity in a desired direction can be achieved than was previously possible.
A sharp directive sound enhancement technique will be described first and then a sound spot enhancement technique will be described.
<<Sharp Directive Sound Enhancement Technique>>
A principle of a sharp directive sound enhancement technique of the present invention will be described. The sharp directive sound enhancement technique of the present invention is based on the nature of a microphone array technique being capable of following sounds from any direction on the basis of signal processing and positively uses reflected sounds to pick up sounds with a high SN ratio. One feature of the present invention is a combined use of reflected sounds and a signal processing technique that enables a sharp directivity.
Prior to the description, symbols will be defined again. The index of a discrete frequency is denoted by ω (The index ω of a discrete frequency may be considered to be an angular frequency ω because a frequency f and an angular frequency ω satisfies the relation ω=2πf. With regard to ω, the “index of a discrete frequency” may be also sometimes simply referred to as a “frequency”) and the index of frame-time number is denoted by k. Frequency-domain representation of a k-th frame of an analog signal received at M microphones is denoted by X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T and a filter that enhances a frequency-domain signal X→(ω, k) of a sound from a target direction θs as viewed from the center of a microphone array with a frequency ω is denoted by X→(ω, θs), where M is an integer greater than or equal to 2 and T represents the transpose. Then, a frequency-domain signal Y(ω, k, θs) resulting from the enhancement of the frequency-domain signal X→(ω, k) of the sound from the target direction θs with the frequency ω (hereinafter the resulting signal is referred to as an output signal) can be given by equation (6):
Y(ω, k, θs)={right arrow over (W)}H(ω, θs){right arrow over (X)}(ω, k) (6)
where H represents the Hermitian transpose.
While the “center of a microphone array” can be arbitrarily determined, typically the geometrical center of the array of the M microphones is treated as the “center of a microphone array”. In the case of a linear microphone array, for example, the point equidistant from the microphones at the both ends of the array is treated as the “center of the microphone array”. In the case of a planar microphone array in which microphones are arranged in a square matrix of m×m (m2=M), for example, the position at which the diagonals linking the microphones at the corners intersect is treated as the “center of the microphone array”.
A filter W→(ω, θs) may be designed in various ways. A design using minimum variance distortionless response (MVDR) method will be described here. In the MVDR method, a filter W→(ω, θs) is designed so that the power of sounds from directions other than a target direction θs (hereinafter sounds from directions other than the target direction θs will be also referred to as “noise”) is minimized at a frequency ω (see equation (7)) by using a spatial correlation matrix Q(ω) under the constraint condition of equation (8). Transfer functions at a frequency ω between a sound source and the M microphones is denoted by a→(ω, θs)=[a1(ω, θs), . . . , aM(ω, θs)]T, where the sound source is assumed to be in a direction θs. In other words, a→(ω, θs)=[a1(ω, θs), . . . , aM(ω, θs)]T represents transfer functions of a sound from the direction θs to the microphones included in the microphone array at frequency ω. The spatial correlation matrix Q(ω) represents the correlation among components X1(ω, k), . . . , XM(ω, k) of a frequency-domain signal X→(ω, k) at frequency ω and has E[Xi(ω, k)Xj * (ω, k) (1≦i≦M, 1≦j≦M) as its (i, j) elements. The operator E[•] represents a statistical averaging operation and the symbol * is a complex conjugate operator. The spatial correlation matrix Q(ω) can be expressed using statistics values of X1(ω, k), . . . , XM(ω, k) obtained from observation or may be expressed using transfer functions. The latter case, where the spatial correlation matrix Q(ω) is expressed using transfer functions, will be described momentarily hereinafter.
It is known that the filter W→(ω, θs) which is an optimal solution of equation (7) can be given by equation (9) (see Reference 1 listed later).
As will be appreciated from the fact that the inverse matrix of the spatial correlation matrix Q(ω) is included in equation (9), the structure of the spatial correlation matrix Q(ω) is important for achieving a sharp directivity. It will be appreciated from equation (7) that the power of noise depends on the structure of the spatial correlation matrix Q(ω).
A set of indices p of directions from which noise arrives is denoted by {1, 2, . . . , P−1}. It is assumed that the index s of the target direction θs does not belong to the set {1, 2, . . . , P−1}. Assuming that P−1 noises come from arbitrary directions, the spatial correlation matrix Q(ω) can be given by equation (10a). In order to design a filter that sufficiently functions in the presence of many noises, it is preferable that P be a relatively large value. It is assumed here that P is an integer on the order of M. While the description is given as if the target direction θs is a constant direction (and therefore directions other than the target direction θs are described as directions from which noise arrives) for the clarity of explanation of the principle of the sharp directive sound enhancement technique of the present invention, the target direction θs in reality may be any direction that can be a target of sound enhancement. Usually, a plurality of directions can be target directions θs. In this light, the differentiation between the target direction θs and noise directions is subjective. It is more correct to consider that one direction selected from P different directions that are predetermined as a plurality of possible directions from which whatever sounds, including a target sound or noise, may arrive is the target direction and the other directions are noise directions. Therefore, the spatial correlation matrix Q(ω) can be represented by transfer functions a→(ω, θφ)=[a1(ω, θφ), . . . , aM(ω, θφ)]T (φ ε Φ) of sounds that come from directions θφ included in a plurality of possible directions from which sounds may arrive to the microphones and can be written as equation (10b), where Φ is the union of set {1, 2, . . . , P−1} and a set {s}. Note that |Φ|=P and |Φ| represents the number of elements of the set Φ.
Here, it is assumed that the transmission characteristic a→(ω, θs) of a sound from the target direction θs and the transfer functions a→(ω, θp)=[a1(ω, θp), . . . , aM(ω, θp)]T of sounds from directions p ε{1, 2, . . . , P−1} are orthogonal to each other. That is, it is assumed that there are P orthogonal basis systems that satisfy the condition given by equation (11). The symbol ⊥ represents orthogonality. If A→⊥B→, the inner product of vectors A→ and B→ is zero. It is assumed here that P≦M. Note that if the condition given by equation (11) can be relaxed to assume that there are P basis systems that can be regarded approximately as orthogonal basis systems, P is preferably a value on the order of M or a relatively large value greater than or equal to M.
{right arrow over (a)}(ω, θs)⊥{right arrow over (a)}(ω, θ1)⊥ . . . ⊥{right arrow over (a)}(ω, θP-1) (11)
Then, the spatial correlation matrix Q(ω) can be expanded as equation (12). Equation (12) means that the spatial correlation matrix Q(ω) can be decomposed into a matrix V(ω)=[a→(ω, θs), a→(ω, θ1), . . . , a→(ω, θp-1)]T made up of P transfer functions that satisfy orthogonality and a unit matrix Λ(ω). Here, ρ is an eigenvalue of a transmission characteristic a→(ω, θφ) that satisfies equation (11) for the spatial correlation matrix Q(ω) and is a real value.
Q(ω)=ρ{right arrow over (V)}(ω){right arrow over (Λ)}(ω){right arrow over (V)}H(ω) (12)
Then, the inverse matrix of the spatial correlation matrix Q(ω) can be given by equation (13).
Substitution of equation (13) into equation (7) shows that the power of noise is minimized. If the power of noise is minimized, it means that the directivity in the target direction θs is achieved. Therefore, orthogonality between the transfer functions of sounds from different directions is an important condition for achieving directivity in the target direction θs.
The reason why it is difficult for conventional techniques to achieve a sharp directivity in a target direction θs will be discussed below.
Conventional techniques assumed in designing filters that transfer functions were made up of those of direct sounds. In reality, there are reflected sounds that are produced by reflection of sounds from the same sound source off surfaces such as walls and a ceiling and arrive at microphones. However, the conventional techniques regarded reflected sounds as a factor that degrade directivity and ignored the presence of reflected sounds. In the conventional techniques, transfer functions a→conv(ω, θ)=[a1(ω, θ), . . . , aM(ω, θ)]T were treated as a→conv(ω, θ)=h→d(ω, θ), where h→d(ω, θ)=[hd1(ω, θ), . . . , hdM(ω, θ)]T represents steering vectors of only a direct sound arriving from a direction θ. Note that a steering vector is a complex vector where phase response characteristics of microphones at a frequency ω with respect to a reference point are arranged for a sound wave from a direction θ viewed from the center from the microphone array.
Assuming that sounds arrive at a linear microphone array as plane waves, an m-th element hdm(ω, θ) of the steering vector h→d(ω, θ) of a direct sound is given by, for example, equation (14a), where m is an integer that satisfies 1≦m≦M, c represents the speed of sound, u represents the distance between adjacent microphones, j is an imaginary unit. The reference point is the midpoint of the full-length of the linear microphone array (the center of the linear microphone array). The direction θ is defined as the angle formed by the direction from which a direct sound arrives and the direction in which the microphones included in the linear microphone array, as viewed from the center of the linear microphone array (see
The inner product γconv(ω, θ) of a transmission characteristic of a direction θ and a transmission characteristic of a target direction θs can be given by equation (15), where θ≠θs.
Hereinafter, γconv(ω, θ) is referred to as coherence. The direction θ in which the coherence γconv(ω, θ) is 0 can be given by equation (16), where q is an arbitrary integer, except 0. Since 0<θ<π/2, the range of q is limited for each frequency band.
Since only parameters relating to the size of the microphone array (M and u) can be changed in equation (16), it is difficult to reduce the coherence γconv(ω, θ) without changing any of the parameters relating to the size of the microphone array if the difference (angular difference) |θ−θs| between directions is small. If this is the case, the power of noise is not reduced to a sufficiently small value and directivity having a wide beam width in the target direction θs as schematically illustrated in
The sharp directive sound enhancement technique of the present invention is based on the consideration described above and is characterized by positively taking into account reflected sounds, unlike in the conventional technique, on the basis of an understanding that in order to design a filter that provides a sharp directivity in the target direction θs, it is important to enable the coherence to be reduced to a sufficiently small value even when the difference (angular difference) |θ−θs| between directions is small.
Two types of plane waves, namely direct sounds from a sound source and reflected sounds produced by reflection of that sound off a reflective object 300, together enter the microphones of a microphone array. Let the number of reflected sounds be denoted by Ξ. Here, Ξ is a predetermined integer greater than or equal to 1. Then, a transmission characteristic a→(ω, θ)=[a1(ω, θ), . . . , aM(ω, θ)]T can be expressed by the sum of a transmission characteristic of a direct sound that comes from a direction that can be a target of sound enhancement and directly arrives at the microphone array and the transmission characteristic(s) of one or more reflected sounds that are produced by reflection of that sound off a reflective object and arrive at the microphone array. Specifically, the transmission characteristic can be represented as the sum of the steering vector of the direct sound and the steering vector of Ξ reflected sounds whose decays due to reflection and arrival time differences from the direct sound are corrected, as shown in equation (17a), where τξ(θ) is the arrival time difference between the direct sound and a ξ-th (1≦ξ≦Ξ) reflected sound and αξ (1≦ξ≦Ξ) is a coefficient for taking into account decays of sounds due to reflection. Here, h→rξ(ω, θ)=[hr1ξ(ω, θ), . . . , hrMξ(ω, θ)]T represents the steering vectors of reflected sounds corresponding to the direct sound from direction θ. Typically, αξ (1≦ξ≦Ξ) is less than or equal to 1 (1≦ξ≦Ξ). For each reflected sound, if the number of reflections in the path from the sound source to the microphones is 1, αξ (1≦ξ≦Ξ) can be considered to represent the acoustic reflectance of the object from which the ξ-th reflected sound was reflected.
Since it is desirable that one or more reflected sounds be provided to the microphone array made up of M microphones, it is preferable that there is one or more reflective objects. From this point of view, a sound source, the microphone array, and one or more reflective objects are preferably in such a positional relation that a sound from the sound source is reflected off at least one reflective object before arriving at the microphone array, assuming that the sound source is located in the target direction. Each of the reflective objects has a two-dimensional shape (for example a flat plate) or a three-dimensional shape (for example a parabolic shape). Each reflective object has preferably about the size of the microphone array or greater (greater by a factor of 1 to 2). In order to effectively use reflected sounds, the reflectance αξ (1≦ξ≦Ξ) of each reflective object is preferably at least greater than 0, and more preferably, the amplitude of a reflected sound arriving at the microphone array is greater than the amplitude of the direct sound by a factor of 0.2 or greater. For example, each reflective object is a rigid solid. Each reflective object may be a movable object (for example a reflector) or an immovable object (such as a floor, wall, or ceiling). Note that if an immovable object is set as a reflective object, the steering vector for the reflective object needs to be changed as the microphone array is relocated (see functions Ψ(θ) and Ψξ(θ) described later) and consequently the filter needs to be recalculated (re-set). Therefore, the reflective objects are preferably accessories of the microphone array for the sake of robustness against environmental changes (in this case, Ξ reflected sounds assumed are considered to be sounds reflected off the reflective objects). Here the “accessories of the microphone array” are “tangible objects capable of following changes of the position and orientation of the microphone array while maintaining the positional relation (geometrical relation) with the microphone array). A simple example may be a configuration where reflective objects are fixed to the microphone array.
In order to concretely describe advantages of the sharp directive sound enhancement technique of the present invention, it is assumed in the following that Ξ=1, sounds are reflected once, and one reflective object exists at a distance of L meters from the center of the microphone array. The reflective object is a thick rigid object. Since Ξ=1 in this case, the symbol representing this is omitted and therefore equation (17a) can be rewritten as equation (17b):
{right arrow over (a)}(ω, θ)={right arrow over (h)}d(ω, θ)+α exp[−jωτ(θ)]·{right arrow over (h)}r(ω, θ) (17b)
An m-the element of the steering vector h→r(ω, θ)=[hr1(ω, θ), . . . , hrM(ω, θ)]T of a reflected sound can be given by equation (18a) in the same way that the steering vector of a direct sound is represented (see equation (14a)). The function Ψ(θ) outputs the direction from which a reflected sound arrives. Note that if the steering vector of a direct sound is written as equation (14b), an m-th element of the steering vector h→r(ω, θ)=[hrξ(ω, θ), . . . , hrM(ω, θ)]T of a reflected sound is given by equation (18b). Typically, an m-th element of a ξ-th (1≦ξ≦Ξ) steering vector h→rξ(ω, θ)=[hr1ξ(ω, θ), . . . , hrMξ(ω, θ)]T is given by equation (18c) or equation (18d). The function Ψξ(θ) outputs the direction from which the ξ-th reflected sound arrives.
Since the location of a reflective object can be set as appropriate, the direction from which a reflected sound arrives can be treated as a variable parameter.
Assuming that a flat-plate reflective object is near the microphone array (the distance L is not extremely large compared with the size of the microphone array), the coherence γ(ω, θ) is given by equation (19), where θ≠θ.
It will be apparent from equation (19) that the coherence γ(ω, θ) of equation (19) can be smaller than coherence γconv(ω, θ) of the conventional technique of equation (15). Since parameters (Ψ(θ) and L) that can be changed by relocating or reorienting the reflective object are included in the second to fourth terms of equation (19), there is a possibility that the first term, h→dH(ω, θ)h→d(ω, θ), can be eliminated.
For example, if a flat reflector is placed in such a position that the direction along which the microphones are arranged in a linear microphone array is normal to the reflector, Ψ(θ)=π−θ holds for the function Ψ(θ) and equation (20) folds for the difference τ(θ) in arrival time between a direct sound and a reflected sound. Therefore, the conditions of equatione (21) and (22) are generated for the elements of equation (19). Here, the symbol * is a complex conjugate operator.
Since the absolute value of h→dH(ω, θ)h→r(ω, θ) is sufficiently smaller than h→dH(ω, θ)h→d(ω, θ), the second and third terms of equation (19) are neglected. Then the coherence γ(ω, θ) can be approximated as equation (23):
{tilde over (γ)}(ω, θ)≈{1+α2 exp[−jω(τ(θ)−τ(θs))]}{right arrow over (h)}dH(ω, θs){right arrow over (h)}d(ω, θ) (23)
Even if h→dH(ω, θ)h→d(ω, θ)≠0, an approximated coherence γ˜(ω, θ) has a minimal solution θ of equation (24), where q is an arbitrary positive integer. The range of q is restricted for each frequency.
That is, not only the coherence in a direction given by equation (16) but also the coherence in a direction given by equation (24) can be suppressed. Since suppression of coherence can reduce the power of noise, a sharp directivity can be achieved as schematically shown in
While
As is apparent from the foregoing description, the essence of the sharp directive sound enhancement technique of the present invention is that the transmission characteristic a→(ω, θs)=[a1(ω, θ), . . . , aM(ω, θ)]T is represented by the sum of the steering vector of a direct sound and the steering vectors of Ξ reflected sounds, as shown in Equation (17a), for example. Since this does not affect the filter design concept, filters W→(ω, θs) can be designed by a method other than the minimum variance distortionless response (MVDR) method.
Methods other than the MVDR method described above will be described. They are: <1> a filter design method based on SNR maximization criterion, <2> a filter design method based on power inversion, <3> a filter design method using MVDR with one or more null directions (directions in which the gain of noise is suppressed) as a constraint condition, <4> a filter design method using delay-and-sum beam forming, <5> a filter design method using the maximum likelihood method, and <6> a filter design method using the adaptive microphone-array for noise reduction (AMNOR) method. For <1> the filter design method based on SNR maximization criterion and <2> the filter design method based on power inversion, refer to Reference 2 listed below. For <3> the filter design method using MVDR with one or more null directions (directions in which the gain of noise is suppressed) as a constraint condition, refer to Reference 3 listed below. For <6> the filter design method using the adaptive microphone-array for noise reduction (AMNOR) method, refer to Reference 4 listed below.
<1> Filter Design Method Based on SNR Maximization Criterion
In the filter design method based on SNR maximization criterion, a filter W→(ω, θs) is determined on the basis of a criterion of maximizing the SN ratio (SNR) in a target direction θs. The spatial correlation matrix for a sound from the target direction θs is denoted by Rss(ω) and the spatial correlation matrix for a sound from a direction other than the target direction θs is denoted by Rnn(ω). Then the SNR can be given by equation (25). Here, Rss(ω) can be given by equation (26) and Rnn(ω) can be given by equation (27). Transfer functions a→(ω, θ)=[a1(ω, θs), . . . , aM(ω, θs)]T can be given by equation (17a) (to be precise, equation (17a) where θ is replaced with θs).
The filter W→(ω, θs) that maximizes the SNR of equation (25) can be obtained by setting the gradient relating to filter W→(ω, θs) to zero, that is, by equation (28).
Thus, the filter W→(ω, θs) that maximizes the SNR of equation (25) can be given by equation (29):
{right arrow over (W)}(ω, θs)=Rnn−1(ω){right arrow over (a)}(ω, θs) (29)
Equation (29) includes the inverse matrix of the spatial correlation matrix Rnn(ω) of a sound from a direction other than the target direction θs. It is known that the inverse matrix of Rnn(ω) can be replaced with the inverse matrix of a spatial correlation matrix Rxx(ω) of a whole input including sounds from the target direction θs and other directions than the target direction θs. Note that Rxx(ω)=Rss(ω)+Rnn(ω)=Q(ω) (see equatione (10a), (26) and (27)). That is, the filter W→(ω, θs) that maximizes the SNR of equation (25) may be obtained by equation (30):
{right arrow over (W)}(ω, θs)=Rxx−1(ω){right arrow over (a)}(ω, θs) (30)
<2> Filter Design Method Based on Power Inversion
In the filter design method based on power inversion, a filter W→(ω, θs) is determined on the basis of a criterion of minimizing the average output power of a beam former while a filter coefficient for one microphone is fixed at a constant value. Here, an example where the filter coefficient for the first microphone among M microphones is fixed will be described. In this design method, a filter W→(ω, θs) is designed that minimizes the power of sounds from all directions (all directions from which sounds can arrive) by using a spatial correlation matrix Rxx(ω) (see equation (31)) under the constraint condition of equation (32). Transfer functions a→(ω, θs)=[a1(ω, θs), . . . , aM(ω, θs)]T can be given by equation (17a) (to be precise, by equation (17a) where θ is replaced with θs). Here, Rxx(ω)=Q(ω) (see equatione (10a), (26) and (27)).
It is known that the filter W→(ω, θs) that is an optimum solution of equation (31) can be given by equation (33):
{right arrow over (W)}(ω, θs)=Rxx−1(ω){right arrow over (G)} (33)
<3> Filter Design Method Using MVDR with One or More Null Directions as Constraint Condition
In the MVDR method described earlier, a filter W→(ω, θs) has been designed under the single constraint condition that a filter is obtained that minimizes the average output power of a beam former given by equation (7) (that is, the power of noise which is sounds from directions other than a target direction) under the constraint condition that the filter passes sounds from a target direction θs in all frequency bands as expressed by equation (8). According to the method, the power of noise can be generally suppressed. However, the method is not necessarily preferable if it is previously known that there is a noise source(s) that has strong power in one or more particular directions. If this is the case, a filter is required that strongly suppresses one or more particular known directions (that is, null directions) in which the noise source(s) exist(s). Therefore, the filter design method described here obtains a filter that minimizes the average output power of the beam former given by equation (7) (that is, minimizes the average output power of sounds from directions other than a target direction and the null directions) under the constraint conditions that (1) the filter passes sounds from the target direction θs in all frequency bands and that (2) the filter suppresses sounds from B known null directions θN1, θN2, . . . , θNB (B is a predetermined integer greater than or equal to 1) in all frequency bands. Let a set of indices φ of directions from which sound arrives be denoted by {1, 2, . . . , P}, then Nj ε{1, 2, . . . , P} (where j ε {1, 2, . . . , B}) and B≦P−1, as has been described earlier.
Let a→(ω, θi)=[a1(ω, θi), . . . , aM(ω, θi)]T be transfer functions between a sound source assumed to be located in a direction θi and the M microphones at a frequency ω, in other words, transfer functions of a sound from a direction θi at a frequency ω arriving at the microphones of a microphone array, then a constraint condition can be given by equation (34). Here, indices i ε {s, N1, N2, . . . , NB}, transfer functions a→(ω, θi)=[a1(ω, θi), . . . , aM(ω, θi)]T can be given by equation (17a) (to be precise, by equation (17a) where θ is replaced with θi), and fi(ω) represents a pass characteristic at a frequency ω for a direction θi.
{right arrow over (W)}H(ω, θs){right arrow over (a)}(ω, θi)=fi(ω) i ε {s, N1, N2, . . . , NB} (34)
Equation (34) can be represented as a matrix, for example as equation (35). Here, A→(ω, θs)=[a→(ω, θs), a→(ω, θN1), a→(ω, θNB)]
{right arrow over (W)}H(ω, θs){right arrow over (A)}(ω, θs)={right arrow over (F)} (35)
where
{right arrow over (F)}=[fs(ω), fN1(ω), . . . , fNB(ω)]
Taking into consideration the constraint conditions that (1) the filter passes sounds from the target direction θs in all frequency bands and that (2) the filter suppresses sounds from B known null directions θN1, θN2, . . . , θNB in all frequency bands, ideally fs(ω)=1.0 and fi(ω)=0.0 (i ε {N1, N2, . . . , NB}) should be set. This means that the filter completely passes sounds in all frequency bands from the target direction θs and completely blocks sounds in all frequency bands from B known null directions θN1, θN2, . . . , θNB. In reality, however, it is difficult in some situations to effect such control as completely passing all frequency bands or completely blocking all frequency bands. In such a case, the absolute value of fs(ω) is set to a value close to 1.0 and the absolute value of fi(ω) (i ε {N1, N2, . . . , NB}) is set to a value close to 0.0. Of course, fi(ω) and fj(ω) (i≠j; i and j ε {N1, N2, . . . , NB}) may be the same or different.
According to the filter design method described here, the filter W→(ω, θs) that is an optimum solution of equation (7) under the constraint condition given by equation (35) can be given by equation (36) (see Reference 3 listed below).
{right arrow over (W)}(ω, θs)=Q−1(ω){right arrow over (A)}(ω, θs)({right arrow over (A)}H(ω, θs)Q−1(ω){right arrow over (A)}(ω, θs))−1{right arrow over (F)} (36)
<4> Filter Design Method Using Delay-And-Sum Beam forming
As apparent from equation (2), assuming that direct and reflected sounds that arrive are plane waves, then a filter W→(ω, θs) can be given by equation (37). That is, the filter W→(ω, θs) can be obtained by normalizing a transmission characteristic a→(ω, θs). The transmission characteristic a→(ω, θs)=[a1(ω, θs), . . . , aM(ω, θs)]T can be given by equation (17a) (to be precise, by equation (17a) where θ is replaced with θs). The filter design method does not necessarily achieve a high filtering accuracy but requires only a small quantity of computation.
<5> Filter Design Method Using Maximum Likelihood Method
By excluding spatial information concerning sounds from a target direction from a spatial correlation matrix Q(ω) in the MVDR method described earlier, flexibility of suppression of noise can be improved and the power of noise can be further suppressed. Therefore, in the filter design method described here, the spatial correlation matrix Q(ω) is written as the second term of the right-hand side of equation (10a), that is, equation (10c). A filter W→(ω, θs) can be given by equation (9) or (36). Here, Q(ω) included in equatione (9) and (36) or Rxx(ω)=Q(ω) included in equatione (30) and (33) is a spatial correlation matrix given by equation (10c).
<6> Filter Design Method Using AMNOR Method
The AMNOR method obtains a filter that allows some amount of decay D of a sound from a target direction by trading off the amount of decay D of the sound from the target direction against the power of noise remaining in a filter output signal (for example, the amount of decay D is maintained at a certain threshold D^ or less) and, when a mixed signal of [a] a signal produced by applying transfer functions between a sound source and microphones to a virtual signal from a target direction (hereinafter referred to as the virtual target signal) and [b] noise (obtained by observation with M microphones in a noisy environment without a sound from the target direction) is input, outputs a filter output signal that reproduces best the virtual target signal in terms of least squares error (that is, the power of noise contained in a filter output signal is minimized). According to the AMNOR method, a filter W→(ω, θs) can be given by equation (38) (see Reference 4 listed below). Here, Rss(ω) can be given by equation (26) and Rnn(ω) can be given by equation (27). Transfer functions a→(ω, θ)=[a1(ω, θs), . . . , aM(ω, θs)]T can be given by equation (17a) (to be precise, by equation (17a) where θ is replaced with θs).
{right arrow over (W)}(ω, θs)=Ps{right arrow over (a)}(ω, θs)(Rnn(ω)+PsRss(ω))−1 (38)
Ps is a coefficient that assigns a weight to the level of the virtual target signal and called the virtual target signal level. The virtual target signal level Ps is a constant that is not dependent on frequencies. The virtual target signal level Ps may be determined empirically or may be determined so that the difference between the amount of decay D of a sound from the target direction and the threshold D^ is within an arbitrarily predetermined error margin. The latter case will be described. The frequency response F(ω) of the filter W→(ω, θs) to a sound from a target direction θs in the AMNOR method can be given by equation (39). Let the amount of decay D(Ps) when using the filter W→(ω, θs) given by equation (38) be denoted by D(Ps), then the amount of decay D(Ps) can be defined by equation (40). Here, ω0 represents the upper limit of frequency ω (typically, a higher-frequency adjacent to a discrete frequency ω). The amount of decay D(Ps) is a monotonically decreasing function of Ps. Therefore, a virtual target signal level Ps such that the difference between the amount of decay D(Ps) and the threshold D^ is within an arbitrarily predetermined error margin can be obtained by repeatedly obtaining the amount of decay D(Ps) while changing Ps with the monotonicity of D(Ps).
<Variation>
In the foregoing description, the spatial correlation matrices Q(ω), Rss(ω) and Rnn(ω) are expressed using transfer functions. However, the spatial correlation matrices Q(ω), Rss(ω) and Rnn(ω) can also be expressed using the frequency-domain signals X→(ω, k) described earlier. While the spatial correlation matrix Q(ω) will be described below, the following description applies to Rss(ω) and Rnn(ω) as well. (Q(ω) can be replaced with Rss(ω) or Rnn(ω)). The spatial correlation matrix Rss(ω) can be obtained using frequency-domain representations of analog signals obtained by observation with a microphone array (including M microphones) in an environment where only sounds from a target direction exist. The spatial correlation matrix Rnn(ω) can be obtained using frequency-domain representations of an analog signal obtained by observation with a microphone array (including M microphones) in an environment where no sounds from a target direction exist (that is, a noisy environment).
The spatial correlation matrix Q(ω) using frequency domain signals X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T can be given by equation (41). Here, the operator E[•] represents a statistical averaging operation. When viewing a discrete time series of an analog signal received with a microphone array (including M microphones) as a stochastic process, the operator E[•] represents a arithmetic mean value (expected value) operation if the stochastic process is a so-called wide-sense stationary process or a second-order stationary process. In this case, the spatial correlation matrix Q(ω) can be given by equation (42) using frequency-domain signals X→(ω, k−i) (i=0, 1, . . . , ζ−1) of a total of ζ current and past frames stored in a memory, for example. When i=0, a k-th frame is the current frame. Note that the spatial correlation matrix Q(ω) given by equation (41) or (42) may be recalculated for each frame or may be calculated at regular or irregular interval, or may be calculated before implementation of an embodiment, which will be described later (especially when Rss(ω) or Rnn(ω) is used in filter design, the spatial correlation matrix Q(ω) is preferably calculated beforehand by using frequency-domain signals obtained before implementation of the embodiment). If the spatial correlation matrix Q(ω) is recalculated for each frame, the spatial correlation matrix Q(ω) depends on the current and past frames and therefore the spatial correlation matrix will be explicitly represented as Q(ω, k) as in equatione (41a) and (42a).
If the spatial correlation matrix Q(ω, k) represented by equation (41a) or (42a) is used, the filter W→(ω, θs) also depends on the current and past frames and therefore is explicitly represented as W→(ω, θs, k). Then, a filter W→(ω, θs) represented by any of equatione (9), (29), (30), (33), (36) and (38) described with the filter design methods described above is rewritten as equatione (9m), (29m), (30m), (33m), (36m) or (38m).
<<First Embodiment of Sharp Directive Sound Enhancement Technique>>
[Step S1]
The filter design section 260 calculates beforehand a filter W→(ω, θi) for each frequency for each of discrete directions from which sounds to be enhanced can arrive. The filter design section 260 calculates filters W→(ω, θ1), . . . , W→(ω, θi), . . . , W→(ω, θ1) (1≦i≦I, ω ε Ω; i is an integer and Ω is a set of frequencies ω), where I is the total number of discrete directions from which sounds to be enhanced can arrive (I is a predetermined integer greater than or equal to 1 and satisfies I≦P).
To do so, transfer functions a→(ω, θi)=[a1(ω, θi), . . . , aM(ω, θi)]T (1≦i≦I, ω ε Ω) need to be obtained except for the case of <Variation> described above. Transmission characteristic a→(ω, θi)=[a1(ω, θi), . . . , aM(ω, θi)]T can be calculated practically according to equation (17a) (to be precise, by equation (17a) where θ is replaced with θi) on the basis of the arrangement of the microphones in the microphone array and environmental information such as the positional relation of reflective objects such as a reflector, floor, walls, or ceiling to the microphone array, the arrival time difference between a direct sound and a ξ-th reflected sound (1≦ξ≦Ξ), and the acoustic reflectance of the reflective object. Note that if the <3> filter design method using MVDR with one or more null directions as constraint condition is used, the indices i of the directions used for calculating the transfer functions a→(ω, θi) (1≦i≦I, ω ε Ω) preferably cover all of indices N1, N2, . . . , NB of directions of at least B null directions. In other words, indices N1, N2, . . . , NB of the directions of B null directions are set to any of different integers greater than or equal to 1 and less than or equal to I.
The number Ξ of reflected sounds is set to an integer that satisfies 1≦Ξ. The number Ξ is not limited and can be set to an appropriate value according to the computational capacity and other factors. If one reflector is placed near the microphone array, the transfer functions a→(ω, θi) can be calculated practically according to equation (17b) (to be precise, by equation (17b) where θ is replaced with θi).
To calculate steering vectors, equatione (14a), (14b), (18a), (18b), (18d) or (18d), for example, can be used. Note that transfer functions obtained by actual measurements in a real environment, for example, may be used for designing the filters instead of using equatione (17a) and (17b).
Then, W→(ω, θi) (1≦i≦I) is obtained according to any of equatione (9), (29), (30), (33), (36), (37) and (38), for example, using the transfer functions a→(ω, θi), except for the case described in <Variation>. Note that if equation (9), (30), (33) or (36) is used, the spatial correlation matrix Q(ω) (or Rxx(ω)) can be calculated according to equation (10b), except for the case described with respect to <5> the filter design method using the maximum likelihood method. If equation (9), (30), (33) or (36) is used according to <5> the filter design method using the maximum likelihood method described earlier, the spatial correlation matrix Q(ω) (or Rxx(ω)) can be calculated according to equation (10c). If equation (29) is used, the spatial correlation matrix Rnn(ω) can be calculated according to equation (27). I×|Ω| filters W→(ω, θi) (1≦i≦I, ω ε Ω) are stored in the storage 290, where |Ω| represents the number of the elements of the set Ω.
[Step S2]
The M microphones 200-1, . . . , 200-M making up the microphone array are used to pick up sounds, where M is an integer greater than or equal to 2.
There is no restraint on the arrangement of the M microphones. However, a two- or three-dimensional arrangement of the M microphones has the advantage of eliminating uncertainty of a direction from which sounds to be enhanced arrive. That is, a planar or steric arrangement of the microphones can avoid the problem with a horizontal linear arrangement of the M microphones that a sound arriving from a front direction cannot be distinguished from a sound arriving from right above, for example. In order to provide a wide range of directions that can be set as sound-pickup directions, each microphone preferably has a directivity capable of picking up sounds with a certain level of sound pressure in potential target directions θs which are sound-pickup directions. Accordingly, microphones having relatively weak directivity, such as omnidirectional microphones or unidirectional microphones are preferable.
[Step S3]
The AD converter 210 converts analog signals (pickup signals) picked up with the M microphones 200-1, . . . , 200-M to digital signals x→(t)=[x1(t), . . . , XM(t)]T, where t represents the index of a discrete time.
[Step S4]
The frame generator 220 takes inputs of the digital signals x→(t)=[x1(t), . . . , xM(t)]T output from the AD converter 210, stores N samples in a buffer on a channel by channel basis, and outputs digital signals x→(k)=[x→1(k), . . . , x→M(k)]T in frames, where k is an index of a frame-time number and x→m(k) [xm((k−1)N+1), . . . , xm(kN)] (1≦m≦M). N depends on the sampling frequency and 512 is appropriate for sampling at 16 kHz.
[Step S5]
The frequency-domain transform section 230 transforms the digital signals x→(k) in frames to frequency-domain signals X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T and outputs the frequency-domain signals, where ω is an index of a discrete frequency. One way to transform a time-domain signal to a frequency-domain signal is fast discrete Fourier transform. However, the way to transform the signal is not limited to this. Other method for transforming to a frequency domain signal may be used. The frequency-domain signal X→(ω, k) is output for each frequency ω and frame k at a time.
[Step S6]
The filter applying section 240 applies the filter W→(ω, θs) corresponding to a target direction θs to be enhanced to the frequency-domain signal X→(ω, k)=[X1(ω, k), . . . , Xm(ω, k)]T in each frame k for each frequency ω ε Ω and outputs an output signal Y(ω, k, θs) (see equation (43)). The index s of the target direction θs is s ε {1, . . . , I} and the filters W→(ω, θs) are stored in the storage 290. Therefore, the filter applying section 240 only has to retrieve the filter W→(ω, θs) that corresponds to the target direction θs to be enhanced from the storage 290. If the index s of the target direction θs does not belong to the set {1, . . . , I}, that is, the filter W→(ω, θs) that corresponds to the target direction θs has not been calculated in the process at step S1, the filter design section 260 may calculate at this moment the filter W→(ω, θs) that corresponds to the target direction θs or a filter W→(ω, θs′) that corresponds to a direction θs′ close to the target direction θs may be used.
Y(ω, k, θs)={right arrow over (W)}H(ω, θs){right arrow over (X)}(ω, k) ∀ω ε Ω (43)
[Step S7]
The time-domain transform section 250 transforms the output signal Y(ω, k, θs) of each frequency ω ε Ω in a k-th frame to a time domain to obtain a time-domain frame signal y(k) in the k-th frame, then combines the obtained frame time-domain signals y(k) in the order of frame-time number index, and outputs a time-domain signal y(t) in which the sound from the target direction θs is enhanced. The method for transforming a frequency-domain signal to a time-domain signal is inverse transform of the transform used in the process at step S5 and may be fast discrete inverse Fourier transform, for example.
While the first embodiment has been described here in which the filters W→(ω, θi) are calculated beforehand in the process at step S1, the filter design section 260 may calculate the filter W→(ω, θi) for each frequency after the target direction θs is determined, depending on the computational capacity of the sharp directive sound enhancement apparatus 1.
<<Second Embodiment of Sharp Directive Sound Enhancement Technique>>
[Step S11]
M microphones 200-1, . . . , 200-M making up a microphone array is used to pick up sounds, where M is an integer greater than or equal to 2. The arrangement of the M microphones is as described in the first embodiment.
[Step S12]
The AD converter 210 converts analog signals (pickup signals) picked up with the M microphones 200-1, . . . , 200-M to digital signals x→(t)=[x1(t), . . . , xM(t)]T, where t represents the index of a discrete time.
[Step S13]
The frame generator 220 takes inputs of the digital signals x→(t)=[x1(t), . . . , xM(t)]T output from the AD converter 210, stores N samples in a buffer on a channel by channel basis, and outputs digital signals x→(k)=[x→1(k), . . . , x→M(k)]T in frames, where k is an index of a frame-time number and x→m(k) [xm((k−1)N+1), . . . , xm(kN)] (1≦m≦M). N depends on the sampling frequency and 512 is appropriate for sampling at 16 kHz.
[Step S14]
The frequency-domain transform section 230 transforms the digital signals x→(k) in frames to frequency-domain signals X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T and outputs the frequency-domain signals, where ω is an index of a discrete frequency. One way to transform a time-domain signal to a frequency-domain signal is fast discrete Fourier transform. However, the way to transform the signal is not limited to this. Other method for transforming to a frequency domain signal may be used. The frequency-domain signal X→(ω, k) is output for each frequency ω and frame k at a time.
[Step S15]
The filter calculating section 261 calculates the filter W→(ω, θs, k) (ω ε Ω; Ω is a set of frequencies ω) that corresponds to the target direction θs to be used in a current k-th frame.
To do so, transfer functions a→(ω, θs)=[a1(ω, θs), . . . , aM(ω, θs)]T (ωε Ω) need to be provided. Transfer functions a→(ω, θs)=[a1(ω, θs), . . . , aM(ω, θs)]T can be calculated practically according to equation (17a) (to be precise, by equation (17a) where θ is replaced with θs) on the basis of the arrangement of the microphones in the microphone array and environmental information such as the positional relation of reflective objects such as a reflector, floor, walls, or ceiling to the microphone array, the arrival time difference between a direct sound and a ξ-th reflected sound (1≦ξ≦Ξ), and the acoustic reflectance of the reflective object. Note that if <3> the filter design method using MVDR with one or more null directions as a constraint condition is used, transfer functions a→(ω, θNj) (1≦j≦B, ω ε Ω) also need to be obtained. The transfer functions can be calculated practically according to equation (17a) (to be precise, by equation (17a) where θ is replaced with θNj) on the basis of the arrangement of the microphones in the microphone array and environmental information such as the positional relation of reflective objects such as a reflector, a floor, a wall, or ceiling to the microphone array, the arrival time difference between a direct sound and a ξ-th reflected sound (1≦ξ≦Ξ), and the acoustic reflectance of the reflective object.
The number Ξ of reflected sounds is set to an integer that satisfies 1≦Ξ. The number Ξ is not limited and can be set to an appropriate value according to the computational capacity and other factors. If one reflector is placed near the microphone array, the transfer functions a→(ω, θs) can be calculated practically according to equation (17b) (to be precise, by equation (17b) where θ is replaced with θs). In this case, transfer functions a→(ω, θNj) (1≦j≦B, ω ε Ω) can be practically calculated according to equation (17b) (to be precise, by equation (17b) where θ is replaced with θNj).
To calculate steering vectors, equatione (14a), (14b), (18a), (18b), (18c) or (18d), for example, can be used. Note that transfer functions obtained by actual measurements in a real environment, for example, may be used for designing the filters instead of using equatione (17a) and (17b).
Then, the filter calculating section 261 calculates filters W→(ω, θs, k) (ω ε Ω) according to any of equatione (9m), (29m)m (30m), (33m), (36m) and (38m) using the transfer functions a→(ω, θs) (ω ε Ω) and, if needed, the transfer functions a→(ω, θNj) (1≦j≦B, ω ε Ω). Note that the spatial correlation matrix Q(ω) (or Rxx(ω)) can be calculated according to equation (41a) or (42a). In the calculation of the spatial correlation matrix Q(ω), frequency-domain signals X→(ω, k−i) (i=0, 1, . . . , ζ−1) of a total of ζ current and past frames stored in the storage 290, for example, are used.
[Step S16]
The filter applying section 240 applies the filter W→(ω, θs, k) corresponding to a target direction θs to be enhanced to the frequency-domain signal X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T in each frame k for each frequency ω ε Ω and outputs an output signal Y(ω, k, θs) (see equation (44)).
Y(ω, k, θs)={right arrow over (W)}H(ω, θs, k){right arrow over (X)}(ω, k) ∀ω ε Ω (44)
[Step S17]
The time-domain transform section 250 transforms the output signal Y(ω, k, θs) of each frequency ω ε Ω of a k-th frame to a time domain to obtain a time-domain frame signal y(k) in the k-th frame, then combines the obtained frame time-domain signals y(k) in the order of frame-time number index, and outputs a time-domain signal y(t) in which the sound from the target direction θs is enhanced. The method for transforming a frequency-domain signal to a time-domain signal is inverse transform of the transform method used in the process at step S14 and may be fast discrete inverse Fourier transform, for example.
[Experimental Example of Sharp Directive Sound Enhancement Technique]
Results of an experiment on the first embodiment of the sharp directive sound enhancement technique of the present invention (the minimum variance distortionless response (MVDR) method under a single constraint condition) will be described. As illustrated in
The same experiment was conducted with the reflector 300 placed so that the flat surface of the reflector 300 formed an angle of 45 degrees with the direction in which the microphones of the linear microphone array were arranged, as shown in
<Example Applications>
Figuratively speaking, the sharp directive sound enhancement technique is equivalent to generation of a clear image from an unsharp, blurred image and is useful for obtaining detailed information about an acoustic field. The following is description of examples of services where the sharp directive sound enhancement technique of the present invention is useful.
A first example is creation of contents that are combination of audio and video. The use of an embodiment of the sharp directive sound enhancement technique of the present invention allows the target sound from a great distance to be clearly enhanced even in a noisy environment with noise sounds (sounds other than target sounds). Therefore, for example sounds in a particular area corresponding to a zoomed-in moving picture of a dribbling soccer player that was shot from outside the field can be added to the moving picture.
A second example is an application to a video conference (or an audio teleconference). When a conference is held in a small room, the voice of a human speaker can be enhanced to a certain degree with several microphones according to a conventional technique. However, in a large conference room (for example, a large space where there are human speakers at a distance of 5 m or more from microphones), it is difficult to clearly enhance the voice of a human speaker at a distance with the conventional techniques by the conventional method and a microphone needs to be placed in front of each human speaker. In contrast, the use of an embodiment of the sharp directive sound enhancement technique of the present invention is capable of clearly enhancing sounds from a great distance and therefore enables construction of a video conference system that is usable in a large conference room without having to place a microphone in front of each human speaker.
<<Principle of Sound Spot Enhancement Technique>>
A principle of a sound spot enhancement technique of the present invention will be described below. The sound spot enhancement technique of the present invention is based on the nature of a microphone array technique being capable of following sounds from any direction on the basis of signal processing and positively uses reflected sounds to pick up sounds with a high SN ratio. One feature of the present invention is a combined use of reflected sounds and a signal processing technique that enables a sharp directivity. In particular, one of the remarkable features of the sound spot enhancement technique of the present invention is the use of a reflective object to increase the difference between in transfer functions of different sound sources to a microphone array, in light of the fact that the transfer functions of sound sources located in nearly the same directions from the microphone array but at different distances from the microphone array to the microphone array are very similar to one another. By extracting differences in transmission characteristic through signal processing, a sound spot enhancement technique capable of enhancing sounds according to the distances from the microphone array can be achieved.
Prior to the description, symbols will be defined again. The index of a discrete frequency is denoted by ω (The index ω of a discrete frequency may be considered to be an angular frequency ω because a frequency f and an angular frequency ω satisfies the relation ω=2πf. With regard to ω, the “index of a discrete frequency” may be also sometimes simply referred to as a “frequency”) and the index of frame-time number is denoted by k. Frequency-domain representation of a k-th frame of an analog signal received at M microphones is denoted by X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T and a filter that enhances a frequency-domain signal X→(ω, k) of a sound from a sound source assumed to be located in a direction θs as viewed from the center of the microphone array at a distance Dh from the center of the microphone array with a frequency ω is denoted by W→(ω, θs, Dh), where M is an integer greater than or equal to 2 and T represents the transpose. It is assumed here that the distance Dh is fixed.
While the “center of a microphone array” can be arbitrarily determined, typically the geometrical center of the array of the M microphones is treated as the “center of a microphone array”. In the case of a linear microphone array, for example, the point equidistant from the microphones at the both ends of the array is treated as the “center of the microphone array”. In the case of a planar microphone array in which microphones are arranged in a square matrix of m×m (m2=M), for example, the position at which the diagonals linking the microphones at the corners intersect is treated as the “center of the microphone array.”
The expression “sound source assumed to be located in . . . ” has been used because the actual presence of a sound source at the location is not essential to the sound spot enhancement technique of the present invention. That is, as will be apparent from the later description, the sound spot enhancement technique of the present invention in essence performs signal processing of applying filters to signals represented by frequencies and enables embodiments in which a filter is created beforehand for each discrete distance Dh. Accordingly, the actual presence of a sound source at the location is not required even at the stage where the sound spot enhancement processing is actually performed. For example, if a sound source actually exists at a location in a direction θs as viewed from the microphone array and at a distance of Dh from the microphone array at the stage where the sound spot enhancement processing is actually performed, a sound from the sound source can be enhanced by choosing an appropriate filter for the location. If the sound source does not actually exist at the location and if it is assumed that there are no sounds and even no noise at all, a sound enhanced by the filter will be ideally complete silence. However, this is no different from enhancing a “sound arriving from the location”.
Under these conditions, a frequency-domain signal Y(ω, k, θs, Dh) resulting from the enhancement of a frequency-domain signal X→(ω, k) of a sound from a sound source assumed to be at a location in a direction θs at a distance of Dh as viewed from the center of the microphone array (hereinafter referred to as a “location (θs, Dh)” unless otherwise stated) with frequency ω can be given by equation (106) (hereinafter the resulting signal is referred to as an output signal).
Y(ω, k, θs, Dh)={right arrow over (W)}H(ω, θs, Dh){right arrow over (X)}(ω, k) (106)
where H represents the Hermitian transpose.
The filter W→(ω, θs, Dh) may be designed in various ways. A design using minimum variance distortionless response (MVDR) method will be described here. In the MVDR method, a filter W→(ω, θs, Dh) is designed so that the power of sounds from directions other than a direction θs (hereinafter sounds from directions other than the direction θs will be also referred to as “noise”) is minimized at a frequency ω by using a spatial correlation matrix Q(ω) under the constraint condition of equation (108). (see equation (107). It should be noted that the spatial correlation matrix Q(ω) is specified as Q(ω, Dh) because it is assumed here that the direction Dh is fixed.) Assuming that a sound source is located in a position (θs, Dh), then a→(ω, θs, Dh)=[a1(ω, θs, Dh), . . . , aM(ω, θs, Dh)]T represents transfer functions at a frequency ω between the sound source and the M microphones. In other words, a→(ω, θs, Dh)=[a1(ω, θs, Dh), . . . , aM(ω, θs, Dh)]T represents transfer functions of a sound from the position (θs, Dh) to the microphones included in the microphone array at frequency ω. The spatial correlation matrix Q(ω) represents the correlation among components X1(ω, k), . . . , XM(ω, k) of a frequency-domain signal X→(ω, k) at frequency ω and has E[Xi(ω, k)Xj * (ω, k) (1≦i≦M, 1≦j≦M) as its (i, j) elements. The operator E[•] represents a statistical averaging operation and the symbol * is a complex conjugate operator. The spatial correlation matrix Q(ω) can be expressed using statistics values of X1(ω, k), . . . , XM(ω, k) obtained from observation or may be expressed using transfer functions. The latter case, where the spatial correlation matrix Q(ω) is expressed using transfer functions, will be described momentarily hereinafter.
It is known that the filter W→(ω, θs, Dh) which is an optimal solution of equation (107) can be given by equation (109) (see Reference 1 listed later).
As will be appreciated from the fact that the inverse matrix of the spatial correlation matrix Q(ω, Dh) is included in equation (109), the structure of the spatial correlation matrix Q(ω, Dh) is important for achieving a sharp directivity. It will be appreciated from equation (107) that the power of noise depends on the structure of the spatial correlation matrix Q(ω, Dh).
A set of indices p of directions from which noise arrives is denoted by {1, 2, . . . , P−1}. It is assumed that the index s of the target direction θs does not belong to the set {1, 2, . . . , P−1}. Assuming that P−1 noises come from arbitrary directions, the spatial correlation matrix Q(ω, Dh) can be given by equation (110a). In order to design a filter that sufficiently functions in the presence of many noises, it is preferable that P be a relatively large value. It is assumed here that P is an integer on the order of M. While the description is given as if the direction θs is a constant direction (and therefore directions other than the direction θs are described as directions from which noise arrives) for the clarity of explanation of the principle of the sound spot enhancement technique of the present invention, the direction θs in reality may be any direction that can be a target of sound enhancement. Usually, a plurality of directions can be directions θs. In this light, the differentiation between the direction θs and noise directions is subjective. It is more correct to consider that one direction selected from P different directions that are predetermined as a plurality of possible directions from which whatever sounds, including a target sound or noise, may arrive is the direction that can be a target of sound enhancement and the other directions are noise directions. Therefore, the spatial correlation matrix Q(ω, Dh) can be represented by transfer functions a→(ω, θφ, Dh)=[a1(ω, θφ, Dh), . . . , aM(ω, θφ, Dh)]T (φ ε Φ) of sounds that come from directions θφ included in a plurality of possible directions that are at a distance Dh from the center of the microphone array and from which sounds may arrive to the microphones and can be written as equation (110b), where Φ is the union of set {1, 2, . . . , P−1} and a set {s}. Note that |Φ|=P and |Φ| represents the number of elements of the set Φ.
Here, it is assumed that the transmission characteristic a→(ω, θs, Dh) of a sound from the direction θs and the transfer functions a→(ω, θp, Dh)=[a1(ω, θp, Dh), . . . , aM(ω, θp, Dh)]T of sounds from directions p ε {1, 2, . . . , P−1} are orthogonal to each other. That is, it is assumed that there are P orthogonal basis systems that satisfy the condition given by equation (111). The symbol ⊥ represents orthogonality. If A→⊥B→, the inner product of vectors A→ and B→ is zero. It is assumed here that P≦M. Note that if the condition given by equation (111) can be relaxed to assume that there are P basis systems that can be regarded approximately as orthogonal basis systems, P is preferably a value on the order of M or a relatively large value greater than or equal to M.
{right arrow over (a)}(ω, θs, Dh)⊥{right arrow over (a)}(ω, θ1, Dh)⊥ . . . ⊥{right arrow over (a)}(ω, θP-1, Dh) (111)
Then, the spatial correlation matrix Q(ω, Dh) can be expanded as equation (112). Equation (112) means that the spatial correlation matrix Q(ω, Dh) can be decomposed into a matrix V(ω, Dh)=[a→(ω, θs, Dh), a→(ω, θ1, Dh), . . . , a→(ω, θp-1, Dh)]T made up of P transfer functions that satisfy orthogonality and a unit matrix Λ(ω, Dh). Here, ρ is an eigenvalue of a transmission characteristic a→(ω, θφ, Dh) that satisfies equation (111) for the spatial correlation matrix Q(ω, Dh) and is a real value.
Q(ω, Dh)=ρ{right arrow over (V)}(ω, Dh){right arrow over (Λ)}(ω, Dh){right arrow over (V)}H(ω, Dh) (112)
Then, the inverse matrix of the spatial correlation matrix Q(ω) can be given by equation (113).
Substitution of equation (113) into equation (107) shows that the power of noise is minimized. If the power of noise is minimized, it means that the directivity in the direction θs is achieved. Therefore, orthogonality between the transfer functions of sounds from different directions is an important condition for achieving directivity in the direction θs.
The reason why it is difficult for conventional techniques to achieve a sharp directivity in a direction θs will be discussed below.
Conventional techniques assumed in designing filters that transfer functions are made up of those of direct sounds. In reality, there are reflected sounds that are produced by reflection of sounds from the same sound source off surfaces such as walls and a ceiling and arrive at microphones. However, the conventional techniques regarded reflected sounds as a factor that degrade directivity and ignored the presence of reflected sounds. Assuming that sounds arrive at a linear microphone array as plane waves, the conventional technique treated transfer functions a→conv(ω, θ)=[a1(ω, θ), . . . , aM(ω, θ)]T as a→conv(ω, θ)=h→d(ω, θ), where h→d(ω, θ)=[hd1(ω, θ), . . . , hdM(ω, θ)]T represents steering vectors of only a direct sound arriving from a direction θ (Since sound waves are considered to be plane waves, the steering vectors do not depend on distance D.) Note that a steering vector is a complex vector where phase response characteristics of microphones at a frequency ω with respect to a reference point are arranged for a sound wave from a direction θ viewed from the center of the microphone array.
It is assumed hereinafter momentarily that sound arrives at the linear microphone as plane waves. Assume that an m-th element hdm(ω, θ) of the steering vector h→d(ω, θ) of a direct sound is given by, for example, equation (114c), where u represents the distance between adjacent microphones, j is an imaginary unit. In this case, the reference point is the midpoint of the full-length of the linear microphone array (the center of the linear microphone array). The direction θ is defined as the angle formed by the direction from which a direct sound arrives and the direction in which the microphones included in the linear microphone array are arranged, as viewed from the center of the linear microphone array (see
The inner product γconv(ω, θ) of a transmission characteristic of a direction θ and a transmission characteristic of a target direction θs can be given by equation (115), where θ≠θs.
Hereinafter, γconv(ω,θ) is referred to as coherence. The direction θ in which the coherence γconv(ω,θ) is 0 can be given by equation (116), where q is an arbitrary integer, except 0. Since 0<θ<π/2, the range of q is limited for each frequency band.
Since only parameters relating to the size of the microphone array (M and u) can be changed in equation (116), it is difficult to reduce the coherence γconv(ω, θ) without changing any of the parameters relating to the size of the microphone array if the difference (angular difference) |θ−θs| between directions is small. If this is the case, the power of noise is not reduced to a sufficiently small value and directivity having a wide beam width in the target direction θs as schematically illustrated in
The sound spot enhancement technique of the present invention is based on the consideration described above and is characterized by positively taking into account reflected sounds, unlike in the conventional technique, on the basis of an understanding that in order to design a filter that provides a sharp directivity in the direction θs, it is important to enable the coherence to be reduced to a sufficiently small value even when the difference (angular difference) |θ−θs| betweendirections is small.
Two types of plane waves, namely direct sounds from a sound source and reflected sounds produced by reflection of that sound off a reflective object 300, together enter the microphones of a microphone array. Let the number of reflected sounds be denoted by Ξ. Here, Ξ is a predetermined integer greater than or equal to 1. Then, a transmission characteristic a→(ω, θ)=[a1(ω, θ), . . . , aM(ω, θ)]T can be expressed by the sum of a transmission characteristic of a direct sound that comes from a direction that can be a target of sound enhancement and directly arrives at the microphone array and the transmission characteristic(s) of one or more reflected sounds that are produced by reflection of that sound off a reflective object and arrive at the microphone array. Specifically, the transmission characteristic can be represented as the sum of the steering vector of the direct sound and the steering vector of Ξ reflected sounds whose decays due to reflection and arrival time differences from the direct sound are corrected, as shown in equation (117a), where τξ(θ) is the arrival time difference between the direct sound and a ξ-th (1≦ξ≦Ξ) reflected sound and αξ (1≦ξ≦Ξ) is a coefficient for taking into account decays of sounds due to reflection. Here, h→rξ(ω, θ)=[hr1ξ(ω, θ), . . . , hrMξ(ω, θ)]T represents the steering vectors of reflected sounds corresponding to the direct sound from direction θ. Typically, αξ (1≦ξ≦Ξ) is less than or equal to 1 (1≦ξ≦Ξ). For each reflected sound, if the number of reflections in the path from the sound source to the microphones is 1, αξ (1≦ξ≦Ξ) can be considered to represent the acoustic reflectance of the object from which the ξ-th reflected sound was reflected.
Since it is desirable that one or more reflected sounds be provided to the microphone array made up of M microphones, it is preferable that there is one or more reflective objects. From this point of view, a sound source, the microphone array, and one or more reflective objects are preferably in such a positional relation that a sound from the sound source is reflected off at least one reflective object before arriving at the microphone array, assuming that the sound source is located in the target direction for sound enhancement. Each of the reflective objects has a two-dimensional shape (for example a flat plate) or a three-dimensional shape (for example a parabolic shape). Each reflective object is preferably about the size of the microphone array or greater (greater by a factor of 1 to 2). In order to effectively use reflected sounds, the reflectance αξ (1≦ξ≦Ξ) of each reflective object is preferably at least greater than 0, and more preferably, the amplitude of a reflected sound arriving at the microphone array is greater than the amplitude of the direct sound by a factor of 0.2 or greater. For example, each reflective object is a rigid solid. Each reflective object may be a movable object (for example a reflector) or an immovable object (such as a floor, wall, or ceiling). Note that if an immovable object is set as a reflective object, the steering vector for the reflective object needs to be changed as the microphone array is relocated (see functions Ψ(θ) and Ψξ(θ) described later) and consequently the filter needs to be recalculated (re-set). Therefore, the reflective objects are preferably accessories of the microphone array for the sake of robustness against environmental changes (in this case, Ξ reflected sounds assumed are considered to be sounds reflected off the reflective objects). Here the “accessories of the microphone array” are “tangible objects capable of following changes of the position and orientation of the microphone array while maintaining the positional relation (geometrical relation) with the microphone array). A simple example may be a configuration where reflective objects are fixed to the microphone array.
In order to concretely describe advantages of the sound spot enhancement technique of the present invention, it is assumed in the following that Ξ=1, sounds are reflected once, and one reflective object exists at a distance of L meters from the center of the microphone array. The reflective object is a thick rigid object. Since Ξ=1 in this case, the symbol representing this is omitted and therefore equation (117a) can be rewritten as equation (117b):
{right arrow over (a)}(ω, θ)={right arrow over (h)}d(ω, θ)+α exp[−jωτ(θ)]·{right arrow over (h)}r(ω, θ) (117b)
An m-th element of the steering vector h→r(ω, θ)=[hr1(ω, θ), . . . , hrM(ω, θ)]T of a reflected sound can be given by equation (118a) in the same way that the steering vector of a direct sound is represented (see equation (114c)). The function Ψ(θ) outputs the direction from which a reflected sound arrives. Note that if the steering vector of a direct sound is written as equation (114d), an m-th element of the steering vector h→r(ω, θ)=[hr1(ω, θ), . . . , hrM(ω, θ)]T of a reflected sound is given by equation (118b). If Ξ≦2, an m-th element of a ξ-th (1≦ξ≦Ξ) steering vector h→rξ(ω, θ)=[hr1ξ(ω, θ), . . . , hrMξ(ω, θ)]T is given by equation (118c) or equation (118d). The function Ψξ(θ) outputs the direction from which the ξ-th (1≦ξ≦Ξ) reflected sound arrives.
Since the location of a reflective object can be set as appropriate, the direction from which a reflected sound arrives can be treated as a variable parameter.
Assuming that a flat-plate reflective object is near the microphone array (the distance L is not extremely large compared with the size of the microphone array), the coherence γ(ω, θ) is given by equation (119), where θ≠θs.
It will be apparent from equation (119) that the coherence γ(ω, θ) of equation (119) can be smaller than coherence γconv(ω, θ) of the conventional technique of equation (115). Since parameters (Ψ(θ) and L) that can be changed by relocating or reorienting the reflective object are included in the second to fourth terms of equation (119), there is a possibility that the first term, h→dH(ω, θ)h→d(ω, θ), can be eliminated.
For example, if a flat reflector is placed in such a position that the direction along which the microphones are arranged in a linear microphone array is normal to the reflector, Ψ(θ)=π−θ holds for the function Ψ(θ) and equation (120) folds for the difference τ(θ) in arrival time between a direct sound and a reflected sound. Therefore, the conditions of equatione (121) and (122) are generated for the elements of equation (119). Here, the symbol * is a complex conjugate operator.
Since the absolute value of h→dH(ω, θ)h→r(ω, θ) is sufficiently smaller than h→dH(ω, θ)h→d(ω, θ), the second and third terms of equation (119) are neglected. Then the coherence γ(ω, θ) can be approximated as equation (123):
{tilde over (γ)}(ω, θ)≈{1+α2exp[−jω(τ(θ)−τ(θs))]}{right arrow over (h)}dH(ω, θs){right arrow over (h)}d(ω, θ) (123)
Even if h→dH(ω, θ)h→d(ω, θ)≠0, an approximated coherence γ˜(ω, θ) has a minimal solution θ of equation (124), where q is an arbitrary positive integer. The range of q is restricted for each frequency.
That is, not only the coherence in a direction given by equation (116) but also the coherence in a direction given by equation (124) can be suppressed. Since suppression of coherence can reduce the power of noise, a sharp directivity can be achieved as schematically shown in
While
While for clarity of explanation of the principle of the sound spot enhancement technique of the present invention, it has been assumed in the foregoing that sounds waves arrive as plane waves, the essence of the spot sound enhancement technique of the present invention is that the transmission characteristic a→(ω, θ, D)=[a1(ω, θ, D), . . . , aM(ω, θ, D)]T is represented by the sum of the steering vector of a direct sound and the steering vectors of Ξ reflected sounds, as shown in Equation (117a), for example, as is apparent from the foregoing description. Accordingly, it will be understood that the technique is not limited to sound waves that arrive as plane waves, but is capable of achieving sound enhancement of sounds arriving as spherical waves with a higher directivity than the conventional technique.
Transfer functions a→(ω, θ, D) of sound waves that arrive as spherical waves will be described. Two types of spherical waves, namely direct sounds from a sound source and reflected sounds produced by reflection of that sound off a reflective object 300, together enter the microphones of a microphone array. Let the number of reflected sounds be denoted by Ξ. Here, Ξ is a predetermined integer greater than or equal to 1. Then, a transmission characteristic a→(ω, θ, D)=[a1(ω, θ, D), . . . , aM(ω, θ, D)]T can be expressed by the sum of a transmission characteristic of a direct sound that comes from a position (θs, D) that can be a target of sound enhancement and directly arrives at the microphone array and the transmission characteristic(s) of one or more reflected sounds that are produced by reflection of that sound off a reflective object and arrive at the microphone array. Specifically, the transmission characteristic can be represented as the sum of the steering vector of the direct sound and the steering vector of Ξ reflected sounds whose decays due to reflection and arrival time differences from the direct sound are corrected, as shown in equation (125), where τξ(θ, D) is the arrival time difference between the direct sound and a ξ-th (1≦ξ≦Ξ) reflected sound and αξ(1≦ξ≦Ξ) is a coefficient for taking into account decays of sounds due to reflection. Here, h→d(ω, θ, Dh)=[hd1(ω, θ, Dh), . . . , hdM(ω, θ, Dh)]T represents the steering vector of a direct sound from position (θs, D) and h→rξ(ω, θ, D)=[hr1ξ(ω, θ, D), . . . , hrMξ(ω, θ, D)]T represents the steering vector of a reflected sound corresponding to the direct sound from position (θs, D). A note about the term “steering vector” will be added here. A “steering vector” is also called “direction vector” and, as the name suggests, represents typically a complex vector that is dependent on “direction”. From this view point, it is more precise to refer a complex vector that is dependent on a position (θs, D) as an “extended steering vector”, for example. However, for the sake of simplicity, the complex vector that is dependent on a position (θs, D) will be also simply referred to as the “steering vector” herein. Typically, αξ (1≦ξ≦Ξ) is less than or equal to 1 (1≦ξ≦Ξ). For each reflected sound, if the number of reflections in the path from the sound source to the microphones is 1, αξ (1≦ξ≦Ξ) can be considered to represent the acoustic reflectance of the object from which the ξ-th reflected sound was reflected.
In equation (125), an m-th element hdm(ω, θ, Dh) of the steering vector h→d(ω, θ, Dh) of the direct sound can be given by equation (125a), for example. Here m is an integer that satisfies 1≦m≦M, c represents the speed of sound, and j is an imaginary unit. In an appropriately set spatial coordinate system, v→θ,D(d) represents a position vector of a position (θ, D), u→m represents a position vector of an m-th microphone, the symbol ∥•∥ represents a norm, and f(∥v→θ,D(d)−u→m∥ is a function representing a distance decay of a sound wave. For example, f(∥v→θ,D(d)−u→m∥)=1/∥v→θ,D(d)−u→m∥ and in this case equation (125a) can be written as equation (125b).
In equation (125), an m-th element hrmξ(ω, θ, D) of the steering vector h→rξ(ω, θ, D)=hr1ξ(ω, θ, D), . . . , hrMξ(ω, θ, D)]T can be given by equation (126a), like the steering vector of the direct sound (see equation (125a)). Here, m is an integer that satisfies 1≦m≦M, c represents the speed of sound, and j is an imaginary unit. In the spatial coordinate system, v→θ,D(ξ) represents a position vector of a position that is an mirror image of a position (θ, D) with respect to the reflecting surface of a ξ-th reflector, u→m, represents the position vector of the m-th microphone, the symbol ∥•∥ represents a norm, and f(∥v→θ,D(ξ)−u→m∥) is a function representing a distance decay of a sound wave. For example, f(∥v→θ,D(ξ)−u→m∥)=1/∥v→θ,D(ξ)−u→m∥ and in this case equation (126a) can be written as equation (126b).
Note that a ξ-th arrival time difference τξ(θ, D) and positional vector v→θ,D(ξ) can be theoretically calculated on the basis of the positional relation among the position (θ, D), the microphone array and the ξ-th reflective object when the positional relation is determined.
Unlike the conventional techniques, the sound spot enhancement technique of the present invention positively takes into account reflected sounds and therefore is capable of a sharp directive sound spot enhancement. This will be described by taking two sound sources by way of example. It is difficult to spot-enhance sounds emanating from two sound sources A and B at different distances from a microphone array but in about the same directions viewed from the microphone array as illustrated in
In contrast, the sound spot enhancement technique of the present invention positively takes into account reflected sounds therefore virtual sound sources A(ξ) and B(ξ) of ξ-th reflected sounds exist at positions of mirror images of sound sources A and B with respect to the reflecting surface of the ξ-th reflector 300 from the view point of the microphone array as illustrated in
Thus far, distance Dh has been fixed in order to explain how high directivity can be achieved. Accordingly, spatial correlation matrices Q(ω) has been written as (110a) and (110b). However, by taking into account the correlation between transfer functions of M channels for different distances Dδ (δ=1, 2, . . . , G), the amount of information concerning a sound field can be increased to construct a spatial correlation matrix that provides more precise filters. The spatial correlation matrix Q(ω) can be given by equation (110c). A set to which indices φ of directions θφ belong is denoted by Φ (|Φ|=P) and a set to which indices δ of distances Dδ belong is denoted by Δ (|Δ|)=G).
Then, by using the spatial correlation matrix Q(ω) given by equation (110c), a filter W→(ω, θs, Dh) designed by the minimum variance distortionless response (MVDR) method can be written as equation (109a) instead of equation (109).
As has been described, the essence of the sound spot enhancement technique of the present invention is that the transmission characteristic a→(ω, θ, D)=[a1(ω, θ, D), . . . , aM(ω, θ, D)]T is represented by the sum of the steering vector of a direct sound and the steering vectors of Ξ reflected sounds. Since this does not affect the filter design concept, filters W→(ω, θs, Dh) can be designed by a method other than the minimum variance distortionless response (MVDR) method.
Methods other than the MVDR method described above will be described. They are: <1> a filter design method based on SNR maximization criterion, <2> a filter design method based on power inversion, <3> a filter design method using MVDR with one or more suppression points (directions in which the gain of noise is suppressed) as a constraint condition, <4> a filter design method using delay-and-sum beam forming, <5> a filter design method using the maximum likelihood method, and <6> a filter design method using the adaptive microphone-array for noise reduction (AMNOR) method. For <1> the filter design method based on SNR maximization criterion and <2> the filter design method based on power inversion, refer to Reference 2 listed below. For <3> the filter design method using MVDR with one or more suppression points (directions in which the gain of noise is suppressed) as a constraint condition, refer to Reference 3 listed below. For <6> the filter design method using the adaptive microphone-array for noise reduction (AMNOR) method, refer to Reference 4 listed below.
<1> Filter Design Method Based on SNR Maximization Criterion
In the filter design method based on SNR maximization criterion, a filter W→(ω, θs, Dh) is determined on the basis of a criterion of maximizing the SN ratio (SNR) from a position (θs, Dh). The spatial correlation matrix for a sound from the position (θs, Dh) is denoted by Rss(ω) and a spatial correlation matrix for a sound from a position other than the position (θs, Dh) is denoted by Rnn(ω). Then the SNR can be given by equation (128). Here, Rss(ω) can be given by equation (129) and Rnn(ω) can be given by equation (130). Transfer functions a→(ω, θs, Dh)=[a1(ω, θs, Dh), . . . , aM(ω, θs, Dh)]T can be given by equation (125), for example (to be precise, equation (125) where θ is replaced with θs and D replaced with Dh). A set to which indices φ of directions θφ belong is denoted by Φ (|Φ|=P) and a set to which indices δ of distances Dδ belong is denoted by Δ (|Δ|=G).
The filter W→(ω, θs, Dh) that maximizes the SNR of equation (128) can be obtained by setting the gradient relating to filter W→(ω, θs, Dh) to zero, that is, by equation (131).
Thus, the filter W→(ω, θs, Dh) that maximizes the SNR of equation (128) can be given by equation (132):
{right arrow over (W)}(ω, θs, Dh)=Rnn−1(ω){right arrow over (a)}(ω, θs, Dh) (132)
Equation (132) includes the inverse matrix of the spatial correlation matrix Rnn(ω) of a sound from a position other than the position (θs, Dh). It is known that the inverse matrix of Rnn(ω) can be replaced with the inverse matrix of a spatial correlation matrix Rxx(ω) of a whole input including sounds from (1) the position (θs, Dh) and (2) sounds from a position other direction (θs, Dh). Here, Rxx(ω)=Rss(ω)+Rnn(ω)=Q(ω). That is, the filter W→(ω, θs, Dh) that maximizes the SNR of equation (128) may be obtained by equation (133):
{right arrow over (W)}(ω, θs, Dh)=Rxx−1(ω){right arrow over (a)}(ω, θs, Dh) (133)
<2> Filter Design Method Based on Power Inversion
In the filter design method based on power inversion, a filter W→(ω, θs, Dh) is determined on the basis of a criterion of minimizing the average output power of a beam former while a filter coefficient for one microphone is fixed at a constant value. Here, an example where the filter coefficient for the first microphone among M microphones is fixed will be described. In this design method, a filter W→(ω, θs, Dh) is designed that minimizes the power of sounds from all positions (all positions that can be assumed to be sound source positions)) by using a spatial correlation matrix Rxx(ω) (see equation (134)) under the constraint condition of equation (135). Transfer functions a→(ω, θs, dh)=[a1(ω, θs, Dh), . . . , aM(ω, θs, Dh)]T can be given by equation (125), for example (to be precise, by equation (125) where θ is replaced with θs and D is replaced with Dh).
It is known that the filter W→(ω, θs, Dh) that is an optimum solution of equation (134) can be given by equation (136) (see Reference 2 listed below).
{right arrow over (W)}(ω, θs, Dh)=Rxx−1(ω){right arrow over (G)} (136)
<3> Filter Design Method Using MVDR with One or More Suppression Points as Constraint Condition
In the MVDR method described earlier, a filter W→(ω, θs, Dh) has been designed under the single constraint condition that a filter is obtained that minimizes the average output power of a beam former given by equation (107) (that is, the power of noise which is sounds from directions other than a position (θs, Dh) under the constraint condition that the filter passes sounds from a position (θs, Dh) in all frequency bands as expressed by equation (108). According to the method, the power of noise can be generally suppressed. However, the method is not necessarily preferable if it is previously known that there is a noise source(s) that has strong power in one or more particular directions. If this is the case, a filter is required that strongly suppresses one or more particular known directions (that is, suppression points) in which the noise source(s) exist. Therefore, the filter design method described here obtains a filter that minimizes the average output power of the beam former given by equation (107) (that is, minimizes the average output power of sounds from directions other than a position (θs, Dh) and the suppression points) under the constraint conditions that (1) the filter passes sounds from the position (θs, Dh) in all frequency bands and that (2) the filter suppresses sounds from B known suppression points (θN1, DG1), (θN2, DG2), . . . , (θNB, DGB). (B is a predetermined integer greater than or equal to 1) in all frequency bands. Let a set of indices φ of directions from which noise arrives be denoted by {1, 2, . . . , P}, then Nj ε {1, 2, . . . , P} (where j ε {1, 2, . . . , B}) and B≦P−1, as has been described earlier. Let a set of indices δ of distances to sound sources be denoted by {1, 2, . . . , G}, then Gj ε {1, 2, . . . , G} (where j ε {1, 2, . . . , B}) and B≦G−1.
Let a→(ω, θi, Dg)=[a1(ω, θi, Dg), . . . , aM(ω, θi, Dg)]T be transfer functions between a sound source assumed to be located in a position (θi, Dg) and the M microphones at a frequency ω, in other words, transfer functions of a sound from a position (θi, Dg) at a frequency ω arriving at the microphones of a microphone array, then a constraint condition can be given by equation (137). Here, for indices i and g, (i, g) ε {(s, h), (N1, G1), (N2, G2), . . . , (NB, GB)}, transfer functions a→(ω, θi, Dg)=[a1(ω, θi, Dg), . . . , aM(ω, θi, Dg)]T can be given by equation (125) (to be precise, by equation (125) where θ is replaced with θi and D is replaced with Dh), and fi,g(ω) represents a pass characteristic at a frequency ω for a position (θi, Dg).
{right arrow over (W)}H(ω, θs, Dh){right arrow over (a)}(ω, θi, Dg)=fi,g(ω)
(i, g)ε{(s, h),(N1, G1),(N2, G2), . . . , (NB, GB)} (137)
Equation (137) can be represented as a matrix, for example written as equation (138). Here, A→(ω, θs, Dh)=[([a→(ω, θs, Dh), a→(ω, θN1, DG1), . . . , a→(ω, θNB, DGB)].
{right arrow over (w)}H(ω, θs, Dh){right arrow over (A)}(ω, θs, Dh)={right arrow over (F)} (138)
where
{right arrow over (F)}=[fs,h(ω,fN1,G1(ω), . . . , fNB,GB(ω)]
Taking into consideration the constraint conditions that (1) the filter passes sounds from the position (θs, Dh) in all frequency bands and that (2) the filter suppresses sounds from B known suppression points (θN1, DG1), (θN2, DG2), . . . , (θNB, DGB), in all frequency bands, ideally fs,h(ω)=1.0 and fi,g(ω)=0.0 ((i, g) ε {(N1, G1), (N2, G2), . . . , (NB, GB)}) should be set. This means that the filter completely passes sounds in all frequency bands from the position (θs, Dh) and completely blocks sounds in all frequency bands from B known suppression points (θN1, DG1), (θN2, DG2), . . . , (θNB, DGB). In reality, however, it is difficult in some situations to effect such control as completely passing all frequency bands or completely blocking all frequency bands. In such a case, the absolute value of fs,h(ω) is set to a value close to 1.0 and the absolute value of fi,g(ω) ((i, g) ε {(N1, G1), (N2, G2), . . . , (NB, GB)}) is set to a value close to 0.0. Of course, fi,g
According to the filter design method described here, the filter W→(ω, θs, Dh) that is an optimum solution of equation (107) under the constraint condition given by equation (138) can be given by equation (139) (see Reference 3 listed below). While a spatial correlation matrix Q(ω) that can be given by equation (110c) has been used, a spatial correlation matrix given by equation (110a) or (110b) may be used.
{right arrow over (W)}(ω, θs, Dh)=Q−1(ω){right arrow over (A)}(ω, θs, Dh)({right arrow over (A)}H(ω, θs, Dh)Q−1(ω){right arrow over (A)}(ω, θs, Dh))−1{right arrow over (F)} (139)
<4> Filter Design Method Using Delay-And-Sum Beam Forming
Assuming that direct and reflected sounds arriving are plane waves, then a filter W→(ω, θs, Dh) can be given by equation (140) according to the delay-and-sum beam forming. That is, the filter W→(ω, θs, Dh) can be obtained by normalizing a transmission characteristic a→(ω, θs, Dh). The transmission characteristic a→(ω, θs, Dh)=[a1(ω, θs, Dh), . . . , aM(ω, θs, Dh)]T can be given by equation (125) (to be precise, by equation (125) where θ is replaced with θs and D is replaced with Dh). The filter design method does not necessarily achieve a high filtering accuracy but requires only a small quantity of computation.
<5> Filter Design Method Using Maximum Likelihood Method
By excluding spatial information concerning sounds from a target direction from a spatial correlation matrix Q(ω, Dh) in the MVDR method described earlier, flexibility of suppression of noise can be improved and the power of noise can be further suppressed. Therefore, in the filter design method described here, the spatial correlation matrix Q(ω, Dh) is written as the second term of the right-hand side of equation (110a), that is, equation (110d). A filter W→(ω, θs, Dh) can be given by equation (109) or (139). Here, the spatial correlation matrix included in equatione (109) and (139) is a spatial correlation matrix given by equation (110d).
Alternatively, spatial information concerning sounds from the position (θs, Dh) may be excluded from the spatial correlation matrix Q(ω). In that case, a spatial correlation matrix Q(ω) is given by equation (110e) in the filter design method described here. A filter W→(ω, θs, Dh) can be given by equation (109) or (139). Here, the spatial correlation matrix included in equatione (109) and (139) is given by equation (110e).
<6> Filter Design Method Using AMNOR Method
The AMNOR method obtains a filter that allows some amount of decay D of a sound from a target direction by trading off the amount of decay D of the sound from the target direction against the power of noise remaining in a filter output signal (for example, the amount of decay D is maintained at a certain threshold D^ or less) and, when a mixed signal of [a] a signal produced by applying transfer functions between a sound source and microphones to a virtual signal (hereinafter referred to as the virtual signal) from a target direction and [b] noise (obtained by observation with M microphones in a noisy environment without a sound from the target direction) is input, outputs a filter output signal that reproduces best the virtual signal in terms of least squares error (that is, the power of noise contained in a filter output signal is minimized).
The filter design method described here incorporates the concept of distance into the AMNOR method and can be considered to be similar to the AMNOR method. Specifically, the method obtains a filter that allows some amount of decay D of a sound from a position (θs, Dh) by trading off the amount of decay D of the sound from the position (θs, Dh) against the power of noise remaining in a filter output signal (for example, the amount of decay D is maintained at a certain threshold D^ or less) and, when a mixed signal of [a] a signal produced by applying transfer functions between a sound source and microphones to a virtual target signal from a position (θs, Dh) (hereinafter referred to as the virtual target signal) and [b] noise (obtained by observation with M microphones in a noisy environment without a sound from the position (θs, Dh)) is input, outputs a filter output signal that reproduces best the virtual target signal in terms of least squares error (that is, the power of noise contained in a filter output signal is minimized).
According to the filter design method described here, a filter W→(ω, θs, Dh) can be given by equation (141) as in the AMNOR method (see Reference 4 listed below). Here, Rss(ω) can be given by equation (126) and Rnn(ω) can be given by equation (127). Transfer functions a→(ω, θs, Dh)=[a1(ω, θs, Dh), . . . , aM(ω, θs, Dh)]T can be given by equation (125) (to be precise, by equation (125) where θ is replaced with θs and D is replaced with Dh).
{right arrow over (W)}(ω, θs, Dh)=Ps{right arrow over (a)}(ω, θs, Dh)(Rnn (ω)+PsRss (ω))−1 (141)
Ps is a coefficient that assigns a weight to the level of the virtual target signal and called the virtual target signal level. The virtual target signal level Ps is a constant that is not dependent on frequencies. The virtual target signal level Ps may be determined empirically or may be determined so that the difference between the amount of decay D of a sound from the position (θs, Dh) and the threshold D^ is within an arbitrarily predetermined error margin. The latter case will be described. The frequency response F(ω) of the filter W→(ω, θs, Dh) to a sound from a position (θs, Dh) can be given by equation (142). Let the amount of decay D(Ps) when using the filter W→(ω, θs, Dh) given by equation (141) be denoted by D(Ps), then the amount of decay D(Ps) can be defined by equation (143). Here, ω0 represents the upper limit of frequency ω (typically, a higher-frequency adjacent to a discrete frequency ω). The amount of decay D(Ps) is a monotonically decreasing function of Ps. Therefore, a virtual target signal level Ps such that the difference between the amount of decay D(Ps) and the threshold D^ is within an arbitrarily predetermined error margin can be obtained by repeatedly obtaining the amount of decay D(Ps) while changing Ps with the monotonicity of D(Ps).
<Variation>
In the foregoing description, the spatial correlation matrices Q(ω), Rss(ω) and Rnn(ω) are expressed using transfer functions. However, the spatial correlation matrices Q(ω), Rss(ω) and Rnn(ω) can also be expressed using the frequency-domain signals X→(ω, k) described earlier. While the spatial correlation matrix Q(ω) will be described below, the following description applies to Rss(ω) and Rnn(ω) as well. (Q(ω) can be replaced with Rss(ω) or Rnn(ω)). The spatial correlation matrix Rss(ω) can be obtained using frequency-domain representations of analog signals obtained by observation with a microphone array (including M microphones) in an environment where only sounds from a position (θs, Dh) exist. The spatial correlation matrix Rnn(ω) can be obtained using frequency-domain representations of an analog signal obtained by observation with a microphone array (including M microphones) in an environment where no sounds from a position (θs, Dh) exist (that is, a noisy environment).
The spatial correlation matrix Q(ω) using frequency domain signals X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T can be given by equation (144). Here, the operator E[•] represents a statistical averaging operation. When viewing a discrete time series of an analog signal received with a microphone array (including M microphones) as a stochastic process, the operator E[•] represents a arithmetic mean value (expected value) operation if the stochastic process is a so-called wide-sense stationary process or a second-order stationary process. In this case, the spatial correlation matrix Q(ω) can be given by equation (145) using frequency-domain signals X→(ω, k−i) (i=0, 1, . . . , ζ−1) of a total of ζ current and past frames stored in a memory, for example. When i=0, a k-th frame is the current frame. Note that the spatial correlation matrix Q(ω) given by equation (1441) or (145) may be recalculated for each frame or may be calculated at regular or irregular interval, or may be calculated before implementation of an embodiment, which will be described later (especially when Rss(ω) or Rnn(ω) is used, the spatial correlation matrix Q(ω) is preferably calculated beforehand by using frequency-domain signals obtained before implementation of the embodiment). If the spatial correlation matrix Q(ω) is recalculated for each frame, the spatial correlation matrix Q(ω) depends on the current and past frames and therefore the spatial correlation matrix will be explicitly represented as Q(ω, k) as in equatione (144a) and (145a).
If the spatial correlation matrix Q(ω, k) represented by equation (144a) or (145a) is used, the filter W→(ω, θs, Dh) also depends on the current and past frames and therefore is explicitly represented as W→(ω, θs, Dh, k). Then, a filter W→(ω, θs, Dh) represented by any of equatione (109), (132), (133), (136), (139) and (141) described with the filter design methods described above is rewritten as equatione (109m), (132m), (133m), (136m), (139m) or (141m).
<<First Embodiment of Sound Spot Enhancement Technique>>
[Step S21]
The filter design section 660 calculates beforehand a filter W→(ω, θi, Dg) for each frequency for each of discrete possible positions (θi, Dg) from which sounds to be enhanced can arrive. The filter design section 660 calculates filters W→(ω, θ1, D1), . . . , W→(ω, θi, D1), . . . , W→(ω, θI, D1), . . . , W→(ω, θ1, D2), . . . , W→(ω, θi, D2), . . . , W→(ω, θI, D2), . . . , W→(ω, θ1, Dg), . . . , W→(ω, θi, Dg), . . . , W→(ω, θI, Dg), . . . , W→(ω, θ1, DG), . . . , W→(ω, θi, DG,), . . . , W→(ω, θI, DG) (1≦i≦I, 1≦g≦G, ω ε Ω; i and g are integers and Ω is a set of frequencies ω), where I is the total number of discrete directions from which sounds to be enhanced can arrive (I is a predetermined integer greater than or equal to 1 and satisfies I≦P) and G is the number of the discrete distances (G is a predetermined integer greater than or equal to 1).
To do so, transfer functions a→(ω, θi, Dg)=[a1(ω, θi, Dg), . . . , aM(ω, θi, Dg)]T (1≦i≦I, 1≦g≦G, ω ε Ω) need to be obtained except for the case of <Variation> described above. The transfer functions a→(ω, θi, Dg)=[a1(ω, θi, Dg), . . . , aM(ω, θi, Dg)]T can be calculated practically according to equation (125) (to be precise, by equation (125) where θ is replaced with θi and D is replaced with Dg) on the basis of the arrangement of the microphones in the microphone array and environmental information such as the positional relation of a reflective object such as a reflector, floor, walls, and ceiling to the microphone array, the arrival time difference between a direct sound and a ξ-th (1≦ξ≦Ξ)reflected sound, and the acoustic reflectance of the reflective object. Note that if <3> the filter design method using MVDR with one or more suppression points as a constraint condition is used, the indices (i, g) of the directions used for calculating the transfer functions a→(ω, θi, Dg) (1≦i≦I, 1≦g≦G, ω ε Ω) preferably cover all of indices (N1, G1), (N2, G2), . . . , (NB, GB) of directions of at least B suppression positions. In other words, B indices N1, N2, . . . , NB are set to any of different integers greater than or equal to 1 and less than or equal to I and the B indices G1, G2, . . . , GB are set to any of different integers greater than or equal to 1 and less than or equal to G.
The number Ξ of reflected sounds is set to an integer that satisfies 1≦Ξ. The number Ξ is not limited and can be set to an appropriate value according to the computational capacity and other factors.
To calculate steering vectors, equatione (125a), (125b), (126a), or (126b), for example, can be used. Note that transfer functions obtained by actual measurements in a real environment, for example, may be used for designing the filters instead of using equatione (125).
Then, W→(ω, θi, Dg) (1≦i≦I, 1≦g≦G) is obtained according to any of equatione (109), (109a), (132), (133), (136), (139), (140) and (141), for example, by using the transfer functions a→(ω, θi, Dg), except for the case described in <Variation>. Note that if equation (109), (109a), (133), (136) or (139) is used, the spatial correlation matrix Q(ω) (or Rxx(ω)) can be calculated according to equation (110b), except for the case described with respect to <5> the filter design method using the maximum likelihood method. If equation (109), (109a), (133), (136) or (139) is used according to <5> the filter design method using the maximum likelihood method described earlier, the spatial correlation matrix Q(ω) (or Rxx(ω)) can be calculated according to equation (110c) or (110d). If equation (132) is used, the spatial correlation matrix Rnn(ω) can be calculated according to equation (130). I×G×|Ω| filters W→(ω, θi, Dg) (1≦i≦I, 1≦g≦G, ω ε Ω) are stored in the storage 690, where |Ω| represents the number of the elements of the set Ω.
[Step S22]
The M microphones 200-1, . . . , 200-M making up the microphone array are used to pick up sounds, where M is an integer greater than or equal to 2.
There is no restraint on the arrangement of the M microphones. However, a two- or three-dimensional arrangement of the M microphones has the advantage of eliminating uncertainty of a direction from which sounds to be enhanced arrive. That is, a planar or steric arrangement of the microphones can avoid the problem with a horizontal linear arrangement of the M microphones that a sound arriving from a front direction cannot be distinguished from a sound arriving from right above, for example. In order to provide a wide range of directions that can be set as sound-pickup directions, each microphone preferably has a directivity capable of picking up sounds with a certain level of sound pressure in potential target directions θs which are sound-pickup directions. Accordingly, microphones having relatively weak directivity, such as omnidirectional microphones or unidirectional microphones are preferable.
[Step S23]
The AD converter 610 converts the analog signals (pickup signals) picked up with the M microphones 200-1, . . . , 200-M to digital signals x→(t)=[x1(t), . . . , xM(t)]T, where t represents the index of a discrete time.
[Step S24]
The frame generator 620 takes inputs of the digital signals x→(t)=[x1(t), . . . , xM(t)]T output from the AD converter 610, stores N samples in a buffer on a channel by channel basis, and outputs digital signals x→(k)=[x→1(k), . . . , x→M(k)]T in frames, where k is an index of a frame-time number and x→m(k)=[xm((k−1)N+1), . . . , xm(kN)] (1≦m≦M). N depends on the sampling frequency and 512 is appropriate for sampling at 16 kHz.
[Step S25]
The frequency-domain transform section 630 transforms the digital signals x→(k) in frames to frequency-domain signals X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T and outputs the frequency-domain signals, where 107 is an index of a discrete frequency. One way to transform a time-domain signal to a frequency-domain signal is fast discrete Fourier transform. However, the way to transform the signal is not limited to this. Other method for transforming to a frequency domain signal may be used. The frequency-domain signal X→(ω, k) is output for each frequency ω and frame k at a time.
[Step S26]
The filter applying section 640 applies the filter W→(ω, θs, Dh) corresponding to a position (θs, Dh) to be enhanced to the frequency-domain signal X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T in each frame k for each frequency ω ε Ω and outputs an output signal Y(ω, k, θs, Dh) (see equation (146)). The indices s and h of the position (θs, Dh) is s ε {1, . . . , I} and h ε {1, . . . , G} and the filter W→(ω, θs, Dh) is stored in the storage 690. Therefore, the filter applying section 640 only has to retrieve the filter W→(ω, θs, Dh) that corresponds to the position (θs, Dh) to be enhanced from the storage 690, for example, in the process at step S26. If the index s of the direction θs does not belong to the set {1, . . . , I} or the index h of direction Dh does not belong to the set {1, . . . , G}, that is, the filter W→(ω, θs, Dh) that corresponds to the position (θs, Dh) has not been calculated in the process at step S21, the filter design section 660 may calculate at this moment the filter W→(ω, θs, Dh) that corresponds to the position (θs, Dh) or filter W→(ω, θs′, Dh) or W→(ω, θs, Dh′) or →(ω, θs′, Dh′) that corresponds to a direction θs′ close to the direction θs and/or a distance Dh′ close to the distance Dh may be used.
Y(ω, k, θs, Dh)={right arrow over (W)}H(ω, θs, Dh){right arrow over (X)}(ω, k) ∀ω ε Ω (146)
[Step S27]
The time-domain transform section 650 transforms the output signal Y(ω, k, θs, Dh) of each frequency ω ε Ω in a k-th frame to a time domain to obtain a time-domain frame signal y(k) in the k-th frame, then combines the obtained frame time-domain signals y(k) in the order of frame-time number index, and outputs a time-domain signal y(t) in which the sound from a position (θs, Dh) is enhanced. The method for transforming a frequency-domain signal to a time-domain signal is inverse transform of the transform used in the process at step S25 and may be fast discrete inverse Fourier transform, for example.
While the first embodiment has been described here in which the filters W→(ω, θi, Dg) are calculated beforehand in the process at step S21, the filter design section 660 may calculate the filter W→(ω, θs, Dh) for each frequency after the position (θs, Dh) is determined, depending on the computational capacity of the sound spot enhancement apparatus 3.
<<Second Embodiment of Sound Spot Enhancement Technique>>
[Step S31]
M microphones 200-1, . . . , 200-M making up a microphone array is used to pick up sounds, where M is an integer greater than or equal to 2. The arrangement of the M microphones is as described in the first embodiment.
[Step S32]
The AD converter 610 converts analog signals (pickup signals) picked up with the M microphones 200-1, . . . , 200-M to digital signals x→(t)=[x1(t), . . . , xM(t)]T, where t represents the index of a discrete time.
[Step S33]
The frame generator 620 takes inputs of the digital signals x→(t)=[x1(t), . . . , xM(t)]T output from the AD converter 610, stores N samples in a buffer on a channel by channel basis, and outputs digital signals x→(k)=[x→1(k), . . . , x→M(k)]T in frames, where k is an index of a frame-time number and x→m(k)=[xm((k−1)N+1), . . . , xm(kN)] (1≦m≦M). N depends on the sampling frequency and 512 is appropriate for sampling at 16 kHz.
[Step S34]
The frequency-domain transform section 630 transforms the digital signals x→(k) in frames to frequency-domain signals X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T and outputs the frequency-domain signals, where ω is an index of a discrete frequency. One way to transform a time-domain signal to a frequency-domain signal is fast discrete Fourier transform. However, the way to transform the signal is not limited to this. Other method for transforming to a frequency domain signal may be used. The frequency-domain signal X→(ω, k) is output for each frequency ω and frame k at a time.
[Step S35]
The filter calculating section 661 calculates the filter W→(ω, θs, Dh, k) (ω ε Ω; Ω is a set of frequencies ω) that corresponds to the position (θs, Dh) to be used in a current k-th frame.
To do so, transfer functions a→(ω, θs, Dh)=[a1(ω, θs, Dh), . . . , aM(ω, θs, Dh)]T (ω ε Ω) need to be provided. Transfer functions a→(ω, θs, Dh)=[a1(ω, θs, Dh), . . . , aM(ω, θs, Dh)]T can be calculated practically according to equation (17a) (to be precise, by equation (125) where θ is replaced with θs and D is replaced with Dh) on the basis of the arrangement of the microphones in the microphone array and environmental information such as the positional relation of a reflective object such as a reflector, floor, walls, or ceiling to the microphone array, the arrival time difference between a direct sound and a ξ-th reflected sound (1≦ξ≦Ξ), and the acoustic reflectance of the reflective object. Note that if <3> the filter design method using MVDR with one or more suppression points as a constraint condition is used, transfer functions a→(ω, θNj, DGj) (1≦j≦B, ω ε Ω) also need to be obtained. The transfer functions can be calculated practically according to equation (125) (to be precise, by equation (125) where θ is replaced with θNj and D is replaced with DGj) on the basis of the arrangement of the microphones in the microphone array and environmental information such as the positional relation of a reflective object such as a reflector, a floor, a wall, or ceiling to the microphone array, the arrival time difference between a direct sound and a ξ-th reflected sound (1≦ξ≦Ξ), and the acoustic reflectance of the reflective object.
The number Ξ of reflected sounds is set to an integer that satisfies 1≦Ξ. The number Ξ is not limited and can be set to an appropriate value according to the computational capacity and other factors.
To calculate steering vectors, equatione (125a), (125b), (126a), or (126b), for example, can be used. Note that transfer functions obtained by actual measurements in a real environment, for example, may be used for designing the filters instead of using equation (125).
Then, the filter calculating section 661 calculates filters W→(ω, θs, Dh, k) (ω ε Ω) according to any of equatione (109m), (132m), (133m), (136m), (139m) and (141m) using the transfer functions a→(ω, θs, Dh) (ω ε Ω) and, if needed, the transfer functions a→(ω, θNj, DGj) (1≦j≦B, ω ε Ω). Note that the spatial correlation matrix Q(ω) (or Rxx(ω)) can be calculated according to equation (144a) or (145a). In the calculation of the spatial correlation matrix Q(ω), frequency-domain signals X→(ω, k−i) (i=0, 1, . . . , ζ−1) of a total of ζ current and past frames stored in the storage 690, for example, are used.
[Step S36]
The filter applying section 640 applies the filter W→(ω, θs, Dh, k) corresponding to the target direction θs to be enhanced to the frequency-domain signal X→(ω, k)=[X1(ω, k), . . . , XM(ω, k)]T in each frame k for each frequency ω ε Ω and outputs an output signal Y(ω, k, θS, Dh) (see equation (147)).
Y(ω, k, θs, Dh)={right arrow over (W)}H(ω, θs, Dh, k){right arrow over (X)}(ω, k) ∀ω ε Ω (147)
[Step S37]
The time-domain transform section 650 transforms the output signal Y(ω, k, θs, Dh) of each frequency ω ε Ω of a k-th frame to a time domain to obtain a time-domain frame signal y(k) in the k-th frame, then combines the obtained frame time-domain signals y(k) in the order of frame-time number index, and outputs a time-domain signal y(t) in which the sound from the position (θs, Dh) is enhanced. The method for transforming a frequency-domain signal to a time-domain signal is inverse transform of the transform used in the process at step S34 and may be fast discrete inverse Fourier transform, for example.
A filter W→(ω, θi) that corresponds to a direction θi can be calculated by Σg=1GβgW→(ω, θi, Dg) in the sound spot enhancement technique, where βg [1≦g≦G] is a weighting factor, which preferably satisfies Σg=1Gβg=1 and preferably 0≦βg [1≦g≦G]. Note that the filter W→(ω, θi, Dg) may be a filter represented using transfer functions measured in a real environment.
[Experimental Example of Sound Spot Enhancement Technique]
Results of experimental examples on the sound spot enhancement according to the first embodiment of the sound spot enhancement technique of the present invention (the minimum variance distortionless response (MVDR) method under a single constraint condition) will be described. The experiments were conducted in the same environment illustrated in
<Example Applications>
Figuratively speaking, the sound spot enhancement technique is equivalent to generation of a clear image from an unsharp, blurred image and is useful for obtaining detailed information about an acoustic field. The following is description of examples of services where the sound spot enhancement technique of the present invention is useful.
A first example is creation of contents that are combination of audio and video. The use of an embodiment of the sound spot enhancement technique of the present invention allows the target sound from a great distance to be clearly enhanced even in a noisy environment with noise sounds (sounds other than target sounds). Therefore, for example sounds in a particular area corresponding to a zoomed-in moving picture of a dribbling soccer player that was shot from outside the field can be added to the moving picture.
A second example is an application to a video conference (or an audio teleconference). When a conference is held in a small room, the voice of a human speaker can be enhanced to a certain degree with several microphones according to a conventional technique. However, in a large conference room (for example, a large space where there are human speakers at a distance of 5 m or more from microphones), it is difficult to clearly enhance the voice of a human speaker at a distance with the conventional techniques by the conventional method and a microphone needs to be placed in front of each human speaker. In contrast, the use of an embodiment of the sound spot enhancement technique of the present invention is capable of clearly enhancing sounds from a particular area farther from a particular area and therefore enables construction of a video conference system that is usable in a large conference room without having to place a microphone in front of each human speaker. Furthermore, since sounds from a particular area can be enhanced, restrictions on the locations of conference participants with respect to the locations of microphones can be relaxed.
<Configurations of Implementation of Sound Enhancement Technique>
Exemplary configurations of implementations of the sound enhancement techniques of the present invention will be described below with reference to
In an exemplary configuration of an implementation illustrated in
In an exemplary configuration illustrated in
In an exemplary configuration illustrated in
An exemplary configuration of an implementation illustrated in
In an extended exemplary configuration illustrated in
If a microphone array is provided in the movable reflector 310, the microphone array may be placed in the movable reflector 310 so that the sound pickup hole of each of the microphones of the microphone array provided in the movable reflector 310 is positioned at the flat surface (the opening surface) opposite the flat surface of the movable reflector 310 that is opposable to the opening surface of the supporting board 400. If a microphone array is provided in the movable reflector 320, the microphone array may be placed in the movable reflector 320 so that the sound pickup hole of each of the microphones of the microphone array provided in the movable reflector 320 is positioned at the flat surface (the opening surface) opposite the flat surface of the movable reflector 320 that can form the same plane as the opening surface of the supporting board 400. Of course, a microphone array may be provided in at least one of the movable reflectors 310 and 320 so that both surfaces of the movable reflector 310 and/or 320 are opening surfaces.
[A] If a microphone array is provided in at least one of the movable reflectors 310 and 320 and, in addition, the opening surface of the movable reflector 310 is a flat surface opposable to the opening surface of the supporting board 400 or the opening surface of the movable reflector 320 is a flat surface that can form the same plane as the opening surface of the supporting board 400, positioning the movable reflector 310 and/or the movable reflector 320 in such a manner that the opening surface of the movable reflector 310 and/or movable reflector 320 is invisible from the direction of sight in the form illustrated in
[B] If a microphone array is provided in at least one of the movable reflectors 310 and 320 and, in addition, the opening surface of the movable reflector 310 is a flat surface opposite the surface opposable to the opening surface of the supporting board 400 or the opening surface of the movable reflector 320 is a flat surface opposite the surface that can form the same plane as the opening surface of the supporting board 400, the same effect as increasing the array size can be provided in the form illustrated in
Providing a microphone array in both surfaces of at least one of the movable reflectors 310 and 320 so that both surfaces of the movable reflector 310 and/or 320 are opening surfaces, can provide the same effects as both of [A] and [B].
<References>
A sound enhancement apparatus relating to the embodiments described above includes an input section to which a keyboard and the like can be connected, an output section to which a liquid-crystal display and the like can be connected, a CPU (Central Processing Unit) (which may include a memory such as a cache memory), memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), an external storage, which is a hard disk, and a bus that interconnects the input section, the output section, the CPU, the RAM, the ROM and the external storage in such a manner that they can exchange data. A device (drive) capable of reading and writing data on a recording medium such as a CD-ROM may be provided in the sound enhancement apparatus as needed. A physical entity that includes these hardware resources may be a general-purpose computer.
Programs for enhancing sounds in a narrow range and data required for processing by the programs are stored in the external storage of the sound enhancement apparatus (the storage is not limited to an external storage; for example the programs may be stored in a read-only storage device such as a ROM.). Data obtained through the processing of the programs is stored on the RAM or the external storage device as appropriate. A storage device that stores data and addresses of its storage locations is hereinafter simply referred to as the “storage”.
The storage of the sound enhancement apparatus stores a program for obtaining a filter for each frequency by using a spatial correlation matrix, a program for converting an analog signal to a digital signal, a program for generating frames, a program for transforming a digital signal in each frame to a frequency-domain signal in the frequency domain, a program for applying a filter corresponding to a direction or position that is a target of sound enhancement to a frequency-domain signal at each frequency to obtain an output signal, and a program for transforming the output single to a time-domain signal.
In the sound enhancement apparatus, the programs stored in the storage and data required for the processing of the programs are loaded into the RAM as required and are interpreted and executed or processed by the CPU. As a result, the CPU implements given functions (the frame design section, the AD converter, the frame generator, the frequency-domain transform section, the filter applying section, and the time-domain transform section) to implement sound enhancement.
<Addendum>
The present invention is not limited to the embodiments described above and modifications can be made without departing from the spirit of the present invention. Furthermore, the processes described in the embodiments may be performed not only in time sequence as is written or may be performed in parallel with one another or individually, depending on the throughput of the apparatuses that perform the processes or requirements.
If processing functions of any of the hardware entities (sound enhancement apparatus) described in the embodiments are implemented by a computer, the processing of the functions that the hardware entities should include is described in a programs. The program is executed on the computer to implement the processing functions of the hardware entity on the computer.
The programs describing the processing can be recorded on a computer-readable recording medium. The computer-readable recording medium may be any recording medium such as a magnetic recording device, an optical disc, a magneto-optical recording medium, and a semiconductor memory. Specifically, for example, a hard disk device, a flexible disk, or a magnetic tape may be used as a magnetic recording device, a DVD (Digital Versatile Disc), a DVD-RAM (Random Access Memory), a CD-ROM (Compact Disc Read Only Memory), or a CD-R (Recordable)/RW (ReWritable) may be used as an optical disk, MO (Magnet-Optical disc) may be used as a magneto-optical recording medium, and an EEP-ROM (Electronically Erasable and Programmable Read Only Memory) may be used as a semiconductor memory.
The program is distributed by selling, transferring, or lending a portable recording medium on which the program is recorded, such as a DVD or a CD-ROM. The program may be stored on a storage device of a server computer and transferred from the server computer to other computers over a network, thereby distributing the program.
A computer that executes the program first stores the program recorded on a portable recording medium or transferred from a server computer into a storage device of the computer. When the computer executes the processes, the computer reads the program stored on the recording medium of the computer and executes the processes according to the read program. In another mode of execution of the program, the computer may read the program directly from a portable recording medium and execute the processes according to the program or may execute the processes according to the program each time the program is transferred from the server computer to the computer. Alternatively, the processes may be executed using a so-called ASP (Application Service Provider) service in which the program is not transferred from a server computer to the computer but process functions are implemented by instructions to execute the program and acquisition of the results of the execution. Note that the program in this mode encompasses information that is provided for processing by an electronic computer and is equivalent to the program (such as data that is not direct commands to a computer but has the nature that defines processing of the computer).
While the hardware entities are configured by causing a computer to execute a predetermined program in the embodiments described above, at least some of the processes may be implemented by hardware.
Number | Date | Country | Kind |
---|---|---|---|
2010-285175 | Dec 2010 | JP | national |
2010-285181 | Dec 2010 | JP | national |
2011-025784 | Feb 2011 | JP | national |
2011-190768 | Sep 2011 | JP | national |
2011-190807 | Sep 2011 | JP | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/JP2011/079978 | 12/19/2011 | WO | 00 | 6/20/2013 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO2012/086834 | 6/28/2012 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
6473733 | McArthur et al. | Oct 2002 | B1 |
6738481 | Krasny et al. | May 2004 | B2 |
6868365 | Balan et al. | Mar 2005 | B2 |
6947570 | Maisano | Sep 2005 | B2 |
8363846 | Li et al. | Jan 2013 | B1 |
20030028372 | McArthur et al. | Feb 2003 | A1 |
20030063759 | Brennan et al. | Apr 2003 | A1 |
20030108214 | Brennan et al. | Jun 2003 | A1 |
20050265563 | Maisano | Dec 2005 | A1 |
20070165879 | Deng et al. | Jul 2007 | A1 |
20080112574 | Brennan et al. | May 2008 | A1 |
20090055170 | Nagahama | Feb 2009 | A1 |
20100119079 | Kim et al. | May 2010 | A1 |
Entry |
---|
Frost, O. L., “An Algorithm for Linearly Constrained Adaptive Array Processing”, IEEE, vol. 60, No. 8, pp. 926-935, (Aug. 1972). |
Flanagan, J. L., et al., “Spatially selective sound capture for speech and audio processing”, Speech Communication, vol. 13, pp. 207-222, (Oct. 1993). |
Nomura, H., et al., “Microphone array for near sound field”, The Journal of the Acoustical Society of Japan, vol. 53, No. 2, pp. 110-116, (1997). |
Hioka, Y., et al., “Enhancement of Sound Sources Located within a Particular Area Using a Pair of Small Microphone Arrays”, IEICE Transcations on Fundamentals, vol. E91-A, No. 2, pp. 561-574, (Feb. 2008). |
Hioka, Y., et al., “A method of separating sound sources located at different distances based on direct-to-reverberation ratio”, Proceedings of Autumn Meeting of the Acoustical Society of Japan, pp. 633-634, (Sep. 2009). |
Haykin, S., “Adaptive Filter Theory”, Kagaku Gijutsu Shuppann, pp. 66-73 and 248-255, (2001). |
Kikuma, N., “Adaptive Antenna Technology”, OHMSHA, pp. 35-90, (2003). |
Asano, F., “Array signal processing-sound source localization/tracking and separation”, Corona Publishing, pp. 88-89 and 259-261, (2011). |
Kaneda, Y., “Directivity characteristics of adaptive microphone-array for noise reduction (AMNOR)”, The Journal of the Acoustical Society of Japan, vol. 44, No. 1, pp. 23-30, (1988). |
International Search Report Issued Feb. 7, 2012 in PCT/JP11/079978 Filed Dec. 19, 2011. |
European Patent Office Action issued Jun. 30, 2015, in Patent Application No. 11 852 100.4. |
Blind Source Separation in Reflective Sound Fields, Asano et al.,Electrotechnical Laboratory, Prest, JST, Tsukuba University, Japan, International Workshop on Hands-Free Speech Communication (HSC2001), Kyoto, Japan, Apr. 9-11, 2001. |
Number | Date | Country | |
---|---|---|---|
20130287225 A1 | Oct 2013 | US |