The present disclosure relates to a signal processing device, a signal processing method, and a program.
Sound heard in a general environment such as a room is reverberated by a sound wave generated from a sound source being repeatedly reflected on a floor, a ceiling, or a wall. Since the frequency characteristics and the duration of the sound change due to this reverberation, a listener can perceive a sense of distance to the sound and a sense of reverberation by these changes. A technique for adding such reverberation is described in Patent Document 1 below.
Patent Document 1: WO 2013/057948 A
However, in the technique described in Patent Document 1, only a delay simulating reverberation is operated as a control target, and thus, it is insufficient to control a sense of reverberation caused by sound reflection or sound duration.
An object of the present disclosure is to provide a signal processing device, a signal processing method, and a program capable of adding appropriate reverberation to a signal of a virtual sound source, for example.
The present disclosure provides, for example,
The present disclosure provides, for example,
The present disclosure provides, for example,
Hereinafter, embodiments and the like of the present disclosure will be described with reference to the drawings. Note that the description will be made in the following order.
The embodiments and the like to be described below are preferred specific examples of the present disclosure, and the contents of the present disclosure are not limited to these embodiments and the like.
First, in order to facilitate understanding of the present disclosure, the background of the present disclosure and problems to be considered in the present disclosure will be described.
In recent years, high-resolution audio recorded at a high bit rate and high sampling rate and virtual surround audio reproduced by headphones have become widespread. These provide new values to conventional digital audio. Meanwhile, research and development of surround audio reproduced by a multichannel speaker are also in progress, and efforts have started to implement a real listening experience by increasing resolution not only in a time direction but also in a spatial direction. There has been proposed a technique of reproducing a wavefront propagating in space with spatially high resolution by using a much larger number of speakers than conventional 2 to 5.1 channel (ch) surround systems to faithfully reproduce the original wavefront. Such a technique is generally called “wavefront synthesis”, and various research and development are being conducted in academic and industrial fields.
Meanwhile, sound heard in a general environment is reverberated by a sound wave generated from a sound source being repeatedly reflected on a floor, a ceiling, or a wall. Since the frequency characteristics and the duration of the sound change due to this reverberation, a listener can perceive a sense of distance to the sound and a sense of reverberation by these changes. However, in the wavefront synthesis, influences of reflection or the like are not physically reproduced. Therefore, even if a virtual sound source is arranged to accurately reproduce a wavefront, it may be difficult for a listener to perceive the sense of distance to the sound source and the sense of reverberation. Therefore, even in the wavefront synthesis, it is considered that appropriately adding reverberation makes it possible to improve a realistic feeling caused by a sound field to be reproduced. In view of the above point, the embodiments of the present disclosure will be described in detail.
The present disclosure is a technique in which reverberation is added in a system to which wavefront synthesis is applied (hereinafter, such a system will be appropriately abbreviated as a “wavefront synthesis system”).
Here, a specific example of adding reverberation to a virtual sound source itself in the wavefront synthesis system will be described. First, the property of reverberation will be described in terms of the distance between a sound source and a sound reception point.
When the two direct sounds are compared, the amplitude value of the direct sound illustrated in
The level and length of reverberation are adjusted according to the distance r, so that a reverberation sound signal is generated. The reverberation sound signal is convolved with a signal of the virtual sound source VS and then processed by a wavefront synthesis filter to generate a drive signal, and the drive signal is reproduced from the speaker array SA, so that the listener L can listen to the sound to which the reverberation is added. As the reverberation to be convolved with the virtual sound source VS, reverberation measured in advance in a concert hall or the like, or reverberation created by use of a simulation algorithm such as a method of mirror images can be used.
The reverberation added according to the distance r is determined according to, for example, a reverberation control function.
Note that the reverberation control function described above can be adjusted by a user. For example, the curvature of the reverberation control function may be adjusted by the user. A configuration may be adopted in which the user can freely change the curvature of the reverberation control function by displaying the reverberation control function on editing software for wavefront synthesis and performing a drag and drop operation or the like with a mouse on the reverberation control function.
A signal of the virtual sound source VS (hereinafter referred to as a virtual sound source signal D0) is input to the signal processing device 1. The virtual sound source signal D0 is, for example, a monaural signal of one channel, and is an object audio signal corresponding to a predetermined object. The signal of the virtual sound source signal VS may be read from an appropriate memory or may be distributed from a network such as the Internet. Note that, in the wavefront synthesis system, information indicating the position of the virtual sound source VS is also input to the signal processing device 1, but illustration of this point is omitted.
Furthermore, distance information I(r) indicating the distance r measured by a distance measuring device or the like and a reverberation control function FU illustrated in
When the signal processing device 1 performs processing, the drive signal for driving the speaker array SA is generated. The drive signal includes signals of channels (ch) corresponding to the number of the speaker units SU included in the speaker array SA. In the example illustrated in
The signal processing device 1 includes, for example, a reverberation reading unit 11, a reverberation correction unit 12, a reverberation convolution unit 13, a wavefront synthesis filter 14, and a digital-to-analog (DA) conversion/amplification unit 15. The reverberation reading unit 11 reads a signal corresponding to reverberation from an appropriate memory. The signal corresponding to the reverberation may be distributed via a network such as the Internet, and in this case, the reverberation reading unit 11 functions as a communication unit connectable to the network. The reverberation reading unit 11 outputs a signal RDA corresponding to the read reverberation to the reverberation correction unit 12.
The reverberation correction unit 12, which is an example of a reverberation sound signal generation unit, specifies, for example, the level of the reverberation corresponding to the distance information I(r) by using the reverberation control function FU. The reverberation correction unit 12 then generates a reverberation sound signal RDB by adjusting the level of the signal RDA corresponding to the reverberation on the basis of the specified level. The reverberation correction unit 12 outputs the generated reverberation sound signal RDB to the reverberation convolution unit 13. That is, in the present embodiment, the reverberation correction unit 12 corresponds to the reverberation sound signal generation unit. Note that the reverberation time may be adjusted instead of the level of the reverberation, or the reverberation time may be adjusted together with the level of the reverberation.
The reverberation convolution unit 13 performs processing of convolving the reverberation sound signal RDB with the virtual sound source signal D0 to generate a convolution signal D1. Since the convolution processing is generally performed by converting a signal into a frequency domain signal, the reverberation convolution unit 13 may include fast Fourier transform (FFT) or the like. The reverberation convolution unit 13 outputs the convolution signal D1 to the wavefront synthesis filter 14.
The wavefront synthesis filter 14, which is an example of a drive signal generation unit, generates a drive signal D2 by performing wavefront synthesis filtering on the convolution signal D1. For example, the wavefront synthesis filter 14 converts the convolution signal D1 into a multichannel signal (eight channels in this example), and appropriately adjusts the phase, gain, delay, and the like of each signal included in the multichannel signal to reproduce such a wavefront that the virtual sound source signal D0 is localized at a predetermined position. As a result, the drive signal D2 including the signals of eight channels is output from the wavefront synthesis filter 14. As an algorithm of the processing performed by the wavefront synthesis filter 14, any algorithm is used according to the processing capability of a calculator, the arrangement of the speakers, or the like. Specifically, a higher order ambisonics (HOA) method, a weighted mode matching method, a spectral division method, or the like can be applied.
The DA conversion/amplification unit 15 converts the drive signal D2 in a digital format into a drive signal D3 in an analog format and amplifies the drive signal D3. As a result, the drive signal D3 in the analog format is generated. The signal of each channel included in the drive signal D3 is supplied to a corresponding one of the speaker units SU and reproduced. Note that a component including the wavefront synthesis filter 14 and the DA conversion/amplification unit 15 may be a component corresponding to the drive signal generation unit.
Next, an operation example of the signal processing device 1 will be described with reference to the flowchart illustrated in
In step ST12, the reverberation reading unit 11 reads the signal RDA corresponding to the reverberation. Note that the signal RDA corresponding to the reverberation may be generated. The processing then proceeds to step ST13.
In step ST13, the reverberation correction unit 12 reads the distance information I(r) and the reverberation control function FU. The processing then proceeds to step ST14.
In step ST14, the reverberation correction unit 12 performs reverberation correction processing. By such processing, the reverberation sound signal RDB is generated. Specifically, the signal RDA corresponding to the reverberation is corrected on the basis of the distance information I(r) and the reverberation control function FU, so that the reverberation sound signal RDB is generated. The generated reverberation sound signal RDB is output to the reverberation convolution unit 13. The processing then proceeds to step ST15.
In step ST15, the reverberation convolution unit 13 performs processing of convolving the virtual sound source signal D0 and the reverberation sound signal RDB. By such processing, the convolution signal D1 is generated. The processing then proceeds to step ST16.
In step ST16, the wavefront synthesis filter 14 performs filtering processing. By such processing, the drive signal D2 in the digital format is generated. The processing then proceeds to step ST17.
In step ST17, the DA conversion/amplification unit 15 performs processing. By such processing, the drive signal D3 in the analog format is generated, and the drive signal D3 is amplified. The signal of each channel of the drive signal D3 is supplied to a corresponding one of the speaker units SU, and the sound is reproduced from the speaker units SU.
Note that the order of the flow of the above-described processing may be appropriately changed, or a plurality of pieces of processing may be performed in parallel.
According to the present embodiment described above, reverberation that does not depend on the position of the listener L can be added in the wavefront synthesis system. Furthermore, even in a case where the position of the listener L changes, appropriate reverberation can be added.
Note that, in a case where there is a plurality of virtual sound sources VS, that is, in a case where there is a plurality of virtual sound source signals D0, the signal corresponding to the reverberation may be the same signal for all the virtual sound source signals D0 or a different signal for each virtual sound source signal D0. Furthermore, in a case where there is a plurality of signals corresponding to reverberation, the reverberation control function FU may be the same function for the signals or a different function for each signal.
In the above description, an example is assumed in which one listener is present for one or a plurality of virtual sound sources, but there may be a plurality of listeners.
For example, adding reverberation to the virtual sound source VSA and the virtual sound source VSB with reference to the listener LA will be considered. At this time, the relationship of the distance from the listener LA to each virtual sound source is rA1<rB1, and thus, in a case where processing similar to that in the embodiment is performed, the reverberation added to the virtual sound source VSB is set to be larger and longer than the reverberation added to the virtual sound source VSA. As described above, this setting represents, by wavefront synthesis, a phenomenon that reverberation of a far sound source is relatively large. However, in a case where the listener LB listens to the sound to which reverberation is added as described above, the reverberation of the virtual sound source VSB is heard more loudly although the relationship of the distance from the listener LB to each virtual sound source is rA2>rB2.
One of advantages of the wavefront synthesis system is that a plurality of people can simultaneously listen to the same wavefront, and it is quite likely that there is a plurality of listeners. In a case where there is a plurality of listeners, using the shortest distance between a sound source and a listener causes reverberation to be optimized only for the listener. Therefore, for example, in a case where the presence of a plurality of listeners is detected by the distance measuring device or the like, or in a case where a predetermined mode is set, the distance that minimizes the distance between the virtual sound source and the speaker array SA is set as the distance r.
Specifically, as illustrated in
As a result, it is possible to avoid a situation in which reverberation differ for each listener as illustrated in
Note that, although the speaker array SA in which the speaker units SU are arranged on a straight line has been described above as an example, the speaker units SU may be arranged in a rectangular shape (four directions), for example, as illustrated in
Next, a second embodiment will be described. Note that, in the description of the second embodiment, the same or similar components in the above description are denoted by the same reference signs, and redundant description will be appropriately omitted. Furthermore, the matters described in the first embodiment can be applied to the second embodiment unless otherwise noted.
In the first embodiment, it is possible to add reverberation without significantly changing the framework of a general wavefront synthesis system. The second embodiment is an embodiment in which a reflection component of reverberation is newly generated as a virtual sound source, and more effective reverberation can be added.
Reverberation includes a reflected sound following a direct sound. This reflected sound is observed when a sound wave reflected on a ceiling, a floor, or a wall reaches a sound reception point. That is, it can be considered that the reflected sound does not arrive from the position of a sound source, but arrives from the reflected point. Therefore, on the assumption that the reflected point is a sound source that generates the reflected sound, it is possible to perform wavefront synthesis while the reflected sound is set as a virtual sound source.
The signal processing device 1A is supplied with a virtual sound source signal D0. The virtual sound source signal D0 is branched and also supplied to the virtual reverberation signal generation unit 22. Furthermore, the virtual reverberation signal generation unit 22 is supplied with virtual wall position information I(VW) indicating the position of a virtual wall with respect to a predetermined virtual sound source VS, which serves as an example of the virtual space information.
The virtual sound source signal wavefront synthesis filter 21 performs filtering processing on the virtual sound source signal D0 by a wavefront synthesis filter. The wavefront synthesis filtering is performed to generate an output signal D5. For example, the virtual sound source signal wavefront synthesis filter 21 converts the virtual sound source signal D0 into a multichannel signal (eight channels in this example), and appropriately adjusts the phase, gain, delay, and the like of each signal included in the multichannel signal to reproduce such a wavefront that the virtual sound source signal D0 is localized at a predetermined position. As a result, the output signal D5 including the signals of eight channels is output from the virtual sound source signal wavefront synthesis filter 21. As an algorithm of the processing performed by the virtual sound source signal wavefront synthesis filter 21, any algorithm is used according to the processing capability of a calculator, the arrangement of speakers, and the like. Specifically, a HOA method, a weighted mode matching method, a spectral division method, or the like can be applied.
The virtual reverberation signal generation unit 22 acquires the virtual sound source signal D0, the sound source position of the virtual sound source signal D0, and the virtual wall position information I (VW), and generates a virtual reverberation signal D6 (D61 to D6N). For example, the virtual reverberation signal generation unit 22 performs processing of specifying the reflection position of the virtual reverberation signal D6, in other words, processing of specifying the position where the virtual reverberation signal D6 is localized. The reflection position of the virtual reverberation signal D6 can be calculated, for example, by simulation of sound propagation using a method of mirror images or the like. For example, the virtual reverberation signal generation unit 22 refers to the virtual wall position information I(VW) and sets the reflection position of the virtual reverberation signal D61 of the primary reflection at the position of a wall closest to the reproduction position of the virtual sound source signal D0. As described above, in the present embodiment, the virtual reverberation signal D6 (D61 to D6N), which is a reverberation sound signal in the present embodiment, is generated according to the sound source position of the virtual sound source VS and the distance to the surface reflecting the virtual sound source signal D0.
The virtual reverberation signal wavefront synthesis filter 23 performs wavefront synthesis filtering processing on the virtual reverberation signal D6 to generate a virtual reverberation signal D6′(D6′1 to D6′N) obtained by making the virtual reverberation signal D6 into a virtual sound source. For example, the virtual reverberation signal wavefront synthesis filter 231 converts the virtual reverberation signal D61 into a multichannel signal (eight channels in this example), and adjusts the phase, gain, delay, and the like of each signal included in the multichannel signal such that the virtual reverberation signal D61 is localized at a predetermined position (along the wall). As a result, the virtual reverberation signal D6′1 including the signals of eight channels is output from the virtual reverberation signal wavefront synthesis filter 231. As an algorithm of the processing performed by the virtual reverberation signal wavefront synthesis filter 231, any algorithm is used according to the processing capability of the calculator, the arrangement of the speakers, and the like. Specifically, a HOA method, a weighted mode matching method, a spectral division method, or the like can be applied.
The adder 24 adds signals of corresponding channels included in the virtual reverberation signal D6′ obtained by making the virtual reverberation signal D6 into the virtual sound source, specifically, the virtual reverberation signals D6′1 to D6′N. As a result of the addition processing by the adder 24, an output signal D7 including signals of eight channels is output from the adder 24.
A drive signal for each of speaker units SU is generated on the basis of the virtual reverberation signal D6′ obtained by making the virtual reverberation signal D6 into the virtual sound source. Specifically, the adder 25 adds the output signal D5 obtained by performing wavefront synthesis filtering on the virtual sound source signal D0 and the output signal D7 output from the adder 24 for each corresponding channel. A drive signal D8 including signals of eight channels in a digital format is generated by the addition processing by the adder 25. The generated drive signal D8 is output from the adder 25.
The DA conversion/amplification unit 15 converts the drive signal D8 in the digital format into a signal in an analog format and then amplifies the converted signal to generate and output a drive signal D9. The signal of each channel of the drive signal D9 in the analog format is supplied to a corresponding one of the speaker units SU, and the sound is reproduced from the speaker units SU.
As described above, in the present embodiment, for example, a component including the virtual reverberation signal generation unit 22 and the virtual reverberation signal wavefront synthesis filter 23 corresponds to a reverberation sound signal generation unit. Furthermore, for example, a component including the virtual sound source signal wavefront synthesis filter 21, the adder 24, and the adder 25 (which may include the DA conversion/amplification unit 15) corresponds to a drive signal generation unit.
According to the second embodiment described above, making reverberation into a virtual sound source makes it possible to arrange the reverberation at any position. Therefore, reverberation can be appropriately reproduced.
Note that, in the second embodiment, the number of reflections (primary, secondary, and tertiary reflections) can be set to any number. Furthermore, since the attenuation rate of reflection varies depending on the material and shape of a wall, virtual reverberation may be generated in consideration of these. Generating and reproducing the virtual reverberation in consideration of the material and shape of the wall makes it possible to give a higher realistic feeling to a listener.
Next, modifications of the second embodiment will be described. In the above-described wavefront synthesis system that makes reverberation into a virtual sound source, a wavefront synthesis filter requires a larger number of calculations as the order of reflection increases, which increases the amount of computation. Such a problem is disadvantageous, for example, in an application that processes the wavefront synthesis system online.
A normal wavefront synthesis filter is designed by the number and arrangement of installed speakers and the order of a wavefront to be reproduced. Thinning out the number of speakers and lowering the reproduction order make it possible to reduce the amount of computation for wavefront synthesis filtering processing. When the order is lowered, the resolution of the wavefront to be reproduced is lowered, and thus, a trade-off occurs between the sound quality and the amount of computation. However, in general, the gain of a reflection component is smaller than that of a direct sound, and a listener does not strictly listen to only the reflection. Therefore, it is considered that performing wavefront synthesis on a reverberation component with a low order contributes to improvement of a realistic feeling without problems. In view of the above point, the modifications of the second embodiment will be described.
First, a first modification will be described.
Operations of the virtual reverberation signal low-order wavefront synthesis filter 27 and the signal duplication unit 28 will be described with reference to
The virtual reverberation signal low-order wavefront synthesis filter 271 performs wavefront synthesis filtering on the virtual reverberation signal D61, but outputs a signal in which channels are thinned out. For example, as illustrated in
Note that interpolation processing may be performed in which signals for speakers on both sides of a speaker having no signal are averaged to generate a signal for the speaker having no signal. Furthermore, only the output of the virtual reverberation signal low-order wavefront synthesis filter 27 may be reproduced while signal duplication is not performed and the thinned-out speakers are left as they are. As described above, the amount of computation can be reduced.
A configuration may be adopted in which wavefront synthesis is not performed on a reflection component and the reflection component is not made into a virtual sound source, so that the amount of computation may be reduced. As illustrated in
Furthermore, a reverberation speaker may be installed separately from the speaker array SA for wavefront synthesis, and reverberation may be reproduced from the speaker. For example, as illustrated in
Next, a third embodiment will be described. Note that, in the description of the first and second embodiments, the same or similar components in the above description are denoted by the same reference signs, and redundant description will be appropriately omitted. Furthermore, the matters described in the first and second embodiments can be applied to the third embodiment unless otherwise noted.
In the first and second embodiments, the reflection of reverberation has been focused. However, other characteristics of reverberation include a feeling of being surrounded by sound and duration of sound. A feeling that the entire space in which a listener is located is sounding and the sound that continues for a long time, which are results of innumerable reflections occurring everywhere, are expressed as a component of late reverberation in
The wavefront synthesis filter 14 outputs a signal D15 by performing wavefront synthesis filtering on a virtual sound source signal D0.
A signal RDA corresponding to reverberation output from the reverberation reading unit 11 is supplied to the late reverberation signal generation unit 31. The late reverberation signal generation unit 31 generates a late reverberation signal D16 on the basis of the signal RDA corresponding to the reverberation. For example, as schematically illustrated in
The reverberation convolution unit 32 performs processing of convolving the late reverberation signal D16 with the virtual sound source signal D0 to generate a convolution signal D17. The reverberation convolution unit 32 then outputs the generated convolution signal D17 to the multichannel processing unit 33.
The multichannel processing unit 33 performs processing of converting the convolution signal D17 into a multichannel signal so that the converted signal corresponds to the number of channels (eight channels in this example) of the speaker array SA. The multichannel processing unit 33 performs the processing to generate a multichannel late reverberation sound (hereinafter, a multichannel late reverberation signal D18). The multichannel late reverberation signal D18 is supplied to the adder 34.
The adder 34 adds the signal D15 output from the wavefront synthesis filter 14 and the multichannel late reverberation signal D18 for each corresponding channel. The adder 34 performs the processing to generate a drive signal D19 in a digital format.
The DA conversion/amplification unit 15 converts the drive signal D19 in the digital format into a signal in an analog format and then amplifies the converted signal to generate and output a drive signal D20. The signal of each channel of the drive signal D20 in the analog format is supplied to a corresponding one of speaker units SU, and the sound is reproduced from the speaker units SU.
According to the present embodiment described above, it is possible to reproduce late reverberation in a wide range, and it is possible to give a listener a feeling of being surrounded by sound. Furthermore, it is possible to reproduce the late reverberation component as if the entire space sounded without performing calculation by the wavefront synthesis filter.
Note that, in an actual space, the waveform of the impulse response differs depending on the observation point. In consideration of this point, a different piece of late reverberation may be used for each position of the speaker units SU, and signals obtained by convolving the pieces of late reverberation with the virtual sound source signal D0 may be reproduced. Furthermore, the late reverberation signal generation unit 31 may generate a pseudo late reverberation signal, and the generated pseudo late reverberation signal may be convolved with the virtual sound source signal D0.
Although the plurality of embodiments of the present disclosure has been specifically described above, the contents of the present disclosure are not limited to the above-described embodiments, and various modifications based on the technical idea of the present disclosure are possible.
The configurations, methods, steps, shapes, materials, numerical values, and the like described in the above-described embodiments and modifications are merely examples, and configurations, methods, steps, shapes, materials, numerical values, and the like different from those described above may be used as necessary, or those described above may be replaced with known ones. In addition, the configurations, methods, steps, shapes, materials, numerical values, and the like in the embodiments and the modifications can be combined with each other within a range in which no technical contradiction occurs.
Note that the contents of the present disclosure are not to be construed as being limited by the effects exemplified in the present specification.
The present disclosure can also adopt the following configurations.
(1)
A signal processing device including:
The signal processing device according to (1), in which
The signal processing device according to (1) or (2), in which
The signal processing device according to (1) or (2), in which
The signal processing device according to (1) or (2), in which
The signal processing device according to any one of (1) to (5), in which
The signal processing device according to (6), in which
The signal processing device according to (7), in which
The signal processing device according to any one of (1) to (8), in which
A signal processing method including:
A program for causing a computer to execute a signal processing method including:
1, 1A, 1B, 1C Signal processing device
12 Reverberation correction unit
14 Wavefront synthesis filter
15 DA conversion/amplification unit
21 Virtual sound source signal wavefront synthesis filter
22 Virtual reverberation signal generation unit
23 Virtual reverberation signal wavefront synthesis filter
24, 24, 34 Adder
33 Multichannel processing unit
SU Speaker unit
SA Speaker array
Number | Date | Country | Kind |
---|---|---|---|
2019-224997 | Dec 2019 | JP | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/JP2020/044855 | 12/2/2020 | WO |