Patent Grant
11094305
This application claims the priority of Japan patent application serial no. 2018-247689, filed on Dec. 28, 2018. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification.
The disclosure relates to a technology for detecting a performance tempo of a musical instrument.
Schemes of generating one music video by imaging singing or the performance of artists and musicians at a plurality of angles and linking obtained videos are known. In the schemes, it is necessary to select appropriate cameras in accordance with the narrative of video content to be generated while pieces of music are in progress.
As a technology related to this, for example, Patent Document 1 (Japanese Patent Laid-Open No. 2005-026739) discloses a system capable of controlling switching between a plurality of cameras disposed on a stage based on a scenario stored in advance. Patent Document 2 (Japanese Patent Laid-Open No. 2005-295431) discloses a technology for recognizing the position of a person who is speaking based on speech acquired by a plurality of microphones and switching between a plurality of cameras to ascertain the speaking person.
According to the system disclosed in Patent Document 1, it is possible to perform automated switching between the cameras in accordance with a preset intention. In the disclosure, it is necessary to associate a switching timing of the cameras with any position in a piece of music. However, when a live performance of a piece of music is played, the association may not be performed in advance. There is a method of switching between cameras autonomously, but there is concern of discomfort being experienced by an audience when cameras are switched at timings irrelevant to a piece of music (for example, beats or bars).
According to an embodiment of the disclosure, an information processing device includes: an acquisition part that acquires samples of musical sound signals in a time series; an evaluation part that has an adaptive filter using the acquired samples of the musical sound signals as reference signals and using samples of musical sound signals acquired a predetermined time earlier than the samples of the musical sound signals as input signals; and a tempo determination part that sequentially inputs the samples of the musical sound signals to the adaptive filter and determines a tempo corresponding to a musical sound based on a filter coefficient of the adaptive filter when a value of the filter coefficient of the adaptive filter converges.
According to an embodiment of the disclosure, the tempo determination part may determine whether the predetermined time is a value corresponding to the tempo of the musical sound based on the converged filter coefficient.
According to an embodiment of the disclosure, the filter coefficient may include a plurality of coefficients. The tempo determination part may input a sample group of the plurality of musical sound signals acquired within a predetermined period as the input signal to the adaptive filter.
According to an embodiment of the disclosure, the tempo determination part may determine a value corresponding to a time difference between a sample of an input signal multiplied by a coefficient indicating a maximum value among the plurality of converged coefficients and a sample of the musical sound signal used as the reference signal as the tempo corresponding to the musical sound.
According to an embodiment of the disclosure, the filter coefficient may include a plurality of coefficients. The tempo determination part may input a sample group of the plurality of musical sound signals acquired within a first period and a sample group of the plurality of musical sound signals acquired within a second period that has a length of a multiple of n (where n is an integer equal to or greater than 2) times the first period and continues from the first period as the input signals to the adaptive filter.
The disclosure provides a video processing system, including: the foregoing information processing device; and a control device that switches between a plurality of video sources respectively corresponding to a plurality of cameras at a timing in accordance with a tempo determined by the information processing device.
According to an embodiment of the disclosure, a tempo detection device is provided. The tempo detection device includes: a musical sound signal acquisition part that acquires musical sound signals; and a tempo detection part. The tempo detection part includes: a sampling part that uses signals obtained after the musical sound signals are sampled at a predetermined frequency, as samples of the musical sound signals; a signal delaying part that delays the samples of the musical sound signals by a predetermined number of time steps; and an adaptive filter unit using a sample of a latest time step as a reference signal, using a sample generated earlier by the predetermined number of time steps as an input signal, and updating a filter coefficient of the adaptive filter unit so that an error between the input signal and the reference signal is a minimum. The tempo detection part sequentially inputs the samples of the musical sound signals and determines a tempo corresponding to a musical sound based on the filter coefficient when a values of the filter coefficient of the adaptive filter unit converges.
The disclosure provides a technology for detecting a beat of a performed musical piece from a musical viewpoint.
The adaptive filter is a digital filter that dynamically updates the filter coefficient so that an error between the input signal (an evaluation target signal) and the reference signal (real signal) becomes minimum. Since a piece of music is configured to have a beat, constant periodicity is observed in the musical sound signal. Accordingly, when samples of musical sound signals with a certain interval are input as the reference signal and the input signal to the adaptive filter, the filter coefficient converges to a value in accordance with the periodicity. Accordingly, the tempo corresponding to the musical sound can be evaluated based on the converged filter coefficient.
When the filter coefficient included in the adaptive filter is a single coefficient, the converged filter coefficient is a value indicating “to what degree the set predetermined time matches a real tempo.”
When the filter coefficient included in the adaptive filter includes the plurality of coefficients, the samples of the plurality of musical sound signals acquired within the predetermined period can be set an input of the adaptive filter. In this case, a timing corresponding to the real tempo can be ascertained in accordance with the value of each converged coefficient.
When there is a coefficient with the largest value among the plurality of coefficients, a sample in which the coefficient is an evaluation target is meant to be the most similar to a sample which is the reference signal. Accordingly, a time difference between the samples can be determined to be a tempo corresponding to the musical sound.
In this way, the sample group evaluated by the adaptive filter may not be included in a single period. By setting the sample group included in the first and second periods as an evaluation target, it is possible to evaluate a period which is n times the first period. That is, it is possible to perform evaluation from a musical viewpoint.
By switching between the video sources (for example, videos obtained by imaging a performer using a plurality of cameras) at timings in accordance with the detected tempos of the piece of music, it is possible to obtain a video with little discomfort.
The disclosure can be specified as an information processing device and a video processing system including at least some of the foregoing parts. The disclosure can also be specified as a method performed by the foregoing information processing device and video processing system. The disclosure can also be specified as a program causing the method to be performed or a non-transitory storage medium on which the program is recorded. The processes or parts can be freely combined to be performed as long as there are no technical contradictions therebetween.
A video processing system according to the embodiment is a system in which the performance of a musical instrument by a performer is videoed by a plurality of cameras and an acquired video is reorganized and output. The video processing system according to the embodiment includes a tempo detection device 100, a video processing device 200, a plurality of cameras 300, and a microphone 400.
The tempo detection device 100 is a device that detects a tempo of a piece of music based on the input musical sound signal. In the embodiment, a tempo is the number of beats per minute and is expressed in beats per minute (BPM). For example, when the BPM is 120, the number of beats per minute is 120 beats. Information regarding the detected tempo is transmitted as tempo information to the video processing device 200.
The video processing device 200 is a device that acquires and records the video signals from the plurality of connected cameras 300, reorganizes the recorded videos in accordance with a predetermined rule, and outputs the reorganized videos. Specifically, a plurality of recorded video sources is sequentially selected in a time series and the selected video sources are combined to be output, as illustrated in
Next, the tempo detection device 100 will be described in detail.
The musical sound signal acquisition part 101 acquires a musical sound signal which is an analog signal from the microphone 400. In the description of the present specification, a musical sound signal has a concept including both an analog signal and a digital signal obtained by sampling the analog signal.
The tempo detection part 102 samples an analog signal at a predetermined rate and detects a tempo based on the obtained digital signal. Specific processing content will be described later. The tempo detection part 102 generates information indicating a tempo of the piece of music (tempo information) and transmits the information to the video processing device 200. In the embodiment, the tempo information is information including a value (for example, 120 BPM) of the detected tempo.
Next, the video processing device 200 will be described.
A video recording part 201 acquires and records video signals and a sound signal from the plurality of cameras 300 and the microphone 400. For example, when the number of cameras is 4, the video recording part 201 is connected to each of the cameras 300A, 300B, 300C, and 300D, and acquires and records a plurality of video signals (video streams). The recorded video signal is also referred to as a video source below. The video recording part 201 and the cameras 300 may be connected in a wired manner or a wireless manner.
A video source selection part 202 links (edits) the plurality of video signals recorded by the video recording part 201 using the tempo information acquired from the tempo detection part 102 to generate an output signal. The video sources may be selected in accordance with a preset predetermined rule. For example, the video source selection part 202 retains data in which association between the number of beats from performance start of a piece of music and the cameras 300 is described (hereinafter referred to as video source selection information), switches between the video sources, as illustrated in
An adaptive algorithm will be described before a principle in which the tempo detection part 102 detects a tempo is described. Since the adaptive algorithm is a known algorithm, detailed description will be omitted and only an outline of the adaptive algorithm will be described.
Here, n indicates a time step. A case of n=0 indicates a latest time step and a case of n=−32 indicates a time step 32 steps earlier.
The tempo detection device 100 according to the embodiment calculates similarity between a processing target sample and a previous sample using characteristics of the adaptive filter.
In the embodiment, a sampling part 1021 samples a musical sound signal at 44,100 Hz and subsequently performs a decimation process on the obtained signal at intervals of 512 samples. That is, a duration time of one sample is about 11.6 milliseconds. In this example, the duration time is about 371 milliseconds in 32 steps and is about 743 milliseconds in 64 steps. These times are equal to intervals of beats in the case of 160 BPM and 80 BPM, respectively.
The tempo detection part 102 detects a tempo using the adaptive filter. Specifically, the adaptive algorithm is executed using x(0) which is a latest sample as a reference signal and using x(−32) to x(−63) which are samples generated 32 steps earlier as input signals.
Here, when beats of a piece of music are in a section from the step 32 steps earlier to the step 63 steps earlier, it is supposed that there is a sample indicating a highest value of similarity with x(0) in one step. In other words, in the section from the step 32 steps earlier to the step 63 steps earlier, a step at which the most similar music pressure to x(0) is observed can be estimated to be a step corresponding to a beat of the piece of music.
In the example of
y(0)=h32(0)x(−32)+h33(0)x(−33)+ . . . +h47(0)x(−47)+ . . . +h63(0)x(−63) Expression (1)
e(0)=x(0)−y(0) Expression (2)
The calculated error is fed back to be used for updating the filter coefficients in a next time step. The following expression is an expression that determines filter coefficients in a next time step. Here, μ is a response sensitivity value obtained empirically.
h32(1)=h32(0)+μe(0)x(−32)
h33(1)=h33(0)+μe(0)x(−33)
. . .
h63(1)=h63(0)+μe(0)x(−63)
When the musical sound signals are sequentially input to the tempo detection part 102 for each time step, the filter coefficients h32(0) to h63(0) are frequently updated to converge to a certain state.
The filter coefficient h indicates similarity of a sound pressure for each time step.
In this example, steps from the step 32 steps earlier to the step 63 steps earlier are set as evaluation targets. That is, T1 in
A value (t1) determined by the tempo detection part 102 is transmitted to the video processing device 200 (the video source selection part 202) to generate an output signal.
First in step S11, the tempo information is acquired from the tempo detection part 102. The tempo information may include information regarding a time stamp or the like in addition to a value indicating the tempo of the piece of music. For example, the tempo information may include information indicating a performance start timing of the piece of music.
As described above, the video processing system according to the first embodiment can calculate a tempo of the piece of music based on periodicity of a waveform of the musical sound signal. Since the videos can be combined in synchronization with the positions of the beats, camera work in which discomfort is less can be realized.
In the first embodiment, the tempo detection device 100 has evaluates the periodicity of the musical sound signal included during the period T1. On the other hand, a second embodiment is an embodiment in which periodicities of musical sound signals included during a plurality of different periods (T1 and T2), the periodicities are integrated to determine a tempo of a piece of music.
In the tempo detection device 100 according to the second embodiment, only a configuration of the tempo detection part 102 is different from that of the first embodiment. Hereinafter, differences will be described.
A period indicated by T1 is a first period and a period indicated by T2 is a second period. In the embodiment, the length of T2 is twice the length of T1. In this way, a timing before one beat earlier and a timing two or more beats earlier can be detected.
In the example of
y1(0)=h32(0)x(−32)+h33(0)x(−33)+ . . . +h63(0)x(−63) Expression (3)
e1(0)=x(0)−y1(0) Expression (4)
When yz is an output signal from the adaptive filter unit 1023B, the output signal can be expressed as in Expression (5). An error between the output signal and the reference signal is expressed as in Expression (6).
y2(0)=h64(0)x(−64)+h66(0)x(−66)+ . . . +h126(0)x(−126) Expression (5)
e2(0)=x(0)−y2(0) Expression (6)
Here, the filter coefficients in Expression (5) are substituted with the filter coefficients in the adaptive filter unit 1023A. As a result, the output signal is expressed as in Expression (7).
y2(0)=h32(0)x(−64)+h33(0)x(−66)+ . . . +h64(0)x(−126) Expression (7)
In the second embodiment, an expression by which the adaptive filter unit 1023A updates the filter coefficients h32 to h63 is described as follows. Parentheses are independent terms in the embodiment.
h32(1)=h32(0)+μ1e1(0)x(−32)+[μ2e2(0)x(−64)]
h33(1)=h33(0)+μ1e1(0)x(−33)+[μ2e2(0)x(−66)]
. . .
h63(1)=h63(0)+μ1e1(0)x(−63)+[μ2e2(0)x(−126)]
That is, in the second embodiment, when the adaptive filter unit 1023A updates the filter coefficients, a correction result of the filter coefficients by the adaptive filter unit 1023B is added. In other words, a result of the determination of the similarity performed during the period T2 by the adaptive filter unit 1023B is added to a result of the determination of the similarity performed during the period T1 by the adaptive filter unit 1023A.
In the first embodiment, the value of the tempo has been calculated from the mathematical viewpoint, but the value of the mathematically calculated tempo does not necessarily match the value of the musical tempo (an intrinsic tempo of the piece of music) in some cases. For example, depending on a configuration of a piece of music, a section in which a tempo is heard at 120 BPM and a section in which a tempo is heard at 60 BPM coexist in some cases. For example, when a ringing way of percussion before and after a musical interlude is changed, an estimation result of a tempo may change despite an unchanged tempo of a piece of music in some cases. In the first embodiment, when a piece of music determined to be mathematically at 120 BPM enters a section determined to be at 60 BPM, the converging filter coefficients are changed again and correct tempo determination may not be performed in some cases. This is because the shape of a peak denoted by reference sign 801 in
In the second embodiment, however, periodicity of a musical sound signal during the period T1 and periodicity of a musical sound signal during the period T2 (of which a length is twice the length of T1) are added for evaluation. In this configuration, even when a sound with a half of a tempo is temporarily heard, the cumulatively evaluated filter coefficients are not considerably changed. That is, a tempo of a piece of music can be determined by adding not only the mathematical viewpoint but also the musical viewpoint.
In the second embodiment, two adaptive filter units have been used to evaluate the periodicities of the musical sound signals during the periods T1 and T2. However, a third embodiment is an embodiment in which four adaptive filter units are used to evaluate four periods.
In the tempo detection device 100 according to the third embodiment, only a configuration of the tempo detection part 102 is different from that of the second embodiment. Hereinafter, differences will be described.
The sampling part 1021A samples a musical sound signal at 44,100 Hz and subsequently performs a process of decimating the obtained signal for every 512 samples as in the sampling part 1021. The sampling part 1021B samples a musical sound signal at 44,100 Hz and subsequently performs a process of decimating the obtained signal for every 2048 samples.
The adaptive filter unit 1023C is a unit evaluating a step 32 steps earlier to a step 64 steps earlier in a low sound area (a range denoted by reference sign 1403: here, since a sampling rate of the low sound area is ¼ of that of a high sound area, one step of the low sound area is equivalent to four steps of the high sound area).
In the following description, a musical sound signal of the low sound area is denoted by xL(n) and is distinguished from a musical sound signal x(n) of the high sound area.
Here, when y3 is an output signal from the adaptive filter unit 1023C, the output signal can be expressed as in Expression (8). An error between the output signal and the reference signal is expressed as in Expression (9).
y3(0)=hL32(0)xL(−32)+hL33(0)xL(−33)+ . . . +hL63(0)xL(−63) Expression (8)
e3(0)=xL(0)−y3(0) Expression (9)
When y4 is an output signal from the adaptive filter unit 1023D, the output signal can be expressed as in Expression (10). An error between the output signal and the reference signal is expressed as in Expression (11).
y4(0)=hL64(0)xL(−64)+hL66(0)x(−66)+ . . . +hL126(0)xL(−126) Expression (10)
e4(0)=xL(0)−y4(0) Expression (11)
Here, the filter coefficients in Expression (8) are substituted with the filter coefficients in the adaptive filter unit 1023A. As a result, the output signal is expressed as in Expression (12).
y3(0)=h32(0)xL(−32)+h33(0)xL(−33)+ . . . +h63(0)xL(−63) Expression (12)
Here, the filter coefficients in Expression (10) are substituted with the filter coefficients in the adaptive filter unit 1023A. As a result, the output signal is expressed as in Expression (13).
y4(0)=h32(0)xL(−64)+h33(0)xL(−66)+ . . . +h63(0)xL(−126) Expression (13)
In the third embodiment, an expression by which the adaptive filter unit 1023A updates the filter coefficients h32 to h63 is described as follows. Parentheses are independent terms in the embodiment.
h32(1)=h32(0)+μ1e1(0)x(−32)+[μ2e2(0)x(−64)+μ3e3(0)xL(−32)+μ4e4(0)xL(−64)]
h33(1)=h33(0)+μ1e1(0)x(−33)+[μ2e2(0)x(−66)+μ3e3(0)xL(−33)+μ4e4(0)xL(−66)]
. . .
h63(1)=h63(0)+[μ1e1(0)x(−63)+μ2e2(0)x(−126)+μ3e3(0)xL(−63)+μ4e4(0)xL(−126)]
That is, in the third embodiment, when the adaptive filter unit 1023A updates the filter coefficients, correction results of the filter coefficients by the adaptive filter units 1023B, 123C, and 123D is added. In other words, results of the determination of the similarity performed during the periods T2, T3, and T4 by the adaptive filter units 1023B, 123C, and 123D are added to a result of the determination of the similarity performed during the period T1 by the adaptive filter unit 1023A.
In the third embodiment, the periods T2, T3, and T4 are equivalent to the second period. The length of the periods T2, T3, and T4 may be n times (where n is an integer equal to or greater than 2) the length of the period T1.
In the third embodiment, as described above, periodicity of a musical sound signal during the period T1 and periodicity of a musical sound signal during the periods T2, T3, and T4 (of which lengths are twice, 4 times, and 8 times the length of T1) are added for evaluation. Further, the musical sound signal is separated into the high sound area and the low sound area, the periods T1 and T2 are evaluated using the musical sound signal of the high sound area, the periods T3 and T4 are evaluated using the musical sound signal of the low sound area. In general, since a musical instrument of a high sound area (for example, a hi-hat or the like) tends to be sounded at a fast tempo and a musical instrument of a low sound area (for example, a bass drum or the like) tends to be sounded at a slow tempo, determination of a tempo with higher precision than in the second embodiment is accordingly possible.
Specific details of the above-exemplified embodiments have been described. Table 1 is a table that shows progress of a piece of music which is an evaluation target. A tempo of the piece of music is assumed to be 120 BPM.
In an intro section, a tempo is estimated to be 120 BPM. Thereafter, when the piece of music is advanced to a section of Melody A or Melody B, a piano of which keys are stroked at random is added and a tempo of percussion is changed. Therefore, in a mathematical method, it is difficult to estimate a tempo correctly.
On the other hand, in a method according to the embodiments, when a tempo of a section of Melody A or B is estimated, an estimation result of the tempo in an intro section is added to perform cumulative evaluation. Thus, even when the piece of music is advanced after Melody A, an estimated tempo of the piece of music does not considerably deviate from 120 BPM consequently.
In a section of Melody C, percussion corresponds to 60 BPM and is performed. However, since an estimation result cumulative until now is added even in evaluation in the section of Melody C, an evaluation result of 120 BPM is maintained as a whole.
In this way, since the tempo detection part according to the embodiments cumulates results obtained by evaluating the plurality of sections and performs comprehensive evaluation, a tempo of a piece of music can be detected with higher precision than when a simple mathematical scheme is used. In other words, a tempo of a piece of music can be evaluated musically in consideration of advance of the piece of music.
The foregoing embodiments are merely exemplary and the disclosure can be modified appropriately within the scope of the disclosure without departing from the gist of the disclosure. For example, the exemplary embodiments may be combined and realized.
For example, in the second embodiment, a musical sound signal may also be separated using a highpass filter and a lowpass filter. In this case, a musical sound signal input to an adaptive filter unit corresponding to a faster tempo may include a frequency component higher than that of a musical sound signal input to an adaptive filter unit corresponding to a slower tempo.
In the description of the embodiments, the plurality of sample groups included within the predetermined period (for example, a step 32 steps earlier to a step 63 steps earlier) have been input as input signals to the adaptive filter, but a target evaluated by an adaptive filter may be a single sample. In this case, the filter coefficient is a single value, as illustrated in
In the second and third embodiments, the plurality of adaptive filter units have been used, but a single adaptive filter unit may be used in a time division manner.
In the description of the embodiments, the video recording part 201 has recorded the video signal and the video source selection part 202 has generated the output signal by combining the plurality of recorded videos. On the other hand, the tempo detection device 100 can also detect beats in real time. In this case, the tempo detection device 100 may generate tempo information whenever a beat is detected, and may transmit the tempo information to the video processing device 200 in real time. In this case, the tempo information is information indicating a beat appearance timing. The video processing device 200 may select a plurality of video sources based on the beat appearance timing notified of in real time without recording the video and may output the selected video source.
In the description of the embodiments, the adaptive filters have been used as parts obtaining similarity of a musical sound signal (between samples). However, when data indicating periodicity of a waveform of a musical sound signal can be acquired, similarity between samples may be obtained using a part other than the exemplified parts.
In the description of the embodiments, the tempo detection device 100 and the video processing device 200 are different devices, but hardware in which both the tempo detection device and the video processing device are integrated may be used.
In the description of the embodiments, the system in which the video processing device 200 switches between the plurality of cameras has been exemplified. However, the video processing device 200 may be omitted and the single tempo detection device 100 may be realized.
It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the disclosure covers modifications and variations provided that they fall within the scope of the following claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| JP2018-247689 | Dec 2018 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 7518054 | McKinney | Apr 2009 | B2 |
| 10303423 | Kaczynski | May 2019 | B1 |
| 20020037083 | Weare | Mar 2002 | A1 |
| 20040094019 | Herre | May 2004 | A1 |
| 20040177746 | Becker | Sep 2004 | A1 |
| 20050217463 | Kobayashi | Oct 2005 | A1 |
| 20090288545 | Mann | Nov 2009 | A1 |
| 20100282045 | Chen | Nov 2010 | A1 |
| 20110023691 | Iwase | Feb 2011 | A1 |
| 20120130516 | Reinsch | May 2012 | A1 |
| 20140307878 | Osborne | Oct 2014 | A1 |
| 20190377539 | O'Donnell | Dec 2019 | A1 |
| 20200211517 | Tsutaki | Jul 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 2005026739 | Jan 2005 | JP |
| 2005295431 | Oct 2005 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20200211517 A1 | Jul 2020 | US |
