Patent Grant
10366684
The present invention relates to a technique of providing information that is synchronized with user's performance of a piece of music.
Conventionally, there have been proposed techniques (referred to as “score alignment”) for performing analysis on a position in a piece of music at which position a user is currently performing the piece of music. Non-Patent Documents 1 and 2, for example, each disclose a technique for analyzing temporal correspondence between positions in a piece of music and sound signals representing performance sounds of the piece of music, by use of a probability model, such as a hidden Markov model (HMM).
Non-Patent Document 1: MAEZAWA, Akira, OKUNO, Hiroshi G. “Non-Score-Based Music Parts Mixture Audio Alignment”, IPSJ SIG Technical Report, Vol. 2013-MUS-100 No. 14, 2013 Sep. 1
Non-Patent Document 2: MAEZAWA, Akira, ITOYAMA, Katsutoshi, YOSHII, Kazuyoshi, OKUNO, Hiroshi G. “Inter-Acoustic-Signal Alignment Based on Latent Common Structure Model”, IPSJ SIG Technical Report, Vol. 2014-MUS-103 No. 23, 2014 May 24
It would be convenient for producing multiple music part performance sounds if, while analyzing a position in a piece of music being performed by a user, an accompaniment instrumental and/or vocal sound can be reproduced synchronously with the performance of the user based on the music information that has been prepared in advance. Performing analysis, etc., on a performance position, however, involves a processing delay. Therefore, if a user is provided with music information that corresponds to a time point corresponding to a performance position in a piece of music, which position is identified based on a performance sound, the music information provided becomes delayed with respect to the user's performance. Described above is a processing delay that is involved in analysis of a performance position, and a delay may also occur in providing music information due to a communication delay among devices in a communication system because in such a communication system a performance sound that has been transmitted from a terminal device is received via a communication network and analyzed, and music information is then transmitted to the terminal device.
In consideration of the circumstances described above, it is an object of the present invention to reduce a delay that is involved in providing music information.
In order to solve the aforementioned problem, an information providing method according to a first aspect of the present invention includes: sequentially identifying a performance speed at which a user performs a piece of music; identifying, in the piece of music, a performance position at which the user is performing the piece of music; setting an adjustment amount in accordance with a temporal variation in the identified performance speed; and providing the user with music information corresponding to a time point that is later, by the set adjustment amount, than a time point that corresponds to the performance position identified in the piece of music. In this configuration, a user is provided with music information that corresponds to a time point that is later, by the adjustment amount, than the time point that corresponds to a position that is being performed by the user in the piece of music.
An information providing method according to a second aspect of the present invention includes: sequentially identifying a performance speed of performance of a user; identifying a beat point of the performance of the user; setting an adjustment amount in accordance with a temporal variation in the identified performance speed; and indicating, to the user, a beat point at a time point that is shifted with respect to the identified beat point by the set adjustment amount.
A performance device 14 is connected to each of the terminal devices 12. Each performance device 14 is an input device, which receives performance of a specific piece of music by each user U (UA or UB) of the corresponding terminal device 12, and generates performance information Q (QA or QB) representing a performance sound of the piece of music. An electrical musical instrument that generates a sound signal representing a time waveform of the performance sound as the performance information Q, or one that generates as the performance information Q time series data representing content of the performance sound (e.g., a MIDI instrument that outputs MIDI-format data in time series), for example, can be used as the performance device 14. Furthermore, an input device included in the terminal device 12 can also be used as the performance device 14. In the following example a case is assumed where the user UA of the terminal device 12A performs a first part of a piece of music and the user UB of the terminal device 12B performs a second part of the piece of music. It is of note, however, that the respective contents of the first and second parts of the piece of music may be identical or differ from each other.
The user UA of the terminal device 12A and the user UB of the terminal device 12B are able to perform music together in ensemble via the communication network 18 (a so-called “network session”). Specifically, as illustrated in
Meanwhile, the information providing device 10 as in the first embodiment sequentially provides each of the terminal devices 12A and 12B with sampling data (discrete data) of music information M synchronously with the performance of the user UA of the terminal device 12A, the music information M representing a time waveform of an accompaniment sound of the piece of music (a performance sound of an accompaniment part that differs from the first part and the second part). As a result of the operation described above, a sound mixture consisting of a performance sound of the first part, a performance sound of the second part, and the accompaniment sound is output from the respective sound output devices 34 of the terminal devices 12A and 12B, with the performance sound of the first part being represented by the performance information QA, the performance sound of the second part by the performance information QB, and the accompaniment sound by the music information M. Each of the users UA and UB are thus enabled to perform the piece of music by operating the performance device 14 while listening to the accompaniment sound provided by the information providing device 10 and to the performance sound of the counterpart user.
The communication device 44 communicates with each of the terminal devices 12 via the communication network 18. Specifically, the communication device 44 as in the first embodiment receives, from the terminal device 12A, the performance information QA of the performance of the user UA. In the meantime, the communication device 44 sequentially transmits the sampling data of the music information M to each of the terminal devices 12A and 12B, such that the accompaniment sound is synchronized with the performance represented by the performance information QA.
By executing the program stored in the storage device 42, the control device 40 realizes multiple functions (an analysis processor 50, an adjustment amount setter 56, and an information provider 58) for providing the music information M to the terminal devices 12. It is of note, however, that a configuration in which the functions of the control device 40 are dividedly allocated to a plurality of devices, or a configuration which employs electronic circuitry dedicated to realize part of the functions of the control device 40, may also be employed.
The analysis processor 50 is an element that analyzes performance information QA received by the communication device 44 from the terminal device 12A, and includes a speed analyzer 52 and a performance analyzer 54. The speed analyzer 52 identifies a speed V of the performance (hereinafter referred to as a “performance speed”) of the piece of music by the user UA. The performance of the user UA is represented by the performance information QA. The performance speed V is identified sequentially and in real time, in parallel with the progress of the performance of the piece of music by the user UA. The performance speed V is identified, for example, in the form of tempo which is expressed as the number of beats per unit time. A freely selected one of publically known techniques can be employed for the speed analyzer 52 to identify the performance speed V.
The performance analyzer 54 identifies, in the piece of music, a position T at which the user UA is performing the piece of music (this position will hereinafter be referred to as a “performance position”). Specifically, the performance analyzer 54 identifies the performance position T by collating the user UA's performance represented by the performance information QA with the time series of a plurality of notes indicated in the score information S that is stored in the storage device 42. The performance position T is identified sequentially and in real time, in parallel with the progress of the performance of the piece of music by the user UA. One of well-known techniques (e.g., the score alignment techniques disclosed in Non-Patent Documents 1 and 2) may be freely selected for employment for the performance analyzer 54 to identify the performance position T. It is of note that, in a case where the users UA and UB perform parts of the piece of music that are different from each other, the performance analyzer 54 first identifies the part performed by the user UA, among a plurality of parts indicated in the score information S, and then identifies the performance position T.
The information provider 58 in
A delay (a processing delay and a communication delay) may occur from a time point at which the user UA performs the piece of music until the music information M is received and played by the terminal device 12A or 12B, because the music information M is transmitted from the information providing device 10 to the terminal device 12A or 12B after the performance information QA is transmitted from the terminal device to the information providing device 10 and is analyzed at the information providing device 10. As illustrated in
A delay (a processing delay and a communication delay) of approximately 30 ms may occur from a time point at which the user UB performs a specific portion of the piece of music until the performance sound of the performance of that specific portion is output from the sound output device 34 of the terminal device 12A, because, after the user UB performs that portion, the performance information QB is transmitted by the terminal device 12B and received by the terminal device 12A before the performance sound is played at the terminal device 12A. The user UA performs his/her own part as follows so that the performance of the user UA and the performance of the user UB become coincident with each other despite occurrence of a delay as illustrated above. With the performance device 14, the user UA performs his/her own part corresponding to a specific portion performed by the user UB in the piece of music, at a (first) time point that temporally precedes (is earlier than) a (second) time point at which the performance sound corresponding to that specific portion is expected to be output from the sound output device 34 of the terminal device 12A. Here, the first time point is earlier than the second time point by a delay amount estimated by the user UA (this delay estimated by the user UA will hereinafter be referred to as a “recognized delay amount”). That is to say, the user UA plays the performance device 14 by temporally preceding, by his/her own recognized delay amount, the performance sound of the user UB that is actually output from the sound output device 34 of the terminal device 12A.
The recognized delay amount is a delay amount that the user UA estimates as a result of listening to the performance sound of the user UB. The user UA estimates the recognized delay amount in the course of performing the piece of music, on an as-needed basis. Meanwhile, the control device 30 of the terminal device 12A causes the sound output device 34 to output the performance sound of the performance of the user UA at a time point that is delayed with respect to the performance of the user UA by a prescribed delay amount (e.g., a delay amount of 30 ms estimated either experimentally or statistically). As a result of the aforementioned process being executed in each of the terminal devices 12A and 12B, each terminal device 12A and 12B outputs a sound in which the performance sounds of the users UA and UB are substantially coincident with each other.
The adjustment amount α set by the adjustment amount setter 56 is preferably set to a time length that corresponds to the recognized delay amounts perceived by the respective users U. However, since the delay amounts are predicted by the respective users U, the recognized delay amount cannot be directly measured. Accordingly, with the results of simulation explained in the following taken into consideration, the adjustment amount setter 56 according to the first embodiment sets the adjustment amount α to be variable in accordance with a temporal variation in the performance speed V identified by the speed analyzer 52.
In accordance with the foregoing, the adjustment amount setter 56 as in the first embodiment sets the adjustment amount α to be variable in accordance with a temporal variation in the performance speed V identified by the speed analyzer 52. Specifically, the adjustment amount setter 56 sets the adjustment amount α in accordance with a temporal variation in the performance speed V, such that the adjustment amount α decreases when the performance speed V increases over time (i.e., when the adjustment amount α is estimated to be greater than the recognized delay amount perceived by the user UA), and such that the adjustment amount α increases when the performance speed V decreases over time (i.e., when the adjustment amount α is estimated to be smaller than the recognized delay amount perceived by the user UA). Accordingly, in a situation where the performance speed V increases over time, a variation in the performance speed V is turned into a decrease by allowing the respective beat points of the accompaniment sound represented by the music information M to move behind a time series of respective beat points predicted by the user UA, whereas in a situation where the performance speed V decreases over time, a variation in the performance speed V is turned into an increase by allowing the respective beat points of the accompaniment sound to move ahead of a time series of respective beat points predicted by the user UA. In other words, the adjustment amount α is set such that the performance speed V of the user UA be maintained essentially constant.
The adjustment amount setter 56 sets the adjustment amount α to be variable in accordance with the variation degree R among the performance speeds V (S34). Specifically, the adjustment amount setter 56 according to the first embodiment calculates a subsequent adjustment amount α (αt+1) through an arithmetic expression F(αt,R) in Expression (1), where a current adjustment amount α (αt) and the variation degree R among the performance speeds V are variables of the expression.
αt+1=F(αt,R)=αt exp(cR) (1)
The symbol “c” in Expression (1) is a prescribed negative number (c<0).
Having calculated the adjustment amount α by the above procedure, the adjustment amount setter 56 clears the N number of the performance speeds V stored in the storage device 42 (S35), and the process then returns to step S31. As will be understood from the explanation given above, calculation of the variation degree R (S33) and update of the adjustment amount α (S34) are performed repeatedly for every set of N number of the performance speeds V identified from the performance information QA by the speed analyzer 52.
In the first embodiment, as explained above, at each terminal device 12, an accompaniment sound is played which corresponds to a portion of the music information M, and this portion corresponds to a time point that is later, by the adjustment amount α, than a time point corresponding to the performance position T of the user UA. Thus, a delay in providing the music information M can be reduced in comparison to a configuration of providing the respective terminal devices 12 with a portion of the music information M that corresponds to a time point corresponding to the performance position T. In the first embodiment, a delay might occur in providing the music information M due to a communication delay because information (e.g., the performance information QA and the music information M) is transmitted and received via the communication network 18. Therefore, an effect of the present invention, i.e., reducing a delay in providing music information M, is particularly pronounced. Moreover, in the first embodiment, since the adjustment amount α is set to be variable in accordance with a temporal variation (the variation degree R) in the performance speed V of the user UA, it is possible to guide the performance of the user UA such that the performance speed V is maintained essentially constant. Frequent fluctuations in the adjustment amount α can also be reduced in comparison to a configuration in which the adjustment amount α is set for each performance speed V.
In a case where multiple terminal devices 12 perform music in ensemble over the communication network 18, it is also possible to adopt a configuration in which performance information Q (e.g., QA) of the user U himself/herself of a prescribed amount is buffered in a corresponding terminal device 12 (e.g., 12A), and a read position of the buffered performance information Q (e.g., QA) is controlled so as to be variable in accordance with the communication delay involved in the actual delay in providing music information M and performance information Q (e.g., QB) of another user U, for the purpose of compensating for fluctuations in a communication delay occurring in the communication network 18. When the first embodiment is applied to this configuration, since the adjustment amount α is controlled so as to be variable in accordance with a temporal variation in the performance speed V, an advantage is obtained in that the amount of delay in buffering the performance information Q can be reduced.
The second embodiment of the present invention will now be explained. In the embodiments illustrated in the following, elements that have substantially the same effects and/or functions as those of the elements in the first embodiment will be assigned the same reference signs as in the description of the first embodiment, and detailed description thereof will be omitted as appropriate.
The first embodiment illustrates a configuration in which the speed analyzer 52 identifies the performance speeds V across all sections of the piece of music. The speed analyzer 52 according to the second embodiment identifies the performance speed V of the user UA sequentially for a specific section (hereinafter referred to as an “analyzed section”) in the piece of music.
The analyzed section is a section in which the performance speed V is highly likely to be maintained essentially constant, and such a section(s) is (are) identified by referring to the score information S stored in the storage device 42. Specifically, the adjustment amount setter 56 identifies, as the analyzed section, a section other than a section on which an instruction is given to increase or decrease a performance speed (i.e., a section on which an instruction is given to maintain the performance speed V) in the score of the piece of music as indicated in the score information S. For each of analyzed section(s) of the piece of music, the adjustment amount setter 56 calculates a variation degree R among performance speeds V. In the piece of music, the performance speeds V are not identified for sections other than the analyzed sections, thus the performance speeds V in those sections other than the analyzed sections are not reflected in the variation degree R (nor in the adjustment amount α).
Substantially the same effects as those of the first embodiment are obtained in the second embodiment. In the second embodiment, since the performance speeds V of the user U are identified for specific sections of the piece of music, a processing load in identifying the performance speeds V is reduced in comparison to a configuration in which the performance speeds V are identified for all sections. Moreover, the analyzed section is identified based on the score information S, i.e., the score information S used to identify the performance position T is also used to identify the analyzed section. Therefore, an amount of data retained in the storage device 42 (hence a storage capacity needed for the storage device 42) is reduced in comparison to a configuration in which information indicating performance speeds of the score of the piece of music and score information S used to identify a performance position T are retained as separate information. In the second embodiment, moreover, since the adjustment amount α is set in accordance with the performance speeds V in the analyzed section(s) of the piece of music, an advantage is obtained in that it is possible to set an appropriate adjustment amount α that is free of the impact of fluctuations in the performance speed V, which fluctuations occur as a result of musical expressivity in performance of the user UA.
In the example described above, the performance speed V is calculated by selecting, as a section to be analyzed, a section in the piece of music in which section the performance speed V is highly likely to be maintained essentially constant. However, a method of selecting an analyzed section is not limited to the above example. For example, by referring to the score information S, the adjustment amount setter 56 may select as the analyzed section a section in the piece of music on which it is easy to identify a performance speed V with good precision. For example, in the piece of music, it tends to be easier to identify a performance speed V with high accuracy in a section in which a large number of short notes are distributed, as opposed to a section in which long notes are distributed. Accordingly, the adjustment amount setter 56 may be preferably configured to identify as the analyzed section a section in the piece of music, in which section there are a large number of short notes, such that performance speeds V are identified for the identified analyzed section. Specifically, in a case where the total number of notes (i.e., appearance frequency of notes) in a section having a prescribed length (e.g., a prescribed number of bars) is equal to or greater than a threshold, the adjustment amount setter 56 may identify that section as the analyzed section. The speed analyzer 52 identifies performance speeds V for that section, and the adjustment amount setter 56 calculates a variation degree R among the performance speeds V in that section. Therefore, the performance speeds V of the performance in a section(s) each having the prescribed length and including the number of notes equal to or greater than the threshold are reflected in the adjustment amount α. Meanwhile, the performance speeds V of the performance in a section(s) each having the prescribed length and including the number of notes smaller than the threshold are not identified, and the performance speeds V of the performance of the section(s) are not reflected in the adjustment amount α.
Substantially the same effects as those of the first embodiment are also obtained in the above configuration. Moreover, as described above, a processing load in identifying the performance speeds V is reduced in comparison to a configuration in which the performance speeds V are identified for all sections. Substantially the same effect as the aforementioned effect achieved by utilizing the score information S to identify the analyzed section can also be obtained. Furthermore, since a section on which it is relatively easy to identify a performance speed with good precision is identified as the analyzed section, an advantage is obtained in that it is possible to set an appropriate adjustment amount α that is based on performance speeds identified with high accuracy.
As described with reference to
The performance analyzer 54 as in the third embodiment sequentially identifies a beat point of the performance (hereinafter referred to as a “performance beat point”) of the user UA, by analyzing the performance information QA received by the communication device 44 from the terminal device 12A. One of well-known techniques can be freely selected for employment for the performance analyzer 54 to identify the performance beat points. Meanwhile, the adjustment amount setter 56 sets the adjustment amount α to be variable in accordance with a temporal variation in the performance speed V identified by the speed analyzer 52, similarly to the first embodiment. Specifically, the adjustment amount setter 56 sets the adjustment amount α in accordance with a variation degree R among the performance speeds V, such that the adjustment amount α decreases when the performance speed V increases over time (R>0), and that the adjustment amount α increases when the performance speed V decreases over time (R<0).
The information provider 58 as in the third embodiment sequentially indicates a beat point to the user UA at a time point that is shifted, by the adjustment amount α, from the performance beat point identified by the performance analyzer 54. Specifically, the information provider 58 sequentially transmits, from the communication device 44 to the terminal device 12A of the user UA, a sound signal representing a sound effect (e.g., a metronome click) for enabling the user UA to perceive a beat point. Specifically, a timing at which the information providing device 10 transmits a sound signal representing a sound effect to the terminal device 12A is controlled in the following manner. That is, in a case where the performance speed V decreases over time, the sound output device 34 of the terminal device 12A outputs a sound effect at a time point preceding a performance beat point of the user UA, and in a case where the performance speed V increases over time, the sound output device 34 of the terminal device 12A outputs a sound effect at a time point that is delayed with respect to a performance beat point of the user UA.
A method of allowing the user UA to perceive beat points is not limited to outputting of sounds. A blinker or a vibrator may be used to indicate beat points to the user UA. The blinker or the vibrator may be incorporated inside the terminal device 12, or attached thereto externally.
According to the third embodiment, a beat point is indicated to the user UA at a time point that is shifted, by the adjustment amount α, from a performance beat point identified by the performance analyzer 54 from the performance of the user UA, and thus an advantage is obtained in that the user UA can be guided such that the performance speed is maintained essentially constant.
Modifications
The embodiments illustrated above can be modified in various ways.
Specific modes of modification will be described in the following. Two or more modes selected from the following examples may be combined, as appropriate, in so far as the modes combined do not contradict one another.
(1) In the first and second embodiments described above, each terminal device 12 is provided with music information M that represents the time waveform of an accompaniment sound of a piece of music, but the content of the music information M is not limited to the above example. For example, music information M representing a time waveform of a singing sound (e.g., a voice recorded in advance or a voice generated using voice synthesis) of the piece of music can be provided to the terminal devices 12 from the information providing device 10. The music information M is not limited to information indicating a time waveform of a sound. For example, the music information M may be provided to the terminal devices 12 in the form of time series data in which operation instructions, to be directed to various types of equipment such as lighting equipment, are arranged so as to correspond to respective positions in the piece of music. Alternatively the music information M may be provided in the form of a moving image (or a time series consisting of a plurality of still images) related to the piece of music.
Furthermore, in a configuration in which a pointer indicating a performance position is arranged in a score image displayed on the terminal device 12, and in which the pointer is moved in parallel with the progress of the performance of the piece of music, the music information M is provided to the terminal device 12 in the form of information indicating a position of the pointer. It is of note that a method of indicating the performance position to the user is not limited to the above example (displaying of a pointer). For example, blinking by a light emitter, vibration of a vibrator, etc., can also be used to indicate the performance position (e.g., a beat point of the piece of music) to the user.
As will be understood from the above example, typical examples of the music information M include data of a time series that is supposed to progress temporally along with the progress of the performance or playback of the piece of music. The information provider 58 is comprehensively expressed as an element that provides music information M (e.g., a sound, an image, or an operation instruction) that corresponds to a time point that is later, by an adjustment amount α, than a time point (a time point on a time axis of the music information M) that corresponds to a performance position T.
(2) The format and/or content of the score information S may be freely selected. Any information representing performance contents of at least a part of the piece of music (e.g., lyrics, or scores consisting of tablature, chords, or percussion notation) may be used as the score information S.
(3) In each of the embodiments described above, an example was given of a configuration in which the information providing device 10 communicates with the terminal device 12A via the communication network 18, but the terminal device 12A may be configured to function as the information providing device 10. In this case, the control device 30 of the terminal device 12A functions as a speed analyzer, a performance analyzer, an adjustment amount setter, and an information provider. The information provider, for example, provides to the sound output device 34 sampling data of a portion corresponding to a time point that is later, by an adjustment amount α, than a time point corresponding to a performance position T identified by the performance analyzer in the music information M of the piece of music, thereby causing the sound output device 34 to output an accompaniment sound of the piece of music. As will be understood from the above explanation, the following operations are comprehensively expressed as operations to provide a user with music information M: an operation of transmitting music information M to a terminal device 12 from an information providing device 10 that is provided separately from the terminal devices 12, as described in the first and second embodiments; and an operation, performed by a terminal device 12A, of playing an accompaniment sound corresponding to the music information M in a configuration in which the terminal device 12A functions as an information providing device 10. That is to say, providing the music information M to a terminal device 12, and indicating the music information M to the user (e.g., emitting an accompaniment sound, or displaying a pointer indicating a performance position), are both included in the concept of providing music information M to the user.
Transmitting and receiving of performance information Q between the terminal devices 12A and 12B may be omitted (i.e., the terminal device 12B may be omitted). Alternatively, the performance information Q may be transmitted and received among three or more terminal devices 12 (i.e., ensemble performance by three or more users U).
In a scene where the terminal device 12B is omitted and in which only the user UA plays the performance device 14, the information providing device 10 may be used, for example, as follows. First, the user UA performs a first part of the piece of music in parallel with the playback of an accompaniment sound represented by music information M0 (the music information M of the first embodiment described above), in the same way as in the first embodiment. The performance information QA, which represents a performance sound of the user UA, is transmitted to the information providing device 10 and is stored in the storage device 42 as music information M1. Then, in the same way as the first embodiment, the user UA performs a second part of the piece of music in parallel with the playback of the accompaniment sound represented by the music information M0 and the performance sound of the first part represented by the music information M1. As a result of the above process being repeated, music information M is generated for each of multiple parts of the piece of music, with these pieces of music information M respectively representing performance sounds that synchronize together at an essentially constant performance speed. The control device 40 of the information providing device 10 synthesizes the performance sounds represented by the multiple pieces of the music information M, in order to generate music information M of an ensemble sound. As will be understood from the above explanation, it is possible to record (overdub) an ensemble sound in which respective performances of multiple parts by the user UA are multiplexed. It is also possible for the user UA to perform processing, such as deleting and editing, on each of the multiple pieces of music information M, each representing the performance of the user UA.
(4) In the first and second embodiments described above, the performance position T is identified by analyzing the performance information QA corresponding to the performance of the user UA; however, the performance position T may be identified by analyzing both the performance information QA of the user UA and the performance information QB of the user UB. For example, the performance position T may be identified by collating, with the score information S, a sound mixture of the performance sound represented by the performance information QA and the performance sound represented by the performance information QB. In a case where the users UA and UB perform mutually different parts of the piece of music, the performance analyzer 54 may identify the performance position T for each user U after identifying a part that is played by each user U among multiple parts indicated in the score information S.
(5) In the embodiments described above, a numerical value calculated through Expression (1) is employed as the adjustment amount α, but a method of calculating the adjustment amount α corresponding to temporal variation in the performance speed V is not limited to the above described example. For example, the adjustment amount α may be calculated by adding a prescribed compensation value to the numerical value calculated through Expression (1). This modification enables the providing of music information M that corresponds to a time point that precedes a time point that corresponds to a performance position T of each user U, by a time length equivalent to a compensated adjustment amount α, and is especially suitable for a case in which positions or content of performance are to be sequentially indicated to a user U, i.e., a case in which music information M must be indicated prior to the performance of a user U. For example, it is especially suitable for a case in which a pointer indicating a performance position is displayed on a score image, as described above. A fixed value set in advance, or a variable value that is in accordance with an indication from the user U may, for example, be set as the compensation value utilized in the calculation of the adjustment amount α. Furthermore, a range of the music information M indicated to the user U may be freely selected. For example, in a configuration in which content to be performed by the user U is sequentially provided to the user U in the form of sampling data of the music information M, it is preferable to indicate, to the user U, music information M that covers a prescribed unit amount (e.g., a range covering a prescribed number of bars constituting the piece of music) from a time point corresponding to the adjustment amount α.
(6) In each of the embodiments described above, the performance speeds V and/or the performance position T are analyzed with respect to the performance of the performance device 14 by the user UA, but performance speeds (singing speeds) V and/or performance position (singing position) T may also be identified for the singing of the user UA, for example. As will be understood from the above example, the “performance” in the present invention includes singing by a user, in addition to instrumental performance (performance in a narrow sense) using a performance device 14 or other relevant equipment.
(7) In the second embodiment, the speed analyzer 52 identifies performance speeds V of the user UA for a particular section in the piece of music. However, the speed analyzer 52 may also identify the performance speeds V across all sections of the piece of music, similarly to the first embodiment. The adjustment amount setter 56 identifies analyzed sections, and calculates, for each of the analyzed sections, a variation degree R among performance speeds V that fall in a corresponding analyzed section, from among the performance speeds V identified by the speed analyzer 52. Since the variation degrees R are not calculated for sections other than the analyzed sections, the performance speeds V in those sections other than the analyzed sections are not reflected in the variation degrees R (nor in the adjustment amounts α). Substantially the same effects as those of the first embodiment are also obtained according to this modification. In addition, the adjustment amounts α are set in accordance with the performance speeds V in the analyzed sections of the piece of music, similarly to the second embodiment, and therefore, an advantage is obtained in that it is possible to set appropriate adjustment amounts α by identifying, as the analyzed sections, sections of the piece of music that are suitable for identifying the performance speeds V (e.g., a section in which the performance speed V is highly likely to be maintained essentially constant, or a section on which it is easy to identify the performance speeds V with good precision).
Programs according to the aforementioned embodiments may be provided, being stored in a computer-readable recording medium and installed in a computer. The recording medium includes a non-transitory recording medium, a preferable example of which is an optical storage medium, such as a CD-ROM (optical disc), and can also include a freely selected form of well-known storage media, such as a semiconductor storage medium and a magnetic storage medium. It is of note that the programs according to the present invention can be provided, being distributed via a communication network and installed in a computer.
The following aspects of the present invention may be derived from the different embodiments and modifications described in the foregoing.
An information providing method according to a first aspect of the present invention includes: sequentially identifying a performance speed at which a user performs a piece of music; identifying, in the piece of music, a performance position at which the user is performing the piece of music; setting an adjustment amount in accordance with a temporal variation in the identified performance speed; and providing the user with music information corresponding to a time point that is later, by the set adjustment amount, than a time point that corresponds to the performance position identified in the piece of music. In this configuration, a user is provided with music information that corresponds to a time point that is later, by the adjustment amount, than the time point that corresponds to a position that is being performed by the user in the piece of music. Accordingly, it is possible to reduce a delay in providing the music information, in comparison to a configuration in which a user is provided with music information corresponding to a time point that corresponds to a performance position by the user. Moreover, the adjustment amount is set to be variable in accordance with a temporal variation in the speed of performance by the user, and therefore, it is possible to guide the performance of the user such that, for example, the performance speed is maintained essentially constant.
Given that the performance speed tends to decrease over time when the adjustment amount is small and that the performance speed tends to increase over time when the adjustment amount is large, for example, a configuration is preferable in which the adjustment amount is set so as to decrease when the identified performance speed increases and increase when the performance speed decreases. According to this aspect of the invention, it is possible to guide the performance of the user such that the performance speed is maintained essentially constant.
According to a preferable embodiment of the present invention, the performance speed at which the user performs the piece of music is identified for a prescribed section in the piece of music. According to this embodiment, a processing load in identifying the performance speed is reduced in comparison to a configuration in which the performance speed is identified for all sections of the piece of music.
The performance position, in the piece of music, at which the user is performing the piece of music may be identified based on score information representing a score of the piece of music, and the prescribed section in the piece of music may be identified based on the score information. This configuration has an advantage in that, since the score information is used to identify not only the performance position but also the prescribed section, an amount of data retained can be reduced in comparison to a configuration in which information used to identify a prescribed section and information used to identify a performance position are retained as separate information.
A section in the piece of music other than a section on which an instruction is given to increase or decrease a performance speed, for example, may be identified as the prescribed section of the piece of music. In this embodiment, the adjustment amount is set in accordance with the performance speed in a section in which the performance speed is highly likely to be maintained essentially constant. Accordingly, it is possible to set an adjustment amount that is free of the impact of fluctuations in the performance speed, which fluctuations occur as a result of musical expressivity in performance of the user.
Furthermore, a section that has a prescribed length and includes notes of the number equal to or greater than a threshold in the piece of music may be identified as the prescribed section in the piece of music. In this embodiment, performance speeds are identified for a section on which it is relatively easy to identify the performance speed with good precision. Therefore, it is possible to set an adjustment amount that is based on performance speeds identified with high accuracy.
The information providing method according to a preferable embodiment of the present invention includes: identifying the performance speed sequentially by analyzing performance information received from a terminal device of the user via a communication network; identifying the performance position by analyzing the received performance information; and providing the user with the music information by transmitting the music information to the terminal device via the communication network. In this configuration, a delay occurs due to communication to and from the terminal device (a communication delay). Thus, a particularly advantageous effect is realized by the present invention in that any delay in providing music information is minimized.
According to a preferable embodiment of the present invention, the information providing method further includes calculating, from a time series consisting of a prescribed number of the performance speeds that are identified, a variation degree which is an indicator of a degree and a direction of the temporal variation in the performance speed, and the adjustment amount is set in accordance with the variation degree. The variation degree may, for example, be expressed as an average of gradients of the performance speeds, each of the gradients being determined based on two consecutive performance speeds in the time series consisting of the prescribed number of the performance speeds. Alternatively, the variation degree may also be expressed as a gradient of a regression line obtained, by linear regression, from the time series consisting of the prescribed number of the performance speeds. In this embodiment, the adjustment amount is set in accordance with the variation degree in the performance speed. Accordingly, frequent fluctuations in the adjustment amount can be reduced in comparison to a configuration in which an adjustment amount is set for each performance speed.
An information providing method according to a second aspect of the present invention includes: sequentially identifying a performance speed of performance of a user; identifying a beat point of the performance of the user; setting an adjustment amount in accordance with a temporal variation in the identified performance speed; and indicating, to the user, a beat point at a time point that is shifted with respect to the identified beat point by the set adjustment amount. According to the second aspect of the present invention, it is possible to guide the performance of the user such that the performance speed is, for example, maintained essentially constant.
The present invention may also be specified as an information providing device that executes the information providing methods set forth in the aforementioned aspects. The information providing device according to the present invention is either realized as dedicated electronic circuitry, or as a general processor, such as a central processing unit (CPU), the processor functioning in cooperation with a program.
100: communication system
10: information providing device
12 (12A, 12B): terminal device
14: performance device
18: communication network
30, 40: control device
32, 44: communication device
34: sound output device
42: storage device
50: analysis processor
52: speed analyzer
54: performance analyzer
56: adjustment amount setter
58: information provider
| Number | Date | Country | Kind |
|---|---|---|---|
| 2014-236792 | Nov 2014 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4484507 | Nakada | Nov 1984 | A |
| 5315911 | Ochi | May 1994 | A |
| 5521323 | Paulson | May 1996 | A |
| 5693903 | Heidorn | Dec 1997 | A |
| 5894100 | Otsuka | Apr 1999 | A |
| 5913259 | Grubb | Jun 1999 | A |
| 5952597 | Weinstock | Sep 1999 | A |
| 6051769 | Brown, Jr. | Apr 2000 | A |
| 6107559 | Weinstock | Aug 2000 | A |
| 6156964 | Sahai | Dec 2000 | A |
| 6166314 | Weinstock | Dec 2000 | A |
| 6333455 | Yanase | Dec 2001 | B1 |
| 6376758 | Yamada | Apr 2002 | B1 |
| 6380472 | Sugiyama | Apr 2002 | B1 |
| 6380474 | Taruguchi | Apr 2002 | B2 |
| 7164076 | McHale | Jan 2007 | B2 |
| 7189912 | Jung | Mar 2007 | B2 |
| 7297856 | Sitrick | Nov 2007 | B2 |
| 7482529 | Flamini | Jan 2009 | B1 |
| 7579541 | Guldi | Aug 2009 | B2 |
| 7649134 | Kashioka | Jan 2010 | B2 |
| 7989689 | Sitrick | Aug 2011 | B2 |
| 8015123 | Barton | Sep 2011 | B2 |
| 8180063 | Henderson | May 2012 | B2 |
| 8338684 | Pillhofer | Dec 2012 | B2 |
| 8367921 | Evans | Feb 2013 | B2 |
| 8440901 | Nakadai | May 2013 | B2 |
| 8445766 | Raveendran | May 2013 | B2 |
| 8629342 | Lee | Jan 2014 | B2 |
| 8660678 | Lavi | Feb 2014 | B1 |
| 8686271 | Wang | Apr 2014 | B2 |
| 8785757 | Pillhofer | Jul 2014 | B2 |
| 8838835 | Hara | Sep 2014 | B2 |
| 8889976 | Nakadai | Nov 2014 | B2 |
| 8990677 | Sitrick | Mar 2015 | B2 |
| 8996380 | Wang | Mar 2015 | B2 |
| 9135954 | Sitrick | Sep 2015 | B2 |
| 9275616 | Uemura | Mar 2016 | B2 |
| 9959851 | Fernandez | May 2018 | B1 |
| 20010023635 | Taruguchi et al. | Sep 2001 | A1 |
| 20020078820 | Miyake | Jun 2002 | A1 |
| 20020118562 | Hiratsuka | Aug 2002 | A1 |
| 20040025676 | Shadd | Feb 2004 | A1 |
| 20040177744 | Strasser | Sep 2004 | A1 |
| 20050115382 | Jung | Jun 2005 | A1 |
| 20080196575 | Good | Aug 2008 | A1 |
| 20080282872 | Ma | Nov 2008 | A1 |
| 20090229449 | Yamada | Sep 2009 | A1 |
| 20110003638 | Lee et al. | Jan 2011 | A1 |
| 20110277615 | Kendler | Nov 2011 | A1 |
| 20150206441 | Brown | Jul 2015 | A1 |
| 20170018262 | Pinuela Irrisarri | Jan 2017 | A1 |
| 20170220855 | Bose et al. | Aug 2017 | A1 |
| 20170230651 | Bose et al. | Aug 2017 | A1 |
| 20170256246 | Maezawa | Sep 2017 | A1 |
| 20170294134 | Angel et al. | Oct 2017 | A1 |
| 20170337910 | Maezawa et al. | Nov 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 2919228 | Sep 2015 | EP |
| S57124396 | Aug 1982 | JP |
| H03253898 | Nov 1991 | JP |
| 2007279490 | Oct 2007 | JP |
| 2011242560 | Dec 2011 | JP |
| 2015079183 | Apr 2015 | JP |
| 9916048 | Apr 1999 | WO |
| 2005022509 | Mar 2005 | WO |
| Entry |
|---|
| Notice of Allowance issued in U.S. Appl. No. 15/597,675 dated Mar. 9, 2018. |
| Extended European Search Report issued in European Application No. 15861046.9 dated Apr. 10, 2018. |
| International Search Report issued in Intl. Appln. No. PCT/JP2015/082514 dated Feb. 2, 2016. English translation provided. |
| Written Opinion issued in Intl. Appln. No. PCT/JP2015/082514 dated Feb. 2, 2016. English translation provided. |
| International Preliminary Report on Patentability issued in Intl. Appln. No. PCT/JP2015/082514 dated May 23, 2017. English translation provided. |
| Maezawa et al. “Non-Score-Based Music Parts Mixture Audio Alignment.” Information Processing Society of Japan. SIG Technical Report. Sep. 1, 2013: 1-6. vol. 2013-MUS-100, No. 14. Cited in Specification. English abstract provided. |
| Maezawa et al. “Inter-Acoustic-Signal Alignment Based on Latent Common Structure Model.” Information Processing Society of Japan. SIG Technical Report. May 24, 2014: 1-6. vol. 2014-MUS-103, No. 23. Cited in Specification. English abstract provided. |
| Shirogane et al. “Description and Verification of an Automatic Accompaniment System by a Virtual Text with Rendezvous.” Information Processing Society of Japan. Mar. 15, 1995: 1-369-1-370. Cited in NPL 1, NPL 2, and NPL 3. English translation of NPL 1, NPL 2, and NPL 3 provided. |
| Inoue et al. “Adaptive Automated Accompaniment System for Human Singing.” Transactions of Information Processing Society of Japan. Jan. 15, 1996: 31-38. vol. 37, No. 1. English abstract provided. Cited in NPL 1, NPL 2, and NPL 3. English translation of NPL 1, NPL 2, and NPL 3 provided. |
| Notice of Allowance issued in U.S. Appl. No. 15/597,675 dated Jul. 18, 2018. |
| Office Action issued in U.S. Appl. No. 15/597,675 dated Oct. 20, 2017. |
| Notice of Allowance issued in U.S. Appl. No. 15/597,675 dated Oct. 30, 2018. |
| Number | Date | Country | |
|---|---|---|---|
| 20170256246 A1 | Sep 2017 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2015/082514 | Nov 2015 | US |
| Child | 15598351 | US |
