The present invention relates to augmented stringed musical instruments, i.e., conventional acoustic stringed instruments enhanced with electrical sensors and actuators.
The internet of things revolution has impacted a variety of fields and industries, including acoustic stringed instruments. As a result new augmented instruments or prototypes were introduced. Prior art shows several examples of augmented stringed musical instruments and different applications (US 20140224099 A1; US20120007884; US2002/0005108A1; PCT/GB2000/000769; U.S. Pat. No. 6,320,113 B1; U.S. Pat. No. 8,389,835 B2).
The problem addressed relates to the need of players of acoustic stringed instruments to enhance the range of sound, timbre and vibration of their instrument in order for instance to replicate the resonance or other characteristics of another instrument or to recreate special sound or vibration effects, or to modify such effects during the act of playing. The existing technologies available to acoustic stringed instruments allow to create special effects only by acting on the physical feature of the instrument, i.e. the size, or the shape or the thickness of the sounding board, the material of the different instrument's parts, or the size of the strings, which is obviously an operation that can be done exclusively before the act of playing and which requires often a recrafting of the instrument.
The present invention provides an innovative solution to the described need. The invention describes a method and a process to synthesize a desired vibration or timbre or sound of another instrument (Target Instrument) in real time. The solution will make possible, for instance, to make vibrate a stringed instrument (Controlled Instrument) similarly to the timbre of another stringed instrument (Target Instrument), this last one having different features and a different timbre than the Controlled Instrument. The capability to control and modify such effects will be accomplished in real-time while the musician is actually playing the instrument.
The Controlled Instrument and the Target Instrument are stringed musical instruments that include a soundboard and/or a sound box, and a bridge. A measurement apparatus of the instruments vibration is placed under the bridge of the Controlled Instrument, and two actuators are attached respectively to the main resonating surface and a secondary, mechanically uncoupled tonal chamber. The signals driving the actuators are determined by a Control Algorithm. While the presence of a secondary actuator does not improve the generality of the system, its contribution may become fundamental in practice when using one of the empiric estimation techniques hereby proposed.
The Control Algorithm imitates the Target Instrument by feeding the actuators with a processed version of the Controlled Instrument's string and soundboard vibration detected by the measurement apparatus while the instrument is being played. The Cloning Procedure gives the parameters of the Control Algorithm when such an algorithm has to control the Controlled Instrument so it sounds similar to the Target Instrument. However, the Control Algorithm is not restricted to only impose the reproduction of a Target Instrument. The Control Algorithm can make the Control Instrument sound in other desired manners by choosing the Control Algorithm parameters.
The patent US2002/0005108A1 describes several sensors and actuators embedded in a musical instrument, which are capable to control the sound emission, among other features, via a DSP (Digital Signal Processing) module. The DSP unit implements the sound controller. The Control Algorithm of the present invention needs the sensor/DSP/actuator architecture described in the patent US2002/0005108A1. However, such a patent does not specify neither the Cloning Procedure, nor the Control Algorithm, which are instead the inventive steps of the present document. The patent PCT/GB2000/000769 has identical claims to the previous patent US2002/0005108A1 for what concerns the architecture sensor/controller unit/actuator. Analogously, the function of the processing unit is different from the Control Algorithm. The expired so patent U.S. Pat. No. 6,320,113 B1 is similar to the architecture sensor/controller unit/actuator of the patent US2002/0005108A1. It describes a system that provides sound control for an acoustic musical instrument comprising actuators, sensors, and closed-loop transfer functions. A sensor is configured to generate sensed signals based on vibrations from a structure. The sensed signals are input to a controller that processes output signals. The output signals are fed to an actuator that alters the sound of the acoustic instrument. The differences of the present invention compared to this patent are the same as the differences compared to patent US2002/0005108A1. In addition, the controller unit is posed outside the instrument as well as the location of the actuators is different from that of the os present invention. The U.S. Pat. No. 8,389,835 B2 has identical claims to the previous patent US2002/0005108A1 for what concerns the architecture sensor/controller unit/actuator. However, the controller unit is less advanced than patent US2002/0005108A1 in that it does not seem to be a DSP. The differences of the present invention compared to this patent are the same as the differences compared to patent US2002/0005108A1.
The method and process of the present invention is described by referring to the figures, which are just an exemplification of the preferred execution and do not limit the invention to just such forms of representation.
The Target Instrument is any instrument composed of a radiating body (i.e., a soundboard (101) and/or a sound box (102)), a bridge (103), and strings (104) (see
The Controlled Instrument is composed of the following components (see
The parameters of the Controlled Instrument are calibrated using a calibration setup that consists of the following items (see
In this section we characterise the mathematical modelling of the Controlled and Target Instruments, which is the essential preliminary step to specify the Control Algorithm in the next section.
Following a classic approach in control systems theory, the input/output relationships of the Controlled Instrument, and the Target Instrument, are detailed respectively in
The physical variables explicited on the signal branches are the following:
The frequency response functions corresponding to mechanical and acoustical parts of the Controlled Instrument and the estimation procedures for their parameters are the following:
H1(ω) (403) models the combined action of the response of the actuator a1 (204) and the mechanical impedance of the instrument body excited in the position of the actuator. It can be measured by feeding the actuator a1 with a test signal such as a Logarithmic Sinusoidal Sweep (LSS) or a Maximum Length Sequence (MLS), recording the output signal measured with sensor s1 (203) and using standard system estimation techniques such as, and not limited to, Least Squares Optimization.
With basic algebra manipulations, the overall matrix of FRFs G(ω) (502) of the Controlled Instrument system can be derived as
The n-th component of G(ω) (502), referred to as Gn(ω), models the FRF from the n-th string to the pressure sensor s2 (301).
The Control Algorithm is based on a mathematical model of the Controlled Instrument we described in the previous section, which is the acoustic system under control. Such an algorithm is representable as a pair of discrete-time Linear Time Invariant (LTI) systems that are specified by K1(ω) (404), K2(ω) (407), respectively, and that are implemented by a microprocessor. The algorithm's behaviour consists of two main parts: (i) the creation of the resonances that are not present among those naturally producible by the Controlled Instrument (i.e., those resonances that could not be produced without using a Control Algorithm); (ii) the cancellation of the resonances that are present among those produced by the Controlled Instrument.
Since there are many ways to choose K1(ω) (404), K2(ω) (407), one may have several instances of the Control Algorithm. Each instance has its own implementation complexity. Finding a set of parameters for an instance of the Control Algorithm can be seen as an optimization problem, where the goal is to minimize in the frequency domain the weighted squared error E (604) between the Controlled Instrument's frequency response G(ω) (401) and the desired frequency response D(ω) (602). The weighting function W(ω) 605 is arbitrarily chosen using e.g. a psycho-acoustic function that tries to give more importance to the frequency region that are most important for the human ear. In the following, the most general instance is specified (the Multi-Objective Control Algorithm), and then some special cases are provided.
Let D(ω) (602) be a desired FRF, namely a FRF that the Controlled Instrument has to exhibit so that it can produces the desired vibrations. The algorithm consists in specifying the control blocks K1(ω) (404), K2(ω) (407) so that the FRF of the Controlled Instrument G(ω) (401), specified in Eq. (4.1), is as much close as possible to the desired FRF D(ω) (602). Since the FRF is a matrix of complex numbers, we need to specify in which sense the two FRF are made equal. This is complicated by that we are dealing with complex numbers.
Let MD(ω) be the matrix whose entries are the modules of the entries of the matrix D(ω) (602), and let ΦD(ω) the matrix whose entries are given by the phases of the entries of the matrix D(ω) (602). Analogously, let MG(ω) be the matrix whose entries are the modules of the entries of the matrix G(ω) (401), and let ΦG(W) the matrix whose entries are given by the phases of the entries of the matrix G(ω) (401). Moreover, let ∥·∥ be any induced matrix norm, for example 1-norm, 2-norm, Frobenius-norm, oc-norm, max-norm, or min-norm, to mention some of the possibilities.
The most general way consists in choosing K1(ω) (404), K2(ω) (407) by a multi-objective optimisation so that the integral over the frequency domain of a weighted squared induced norm of the matrix difference among MD(ω) and MG(ω) is as small as possible, while the integral over the frequencies of a weighted squared induced norm of the matrix difference among ΦD(ω) and ΦG(ω) is as small as possible. Additionally, this optimisation has to ensure that the choice of K1(ω) (404), K2(ω) (407) give a FRF matrix G(ω) must be BIBO (Bounded-Input, Bounded-Output) stable. Otherwise, the resulting system would present self-sustained oscillations due to errors in feedback control, which are usually referred to as “Larsen effect” by musicians. The constraint can be satisfied by exploiting well-known results in control systems theory concerning poles and zero placements of FRF. Formally, we have the following multi-objective optimisation problem:
In this optimisation problem, the weighing matrixes WM(ω) and WΦ(ω) are chosen arbitrarily. For example, they can be a psychoacoustic weighting function that gives more importance they can give more importance to the frequencies important for the human hearing system, such as A-weighting, ITU-R 468 or similar functions. The decision variabler of the optimisation problem are K1(ω) (404), K2(ω) (407). In the problem, ωmax 229 is the maximal frequency of interest for the application, usually close to the human hearing frequency limit (e.g. 20,000/2π rad/s). Note that in the optimisation problem we have two objectives: the simultaneous minimisation of the module and the simultaneous minimisation of the phases. The solution of the optimisation problem can be obtained by any solution method for multi-objective optimisation. This should not be a problem, since the solution can be achieved off-line. Nevertheless, in the following, we present some other approaches that are of reduced computational complexity.
A computationally more affordable way to solve optimisation problem (4.2) is via a scalarizarion procedure that leads to a Pareto optimisation as follows: First, we define a secularised cost function weighted by the Pareto coefficient 0≤ρ≤1:
p(K1(ω),K2(ω))=∫0ω
Then, the Control Algorithm parameters are give by the solution to the following optimisation problem:
minK
subject to BIBO stability of G(ω). (4.6)
Note that in this optimisation problem, the choice of the coefficient ρ is left to the implementer. For example, one could even choose ρ=1 so to give no relevance to the phase minimisation. Alternatively, it can be done by constructing the standard Pareto trade-off curve and looking for the knee-point of the courve. Finally, observe that if optimisation problem (4.2) is convex in the decision variables K1(ω) (404), K2(ω) (407), then the optimal solution returned by (4.5) is identical to the one returned by (4.2). However, if problem (4.2) is non convex, then the solution of problem (4.5) is a feasible solution for problem (4.2) and in general is sub-optimal for problem (4.2).
The methods to determine the values of K1(ω) (404), K2(ω) (407) can be computational demanding. Here, we describe a sub-optimal method that is of easier implementation. This method is sub-optimal compared to the more general method given by optimisation problems (4.2) and (4.5).
Once a desired FRF D(ω) (602) is set, the Multi-Objective Sub-Optimal Control Algorithm consists in finding the control blocks K1(ω) (404), K2(ω) (407) that simultaneously minimise the squared error between the spectral magnitudes of the controlled and target FRFs, where the errors are defined component-wise:
E
n=∫0ω
where ωmax is the maximal frequency of interest for the application, as for the previous problems, and Wp(ω) is a psychoacoustic weighting function that gives more importance to the frequencies relevant for the human hearing system, such as A-weighting, ITU-R 468 or similar functions. The optimization is subjected to the constraint that the resulting system defined by G(ω) (401) must be BIBO (Bounded-Input, Bounded-Output) stable. The optimisation problem is
As for the previous section, the solution to this problem can via standard multi-objective optimisation methods. A computationally simple method of finding a feasible solution, which is optimal if the problem is convex, is by Pareto scalarization, where the solution to (4.8) is achieved by solving the following Pareto optimisation problem
Note that in this problem, the choice of the Pareto weighting coefficients ρi is left to the implementer. Alternatively, one can draw the usual Pareto trade-off curve and choose the ρi that give the knee-point.
Here we propose a simpler method to determine K1(ω) (404), K2(ω) (407). The method is heuristic in the sense that it is not analytically derived from the approach of the previous section, although it is inspired from it. Here, the determination of the values of K1(ω) (404), K2(ω) (407) consists of two steps. First, the parameters of the controller K2(ω) (407) are estimated exploiting that the actuator a2 (205) is placed in an independent tone chamber. In this way it is possible to reproduce, in the Controlled Instrument, the acoustic resonances that are given by D(ω) (602) but not in the Controlled Instrument itself when both the actuators are not active. In a formal way, the feed-forward FRFs of the Controlled Instrument when only actuator a2 (205) is active, is defined as:
{tilde over (G)}
2,n(ω)=[At(ω)K2(ω)+Ac(ω)]Hc,n(ω). (4.14)
In the first step, an optimization problem is posed to find only the controller K2(ω) (407) that minimize the weighted target error functions Et,n defined as
E
t,n=∫0ω
where the exponent βt>1 controls the weight given to the frequency points where the spectral magnitude |Dn(ω)| is larger than |{tilde over (G)}2,n(ω)|. In other words, the error Et,n 282 is subjected to an additional weight that gives more relevance to the frequency points where target response has resonances that are not present in the Controlled Instrument's response. At the same time, less effort is spent trying to suppress resonances that are present in the Controlled Instrument but not in the Target Instrument. Formally, the optimisation problem posed to determine K2(ω) (407) is
As for the previous section, the solution to this problem can via standard multi-objective optimisation methods. A computationally simple method of finding a feasible solution, which is optimal if the problem is convex, is by Pareto scalarization, where the solution to (4.16) is achieved by solving the following Pareto optimisation problem
Let denote by K2*(ω) the solution to this problem, namely K2*(ω) is the transfer function of the calibrated controller that drives the actuator a2 (205) computed as the result of the first step of the algorithm.
If we now use K2*(ω), the overall FRFs of the Controlled Instrument depends now only on K1(ω) (404), and results defined as
Then, we can determine K1(ω) (404) by formulating an optimisation problem that minimizes the overall resulting error:
E
r,n=∫0ω
where 0<βr<1 is the exponent that weights the effort towards the suppression of the unwanted resonances in the Controlled Instrument's response and, at the same time, simplify the task of designing a controller that maintains the constraint of BIBO stability. Formally, the optimisation problem posed to determine K1(ω) (404) is
As for the optimisation problem of the first step, the solution to this problem can be done via standard multi-objective optimisation methods. A computationally simple method of finding a feasible solution, which is optimal if the problem is convex, is by Pareto scalarization, where the solution to (4.24) is achieved by solving the following Pareto optimisation problem
Let denote by K1*(a)) the solution to this problem, namely K1*(a)) is the transfer function of the calibrated controller that drives the actuator a1 (204) computed as the result of the second step of the algorithm.
The Cloning Procedure provides the Control Algorithm with the parameters that regulate actuators of the Controlled Instrument when the Control Algorithm allows the reproduction of the vibrations of the Target Instrument. Otherwise, the Control Algorithm can have its parameters set so that the Controlled Instrument can reproduce any desired vibration. The Control Algorithm's behaviour for cloning a Target Instrument consists of two main parts: (i) the creation of the resonances that are not present among those naturally producible by the Controlled Instrument (i.e., those resonances that could not be produced without using a Control Algorithm), but that are present in the Target Instrument's timbre; (ii) the cancellation of the resonances that are present among those produced by the Controlled Instrument, but that are not present in the Target Instrument's timbre.
The objective of the Control Algorithm for cloning a Target Instrument is the imitation of the acoustic properties of a given target acoustic musical instrument by finding a proper set of parameters for the controllers K1(ω) (404), K2(ω) (407). A block diagram for the model of the Target Instrument is presented in
The Cloning Procedure is then finalised by imposing that D(ω) (602)=G*(ω) (502) and applying any Control Algorithm of the previous section.
Number | Date | Country | Kind |
---|---|---|---|
1530095-7 | Jun 2015 | SE | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/SE2016/050494 | 5/29/2016 | WO | 00 |