Sound mixing device

Abstract

A system for interactively controlling multiple parameters affecting an audio output, the system employing a controller with a visual display (e.g., a cathode ray tube) for displaying an icon that is a visual representation of an input sound signal in a multidimensional space and an input device (e.g., a mouse) used to control the location of the icon on the display. The controller generates a multiple parameter control signal that is based upon the location of the icon and is used by a sound signal processing circuit to control multiple parameters affecting an audio output. The icons are images of the sources of sound input signals. The sound signal processing circuit has M times N controllable amplifiers arranged in an M by N matrix in which each of the M inputs is distributed to N controllable amplifiers and the outputs of M controllable amplifiers are combined to provide each of the N outputs. Each controllable amplifier has a unique combination of an input signal and an output signal and receives a unique, continuously variable gain control signal.

Description

BACKGROUND OF THE INVENTION

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever.

The present invention relates to interactively controlling multiple parameters affecting an audio output.

It is known to mix sound input signals into sound output signals using a sound mixing board which includes separate manually activated controls for each sound input signal. The sound input signals may include signals from a plurality of musical instruments and voices as well as sound effects. The controls may include pan (i.e., position) and gain (i.e., amplitude) controls. The sound output signals may be provided to speakers or any recording device.

Sound input signals are mixed to control the position of the signal relative to a listener as well as to control the relative level of an input signal with respect to other input signals. The position of an input signal is controlled by adjusting the amount of the input signal which is provided to each output channel. E.g., for a stereo mix environment (i.e., an environment which includes a left output channel and a right output channel) an input signal which is located directly to the right of the listener is provided to the right output channel while the left output channel is attenuated. The relative level/position of the input signal is controlled by adjusting the amplitude of the input signal which is provided to each output channel.

It is also known to mix sound input signals using a computer system which imitates the controls of a sound mixing board. In such a computer system, the controls are displayed on the screen of the computer as they might appear on the mixing board. Steinberg/Jones, Northridge, Calif. produces such a computer system under the trade designation MIMIX.

It is also known to use external processors such as compressors, limiters, equalizers, reverb units, delay units, to achieve certain effects, e.g., in synthesizing sound or in editing of audio and audiovisual works.

SUMMARY OF THE INVENTION

Our invention features in general a system for controlling multiple parameters affecting an audio output. The system employs a controller with a visual display (e.g., a cathode ray tube) for displaying an icon that is a visual representation of an input sound signal in a multidimensional space. An input device (e.g., a mouse) is used to control the location of the icon on the display. The controller generates a multiple parameter control signal that is based upon the location of the icon and is used by a sound signal processing circuit to control multiple parameters affecting an audio output. With the invention, the states of multiple parameters can be simultaneously controlled and usefully displayed, permitting precision and flexibility in real time dynamic sound control, and permitting the sound engineer to achieve complex mixes with a large number of parameters.

In preferred embodiments a plurality of icons can be used to control a plurality of input sound signals. The parameters could include gains for respective outputs in a multiple output channel environment (e.g., a stereo mix or Dolby system), one or more reverb parameters, equalization parameters, filtering parameters, and compression parameters. The multidimensional space could represent the physical location of a listener in a stereo environment relative to the sources of sound, and the display could include hard left and hard right regions and a nonlinear fadeout region. The units of gain associated with the display could be rescalable. The multidimensional space could alternatively represent a scene of an audiovisual work; locations in the space associated with locations in the scene would have associated audio parameter sets, e.g., stored in a look-up table. State information as to positions of the icons at any instantaneous time and changes in position with time can be stored for later playback, editing and synthesis.

In another aspect, our invention features, in general, a system employing icons that are images of the sources of input sound signals on a visual display for interactively controlling multiple parameters affecting an audio output. This facilitates identification of the source of the input sound signal by the sound engineer, something that is particularly advantageous when there are a large number of input sound signals to control and monitor.

In another aspect, our invention features, in general, a system for interactively controlling the modification of M input sound signals to provide N output sound signals. The system includes a sound signal processing circuit having M times N controllable amplifiers arranged in an M by N matrix in which each of the M inputs is distributed to N controllable amplifiers and the outputs of M controllable amplifiers are combined to provide each of the N outputs. Each controllable amplifier has a unique set of an input signal and an output signal and receives a unique, continuously variable gain control signal. Such a system provides great flexibility in achieving complex sound mixes and complex real time changes in sound mixes.

In preferred embodiments, each controllable amplifier includes a digital-to-analog converter (DAC) that receives a multibit gain control signal and a voltage controlled amplifier that receives the control voltage output of the DAC; the DACs have slew limiting of the control voltage to eliminate control voltage artifacts in the output signals; and a decoder circuit receives gain control signals for all controllable amplifiers with addresses identifying which gain control signals apply to which amplifiers.

Other advantages and features of the invention will be apparent from the following description in a preferred embodiment thereof and from the claims.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The drawings are first briefly described.

DRAWINGS

FIG. 1 shows a diagrammatic-block diagram of a sound mixing system according to the invention.

FIG. 2 shows a schematic-block diagram of a decoder circuit of the FIG. 1 sound mixing system.

FIG. 3 shows a schematic-block diagram of an arrangement of controllable amplifiers of the FIG. 1 sound mixing system.

FIG. 4 shows a schematic-block diagram of line level trim circuits of the FIG. 1 sound mixing system.

FIGS. 5A and 5B show examples of displays for the FIG. 1 mixing system.

FIGS. 6A to 6C show a flow chart of a partial control module of the FIG. 1 sound mixing system, which is responsible for the real time mixing.

FIG. 7 is a block diagram of an alternative sound signal processing matrix configuration for the FIG. 1 system.

STRUCTURE

Referring to FIG. 1, in real time, interactive sound mixing system 10, computer system 12 communicates with mixing circuit 14 via 16-bit data bus 16 (only 12 bits are used in this embodiment) and 8-bit address and control bus 17 (6 bits for address and 2 bits for control), as well as with display 18 (e.g., a color cathode ray tube (CRT)) via display interconnect 20. Computer system 12 includes computer 30 (e.g., an Atari 1040ST), which may be controlled via either keyboard 32 or mouse 34, as well as storage media 36 (e.g., a disk drive), RGB (red, green, blue) interface 38 and control module 40, which is software loaded on computer 30. (A computer listing for the software is incorporated herein as the Appendix.) Computer 30 utilizes a GEM operating system. The software for control module 40 is implemented in an interpretative and compilable GFA Basic (3.0 Version) available from Antic Software, San Francisco, Calif. Mouse 34 includes two activation buttons, right button 35a and left button 35b. Computer system 12 acts as an interactive controller that generates multiple parameter control signals provided to mixing circuit 14, which acts as a sound signal processing circuit that modifies input sound signals to provide output sound signals based upon the multiple parameter control signal.

Mixing circuit 14 includes voltage-controlled amplifier (VCA) cross-point matrix 44, a programmable-gain matrix, which receives sound input signals (e.g., from analog sound sources such as tapes, compact disks, synthesizers, musical instruments and voices) and provides sound output signals to right and left line level trim circuits 46, 48. Trim circuits 46, 48 provide sound output signals to audio amplifier 50, which provides audio output signals to left and right speakers 52, 54 or recording devices.

Cross-point matrix 44 includes address decoder circuit 60, which receives address and control information over address and control bus 17, as well as a plurality of controllable amplifiers 62, which receive amplitude control information over data bus 16 as well as sound input signals via input channels 64, 66, 68, 70. Controllable amplifiers 62 provide sound output signals via output channels 72, 74. Each input channel 64, 66, 68, 70 provides its respective input signal to a pair of controllable amplifiers 62, which each provide an output signal on a respective output channel 72, 74. Accordingly, the number of controllable amplifiers 62 in the VCA cross-point matrix is defined by the number of input channels times the number of output channels. (I.e., 2 output channels.times.4 input channels=8 amplifiers.)

Referring to FIG. 2, decoder circuit 60 includes an arrangement of three 3:8 decoders. More specifically, enable decoder 80 (available under the trade designation 74LS138) receives three enable bits via address and control bus 17 and provides enable signals to decoders 82, 84 (available under the trade designation 74LS138), which also receive three address bits and DATA CLK (one of the control bits) via bus 17. Decoders 82, 84 provide a plurality of active low enable signals LATCH DAC 0/ to LATCH DAC 9/.

Referring to FIG. 3, each controllable amplifier 62 includes register circuit 90, which receives twelve bits of data via data bus 16, digital-to-analog converter (DAC) 92 (available from Analog Devices under the trade designation AD565), which receives registered data from register circuit 90, and voltage controlled amplifier (VCA) 94 (available from P.M.I., Inc. under the trade designation 2120), which receives a V DAC control signal from DAC 92 as well as a sound input signal via an input channel. VCAs 94 provide a logarithmic output gain for a linear voltage control input signal. A remote ground signal (VGND) is also provided to the gain adjust input terminal of each VCA 94; thus, the gain of all the VCAs is adjusted equally.

Still referring to FIG. 3, the output terminals of four controllable amplifiers 62 are tied together to provide a summation of output currents and provide a signal to operational amplifier OPI which provides a first sound output signal (e.g., the right sound output signal). The output terminals of four other controllable amplifiers 62 are tied together to provide a second sound output signal (e.g. the left sound output signal).

Referring to FIG. 4, line level trim circuits 46, 48 each include a DAC (not shown), which receives a digital signal from data bus 16 and provides an input control signal (VDAC 8 or VDAC 9) to a respective VCA 194. Each VCA also receives a sound output signal from one of the chains of controllable amplifiers. Each level trim circuit VCA 194 provides a sound output to the respective left or right channel of audio amplifier 50. Each channel of audio amplifier 50 includes a pair of cascaded operational amplifiers.

Referring to FIGS. 5A and 5B, sound mixing system 10 uses display 18 to provide video display 100 which displays sound input signals in a two-dimensional sound output channel space. FIG. 5A shows an empty video display (i.e., the sound output channel space), while FIG. 5B shows a video display 100 which displays the mix of four input signals. The video display 100 of an output signal space for a stereo mix environment includes a mix portion 102 (the output channel space) which displays angle of displacement from center as .theta. and radial distance, which is related to relative sound amplitude, in decibels (dB).

Mix portion 102 (where icons 106 are moved to actively mix an input signal) includes elliptical boundary 103 which delineates main mix portion 104 (where the majority of the mixing is accomplished) and nonlinear fadeout portion 107 (where icons are moved to gradually fade an input sound signal to mute, i.e., -90 dB). At the bottom of main mix portion 104 are hard left portion 104a and hard right portion 104b. Main mix portion 102 also includes amplitude indicators 109, which generally indicate attenuation relative to listener origin 111, and position indicators 112, which generally indicate angular position relative to listener origin 111. Amplitude indicators 109 could be rescaled (e.g, to show 3 dB increments instead of 5 dB increments), if desired. The position of mouse 34 is displayed by cursor 105, and input channels are displayed as icons 106, which are visual representations of the input sound signal sources. Icons 106 may have the appearance of the element providing the respective sound input signal. (E.g., a guitar which provides a sound input signal would have a guitar shaped icon.)

Video display 100 also includes information portion 99, which provides information about input channels which are displayed on video display 100. Information portion 99 includes active icon portion 108, which displays the "active" icon as well as mix information (e.g., left and right gain and angular displacement relating to that icon), output channel portion 110, which provides information regarding the status of the output channels, events portion 114, which displays different stored states (i.e., a particular configuration of icons) of a particular mix, groups (GRPS) portion 115, which displays whether the active icon is part of a group of icons which are configured to move together, and SMPTE (Society of Motion Picture and Television Engineers) time-code readout 117, which provides information relating to time reference coding of a particular mix.

OPERATION

Referring to FIGS. 1, 5A and 5B, sound mixing system 10 is used to mix one or more sound input signals into a plurality of sound output signals by displaying the sound input signals in a multidimensional sound output signal space. (FIGS. 5A and 5B show the output signal space for a stereo mix environment.) Mouse 34 and keyboard 32 are used to move icons representing sound input signals around the screen display representing the space of the sound output signals. As the icons are moved, the mix of the input signals changes.

More specifically, when cursor 105 is positioned over an icon 106, right button 35a of mouse 34 is depressed, and that icon 106 becomes the active icon (i.e., the icon 106 which represents the input signal which is being mixed). Left button 35b is depressed, and the active icon is moved around display 100 by moving mouse 34 according to known techniques. If both right and left buttons are depressed over the active icon, then the icon may be moved along or around a constant radius. For example, in the stereo mix environment, as an icon moves around the central radius, the relative position of the sound input signal (i.e., the amount of the sound input signal being supplied to the right output channel and the amount of the sound input signal being supplied to the left output channel) changes. Also, as the icon moves along the radius, the strength (i.e., the gain) of the sound input signal changes. Thus, by simply moving an icon to a desired mix position, the mix of the input sound signal is accomplished. After the mix of the input signal has been completed, the position of the icon on the display may be stored for future use with disk drive 36.

Referring to FIGS. 6A-6C, a flow chart of control module 40 of real time mix system 10 is shown. Line references in the flow chart are to the lines of the listing in the Appendix hereto. More specifically, at initialization step 120, normalized coefficients are precomputed and ranges of variables are initialized. The left and right gain coefficients are computed using the following equations:

left.sub.-- dB.sub.-- coef=20.times.LOG.sub.10 (MAX (SIN (90-i)/2),0.001) right.sub.-- dB.sub.-- coef=20.times.LOG.sub.10 (MAX (COS (90-i)/2),0.001) where: MAX=a function which provides the greater of the two numbers in the parenthetical, and i=an angular position relative to a listener. The coefficients of dB v. angular position (relative to listener) are stored in a two-dimensional look-up table, thus, allowing module 40 to function faster because these coefficients need not be generated for every mix action. At real time mix step 122, control module 40 has finished its initialization and starts a mix of input sound signals to stereo output signals.

At setup graphic buffers step 124, graphic buffers are provided so that a previous mix (such as a mix which is retrieved from disk drive 36) may be stored while a present mix is being generated. Also, in step 124, the resolution of display 18 is configured to allow computer 30 to function in an environment having 320 locations in the x direction by 200 locations in the y direction. Also, computer 30 causes 320 by 200 pixels to be displayed on display 18 and mouse 34 to be initialized for this configuration. By initializing mouse 34 for this configuration, a location on display 18 corresponds to a functional location of computer 30. Thus, computer 30 may be controlled simply by referring to cursor 105 which is controlled by mouse 34 and displayed on display 18.

At icon step 126, control module 40 determines if any icons are present on display 100. If icons are present, then there is a preexisting mix of information; if no icons are present, then there is no mix present and, thus, the mix need not be updated at this time.

If there is an icon present, then at step 128, the graphics are updated. More specifically, the x and y coordinates of the icon are used to update all the values displayed on screen 100 as well as to update each controllable amplifier 62.

At step 130, mouse 34 and keyboard 32 are sampled to determine if any action has been taken. At step 132, the results of the sample are analyzed to determine if any special function keys have been pressed. If so, then step 134, which includes subroutines for the special functions, is used to instruct computer 30 how to react to a particular function key. Functions include: increase gain, decrease gain, pan left, pan right, etc.

At step 136, video display 100 is scanned to determine if cursor 105, which is controlled by mouse 34, is in mix portion 102. If cursor 105 is not in the mix area, then at step 138, the mouse may be used to control special functions of control module 40 (e.g., the mouse may be used to control a pull-down menu). The pull-down menu is an alternative to the special function keys used in step 132.

At step 140, control module 40 checks to see if right button 35a of mouse 34 has been activated. If so then, at step 142, control module 40 checks to see if any icon is targeted by mouse 34 (i.e., to see if the cursor, which is controlled by mouse 34, is positioned over an icon). If so then, at step 144, control module 40 initializes a new icon and updates the graphics of active icon portion 108 of display 100 to indicate the new active icon. If no new icon is activated, then control module 40 continues to operate with a prior icon.

At step 146, a moving average is set up for the moving mouse. More specifically, seven previous locations of the mouse are stored and weighted together in a conventional manner. Because module 40 is functioning so quickly, the seven previous locations are generated at a speed which is virtually transparent to the user and listener. Each time the mouse is updated (i.e., moved) then the standard traveling average is automatically updated. Thus, movements of the mouse provide a substantially continuously flowing output signal. The weighing function may be adjusted in a conventional manner to provide a faster or slower ramping movement of mouse 34, and thus of the mix.

At step 148, the mouse position is analyzed to determine if DACs 92 need to be updated (e.g., due to movement of mouse 34 or use of special functions). If so then, at step 150, control module 40 updates the coordinates displayed by video display 100 and provided to controllable amplifiers 62; mouse cursor 105 is also hidden so that mouse 34 moves an icon 106 rather than cursor 105.

At step 152, control module 40 determines whether mouse 34 is activated by toggling one or both of its buttons 35a, 35b. If mouse 34 is activated, then at step 154, the x, y coordinates of mouse 34 are sampled, a weighted path for mouse 34 is computed, and a limit is imposed to prevent the icon from moving beyond the mix zone.

At step 156, gain information (r) and pan information (.theta.) are computed relative to listener origin 111 by converting standard x, y location values to polar coordinates relative to the listener origin. The listener origin may be different from the computer screen origin. E.g., in the stereo environment shown in FIGS. 5A and 5B, the listener origin is at the center, bottom screen.

At step 158, left dB and right dB values are computed based upon the r and .theta. values. In general, the r value is first used to calculate a value for a "bels" variable; if the icon is within region 104, r is simply multiplied times a scaling factor "d-scale", which has a value of 0.1875; if the icon is outside of region 104 in region 107, bels is calculated by first multiplying r times d-scale and then rescaling the value so obtained in a manner that provides realistic fadeout of the signal, the rescaling being a function of the value of bels and .theta.. Following the computation of bels, it is determined if the icon is in an angle of acceptance 159 (see FIG. 5B). If not, left dB and right dB values are now computed using the coefficients stored in the look-up table which was generated at step 120. More specifically,

dB.sub.-- left=MAX(bels+dB.sub.-- coef(a+90,0),-90) dB.sub.-- right=MAX(bels+dB.sub. coef(a+90,1),-90) where: dB.sub.-- left=the variable left dB component for the active icon, dB.sub.-- right=the variable right dB component for the active icon, MAX=a function which provides the greater of the two numbers in parenthetical, a=angular location.

If the icon is within angle of acceptance 159, then mono output signals (left and right gain equal to -3 dB below the computed gain) are provided. More specifically, if the icon is located in the angle of acceptance, then:

dB.sub.-- left=MAX(bels-3,-90), and dB.sub.-- right=MAX(bels-3,-90), where MAX=a function which provides the greater of the two numbers in parenthetical. Thus, when an icon is in the angle of acceptance, dB left and dB right are equal, each being 3 dB less than the bels radius value.

At step 160, the left and right dB values, which were computed in step 158, are displayed.

At step 166, mouse 34 is checked to see if it is moving by checking the x, y location values. If it is, then, at step 163, a final check is performed to determine if cross-point matrix 44 should be enabled. If matrix 44 is enabled then, at step 164, a six-bit address is generated; the address represents controllable amplifiers 62 which are assigned to active channels. This address is provided to decoder circuit 60. Decoder circuit 60 then provides enable signals to the DACs associated with the appropriate controllable amplifiers 62.

At step 166, control module 40 checks to confirm that the addressed controllable amplifiers 62 correspond to active channels. If the channels are inactive then at step 168, control module 40 instructs computer 30 to provide a mute signal to the addressed controllable amplifiers 62.

If the channels are active, then, at step 170, a 12-bit data word, which represents right gain, is generated for the right output VCA 194, and a 12-bit data word, which represents left gain, is generated for the left output 194. A unity offset is included in these computations. The unity offset compensates for the offset of individual controllable amplifiers 62. More specifically,

right.sub.-- gain=(offset1-dB.sub.-- right).times.20, and left.sub.-- gain=(offset2-dB.sub.-- left).times.20, where offset1=the offset of the right controllable amplifier, and offset2=the offset of the left controllable amplifier. Because VCAs 94 of controllable amplifiers 62 provide a logarithmic output level for a linear input, the left and right input gain signals may be linear.

At step 172, these data words are provided to controllable amplifiers 62. More specifically, DAC 92 of each activated controllable amplifier 62 receives this data word via register circuit 90 and provides a voltage (V DAC) to a respective VCA 94. VCA 94 receives the voltage which is linear and provides an output signal which is the appropriate dB level. DACs 92 provide slew limiting of the control voltage to eliminate control voltage artifacts in the signal output.

At step 174, the mouse coordinates of the active icon are updated, and, at step 176, the entire mix is updated. At step 178, the x and y coordinates are updated and the r and .theta. values are stored for the active icon. At step 180, the keyboard is sampled to determine if there was any keyboard activity and at step 182 the mouse is scanned to determine if a constant R or .theta. mode is chosen. If so then, at step 184, the data is computed to provide the icon along a constant R path or along a constant .theta. radius.

Control module then returns to step 152, and if mouse 34 is activated, repeats the mix for another icon. If mouse 34 is not activated then, at step 186, R and .theta. are computed for the mouse location and, at step 188, the keyboard is scanned to determine if any special commands were generated. Special commands may include whether to store the mix and prepare to end the present mix session. If not, then control module 40 returns to sample mouse 34 at step 130. If so, then the special commands are processed at step 190. Control module 40 then checks to see if the ESC key was pressed at step 192. If not, then control module 40 again determines whether any icons are present at step 126. If so, then control module 40 terminates the mix at step 194.

OTHER EMBODIMENTS

Other embodiments of the invention are within the scope of the following claims.

For example, system 10 may be used to mix input signals into greater than two output channels, e.g., a quadraphonic mix environment or a Dolby surround sound mix environment (having left front, center front, right front, and rear outputs), and the visual display could be in three or more dimensions.

Also, the icons could change in appearance (e.g., color, size, brightness) to indicate a change in parameter value (e.g., compression, equalization, grouping).

Additionally, regions may be defined in the multidimensional space, such that movement of an icon to that location will cause automatic generation of a control signal affecting any number of parameters.

This mapping ability of the invention can, e.g., be used in mixing voice and music for film or television. When mixing for film or television, often music is mixed with the narrative to enhance the presentation of the work. The music is often mixed somewhat lower than the narrative when narrative is present and is brought to near full level when there is no narrative, to fill in the gap. It is often necessary to limit the dynamic ranges of the narrative and the music to achieve better intelligibility. The narrative often has compression and limiting employed to contain the dynamic peaks of the announcer while the music is heavily compressed, so that no offending peaks interfere with the voice over and to keep the music from "dropping out" or disappearing in the mix. This compression of the music is made even more necessary if the music has a great dynamic range. When the music level is brought up to full level, in a case where there is no narrative, the excessive compression used for mixing under will be heard, often resulting in music without dynamics and sometimes audible "pumping" from the compressor's inability to deal with the complex waveform. The engineer typically selects a compression ratio that achieves punch when the music is full and contains the music sufficiently when it is mixed under. Unless the engineer has special equipment at his disposal, the compression ratio parameter is not changed during a typical mix because it is too complicated to change that parameter or many parameters dynamically. Tracking a change in the mix of voice and music with a change in a parameter, such as compression ratio, is very difficult, especially if many changes are required.

With the interactive control apparatus according to the invention, the change in compression slope can be mapped into the output space so that simply moving the icon to change gain also outputs control data to an external, dynamically controlled compressor that has its compression slope altered to fit the requirements for the music being full level or being mixed under to accommodate the narrative. In such a case, the compression slope can be adjusted to precisely provide the optimum compression for the music under and music full periods.

Another example for film mixing is the use of mapping templates for each scene of the movie. A template would consist of a 2-D or 3-D "floor plan" of the scene, setup by the engineer. Different areas would be outputted to dynamically control external audio processing devices, e.g., compressors, limiters, equalizers, reverb and delay units. If an icon is placed into any of these regions so mapped, the coordinates of the icon position are used to address data in a look-up table or derived from a function, and the data are outputted dynamically to the external device processing the audio channel represented by the icon. E.g., if the actor were to walk behind an object, the parameters for reverb, early reflection, and equalization could be altered to achieve the effect of someone talking from behind the object. These data outputted are in addition to the gain change to one or more output channels, e.g., based upon the position of the actor. The multiple parameter control data would also be layered, so that an icon placed in a region so mapped, would effect the change of one or more external devices associated with that channel or any other channel. Movement of one sound source could also be programmed to affect another sound source; e.g., if one sound source (associated with the active icon) moves in front of another sound source, the second source could be automatically attenuated even though its icon is not active. Different layers could also be selected or deactivated at will. An extra output bus of the gain matrix could be the source of the signal to the external processing, allowing one or more channels to contribute to the signal sent to an external device, e.g., a reverb unit.

The overall mapping concept could involve primary gain settings to the mix, secondary gain settings (dependent on or independent of the primary gain setting) to feed external processing, and control information for the dynamic alteration of external processing parameters (reverb decay, delay time, compression slope, limiter attack time, limiter release time, etc.).

The return from the external processing device could re-enter the gain matrix on another previously unused or additional input channel and also be represented by an icon. As shown in FIG. 7, two output busses 202, 204 from matrix 206 feed signals into two independent audio effects processors 208, 210 that are dynamically reprogrammable via MIDI control line 212.

Output busses 224a, 224b are for the main mix and are used to feed the recording device or listening apparatus. Busses 226a, 226b are the effect busses used to provide primary feeding back into processors 208, 210. Module 220 allows monitoring the output of mix busses and effect busses to confirm proper operation.

Due to the universal nature of the variable gain matrix, all input channels can contribute to a main stereo mix while one or more dependent or independent submixes can be outputted to the external effects (reverb, delay, pitch . . . ), as well as monitoring busses (222a, 222b). Dependent means software simulation of postfade, and independent means software simulation of prefade. Prefade and postfade are terms used on conventional consoles to indicate if a submix is after the main channel fader (dependent) or before the main channel fader (independent). The stereo returns from the external effect devices are re-entered into the variable gain matrix, and can therefore contribute to the main mix or one or more submixes. The software will disallow or mute the various nodes that might contribute to feedback.

With an M.times.N variable gain matrix employed, main channel signals as well as effect returns can be uniquely controlled from the interactive controller. The outputs of the matrix could feed recording devices, effect processors, monitoring busses. For instance two busses 222a, 222b could be dedicated as a monitoring buss. Because the buss is easily reconfigurable via software, this monitoring buss could be dynamically reconfigured for stereo "in-place" solo, normal stereo monitoring, group solo, mute in monitor but not in main mix, prefade (independent) listen, etc.

The variable gain matrix in a large enough configuration functions as a universal, dynamically reprogrammable "console" with a graphics front end. Under software control, configurations that must be "hardwired" into a current console, can be simulated via manipulation of the various gains at various matrix nodes.

The system may also be used for theater playback to play back special effects, in recording studios for the main mix or for special mix effects, in television post production for the main mix or for easy repeatable automated stereo mixing to timecode, in live performances to allow the mixing to be remote from the amplifiers (e.g., by using an optical link for busses 16, 17), in pavilions to allow live mixing to the audience from one or more tracks, in radio production rooms where space and versatility are a factor, in mixing sound effects for simulators (e.g., aircraft, space, submarine) or in satellite mix-minus setups to allow easy recall of complicated mixes.

Besides interactive input devices that permit direct interaction with the sound engineer, other sources of controlling information can be used to change the position of icons and appearance, e.g., a completely independent computer drive console, a source of MIDI data, a MIDI sequencer, a source of computer generated data, and an external audio or visual editing system. ##SPC1##

Claims

1. An apparatus for controlling multiple parameters effecting an audio mix output, said apparatus comprising

a controller including a visual display for displaying a visual representation of an input sound signal in a multidimensional space and generating a multiple parameter control signal controlling multiple parameters effecting said audio mix output, each dimension of said multidimensional space separately controlling a mixing parameter or mixing parameters, said control signal being based upon the state of said representation in said multidimensional space, at least two said dimensions being the location of said representation on said visual display, a change in location of said representation on said display causing change in two or more said mixing parameters,

an input device connected to said controller to control the state of said representation in said multidimensional space so as to affect said audio mix output, and

a sound signal processing circuit having a sound signal input for receiving a sound input signal, a sound signal output for providing a sound output signal, and a control signal input for receiving said multiple parameter control signal, said circuit modifying said sound input signal based upon said multiple parameter control signal to provide said sound output signal,

wherein said multidimensional space represents a multiple channel output mix environment, said multiple parameters include gains associated with each said output, and said sound signal processing circuit provides a plurality of output signals,

wherein said visual representation is displayed with respect to a listener origin, and the location of said representation is characterized on said display in polar coordinates including a radius component and an angular displacement component.

2. The apparatus of claim 1 wherein said controller is an interactive controller, and said input device is an interactive input device.

3. The apparatus of claim 1 wherein said display displays a plurality of visual representations of respective input signals, and said sound signal processing circuit has a plurality of sound signal inputs.

4. The apparatus of claim 3 wherein said multidimensional space represents a multiple channel output mix environment with N output channels, said multiple parameters include a gain associated with each unique combination of an input sound signal and an output sound signal, and said sound signal processing circuit provides N output signals and includes M times N controllable amplifiers, where M is the number of input signals, the outputs of said amplifiers that are associated with the same output channel being added to each other to provide said N output signals, each said controllable amplifier being connected to receive a said input sound signal and a unique gain control signal, said multiple parameter control signal including said gain control signals.

5. The apparatus of claim 4 wherein

each said controllable amplifier includes a digital-to-analog converter and a voltage controlled amplifier,

said gain signal includes a multibit digital word that is provided to said digital-to-analog converter,

said digital-to-analog converter provides a control voltage to said voltage controlled amplifier, and

said voltage controlled amplifier controls said amplitude of said output signal based upon said control voltage.

6. The apparatus of claim 5 wherein said digital-to-analog converters provide slew limiting of said control voltage to eliminate control voltage artifacts in the sound output signal.

7. The apparatus of claim 4 wherein said controller provides address and control information to said sound signal processing circuit, and

said address and control information controls gain updating of particular controllable amplifiers.

8. The apparatus of claim 1 wherein said multiple parameters include one or more reverb parameters, and said sound signal processing circuit includes a reverb unit that is controlled by said multiple parameter control signal.

9. The apparatus of claim 1 wherein said multiple parameters include one or more equalization parameters, and said sound signal processing circuit includes an equalizer that is controlled by said multiple parameter control signal.

10. The apparatus of claim 1 wherein said multiple parameters include one or more filtering parameters, and said sound signal processing circuit includes a filter that is controlled by said multiple parameter control signal.

11. The apparatus of claim 1 wherein said multiple parameters include one or more compression parameters, and said sound signal processing circuit includes a dynamics processor that is controlled by said multiple parameter control signal.

12. The apparatus of claim 1 wherein said multidimensional space represents a scene of an audiovisual work, and locations in said space associated with locations in said scene have associated audio parameter sets.

13. The apparatus of claim 12 wherein said audio parameter sets include values for compression, equalization, reverb or delay associated with locations and physical objects in said scene, and said sound processing circuit includes compression, equalization, reverb or delay circuitry.

14. The apparatus of claim 13 wherein said multiple parameters include multidimensional gain values associated with a location of a source of a sound input signal in said scene.

15. The apparatus of claim 1 further comprising a MIDI control line that is connected between said controller and said sound signal processing circuit and transmits said multiple parameter control signal.

16. The apparatus of claim 1 wherein said visual representation is displayed on a display which simulates three dimensions with respect to a listener origin, and the location of said representation is characterized on said display in polar coordinates including a radius component and two angular displacement components.

17. The apparatus of claim 1 wherein said radius component corresponds to a gain for a mix of said input signal, and said angular displacement component corresponds to relative amounts of said input signal in said output signals.

18. The apparatus of claim 17 wherein units of gain associated with said radius component are rescalable.

19. The apparatus of claim 3 wherein said visual representations are icons that are images of sources of the input sound signals.

20. The apparatus of claim 19 wherein said sources are musical instruments, and said icons are images of said musical instruments.

21. The apparatus of claim 1 wherein said multidimensional space represents a stereo mix environment, said multiple parameters include left and right gain, and said sound signal processing circuit provides a left sound output signal and a right sound output signal,

wherein said multiple parameters include left front, center front, right front, and rear gain, and said sound signal processing circuit provides left front, center front, right front, and rear output signals.

22. The apparatus of claim 1 wherein said sound signal processing circuit includes

a controllable amplifier for each said output signal,

each said controllable amplifier being connected to receive said input sound signal and a gain control signal to provide a respective amplified sound input signal as said sound output signal, said multiple parameter control signal including said gain control signal.

23. The apparatus of claim 22 wherein

each said controllable amplifier includes a digital-to-analog converter and a voltage controlled amplifier,

said gain signal includes a multibit digital word that is provided to said digital-to-analog converter,

said digital-to-analog converter provides a control voltage to said voltage controlled amplifier, and

said voltage controlled amplifier controls said amplitude of said output signal based upon said control voltage.

24. The apparatus of claim 23 wherein said digital-to-analog converters provide slew limiting of said control voltage to eliminate control voltage artifacts in the sound output signal.

25. The apparatus of claim 22 wherein said controller provides address and control information to said sound signal processing circuit, and

said address and control information controls gain updating of particular controllable amplifiers.

26. The apparatus of claim 1 wherein said display has one or more mappable regions defined in said multidimensional space such that movement of a representation to a said region will cause automatic generation of a control signal affecting one or more parameters.

27. The apparatus of claim 26 wherein each said mappable region has an associated control signal including an audio parameter set.

28. The apparatus of claim 1 wherein said display has a fadeout region, and movement of a representation within the fadeout region in a predetermined direction results in accelerated fadeout of the output signal.

29. The apparatus of claim 1 wherein said display has amplitude indicators related to radial position relative to listener origin and angular position indicators related to angular position relative to listener origin.

30. The apparatus of claim 1 wherein said display includes a mix portion in which said visual representations are displayed and an information portion that displays values of left and right gain and angular position for said visual representation.

31. The apparatus of claim 1 wherein said controller includes means for storing state information as to the positions of said visual representations on said display, and means for recalling said state information to control the positions of said visual representations.

32. The apparatus of claim 31 wherein said state information is static state information indicating the instantaneous positions of said visual representations.

33. The apparatus of claim 31 wherein said state information is time dependent state information indicating the changes with time of the positions of said visual representations on said display.

34. The apparatus of claim 31 further comprising means to edit stored state information.

35. The apparatus of claim 33 wherein said interactive controller includes means for creating time dependent state information from static state information by determining gradual transitions between specified states in said static state information.

36. An apparatus for controlling multiple parameters effecting an audio mix output, said apparatus comprising

a controller including a visual display for displaying a visual representation of an input sound signal in a multidimensional space and generating a multiple parameter control signal controlling multiple parameters effecting said audio mix output, each dimension of said multidimensional space separately controlling a mixing parameter or mixing parameters, said control signal being based upon the state of said representation in said multidimensional space, at least two said dimensions being the location of said representation on said visual display, a change in location of said representation on said display causing a change in two or more said mixing parameters,

an input device connected to said controller to control the state of said representation in said multidimensional space so as to affect said audio mix output, and

a sound signal processing circuit having a sound signal input for receiving a sound input signal, a sound signal output for providing a sound output signal, and a control signal input for receiving said multiple parameter control signal, said circuit modifying said sound input signal based upon said multiple parameter control signal to provide said sound output signal,

wherein said display has a fadeout region, and movement of a representation within the fadeout region in a predetermined direction results in accelerated fadeout of the output signal.

37. The apparatus of claim 36 wherein said multidimensional space represents a multiple channel output mix environment, said multiple parameters include gains associated with each said output, and said sound signal processing circuit provides a plurality of output signals.

38. The apparatus of claim 37 wherein said multidimensional space represents a stereo mix environment, said multiple parameters include left and right gain, and said sound signal processing circuit provides a left sound output signal and a right sound output signal.

39. The apparatus of claim 38 wherein said multiple parameters include left front, center front, right front, and rear gain, and said sound signal processing system provides left front, center front, right front, and rear output signals.

40. The apparatus of claim 37 wherein said display has a hard left region with which a muted left gain control signal is associated and a hard right region with which a muted right gain control signal is associated.

41. The apparatus of claim 37 wherein said visual representation includes text indicating said sound input signal.

42. The apparatus of claim 36 wherein said controller is an interactive controller, and said input device is an interactive input device.

43. The apparatus of claim 36 wherein there are plural output channels, said multiple parameters include gains associated with respective outputs, and said display has a region which causes a muted gain signal for a said channel.

44. The apparatus of claim 36 further comprising a MIDI control line that is connected between said controller and said sound processing circuit and transmits said multiple parameter control signal.

45. The apparatus of claim 36 wherein said display has one or more mappable regions defined in said multidimensional space such that movement of a representation to a said region will cause automatic generation of a control signal affecting one or more parameters.

46. The apparatus of claim 45 wherein each said mappable region has an associated control signal including an audio parameter set.

47. An apparatus for controlling multiple parameters effecting an audio mix output, said apparatus comprising

a controller including a visual display for displaying a visual representation of an input sound signal in a multidimensional space and generating a multiple parameter control signal controlling multiple parameters effecting said audio mix output, each dimension of said multidimensional space separately controlling a mixing parameter or mixing parameters, said control signal being based upon the state of said representation in said multidimensional space, at least two said dimensions being the location of said representation on said visual display, a change in location of said representation on said display causing a change in two or more said mixing parameters,

an input device connected to said controller to control the state of said representation in said multidimensional space so as to affect said audio mix output, and

a sound signal processing circuit having a sound signal input for receiving a sound input signal, a sound signal output for providing a sound output signal, and a control signal input for receiving said multiple parameter control signal, said circuit modifying said sound input signal based upon said multiple parameter control signal to provide said sound output signal,

wherein said display displays a plurality of visual representations of respective input signals, and said sound signal processing circuit has a plurality of sound signal inputs,

wherein said multidimensional space represents a multiple channel output mix environment with N output channels, said multiple parameters include a gain associated with each unique combination of an input sound signal and ann output sound signal, and said sound signal processing circuit provides N output signals and includes M times N controllable amplifiers, where M is the number of input signals, the outputs of said amplifiers that are associated with the same output channel being added to each other to provide said N output signals, each said controllable amplifier being connected to receive a said input sound signal and a unique gain control signal, said multiple parameter control signal including said gain control signal,

wherein said controller provides address and control information to said sound signal processing circuit, and said address and control information controls gain updating of particular controllable amplifiers,

wherein

said sound signal processing circuit includes a decoder circuit,

said decoder circuit receives said address and control information and provides enable signals to said controllable amplifiers, and

said enable signals enable the gain updating of particular controllable amplifiers.

48. An apparatus for controlling multiple parameters effecting an audio mix output, said apparatus comprising

a controller including a visual display for displaying a visual representation of an input sound signal in a multidimensional space and generating a multiple parameter control signal controlling multiple parameters effecting said audio mix output, each dimension of said multidimensional space separately controlling a mixing parameter or mixing parameters, said control signal being based upon the state of said representation in said multidimensional space, at least two said dimensions being the location of said representation on said visual display, a change in location of said representation on said display causing a change in two or more said mixing parameters,

an input device connected to said controller to control the state of said representation in said multidimensional space so as to affect said audio mix output, and

a sound signal processing circuit having a sound signal input for receiving a sound input signal, a sound signal output for providing a sound output signal, and a control signal input for receiving said multiple parameter control signal, said circuit modifying said sound input signal based upon said multiple parameter control signal to provide said sound output signal,

wherein said multidimensional space represents a multiple channel output mix environment, said multiple parameters include gains associated with each said output, and said sound signal processing circuit provides a plurality of output signals,

wherein said sound signal processing circuit includes a controllable amplifier to each said output signal, each said controllable amplifier being connected to receive said input sound signal and a gain control signal to provide a respective amplified sound input signal as said sound output signal, said multiple parameter control signal including said gain control signal,

wherein said controller provides address and control information to said sound signal processing circuit, and said address and control information controls gain updating of particular controllable amplifiers,

wherein said sound signal processing circuit includes a decoder circuit, said decoder circuit receives said address and control information and provides enable signals to said controllable amplifiers, and said enable signals enable the gain updating of particular controllable amplifiers.

49. An apparatus for controlling multiple parameters effecting an audio mix output, said apparatus comprising

a controller including a visual display for displaying a visual representation of an input sound signal in a multidimensional space and generating a multiple parameter control signal controlling multiple parameters effecting said audio mix output, each dimension of said multidimensional space separately controlling a mixing parameter or mixing parameters, said control signal being based upon the state of said representation in said multidimensional space, at least two said dimensions being the location of said representation on said visual display, a change in location of said representation on said display causing a change in two or more said mixing parameters,

an input device connected to said controller to control the state of said representation in said multidimensional space so as to affect said audio mix output, and

a sound signal processing circuit having a sound signal input for receiving a sound input signal, a sound signal output for providing a sound output signal, and a control signal input for receiving said multiple parameter control signal, said circuit modifying said sound input signal based upon said multiple parameter control signal to provide said sound output signal,

wherein said display displays a plurality of visual representations of respective input signals, and said sound signal processing circuit has a plurality of sound signal inputs,

wherein said visual representations have changes in appearance that are associated with changes in value of a said multiple parameter.

50. The apparatus of claim 49 wherein said change in appearance is a change in color.

51. The apparatus of claim 49 wherein said change in appearance is a change in brightness.

52. The apparatus of claim 49 wherein said change in appearance is a change in size.

53. The apparatus of claim 49 wherein said change in appearance is a change in shape.