Patent Grant
8301076
This invention relates generally to wireless devices capable of audio recording, and more specifically to distributed audio recording and collaborative mixing by two or more wireless devices.
Wireless devices, such as laptop computers, personal digital assistants (PDAs), cellular phones, etc., bring new resources to distributed computing. In addition to typical computational resources such as CPU, disk space, and applications, wireless devices increasingly employ cameras, microphones, GPS receivers, and other types of sensors. A wireless device by definition has at least one wireless communication interface (e.g., cell, radio frequency, Wi-Fi, or Bluetooth™). Users increasingly take wireless devices with them to new places, in both their personal and professional lives. The ability of wireless devices to form ad-hoc grids allows using the available resources in a collaborative manner, by aggregating information from the range of input/output interfaces found in wireless devices, by leveraging the locations and contexts in which wireless devices are located, and finally, by leveraging the mesh network capabilities of wireless devices. Wireless grids allow coordinated collaboration of heterogeneous inherently unreliable devices, across unreliable network connections. The inherent unreliability of wireless devices is primarily caused by the fact that those devices are, due to their mobile nature, battery-powered. Thus, reducing the power consumption and mitigating the inherent unreliability are two goals of a paramount importance. Thus, there is a need in distributed systems and applications which can assist in achieving both goals by off-loading processing and data management to non-mobile devices, or to wireless devices which can be reachable with less transmitter power.
There is provided a system for distributed audio recording and collaborative mixing by combining audio streams from two or more sources into a single stream that is composed of two or more channels. Leveraging the spatial location of the devices allows the producing of high quality multi-channel sound (e.g., stereo sound or surround sound).
Two or more wireless devices can be located near a sound source, e.g., at a business meeting, symphony concert, or a live lecture. The wireless devices can be independently controlled by their respective users, by a mixer component, or by a leader wireless device. The wireless devices can convert the recorded audio into a standard or proprietary audio stream format, and transmit the audio stream to a mixer component which can run on a remote computer.
The real-time clocks of two or more participating wireless devices can be synchronized. A wireless device can insert timestamps into the audio stream to facilitate the mixing operation.
Mixing of the two or more audio streams recorded by wireless devices can be performed by a mixer component either in real time (contemporaneously with the recording) or asynchronously with respect to the recording. The mixing can be performed in a fully automated mode, and/or in an operator-assisted mode.
The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the drawings, like numerals are used to indicate like parts throughout the various views.
There is provided a system for distributed audio recording and collaborative mixing, by combining audio streams from multiple sources into a single stream that is composed of multiple channels. Leveraging the spatial location of the devices allows to produce high quality multi-channel sound (e.g., stereo sound or surround sound).
The wireless devices 101a-101z can have a user interface and/or an application programming interface (API) allowing to at least start and stop the audio recording and streaming operations. In one embodiment, the wireless devices 101a-101z can be independently controlled by their respective users via a user interface. In another embodiment, the wireless devices 101a-101z can register with and be controlled by a mixer component (not shown in
The mixer component can run on a remote computer 110. A “computer” herein shall refer to a programmable device for data processing, including a central processing unit (CPU), a memory, and at least one communication interface. A computer can be provided, e.g., by a personal computer (PC) running the Linux operating system.
Computer 110 can be connected to network 180. While different networks are designated herein, it is recognized that a single network as seen from the network layer of the Open System Interconnection (OSI) model can comprise a plurality of lower layer networks (e.g., what can be regarded as a single IP network, can include a plurality of different physical networks).
In one aspect, wireless device 101a-101c can be provided by a PDA and can connect to network 180 via a wireless access point 114a-114z. In another aspect, wireless device 101d-101z can be provided by a cellular phone and can connect to network 180 via General Packet Radio Service (GPRS) gateway 150.
The mixer component can transmit control messages to the wireless devices 101a-101z. The control messages can be encapsulated into, e.g., Blocks Extensible Exchange Protocol (BEEP). The control messages can include a start recording message and a stop recording command.
Upon receiving a start recording command, the wireless device 101 can activate its microphone to start recording. In one embodiment, the wireless device can start transmitting the recorded audio stream back to the mixer component in real time (synchronously with the recording). In another embodiment, the wireless device can buffer the audio stream being recorded and, asynchronously with respect to the recording, transmit the buffered stream back to the mixer component. In a yet another embodiment, the wireless device can store the recorded audio stream in its memory for later transmission to a mixer component.
Upon receiving a stop recording command, the wireless device 101 might stop recording audio stream. In one embodiment, the wireless device might further stop any synchronous transmission of the audio stream to the mixer component. In another embodiment, the wireless device 101 can further complete any asynchronous transmission of a buffered audio stream to a mixer component.
In another embodiment, the wireless devices 101a-101z can elect a leader wireless device which will coordinate the recording by other participating wireless devices. The leader election can be performed, e.g., using an algorithm described in “A Leader Election Protocol For Fault Recovery In Asynchronous Fully-Connected Networks” by M. Franceschetti and J. Bruck, available at http://caltechparadise.library.caltech.edu/31/00/etr024.pdf.
A skilled artisan would appreciate the fact that any other suitable algorithm of the leader election can be used without departing from the scope and spirit of the invention.
The wireless devices 101a-101z can convert the recorded audio into a standard or proprietary audio stream format, e.g., MPEG-3, RealAudio, Windows Media Audio, etc. The resulting audio stream can be stored by the recording device locally, and/or transmitted to a remote computer 110 via a wireless access point 114 and network 180. Wireless devices with no direct connection to wireless access point can leverage the mesh network capability of a group of wireless devices, e.g., by establishing a wireless mesh network defined in IEEE 80211s.
In one embodiment, wireless devices 101a-101z can have their real-time clocks unsynchronized. In another embodiment, the real-time clocks of two or more participating wireless devices 101a-101z can be synchronized using, e.g., Network Time Protocol (NTP) by Network Working Group, available at ftp://ftp.rfc-editor.org/in-notes/rfc1305.pdf. A wireless device can insert timestamps into the audio stream to facilitate the mixing operation.
Mixing of the two or more audio streams recorded by wireless devices 101a-101z can be performed by a mixer component (not shown in
The mixing can be performed in a fully automated mode, and/or in an operator-assisted mode.
Operation of the mixer component in a fully automated mode is now described with reference to
In another embodiment, where the real-time clocks of the participating wireless devices can not be synchronized reliably, the individual recordings can be synchronized in time based upon one or more clearly distinguishable events present in all the recordings being synchronized. A clearly distinguishable event can be, e.g., a change in the signal amplitude at a given frequency range where the amplitude level changes by a value exceeding a pre-defined amplitude threshold within a time period not exceeding a pre-defined duration.
The operation of the mixer component in an operator-assisted mode is now described. Graphical representations of the sound waves over two or more sound channels, e.g., graphs of the audio signal amplitude over time, can be presented to the user via a graphical user interface (GUI) as shown in
The user can choose a common point of visual distinction (e.g., a point of rapid signal amplitude change 310a, 310b) and align the graphs using the view slide controls and then pressing the Sync button 320, so that two or more sound channels are synchronized at the common point 310a, 310b.
A small sample of systems methods and apparatus that are described herein is as follows:
two or more wireless devices capable of audio recording, wherein said two or more wireless devices are located near a sound source to be recorded;
wherein each wireless device of said two or more wireless devices having an interface allowing at least start and stop audio recording and streaming operations;
wherein each wireless device of said two or more wireless devices being configured to transmit a recorded audio stream to a mixer component; and
a mixer component configured to combine two or more audio streams received from said two or more wireless devices into a multi-channel audio stream by synchronizing in time said two or more audio streams, said synchronization being performed based upon one or more clearly distinguishable events present in all said two or more audio streams.
two or more wireless devices capable of audio recording,
wherein said two or more wireless devices are located near a sound source to be recorded, each wireless device of said two or more wireless devices having an interface allowing at least start and stop audio recording and streaming operations, each wireless device of said two or more wireless devices being configured to transmit a recorded audio stream to a mixer component, each wireless device of said two or more wireless devices having a real-time clock, each wireless device of said two or more wireless devices being further configured insert timestamps into said recorded audio stream; and
a mixer component configured to combine two or more audio streams received from said two or more wireless devices into a multi-channel audio stream by synchronizing in time said two or more audio streams based upon said timestamps.
While the present invention has been particularly shown and described with reference to certain exemplary embodiments, it will be understood by one skilled in the art that various changes in detail may be affected therein without departing from the spirit and scope of the invention as defined by claims that can be supported by the written description and drawings. Further, where exemplary embodiments are described with reference to a certain number of elements it will be understood that the exemplary embodiments can be practiced utilizing less than the certain number of elements.
This application claims priority under 35 U.S.C §119(e) of the following provisional application: U.S. Ser. No. 60/965,581, filed Aug. 21, 2007, entitled “SYSTEM AND METHOD FOR DISTRIBUTED AUDIO RECORDING AND COLLABORATIVE MIXING”, the content of which is incorporated herein by reference.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT The disclosed invention was made with government support under a grant awarded by the National Science Foundation under Grant No. 0227879. The United States Government has certain rights in the invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7084898 | Firestone et al. | Aug 2006 | B1 |
| 7191023 | Williams | Mar 2007 | B2 |
| 7711443 | Sanders et al. | May 2010 | B1 |
| 7825322 | Classen et al. | Nov 2010 | B1 |
| 20020015504 | Kohno et al. | Feb 2002 | A1 |
| 20030169330 | Shachar et al. | Sep 2003 | A1 |
| 20050201301 | Bridgelall | Sep 2005 | A1 |
| 20060130636 | Toledano et al. | Jun 2006 | A1 |
| 20070039449 | Redmann | Feb 2007 | A1 |
| 20070111657 | Yamada et al. | May 2007 | A1 |
| 20070127671 | Chua et al. | Jun 2007 | A1 |
| 20070140510 | Redmann | Jun 2007 | A1 |
| Number | Date | Country |
|---|---|---|
| 1020010004504 | Jan 2001 | KR |
| 100469472 | Jan 2005 | KR |
| 1020060128305 | Dec 2006 | KR |
| Number | Date | Country | |
|---|---|---|---|
| 20090068943 A1 | Mar 2009 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60965581 | Aug 2007 | US |
