Patent Grant
RE44737
Embodiments described herein relate to position calibration of audio sensors and actuators in a distributed computing platform.
Many emerging applications like multi-stream audio/video rendering, hands free voice communication, object localization, and speech enhancement, use multiple sensors and actuators (like multiple microphones/cameras and loudspeakers/displays, respectively). However, much of the current work has focused on setting up all the sensors and actuators on a single platform. Such a setup would require a lot of dedicated hardware. For example, to set up a microphone array on a single general purpose computer, would typically require expensive multichannel sound cards and a central processing unit (CPU) with larger computation power to process all the multiple streams.
Computing devices such as laptops, personal digital assistants (PDAs), tablets, cellular phones, and camcorders have become pervasive. These devices are equipped with audio-visual sensors (such as microphones and cameras) and actuators (such as loudspeakers and displays). The audio/video sensors on different devices can be used to form a distributed network of sensors. Such an ad-hoc network can be used to capture different audio-visual scenes (events such as business meetings, weddings, or public events) in a distributed fashion and then use all the multiple audio-visual streams for an emerging applications. For example, one could imagine using the distributed microphone array formed by laptops of participants during a meeting in place of expensive stand alone speakerphones. Such a network of sensors can also be used to detect, identify, locate and track stationary or moving sources and objects.
To implement a distributed audio-visual I/O platform, includes placing the sensors, actuators and platforms into a space coordinate system, which includes determining the three-dimensional positions of the sensors and actuators.
Embodiments of a three-dimensional position calibration of audio sensors and actuators in a distributed computing platform are disclosed. In the following description, numerous specific details are set forth. However, it is understood that embodiments may be practiced without these specific details. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.
Reference throughout this specification to “one embodiment” or “an embodiment” indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
For example, in one embodiment, given a set of M acoustic sensors and S acoustic actuators in unknown locations, one embodiment estimates their respective three dimensional coordinates. The acoustic actuators are excited using a predetermined calibration signal such as a maximum length sequence or chirp signal, and the time of arrival (TOA) is estimated for each pair of the acoustic actuators and sensors. In one embodiment, the TOA for a given pair of microphone and speakers is defined as the time for the acoustic signal to travel from the speaker to the microphone. Measuring the TOA and knowing the speed of sound in the acoustical medium, the distance between each acoustical signal source and the acoustical sensors can be calculated, thereby determining the three dimensional positions of the actuators and the sensors.
In block 202, a first GPC 102, which may be considered the master GPC of the distributed platform, transmits a wireless signal to a surrounding set of GPCs in the distributed platform (the actual number of GPCs included in the distributed platform may vary based on implementation). The signal from the first GPC 102 includes a request that a specific actuator of one of the GPCs (e.g., second GPC 103) be excited to generate an acoustic signal to be received by the sensors of the surrounding GPCs (e.g., GPC 102, 104-106). In one embodiment, the initial wireless signal from the master GPC 102 identifies the specific actuator 110 to be excited.
In response to the signal from the master GPC 102, in block 204 the second GPC 103 excites the actuator 110 to generate an acoustic signal. In one embodiment, the acoustic signal may be a maximum length sequence or chirp signal, or another predetermined signal. In block 206, the second GPC 103 also transmits a first global time stamp to the other GPCs 104-106. In one embodiment, the global time stamp identifies when the second GPC 103 initiated the actuation of the actuator 110 for the second GPC 103. In block 208, the sensors of the GPCs 102, 104-106, receive the acoustic signal generated by the second GPC 103.
In block 210, the time for the acoustic signal to travel from the actuator 110 of the second GPC 103 to the respective sensors (hereinafter referred to as Time of Arrival (TOA)), is estimated. In one embodiment, the TOA for a given pair of a microphone and speaker is defined as the time taken by the acoustic signal to travel form the speaker to the microphone.
In one embodiment, the GPCs that receive the acoustic signal via their sensors, proceed to estimate the respective TOAs. In one embodiment, there exists a common clock in the distributed platform so that GPCs 102-106 are able to determine the time of arrival of audio samples captured by the respective sensors. As a result, the TOA can be estimated based on the difference between the first global time stamp issued by the second GPC 103 and the time of when the acoustic signal is received by a sensor.
Considering, however, that sensors are distributed on different platforms, the audio stream among the different GPCs are typically not synchronized in time (e.g., analog-to-digital and digital-to-analog converters of actuators and sensors of the different GPCs are unsynchronized). As a result, the estimated TOA does not necessarily correspond to the actual TOA. In particular, the TOA of the acoustic signal may include an emission start time, which is defined as the time after which the sound is actually emitted from the speaker (e.g., actuator 110) once the command has been issued from the respective GPC (e.g., GPC 103). The actual emission start time is typically never zero and can actually vary in time depending on the sound card and processor load of the respective GPC.
Therefore, to account for the variations in the emission start time, multiple alternatives may be used. For example, in one embodiment, if multiple audio input channels are available on the GPC exciting an actuator, then one of the output channels can be connected directly to one of the input channels forming a loop-back. Source emission start time can then be estimated for a given speaker, and can be globally transmitted to the other GPCs 102, 104-106 to more accurately determine the respective TOAs. Furthermore, in one embodiment, in the case of using the loop-back, the estimated emission start time will be included in the global time stamp transmitted by the respective GPC.
Once the TOAs for the acoustic signal have been estimated by the receiving GPCs 104-106, which may include accounting for the unknown emission start time as described above, the TOAs are transmitted to the master GPC 102. In an alternative embodiment, the TOAs can be computed by the master GPC 102, in which case each sensor of GPCs 104-106 generate a second global timestamp of when the acoustic signals arrived, respectively. In the alternative embodiment, the master GPC 102 uses the first global time stamp (identifying when the second GPC 103 initiated the actuation of the actuator 110) and the second global time stamps to estimate the TOAs for the respective pairs of actuators and sensors. In such as case, the master GPC 102 may also estimate the emission start time of the acoustic signal to estimate the TOAs.
In decision block 212, if additional actuators remain in the distributed platform, the processes of blocks 202-210 are repeated to have each of the actuators in the platform generate an acoustic signal to determine the TOAs with respective receiving sensors. In an alternative embodiment, multiple separate actuators may be actuated in parallel, wherein the actuator signals are multiplexed by each actuator using a unique signal (e.g., different parameters for chirp or MLS signals). In the case of actuating the multiple separate actuators in parallel, the master GPC 102 identifies to each actuator a unique signal parameters to be used when exciting the actuator.
Once all of the TOAs for the different pairs of actuators and sensors have been computed and transmitted to the master GPC 102, in block 214 the master GPC 102 computes the coordinates of the sensors and the actuators. More specifically, as illustrated in the position computation scheme of
To provide the initial estimation as used by the NLS, several alternatives are available. For example, if an approximate idea of the microphone and speaker positions is available, then the initialization may be done manually. In another embodiment, the use of one or more cameras may provide a rough estimate to be used as the initial estimation.
An additional embodiment to generate an initial estimation includes assuming that microphones and speakers on a given computing platform are approximately at the same position, and given all estimates of the pairwise distances between the separate GPCs, a multidimensional scaling approach may be used to determine the coordinates from, in one embodiment, the Euclidean distance matrix. The approach involves converting the symmetric pairwise distance matrix to a matrix of scalar products with respect to some origin and then performing a singular value decomposition to obtain the matrix of coordinates. The matrix coordinates in turn, may be used as the initial guess or estimate of the coordinates for the respective GPCs, and the microphones and speakers located on them.
The techniques described above can be stored in the memory of one of the computing devices or GPCs as a set of instructions to be executed. In addition, the instructions to perform the processes described above could alternatively be stored on other forms of computer and/or machine-readable media, including magnetic and optical disks. Further, the instructions can be downloaded into a computing device over a data network in a form of compiled and linked version.
Alternatively, the logic to perform the techniques as discussed above, could be implemented in additional computer and/or machine readable media, such as discrete hardware components as large-scale integrated circuits (LSI's), application-specific integrated circuits (ASIC's), firmware such as electrically erasable programmable read-only memory (EEPROM's); and electrical, optical, acoustical and other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.); etc.
These embodiments have been described with reference to specific exemplary embodiments thereof. It will, however, be evident to persons having the benefit of this disclosure that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the embodiments described herein. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.
| Number | Name | Date | Kind |
|---|---|---|---|
| 3559161 | Raudsep | Jan 1971 | A |
| 4555779 | Roberts | Nov 1985 | A |
| 5691922 | McEwan et al. | Nov 1997 | A |
| 5778082 | Chu et al. | Jul 1998 | A |
| 5859595 | Yost | Jan 1999 | A |
| 5959568 | Woolley | Sep 1999 | A |
| 5970413 | Gilhousen | Oct 1999 | A |
| 5986971 | Kim et al. | Nov 1999 | A |
| 6201499 | Hawkes et al. | Mar 2001 | B1 |
| 6243471 | Brandstein et al. | Jun 2001 | B1 |
| 6643516 | Stewart | Nov 2003 | B1 |
| 6661342 | Hall et al. | Dec 2003 | B2 |
| 6662137 | Squibbs | Dec 2003 | B2 |
| 6677895 | Holt | Jan 2004 | B1 |
| 6941246 | Raykar et al. | Sep 2005 | B2 |
| 20020077772 | Squibbs | Jun 2002 | A1 |
| 20020097885 | Birchfield et al. | Jul 2002 | A1 |
| 20020150263 | Rajan | Oct 2002 | A1 |
| 20020155845 | Martorana | Oct 2002 | A1 |
| 20020168989 | Dooley et al. | Nov 2002 | A1 |
| 20030012168 | Elson et al. | Jan 2003 | A1 |
| 20030014486 | May | Jan 2003 | A1 |
| 20030114170 | Rick et al. | Jun 2003 | A1 |
| 20030129996 | Maloney et al. | Jul 2003 | A1 |
| 20030174086 | Hirt | Sep 2003 | A1 |
| 20030236866 | Light | Dec 2003 | A1 |
| 20040109417 | Castro et al. | Jun 2004 | A1 |
| 20040170289 | Whan | Sep 2004 | A1 |
| Number | Date | Country |
|---|---|---|
| 2 329 778 | Mar 1999 | GB |
| 2 329 778 | Mar 1999 | GB |
| WO 0126335 | Oct 2000 | WO |
| WO 0126335 | Apr 2001 | WO |
| PCTUS2004008587 | Jan 2005 | WO |
| Entry |
|---|
| Office Action for corresponding European Application No. 04 760 779.1-2212, dated Dec. 21, 2009. |
| Written Opinion in corresponding Singapore application No. 200505757-4, issued by the Austrian Patent Office on Jun. 29, 2007. |
| Written Opinion in corresponding Singapore application No. 200506757-4, issued by the Austrian Patent Office on May 30, 2008. |
| First Office Action in corresponding Chinese application No. 2004100347403 issued by the State Intellectual Property Office of the People's Republic of China, Oct. 27, 2006. |
| Second Office Action in corresponding Chinese application No. 2004100347403 issued by the State Intellectual Property Office of the People's Republic of China, Jun. 6, 2008. |
| International Search Report for PCT US2004/008587, dated Jan. 7, 2005. |
| Written Opinion for PCT US2004/008587, dated Jan. 7, 2005. |
| Bulusu, Nirupama, et al., “Scalable Coordination for Wireless Sensor Networks: Self-Configuring Localization Systems,” Proceedings of the 6th International Symposium on Communication Theory and Applications (ISCTA '01), Ambleside, Lake District, UK, Jul. 2001, 6 pages. |
| Girod, Lewis, et al., “Locating tiny sensors in time and space: A case study,” Proceedings of the 2002 IEEE International Conference on Computer Design: VLSI in Computers and Processors (ICCD '02), Freiburg, Germany, Sep. 16-18, 2002, 6 pages. |
| Moses, Randolph L., et al., “A Self-Localization Method for Wireless Sensor Networks,” EURASIP Journal on Applied Signal Processing 2003-4, .COPYRGT. 2003 Hindawai Publishing Corporation, pp. 348-358. |
| Raykar, Vikas C., et al., “Self Localization of acoustic sensors and actuators on Distributed platforms,” Proceedings. 2003 International Workshop on Multimedia Technologies in E-Learning and Collaboration, Nice, France, Oct. 2003, 8 pages. |
| Sachar, Joshua M., et al., “Position Calibration Of Lage-Aperture Microphone Arrays,” 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 2, 2002, pp. 1797-1800. |
| Savvides, Andreas, et al., “Dynamic Fine-Grained Localization in Ad-Hoc Networks of Sensors,” in the proceedings of the International Conference on Mobile Computing and Networking (MobiCom) 2001, Rome, Italy, Jul. 2001, pp. 166-179. |
| Examination Report in corresponding Singapore application No. 200506757-4, issued by the Austrian Patent Office on Jul. 23, 2009. |
| Office Action for corresponding European Application No. 04 760 779.1-2212, dated Apr. 11, 2011. |
| Result of Consultation for corresponding European Application No. 04 760 779.1, dated Nov. 21, 2011. |
| Communication Under Rule 73(3) EPC for corresponding European Application No. 04 760 779.1, dated Dec. 19, 2011. |
| Office Action for corresponding European Application No. 11 009 918.1-2212, dated Dec. 20, 2012. |
| European Search Report for corresponding European Application No. 11 00 9918,1, dated Apr. 23, 2012. |
| Morset, “How to measure the initial time delay (distance between the loudspeaker and the microphone)?”, Morset Sound Development, Jul. 25, 2002, pp. 1-3, available at http://www.nvo.com/winmls/nss-folder/discussion/How%20to%20measure%20distance%20loudspeaker1miscrophone.doc. |
| Walker, “The Truth About Latency,” Sound on Sound, Sep. 2002 pp. 1-7, available at http://www.soundonsound.com/sos/Sep02/articles/pcmusician0902.asp?print=yes. |
| Office Action for corresponding European Application No. 11 009 918.1, dated Sep. 27, 2013. |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10435231 | May 2003 | US |
| Child | 12110216 | US |
