Patent Application
20170303062
The present application relates generally to a method of synchronizing electronic devices. More specifically, the present application is capable of precisely synchronizing electronic devices equipped with audio capabilities, such as mobile devices, desktop and laptop computers, smart TVs, and Bluetooth speakers.
In recent years, the use of the aforementioned devices has become increasingly widespread. When two or more of these electronic devices are situated in close proximity, e.g. in the same room, it is desirable to use their audio capabilities collaboratively, so that either better audio quality or new audio features can be realized. By way of example, binaural sound (also known as 3D sound) recording and playback are important for Virtual Reality applications; but due to audio component and form factor constraints, binaural sound is difficult to be produced or reproduced on a single device, even if such device is equipped with multiple microphones and/or speakers.
Interaural Time Difference (or ITD), which is the difference in arrival time of a sound between two ears of humans or animals, represents an important factor that affects how a sound may be perceived by a human or an animal. The maximum ITD for humans is approximately 500 microseconds. Humans can perceive the time difference of a small fraction of the maximum ITD. ITD is important in the localization of sounds, as the time difference provides a cue to the direction or angle of the sound source from the head. Consequently precise synchronization, characterized by a synchronization accuracy of less than the maximum ITD, of multiple microphones is required for binaural sound recording. Likewise, precise synchronization of multiple speakers is required for playback of binaural sound playback. Precise synchronization is also a requirement in other applications. By way of example, seamless animation across multiple display screens can be achieved with precise synchronization—to an observer, each screen acts in unison with other screens as if it were part of one single, larger display screen. By way of another example, devices such as smartphones and tablets equipped with cameras can capture videos and photos synchronously, and the output from each camera can subsequently be stitched/combined to form stereoscopic 3D and/or panoramic videos and photos. The synchronization accuracy required by the latter two use case examples is less stringent comparing to that of binaural audio recording and playback, but still beyond the capabilities of existing methods based on Wi-Fi, Bluetooth or Network Time Protocol.
While precise synchronization can often be achieved trivially in a hardwired setup, e.g. in a home theater system where two or more speakers are tethered to the main control unit through audio cables, it is still quite challenging for existing methods to wirelessly synchronize two discrete or independent electronic devices to a precision that is sufficient for binaural (3D) sound and/or for other applications.
Moreover, once initial precise synchronization is achieved, it is often required to precisely measuring the clock drift between audio channels of electronic devices, in order to maintain the synchronization. Clock drift refers to the fact that clocks used by these devices, in general, do not run at the same speed, and after some time they “drift apart,” causing the audio channels to gradually desynchronize from each other. This clock drift needs to be measured precisely, in order to correct its effects and maintain tight synchronization, without resorting to frequent re-synchronization of the devices involved. It is difficult, with existing methods, to detect the relative clock drift between two discrete or independent electronic devices wirelessly, especially when one or both devices are subject to movement (i.e. non-stationary) relative to each other, to a precision sufficient for certain applications, such as binaural (3D) sound.
The present application discloses a method for synchronizing a plurality of electronic devices. The plurality of electronic devices are preferably close in distance to one another such as special sound effects or visual effects may be generated by the plurality of electronic devices. In one embodiment according to the present application, the achieved precision of synchronization is such that binaural sound recording can be made across multiple electronic devices that are not tethered (i.e. hard-wired), with each device recording or reproducing a separate audio stream. In another embodiment, the level of precision also enables the faithful reproduction of a pre-recorded binaural sound recording across multiple electronic devices, with each device playing a separate audio stream of the recording. Once synchronized and once a trigger event has occurred, each electronic device is able to start emitting or recording sound through its audio subsystem, in such a way that the time difference of the start of playback or recording of sound between any two devices is significantly less than the maximum human ITD.
The present application also discloses method for measuring relative clock drift between multiple electronic devices, that may or may not be stationary relative to each other. Based on the clock drift measurement, audio samples can be discarded or inserted into the audio input or output streams on either or both devices, such that synchronization is maintained. The accuracy achieved by this method is such that binaural sound recording and playback can be made across multiple electronic devices on a continuous basis without loss of initial synchronization.
An aspect of the present application is directed to a method of wirelessly synchronizing a plurality of discrete electronic devices each having an audio input channel and audio output channel. The method comprises providing a wireless communication channel among the plurality of electronic devices; detecting an output injection parameter of the audio output channel of a first electronic device at the time when an audio signal is output by the audio output channel of the first electronic device; detecting an input injection parameter of the audio input channel of a first electronic device at the time when the audio signal is injected to the audio input channel of the first device; receiving an input injection parameter of the audio input channel of a second device; receiving an output injection parameter of the audio output channel of a second device; and determine a synchronization parameter for synchronizing the first electronic device and the second electronic device on the basis of the detected parameters and the receiving parameters of audio channels. According to various embodiments, the input injection parameter includes a sample number. The method further comprises detecting sample frequencies of the audio channels of the first and second electronic devices; and determining the synchronization parameter on the basis of the sample frequencies. The method further comprises generating a 3D audio signal based on the synchronization parameter. The method further comprises recording a 3D audio signal based on the synchronization parameter. Another aspect of the present application is directed to a method of determining the clock drift between a first electronic device and a second electronic device. The method comprises injecting a plurality of audio signals to the audio output channel of a first device and detecting injection parameters of the plurality of audio signals at the time when the plurality of audio signals are injected to the audio output channel of the first device; detecting injection parameters of the plurality of audio signals at the time when the plurality of audio signals are injected to the audio input channel of the second device; injecting a plurality of audio signals generated by the second device into the audio input channel of the first device and detecting injection parameters at the time when the plurality of audio signals generated by the second device are injected into the audio input channel of the first device, injecting the plurality of audio signals generated by the second device into the audio input channel of the second electronic device and detecting injection parameters at the time when the plurality of audio signals generated by the second electronic device are injected into the audio input channel of the second electronic device; and determining a clock drift between the first electronic device and the second electronic device based on the detected injection parameters. According to various embodiment, the injection parameter includes a sample number. The plurality of audio signals include four audio signals. The first electronic device and the second electronic device generate the plurality of audio signals alternately. The two electronic devices are relatively stationary to each other. The two electronic devices are subject to movement to each other.
An advantage that may be achieved by the methods as set forth in the present application is that, according to particular embodiments, it does not require nor involve the use or assistance of any external system or apparatus, e.g. a server or a signal generation apparatus, that is not present on the electronic devices to implement these disclosed synchronization methods (i.e., offline in nature).
In another embodiment, the methods as disclosed in the present application may be implemented by hardware or software. When software is used for carrying out the methods, a non-transitory medium may be used to record an executable program that, when executed, causing a computer or a processor to implement the synchronization methods as disclosed in the present application.
Various exemplary embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements and in which:
In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings can be practiced without such details or with an equivalent arrangement. In other instances, well known methods, procedures, components, and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.
According to one embodiment, steps to obtain precise synchronization between electronic device A 101 and electronic device B 105 are comprised of:
Referring now to
It is noted that the sample numbers on the audio input/output channels 201, 202, 203, and 204 are independent of one another, and are incremented relative to their respective and in general, different starting points. In one embodiment, the synchronization method according to the present application determines a sample number T, on device A 101's audio output channel (speaker) 201, that corresponds in time to sample number B2 on device B 105's audio output channel (speaker) 204. The method also determines a sample number T_prime, on device A 101's audio input channel (microphone) 202, that corresponds in time to sample number B2_prime on device B 105's audio input channel (microphone) 203.
According to one embodiment, sample number T on device A 101's audio output channel (speaker) 201 is calculated according to the following formula:
T=[(A2_prime−A1_prime)+(B2_prime−B1_prime)*Sa/Sb]/2+A1,
wherein Sa is audio channel sampling frequency used by device A 101, and Sb is audio channel sampling frequency used by device B 105.
According to one embodiment, sample number T_prime on device A 101's audio input channel (microphone) 202 is calculated according to the following formula:
T_prime=T−A1+A1_prime.
According to one embodiment, device B 105 supplies Sb, B2_prime and B1_prime to device A 101 over the communication channel 109, in order for device A 101 to apply the aforementioned formula. According to another embodiment, device B 105 supplies the Sb, and difference between B2_prime and B1_prime, in order for device A 101 to apply the aforementioned formula.
Once device A 101 obtains T, devices A 101 and B 105 are considered to be in synchronization with each other in terms of audio output. According to one embodiment, in response to a trigger event, e.g. a user pressing a “play” button on the User Interface of a music application, device A 101 chooses a sample number Tp, and injects an audio stream to its audio output channel (speaker) 201 starting at Tp. Device A 101 also calculates the difference between Tp and T, i.e., D1=Tp−T, and send D1 along with Sa to device B 105 over the communications channel 109. Device B 105 then injects an audio stream to its auto output channel (speaker) 204 starting at the following sample number:
B2+D1*Sb/Sa.
In this way, the audio streams output by devices A 101 and B 105 are considered to be in synchronization with each other. The synchronization method is capable of reducing signal difference produced by these devices down to one sample interval. For example, for a commonly used audio channel sampling frequency of 44.1 kHz in electronic devices, the synchronization resolution is the duration of one digitized sample, or approximately 23 microseconds.
The method disclosed in the present application may also be used for synchronization when a plurality of electronic devices are recording audio signals. Once device A 101 obtains T_prime, devices A 101 and B 105 are considered to be in synchronization with each other in terms of audio input. According to one embodiment, in response to a trigger event, e.g. a user pressing a “record” button on the User Interface of a recording application, device A 101 chooses a sample number Tr, and starts to record an audio stream through its audio input channel (microphone) 202 starting at Tr. Device A 101 also calculates the difference between Tr and T_prime, i.e., D2=Tr−T_prime, and sends D2 along with Sa to device B 105 over the communications channel 109. Device B 105 then starts recording an audio stream through its auto input channel (microphone) 203 starting at sample number B2_prime+D2*Sb/Sa. The audio streams captured by devices A 101 and B 105 will be in synchronization with each other. For a commonly used audio channel sampling frequency of 44.1 kHz in electronic devices, the synchronization resolution is the duration of one digitized sample, or approximately 23 microseconds.
While the information exchanged between electronic devices and method employed to process the information have been described in accordance with the depicted embodiment of
While system 100 and synchronization steps have been described in accordance with the depicted embodiment of
In the foregoing sections, we described methods to wireless synchronize two electronic devices in close proximity to each other. We now turn to the case wherein multiple (>2) devices nearby need to be synchronized.
Refer now to
In another embodiment, the three devices may be synchronized collectively.
The synchronization steps include similar processing as those in the two devices case, except for the following. When each device sends an acoustic signal through its audio output channel (104, 108, 113), itself and all other devices are recording and detecting said signal in their respective audio input channel (microphone) (103, 107, 112). Each device reports detected sample numbers or the difference between sample numbers to relevant devices. Here a “relevant device” refers to the device that generated the corresponding acoustic signals in the report. According to one embodiment, each device employs different acoustic signal characteristics so that another device can distinguish from which device an acoustic signal is originated. In addition, it is preferable to avoid two or more devices sending acoustic signals that overlap in time which may cause interference. According to one embodiment, a time-division approach can be taken, wherein each device has an assigned time slot to send its acoustic signal. According to another embodiment, a carrier sensing and random backoff mechanism can be employed.
While we used 3 devices as an example, it is apparent that the same method described in in the previous sections can be extended to apply to a greater number (>3) of devices.
Referring to
PPM
AB
=CD
AB*1000000 , PPMBA=CDBA*1000000.
It is desirable to obtain an accurate estimate of the clock drift as quickly as possible, which entails choosing Td1 to be as small as possible. Depending on clock crystals and circuits used in each device, the relative clock drift can range from less than 1 ppm for TCXO-driven clocks to more than 100 ppm for typical crystal-driven clocks. As it is generally not known a priori the approximate range of the clock drift between any given pair of devices, Td1 may be set too small to obtain an accurate estimate in some cases. By way of example, assuming a sampling frequency of 48 KHz used by both devices (i.e. 20.8 μs for each digitized audio sample), and Td1 set to 10 seconds. Further assuming measured time difference between Td1 and T′d1 to be 104 μs (i.e. 5 samples), in which case the clock drift rate is calculated to be 10.4 ppm. Because the time difference measurement has the resolution or granularity of 1 sample, this result can be off by 20%, or in other words, the true clock drift rate can fall in the range between 8.3 ppm to 12.5 ppm. A coarse-to-fine approach is therefore contemplated to solve this issue, with steps comprised of:
Referring now to
The time elapsed between the first and second acoustic signals and between the third and fourth acoustic signals are controlled to be sufficiently small such that the two devices 101 and 105 can be considered stationary relative to each other in between the issuance of said signals. It is noted that the time elapsed between the second and third acoustic signals can be arbitrary, and the relative position of devices 101 and 105 can change during this time period.
According to one embodiment, sample number A2 on device A 101's audio output channel (speaker) 201 is calculated according to the following formula:
A2=[(A2′−A1′)+(B2′−B1′)]/2+A1;
sample number B4 on device B 105's audio output channel (speaker) 204 is calculated according to the following formula:
B4=[(A4′−A3′)+(B4′−B3′)]/2+B3.
Time difference between A2 and A4, is calculated as
TA
d=(A4−A2)/SA,
and time difference between B2 and B4 is calculated as
TB
d=(B4−B2)/SB.
The drift rate of device B 105's audio clock relative to device A 101's is calculated as
CD
AB=(TBd−TAd)/TAd.
Similarly the drift rate of device A 101's audio clock relative to device B 105's is calculated as
CD
BA=(TAd−TBd)/TBd.
Referring now to
The time elapsed between the first and second acoustic signals and between the third and fourth acoustic signals are controlled to be sufficiently small such that the two devices 101 and 105 can be considered stationary relative to each other in between the issuance of said signals. It is noted that the time elapsed between the second and third acoustic signals can be arbitrary, and the relative position of devices 101 and 105 can change during this time period.
According to one embodiment, sample number A2 on device A 101's audio output channel (speaker) 201 is calculated according to the following formula:
A2=[(A2′−A1′)+(B2′−B1′)]/2+A1;
sample number A4 is calculated according to the following formula:
A4=[(A4′−A3′)+(B4′−B3′)]/2+A3.
Time difference between A2 and A4, is calculated as
TA
d=(A4−A2)/SA, and
time difference between B2 and B4 is calculated as
TB
d=(B4−B2)/SB.
The drift rate of device B 105's audio clock relative to device A 101's is calculated as
CD
AB=(TBd−TAd)/TAd.
Similarly the drift rate of device A 101's audio clock relative to device B 105's is calculated as
CD
BA=(TAd−TBd)/TBd.
In a manner similar to that described in previous sections of the present application, if the time delay between the second and third acoustic signals is not adequate to obtain an accurate estimate of the clock drift, due to the measurement resolution which is the duration of one audio sample, a larger time delay can be chosen and two more acoustic signals can be issued, and the procedures as described in previous sections can be carried out using the timing of the fifth signal in place of the third signal and the sixth signal in place of the fourth.
A typical electronic device contains multiple clocks, driven by separate crystal oscillators. By way of example, a different clock than the one for the audio subsystem is used by the Operating System of an electronic device. Although the aforementioned methods apply to measuring the relative clock drift between the audio clocks of different devices, it is contemplated that they can be extended to measure the relative clock drift between other clocks, e.g. between the OS clocks. This is achieved, by way of example and not by way of limitation, by first measuring on each device the clock drift between the audio clock and the other clock that is of interest, then combine the results in a straightforward manner.
While the present invention has been described in detail with respect to specific embodiments thereof, it will be appreciated that those skilled in the art, upon attaining an understanding of the foregoing, may readily conceive of alterations to, variations of and equivalents to these embodiments. Accordingly, the scope of the present invention should be assessed as that of the appended claims and any equivalents thereto.
The present application claims the benefit of priority under 35 U.S.C. sctn 119 to U.S. Provisional Application No. 62/278,411, titled “Method to wirelessly synchronize electronic devices,” filed on Mar. 13, 2016, the entirety of which is incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62278411 | Jan 2016 | US |
