Patent Application
20120327746
Acoustic signals such as handclaps or fingersnaps may be used as input within augmented reality environments. In some instances, systems and techniques may attempt to determine the location of these acoustic signals within these environments. Traditional methods of localizing, or determining the spatial coordinates, of an acoustic source are sensitive to distortions introduced by the environment and frequently produce erroneous results. What is desired is a way to effectively and accurately determine the spatial coordinates of an acoustic signal.
The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.
Augmented reality environments may utilize acoustic signals such as audible gestures, human speech, audible interactions with objects in the physical environment, and so forth for input. Detection of these acoustic signals provides for minimal input, but richer input modes are possible where the acoustic signals may be localized, or located in space. For example, a handclap at chest height may be ignored as applause while a handclap over the user's head may call for execution of a special function. In another example, a person speaking while standing may be given priority for speech recognition over a person seated while speaking In one example, the localization may be of acoustic signals propagated through a medium having a density less than 1000 kilograms per cubic meter. For example, gaseous air at about one standard atmosphere of pressure (about 100 kilopascals).
A plurality of microphones may be used to detect an acoustic signal. By measuring the time of arrival of the acoustic signal at each of the microphones, and given a known position of each microphone relative to one another, time-difference-of-arrival data is generated. This time-difference-of-arrival (TDOA) data may be used for hyperbolic positioning to calculate the location of the acoustic signal. The acoustic environment, particularly with audible frequencies (including those extending from about 300 Hz to about 3 KHz), are signal and noise rich. Furthermore, acoustic signals interact with various objects in the physical environment, including users, furnishings, walls, and so forth. This interaction can result in direct sound as well as indirect sounds. Direct sound is sound which travels directly from the acoustic source to a sensing microphone, free from obstruction or interaction with other objects or surfaces. Indirect sounds, in contrast, travel via one or more indirect routes, such as via reflection, refraction, and so forth. Indirect sounds such as reverberations may manifest as a series of echoes of the direct signal, received over a relatively short span of time.
Generally, a path length between the acoustic source and the sensor is longer for indirect sounds than for direct sounds. Stated another way, direct sounds generally travel a shorter distance from the source to the sensor. Given a reasonably homogenous medium for transfer of the signal, the longer path length results in more time taken by the indirect sound to transit from the source to the sensor, compared to the direct sound. Given the importance of accurate time differentials in hyperbolic positioning, such variances in timing may lead to significant errors in a calculated position. The presence of multiple similar signals such as in the case of reverberation may thus cause significant variance in the calculated time delay and corresponding introduction of errors in calculated location.
Disclosed herein are devices and techniques for generating time-difference-of-arrival (TDOA) data suitable for use in determining location of a signal source within an environment that may cause signal reverberation. Within such an environment, microphones that acquire signals from an acoustic source are disposed in a pre-determined physical arrangement having relative locations to one another which are known. In this regard, an origin point may be specified relative to the microphones. The spatial coordinates of the acoustic signal may then be defined relative to the origin. A time-difference-TDOA module is configured to generate from the signals a set of time delays suitable for use by a TDOA localization module in determining spatial coordinates of the acoustic signal.
As described herein, the TDOA module is configured to process audio signals to determine direct sounds emanating from the acoustic source, and provide time-difference-of-arrival delay intervals. The TDOA module is configured to look for direct sound components of the signals and disregard reverberations, echoes, and so forth. A portion of the signal may be used to determine the direct sound. For example, a first two milliseconds of an event of interest may be used to determine whether the signal is a direct or indirect sound.
In some implementations, known physical attributes may be used to constrain or filter possible TDOA results. Physical attributes may include spacing and positioning of microphones, room dimensions, and so forth. For example, where the microphone spacing is known, TDOA results which would result in a sound exceeding the speed of sound when travelling between the two microphones may be disregarded.
As shown here, the sensor node 102 incorporates or is coupled to a plurality of microphones 104 configured to receive acoustic signals. A ranging system 106 may also be present which provides another method of measuring the distance to objects within the room. The ranging system 106 may comprise laser range finder, acoustic range finder, optical range finder, structured light module, and so forth. The structured light module may comprise a structured light source and camera configured to determine position, topography, or other physical characteristics of the environment or objects therein based at least in part upon the interaction of structured light from the structured light source and an image acquired by the camera.
A network interface 108 may be configured to couple the sensor node 102 with other devices placed locally such as within the same room, on a local network such as within the same house or business, or remote resources such as accessed via the internet. In some implementations, components of the sensor node 102 may be distributed throughout the room and configured to communicate with one another via cabled or wireless connection.
The sensor node 102 may include a computing device 110. The computing device 110 may comprise one or more processors 112, one or more input/output interfaces 114, and a memory 116. The memory 116 may store an operating system 118, time-difference-of-arrival (TDOA) module 120, and TDOA localization module 122. In some implementations, the resources among a plurality of computing devices 110 may be shared. These resources may include input/output devices, processors 112, memory 116, and so forth. The memory 116 may include computer-readable storage media (“CRSM”). The CRSM may be any available physical media accessible by a computing device to implement the instructions stored thereon. CRSM may include, but is not limited to, random access memory (“RAM”), read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technology, compact disk read-only memory (“CD-ROM”), digital versatile disks (“DVD”) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computing device.
The input/output interface 114 may be configured to couple the computing device 110 to microphones 104, ranging system 106, network interface 108, or other devices such as an atmospheric pressure sensor, temperature sensor, hygrometer, barometer, an image projector, camera, and so forth. The coupling between the computing device 110 and the external devices such as the microphones 104 and the network interface 108 may be via wire, fiber optic cable, wirelessly, and so forth.
The TDOA module 120 is configured to determine time-difference-of-arrival delay values using direct sound acoustic signals received by the microphones 104. In some implementations the TDOA module 120 may be configured to accept data from the sensors accessible to the input/output interface 114. For example, the TDOA module 120 may determine time-difference-of-arrival based at least in part upon temperature and humidity data. As described below in more detail with regards to
Once available by the TDOA module 120, the TDOA localization module 122 is configured to use the TDOA data to determine source locations of the acoustic signals. This determination may include hyperbolic positioning based on the TDOA data and known positions of the microphones.
The support structure 202 may comprise part of the structure of a room. For example, the microphones 104 may be mounted to the walls, ceilings, floor, and so forth at known locations within the room. In some implementations the microphones 104 may be emplaced, and their position relative to one another determined through other sensing means, such as via the ranging system 106, structured light scan, manual entry, and so forth.
The ranging system 106 is also depicted as part of the sensor node 102. As described above, the ranging system 106 may utilize optical, acoustic, radio, or other range finding techniques and devices. The ranging system 106 may be configured to determine the distance, position, or both between objects, users, microphones 104, and so forth. For example, in one implementation the microphones 104 may be placed at various locations within the room and their precise position relative to one another determined using an optical range finder configured to detect an optical tag disposed upon each.
In another implementation, the ranging system 106 may comprise an acoustic transducer and the microphones 104 may be configured to detect a signal generated by the acoustic transducer. For example, a set of ultrasonic transducers may be disposed such that each projects ultrasonic sound into a particular sector of the room. The microphones 104 may be configured to receive the ultrasonic signals, or dedicated ultrasonic microphones may be used. Given the known location of the microphones relative to one another, active sonar ranging and positioning may be provided.
A first direct signal 506 which results from direct sound being received at a first microphone 104 is indicated by a solid line. A second direct signal 508 from the direct sound being received at a second microphone 104 is indicated with a dotted line. A time difference between a first peak of the first direct signal 506 and the second direct signal 508 is a time-difference-of-arrival interval 510. For example, this may be the time difference in receipt of the signal as compared between the central microphone 104(5) and another microphone such as 104(2). Other methods, including generalized cross correlation may also be used to determine the time-difference-of-arrival.
The graph 500 also illustrates that the microphones have picked up an indirect signal 512. A single indirect signal is shown here for ease of illustration and not by way of limitation. For example, in a small room with hard walls and floor, several indirect reverberating acoustic signals may be detected at the microphone 104. A signal received by the microphone 104 is a combination of these direct and indirect signals. As a result, amplitude of the received signal at any microphone 104 may be greater than or less than that of the direct signals which have not been interfered with by the indirect signals. For example, destructive interference may result in a direct signal with lower amplitude than the indirect signal at a particular microphone.
If the TDOA data were to be based on a received signal as corrupted by the indirect signal 512, an incorrect TDOA interval 510 may be determined. This incorrect TDOA interval would then lead to an incorrect localization of the signal.
The graph 500 also illustrates a noise floor 514 of the system. This noise floor may include ambient noise detected by the sensors. Where the sensors are microphones 104, this might include the sound of an air conditioning system, running motor in a refrigerator, external wind sounds, and so forth. The noise floor 514 may also include noise generated by operation of the sensing equipment, thermal noise in solid state devices, and so forth. In this context, signals may be considered meaningful when their amplitude exceeds that of the noise floor 514. The noise floor 514 may be static or dynamically adjustable.
The graph 500 further illustrates a quiet period 516 where signals from the sensors, if any, are at or below the noise floor 514. Continuing the example, the signal from the first direct signal 506 is shown in the quiet period 516, but during the quiet period 516 is below the noise floor 514. In some implementations, the quiet period 516 may vary from about 10 ms to about 150 ms.
As illustrated in this example, the acoustic source has generated a signal with amplitude which increases over time. Here, the initial peak 518 of the first direct signal 506 is shown where the amplitude of the signal exceeds the noise floor 514. As described below, the initial peak 518 of the first direct signal 506 and the second direct signal 508 may be separated from the peaks of the indirect signal 512. By measuring the TDOA interval 510 between direct signals, overall TDOA data quality is improved, improving accuracy of localization.
Overall duration of a direct signal 520 for one sensor is indicated. Some signals may be transient or sustained. A transient direct signal may have a duration of 250 milliseconds (ms) or less. Transient signals include taps of a user's fingers, a hand clap, individual footsteps, click, and so forth. Sustained signals extend for an overall duration in excess of the transient signals. In some instances, signals that last more than 250 ms may be deemed sustained signals, while signals less than this threshold may be deemed transient. Of course while one example duration threshold has been given, other embodiments may utilize any other threshold value. Sustained signals include human speech, whistles, breath noise, an audio beacon, and so forth.
The following processes may be implemented by the architectures herein, or by other architectures. These processes are illustrated as a collection of blocks in a logical flow graph. Some of the blocks represent operations that can be implemented in hardware, software, or a combination thereof In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order or in parallel to implement the processes. Furthermore, while the following process describes estimation of TDOA for acoustic signals, non-acoustic signals may be processed as described herein.
At 604, noise may be removed from the audio signals. Noise removal may take place using analog devices, digital processing, or a combination thereof At 606, the audio signals are filtered with a narrow band-pass filter. For example, the band-pass filter may have a bandwidth extending from about 800 Hertz (Hz) to about 2 Kilohertz (KHz). In some implementations other band-pass filter bandwidths may be used. The bandwidth is selected to pass events of interest while removing unwanted frequency bands.
At 608, a noise floor of the filtered signals is estimated, either statically or dynamically. Static noise floor estimation may set a pre-determined noise floor level. In contrast, dynamic noise floor may be configured to track and adjust over time. In one implementation, an average-up/track down approach may be used, such that when the signal rises above the estimated noise floor, the noise floor is increased slowly but when the signal goes below the estimated noise floor, the noise floor may decrease more quickly. As a result, the noise floor may slowly rise and quickly fall.
At 610, an event of interest within the filtered signals is determined to occur at a particular time. For example, a sharp peak corresponding to part of the sound associated with a user tapping fingers on a table may be determined at time 150 milliseconds. In some implementations, a lookback may take place to determine where the signal exceeds the noise floor. In other situations, any signal having a signal-to-noise ratio (SNR) exceeding a pre-determined SNR may be designated as an event of interest.
At 612, when a quiet period exists before the event of interest, the process proceeds to 614. The quiet period may be static or dynamically adjusted, as described above. When no quiet period exists, the process may return to 610 and continue attempting to identify an event of interest.
At 614, a time when the signal of the event of interest begins to rise above the noise floor is determined. At 616, an initial peak associated with the signal corresponding to the event of interest which has exceeded the noise floor is detected. A pre-determined number or duration of the initial peaks corresponding to the direct sound may be selected. The pre-determined number or duration may be set statically or dynamically. For example, where set statically, in some implementations the first three peaks may be considered to be the direct sound. Later peaks, owing to the increased propagation time involved in the longer path lengths of indirect sound are disregarded.
At 618, TDOA values based at least in part on the first peak of a plurality of signals are determined. Being free from reverberation or other distorting signals, the TDOA values based on the direct sound are more accurate.
At 620, TDOA values are filtered based at least in part on one or more physical attributes. These physical attributes may include known locations of the microphones 104, dimensions of the room, temperature, atmospheric pressure, and so forth. For example, TDOA values corresponding to a location beyond the confines determined by the ranging system 106 may be filtered and removed.
At 702, a plurality of audio signals associated with an acoustic source is acquired from a plurality of microphones. For example, microphones 104(1)-(5) in the room may gather audio signals from an acoustic source comprising the user speaking aloud.
At 704, noise may be removed from the audio signals. In some implementations a band-pass filter may be applied. For example, the band-pass filter may have a bandwidth extending from about 2 Kilohertz (KHz) to about 8 Kilohertz (KHz). At 706, a noise floor of the signals is estimated, either statically or dynamically. Static noise floor estimation may set a pre-determined noise floor level. In contrast, the dynamic noise floor may be configured to track and adjust over time, such as described above.
At 708, an event of interest is identified at a particular time. For example, the event of interest may comprise the user speaking. This event of interest may be determined when a signal exceeds a pre-determined SNR.
At 710, when a quiet period exists before the event of interest, the process proceeds to 712. The quiet period may be static or dynamically adjusted, as described above. When no quiet period exists, the process may return to 708 and continue attempting to identify an event of interest.
At 712, a signal rise time is determined when the signal begins to rise above the noise floor of the quiet period. At 714, a time window of samples is designated starting at or around the signal rise time. At 716, a set of TDOA values based at least in part on the samples within the window is determined. At 718, the window of samples is slid forward by time t. For example, in some implementations the window may be advanced by about 2 milliseconds.
At 720, when the samples are within a region of interest which contains at least a portion of the direct signal, the process proceeds to 722. When the samples are not within the region of interest, the process may return to 708 and identify an event of interest at a particular time.
At 722, the TDOA values based at least in part on the samples within the now-slid window are determined. At 724, when a stable set of TDOA values among consecutive windows are reached, the process continues to 726. In some implementations a stable set of TDOA values may comprise four windows of TDOA values which are within 2% of one another. At 726, the stable TDOA values are provided for localization. When at 724 the TDOA values are not stable, that is they vary beyond a pre-determined threshold from one another, the process may return to 718 and continue sliding the window and calculating TDOAs. In some implementations, the entire process 700 may be iterated to provide for continuous tracking of a sustained audio source, such as a speaking user.
At 806, cross-correlation values between the audio signals in the region are calculated with a phase transform (PHAT) to generate TDOA values. By confining the computation of the PHAT to the region of interest, computational requirements are reduced compared to continuous calculation of the entire signal.
At 808, a location of the acoustic source is determined based at least in part upon data resulting from the cross-correlation values of the region of interest. Because the TDOA data reflects the direct sound, degradation due to processing of reverberations is not present. As a result, the TDOA data is more accurate, leading to a more accurate localization.
Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims
