Patent Application
20120327115
Augmented reality environments allow interaction among users and real-world objects and virtual or computer-generated objects and information. This merger between the real and virtual worlds paves the way for new interaction opportunities. However, acquiring data about these interactions, such as audio data including speech or audible gestures, may be impaired by noise or multiple signals present in the physical environment.
The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.
An augmented reality system may be configured to interact with objects within a scene and generate an augmented reality environment. The augmented reality environment allows for virtual objects and information to merge and interact with tangible real-world objects, and vice versa. Disclosed herein are techniques and devices suitable for using an acoustic microphone array with beamforming to acquire or reject audio signals occurring within the physical environment of an augmented reality environment. Audio signals include useful information such as user speech, audible gestures, audio signaling devices, as well as noise sources such as street noise, mechanical systems, and so forth. The audio signals may include frequencies generally audible to the human ear or inaudible to the human ear, such as ultrasound.
Signal data is received from a plurality of microphones arranged in a microphone array. The microphones may be distributed in regular or irregular linear, planar, or three-dimensional arrangements. The signal data is then processed by a beamformer module to generate processed data. In some implementations the signal data may be stored for later processing.
Beamforming is the process of applying a set of beamformer coefficients to the signal data to create beampatterns, or effective volumes of gain or attenuation. In some implementations, these volumes may be considered to result from constructive and destructive interference between signals from individual microphones in the microphone array.
Application of the set of beamformer coefficients to the signal data results in processed data expressing the beampattern associated with those beamformer coefficients. Application of different beamformer coefficients to the signal data generates different processed data. Several different sets of beamformer coefficients may be applied to the signal data, resulting in a plurality of simultaneous beampatterns. Each of these beampatterns may have a different shape, direction, gain, and so forth.
Beamformer coefficients may be pre-calculated to generate beampatterns with particular characteristics. Such pre-calculation reduces overall computational demands. In other instances, meanwhile, the coefficients may be calculated on an on-demand basis. In either instance, the coefficients may be stored locally, remotely such as within cloud storage, or distributed across both.
A given beampattern may be used to selectively gather signals from a particular spatial location where a signal source is present. Localization data available within the augmented reality environment which describes the location of the signal source may be used to select a particular beampattern focused on that location. The signal source may be localized, that is have its spatial position determined, in the physical environment by various techniques including structured light, image capture, manual entry, trilateration of audio signals, and so forth. Structured light may involve projection of a pattern onto objects within a scene and may determine position based upon sensing the interaction of the objects with the pattern using an imaging device. The pattern may be regular, random, pseudo-random, and so forth. For example, a structured light system may determine a user's face is at particular coordinates within in the room.
The selected beampattern may be configured to provide gain or attenuation for the signal source. For example, the beampattern may be focused on a particular user's head allowing for the recovery of the user's speech while attenuating noise from an operating air conditioner across the MOM.
Such spatial selectivity by using beamforming allows for the rejection or attenuation of undesired signals outside of the beampattern. The increased selectivity of the beampattern improves signal-to-noise ratio for the audio signal. By improving the signal-to-noise ratio, the interpretation of audio signals within the augmented reality environment is improved.
The processed data from the beamformer module may then undergo additional filtering or be used directly by other modules. For example, a filter may be applied to processed data which is acquiring speech from a user to remove residual audio noise from a machine running in the environment.
The beamforming module may also be used to determine a direction or localize the audio signal source. This determination may be used to confirm a location determined in another fashion, such as from structured light, or when no initial location data is available. The direction of the signal source relative to the microphone array may be identified in a planar manner, such as with reference to an azimuth, or in a three-dimensional manner, such as with reference to an azimuth and an elevation. In some implementations the signal source may be localized with reference to a particular set of coordinates, such as azimuth, elevation, and distance from a known reference point.
Direction or localization may be determined by detecting a maximum signal among a plurality of beampatterns. Each of these beampatterns may have gain in different directions, have different shapes, and so forth. Given the characteristics such as beampattern direction, topology, size, relative gain, frequency response, and so forth, the direction and in some implementations location of a signal source may be determined.
A microphone array 104, input/output devices 106, network interface 108, and so forth may couple to a computing device 110 containing a processor 112 via an input/output interface 114. The microphone array 104 comprises a plurality of microphones. The microphones may be distributed in regular or irregular pattern. The pattern may be linear, planar, or three-dimensional. Microphones within the array may have different capabilities, patterns, and so forth. The microphone array 104 is discussed in more detail below with regards to
The ARFN 102 may incorporate or couple to input/output devices 106. These input/output devices include projectors, cameras, microphones, other ARFNs 102, other computing devices 110, and so forth. The coupling between the computing device 110 and the input/output devices 106 may be via wire, fiber optic cable, or wireless connection. Some of the input/output devices 106 of the ARFN 102 are described below in more detail with regards to
The network interface 108 is configured to couple the computing device 110 to a network such as a local area network, wide area network, wireless wide area network, and so forth. For example, the network interface 108 may be used to transfer data between the computing device 110 and a cloud resource via the internet.
The processor 112 may comprise one or more processors configured to execute instructions. The instructions may be stored in memory 116, or in other memory accessible to the processor 112 such as in the cloud via the network interface 108.
The memory 116 may include computer-readable storage media (“CRSM”). The CRSM may be any available physical media accessible by a computing device to implement the instructions stored thereon. CRSM may include, but is not limited to, random access memory (“RAM”), read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), flash memory or other memory technology, compact disk read-only memory (“CD-ROM”), digital versatile disks (“DVD”) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computing device.
Several modules such as instructions, datastores, and so forth may be stored within the memory 116 and configured to execute on a processor, such as the processor 112. An operating system module 118 is configured to manage hardware and services within and coupled to the computing device 110 for the benefit of other modules. An augmented reality module 120 is configured to maintain the augmented reality environment.
A localization module 122 is configured to determine a location or direction of a signal source relative to the microphone array 104. The localization module 122 may utilize, at least in part, data including structured light, ranging data, and so forth as acquired via the input/output device 106 or the microphone array 104 to determine a location of the audio signal source. For example, a structured light projector and camera may be used to determine the physical location of the user's head, from which audible signals may emanate. In another example, audio time difference of arrival techniques may be used to determine the location.
A beamforming module 124 is configured to accept signal data from the microphone array 104 and apply beamformer coefficients to the signal data to generate processed data. By applying the beamformer coefficients to the signal data, a beampattern is formed which may exhibit gain, attenuation, directivity, and so forth. Such gain, attenuation, directivity and so forth is exhibited in the processed data. For example, the beampattern may focus and increase gain for speech coming from the user. By applying beamformer coefficients configured to produce a beampattern having gain focused on the user's physical location, the acquired signal may be improved in several ways. For example, the resulting processed data exhibits a speech signal with a greater signal-to-noise ratio compared to non-beamformer signals. In another example, the processed data may exhibit reduced noise from other spatial locations. In other implementations, other improvements may be exhibited. This increase in gain is discussed in more detail below with regards to
Beamformer coefficients may be calculated on-the-fly, or at least a portion of the coefficients may be pre-calculated before use. The pre-calculated beamformer coefficients may be stored within a beamformer coefficients datastore 126, described in more depth below with regards to
In some implementations the signal data from the microphone array 104 and/or other input devices in the augmented reality environment may be stored in a signal datastore 128. For example, data about objects within the environment which generate audio signals may be stored, such as their size, shape, motion, and so forth. This stored data may be accessed for later processing by the beamforming module 124 or other modules. Modules may be stored in the memory of the ARFN 102, storage devices accessible on the local network, or cloud storage accessible the network interface 108. For example, a dictation module may be stored and operated from within a cloud resource.
A chassis 204 holds the components of the ARFN 102. Within the chassis 204 may be disposed a projector 206 that generates and projects images into the scene 202. These images may be visible light images perceptible to the user, visible light images imperceptible to the user, images with non-visible light, or a combination thereof. This projector 206 may be implemented with any number of technologies capable of generating an image and projecting that image onto a surface within the environment. Suitable technologies include a digital micromirror device (DMD), liquid crystal on silicon display (LCOS), liquid crystal display, 3LCD, and so forth. The projector 206 has a projector field of view 208 which describes a particular solid angle. The projector field of view 208 may vary according to changes in the configuration of the projector. For example, the projector field of view 208 may narrow upon application of an optical zoom to the projector. In some implementations, a plurality of projectors 206 may be used.
A camera 210 may also be disposed within the chassis 204. The camera 210 is configured to image the scene in visible light wavelengths, non-visible light wavelengths, or both. The camera 210 has a camera field of view 212 which describes a particular solid angle. The camera field of view 212 may vary according to changes in the configuration of the camera 210. For example, an optical zoom of the camera may narrow the camera field of view 212. In some implementations, a plurality of cameras 210 may be used. The chassis 204 may be mounted with a fixed orientation, or be coupled via an actuator to a fixture such that the chassis 204 may move. Actuators may include piezoelectric actuators, motors, linear actuators, and other devices configured to displace or move the chassis 204 or components therein such as the projector 206 and/or the camera 210. For example, in one implementation the actuator may comprise a pan motor 214, tilt motor 216, and so forth. The pan motor 214 is configured to rotate the chassis 204 in a yawing motion changing the azimuth. The tilt motor 216 is configured to change the pitch of the chassis 204 changing the elevation. By panning and/or tilting the chassis 204, different views of the scene may be acquired.
One or more microphones 218 may be disposed within the chassis 204, or elsewhere within the scene such in the microphone array 104. These microphones 218 may be used to acquire input from the user, for echolocation, location determination of a sound, or to otherwise aid in the characterization of and receipt of input from the scene. For example, the user may make a particular noise, such as a tap on a wall or snap of the fingers, which are pre-designated as attention command inputs. The user may alternatively use voice commands. In some implementations audio inputs may be located within the scene using time-of-arrival differences among the microphones, and/or with beamforming as described below with regards to
One or more speakers 220 may also be present to provide for audible output. For example, the speakers 220 may be used to provide output from a text-to-speech module or to playback pre-recorded audio.
A transducer 222 may be present within the ARFN 102, or elsewhere within the environment, and configured to detect and/or generate inaudible signals, such as infrasound or ultrasound. These inaudible signals may be used to provide for signaling between accessory devices and the ARFN 102.
A ranging system 224 may also be provided in the ARFN 102. The ranging system 224 may be configured to provide distance, location, or distance and location information from the ARFN 102 to a scanned object or set of objects. The ranging system 224 may comprise radar, light detection and ranging (LIDAR), ultrasonic ranging, stereoscopic ranging, and so forth. The ranging system 224 may also provide direction information in some implementations. The transducer 222, the microphones 218, the speaker 220, or a combination thereof may be configured to use echolocation or echo-ranging to determine distance and spatial characteristics.
In another implementation, the ranging system 224 may comprise an acoustic transducer and the microphones 218 may be configured to detect a signal generated by the acoustic transducer. For example, a set of ultrasonic transducers may be disposed such that each projects ultrasonic sound into a particular sector of the room. The microphones 218 may be configured to receive the ultrasonic signals, or dedicated ultrasonic microphones may be used. Given the known location of the microphones relative to one another, active sonar ranging and positioning may be provided.
In this illustration, the computing device 110 is shown within the chassis 204. However, in other implementations all or a portion of the computing device 110 may be disposed in another location and coupled to the ARFN 102. This coupling may occur via wire, fiber optic cable, wirelessly, or a combination thereof. Furthermore, additional resources external to the ARFN 102 may be accessed, such as resources in another ARFN 102 accessible via the network interface 108 and a local area network, cloud resources accessible via a wide area network connection, or a combination thereof.
Also shown in this illustration is a projector/camera linear offset designated “O”. This is a linear distance between the projector 206 and the camera 210. Placement of the projector 206 and the camera 210 at distance “O” from one another aids in the recovery of structured light data from the scene. The known projector/camera linear offset “O” may also be used to calculate distances, dimensioning, and otherwise aid in the characterization of objects within the scene 202. In other implementations the relative angle and size of the projector field of view 208 and camera field of view 212 may vary. Also, the angle of the projector 206 and the camera 210 relative to the chassis 204 may vary.
In other implementations, the components of the ARFN 102 may be distributed in one or more locations within the environment 100. As mentioned above, the microphones 218 and the speakers 220 may be distributed throughout the scene. The projector 206 and the camera 210 may also be located in separate chassis 204. The ARFN 102 may also include discrete portable signaling devices used by users to issue command attention inputs. For example, these may be acoustic clickers (audible or ultrasonic), electronic signaling devices such as infrared emitters, radio transmitters, and so forth.
Microphones 218(1)-(M) are distributed along the support structure 302. The distribution of the microphones 218 may be symmetrical or asymmetrical. It is understood that the number and placement of the microphones 218 as well as the shape of the support structure 302 may vary. For example, in other implementations the support structure may describe a triangular, circular, or another geometric shape. In some implementations an asymmetrical support structure shape, distribution of microphones, or both may be used.
The support structure 302 may comprise part of the structure of a room. For example, the microphones 218 may be mounted to the walls, ceilings, floor, and so forth within the room. In some implementations the microphones 218 may be emplaced, and their position relative to one another determined through other sensing means, such as via the ranging system 224, structured light scan, manual entry, and so forth. For example, in one implementation the microphones 218 may be placed at various locations within the room and their precise position relative to one another determined by the ranging system 224 using an optical range finder configured to detect an optical tag disposed upon each.
In one implementation the microphones 218 and microphone array 104 are configured to operate in a non-aqueous and gaseous medium having a density of less than about 100 kilograms per cubic meter. For example, the microphone array 104 is configured to acquire audio signals in a standard atmosphere.
The direction to a signal source may be designated in three-dimensional space with an azimuth and elevation angle. The azimuth angle 506 indicates an angular displacement relative to an origin. The elevation angle 508 indicates an angular displacement relative to an origin, such as local vertical.
The beampattern 504 may exhibit a plurality of lobes, or regions of gain, with gain predominating in a particular direction designated the beampattern direction 602. A main lobe 604 is shown here extending along the beampattern direction 602. A main lobe beam-width 606 is shown, indicating a maximum width of the main lobe 604. A plurality of side lobes 608 is also shown. Opposite the main lobe 604 along the beampattern direction 602 is the back lobe 610. Disposed around the beampatttern 504 are null regions 612. These null regions are areas of attenuation to signals. For example, as shown here the signal source location 502(1) of the first speaker is within the main lobe 604 and benefits from the gain provided by the beampattern 504 and exhibits improved a signal-to-noise ratio compared to a signal acquired with non-beamforming. In contrast, the signal source location 502(2) of the second speaker is in a null region 612 behind the back lobe 610. As a result the signal from the signal source location 502(2) is significantly reduced relative to the first signal source location 502(1).
As shown in this illustration, the use of the beampatterns provides for gain in signal acquisition compared to non-beamforming. Beamforming also allows for spatial selectivity, effectively allowing the system to “turn a deaf ear” on a signal which is not of interest. Furthermore, because multiple beampatterns may be applied simultaneously to the same set of signal data from the microphone array 104, it is possible to have multiple simultaneous beampatterns. For example, a second beampattern 504(2) may be generated simultaneously allowing for gain and signal rejection specific to the signal source location 502(2), as discussed in more depth below with regards to
As shown here, our two signal source locations 502(1) and 502(2) from the first and second users respectively are present in the single room. In this example, assume the second user is a loud talker, producing a high-amplitude audio signal at the signal source location 502(2). The use of the beampattern 504 shown here which is focused on the first user provides gain for the signal source location 502(1) of the first speaker while attenuating the second speaker at the second signal source location 502(2). However, consider that even with this attenuation resulting from the beampattern, the second user is such a loud talker that his speech continues to interfere with the speech signal from the first user.
To alleviate this situation, or provide other benefits, gain to the microphones 218 may be applied differentially across the microphone array 104. In this illustration, a graph of microphone gain 702 is shown associated with each microphone 218 in the array 104. As shown here, gain is reduced in the microphones 218 closest to the second signal source location 502(2). This reduces the signal input from the second user, minimizing the signal amplitude of their speech captured by the beampattern. Similarly, the gain of the microphones 218 proximate to the first speaker's first signal source location 502(1) are increased to provide greater signal amplitude.
In other implementations depending upon microphone response, position of the speaker, and so forth, the gain of the individual microphones may be varied to produce a beampattern which is focused on the signal source location of interest. For example, in some implementations signal-to-noise ratio may be improved by decreasing gain of a microphone proximate to the signal source location of interest.
Shown here with a dotted line is an aggregate signal 806 from the microphone array 104 without beamforming applied. In the aggregate signal 806, the signal of interest 808 shows an amplitude comparable to the noise the signal. A noise signal from machinery such as an air conditioner operating elsewhere in the room 810 is shown here. Attempting to analyze the signal 808, such as processing for speech recognition would likely result in poor results given the low signal-to-noise ratio.
In contrast, the signal with the beamformer 812 clearly elevates the signal of interest 808 well above the noise. Furthermore, the spatial selectivity of the signal with beamformer 812 has effectively eliminated the machinery noise 810 from the signal. As a result of the improved signal quality, additional analysis of the signal such as for speech recognition experiences improved results.
The beamformer coefficients datastore 126 may be configured to store a beampattern name 902, as well as the directionality of the beampattern 504. This directionality may be designated for one or more lobes of the beampattern 504, relative to the physical arrangement of the microphone array 104. For illustration only and not by way of limitation, the directionality of the beampattern is the beampattern direction 602, that is the direction of the main lobe 604.
The directionality may include the azimuth direction 904 and elevation direction 906, along with size and shape 908 of the beampattern. For example, beampattern A is directed in an azimuth of 0 degrees and an elevation of 30 degrees, and has six lobes. In other implementations, size and extent of each of the lobes may be specified. Other characteristics of the beampattern such as beampattern direction, topology, size, relative gain, frequency response, and so forth may also be stored.
Beamformer coefficients 910 which generate each beampattern are stored in the beamformer coefficients datastore 126. When applied to signal data which includes signals from the microphones 218(M) to generate processed data, these coefficients act to weigh or modify those signals to generate a particular beampattern.
The beamformer coefficients datastore 126 may store one or more beampatterns. For example, beampatterns having gain in different directions may be stored. By pre-computing, storing, and retrieving coefficients computational demands are reduced compared to calculation of the beamformer coefficients during processing. As described above, in some implementations one portion of the beamformer coefficients datastore 126 may be stored within the memory 116, while another portion may be stored in cloud resources.
As shown here, a first beampattern 1002 is shown as generated by application beampattern A 902 having beamformer coefficients 910(1). A second beampattern 1004 having gain in a different direction and resulting from beampattern B 902 is also shown. A third beampattern 1006 resulting from application of beampattern C's 902 beamformer coefficients 910(3) points in a direction different from the first and second beampatterns.
As shown at 1008, all three or more beampatterns may be simultaneously active. Thus, as shown in this example, three separate signal sources may be tracked, each with a different beampattern with associated beamformer coefficients. So long as the beamforming module 124 has access to computational capacity to process the incoming signal data from the microphone 104, additional beampatterns may be generated.
The localization module 122 may provide source directional data 1104 to the beamforming module 124. For example, the localization module 122 may use structured light to determine the signal source location 502 of the user is at certain spatial coordinates. The source directional data 1104 may comprise spatial coordinates, an azimuth, an elevation, or an azimuth and elevation relative to the microphone array 104.
The beamforming module 124 may generate or select a set of beamformer coefficients 910 from the beamformer coefficients datastore 126. The selection of the beamformer coefficients 910 and their corresponding beampatterns 504 may be determined based at least in part upon the source directional data 1104 for the signal source. The selection may be made to provide gain or attenuation to a given signal source. For example, beamformer coefficients 910 resulting in the beampattern 504 which provides gain to the user's speech while attenuating spatially distinct noise sources may be selected. As described above, the beamformer coefficients 910 may be pre-calculated at least in part.
The beamforming module 124 applies one or more sets of beamformer coefficients 910 to the signal data 1102 to generate processed data 1106. For example and not by way of limitation, the beamforming module 124 may use four sets of beamformer coefficients 910(1)-(4) and generate four sets of processed data 1106(1)-(4). While originating from the same signal data, each of these sets of processed data 1106 may be distinct due to their different beampatterns 504.
The processed data may be analyzed or further manipulated by additional processes. At shown here, the processed data 1106(1) is filtered by filter module 1108(1). The filtered processed data 1106(1) is then provided to a speech recognition module 1110. The filter module 1108(1) may comprise a band-pass filter configured to selectively pass frequencies of human speech. The filter modules herein may be analog, digital, or a combination thereof. The speech recognition module 110 is configured to analyze the processed data 1106 which may or may not be filtered by the filter module 1108(1) and recognize human speech as input to the augmented reality environment.
The second set of processed data 1106(2) may or may not be processed by a second filter module 1108(2) and provided to an audible gesture recognition module 1112 for analysis. The audible gesture recognition module 1112 may be configured to determine audible gestures such as claps, fingersnaps, tapping, and so forth as input to the augmented reality environment.
So long as the beamforming module 124 has access to processing capability to apply beamforming coefficients 910 to the signal data 1102, multiple simultaneous beampatterns may be produced, each with processed data output. The third set of processed data 1106(3) such as generated by a third set of beamformer coefficients 910 may be provided to some other module 1114. The other module 1114 may provide other functions such as audio recording, biometric monitoring, and so forth.
In some implementations the source directional data 1104 may be unavailable, unreliable, or it may be desirable to confirm the source directional data independently. The ability to selectively generate beampatterns simultaneously may be used to localize a sound source.
A source direction determination module 1116 may be configured as shown to accept multiple processed data inputs 1106(1), . . . 1106(Q). Using a series of different beampatterns 504, the system may search for signal strength maximums. By using successively finer resolution beampatterns 504, the source direction determination module 116 may be configured to isolate a direction to the signal source, relative to the microphone array 104. In some implementations the signal source may be localized to a particular region in space. For example, a set of beampatterns each having different origin points may be configured to triangulate the signal source location, as discussed in more detail below with regards to
The beamforming module 124 may also be configured to track a signal source. This tracking may include modification of pre-calculated set of beamformer coefficients 910, or the successive selection of different sets of beamformer coefficients 910.
The beamforming module 124 may operate in real-time, near-real-time, or may be applied to previously acquired and stored data such as in the signal datastore 128. For example, consider a presentation which took place in the augmented reality environment. The signal data 1102 from the presentation was stored in the signal datastore 128. During the presentation by a presenter, two colleagues in the back of the room conversed with one another, discussing a point raised by the presenter. Upon request for a recording of their side conversation, the beamforming module 124 uses one or more beampatterns to focus on the signal from their position in the room during the conversation and generate processed data 1106 of their conversation. In contrast, other users requesting playback of the presentation may hear audio resulting from beampatterns focused on the presenter.
The processes described in this disclosure may be implemented by the architectures described herein, or by other architectures. These processes are illustrated as a collection of blocks in a logical flow graph. Some of the blocks represent operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order or in parallel to implement the processes. It is understood that the following processes may be implemented on other architectures as well.
At 1204, a location of the signal source relative to the microphone array 104 is determined. Continuing the example, the ARFN 102 may use structured light from the projector 206 and received by the camera 210 to determine the source directional data 1104 showing the user is located at spatial coordinates X, Y, Z in the room, which is at a relative azimuth of 300 degrees and elevation of 45 degrees relative to the microphone array 104.
At 1206, a set of beamformer coefficients 910 are applied to the signal data to generate processed data 1106 having a beampattern 504 focused on the location or the direction of the signal source. In some implementations, at least a portion of the beamformer coefficients 910 may be pre-calculated and retrieved from the beamformer coefficients datastore 126. Selection of the set of beamformer coefficients 910 may be determined at least in part by resolution of the source directional data 1104. For example, where the source directional data has a margin of error of ±1 meter, a beampattern having a larger main lobe beam-width 606 may be selected over a beampattern having a smaller main lobe beam-width 606 to insure capture of the signal. At 1208, the processed data 1106 may be analyzed. For example, the processed data may be analyzed by the speech recognition module 1110, audible gesture recognition module 1112, and so forth. Continuing the example, the speech recognition module 1110 may generate text data from the user's speech. Likewise, the audible gesture recognitions module 1112 may determine a hand clap has taken place and produce this as a user input.
In some implementations the set of beamformer coefficients 910 may be updated at least partly in response to changes in the determined location or direction of the signal source. For example, where the signal source is a user speaking while walking, the set of beamformer coefficients 910 applied to the signal data 1102 may be successively updated to provide a primary lobe with gain focused on the user while in motion.
While a single signal and beampattern have been described here, it is understood that multiple signals may be acquired and multiple simultaneous beampatterns may be present.
Shown here is a room with a set of four coarse beampatterns 1302 deployed therein. These beampatterns 504 are configured to cover four quadrants of the room. As mentioned above, these beampatterns 504 may be exist simultaneously. The signal source location 502 is indicated with an “X” in the upper right quadrant of the room. The processed data 1106 from each of the beampatterns 504 may be compared to determine in which of the beampatterns a signal maximum is present. For example, the beamforming module 124 may determine which beampattern has the loudest signal.
As shown here, the beampattern 504 having a main lobe and beamdirection to the upper right quadrant is shaded, indicating it is the beampattern which contains the maximum signal. A first beampattern direction 1304 is shown at a first angle 1306. Because the coarse beampatterns 1302 are relatively large, at this point the direction to the signal source location 502 is imprecise.
Based upon the determination that the upper right beampattern contains the signal maximum, a set of intermediate beampatterns 1308 is then applied to the signal data 1102. As shown here, this set of intermediate beampatterns are contained predominately within the volume of upper right quadrant of interest, each having smaller primary lobes than the coarse beampatterns 1302. A signal maximum is determined from among the intermediate beampatterns 1308, and as shown here by the shaded primary lobe having a second beampattern direction 1310 at a second angle 1312.
A succession of beampatterns having different gain, orientation, and so forth may continue to be applied to the signal data 1102 to refine the signal source location 502. As shown here, a set of fine beampatterns 1314 are focused around the second beampattern direction 1310. Again, from these beampatterns a signal maximum is detected. For example, as shown here, the shaded lobe of one of the fine beampatterns 1314 contains the signal maximum. A third beampattern direction 1316 of this beampattern is shown having a third angle 1318. The direction to the signal source location 502 may thus be determined as the third angle 1318.
At 1404, a first set of beamformer coefficients 910 describing a first set of beampatterns 504 encompassing a first volume is applied to the signal data 1102. For example, the coarse beampatterns 1302 of
At 1406, a determination is made as to which of the beampatterns within the first set of beampatterns contains a maximum signal strength from the signal. Continuing the example from
At 1408, a second set of beamformer coefficients 910 describing a second set of beampatterns within the first volume is applied to the signal data 1102. For example, the intermediate beampatterns 1308 within the upper right quadrant. In some implementations the beampatterns in the second set may extend outside the first volume. However, the beampatterns in the second set of beamformer coefficients 910 may be configured to be disposed predominately within the first volume.
At 1410, a determination is made as to which of the beampatterns within the second set of beampatterns contains a maximum signal strength from the signal. For example, the beampattern having the second beampattern direction 1310.
At 1412, a direction to the source relative to the microphone array 104 is determined based at least in part upon the characteristics of the beampattern within the second set containing the signal strength maximum. The characteristics of the beampattern may include the beampattern direction 602, main-lobe beamwidth 606, gain pattern, beampattern geometry, location of null regions 612, and so forth.
In some implementations additional iterations of successively finer beampatterns may be used to further refine the direction to the signal source. Furthermore, in some implementations the beampatterns may be configured to have origins disposed in different physical locations. The origin of the beampattern is the central point about which the lobes may be considered to extend from.
Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features described. Rather, the specific features are disclosed as illustrative forms of implementing the claims.
