Patent Grant
10824921
Interacting with computing systems via natural interactions, such as one or more of voice recognition, text, gesture recognition, motion detection, gaze detection, intent recognition, brain activity assessment, text, the state of a home automated device, etc., enables natural user interface experiences.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
A first intelligent assistant computing device configured to receive and respond to natural language inputs provided by human users syncs to a reference clock of a wireless computer network. The first intelligent assistant computing device receives a communication sent by a second intelligent assistant computing device indicating a signal emission time at which the second intelligent assistant computing device emitted a position calibration signal. The first intelligent assistant computing device records a signal detection time at which the position calibration signal was detected. Based on a difference between 1) the signal emission time and the signal detection time, and 2) a known propagation speed of the position calibration signal, a distance between the first and second intelligent assistant computing devices is calculated.
As discussed above, many computing devices support natural language inputs, through which a human user can address and interact with a computing device in a manner that feels similar to addressing and interacting with another human. For example, a user may ask a virtual assistant provided by software running on an intelligent assistant computing device to provide a weather forecast, save a reminder, initiate a communication with another human, change the state of a device, etc. In a typical example, an intelligent assistant computing device may be implemented as an all-in-one computing device that a human user may place in their home, office, conference room, etc. Such devices may be equipped with microphones, cameras, speakers, video displays, and/or other suitable input/output devices useable for receiving, processing, and responding to natural language inputs.
However, such intelligent assistant computing devices are typically confined to single rooms, or individual subdivisions of rooms, outside of which their usefulness is limited. For example, a device's microphone can only record a human user's spoken natural language input when the user is close enough to be heard. This problem is often addressed by placing multiple intelligent assistant computing devices in the same room/building/environment. For instance, in a conference room setting, two or more intelligent assistant computing devices may be utilized to ensure that all individuals in the conference room are positioned close enough to an intelligent assistant computing device for their natural language inputs to be appropriately received and responded to.
However, this multi-device arrangement has its own associated set of challenges. For example, if a human user provides a natural language input that is detected by two intelligent assistant computing devices at the same time, both devices may attempt to respond, which can be both disconcerting for the human user, as well as a waste of computational resources. In another example, the human user may provide a natural language input to a first intelligent assistant computing device, then leave for a different room where a second, different intelligent assistant computing device is located. In this example, the first intelligent assistant computing device may attempt to respond to the natural language input, even though the human user is no longer present.
These problems could be at least partially mitigated if individual intelligent assistant computing devices had at least some information regarding their own positions relative to one another and/or their local environment. Accordingly, the present disclosure is directed to techniques for position calibration for an intelligent assistant computing device, by which the intelligent assistant computing device can calculate the distance between itself and another intelligent assistant computing device in its environment. With this information, multiple intelligent assistant computing devices can collectively respond to natural language inputs more efficiently. To reuse the examples from above, if two different intelligent assistant computing devices detect the same natural language input, the devices can determine which of them is closer to the human user who provided the input, enabling the better-positioned device to respond while the other device remains dormant. Similarly, as a user moves throughout multiple rooms in a house, responsibility for interacting with the user can be dynamically migrated from one intelligent assistant computing device to another, depending on which device the user is currently closest to. In other words, position calibration as described herein addresses numerous problems that arise in the field of computer technology, for example by allowing intelligent assistant computing devices to conserve both power and processing resources by only activating when appropriate.
The user may ask the system for information about a wide range of topics, such as the weather, personal calendar events, movie show times, etc. In some examples, the intelligent assistant computing device also may be configured to control elements in the living room 102, such as external speakers 108A/108B, television 114, or motorized curtains 116.
The intelligent assistant computing device also may be utilized to receive and store messages and/or reminders to be delivered at an appropriate future time. Using data received from sensors, the intelligent assistant computing device may track and/or communicate with one or more users or other entities.
In some examples, the computing device 104 may be operatively connected with one or more other computing devices using a wired connection, or may employ a wireless connection via Wi-Fi, Bluetooth, or any other suitable wireless communication protocol. For example, the computing device 104 may be communicatively coupled to one or more other computing devices via a computer network. The network may take the form of a local area network (LAN), wide area network (WAN), wired network, wireless network, personal area network, or a combination thereof, and may include the Internet. As will be discussed in more detail, in some settings it may be beneficial for multiple intelligent assistant computing devices in an environment to communicate over a wireless computer network, such that each device learns its position relative to the other computing devices.
It will be appreciated that the computing device 104 of
In this example the computing system 200 includes at least one sensor 202, an entity tracker 212, a voice listener 216, a parser 218, an intent handler 220, a commitment engine 222, and at least one output device 226. In some examples the sensors 202 may include one or more microphones 204, visible light cameras 206, infrared (IR) cameras 208, and connectivity devices 210, such as Wi-Fi or Bluetooth modules. In some examples sensor(s) 202 may comprise stereoscopic and/or depth cameras, head trackers, eye trackers, accelerometers, gyroscopes, gaze detection devices, electric-field sensing componentry, GPS or other location tracking devices, temperature sensors, device state sensors, and/or any other suitable sensor.
The entity tracker 212 is configured to detect entities and their activities, including people, animals, or other living things, as well as non-living objects (e.g., intelligent assistant computing devices). Entity tracker 212 includes an entity identifier 214 that is configured to recognize individual users and/or non-living objects. Voice listener 216 receives audio data and utilizes speech recognition functionality to translate spoken utterances into text. Voice listener 216 also may assign confidence value(s) to the translated text, and may perform speaker recognition to determine an identity of the person speaking, as well as assign probabilities to the accuracy of such identifications. Parser 218 analyzes text and confidence values received from voice listener 216 to derive user intentions and generate corresponding machine-executable language.
Intent handler 220 receives machine-executable language representing user intentions from the parser 218, and resolves missing and ambiguous information to generate commitments. Commitment engine 222 stores commitments from the intent handler 220. At a contextually appropriate time, the commitment engine may deliver one or more messages and/or execute one or more actions that are associated with one or more commitments. Commitment engine 222 may store messages in a message queue 224 or cause one or more output devices 226 to generate output. The output devices 226 may comprise one or more of speaker(s) 228, video display(s) 230, light emitter(s) 232, haptic device(s) 234, and/or other suitable output devices. In other examples, output devices 226 may comprise one or more other devices or systems, such as home lighting, thermostats, media programs, door locks, etc., that may be controlled via actions executed by the commitment engine 222.
In different examples the voice listener 216, parser 218, intent handler 220, commitment engine 222, and/or entity tracker 212 may be embodied in software that is stored in memory and executed by one or more processors of a computing device. In some implementations, specially programmed logic processors may be utilized to increase the computational efficiency and/or effectiveness of the intelligent assistant computing device. Additional details regarding the components and computing aspects of computing devices that may store and execute these modules are described in more detail below with reference to
With reference again to
Furthermore, in some examples, entity tracker 212 may be configured to interface with and/or coordinate the activity of one or more of the output devices 226. As discussed above, it is often desirable for individual intelligent assistant computing devices in an environment to have at least some information regarding their positions relative to each other and/or relative to the environment. In this manner, the intelligent assistant computing devices can respond more effectively to human user input collectively, for instance by automatically handing off interaction responsibilities as the human user moves throughout the environment. To this end, the entity tracker may utilize one or both of the sensors 202 and output devices 226 to emit and/or receive position calibration signals useable to calculate the distance between intelligent assistant computing devices in the same environment. The entity tracker may save a map or other information useable to track the relative positions of the various intelligent assistant computing devices. In some implementations, such a map or other information may be saved locally on one or more intelligent assistant computing devices. In some implementations, such a map or other information may be saved remotely. When saved remotely, one or more intelligent assistant computing devices may access the remotely saved map and/or other information.
The word “entity” as used in the context of the entity tracker 300 may refer to people, animals, or other living things, as well as non-living objects. For example, the entity tracker may be configured to identify furniture, appliances, autonomous robots, other intelligent assistant computing devices, structures, landscape features, vehicles, and/or any other physical object, and determine the position/location and current status of such physical objects.
Entity tracker 300 receives sensor data from one or more sensors 302, such as sensor A 302A, sensor B 302B, and sensor C 302C, though it will be understood that an entity tracker may be used with any number and variety of suitable sensors. As examples, sensors usable with an entity tracker may include cameras (e.g., visible light cameras, UV cameras, IR cameras, depth cameras, thermal cameras), microphones, directional microphone arrays (i.e., a beamforming microphone array), pressure sensors, thermometers, motion detectors, proximity sensors, accelerometers, global positioning satellite (GPS) receivers, magnetometers, radar systems, lidar systems, environmental monitoring devices (e.g., smoke detectors, carbon monoxide detectors), barometers, health monitoring devices (e.g., electrocardiographs, sphygmomanometers, electroencephalograms), automotive sensors (e.g., speedometers, odometers, tachometers, fuel sensors), and/or any other sensors or devices that collect and/or store information pertaining to the identity, position, and/or current status of one or more people or other entities. In some examples, the entity tracker 300 may occupy a common device housing with one or more of the plurality of sensors 302, and/or the entity tracker and its associated sensors may be distributed across multiple devices configured to communicate via one or more network communications interfaces (e.g., Wi-Fi adapters, Bluetooth interfaces).
As shown in the example of
Depending on the specific implementation, any or all of the functions associated with the entity identifier, person identifier, position identifier, and status identifier may be performed by the individual sensors 302A-302C. Though the present description generally describes the entity tracker 300 as receiving data from sensors, this does not require that the entity identifier 304, as well as other modules of the entity tracker, must be implemented on a single computing device that is separate and distinct from the plurality of sensors associated with the entity tracker. Rather, functions of the entity tracker 300 may be distributed amongst the plurality of sensors, or other suitable devices. For example, rather than sending raw sensor data to the entity tracker, individual sensors may be configured to attempt to identify entities that they detect, and report this identification to the entity tracker 300, and/or other modules of computing system 200. Furthermore, to simplify descriptions below, the term “sensor” is sometimes used to describe not only the physical measurement device (e.g., microphone or camera), but also the various logic processors configured and/or programmed to interpret signals/data from the physical measurement devices. For example, a “microphone” may be used to refer to the device that translates acoustic energy to an electrical signal, the analog-to-digital converter that converts the electrical signal to digital data, the on-board application-specific-integrated-circuit that pre-processes the digital data, and the downstream modules described herein (e.g., entity tracker 300, entity identifier 304, voice listener 216, or parser 218). As such, reference to a generic “sensor” or a particular sensor (e.g., “microphone” or “camera”) should not be construed to mean only the physical measurement device, but also the cooperating modules/engines, which can be distributed across one or more computers.
Each of the entity identifier 304, person identifier 305, position identifier 306, and status identifier 308 is configured to interpret and evaluate sensor data received from the plurality of sensors 302, and to output context information 310 based on the sensor data. Context information 310 may include the entity tracker's guesses/predictions as to an identity, position, and/or status of one or more detected entities based on received sensor data. As will be described in more detail below, each of the entity identifier 304, person identifier 305, position identifier 306, and status identifier 308 may output their predictions/identifications along with a confidence value.
The entity identifier 304, person identifier 305, position identifier 306, status identifier 308, and other processing modules described herein may utilize one or more machine-learning technologies. Non-limiting examples of such machine-learning technologies can include Feedforward Networks, Recurrent Neural Networks (RNN), Long Short-term Memory (LSTM), Convolutional Neural Networks, Support-vector Machines (SVM), Generative-Adversarial Networks (GAN), Variational Autoencoders, Q-Learning, and Decision Trees. The various identifiers, engines, and other processing blocks described herein may be trained via supervised and/or unsupervised learning utilizing these, or any other appropriate, machine learning technologies to make the described assessments, decisions, identifications, etc. It should be understood, however, that this description is not intended to put forth new technologies for making such assessments, decisions, identifications, etc. Instead, this description is intended to manage computational resources, and as such, is meant to be compatible with any type of processing module.
The entity identifier 304 may output an entity identity 312 of a detected entity, and such entity identity may have any suitable degree of specificity. In other words, based on received sensor data, the entity tracker 300 may predict the identity of a given entity, and output such information as entity identity 312. For example, the entity identifier 304 may report that a particular entity is a piece of furniture, a computing device, a dog, a human male, etc. Additionally, or alternatively, the entity identifier 304 may report that a particular entity is an intelligent assistant computing device with a particular model number and/or user-assigned device name; a pet dog with a specific name and breed; an owner or known user of an intelligent assistant computing device, the owner/known user having a particular name and profile; etc. In some examples, the degree of specificity with which the entity identifier 304 identifies/classifies detected entities may depend on one or more of user preferences and sensor limitations. In some cases, the entity identity output by the entity identifier may simply be a generic identifier that provides no information regarding the nature of the tracked entity, but rather is used to distinguish one entity from another.
The position identifier 306 may be configured to output an entity position (i.e., location) 314 of a detected entity. In other words, the position identifier 306 may predict the current position of a given entity based on collected sensor data, and output such information as entity position 314. As with the entity identity 312, the entity position 314 may have any suitable level of detail, and this level of detail may vary with user preferences and/or sensor limitations. For example, the position identifier 306 may report that a detected entity has a two-dimensional position defined on a plane such as a floor or wall. Additionally, or alternatively, the reported entity position 314 may comprise a three-dimensional position of a detected entity within a real world, three-dimensional environment. In some examples an entity position 314 may comprise a GPS position, a location within an environment-relative coordinate system, etc. In other examples, the entity position may be defined relative to the position of the intelligent assistant computing device. As will be described in more detail below, after performing position calibration, an entity tracker may in some cases calculate a distance between two different intelligent assistant computing devices.
The reported entity position 314 for a detected entity may correspond to the entity's geometric center, a particular part of the entity that is classified as being important (e.g., the head of a human), a series of boundaries defining the borders of the entity in three-dimensional space, etc. The position identifier 306 may further calculate one or more additional parameters describing the position and/or orientation of a detected entity, such as a pitch, roll, and/or yaw parameter. In other words, the reported position of a detected entity may have any number of degrees-of-freedom, and may include any number of coordinates defining the position of the entity in an environment. In some examples, an entity position 314 of a detected entity may be reported even if the entity tracker 300 is unable to identify the entity, and/or determine the current status of the entity.
Status identifier 308 may be configured to output an entity status 316 of a detected entity. In other words, the entity tracker 300 may be configured to predict the current status of a given entity based on received sensor data, and output such information as entity status 316. “Entity status” can refer to virtually any measurable or classifiable property, activity, or behavior of a given entity. For example, when applied to a person, the entity status of the person can indicate a posture of the person (e.g., standing, sitting, laying down), a speed at which the person is walking/running, a current activity of the person (e.g., sleeping, watching TV, working, playing a game, swimming, talking on the phone), a current mood of the person (e.g., by evaluating the person's facial expression or tone of voice), biological/physiological parameters of the person (e.g., the person's heart rate, respiration rate, oxygen saturation, body temperature, neurological activity), whether the person has any current or upcoming calendar events/appointments, etc. “Entity status” can refer to additional/alternative properties or behaviors when applied to other creatures or non-living objects, such as a current temperature of an oven or kitchen sink, whether a device (e.g., television, lamp, microwave) is powered on, whether a door is open, etc.
Upon determining one or more of the entity identity 312, entity position 314, and entity status 316, such information may be sent as context information 310 to any of a variety of external modules or devices, where it may be used in a variety of ways. For example, context information 310 may be used by commitment engine 222 to manage commitments and associated messages and notifications. In some examples, context information 310 may be used by commitment engine 222 to determine whether a particular message, notification, or commitment should be executed and/or presented to a user. Similarly, context information 310 may be utilized by voice listener 216 when interpreting human speech or activating functions in response to a keyword trigger.
Each of entity identity 312, entity position 314, and entity status 316 may take any suitable form. For example, each of the entity identity 312, position 314, and status 316 may take the form of a discrete data packet including a series of values and/or labels describing the information gathered by the entity tracker. Each of the entity identity 312, position 314, and status 316 may additionally include a confidence value defining a statistical likelihood that the information is accurate. For example, if the entity identifier 304 receives sensor data that strongly indicates that a particular entity is a human male named “John Smith,” then entity identity 312 may include this information along with a corresponding relatively high confidence value, such as 90% confidence. If the sensor data is more ambiguous, then the confidence value included in entity identity 312 correspondingly may be relatively lower, such as 62%. In some examples, separate predictions may be assigned separate confidence values. For example, the entity identity 312 may indicate with 95% confidence that a particular entity is a human male, and indicate with a 70% confidence that the entity is John Smith. Such confidence values (or probabilities) may be utilized by a cost function in generating cost calculations for providing messages or other notifications to a user and/or performing action(s).
As noted above, in some examples the entity tracker 300 may be implemented in a single computing device. In other examples, one or more functions of the entity tracker 300 may be distributed across multiple computing devices working cooperatively. For example, one or more of the entity identifier 304, person identifier 305, position identifier 306, and status identifier 308 may be implemented on different computing devices, while still collectively comprising an entity tracker configured to perform the functions described herein. As indicated above, any or all of the functions of the entity tracker may be performed by individual sensors 302. Further, in some examples entity tracker 300 may omit one or more of the entity identifier 304, person identifier 305, position identifier 306, and status identifier 308, and/or include one or more additional components not described herein, while still providing context information 310. Additional details regarding components and computing aspects that may be used to implement entity tracker 300 are described in more detail below with respect to
Returning briefly to
This problem may be mitigated by distributing additional intelligent assistant computing devices throughout the environment, thereby increasing the available space in which intelligent assistance is available. Such a distributed arrangement may be referred to as a “mesh” of devices. In other words, should human user 100 leave living room 102, she may direct additional natural language inputs to a different intelligent assistant computing device located in a different room. However, as discussed above, this scenario has its own associated drawbacks, for example when two different intelligent assistant computing devices receive the same natural language input and attempt to respond, or when intelligent assistant computing device 104 remains active even when the human user has left living room 102.
This can be at least partially alleviated when individual intelligent assistant computing devices in the environment each have at least some information regarding their positions relative to each other and/or the overall environment. Such information may be derived in a variety of ways, as will be discussed in more detail below. In some examples, one or both of the intelligent assistant computing devices may emit a position calibration signal detectable by the other intelligent assistant computing device(s). Upon receiving the position calibration signal, an intelligent assistant computing device can analyze the signal to calculate the distance between the intelligent assistant computing device that received the position calibration signal and the intelligent assistant computing device that emitted the position calibration signal. This may involve, for example, multiplying the time it took the position calibration signal to travel between the two devices with a known propagation speed of the position calibration signal. As used herein, processes through which an intelligent assistant computing device receives information regarding the position of another intelligent assistant computing device are referred to as “position calibration.”
Depending on the nature of the position calibration signal, calculating the distance between the two intelligent assistant computing devices will sometimes be more accurate when one intelligent assistant computing device has accurate information regarding when the position calibration signal was emitted by the other intelligent assistant computing device. With respect to
At 402, method 400 includes, at the first intelligent assistant computing device, syncing to a reference clock of a wireless computer network. Notably, the reference clock is supported by the second intelligent assistant computing device, which is communicatively coupled to the first intelligent assistant computing device via the wireless computer network.
Different intelligent assistant computing devices may each maintain their own internal clock., which can complicate the coordination between the devices involved in position calibration. Accordingly, during synchronization, the internal clock of either or both devices may be adjusted to match the reference clock. Synchronizing to the reference clock can therefore involve one device adjusting its internal clock to match the internal clock of the other device (i.e., the second intelligent assistant computing device provides the reference clock), or both devices adjusting their internal clocks to match a reference clock provided by a 3rd party. In this manner, the two intelligent assistant computing devices can achieve a shared sense of time.
In other examples, synchronization may occur without adjusting or altering a device's internal clock. Rather, for example, an intelligent assistant computing device may merely record an offset between its own internal clock and the reference clock, such that it can report events to other devices relative to the reference clock rather than its own internal clock.
It will be understood that the reference clock can take any suitable form. As examples, the reference clock may be the internal clock maintained by an intelligent assistant computing device, the reference clock as modified by a sync offset, a clock maintained by local network infrastructure (e.g., a wireless router or modem), a clock maintained by a remote timekeeping service (e.g., an Internet Service Provider or intelligent assistant computing device manufacturer), etc.
Method 400 is schematically illustrated with respect to
As discussed above, during position calibration, one intelligent assistant computing device may emit a position calibration signal that is received and analyzed by the other intelligent assistant computing device to calculate the distance between the two devices, and/or derive information regarding the devices' relative positions.
Turning back to
In some cases, the signal emission time may be defined relative to the reference clock. For example, if the reference clock is currently at 0:00:00, then the signal emission time may indicate that the position calibration signal was emitted at 0:00:00, even if, for example, the internal clock of the second intelligent assistant computing device currently reports a different time. In a different scenario, the signal emission time may be reported according to the internal clock of the second intelligent assistant, along with accompanying data describing the offset between the reported time and the reference clock. The signal emission time may be expressed in other suitable ways, for instance by reference to a numbered time frame recorded by the second intelligent assistant computing device. In an example scenario, both intelligent assistant computing devices may establish the moment of synchronization with the reference clock as “time frame 0,” and begin tracking time frames synchronously thereafter. Each time frame may last any suitable length of time, such as, for example, one millisecond.
In some examples, the communication may be sent before the position calibration signal is emitted, in which case the signal emission time will be a future time. In other examples, the communication may be sent simultaneously with the emission of the position calibration signal, or after the emission of the position calibration signal. In general, the signal emission time may be reported in any manner or format useable by the first intelligent assistant computing device to accurately infer when the position calibration signal was or will be emitted.
Second intelligent assistant computing device 502B has also emitted a position calibration signal 512. The position calibration signal may take a variety of suitable forms, and be transmitted over any suitable medium. In the example of
Returning briefly to
In
In addition to or as an alternative to recording the signal detection time, the first intelligent assistant computing device may record a direction from which the position calibration signal was received (e.g., when the position calibration signal is detected via a beamforming microphone array), a volume/amplitude of the detected position calibration signal, and/or other suitable parameters. The direction from which the position calibration signal was detected can in some cases correspond to the direction at which the second intelligent assistant computing device is positioned relative to the first intelligent assistant computing device. Similarly, the volume/amplitude of the detected position calibration signal can aid in calculating the distance between the first and second intelligent assistant computing devices when an original emission volume/amplitude of the position calibration signal is also known.
Returning briefly to
Furthermore, the term “calculation” is used herein broadly, and may refer to one or more operations not locally performed by the first intelligent assistant computing device. Calculating may in some cases include sending data (e.g., the signal detection and emission times, other data relating to the position calibration signal, environment, or intelligent assistant computing devices) to one or more other local or remote computing devices for processing and analysis. In one example, such data may be sent to a cloud service, which may perform some or all of the calculations described herein, and transmit the results of such calculations to any or all intelligent assistant computing devices in the environment.
As indicated above, an intelligent assistant computing device may emit other types of position calibration signals besides sound.
At 602, method 600 includes, at a first intelligent assistant computing device, emitting a position calibration signal via a signal emitter. With respect to
In the example of
Returning to
This is illustrated in
Turning again to
The specific calculations performed when estimating the relative positions of the intelligent assistant computing devices will generally depend on the specific parameters recorded. In some cases, when the position calibration signal is detected at a particular position within a FOD of a sensor (e.g., defined by a sensor-relative coordinate system), the FOD-relative position can be translated into a real-world position defined relative to a real-world environment of the intelligent assistant computing device. For example, when a camera detects an emission of light, then the detected intensity can be used to confirm whether the detection corresponds to a direct line-of-sight between the two intelligent assistant computing devices or a reflection of the emission of light. If the zoom parameters and view direction of the camera are known, then the relative position of an intelligent assistant device can be inferred based on detection of a position calibration signal at a particular FOD-relative position. Such calculations may be easier and/or more accurate when multiple instances of the position calibration signal are detected. It will be understood that the relative real-world positions of the intelligent assistant computing devices may be estimated in any of a variety of suitable ways, and the specific approaches used will vary depending on the available sensors and sensor data.
In some cases, estimating the relative position of the first intelligent assistant computing device can include calculating an orientation of the first intelligent assistant computing device relative to the second. For example, when the first intelligent assistant computing device includes a set of signal emitters, each signal emitter emitting a separate instance of the position calibration signal, then the second intelligent assistant device may record multiple detections of the position calibration signal corresponding to the separate instances. An observed spatial relationship of the multiple detections can then be compared to a known spatial relationship of the signal emitters to calculate the orientation of the first intelligent assistant computing device. In an example scenario, the set of signal emitters may be arranged in a triangle. If the multiple detections are arranged in a distorted triangle, then a computer-vision transform may be used to estimate the orientation of the intelligent assistant computing device that would produce the observed distorted triangle.
It will be understood that such orientation estimation may be performed in any suitable way, using any suitable publicly-known pose estimation techniques or computer vision algorithms. As an example, orientation estimation may be performed using the solvePnP algorithm from the Open Source Computer Vision Library. This algorithm generally works by comparing the observed positions of marker points (e.g., corners of a triangle, features of a human face) in a two-dimensional image to a known three-dimensional relationship of the marker points in an arbitrary coordinate system. Specifically, the known 3D relationship of the marker points is projected onto the 2D image plane, and, provided that intrinsic properties of the camera are known, the six-degree of freedom (6DOF) pose of the object (e.g., intelligent assistant computing device) is determined. Another suitable algorithm may be the posest algorithm, provided under an open source license by the Institute of Computer Science of the Foundation for Research and Technology—Hellas.
This is illustrated in
Returning to
Returning to
It will be understood that identifying the position of the human user may be done in any suitable way. In some examples, the position of the human user may be determined by an entity tracker of an intelligent assistant computing device, using any or all of the entity tracking techniques described above. Furthermore, the position of the human user may be compared to the positions of the intelligent assistant computing devices in any suitable way. In some examples, each of these positions may be expressed relative to the environment (e.g., using an environment-relative coordinate system), in which case it may be relatively trivial to calculate a shortest Euclidean distance between the positions of the human user and each of the intelligent assistant computing devices. When the positions are not all expressed relative to the environment, then the closest device to the human user may be identified in another suitable way, for instance by comparing the amplitude of the natural language input detected at each of the intelligent assistant computing devices, or identifying which of the intelligent assistant computing devices has a better view of the human user (e.g., via a camera). The decision of which of the intelligent assistant computing devices is closer to the position of the human user may be made by any suitable computing device, including either or both of the intelligent assistant computing devices, another computing device in the environment, a remote server, etc.
Returning to
In some scenarios, emission of a position calibration signal may be used for purposes in addition to or instead of determining the position of an intelligent assistant computing device. For example, emission of a position calibration signal may be used to determine information regarding the positions of one or more surfaces in an environment of the intelligent assistant computing devices. After one intelligent assistant computing device emits a position calibration signal, the position calibration signal may in some cases reflect off one or more reflection surfaces in the environment prior to being detected by the other intelligent assistant computing device. Based on analysis of the detected position calibration signal, information regarding the path taken by the position calibration signal can be derived, thereby providing at least some information regarding the layout of the environment. When a plurality of instances of the position calibration signal are detected, each having reflected off a different reflection surface, the intelligent assistant computing device may have sufficient information to estimate a layout of the environment. Further, other information, such as camera images, may be considered together with position calibration signal information to model an environment. Modelling the environment can further improve the ability of the intelligent assistant computing device to provide intelligent assistance, for instance by allowing the intelligent assistant computing device to identify points of entry/egress, notable furniture/appliances, better understand movements of human users throughout the environment, etc.
Reflection of a position calibration signal off a reflection surface in an environment is schematically illustrated in
The two emissions of sound are detected by a second intelligent assistant computing device 804B. While emission of sound 804A takes a direct path between the first and second intelligent assistant computing devices, emission of sound 804B reflects off reflection surface 806 en route. In other words, second emission of sound 804B may correspond to an echo of the original emission of sound, reaching the second intelligent assistant device at a measurable length of time after the initial detection of the position calibration signal. The second intelligent assistant computing device will therefore record a first signal detection time of a first detection of the emission of sound (i.e., emission of sound 804A), and a second detection of the emission of sound (i.e., emission of sound 804B), the second detection corresponding to a reflection of the emission of sound off a reflection surface in the environment. The second intelligent assistant computing device may then compare the time-of-flight for the second detection of the emission of sound to the known propagation speed of sound to calculate the total distance traveled by the second emission of sound, as described above with respect to
As discussed above, an emission of sound from an intelligent assistant computing device will typically take the form of a complex three-dimensional sound waveform, portions of which may end up reflecting off any or all of the surfaces in the local environment prior to being detected by the second intelligent assistant computing device. Because the emission time of the emission of sound is known, the multiple detections of the emission of sound can be used as the basis for performing echolocation using suitable sound processing techniques. In this manner, based on a plurality of detections of the emission of sound, the second intelligent assistant computing device and/or another suitable computing device can calculate a plurality of distances between the first intelligent assistant computing devices and a plurality of reflection surfaces in the environment. This information may be useable to estimate a layout of the environment, which can present advantages with respect to intelligent assistance as described above.
Reflections of the position calibration signal may also be observed when the position calibration signal is an emission of light. This is illustrated in
As discussed above with respect to
At 902, method 900 includes, at a first intelligent assistant computing device, detecting presence of a human user at a first detection position within a field-of-detection (FOD) of the first intelligent assistant computing device. As an example, the first intelligent assistant computing device may include one or more cameras, each having a FOD of an environment of the intelligent assistant computing device. The human user may be detected at a particular position (e.g., pixel position) within the FOD of the camera, corresponding to the first detection position.
This is illustrated in
Returning to
Returning to
Returning to
In some cases, after position calibration is complete, the position of an intelligent assistant computing device may change, either through accidental movement or deliberate repositioning by a human user. When this occurs, the position information obtained via position calibration may be rendered out-of-date, which can diminish or negate the advantages achieved through performing position calibration in the first place.
Accordingly, in some cases, upon detecting a change in position of an intelligent assistant computing device, an updated distance between the intelligent assistant computing devices may be calculated. This may be done in any of a variety of suitable ways, including updating the distance based on changes detected in a FOD of a camera, updating the distance based on data output by a motion sensor, simply repeating any or all of the position calibration steps described above, and/or other suitable processes.
In
Additionally, or alternatively, the change in position of the first intelligent assistant computing device may be detected by one or more motion sensors of the first intelligent assistant computing device (e.g., accelerometers, magnetometers, gyroscopes, inertial measurement units). The change in position of the first intelligent assistant computing device may then be calculated based on data output by the one or more motion sensors, and such data may be processed, fused, combined, etc., in any suitable way.
This is schematically illustrated in
Furthermore, additionally or alternatively to the processes described above, upon detecting a change in its position, the first intelligent assistant computing device may instruct the second intelligent assistant computing device to output a new position calibration signal. This may result in, for example, steps of methods 400 and/or 600 described above being performed, allowing the first intelligent assistant computing device to calculate an updated distance from and/or position relative to the second intelligent assistant computing device.
Though the above description focused on calculating distances between and/or relative positions of pairs of intelligent assistant computing devices, it will be understood that the position calibration techniques described herein may be applied to any arbitrary number of intelligent assistant computing devices. For example, some implementations may feature three, four, five, or more intelligent assistant computing devices. When more than two intelligent assistant computing devices are used, two different position calibration signals may be received by the same intelligent assistant computing device, potentially allowing the device to more accurately triangulate its distance from and/or position relative to the other intelligent assistant devices. Furthermore, though the position calibration techniques described above were discussed separately (e.g., in methods 400, 600, and 900), it will be understood that a single intelligent assistant computing device may utilize any or all of the described techniques, either in sequence or in parallel, and/or use position calibration techniques not explicitly described herein.
In some embodiments, the methods and processes described herein may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.
Computing system 1300 includes a logic machine 1302 and a storage machine 1304. Computing system 1300 may optionally include a display subsystem 1306, input subsystem 1308, communication subsystem 1310, and/or other components not shown in
Logic machine 1302 includes one or more physical devices configured to execute instructions. For example, the logic machine may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.
The logic machine may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. Processors of the logic machine may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic machine optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic machine may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.
Storage machine 1304 includes one or more physical devices configured to hold instructions executable by the logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machine 1304 may be transformed—e.g., to hold different data.
Storage machine 1304 may include removable and/or built-in devices. Storage machine 1304 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage machine 1304 may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.
It will be appreciated that storage machine 1304 includes one or more physical devices. However, aspects of the instructions described herein alternatively may be propagated by a communication medium (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by a physical device for a finite duration.
Aspects of logic machine 1302 and storage machine 1304 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.
The terms “module,” “program,” and “engine” may be used to describe an aspect of computing system 1300 implemented to perform a particular function. In some cases, a module, program, or engine may be instantiated via logic machine 1302 executing instructions held by storage machine 1304. It will be understood that different modules, programs, and/or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and/or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms “module,” “program,” and “engine” may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.
It will be appreciated that a “service”, as used herein, is an application program executable across multiple user sessions. A service may be available to one or more system components, programs, and/or other services. In some implementations, a service may run on one or more server-computing devices.
When included, display subsystem 1306 may be used to present a visual representation of data held by storage machine 1304. This visual representation may take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 1306 may likewise be transformed to visually represent changes in the underlying data. Display subsystem 1306 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machine 1302 and/or storage machine 1304 in a shared enclosure, or such display devices may be peripheral display devices.
When included, input subsystem 1308 may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may comprise or interface with selected natural user input (NUI) componentry. Such componentry may be integrated or peripheral, and the transduction and/or processing of input actions may be handled on- or off-board. Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.
When included, communication subsystem 1310 may be configured to communicatively couple computing system 1300 with one or more other computing devices. Communication subsystem 1310 may include wired and/or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem may allow computing system 1300 to send and/or receive messages to and/or from other devices via a network such as the Internet.
In an example, a method comprises: at a first intelligent assistant computing device configured to receive and respond to natural language inputs provided by human users, syncing to a reference clock of a wireless computer network, the reference clock being supported by a second intelligent assistant computing device communicatively coupled to the first intelligent assistant computing device via the wireless computer network; receiving a communication sent by the second intelligent assistant computing device over the wireless computer network, the communication indicating a signal emission time at which the second intelligent assistant computing device emitted a position calibration signal, the signal emission time being defined relative to the reference clock; recording a signal detection time at which the position calibration signal was detected by the first intelligent assistant computing device, the signal detection time being defined relative to the reference clock; and based at least on a difference between the signal emission time and the signal detection time and a known propagation speed of the position calibration signal, calculating a distance between the first intelligent assistant computing device and the second intelligent assistant computing device. In this example or any other example, the position calibration signal is an emission of sound. In this example or any other example, the emission of sound includes a plurality of different frequencies, one or more of the plurality of frequencies being an ultrasonic frequency. In this example or any other example, the method further comprises recording an amplitude of the detected position calibration signal, and calculating the distance further includes comparing the recorded amplitude to a known emission amplitude of the position calibration signal. In this example or any other example, the first intelligent assistant computing device detects the emission of sound via a beamforming microphone array, and the method further comprises recording a direction from which the emission of sound was detected. In this example or any other example, recording the signal detection time includes recording a first signal detection time of a first detection of the emission of sound and a second signal detection time of a second detection of the emission of sound, the second detection of the emission of sound corresponding to a reflection of the emission of sound off a reflection surface in an environment of the first intelligent assistant computing device, and the method further comprises calculating a cumulative distance between a) the second intelligent assistant computing device and the reflection surface, and b) the reflection surface and the first intelligent assistant computing device. In this example or any other example, the method further comprises calculating a plurality of distances between the first intelligent assistant computing device and a plurality of reflection surfaces in the environment based on a plurality of detections of the emission of sound, and estimating a layout of the environment based on the plurality of distances. In this example or any other example, the method further comprises, upon detecting a change in position of the first intelligent assistant computing device, calculating an updated distance between the first and second intelligent assistant computing devices. In this example or any other example, the change in position of the first intelligent assistant computing device is detected by one or more motion sensors of the first intelligent assistant computing device, and the updated distance is calculated based on data output by the one or more motion sensors. In this example or any other example, the first intelligent assistant computing device includes a camera, detecting the change in position of the first intelligent assistant computing device includes detecting a change in images captured by the camera, and the updated distance is calculated based on the detected change. In this example or any other example, the method further comprises, upon detecting the change in position of the first intelligent assistant computing device, instructing the second intelligent assistant computing device to emit a new position calibration signal. In this example or any other example, the method further comprises: detecting a natural language input provided by a human user; based on the distance between the first and second intelligent assistant computing devices, determining which of the first and second intelligent assistant computing devices is closer to a position of the human user; and responding to the natural language input via the closer intelligent assistant computing device.
In an example, a method comprises: at a first intelligent assistant computing device, emitting a position calibration signal via a signal emitter, the position calibration signal being an emission of light; at a second intelligent assistant computing device, via one or more cameras, recording a set of parameters describing a detection of the position calibration signal; based on the set of parameters describing the detection, estimating relative positions of the first and second intelligent assistant computing devices; receiving a natural language input from a human user in an environment of the first and second intelligent assistant computing devices; based on the relative positions of the first and second intelligent assistant computing devices, determining which of the first and second intelligent assistant computing devices is closer to a position of the human user; and responding to the natural language input via the closer intelligent assistant computing device. In this example or any other example, the signal emitter is an infrared (IR) light emitter, and the emission of light is an emission of IR light. In this example or any other example, the first intelligent assistant computing device includes a set of signal emitters having a known spatial relationship, emitting the position calibration signal includes emitting a separate instance of the position calibration signal from each of the set of signal emitters, the second intelligent assistant computing device records multiple detections of the position calibration signal corresponding to separate instances of the position calibration signal, and the method further comprises, based on comparing a spatial relationship of the multiple detections to the known spatial relationship of the set of signal emitters, calculating an orientation of the first intelligent assistant computing device relative to the second intelligent assistant computing device. In this example or any other example, emitting the position calibration signal includes emitting at least one instance of the position calibration signal differently from other instances of the position calibration signal. In this example or any other example, the method further comprises recording a set of parameters describing a second detection of the position calibration signal corresponding to a reflection of the position calibration signal off a reflection surface in the environment, and, based on the set of parameters describing the second detection, estimating a relative position of the reflection surface.
In an example, a method comprises: at a first intelligent assistant computing device, detecting presence of a human user at a first detection position within a field-of-detection (FOD) of the first intelligent assistant computing device; localizing the human user to a real-world position relative to an environment of the first intelligent assistant computing device corresponding to the first detection position; via a wireless computer network, receiving, from a second intelligent assistant computing device, an indication that the second intelligent assistant computing device detected the human user at a second detection position within a FOD of the second intelligent assistant computing device, the second detection position also corresponding to the real-world position; and using the real-world position of the human user as a landmark, estimating real-world positions for each of the first and second intelligent assistant computing devices. In this example or any other example, the method further comprises: detecting a natural language input provided by the human user; comparing the real-world positions of the first and second intelligent assistant computing devices to the real-world position of the human user to identify which of the first and second intelligent assistant computing devices are closer to the human user; and responding to the natural language input via the closer intelligent assistant computing device. In this example or any other example, the method further comprises, upon detecting a change in position of the first intelligent assistant computing device, calculating an updated distance between the first and second intelligent assistant computing devices.
It will be understood that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated and/or described may be performed in the sequence illustrated and/or described, in other sequences, in parallel, or omitted. Likewise, the order of the above-described processes may be changed.
The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.
This application claims priority to U.S. Provisional Patent Application Ser. No. 62/459,020 filed Feb. 14, 2017, and to U.S. Provisional Patent Application Ser. No. 62/482,165 filed Apr. 5, 2017, the entirety of which are hereby incorporated herein by reference for all purposes.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6067673 | Paese et al. | May 2000 | A |
| 6119088 | Ciluffo | Sep 2000 | A |
| 6332122 | Ortega et al. | Dec 2001 | B1 |
| 6442524 | Ecker et al. | Aug 2002 | B1 |
| 6477500 | Maes | Nov 2002 | B2 |
| 6496799 | Pickering | Dec 2002 | B1 |
| 6574601 | Brown et al. | Jun 2003 | B1 |
| 6727925 | Bourdelais | Apr 2004 | B1 |
| 6728679 | Strubbe et al. | Apr 2004 | B1 |
| 6816730 | Davies et al. | Nov 2004 | B2 |
| 6873953 | Lennig | Mar 2005 | B1 |
| 7019749 | Guo et al. | Mar 2006 | B2 |
| 7050110 | Lienhart et al. | May 2006 | B1 |
| 7330566 | Cutler | Feb 2008 | B2 |
| 7475010 | Chao | Jan 2009 | B2 |
| 7610365 | Kraft et al. | Oct 2009 | B1 |
| 7716056 | Weng et al. | May 2010 | B2 |
| 7803050 | Mao et al. | Sep 2010 | B2 |
| 8139945 | Amir et al. | Mar 2012 | B1 |
| 8165087 | Panabaker | Apr 2012 | B2 |
| 8170875 | Hetherington et al. | May 2012 | B2 |
| 8213689 | Yagnik et al. | Jul 2012 | B2 |
| 8265252 | Ducheneaut et al. | Sep 2012 | B2 |
| 8326627 | Kennewick et al. | Dec 2012 | B2 |
| 8340975 | Rosenberger | Dec 2012 | B1 |
| 8374879 | Falcon et al. | Feb 2013 | B2 |
| 8453402 | Huang | Jun 2013 | B2 |
| 8457959 | Kaiser | Jun 2013 | B2 |
| 8543402 | Ma | Sep 2013 | B1 |
| 8639762 | Rasmussen et al. | Jan 2014 | B2 |
| 8644842 | Arrasvuori et al. | Feb 2014 | B2 |
| 8712758 | Crouch et al. | Apr 2014 | B2 |
| 8752145 | Dotan et al. | Jun 2014 | B1 |
| 8762150 | Edgington et al. | Jun 2014 | B2 |
| 8762156 | Chen | Jun 2014 | B2 |
| 8779965 | Sentelle et al. | Jul 2014 | B2 |
| 8805691 | Genly | Aug 2014 | B2 |
| 8861924 | Meads et al. | Oct 2014 | B2 |
| 8862156 | Bell et al. | Oct 2014 | B2 |
| 8885882 | Reale et al. | Nov 2014 | B1 |
| 8903128 | Shet et al. | Dec 2014 | B2 |
| 8913103 | Sargin et al. | Dec 2014 | B1 |
| 8942986 | Cheyer et al. | Jan 2015 | B2 |
| 8949359 | Rasmussen et al. | Feb 2015 | B2 |
| 9037601 | Palay | May 2015 | B2 |
| 9070366 | Mathias et al. | Jun 2015 | B1 |
| 9085303 | Wolverton et al. | Jul 2015 | B2 |
| 9119512 | Martins, Jr. et al. | Sep 2015 | B2 |
| 9123330 | Sharifi et al. | Sep 2015 | B1 |
| 9171542 | Gandrabur et al. | Oct 2015 | B2 |
| 9230544 | Kwon et al. | Jan 2016 | B2 |
| 9268406 | Geisner et al. | Feb 2016 | B2 |
| 9300925 | Zhang | Mar 2016 | B1 |
| 9307355 | Nehrenz et al. | Apr 2016 | B2 |
| 9311932 | Carter | Apr 2016 | B2 |
| 9318105 | Khosla | Apr 2016 | B1 |
| 9348990 | Chuaprasert et al. | May 2016 | B2 |
| 9368114 | Larson et al. | Jun 2016 | B2 |
| 9378740 | Rosen et al. | Jun 2016 | B1 |
| 9380177 | Rao et al. | Jun 2016 | B1 |
| 9389681 | Sankar et al. | Jul 2016 | B2 |
| 9412392 | Lindahl | Aug 2016 | B2 |
| 9424840 | Hart et al. | Aug 2016 | B1 |
| 9466286 | Hart et al. | Oct 2016 | B1 |
| 9495331 | Govrin et al. | Nov 2016 | B2 |
| 9495613 | Holz et al. | Nov 2016 | B2 |
| 9507977 | Mor et al. | Nov 2016 | B1 |
| 9508341 | Parlikar et al. | Nov 2016 | B1 |
| 9514227 | Garrett et al. | Dec 2016 | B1 |
| 9558749 | Secker-Walker et al. | Jan 2017 | B1 |
| 9576574 | van Os | Feb 2017 | B2 |
| 9622059 | Bouzid et al. | Apr 2017 | B2 |
| 9626352 | Allen et al. | Apr 2017 | B2 |
| 9633652 | Kumiawati et al. | Apr 2017 | B2 |
| 9669296 | Hibbert et al. | Jun 2017 | B1 |
| 9749583 | Fineberg et al. | Aug 2017 | B1 |
| 9761055 | Miller | Sep 2017 | B2 |
| 9767616 | Miller | Sep 2017 | B2 |
| 9842299 | Stolarz et al. | Dec 2017 | B2 |
| 9898250 | Williams et al. | Feb 2018 | B1 |
| 9965247 | Jarvis et al. | May 2018 | B2 |
| 10178301 | Welbourne et al. | Jan 2019 | B1 |
| 10276149 | Liang et al. | Apr 2019 | B1 |
| 10482885 | Moniz | Nov 2019 | B1 |
| 10599390 | Brahmbhatt et al. | Mar 2020 | B1 |
| 20030103647 | Rui et al. | Jun 2003 | A1 |
| 20030131064 | Bell et al. | Jul 2003 | A1 |
| 20050182627 | Tanaka et al. | Aug 2005 | A1 |
| 20050216264 | Attwater et al. | Sep 2005 | A1 |
| 20050225427 | Bell et al. | Oct 2005 | A1 |
| 20050285774 | Wittenberg et al. | Dec 2005 | A1 |
| 20060028552 | Aggarwal et al. | Feb 2006 | A1 |
| 20060067536 | Culbert | Mar 2006 | A1 |
| 20070024487 | Zemany et al. | Feb 2007 | A1 |
| 20070100480 | Sinclair et al. | May 2007 | A1 |
| 20070152157 | Page | Jul 2007 | A1 |
| 20070198245 | Kamatani et al. | Aug 2007 | A1 |
| 20070271086 | Peters et al. | Nov 2007 | A1 |
| 20080030345 | Austin et al. | Feb 2008 | A1 |
| 20080071547 | Prieto et al. | Mar 2008 | A1 |
| 20080077015 | Boric-lubecke et al. | Mar 2008 | A1 |
| 20080195387 | Zigel et al. | Aug 2008 | A1 |
| 20080288251 | Cooper et al. | Nov 2008 | A1 |
| 20090066690 | Harrison | Mar 2009 | A1 |
| 20090303342 | Corcoran et al. | Dec 2009 | A1 |
| 20090319269 | Aronowitz | Dec 2009 | A1 |
| 20100073363 | Densham et al. | Mar 2010 | A1 |
| 20100100851 | Clark et al. | Apr 2010 | A1 |
| 20100179813 | Summerfield et al. | Jul 2010 | A1 |
| 20100195906 | Uliyar et al. | Aug 2010 | A1 |
| 20110010170 | Burns et al. | Jan 2011 | A1 |
| 20110119060 | Aronowitz | May 2011 | A1 |
| 20110184735 | Flaks et al. | Jul 2011 | A1 |
| 20110216090 | Woo et al. | Sep 2011 | A1 |
| 20110219339 | Densham | Sep 2011 | A1 |
| 20110298967 | Clavin et al. | Dec 2011 | A1 |
| 20110302535 | Clerc et al. | Dec 2011 | A1 |
| 20120026335 | Brown et al. | Feb 2012 | A1 |
| 20120253791 | Heck et al. | Oct 2012 | A1 |
| 20120265535 | Bryant-rich et al. | Oct 2012 | A1 |
| 20120268604 | Tree | Oct 2012 | A1 |
| 20130110519 | Cheyer et al. | May 2013 | A1 |
| 20130117377 | Miller | May 2013 | A1 |
| 20130144616 | Bangalore | Jun 2013 | A1 |
| 20130212501 | Anderson et al. | Aug 2013 | A1 |
| 20130253936 | Harvey | Sep 2013 | A1 |
| 20130259456 | Viswanathan | Oct 2013 | A1 |
| 20130304479 | Teller et al. | Nov 2013 | A1 |
| 20130342568 | Ambrus et al. | Dec 2013 | A1 |
| 20140033071 | Gruber et al. | Jan 2014 | A1 |
| 20140067679 | O'reilly et al. | Mar 2014 | A1 |
| 20140100997 | Mayerle et al. | Apr 2014 | A1 |
| 20140156276 | Nakano et al. | Jun 2014 | A1 |
| 20140160290 | Wu | Jun 2014 | A1 |
| 20140180629 | Dokmanic et al. | Jun 2014 | A1 |
| 20140214421 | Shriberg et al. | Jul 2014 | A1 |
| 20140214429 | Pantel | Jul 2014 | A1 |
| 20140222422 | Sarikaya et al. | Aug 2014 | A1 |
| 20140244263 | Pontual et al. | Aug 2014 | A1 |
| 20140272821 | Pitschel et al. | Sep 2014 | A1 |
| 20140330569 | Kolavennu et al. | Nov 2014 | A1 |
| 20140341440 | Walch | Nov 2014 | A1 |
| 20140365226 | Sinha | Dec 2014 | A1 |
| 20150016642 | Walsh et al. | Jan 2015 | A1 |
| 20150019714 | Shaashua et al. | Jan 2015 | A1 |
| 20150025887 | Sidi et al. | Jan 2015 | A1 |
| 20150032254 | Ishiguro | Jan 2015 | A1 |
| 20150032456 | Wait | Jan 2015 | A1 |
| 20150035976 | Mayuzumi | Feb 2015 | A1 |
| 20150102996 | Yim et al. | Apr 2015 | A1 |
| 20150134547 | Oikonomidis | May 2015 | A1 |
| 20150138332 | Cheng et al. | May 2015 | A1 |
| 20150149179 | Korbecki | May 2015 | A1 |
| 20150149182 | Kalns et al. | May 2015 | A1 |
| 20150162000 | Di censo et al. | Jun 2015 | A1 |
| 20150172285 | Lo et al. | Jun 2015 | A1 |
| 20150195666 | Massey | Jul 2015 | A1 |
| 20150220244 | Vats et al. | Aug 2015 | A1 |
| 20150249664 | Talhami et al. | Sep 2015 | A1 |
| 20150279368 | Contolini et al. | Oct 2015 | A1 |
| 20150340033 | Di fabbrizio et al. | Nov 2015 | A1 |
| 20150347114 | Yoon | Dec 2015 | A1 |
| 20150371639 | Foerster et al. | Dec 2015 | A1 |
| 20150382047 | Van os et al. | Dec 2015 | A1 |
| 20160019889 | Alvarez guevara et al. | Jan 2016 | A1 |
| 20160063989 | Deleeuw | Mar 2016 | A1 |
| 20160086018 | Lemoff | Mar 2016 | A1 |
| 20160088043 | Jiang et al. | Mar 2016 | A1 |
| 20160092732 | Black | Mar 2016 | A1 |
| 20160138247 | Conway et al. | May 2016 | A1 |
| 20160148417 | Kim et al. | May 2016 | A1 |
| 20160155443 | Khan et al. | Jun 2016 | A1 |
| 20160171289 | Lee | Jun 2016 | A1 |
| 20160173293 | Kennedy | Jun 2016 | A1 |
| 20160179831 | Gruber et al. | Jun 2016 | A1 |
| 20160187961 | Elibol et al. | Jun 2016 | A1 |
| 20160203002 | Kannan et al. | Jul 2016 | A1 |
| 20160210411 | Mentis | Jul 2016 | A1 |
| 20160217783 | Konuma et al. | Jul 2016 | A1 |
| 20160225373 | Casado et al. | Aug 2016 | A1 |
| 20160234595 | Goran et al. | Aug 2016 | A1 |
| 20160234616 | Gateau | Aug 2016 | A1 |
| 20160259623 | Sumner | Sep 2016 | A1 |
| 20160283185 | Mclaren et al. | Sep 2016 | A1 |
| 20160342702 | Barve et al. | Nov 2016 | A1 |
| 20160358598 | Williams et al. | Dec 2016 | A1 |
| 20160360336 | Gross et al. | Dec 2016 | A1 |
| 20160380929 | Katis et al. | Dec 2016 | A1 |
| 20170013409 | Cerchio et al. | Jan 2017 | A1 |
| 20170025124 | Mixter et al. | Jan 2017 | A1 |
| 20170032021 | Watanachote | Feb 2017 | A1 |
| 20170032787 | Dayal | Feb 2017 | A1 |
| 20170039423 | Cork et al. | Feb 2017 | A1 |
| 20170039602 | Shi-nash et al. | Feb 2017 | A1 |
| 20170068423 | Napolitano et al. | Mar 2017 | A1 |
| 20170078573 | Chen et al. | Mar 2017 | A1 |
| 20170133011 | Chen et al. | May 2017 | A1 |
| 20170140760 | Sachdev | May 2017 | A1 |
| 20170169476 | Nomula et al. | Jun 2017 | A1 |
| 20170185375 | Martel et al. | Jun 2017 | A1 |
| 20170186290 | Li et al. | Jun 2017 | A1 |
| 20170194000 | Itani et al. | Jul 2017 | A1 |
| 20170206900 | Lee et al. | Jul 2017 | A1 |
| 20170230705 | Pardue et al. | Aug 2017 | A1 |
| 20170236512 | Williams et al. | Aug 2017 | A1 |
| 20170242651 | Lang et al. | Aug 2017 | A1 |
| 20170249309 | Sarikaya | Aug 2017 | A1 |
| 20170255450 | Mullins et al. | Sep 2017 | A1 |
| 20170262472 | Goldenberg | Sep 2017 | A1 |
| 20170269975 | Wood et al. | Sep 2017 | A1 |
| 20170278480 | Sung et al. | Sep 2017 | A1 |
| 20170286530 | Paruchuri et al. | Oct 2017 | A1 |
| 20170287490 | Biswal et al. | Oct 2017 | A1 |
| 20170315208 | Sadr | Nov 2017 | A1 |
| 20170322939 | Byron et al. | Nov 2017 | A1 |
| 20170351749 | Quirk et al. | Dec 2017 | A1 |
| 20170357637 | Nell | Dec 2017 | A1 |
| 20170359666 | Lyren et al. | Dec 2017 | A1 |
| 20180009118 | Yamaga et al. | Jan 2018 | A1 |
| 20180047394 | Tian et al. | Feb 2018 | A1 |
| 20180048768 | Spittle et al. | Feb 2018 | A1 |
| 20180074785 | Ohmura | Mar 2018 | A1 |
| 20180090143 | Saddler | Mar 2018 | A1 |
| 20180091782 | Bashkin | Mar 2018 | A1 |
| 20180096696 | Mixter | Apr 2018 | A1 |
| 20180158454 | Campbell et al. | Jun 2018 | A1 |
| 20180199123 | Rao et al. | Jul 2018 | A1 |
| 20180218080 | Krishnamurthy et al. | Aug 2018 | A1 |
| 20180231653 | Pradeep et al. | Aug 2018 | A1 |
| 20180232201 | Holtmann | Aug 2018 | A1 |
| 20180232563 | Albadawi et al. | Aug 2018 | A1 |
| 20180232571 | Bathiche et al. | Aug 2018 | A1 |
| 20180232608 | Pradeep et al. | Aug 2018 | A1 |
| 20180232645 | Finkelstein et al. | Aug 2018 | A1 |
| 20180232662 | Solomon et al. | Aug 2018 | A1 |
| 20180232902 | Albadawi et al. | Aug 2018 | A1 |
| 20180233132 | Herold et al. | Aug 2018 | A1 |
| 20180233139 | Finkelstein et al. | Aug 2018 | A1 |
| 20180233140 | Koishida et al. | Aug 2018 | A1 |
| 20180233141 | Solomon et al. | Aug 2018 | A1 |
| 20180233142 | Koishida et al. | Aug 2018 | A1 |
| 20180260680 | Finkelstein et al. | Sep 2018 | A1 |
| 20180293221 | Finkelstein et al. | Oct 2018 | A1 |
| 20180314689 | Wang et al. | Nov 2018 | A1 |
| 20180333862 | Hayashi | Nov 2018 | A1 |
| 20190057703 | Zeinstra | Feb 2019 | A1 |
| 20200012906 | Albadawi et al. | Jan 2020 | A1 |
| 20200042839 | Herold et al. | Feb 2020 | A1 |
| 20200104653 | Solomon et al. | Apr 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 2947476 | Nov 2015 | EP |
| 2522922 | Aug 2015 | GB |
| 1020070016280 | Feb 2007 | KR |
| 2007018523 | Feb 2007 | WO |
| 2010104772 | Sep 2010 | WO |
| 2013061268 | May 2013 | WO |
| 2015012449 | Jan 2015 | WO |
| 2016114922 | Jul 2016 | WO |
| 2016162678 | Oct 2016 | WO |
| 2016205419 | Dec 2016 | WO |
| Entry |
|---|
| “Final Office Action Issued in U.S. Appl. No. 15/640,251”, dated Apr. 2, 2019, 22 Pages. |
| Miro, et al., “Speaker Diarization: A review of Recent Research”, In the Proceedings of IEEE Transactions on Audio, Speech and Language Processing, vol. 20, Issue 2, Feb. 1, 2012, 15 Pages. |
| Moattar, et al., “A Review on Speaker Diarization Systems and Approaches”, In the Publication of Speech Communication , vol. 54, Issue 10, Dec. 12, 2010, 39 Pages. |
| “International Search Report & Written Opinion for PCT Patent Application No. PCT/US2018/062384”, dated Feb. 15, 2019, 12 Pages. |
| Yu, et al., “Smart Meeting Systems: A Survey of State of the Art and Open Issues”, In the Proceedings of ACM Computing Surveys, vol. 42, No. 2, Mar. 5, 2010, 20 Pages. |
| “Non Provisional Application Filed in U.S. Appl. No. 15/885,518”, filed Jan. 31, 2018, 40 Pages. |
| “Non-Final Office Action Issued in U.S. Appl. No. 15/646,871”, dated Sep. 3, 2019, 23 Pages. |
| “Final Office Action Issued in U.S. Appl. No. 15/657,822”, dated Aug. 22, 2019, 22 Pages. |
| “Final Office Action Issued in U.S. Appl. No. 15/832,656”, dated Aug. 23, 2019, 10 Pages. |
| Constine, Jose, “Instagram launches selfie filters, copying the last big Snapchat feature”, Retrieved From. https://techcrunch.com/2017/05/16/instagram-face-filters/, May 16, 2017, 8 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2019/022836”, dated Jun. 24, 2019, 15 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2019/029558”, dated Jun. 28, 2019, 10 Pages. |
| “Sara: the Socially Aware Robot Assistant”, Retrieved from: https://web.archive.org/web/20160707141922/http:/articulab.hcii.cs.cmu.edu:80/projects/sara/, Jul. 7, 2017, 10 Pages. |
| Arsikere, et al., “Computationally-efficient Endpointing Features for Natural Spoken Interaction with Personal-Assistant Systems”, In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, May 4, 2014, pp. 3241-3245. |
| Ferrer, et al., “Is the Speaker Done Yet? Faster and More Accurate End-of-Utterance Detection using Prosody”, In the proceedings of Seventh International Conference on Spoken Language Processing, Sep. 16, 2002, pp. 2061-2064. |
| Lacharite, Noelle, “Updated: Alexa Skills Kit Fact Template: Step-by-Step Guide to Build a Fact Skill”, Retrieved from: https://developer.amazon.com/blogs/post/Tx3DVGG0K0TPUGQ/New-Alexa-Skills-Kit-Template:-Step-by-Step-Guide-to-Build-a-Fact-Skill, Mar. 29, 2016, 33 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017140”, dated May 18, 2018, 12 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017514”, dated May 17, 2018, 12 Pages. |
| Porcheron, et al., “Do Animals Have Accents?”: Talking with Agents in Multi-Party Conversation, In Proceedings of 20th ACM Conference on Computer-Supported cooperative Work and Social Computing, Feb. 25, 2017, 14 Pages. |
| Xiang, Li, “Improving Knowledge Base Population With Information Extraction”, A Thesis Submitted in Partial fullfillment of the Requirements of the University of New York for the Degree of Doctor of Philosophy, May 2016, 131 Pages. |
| Yun-Nung, Chen, “Unsupervised Learning and Modeling of Knowledge and Intent for Spoken Dialogue Systems”, In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, Jul. 28, 2015, 8 Pages. |
| Zhang, et al., “A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding”, In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, Jul. 9, 2016, pp. 2993-2999. |
| “Notice of Allowance Issued in U.S. Appl. No. 16/573,677”, dated Nov. 6, 2019, 9 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/832,656”, dated Jan. 6, 2020, 9 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/657,822”, dated Feb. 6, 2020, 25 Pages. |
| “Final Office Action Issued in U.S. Appl. No. 15/646,871”, dated Jan. 21, 2020, 23 Pages. |
| “Final Office Action Issued in U.S. Appl. No. 15/640,251”, dated Jan. 30, 2020, 21 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 16/005,470”, dated Feb. 24, 2020, 11 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/640,251”, dated Sep. 12, 2019, 21 Pages. |
| Staff, Appleinsider, “Amazon Alexa's ‘Follow-Up Mode’ Enables Successive Requests Without Trigger Word”, Retrieved From https://appleinsider.com/articles/18/03/09/amazon-alexas-follow-up-mode-enables-successive-requests-without-trigger-word, Mar. 9, 2018, 7 Pages. |
| “Multiple Agents (Each Trained for Different Domain) for One Chat Bot?”, Retrieved From https://discuss.api.ai/t/multiple-agents-each-trained-for-different-domain-for-one-chat-bot/1002, Jul. 1, 2016, 1 Page. |
| “Train the Natural Language Processing Classifiers”, Retrieved From https://www.mindmeld.com/docs/train_the_natural_language_processing_classifiers.html, Retrieved on: May 2, 2017, 10 Pages. |
| “Using Multiple Alexa Devices”, Retrieved From https://www.amazon.com/gp/help/customer/display.html?nodeId=202013740, Apr. 24, 2017, 2 Pages. |
| “Application Filed in U.S. Appl. No. 15/395,961”, filed Dec. 30, 2016, 79 Pages. |
| Ballan, et al., “Event Detection and Recognition for Semantic Annotation of Video”, In Journal of Multimedia Tools and Applications, vol. 51, Issue 1, Nov. 10, 2010, pp. 279-302. |
| Beltagy, et al., “Improved Semantic Parsers for If-Then Statements”, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol. 1, Aug. 7, 2016, pp. 726-736. |
| Boakye, et al., “Overlapped Speech Detection for Improved Speaker Diarization in Multiparty Meetings”, In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, Mar. 31, 2008, 4 Pages. |
| Cho, et al., “A Multi-Sensor Fusion System for Moving Object Detection and Tracking in Urban Driving Environments”, In IEEE International Conference on Robotics & Automation, May 31, 2014, 8 Pages. |
| Fossard, et al., Between Anaphora and Deixis . . . The Resolution of the Demonstrative Noun Phrase “that N”, In Journal of Language and Cognitive Processes, vol. 27, Issue 9, Nov. 2, 2011, 3 Pages. |
| Gebhart, Andrew, “How to bring Alexa into every room of your home”, Retrieved From https://www.cnet.com/how-to/how-to-install-alexa-in-every-room-of-your-home/, Feb. 2, 2017, 7 Pages. |
| Goncalves, et al., “Assessing Users' Emotion at Interaction Lime: A Multimodal Approach With Multiple Sensors”, In Proceedings of Soft Computing, vol. 21, Issue 18, Sep. 1, 2017, 8 Pages. |
| Goswami, et al., “A Reviewon Low Light Image Enhancement Using Image Processing Technique”, In International Journal of Technical Research, vol. 5, Issue 1, Mar. 2016, pp. 60-62. |
| He, et al., “Sensor Scheduling for Target Tracking: A Monte Carlo sampling approach”, In Journal of Digital Signal Processing, vol. 16, Issue 5, Sep. 2006, pp. 533-545. |
| Huijbregts, et al., “Speech Overlap Detection in a Two-Pass Speaker Diarization System”, In Proceedings of 10th Annual Conference of the International Speech Communication, Sep. 6, 2009, pp. 1063-1066. |
| Kabadjov, Mijail Alexandrov., “A Comprehensive Evaluation of Anaphora Resolution and Discourse—new Classification”, In thesis of University of Essex, May 2007, 266 Pages. |
| Kang, et al., “Detection and Tracking of Moving Objects from Overlapping EO and IR Sensors”, In Conference on Computer Vision and Pattern Recognition Workshop, Jun. 27, 2004, 6 Pages. |
| Kozhaya, Joe, “10 Steps to Train an Effective Chatbot and its Machine Learning Models”, Retrieved From https://developer.ibm.com/dwblog/2016/10-steps-train-chat-bot-chatbot-machine-learning/, Dec. 12, 2016, 7 Pages. |
| Li et al., “A Multiple-Camera System Calibration Toolbox Using a Feature Descriptor-based Calibration Pattern”, In Proceedings of IEEE International Conference on Intelligent Robots and Systems, Nov. 3, 2013, pp. 1301-1307. |
| Liu, et al., “Reliable Multiple Object Tracking under Heavy Occlusions”, In Proceedings on Intelligence Information Processing and Trusted Computing (IPTC), International Symposium, Oct. 28, 2010, 3 Pages. |
| Mengusoglu, Erhan, “Confidence Measures for Speech/Speaker Recognition and Applications on Turkish LVCSR”, Retrieved From https://web.archive.org/web/20040619044603/http://www.tcts.fpms.ac.be/publications/phds/mengusoglu/thesis_mengus.pdf, Apr. 20, 2004, 143 Pages. |
| Verma et al., “Face Detection and Tracking in a Video by Propagating Detection Probabilities”, In Proceedings of IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, Issue 10, Oct. 1, 2003, pp. 1215-1228. |
| M. K., et al., “Ambiguities in Natural Language Processing”, In International Journal of Innovative Research in Computer and Communication Engineering, vol. 2, Special Issue 5, Oct. 2014, pp. 392-394. |
| Pan, et al., “Robust Occlusion Handling in Object Tracking”, In IEEE Conference on Computer Vision and Pattern Recognition, Jun. 17, 2007, 8 Pages. |
| Panzarino, Matthew, “Here's an Actual 3D Indoor Map of a Room Captured With Google's Project Tango Phone”, Retrieved From https://techcrunch.com/2014/02/21/heres-an-actual-3d-indoor-map-of-a-room-captured-with-googles-project-tango-phone/, Feb. 21, 2014, 6 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017139”, dated May 8, 2018, 13 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017506”, dated May 4, 2018, 13 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017508”, dated May 8, 2018, 13 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017509”, dated May 11, 2018, 11 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017510”, dated Apr. 20, 2018, 14 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017511”, dated May 17, 2018, 12 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017512”, dated May 4, 2018, 15 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017513”, dated Apr. 12, 2018, 15 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017515”, dated May 9, 2018, 12 Pages. |
| “International Search Report and Written Opinion Issued in PCT Application No. PCT/US2018/017517”, dated May 11, 2018, 12 Pages. |
| Pullen, John Patrick., “Amazon Echo Tip: How to Add Multiple Users ! Time”, Retrieved From http://time.com/4668359/amazon-echo-alexa-multiple-accounts/, Feb. 13, 2017, 3 Pages. |
| Quirk, et al., “Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes”, In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, Jul. 26, 2015, pp. 878-888. |
| Rizwan, et al., “Local Enhancement for Robust Face Detection in Poor SNR Images”, In International Journal of Computer Science and Network Security, vol. 9, Issue 6, Jun. 2009, pp. 93-96. |
| Sinha, et al., “An Analysis Engine for Dependable Elicitation on Natural Language Use Case Description and its Application to Industrial Use Cases”, In IBM Research Report, RC242712, Dec. 18, 2008, 12 Pages. |
| Toutanova, et al., “Compositional Learning of Embeddings for Relation Paths in Knowledge Bases and Text”, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Aug. 7, 2016, pp. 1434-1444. |
| Wagner, Martin, “Tracking with Multiple Sensors”, Faculty of Computer Science at the Technical University of Munich, Sep. 12, 2004, 202 Pages. |
| Wheeler, et al., “Face Recognition at a Distance”, In Publication of Springer, Jan. 2011, pp. 353-381. |
| Yamamoto, S, et al., “Algorithm Optimizations for Low-Complexity Eye Tracking”, In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, Oct. 2009, pp. 18-22. |
| Kalal, et al., “Face-TLD: Tracking-Learning-Detection Applied to Faces”, In Proceedings of 17th IEEE International Conference on Image Processing, Sep. 26, 2010, pp. 3789-3792. |
| Zotkin, et al., “Joint Audio-Visual Tracking Using Particle Filters”, In EURASIP Journal on Applied Signal Processing, vol. 2002, Issue 1, Jan. 2002, pp. 1154-1164. |
| “Final Office Action Issued in U.S. Appl. No. 15/646,871”, dated Apr. 19, 2019, 22 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/682,407”, dated Jun. 26, 2019, 15 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/682,425”, dated May 6, 2019,12 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/636,422”, dated Sep. 4, 2018, 11 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/640,251”, dated Oct. 15, 2018, 22 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/646,871”, dated Dec. 19, 2018, 22 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/656,994”, dated Jan. 22, 2019, 8 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/657,031”, dated Oct. 5, 2018, 16 Pages. |
| “Non-Final Office Action Issued in U.S. Appl. No. 15/657,822”, dated Feb. 21, 2019, 25 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/832,656”, dated Feb. 7, 2019, 8 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/640,113”, dated May 14, 2020, 13 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/640,201”, dated May 27, 2020, 11 Pages. |
| “Notice of Allowance Issued in U.S. Appl. No. 15/832,656”, dated Apr. 22, 2020, 8 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/936,076”, dated Apr. 15, 2020, 23 Pages. |
| “Office Action Issued in European Patent Application No. 18708508.9”, dated May 28, 2020, 6 Pages. |
| “Non Final Office Action Issued in U.S. Appl. No. 15/646,871”, dated Jul. 1, 2020, 24 Pages. |
| “Final Office Action Issued in U.S. Appl. No. 15/657,822”, dated Aug. 7, 2020, 22 Pages. |
| “Notice of Allowance Issued in U.S. Appl. No. 15/640,251”, dated Jul. 31, 2020, 11 Pages. |
| “Final Office Action Issued in U.S. Appl. No. 16/005,470”, dated Sep. 4, 2020, 15 Pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20180233145 A1 | Aug 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62459020 | Feb 2017 | US | |
| 62482165 | Apr 2017 | US |
