Patent Grant
11567573
Augmented reality (AR) systems provide users with an interactive experience of a real-world environment supplemented with virtual information by overlaying computer-generated perceptual or virtual information on aspects of the real-world environment. Various techniques exist for controlling operations of an AR system. Typically, one or more input devices, such as a controller, a keyboard, a mouse, a camera, a microphone, and the like, may be used to control operations of the AR system. For example, a user may manipulate a number of buttons on an input device, such as a controller or a keyboard, to effectuate control of the AR system. In another example, a user may use voice commands to control operations of the AR system. The current techniques for controlling operations of an AR system have many flaws, so improved techniques are needed.
Some embodiments are directed to coupling a system that senses neuromuscular signals via neuromuscular sensors with a system that performs extended reality (XR) functions. As will be appreciated, XR functions may include augmented reality (AR), virtual reality (VR) functions, mixed reality (MR) functions, and the like. In particular, a system that senses neuromuscular signals may be used in conjunction with an XR system to provide an improved XR experience for a user. Neuromuscular signals may be used directly as an input to an XR system (e.g., by using motor unit action potentials as an input signal) and/or the neuromuscular signals may be processed (including by using an inference model as described herein) for purpose of determining a movement, force, and/or position of a part of the user's body (e.g., the fingers, hand, and wrist). For instance, information gained within both systems may be used to improve the overall XR experience. In embodiments where a musculoskeletal representation associated with the body part is generated based on sensor data, a camera in an XR system may capture data that is used to improve the accuracy of a model of the musculoskeletal representation and/or used to calibrate the model. Further, in another implementation, muscle activation data may be visualized and displayed to a user in an XR environment. In yet another example, display information in the XR environment may be used as feedback to the user to permit the user to more accurately control their musculoskeletal input to the system. Further, control features may be provided that permit neuromuscular signals to control XR system elements including operation of the XR system itself. In addition, various forms of input (e.g., text, writing, and/or drawing) identified based on the neuromuscular signals may be provided as input to the XR system, as well as inputs to the XR system based on specific gestures.
According to aspects of the technology described herein, a computerized system for providing input to an extended reality system is provided. The system may comprise one or more neuromuscular sensors configured to detect neuromuscular signals from a user, wherein the one or more neuromuscular sensors are arranged on one or more wearable devices; and at least one computer processor. The at least one computer processor may be programmed to: determine that the computerized system is in a mode configured to provide an input to the extended reality system; identify the input based at least in part on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals; and provide the identified input to the extended reality system.
In an aspect, the mode is determined based on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals.
In an aspect, the mode is determined based on a gesture detected from the user, wherein the gesture is identified based on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals.
In an aspect, the mode is determined by receiving a selection of the mode from a user interface displayed in an extended reality environment provided by the extended reality system, wherein the user interface is configured to identify a plurality of modes from which the user may select.
In an aspect, the mode is determined based on one or more typing, writing, drawing actions, or one-handed actions detected from the user.
In as aspect, the mode is determined in response to receiving a signal from the extended reality system. In an aspect, the signal is generated by the extended reality system in response to detection of an event for which input within an extended reality environment provided by the extended reality system is desired.
In an aspect, the mode comprises a typing mode and identifying the input comprises identifying one or more tapping or typing actions based, at least in part, on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals.
In an aspect, the computerized system further comprises a physical keyboard or a virtual keyboard.
In an aspect, identifying one or more tapping or typing actions comprises detecting the user tapping or typing on the physical keyboard or the virtual keyboard.
In an aspect, the mode comprises a writing mode and identifying the input comprises identifying one or more writing actions as detected from the user based, at least in part, on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals.
In an aspect, the computerized system further comprises a physical stylus, physical writing implement, virtual stylus and/or virtual writing implement.
In an aspect, identifying the one or more writing actions comprises detecting the user using the physical stylus, physical writing implement, virtual stylus and/or virtual writing implement.
In an aspect, the one or more writing actions are identified as detected in mid-air.
In an aspect, the mode comprises a drawing mode and identifying the input comprises identifying one or more drawing actions as detected from the user based, at least in part, on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals.
In an aspect, the one or more drawing actions detected from the user comprise a plurality of line segments and/or curves.
In an aspect, identifying the input for the drawing mode comprises identifying input as drawings and/or text.
In an aspect, the at least one computer processor is further programmed to combine drawing and text inputs such that the text overlays or annotates the drawing.
In an aspect, the mode comprises a one-handed mode and identifying the input comprise identifying one or more one-handed actions as detected from the user based, at least in part, on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals.
In an aspect, the extended reality system is configured to display an indication of the identified input to the user. In an aspect, the indication of the identified input comprises text input identified based on one or more typing, writing, drawing actions, or one-handed actions detected from the user.
In an aspect, the computerized system further comprises a visual display or interface configured to present one or more suggested or predicted words or phrases for text input.
In an aspect, the computerized system further comprises a visual display or interface configured to present one or more virtual ink marks associated with one or more strokes as detected from the user.
In an aspect, the computerized system further comprises a visual display or interface configured to present a drawing as identified based on one or more drawing actions detected from the user.
In an aspect, the computerized system further comprises at least one inertial measurement unit (IMU) sensor, wherein identifying the input comprises identifying the input based, at least in part, on at least one output signal associated with the at least one IMU sensor.
In an aspect, the computerized system further comprises at least one camera, wherein identifying the input comprises identifying the input based, at least in part, on at least one output signal associated with the at least one camera.
In an aspect, the mode comprises a first mode, and the at least one computer processor is further programmed to: identify a second input based at least in part on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, wherein the second input is further identified based, at least in part, on a second mode; and provide the identified second input to the extended reality system.
In an aspect, the one or more wearable devices comprises a first wearable device configured to detect neuromuscular signals from a first arm of the user and a second wearable device configured to detect neuromuscular signals from a second arm of the user.
In an aspect, the extended reality system is an augmented reality system.
According to aspects of the technology described herein, a method performed by a computerized system for providing input to an extended reality system is provided. The method comprises detecting, using one or more neuromuscular sensors arranged on one or more wearable devices, a plurality of neuromuscular signals from a user; determining that the computerized system is in a mode configured to provide input to the extended reality system; identifying the input based at least in part on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals; and providing the identified input to the extended reality system.
According to aspects of the technology described herein, a system for providing one or more inputs to an extended reality (XR) system. The system comprises one or more neuromuscular sensors configured to detect neuromuscular signals from a user, wherein the one or more neuromuscular sensors are arranged on one or more wearable devices; one or more auxiliary sensors configured to detect information regarding a physiological state and/or behavior from the user; and at least one computer processor. The at least one computer processor is programmed to: determine that the system is in a mode configured to provide one or more inputs to the XR system, associate the neuromuscular signals with the information detected from the one or more auxiliary sensors, process the neuromuscular signals and/or the information detected from the one or more auxiliary sensors using one or more inference models; identify the one or more inputs based on the processed neuromuscular signals and/or the processed information detected from the one or more auxiliary sensors; and provide the identified one or more inputs to the XR system.
According to aspects of the technology described herein, a kit for use with an extended reality (XR) system is provided. The kit comprises one or more neuromuscular sensors configured to detect neuromuscular signals from a user, wherein the one or more neuromuscular sensors are arranged on one or more wearable devices; one or more auxiliary sensors configured to detect information regarding a physiological state and/or behavior from the user; and at least one storage medium storing instructions. The instructions, when executed by at least one computer processor, cause the at least one computer processor to: process the neuromuscular signals from the neuromuscular sensors, process the information detected from the one or more auxiliary sensors, identify one or more user inputs based on the processed neuromuscular signals and/or the processed information detected from the one or more auxiliary sensors, and communicate the identified one or more user inputs to the XR system.
According to aspects of the technology described herein, a computerized system for providing input to an augmented reality system based, at least in part, on neuromuscular signals is provided. The system comprises a plurality of neuromuscular sensors configured to record a plurality of neuromuscular signals from a user, wherein the plurality of neuromuscular sensors are arranged on one or more wearable devices; and at least one computer processor. The at least one computer processor is programmed to: determine that the computerized system is in a mode configured to provide input including text to the augmented reality system; identify, based at least in part, on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, the input, wherein the input is further identified based, at least in part, on the mode; and provide the identified input to the augmented reality system.
In an aspect, the mode comprises a typing mode and identifying the input comprises identifying one or more tapping or typing actions based, at least in part, on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals; and identifying text input for the typing mode based on the one or more tapping or typing actions.
In an aspect, identifying the one or more tapping or typing actions comprises identifying the one or more tapping or typing actions on a surface of a physical keyboard.
In an aspect, identifying the one or more tapping or typing actions comprises identifying the one or more tapping or typing actions on a surface that has a virtual keyboard projected thereon by the augmented reality system.
In an aspect, the mode comprises a writing mode and identifying the input comprises identifying one or more writing actions performed by the user based, at least in part, on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals; and identifying text input for the writing mode based on the one or more writing actions.
In an aspect, identifying the one or more writing actions performed by the user comprises identifying the one or more writing actions performed on a surface with a physical stylus, a physical writing implement, or fingertip or fingertips of the user.
In an aspect, identifying the one or more writing actions performed by the user comprises identifying the one or more writing actions performed on a surface with a virtual stylus or virtual writing implement.
In an aspect, identifying the one or more writing actions performed by the user comprises identifying the one or more writing actions performed in mid-air.
In an aspect, the mode comprises a drawing mode and identifying the input comprises identifying one or more drawing actions performed by the user based, at least in part, on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals; and identifying the input for the drawing mode based on the one or more drawing actions.
In an aspect, the input for the drawing mode comprises a plurality of line segments and/or curves.
In an aspect, the input for the drawing mode comprises input determined based on a sequence of pixel positions controlled by the one or more drawing actions performed by the user.
In an aspect, identifying the input for the drawing mode comprises identifying a drawing based on the one or more drawing actions performed by the user; and identifying the text based on the one or more drawing actions performed by the user.
In an aspect, the at least one computer processor is further programmed to combine the drawing and the text such that the text overlays or annotates the drawing.
In an aspect, identifying the input for the drawing mode comprises identifying a drawing based on the one or more drawing actions performed by the user; and identifying the text from the drawing.
In an aspect, the mode comprises a one-handed mode and identifying the input comprises identifying one or more one-handed actions performed by the user based, at least in part, on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals; and identifying the input for the one-handed mode based on the one or more one-handed actions.
In an aspect, determining that the computerized system is in the mode configured to provide the input comprises receiving a user selection of the mode.
In an aspect, receiving the user selection of the mode comprises receiving the user selection from a user interface displayed in an augmented reality environment provided by the augmented reality system, wherein the user interface is configured to identify a plurality of modes from which the user may select.
In an aspect, determining that the computerized system is in the mode configured to provide the input comprises determining the mode from the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals.
In an aspect determining the mode comprises identifying at least one gesture performed by the user based on the plurality of neuromuscular signals and/or the information based on the plurality of neuromuscular signals; and determining the mode corresponding to the at least one gesture.
In an aspect, determining the mode comprises determining the mode based on one or more typing, writing, drawing actions, or one-handed actions performed by the user.
In an aspect, determining that the computerized system is in the mode configured to provide the input comprises determining the mode in response to receiving a signal from the augmented reality system. In an aspect, the signal is generated at the augmented reality system in response to detection of an event for which input within an augmented reality environment provided by the augmented reality system is desired.
In an aspect, the augmented reality system is configured to display an indication of the identified input to the user.
In an aspect, the indication of the identified input comprises text input identified based on one or more typing, writing, drawing actions, or one-handed actions performed by the user.
In an aspect, the indication of the identified input comprises a listing of one or more suggested or predicted words or phrases for the text input.
In an aspect, the indication of the identified input comprises one or more virtual ink marks associated with one or more strokes made by a writing implement.
In an aspect, the indication of the identified input comprises a drawing identified based on one or more drawing actions performed by the user.
In an aspect, the indication is displayed via a user interface presented within an augmented reality environment provided by the augmented reality system.
In an aspect, the indication is rendered onto a surface that the user is interacting with by the augmented reality system.
In an aspect, the computerized system further comprises at least one inertial measurement unit (IMU) sensor, wherein identifying the text input comprises identifying the text input based, at least in part, on at least one output signal associated with the at least one IMU sensor.
In an aspect, the computerized system further comprises at least one camera, wherein identifying the text input comprises identifying the text input based, at least in part, on at least one output signal associated with the at least one camera.
In an aspect, the mode comprises a first mode, and wherein the at least one computer processor is further programmed to: identify, based at least in part, on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, a second input, wherein the second input is further identified based, at least in part, on a second mode; and provide the identified second input to the augmented reality system.
In an aspect, the identified input provided to the augmented reality system comprises input identified from a plurality of sources, wherein the plurality of sources include the plurality of neuromuscular signals and at least one source other than the plurality of neuromuscular signals.
According to aspects of the technology described herein, a method performed by a computerized system for providing input to an augmented reality system based, at least in part, on neuromuscular signals is provided. The method comprises recording, using a plurality of neuromuscular sensors arranged on one or more wearable devices, a plurality neuromuscular signals from a user; determining that the computerized system is in a mode configured to provide input including text to the augmented reality system; identifying based, at least in part, on the plurality of neuromuscular signals and/or information based on the plurality of neuromuscular signals, the input, wherein the input is further identified based, at least in part, on the mode; and providing the identified input to the augmented reality system.
According to aspects of the technology described herein, a computerized system for providing input to an augmented reality system is provided. The computerized system comprises one or more neuromuscular sensors configured to detect neuromuscular signals from a user, wherein the one or more neuromuscular sensors are arranged on one or more wearable devices; and at least one computer processor. The at least one computer processor is programmed to: determine that the computerized system is in a mode configured to provide input including text to the augmented reality system, wherein the mode is determined based on the neuromuscular signals and/or information based on the neuromuscular signals, and the mode is selected from the group consisting of a typing mode, a writing mode, a drawing mode, and a one-handed mode; identify, based at least in part, on the neuromuscular signals and/or information based on the neuromuscular signals, the input, wherein the input is further identified based, at least in part, on the mode; and provide the identified input to the augmented reality system.
In an aspect, the mode is determined based on a gesture as detected from the user based on the neuromuscular signals and/or information based on the neuromuscular signals.
In an aspect, the mode comprises a typing mode and identifying the input comprises identifying one or more tapping or typing actions based, at least in part, on the neuromuscular signals and/or the information based on the neuromuscular signals; and identifying text input for the typing mode based on the one or more tapping or typing actions.
In an aspect, identifying the one or more tapping or typing actions comprises identifying the one or more tapping or typing actions on a surface of a physical keyboard or a surface that has a representation of a keyboard projected thereon.
In an aspect, the mode comprises a writing mode and identifying the input comprises identifying one or more writing actions detected from the user based, at least in part, on the neuromuscular signals and/or the information based on the neuromuscular signals; and identifying text input for the writing mode based on the one or more writing actions.
In an aspect, identifying the one or more writing actions detected from the user comprises identifying the one or more writing actions performed on a surface with a physical stylus, physical writing implement, virtual stylus, or virtual writing implement.
In an aspect, identifying the one or more writing actions detected from the user comprises identifying the one or more writing actions as detected in mid-air.
In an aspect, the mode comprises a drawing mode and identifying the input comprises identifying one or more drawing actions detected from the user based, at least in part, on the neuromuscular signals and/or the information based on the neuromuscular signals; and identifying the input for the drawing mode based on the one or more drawing actions.
In an aspect, the one or more drawing actions comprises a plurality of line segments and/or curves.
In an aspect, identifying the input for the drawing mode comprises identifying a drawing based on the one or more drawing actions detected from the user; and identifying the text based on the one or more drawing actions detected from the user.
In an aspect, the at least one computer processor is further programmed to combine the drawing and the text such that the text overlays or annotates the drawing.
In an aspect, identifying the input for the drawing mode comprises identifying a drawing based on the one or more drawing actions detected from the user; and identifying the text from the drawing.
In an aspect, the mode comprises a one-handed mode and identifying the input comprises identifying one or more one-handed actions detected from the user based, at least in part, on the neuromuscular signals and/or the information based on the neuromuscular signals; and identifying the input for the one-handed mode based on the one or more one-handed actions.
In an aspect, the augmented reality system is configured to display an indication of the identified input to the user.
In an aspect, the indication of the identified input comprises text input identified based on one or more typing, writing, drawing actions, or one-handed actions detected from the user.
In an aspect, the indication of the identified input comprises one or more suggested or predicted words or phrases for the text input.
In an aspect, the indication of the identified input comprises one or more virtual ink marks associated with one or more strokes detected from the user.
In an aspect, the indication of the identified input comprises a drawing identified based on one or more drawing actions detected from the user.
In an aspect, the indication is displayed via a user interface presented within an augmented reality environment provided by the augmented reality system.
In an aspect, the indication is rendered onto a surface that the user is interacting with by the augmented reality system.
In an aspect, the computerized system further comprises at least one inertial measurement unit (IMU) sensor, wherein identifying the text input comprises identifying the text input based, at least in part, on at least one output signal associated with the at least one IMU sensor.
In an aspect, the computerized system further comprises at least one camera, wherein identifying the text input comprises identifying the text input based, at least in part, on at least one output signal associated with the at least one camera.
In an aspect, the mode comprises a first mode, and wherein the at least one computer processor is further programmed to: identify, based at least in part, on the neuromuscular signals and/or information based on the neuromuscular signals, a second input, wherein the second input is further identified based, at least in part, on a second mode; determine that the computerized system is to be switched from the first mode to the second mode; switch the computerized system from the first mode to the second mode in response to determining that the computerized system is to be switched from the first mode to the second mode; and provide the identified second input to the augmented reality system.
In an aspect, the identified input provided to the augmented reality system comprises input identified from a plurality of sources, wherein the plurality of sources include the neuromuscular signals and at least one source other than the neuromuscular signals.
In an aspect, the at least one source other than the neuromuscular signals comprises at least one physical input device, and the identified input provided to the augmented reality system comprises a combination of the input identified from the plurality of sources.
In an aspect, the one or more wearable devices comprises a first wearable device configured to detect neuromuscular signals from a first arm of the user and a second wearable device configured to detect neuromuscular signals from a second arm of the user.
According to aspects of the technology described herein, a method performed by a computerized system for providing input to an augmented reality system is provided. The method comprises detecting, using one or more neuromuscular sensors arranged on one or more wearable devices, neuromuscular signals from a user; determining that the computerized system is in a mode configured to provide input including text to the augmented reality system, wherein the mode is determined based on the neuromuscular signals and/or information based on the neuromuscular signals, and the mode is selected from the group consisting of a typing mode, a writing mode, a drawing mode, and a one-handed mode; identifying based, at least in part, on the neuromuscular signals and/or information based on the neuromuscular signals, the input, wherein the input is further identified based, at least in part, on the mode; and providing the identified input to the augmented reality system.
In an aspect, the mode is determined based on a gesture detected from the user, wherein the gesture is detected based on the neuromuscular signals and/or information based on the neuromuscular signals.
According to aspects of the technology described herein, a non-transitory computer-readable medium encoded with instructions that, when executed by at least one computer processor performs a method of: detecting, using a plurality of neuromuscular sensors arranged on one or more wearable devices, neuromuscular signals from a user; determining that a computerized system is in a mode configured to provide input including text to the augmented reality system, wherein the mode is determined based on the neuromuscular signals and/or information based on the neuromuscular signals, and the mode is selected from the group consisting of a typing mode, a writing mode, a drawing mode, and a one-handed mode; identifying based, at least in part, on the neuromuscular signals and/or information based on the neuromuscular signals, the input, wherein the input is further identified based, at least in part, on the mode; and providing the identified input to the augmented reality system.
It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein.
Various non-limiting embodiments of the technology will be described with reference to the following figures. It should be appreciated that the figures are not necessarily drawn to scale.
The inventors have developed novel techniques for providing input to extended reality (XR) systems, which include inter alia augmented reality (AR), virtual reality (VR), and mixed reality (MR) systems. Various embodiments described herein offer certain advantages, including, but not limited to, avoiding the use of an undesirable or burdensome physical keyboard, joystick, or other controller; overcoming issues associated with time consuming and high latency processing of low quality images of the user captured by a camera; allowing for the capture and detection of subtle, small, or fast movements and/or variations in force exerted by a user (e.g., varying amounts of force exerted through a stylus, writing instrument, or finger being pressed against a surface) that can be important for resolving text input and other control signals; collecting and analyzing various physiological and/or behavioral information detected from the user that enhances the identification process and is not readily obtained by conventional input devices; allowing instances where the user's hand is obscured or outside the camera's field of view, e.g., in the user's pocket, or while the user is wearing a glove; and allowing better user operability and navigability within the XR environment.
Other embodiments account for scenarios in which an individual either does not have access to input devices or may otherwise want to provide input to the XR system without the use of input devices. For example, an individual may want to provide input to the XR system in a covert manner without being noticed by other individuals.
In accordance with some embodiments, signals recorded or detected from wearable sensors are used to identify and provide input to an XR system. Various forms of input, for example, discrete control signals, continuous (e.g., 2D) control signals, text entry via a keyboard or other mode of text entry, writing, and/or drawing, may be identified from the recorded or detected signals and/or information based on the recorded or detected signals to enable improved techniques for providing input (such as text) to the XR system. In some embodiments, various forms of input may be identified based on a mode of the system that senses signals via the wearable sensors and provides input to the XR system. The user can manually, or the system can automatically, switch between input modes based, at least in part, on neuromuscular data detected from the user. In one embodiment, the system can enter a typing mode and can identify text from the user to be provided to the XR system based on one or more tapping or typing actions performed by the user (e.g., tapping on a surface of a physical keyboard, tapping on a surface that has a virtual keyboard projected thereon by the XR system, tapping on a surface that does not have a keyboard projected on it, or performing gestures in mid-air that correspond to typing-style movements). The systems and methods described herein can identify text input from the user based on the recorded or detected signals and/or information based on the recorded or detected signals. In another embodiment, the system can enter a writing mode and text input can be provided to the XR system by identifying one or more writing actions performed by the user (e.g., writing on a surface with a physical or virtual writing implement) based on the recorded or detected signals and/or information based on the recorded or detected signals. In yet another embodiment, the system can enter a drawing mode and input can be provided to the XR system by identifying one or more drawing actions (e.g., drawing one or more line segments and/or curves on a surface) performed by the user based on the recorded or detected signals and/or information based on the recorded or detected signals. In another embodiment, the system can enter a one-handed mode (i.e., a mode where the user uses only one hand to provide input), and input can be provided to the XR system by identifying one or more one-handed actions (for example, gestures such as squeezing, pinching, and/or tapping of various fingers and combinations of fingers) performed by the user based on the recorded or detected signals and/or information based on the recorded or detected signals.
In some embodiments, the XR system may provide visual feedback by displaying an indication of the identified input to the user, which may facilitate text entry or other information provided as input to the XR system. The indication can be displayed via a user interface presented within an XR environment provided by the XR system. For example, a display associated with the XR system can overlay a visual representation of the identified input in the user interface or provide audio feedback to the user about the identified input. In some embodiments, the indication may be rendered by the AR system onto a surface with which the user is interacting.
In some embodiments, the system described herein senses signals via the wearable sensors and provides input to the XR system such that the system smoothly transitions from a first input mode to a second input mode without requiring an explicit mode switch instruction from the user. This provides for a flexible approach to providing input to the XR system. For example, the system described herein may be operating in a typing mode where the user is providing text input to the system by typing on a physical keyboard. The user may stop typing on the physical keyboard and resume providing text input by writing with a stylus. In response, the system may automatically detect the change in input mode and seamlessly switch from the typing mode to a writing mode. In some embodiments, the user may switch to different forms of text entry while the system is in the same mode. For example, the user may begin by typing on a physical keyboard, and resume text entry by typing on a virtual keyboard or using typing motions without any virtual representation of a keyboard. In this scenario, the manner in which the user is providing text input has changed even though the system remains in the typing mode. In some embodiments, the visual feedback provided by the XR system may continue uninterrupted regardless of the mode or the form of text entry.
According to some embodiments, the input to be provided to the XR system may be identified, at least in part, from raw (e.g., unprocessed) sensor signals collected by one or more of the wearable sensors. In some embodiments, the input to be provided to the XR system may be identified, at least in part, from information based on the raw sensor signals (e.g., processed sensor signals), where the raw sensor signals collected by one or more of the wearable sensors are processed to perform amplification, filtering, rectification, and/or other form of signal processing, examples of which are described in more detail below. In some embodiments, the input to be provided to the XR system may be identified, at least in part, from an output of one or more trained inference models that receive the sensor signals (or processed versions of the sensor signals) as input.
As described herein, in some embodiments, various muscular activation states may be identified directly from recorded or detected sensor data. In other embodiments, handstates, gestures, postures, and the like (collectively or individually referred to as muscular activation states) may be identified based, at least in part, on the output of a trained inference model. In some embodiments, various forms of input can be provided to the AR system and may be identified directly from recorded sensor data. In other embodiments, the input can be provided to the AR system and may be identified based, at least in part, on the output of one or more trained inference models. In some embodiments, a trained inference model may output motor unit or muscle activations and/or position, orientation, and/or force estimates for segments of a computer-generated musculoskeletal model. In one example, all or portions of a human musculoskeletal system can be modeled as a multi-segment articulated rigid body system, with joints forming the interfaces between the different segments and joint angles defining the spatial relationships between connected segments in the model. Constraints on the movement at the joints are governed by the type of joint connecting the segments and the biological structures (e.g., muscles, tendons, ligaments) that restrict the range of movement at the joint. For example, the shoulder joint connecting the upper arm to the torso and the hip joint connecting the upper leg to the torso are ball and socket joints that permit extension and flexion movements as well as rotational movements. By contrast, the elbow joint connecting the upper arm and the forearm and the knee joint connecting the upper leg and the lower leg allow for a more limited range of motion. In this example, a multi-segment articulated rigid body system is used to model portions of the human musculoskeletal system. However, it should be appreciated that some segments of the human musculoskeletal system (e.g., the forearm), though approximated as a rigid body in the articulated rigid body system, may include multiple rigid structures (e.g., the ulna and radius bones of the forearm) that provide for more complex movement within the segment that is not explicitly considered by the rigid body model. Accordingly, a model of an articulated rigid body system for use with some embodiments of the technology described herein may include segments that represent a combination of body parts that are not strictly rigid bodies. It will be appreciated that physical models other than the multi-segment articulated rigid body system may be used to model portions of the human musculoskeletal system without departing from the scope of this disclosure.
Continuing with the example above, in kinematics, rigid bodies are objects that exhibit various attributes of motion (e.g., position, orientation, angular velocity, acceleration). Knowing the motion attributes of one segment of the rigid body enables the motion attributes for other segments of the rigid body to be determined based on constraints in how the segments are connected. For example, the hand may be modeled as a multi-segment articulated body with the joints in the wrist and each finger forming the interfaces between the multiple segments in the model. In some embodiments, movements of the segments in the rigid body model can be simulated as an articulated rigid body system in which position (e.g., actual position, relative position, or orientation) information of a segment relative to other segments in the model are predicted using a trained inference model, as described in more detail below.
In one non-limiting example, the portion of the human body approximated by a musculoskeletal representation is a hand or a combination of a hand with one or more arm segments. The information used to describe a current state of the positional relationships between segments, force relationships for individual segments or combinations of segments, and muscle and motor unit activation relationships between segments, in the musculoskeletal representation is referred to herein as the “handstate” of the musculoskeletal representation. It should be appreciated, however, that the techniques described herein are also applicable to musculoskeletal representations of portions of the body other than the hand including, but not limited to, an arm, a leg, a foot, a torso, a neck, or any combination of the foregoing.
In addition to spatial (e.g., position and/or orientation) information, some embodiments are configured to predict force information associated with one or more segments of the musculoskeletal representation. For example, linear forces or rotational (torque) forces exerted by one or more segments may be estimated. Examples of linear forces include, but are not limited to, the force of a finger or hand pressing on a solid object such as a table, and a force exerted when two segments (e.g., two fingers) are pinched together. Examples of rotational forces include, but are not limited to, rotational forces created when segments in the wrist or fingers are twisted or flexed. In some embodiments, the force information determined as a portion of a current handstate estimate includes one or more of pinching force information, grasping force information, or information about co-contraction forces between muscles represented by the musculoskeletal representation.
As the tension of a muscle increases during performance of a motor task, the firing rates of active neurons increases and additional neurons may become active, which is a process referred to as motor unit recruitment. The pattern by which neurons become active and increase their firing rate is stereotyped, such that the expected motor unit recruitment patterns define an activity manifold associated with standard or normal movement. Some embodiments record activation of a single motor unit or a group of motor units that are “off-manifold,” in that the pattern of motor unit activation is different than an expected or typical motor unit recruitment pattern. Such off-manifold activation is referred to herein as, “sub-muscular activation” or “activation of a sub-muscular structure,” where a sub-muscular structure refers to the single motor unit or the group of motor units associated with the off-manifold activation. Examples of off-manifold motor unit recruitment patterns include, but are not limited to, selectively activating a high-threshold motor unit without activating a lower-threshold motor unit that would normally be activated earlier in the recruitment order and modulating the firing rate of a motor unit across a substantial range without modulating the activity of other neurons that would normally be co-modulated in typical motor recruitment patterns. In some embodiments, the neuromuscular sensor(s) may be used to sense sub-muscular activation(s) without observable movement. Sub-muscular activation(s) may be used, at least in part, to identify and provide input to an augmented reality system in accordance with some embodiments of the technology described herein.
In some embodiments, sensors or sensing components 102 include one or more neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensors 102 include one or more auxiliary sensors such as Inertial Measurement Units (IMUs), which measure a combination of physical aspects of motion, using, for example, an accelerometer, a gyroscope, a magnetometer, or any combination of one or more accelerometers, gyroscopes and magnetometers, or any other components or devices capable of detecting spatiotemporal positioning, motion, force, or other aspects of a user's physiological state and/or behavior. In some embodiments, IMUs may be used to sense information about the movement of the part of the body on which the IMU is attached and information derived from the sensed data (e.g., position and/or orientation information) may be tracked as the user moves over time. For example, one or more IMUs may be used to track movements of portions of a user's body proximal to the user's torso relative to the sensor (e.g., arms, legs) as the user moves over time. In other embodiments, sensors 102 include a plurality of neuromuscular sensors and at least one auxiliary sensor configured to continuously record a plurality of auxiliary signals. Examples of other auxiliary sensors include, but are not limited to, microphones, imaging devices (e.g., a camera), radiation-based sensors for use with a radiation-generation device (e.g., a laser-scanning device), or other types of sensors such as thermal sensors, infrared sensors, heart-rate or blood pressure monitors, and/or video eye trackers.
In embodiments that include at least one IMU and one or more neuromuscular sensors, the IMU(s) and neuromuscular sensors may be arranged to detect movement of the same or different parts of the human body. For example, the IMU(s) may be arranged to detect movements of one or more body segments proximal to the torso (e.g., an upper arm), whereas the neuromuscular sensors may be arranged to detect movements of one or more body segments distal to the torso (e.g., a forearm or wrist). It should be appreciated, however, that the sensors may be arranged in any suitable way, and embodiments of the technology described herein are not limited based on the particular sensor arrangement. For example, in some embodiments, at least one IMU and a plurality of neuromuscular sensors may be co-located on a body segment to track movements of body segment using different types of measurements. In one implementation described in more detail below, an IMU sensor and a plurality of EMG sensors are arranged on a wearable device configured to be worn around the lower arm or wrist of a user. In such an arrangement, the IMU sensor may be configured to track movement information (e.g., positioning and/or orientation over time) associated with one or more arm segments, to determine, for example whether the user has raised or lowered their arm, whereas the EMG sensors may be configured to determine movement information associated with wrist or hand segments to determine, for example, whether the user has an open or closed hand configuration or sub-muscular information associated with activation of sub-muscular structures in muscles of the wrist or hand.
Each of the sensors 102 includes one or more sensing components configured to sense information about a user. In the case of IMUs, the sensing components may include one or more accelerometers, gyroscopes, magnetometers, or any combination thereof to measure characteristics of body motion and/or characteristics related to body motion, examples of which include, but are not limited to, acceleration, angular velocity, and sensed magnetic field around the body. In the case of neuromuscular sensors, the sensing components may include, but are not limited to, electrodes configured to detect electric potentials on the surface of the body (e.g., for EMG sensors), vibration sensors configured to measure skin surface vibrations (e.g., for MMG sensors), and acoustic sensing components configured to measure ultrasound signals (e.g., for SMG sensors) arising from muscle activity.
In some embodiments, at least some of the plurality of sensors 102 are arranged as a portion of a wearable device configured to be worn on or around part of a user's body. For example, in one non-limiting example, an IMU sensor and a plurality of neuromuscular sensors are arranged circumferentially around an adjustable and/or elastic band such as a wristband or armband configured to be worn around a user's wrist or arm. Alternatively, at least some of the sensors may be arranged on a wearable patch configured to be affixed to a portion of the user's body. In some embodiments, multiple wearable devices, each having one or more IMUs and/or neuromuscular sensors included thereon may be used to detect neuromuscular data and generate control information based on activation of muscular and sub-muscular structures and/or movement(s) that involve(s) multiple parts of the body.
In one embodiment, sixteen EMG sensors are arranged circumferentially around an elastic band configured to be worn around a user's lower arm. For example,
In some embodiments, the output of one or more of the sensing components may be optionally processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and/or rectification). In other embodiments, at least some signal processing of the output of the sensing components may be performed in software. Thus, signal processing of signals recorded by the sensors may be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect.
In some embodiments, the sensor data recorded by the sensors 102 may be optionally processed to compute additional derived measurements that are then provided as input to one or more inference models, as described in more detail below. For example, recorded signals from an IMU sensor may be processed to derive an orientation signal that specifies the orientation of a rigid body segment over time. Sensors 102 may implement signal processing using integrated components, or at least a portion of the signal processing may be performed by one or more components in communication with, but not directly integrated with the sensors.
System 100 can also include one or more computer processors 104 programmed to communicate with sensors 102 through either a one-way or two-way communication pathway. For example, signals recorded by one or more of the sensors may be provided to the processor(s), which may be programmed to execute one or more machine learning algorithms that process signals output by the sensors 102 to train (or retrain) one or more inference models 106, and the resulting trained (or retrained) inference model(s) 106 may be stored for later use in identifying and providing input to an XR system, as described in more detail below. As will be appreciated, an inference model may be a model that utilizes a statistical inference based on a probability distribution to deduce a result; in this regard, an inference model may comprise a statistical model.
In some embodiments, signals recorded by sensors arranged on a first wearable device worn on one hand/arm and signals recorded by sensors arranged on a second wearable device worn on the other hand/arm may be processed using the same inference model(s) or separate inference model(s).
System 100 also optionally includes one or more controllers 108. For example, controller 108 may be a display controller configured to display a visual representation (e.g., a representation of a hand). As discussed in more detail herein, one or more computer processors may implement one or more trained inference models that receive as input signals recorded by sensors 102 and provide as output information (e.g., predicted handstate information) that may be used to identify and provide input to an XR system. In some embodiments, as a user performs different movements, a trained inference model interprets neuromuscular signals recorded by wearable neuromuscular sensors into position and force estimates (e.g., handstate information) that are used to update the musculoskeletal representation. Because the neuromuscular signals are continuously sensed and recorded, the musculoskeletal representation is updated in real time and a visual representation of a hand (e.g., within an XR environment) may be rendered based on the current handstate estimates. As will be appreciated, an estimate of a user's handstate may be used to determine a gesture being performed by the user and/or to predict a gesture that the user will perform.
According to some embodiments, musculoskeletal representations (e.g., hand-rendering) may include actual visual representations of biomimetic (realistic) hands, synthetic (robotic) hands, as well as abstract “internal representations” that serve as input for gesture control (e.g., to other applications, systems, etc.). That is, the position and/or force of the hand may be provided to downstream algorithms (e.g., control algorithms in an XR system) but may not be directly rendered.
In some embodiments, as shown in
Information generated in either system (e.g., XR camera inputs from an XR system, neuromuscular sensor inputs from a computer-based system that generates the musculoskeletal representation based on sensor data) can be used to improve the user experience, accuracy, feedback, inference models, calibration functions, and other aspects in the overall system. To this end, in an XR environment for example, system 100 may include an XR system that includes one or more of the following: processors, a camera (e.g., one or more camera(s) contained in a head-mounted display), a display (e.g., via XR glasses or other viewing device), or any other auxiliary sensor(s) that provides XR information within a view of the user or provides XR information to the user. In some embodiments, information from a camera contained in the head-mounted display in the XR system may be used in combination with information from the neuromuscular sensors to interpret movement, gestures, and/or actions performed by the user. System 100 may also include system elements that couple the XR system with a computer-based system that generates the musculoskeletal representation based on sensor data. For example, the systems may be coupled via a special-purpose or other type of computer system that receives inputs from the XR system and the system that generates the computer-based musculoskeletal representation. Such a system may include a gaming system, robotic control system, personal computer, or other system that is capable of interpreting XR and musculoskeletal information. The XR system and the system that generates the computer-based musculoskeletal representation may also be programmed to communicate directly. Such information may be communicated using any number of interfaces, protocols, or media.
In some embodiments, inference model 106 may be a neural network and, for example, may be a recurrent neural network. In some embodiments, the recurrent neural network may be a long short-term memory (LSTM) neural network. It should be appreciated, however, that the recurrent neural network is not limited to being an LSTM neural network and may have any other suitable architecture. For example, in some embodiments, the recurrent neural network may be a fully recurrent neural network, a gated recurrent neural network, a recursive neural network, a Hopfield neural network, an associative memory neural network, an Elman neural network, a Jordan neural network, an echo state neural network, a second order recurrent neural network, and/or any other suitable type of recurrent neural network. In other embodiments, neural networks that are not recurrent neural networks may be used. For example, deep neural networks, convolutional neural networks, and/or feedforward neural networks, may be used.
In some embodiments, the output of one or more inference models comprises one or more discrete outputs. Discrete outputs (e.g., classification labels) may be used, for example, when a user performs a particular gesture or causes a particular pattern of activation (including individual neural spiking events) to occur. For example, a model may be trained to estimate or infer whether the user is activating a particular motor unit, activating a particular motor unit with a particular timing, activating a particular motor unit with a particular firing pattern, or activating a particular combination of motor units. On a shorter timescale, discrete classification is used in some embodiments to estimate whether a particular motor unit fired an action potential within a given amount of time. In such embodiments, these estimates may then be accumulated to obtain an estimated firing rate for that motor unit.
In embodiments in which an inference model is implemented as a neural network configured to output a discrete signal, the neural network may include a softmax layer such that the outputs add up to one and may be interpreted as probabilities. The output of the softmax layer may be a set of values corresponding to a respective set of control signals, with each value indicating a probability that the user want to perform a particular control action. As one non-limiting example, the output of the softmax layer may be a set of three probabilities (e.g., 0.92, 0.05, and 0.03) indicating the respective probabilities that the detected pattern of activity is one of three known patterns.
It should be appreciated that when the inference model is a neural network configured to output a discrete output (e.g., a discrete signal), the neural network is not required to produce outputs that add up to one. For example, instead of a softmax layer, the output layer of the neural network may be a sigmoid layer (which does not restrict the outputs to probabilities that add up to one). In such embodiments, the neural network may be trained with a sigmoid cross-entropy cost. Such an implementation may be advantageous in the case when multiple different control actions may occur within a threshold amount of time and it is not important to distinguish the order in which these control actions occur (e.g., a user may activate two patterns of neural activity within the threshold amount of time). In some embodiments, any other suitable non-probabilistic multi-class classifier may be used, as aspects of the technology described herein are not limited in this respect.
In some embodiments, the output(s) of the inference model(s) may be continuous signals rather than discrete signals. For example, the model may output an estimate of the firing rate of each motor unit or the model may output a time-series electrical signal corresponding to each motor unit or sub-muscular structure.
It should be appreciated that aspects of the technology described herein are not limited to using neural networks, as other types of inference models may be employed in some embodiments. For example, in some embodiments, the inference model may comprise a hidden Markov model (HMM), a switching HMM with the switching allowing for toggling among different dynamic systems, dynamic Bayesian networks, and/or any other suitable graphical model having a temporal component. Any such inference model may be trained using recorded sensor signals.
As another example, in some embodiments, the inference model is a classifier taking as input, features derived from the recorded sensor signals. In such embodiments, the classifier may be trained using features extracted from the sensor data. The classifier may be a support vector machine, a Gaussian mixture model, a regression based classifier, a decision tree classifier, a Bayesian classifier, and/or any other suitable classifier, as aspects of the technology described herein are not limited in this respect. Input features to be provided to the classifier may be derived from the sensor data in any suitable way. For example, the sensor data may be analyzed as time series data using wavelet analysis techniques (e.g., continuous wavelet transform, discrete-time wavelet transform, etc.), Fourier-analytic techniques (e.g., short-time Fourier transform, Fourier transform, etc.), and/or any other suitable type of time-frequency analysis technique. As one non-limiting example, the sensor data may be transformed using a wavelet transform and the resulting wavelet coefficients may be provided as inputs to the classifier.
In some embodiments, values for parameters of the inference model may be estimated from training data. For example, when the inference model is a neural network, parameters of the neural network (e.g., weights) may be estimated from the training data. In some embodiments, parameters of the inference model may be estimated using gradient descent, stochastic gradient descent, and/or any other suitable iterative optimization technique. In embodiments where the inference model is a recurrent neural network (e.g., an LSTM), the inference model may be trained using stochastic gradient descent and backpropagation through time. The training may employ a cross-entropy loss function and/or any other suitable loss function, as aspects of the technology described herein are not limited in this respect.
As discussed above, some embodiments are directed to using an inference model for predicting musculoskeletal information based on signals recorded from wearable sensors. As discussed briefly above in the example where portions of the human musculoskeletal system can be modeled as a multi-segment articulated rigid body system, the types of joints between segments in a multi-segment articulated rigid body model constrain movement of the rigid body. Additionally, different human individuals tend to move in characteristic ways when performing a task that can be captured in inference patterns of individual user behavior. At least some of these constraints on human body movement may be explicitly incorporated into inference models used for prediction in accordance with some embodiments. Additionally or alternatively, the constraints may be learned by the inference model though training based on recorded sensor data, as discussed briefly above.
As discussed above, some embodiments are directed to using an inference model for predicting handstate information to enable the generation and/or real-time update of a computer-based musculoskeletal representation. The inference model may be used to predict the handstate information based on IMU signals, neuromuscular signals (e.g., EMG, MMG, and/or SMG signals), external device signals (e.g., camera or laser-scanning signals), or a combination of IMU signals, neuromuscular signals, and external device signals detected as a user performs one or more movements. For instance, as discussed above, a camera associated with an XR system may be used to capture actual position data relating to a human subject of the computer-based musculoskeletal representation and such actual position information may be used to improve the accuracy of the representation. Further, outputs of the inference model may be used to generate a visual representation of the computer-based musculoskeletal representation in an XR environment. For example, a visual representation of muscle groups firing, force being applied, text being entered via movement, or other information produced by the computer-based musculoskeletal representation may be rendered in a visual display of an XR system. In some embodiments, other input/output devices (e.g., auditory inputs/outputs, haptic devices, etc.) may be used to further improve the accuracy of the overall system and/or user experience.
Some embodiments of the technology described herein are directed to using an inference model, at least in part, to map one or more actions identified from the neuromuscular signals (e.g., map muscular activation state information identified from the neuromuscular sensors) to input signals including text. The inference model may receive as input IMU signals, neuromuscular signals (e.g., EMG, MMG, and/or SMG signals), external device signals (e.g., camera or laser-scanning signals), or a combination of IMU signals, neuromuscular signals, and external device signals detected as a user performs one or more sub-muscular activations, one or more movements, and/or one or more gestures. The inference model may be used to predict the input to be provided to the AR system without the user having to make perceptible movements.
Generally, an augmented reality (AR) system 201 may take the form of a pair of goggles, glasses, or other type(s) of device that shows display elements to the user that may be superimposed on “reality” which in some cases could be a user's view of the environment (e.g., as viewed through the user's eyes), or a captured (e.g., by cameras, for example) version of a user's view of the environment. In some embodiments, AR system 201 may include one or more cameras (e.g., camera(s) 204) mounted within a device worn by a user that captures one or more views experienced by the user in their environment. System 201 may have one or more processors 205 operating within the user device and/or within a peripheral device or computer system, and such processor(s) may be capable of transmitting and receiving video information and other types of sensor data.
AR system 201 may also include one or more sensors 207 such as microphones, GPS elements, accelerometers, infrared detectors, haptic feedback elements or any other type of sensor, or any combination thereof. In some embodiments, the AR system 201 may be an audio-based AR system and the one or more sensors 207 may also include one or more headphones or speakers. Further, AR system 201 may also have one or more displays 208 that permit the AR system to overlay and/or display information to the user in addition to the view of the user's environment presented by the AR system. AR system 201 may also include one or more communication interfaces (e.g. interfaces 206) for the purpose of communicating information to one or more computer systems (e.g., a gaming system or other systems capable of rendering or receiving AR data). AR systems can take many forms and are provided by a number of different manufacturers. For example, various embodiments may be implemented in association with one or more types of AR systems. For example, various embodiments may be implemented with the HoloLens holographic reality glasses available from the Microsoft Corporation, the Lightwear AR headset from Magic Leap, the Google Glass AR glasses available from Alphabet, the R-7 Smartglasses System available from ODG, or any other type of AR and/or VR device. Although discussed by way of example, it should be appreciated that one or more embodiments may be implemented within VR or XR systems.
AR system 201 may be operatively coupled to the neuromuscular activity system 202 through one or more communication methods, including but not limited to, the Bluetooth protocol, Wi-Fi, Ethernet-like protocols, or any number of connection types, wireless and/or wired. It should be appreciated that, for example, systems 201 and 202 may be directly connected or coupled through one or more intermediate computer systems or network elements. The double-headed arrow in
Neuromuscular activity system 202 may be similar in structure and function to system 100 described above with reference to
Although
As used herein, the term “gestures” refers to a static or dynamic configuration of one or more body parts including the position of the one or more body parts and forces associated with the configuration. For example, gestures performed by the user include static/discrete gestures (also referred to as “pose”) that indicate a static configuration of one or more body parts. For example, a pose can include a fist, an open hand, statically placing or pressing the palm of the hand down on a solid surface or grasping a ball. A pose can indicate the static configuration by providing positional information (e.g., segment coordinates, joint angles, or similar information) for the pose, or by providing an identifier corresponding to a pose (e.g., a parameter, function argument, or variable value). The gestures performed by the user may include dynamic/continuous gestures that indicate a dynamic configuration of one or more body parts. The dynamic configuration can describe the position of the one or mode body parts, the movement of the one or more body parts, and forces associated with the dynamic configuration. For example, a dynamic gesture can include waving a finger back and forth, throwing a ball or grasping and throwing a ball. Gestures may include covert gestures that are imperceptible to another person, such as slightly tensing a joint by co-contracting opposing muscles, or using sub-muscular activations. Gestures may be defined by an application configured to prompt a user to perform the gestures or, alternatively, gestures may be arbitrarily defined by a user. The gestures performed by the user may include symbolic gestures (e.g., gestures mapped to other gestures, interactions, or commands, for example, based on a gesture vocabulary that specifies the mapping). In some cases, hand and arm gestures may be symbolic and used to communicate according to cultural standards.
In some embodiments, the movements or gestures performed by the user may include tapping or typing actions such as, tapping or typing actions on a surface of a physical keyboard, tapping or typing actions on a surface that has a virtual keyboard projected thereon by the AR system 201, tapping or typing actions without any virtual representation of a keyboard and/or typing actions or other gestures performed in mid-air (e.g., not on a surface).
In some embodiments, the movements or gestures performed by the user may include writing actions such as, writing actions performed on a surface with a physical stylus, a physical writing implement, or fingertip or fingertips of the user (e.g., a user might be imagining that he is holding a pen or stylus by holding his fingertips together in a writing position), writing actions performed on a surface with a virtual stylus or virtual writing implement, and/or writing actions performed with a physical writing implement, a virtual writing implement, or fingertip(s) of the user in mid-air and not on a particular surface.
In some embodiments, the movements or gestures performed by the user may include drawing actions such as, drawing actions performed on a surface including drawing one or more line segments and/or curves and/or swiping through a virtual keyboard (e.g., virtual swipe keyboard) projected by the AR system 201.
In some embodiments, the movements or gestures performed by the user may include one-handed actions such as one-handed chord gestures including squeezes, taps or pinches with various fingers or combinations of fingers of one hand.
In addition to a plurality of neuromuscular sensors, some embodiments include one or more auxiliary sensors configured to record auxiliary signals that may also be provided as input to the one or more trained inference models. Examples of auxiliary sensors include IMU sensors, imaging devices, radiation detection devices (e.g., laser scanning devices), heart rate monitors, or any other type of biosensors configured to record biophysical information from the user during performance of one or more movements or gestures mentioned above. In some embodiments, the neuromuscular signals may be associated or correlated with information detected from the auxiliary sensors (e.g., auxiliary signals providing information indicative of a user's physiological state and/or behavior). For example, the auxiliary signals may be used together with the neuromuscular signals to interpret the user's movements, gestures, actions or otherwise augment and enhance the neuromuscular signals or the input identification process described in detail below.
Process 300 then proceeds to act 304, where the raw sensor signals recorded by the sensors 102 are optionally processed. In some embodiments, the raw sensor signals may be processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and/or rectification). In other embodiments, at least some signal processing of the raw sensor signals may be performed in software. Accordingly, signal processing of the raw sensor signals recorded by the sensors may be performed in hardware, software, or by any suitable combination of hardware and software. In some implementations, the raw sensor signals may be processed to derive other signal data. For example, accelerometer data recorded by one or more IMU sensors may be integrated and/or filtered to determine derived signal data associated with one or more muscles during activation of a muscle or performance of a gesture.
Process 300 then proceeds to act 306, where the raw sensor signals or the processed sensor signals are optionally provided as input to a trained inference model(s) configured to output information representing user activity, such as handstate information and/or muscular activation state information (e.g., a gesture or pose), as described above.
Process 300 then proceeds to act 308, where input to be provided to the AR system 201 is identified based on the raw sensor signals, the processed sensor signals, and/or the outputs of the trained inference model(s) (e.g., the handstate information). In some embodiments, input to be provided to the AR system 201 may be identified based on the movements, gestures, or actions identified from the raw sensor signals, the processed sensor signals, and/or the outputs of the trained inference model(s). For example, text input to be provided to the AR system 201 may be identified based on the tapping or typing actions, writing actions, drawing actions, and/or one-handed actions. Input other than or in addition to text input may be identified, for example, a drawing may be identified based on the drawing actions.
According to some embodiments, the one or more computer processors 104 of system 100 may be programmed to identify the input to be provided to the AR system 201 from signals recorded by sensors 102 (e.g., the raw sensor signals) and/or information based on these signals. The information based on the signals recorded by sensors 102 may include information associated with processed sensor signals (e.g., processed EMG signals) and/or information associated with outputs of the trained inference model (e.g., handstate information).
According to some embodiments, input to be provided to the AR system 201 may be identified based on signals output from the auxiliary sensors (e.g., one or more IMU sensors, one or more cameras or imaging devices associated with neuromuscular activity system 202 or augmented reality system 201) in addition to the signals recorded by the neuromuscular sensors. Such auxiliary sensors can provide additional information regarding the movement of the pen, stylus, fingertip(s), when the user performs the various movements, gestures and/or actions. The additional information can be used to improve the accuracy of the identification process.
In some embodiments, the identified input may be provided to the AR system 201. The AR system 201 may provide visual feedback by displaying an indication of the identified input to the user (and/or may provide other forms of feedback such as audio or haptic feedback). The visual feedback may facilitate text entry, for example, by prompting the user to adjust the way various movements, gestures, and/or actions are performed. The visual feedback may be useful in situations where the user provides input using an object or the user's hand/fingertip, which does not leave physical marks when writing or drawing on a surface, for example. In some embodiments, the indication of the identified input includes text input identified based on the tapping or typing actions, writing actions, drawing actions, and/or one-handed actions performed by the user. In some embodiments, the indication of the identified input includes a listing of one or more suggested or predicted words or phrases for text input. For example, multiple options, guesses or alternative words may be presented to the user. The user may select from among the presented items by, for example, performing certain movements or gestures (that are identified based on neuromuscular signals) or using alternative control schemes (e.g., a cursor/pointer). In some embodiments, the indication of the identified input includes one or more virtual ink marks associated with one or more strokes made by a writing implement. In some embodiments, the indication of the identified input includes a drawing identified based on drawing actions performed by the user. In some embodiments, the indication may be displayed via a user interface presented with an augmented reality environment provided by the AR system 201. For example, the indication may be provided on a virtual document in the user interface or as a representation shown in the AR environment to be floating in space. In some embodiments, the indication may be rendered onto a surface that the user is interacting with by the AR system 201. The indication may be rendered onto the surface where the user is typing, for example, as a scrolling tickertape or a line-oriented typewriter. The indication may be rendered onto the surface where the user is writing, for example, as virtual ink on the surface.
In act 404, a determination may be made that the neuromuscular activity system 202 is in a mode configured to provide input including text to the AR system 201. The mode may include a typing mode in which a user may perform tapping or typing actions on a physical or virtual keyboard to provide text input, a writing mode in which a user may perform writing actions with a physical or virtual writing implement (e.g., pen, stylus, etc.) and/or fingertip(s) to provide text input, a drawing mode in which a user may perform drawing actions with a physical or virtual writing implement (e.g., pen, stylus, etc.) and/or fingertip(s) to provide text and/or drawing input, a one-handed mode in which a user may perform one-handed actions to provide text input, and/or a mode in which discrete and/or continuous control signals may be provided as input to the AR system 201.
In some embodiments, the mode determination may be made based on a user selection of the mode. In other words, the mode that the neuromuscular activity system 202 is in may be determined in response to receiving a user selection of the mode. The user selection may be received from a user interface displayed in an AR environment provided by the AR system 201. The user interface may identify and display a number of modes from which the user may select a particular mode. For example, a list of available modes, such as, typing mode, writing mode, drawing mode, and/or one-handed mode may be provided and the user may select a mode from the list.
In some embodiments, the mode determination may be made based on the sensor signals and/or information based on the sensor signals. In other words, the mode that the neuromuscular activity system 202 is in may be determined based on the sensor signals and/or information based on the sensor signals. In one embodiment, a particular gesture performed by the user may be identified based on the sensor signals and/or information based on the sensor signals, and the mode may be determined by identifying the mode corresponding to the particular gesture. For example, different gestures may be mapped to different modes and a particular mode may be determined based on a corresponding gesture performed by the user. The mode entered based on a particular gesture or muscular activation state may depend on the state of the system (e.g., a current mode of the system) and/or may be personalized according to a user's preferred settings. In some embodiments, the mode may be determined as the user performs one or more actions associated with the corresponding mode. For example, when the user starts performing typing actions, the neuromuscular activity system 202 may be configured to recognize that the input mode is a typing mode and when the user starts performing writing actions, the neuromuscular activity system 202 may be configured to recognize that the input mode is a writing mode. The neuromuscular activity system 202 may switch from one mode to another mode based on detection of different actions performed by the user. For example, the user may switch between performing typing actions and writing actions and the system may determine that the input mode should switch between the typing mode and the writing mode accordingly without interrupting text entry.
In some embodiments, the mode determination may be made based on a signal received from the AR system 201. In other words, the neuromuscular activity system 202 may be configured to operate in a mode determined in response to receiving a signal from the AR system. The AR system 201 may generate the signal in response to detection of an event for which input within an AR environment provided by the AR system is desired. For example, text input may be desired to complete a portion of a form presented in a user interface displayed in the AR environment. Presentation of the form may trigger a signal to be generated by the AR system indicating that text input is desired. The signal may identify the various modes that are available for providing the input. The AR system 201 may communicate the signal to the neuromuscular activity system 202 and the neuromuscular activity system 202 may switch to particular available mode to provide the text input.
In act 406, the input to be provided to the AR system 201 may be identified based on the raw or processed signals and/or information based on the recorded signals (e.g., handstate and/or muscular activation state information). In some embodiments, the one or more computer processors of system 100 may be programmed to identify the input based on the sensor signals, the handstate information, detection of a gesture or muscular activation state, and/or a combination of any of the foregoing.
In some embodiments, the input to be provided to the AR system 201 may be further identified based on the current mode of the neuromuscular activity system 202. When the neuromuscular activity system 202 is in a typing mode, input to be provided to the AR system 201 for the typing mode may be identified by identifying one or more tapping or typing actions performed by a user based on the sensor signals and/or information based on the sensor signals. For example, tapping or typing actions performed on a surface of a physical keyboard or a surface that has a virtual keyboard projected thereon by the AR system may be identified based on the sensor signals and text input for the typing mode may be identified based on the tapping/typing actions.
When the neuromuscular activity system 202 is in a writing mode, input to be provided to the AR system 201 for the writing mode may be identified by identifying one or more writing actions performed by the user based on the sensor signals and/or information based on the sensor signals. For example, writing actions performed on a surface with a physical writing implement, a virtual writing implement and/or fingertip(s) of the user may be identified based on the sensor signals and text input for the writing mode may be identified based on the writing actions.
When the neuromuscular activity system 202 is in a drawing mode, input to be provided to the AR system for the drawing mode may be identified by identifying one or more drawing actions (e.g., drawing a number of line segments and/or curves on a surface) performed by the user based on the sensor signals and/or information based on the sensor signals. Input (e.g., text input and/or drawing input) for the drawing mode may be identified based on the drawing actions. In some embodiments, the input for the drawing mode may include one or more line segments and/or curves. In some embodiments, the input for the drawing mode may include input determined based on a sequence of pixel positions controlled by the drawing actions performed by the user.
In some embodiments, both the text input and the drawing may be identified based on the drawing actions performed by the user. In some implementations, processing of the sensor signals may be performed by multiple processors. The neuromuscular sensors may be configured to communicate at least some of the sensor signals to a first computer processor and a second computer processor, where drawings may be identified by the first computer processor and text input (e.g., handwriting) may be identified by the second computer processor. The text input and the drawing from the first and second computer processors may be combined such that the text overlays or annotates the drawing, or is stored as metadata for later processing (e.g., search and filtering). In other implementations, the drawing may be identified based on the drawing actions performed by the user and the text input may be identified from the drawing. For example, the drawing may be identified from the sensor signals and text may be identified from the drawing by running a handwriting recognition process on the drawing.
When the neuromuscular activity system 202 is in a one-handed mode (i.e., a mode where the user uses only one hand to provide input), input to be provided to the AR system for the one-handed mode may be identified by identifying one or more one-handed actions (for example, squeezing, pinching, and/or tapping of various fingers and combinations of fingers) performed by the user based on the sensor signals and/or information based on the sensor signals. Text input for the one-handed mode may be identified based on the one-handed actions.
In some embodiments, one or more gestures may be identified in addition to the typing/tapping, writing, drawing, and/or one-handed actions to allow editing and/or correction of identified text. For example, one or more delete gestures may be recognized in addition to writing actions (based on which text input is identified) that allow deletion of identified letters or words in the text input. The one or more delete gestures may include a gesture to delete a single letter, a gesture to delete a previous word, and/or a gesture to delete a selected word. In some embodiments, the selection of the word to be deleted may be accomplished using neuromuscular controls, for example, cursor navigation. The one or more delete gestures may involve manipulating an object being held by a user (e.g., a stylus or pencil). For example, the one or more delete gestures may include flipping the object, such as a pencil, to an eraser position and then swiping or pressing an imaginary button on the object with a particular finger to initiate deletion of one or more letters or words.
In some embodiments, one or more gestures (such as newline gestures that indicate the end of a line of text and start of a new line of text, space gestures that indicate a space break in text, and/or other gestures) may be identified and combined with recognizing text input to allow the user to compose longer sequences of text without having to physically move his hand (e.g., to the right or down a virtual page in a virtual document). For example, a swipe or flick in a particular direction may be used as a newline gesture and a “pen up” motion may be used for space or word breaks.
In act 408, the input identified in act 406 may be provided to the AR system 201. Text input and/or drawing input identified based on the sensor signals and/or information based on the sensor signals may be provided to the AR system 201. The one or more computer processors of system 100 may identify and provide the input to the AR system.
In some embodiments, the neuromuscular activity system 202 may switch between different modes, for example typing, writing, drawing, and/or one-handed modes, for providing input. For example, a user may provide text-based input by tapping on a surface of a physical keyboard, writing on a surface with a stylus, swiping though a virtual swipe keyboard projected in the AR environment, or using a custom movement-free mapping from neuromuscular signals to text. These different approaches may all be integrated with the AR system 201 though a common application programming interface (API). In other words, the different forms of text input may be identified by the neuromuscular activity system 202 and provided to the AR system 201, where the AR system receives the different forms of text input via a common text API.
In some embodiments, the input to be provided to the AR system 201 may be identified from multiple sources, where the sources may include the neuromuscular signals and at least one source other than the neuromuscular signals. For example, the at least one source may include a physical input device such as a physical keyboard or stylus. Input received from the multiple sources may be combined and the combined input may be provided to the AR system 201. In some implementations, the common API may receive input from the multiple sources. In some embodiments, visual feedback provided by the AR system may continue regardless of the source, the mode or the form of text entry.
In some embodiments, when used in combination with physical input devices, the neuromuscular activity system 202 may learn to emulate the physical input devices using the neuromuscular signals, thereby allowing seamless switching between the physical input devices and their virtual emulations.
In some embodiments, sensors 710 include a set of neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensors 710 can include a set of neuromuscular sensors and at least one “auxiliary” sensor configured to continuously record auxiliary signals. Examples of auxiliary sensors include, but are not limited to, other sensors such as IMU sensors, microphones, imaging sensors (e.g., a camera), radiation based sensors for use with a radiation-generation device (e.g., a laser-scanning device), or other types of sensors such as a heart-rate monitor. As shown the sensors 710 may be coupled together using flexible electronics 730 incorporated into the wearable device.
In some embodiments, the output of one or more of the sensing components can be optionally processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and/or rectification). In other embodiments, at least some signal processing of the output of the sensing components can be performed in software. Thus, signal processing of signals sampled by the sensors can be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect. A non-limiting example of a signal processing chain used to process recorded data from sensors 710 are discussed in more detail below in connection with
Dongle portion 820 includes antenna 852 configured to communicate with antenna 850 included as part of wearable portion 810. Communication between antenna 850 and 852 may occur using any suitable wireless technology and protocol, non-limiting examples of which include radiofrequency signaling and Bluetooth. As shown, the signals received by antenna 852 of dongle portion 820 may be provided to a host computer for further processing, display, and/or for effecting control of a particular physical or virtual object or objects.
Although the examples provided with reference to
It will be appreciated that the disclosure is not limited to the use of typing, writing, drawing, and/or one-handed modes or identifying input based on tapping/typing actions, writing actions, drawing actions, and/or one-handed actions, and other modes or actions can be used. For example, two-handed actions other that typing, tapping, writing, or drawing on a surface, such as, combinations of fingertip squeezes, hand gestures, or finger movements on both hands may be used without departing from the scope of this disclosure.
The above-described embodiments can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. It should be appreciated that any component or collection of components that perform the functions described above can be generically considered as one or more controllers that control the above-discussed functions. The one or more controllers can be implemented in numerous ways, such as with dedicated hardware or with one or more processors programmed using microcode or software to perform the functions recited above.
In this respect, it should be appreciated that one implementation of the embodiments of the present invention comprises at least one non-transitory computer-readable storage medium (e.g., a computer memory, a portable memory, a compact disk, etc.) encoded with a computer program (i.e., a plurality of instructions), which, when executed on a processor, performs the above-discussed functions of the embodiments of the present invention. The computer-readable storage medium can be transportable such that the program stored thereon can be loaded onto any computer resource to implement the aspects of the present invention discussed herein. In addition, it should be appreciated that the reference to a computer program which, when executed, performs the above-discussed functions, is not limited to an application program running on a host computer. Rather, the term computer program is used herein in a generic sense to reference any type of computer code (e.g., software or microcode) that can be employed to program a processor to implement the above-discussed aspects of the present invention.
Various aspects of the present invention may be used alone, in combination, or in a variety of arrangements not specifically discussed in the embodiments described in the foregoing and are therefore not limited in their application to the details and arrangement of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.
Also, embodiments of the invention may be implemented as one or more methods, of which an example has been provided. The acts performed as part of the method(s) may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.
The foregoing features may be used, separately or together in any combination, in any of the embodiments discussed herein.
Further, although advantages of the present invention may be indicated, it should be appreciated that not every embodiment of the invention will include every described advantage. Some embodiments may not implement any features described as advantageous herein. Accordingly, the foregoing description and attached drawings are by way of example only.
Variations on the disclosed embodiment are possible. For example, various aspects of the present technology may be used alone, in combination, or in a variety of arrangements not specifically discussed in the embodiments described in the foregoing, and therefore they are not limited in application to the details and arrangements of components set forth in the foregoing description or illustrated in the drawings. Aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.
Use of ordinal terms such as “first,” “second,” “third,” etc., in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed. Such terms are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term).
The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”
Any use of the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.
Any use of the phrase “equal” or “the same” in reference to two values (e.g., distances, widths, etc.) means that two values are the same within manufacturing tolerances. Thus, two values being equal, or the same, may mean that the two values are different from one another by ±5%.
The phrase “and/or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and/or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and/or” clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to “A and/or B”, when used in conjunction with open-ended language such as “comprising” can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); etc.
As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and/or” as defined above. For example, when separating items in a list, “or” or “and/or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of” or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.” “Consisting essentially of,” when used in the claims, shall have its ordinary meaning as used in the field of patent law.
The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” “having,” “containing”, “involving”, and variations thereof, is meant to encompass the items listed thereafter and additional items.
Having described several embodiments of the invention in detail, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to be within the spirit and scope of the invention. Accordingly, the foregoing description is by way of example only, and is not intended as limiting. The invention is limited only as defined by the following claims and the equivalents thereto.
This application claims the benefit under 35 U.S.C. § 119(e) of U.S. Provisional Patent Application Ser. No. 62/734,138, filed Sep. 20, 2018, entitled “NEUROMUSCULAR TEXT ENTRY, WRITING AND DRAWING IN AUGMENTED REALITY SYSTEMS,” the entire contents of which is incorporated by reference herein.
| Number | Name | Date | Kind |
|---|---|---|---|
| 1411995 | Dull | Apr 1922 | A |
| 3580243 | Johnson | May 1971 | A |
| 3620208 | Wayne et al. | Nov 1971 | A |
| 3735425 | Hoshall et al. | May 1973 | A |
| 3880146 | Everett et al. | Apr 1975 | A |
| 4055168 | Miller et al. | Oct 1977 | A |
| 4602639 | Hoogendoorn et al. | Jul 1986 | A |
| 4705408 | Jordi | Nov 1987 | A |
| 4817064 | Milles | Mar 1989 | A |
| 4896120 | Kamil | Jan 1990 | A |
| 5003978 | Dunseath, Jr. | Apr 1991 | A |
| D322227 | Warhol | Dec 1991 | S |
| 5081852 | Cox | Jan 1992 | A |
| 5251189 | Thorp | Oct 1993 | A |
| D348660 | Parsons | Jul 1994 | S |
| 5445869 | Ishikawa et al. | Aug 1995 | A |
| 5462065 | Cusimano | Oct 1995 | A |
| 5482051 | Reddy et al. | Jan 1996 | A |
| 5605059 | Woodward | Feb 1997 | A |
| 5625577 | Kunii et al. | Apr 1997 | A |
| 5683404 | Johnson | Nov 1997 | A |
| 6005548 | Latypov et al. | Dec 1999 | A |
| 6009210 | Kand | Dec 1999 | A |
| 6032530 | Hock | Mar 2000 | A |
| 6066794 | Longo | May 2000 | A |
| 6184847 | Fateh et al. | Feb 2001 | B1 |
| 6238338 | DeLuca et al. | May 2001 | B1 |
| 6244873 | Hill et al. | Jun 2001 | B1 |
| 6377277 | Yamamoto | Apr 2002 | B1 |
| D459352 | Giovanniello | Jun 2002 | S |
| 6411843 | Zarychta | Jun 2002 | B1 |
| 6487906 | Hock | Dec 2002 | B1 |
| 6510333 | Licata et al. | Jan 2003 | B1 |
| 6527711 | Stivoric et al. | Mar 2003 | B1 |
| 6619836 | Silvant et al. | Sep 2003 | B1 |
| 6658287 | Litt et al. | Dec 2003 | B1 |
| 6720984 | Jorgensen et al. | Apr 2004 | B1 |
| 6743982 | Biegelsen et al. | Jun 2004 | B2 |
| 6771294 | Pulli et al. | Aug 2004 | B1 |
| 6774885 | Even-Zohar | Aug 2004 | B1 |
| 6807438 | Brun Del Re et al. | Oct 2004 | B1 |
| D502661 | Rapport | Mar 2005 | S |
| D502662 | Rapport | Mar 2005 | S |
| 6865409 | Getsla et al. | Mar 2005 | B2 |
| D503646 | Rapport | Apr 2005 | S |
| 6880364 | Vidolin et al. | Apr 2005 | B1 |
| 6901286 | Sinderby et al. | May 2005 | B1 |
| 6927343 | Watanabe et al. | Aug 2005 | B2 |
| 6942621 | Avinash et al. | Sep 2005 | B2 |
| 6965842 | Rekimoto | Nov 2005 | B2 |
| 6972734 | Ohshima et al. | Dec 2005 | B1 |
| 6984208 | Zheng | Jan 2006 | B2 |
| 7022919 | Brist et al. | Apr 2006 | B2 |
| 7086218 | Pasach | Aug 2006 | B1 |
| 7089148 | Bachmann et al. | Aug 2006 | B1 |
| D535401 | Travis et al. | Jan 2007 | S |
| 7173437 | Hervieux et al. | Feb 2007 | B2 |
| 7209114 | Radley-Smith | Apr 2007 | B2 |
| D543212 | Marks | May 2007 | S |
| 7265298 | Maghribi et al. | Sep 2007 | B2 |
| 7271774 | Puuri | Sep 2007 | B2 |
| 7333090 | Tanaka et al. | Feb 2008 | B2 |
| 7351975 | Brady et al. | Apr 2008 | B2 |
| 7450107 | Radley-Smith | Nov 2008 | B2 |
| 7491892 | Wagner et al. | Feb 2009 | B2 |
| 7517725 | Reis | Apr 2009 | B2 |
| 7558622 | Tran | Jul 2009 | B2 |
| 7574253 | Edney et al. | Aug 2009 | B2 |
| 7580742 | Tan et al. | Aug 2009 | B2 |
| 7596393 | Jung et al. | Sep 2009 | B2 |
| 7618260 | Daniel et al. | Nov 2009 | B2 |
| 7636549 | Ma et al. | Dec 2009 | B2 |
| 7640007 | Chen et al. | Dec 2009 | B2 |
| 7660126 | Cho et al. | Feb 2010 | B2 |
| 7761390 | Ford | Jul 2010 | B2 |
| 7787946 | Stahmann et al. | Aug 2010 | B2 |
| 7805386 | Greer | Sep 2010 | B2 |
| 7809435 | Ettare et al. | Oct 2010 | B1 |
| 7844310 | Anderson | Nov 2010 | B2 |
| 7870211 | Pascal et al. | Jan 2011 | B2 |
| 7901368 | Flaherty et al. | Mar 2011 | B2 |
| 7925100 | Howell et al. | Apr 2011 | B2 |
| 7948763 | Chuang | May 2011 | B2 |
| D643428 | Janky et al. | Aug 2011 | S |
| D646192 | Woode | Oct 2011 | S |
| 8054061 | Prance et al. | Nov 2011 | B2 |
| D654622 | Hsu | Feb 2012 | S |
| 8170656 | Tan et al. | May 2012 | B2 |
| 8179604 | Prada Gomez et al. | May 2012 | B1 |
| 8188937 | Amafuji et al. | May 2012 | B1 |
| 8190249 | Gharieb et al. | May 2012 | B1 |
| D661613 | Demeglio | Jun 2012 | S |
| 8203502 | Chi et al. | Jun 2012 | B1 |
| 8207473 | Axisa et al. | Jun 2012 | B2 |
| 8212859 | Tang et al. | Jul 2012 | B2 |
| 8311623 | Sanger | Nov 2012 | B2 |
| 8348538 | Van Loenen et al. | Jan 2013 | B2 |
| 8351651 | Lee | Jan 2013 | B2 |
| 8355671 | Kramer et al. | Jan 2013 | B2 |
| 8384683 | Luo | Feb 2013 | B2 |
| 8389862 | Arora et al. | Mar 2013 | B2 |
| 8421634 | Tan et al. | Apr 2013 | B2 |
| 8427977 | Workman et al. | Apr 2013 | B2 |
| D682727 | Bulgari | May 2013 | S |
| 8435191 | Barboutis et al. | May 2013 | B2 |
| 8437844 | Syed Momen et al. | May 2013 | B2 |
| 8447704 | Tan et al. | May 2013 | B2 |
| 8467270 | Gossweiler, III et al. | Jun 2013 | B2 |
| 8469741 | Oster et al. | Jun 2013 | B2 |
| 8484022 | Vanhoucke | Jul 2013 | B1 |
| D689862 | Liu | Sep 2013 | S |
| 8591411 | Banet et al. | Nov 2013 | B2 |
| D695454 | Moore | Dec 2013 | S |
| 8620361 | Bailey et al. | Dec 2013 | B2 |
| 8624124 | Koo et al. | Jan 2014 | B2 |
| 8702629 | Giuffrida et al. | Apr 2014 | B2 |
| 8704882 | Turner | Apr 2014 | B2 |
| 8718980 | Garudadri et al. | May 2014 | B2 |
| 8743052 | Keller et al. | Jun 2014 | B1 |
| 8744543 | Li et al. | Jun 2014 | B2 |
| 8754862 | Zaliva | Jun 2014 | B2 |
| 8777668 | Ikeda et al. | Jul 2014 | B2 |
| D716457 | Brefka et al. | Oct 2014 | S |
| D717685 | Bailey et al. | Nov 2014 | S |
| 8879276 | Wang | Nov 2014 | B2 |
| 8880163 | Barachant et al. | Nov 2014 | B2 |
| 8883287 | Boyce et al. | Nov 2014 | B2 |
| 8890875 | Jammes et al. | Nov 2014 | B2 |
| 8892479 | Tan et al. | Nov 2014 | B2 |
| 8895865 | Lenahan et al. | Nov 2014 | B2 |
| 8912094 | Koo et al. | Dec 2014 | B2 |
| 8914472 | Lee et al. | Dec 2014 | B1 |
| 8922481 | Kauffmann et al. | Dec 2014 | B1 |
| 8970571 | Wong et al. | Mar 2015 | B1 |
| 8971023 | Olsson et al. | Mar 2015 | B2 |
| 9018532 | Wesselmann et al. | Apr 2015 | B2 |
| 9037530 | Tan et al. | May 2015 | B2 |
| 9086687 | Park et al. | Jul 2015 | B2 |
| 9092664 | Forutanpour et al. | Jul 2015 | B2 |
| D736664 | Paradise et al. | Aug 2015 | S |
| 9146730 | Lazar | Sep 2015 | B2 |
| D741855 | Park et al. | Oct 2015 | S |
| 9170674 | Forutanpour et al. | Oct 2015 | B2 |
| D742272 | Bailey et al. | Nov 2015 | S |
| D742874 | Cheng et al. | Nov 2015 | S |
| D743963 | Osterhout | Nov 2015 | S |
| 9182826 | Powledge et al. | Nov 2015 | B2 |
| 9211417 | Heldman et al. | Dec 2015 | B2 |
| 9218574 | Phillipps et al. | Dec 2015 | B2 |
| D747714 | Erbeus | Jan 2016 | S |
| 9235934 | Mandella et al. | Jan 2016 | B2 |
| 9240069 | Li | Jan 2016 | B1 |
| D750623 | Park et al. | Mar 2016 | S |
| D751065 | Magi | Mar 2016 | S |
| 9278453 | Assad | Mar 2016 | B2 |
| 9299248 | Lake et al. | Mar 2016 | B2 |
| D756359 | Bailey et al. | May 2016 | S |
| 9329694 | Slonneger | May 2016 | B2 |
| 9341659 | Poupyrev et al. | May 2016 | B2 |
| 9351653 | Harrison | May 2016 | B1 |
| 9367139 | Ataee et al. | Jun 2016 | B2 |
| 9372535 | Bailey et al. | Jun 2016 | B2 |
| 9389694 | Ataee et al. | Jul 2016 | B2 |
| 9393418 | Giuffrida et al. | Jul 2016 | B2 |
| 9402582 | Parviz et al. | Aug 2016 | B1 |
| 9408316 | Bailey et al. | Aug 2016 | B2 |
| 9418927 | Axisa et al. | Aug 2016 | B2 |
| 9439566 | Arne et al. | Sep 2016 | B2 |
| 9459697 | Bedikian et al. | Oct 2016 | B2 |
| 9472956 | Michaelis et al. | Oct 2016 | B2 |
| 9477313 | Mistry et al. | Oct 2016 | B2 |
| 9483123 | Aleem et al. | Nov 2016 | B2 |
| 9529434 | Choi et al. | Dec 2016 | B2 |
| 9597015 | McNames et al. | Mar 2017 | B2 |
| 9600030 | Bailey et al. | Mar 2017 | B2 |
| 9612661 | Wagner et al. | Apr 2017 | B2 |
| 9613262 | Holz | Apr 2017 | B2 |
| 9654477 | Kotamraju | May 2017 | B1 |
| 9659403 | Horowitz | May 2017 | B1 |
| 9687168 | John | Jun 2017 | B2 |
| 9696795 | Marcolina et al. | Jul 2017 | B2 |
| 9720515 | Wagner et al. | Aug 2017 | B2 |
| 9741169 | Holz | Aug 2017 | B1 |
| 9766709 | Holz | Sep 2017 | B2 |
| 9785247 | Horowitz et al. | Oct 2017 | B1 |
| 9788789 | Bailey | Oct 2017 | B2 |
| 9864431 | Keskin et al. | Jan 2018 | B2 |
| 9867548 | Le et al. | Jan 2018 | B2 |
| 9880632 | Ataee et al. | Jan 2018 | B2 |
| 9891718 | Connor | Feb 2018 | B2 |
| 9921641 | Worley, III et al. | Mar 2018 | B1 |
| 10042422 | Morun et al. | Aug 2018 | B2 |
| 10070799 | Ang et al. | Sep 2018 | B2 |
| 10078435 | Noel | Sep 2018 | B2 |
| 10101809 | Morun et al. | Oct 2018 | B2 |
| 10152082 | Bailey | Dec 2018 | B2 |
| 10188309 | Morun et al. | Jan 2019 | B2 |
| 10199008 | Aleem et al. | Feb 2019 | B2 |
| 10203751 | Keskin et al. | Feb 2019 | B2 |
| 10216274 | Chapeskie et al. | Feb 2019 | B2 |
| 10251577 | Morun et al. | Apr 2019 | B2 |
| 10310601 | Morun et al. | Jun 2019 | B2 |
| 10331210 | Morun et al. | Jun 2019 | B2 |
| 10362958 | Morun et al. | Jul 2019 | B2 |
| 10409371 | Kaifosh et al. | Sep 2019 | B2 |
| 10437335 | Daniels | Oct 2019 | B2 |
| 10460455 | Giurgica-Tiron et al. | Oct 2019 | B2 |
| 10489986 | Kaifosh et al. | Nov 2019 | B2 |
| 10496168 | Kaifosh et al. | Dec 2019 | B2 |
| 10504286 | Kaifosh et al. | Dec 2019 | B2 |
| 10520378 | Brown et al. | Dec 2019 | B1 |
| 10528135 | Bailey et al. | Jan 2020 | B2 |
| 10558273 | Park et al. | Feb 2020 | B2 |
| 10592001 | Berenzweig et al. | Mar 2020 | B2 |
| 10610737 | Crawford | Apr 2020 | B1 |
| 10676083 | De Sapio et al. | Jun 2020 | B1 |
| 10687759 | Guo et al. | Jun 2020 | B2 |
| 10905350 | Berenzweig et al. | Feb 2021 | B2 |
| 10905383 | Barachant | Feb 2021 | B2 |
| 10937414 | Berenzweig et al. | Mar 2021 | B2 |
| 10990174 | Kaifosh et al. | Apr 2021 | B2 |
| 11009951 | Bailey et al. | May 2021 | B2 |
| 11150730 | Anderson et al. | Oct 2021 | B1 |
| 20020032386 | Sackner et al. | Mar 2002 | A1 |
| 20020077534 | DuRousseau | Jun 2002 | A1 |
| 20020094701 | Biegelsen et al. | Jul 2002 | A1 |
| 20020198472 | Kramer | Dec 2002 | A1 |
| 20030036691 | Stanaland et al. | Feb 2003 | A1 |
| 20030051505 | Robertson et al. | Mar 2003 | A1 |
| 20030144586 | Tsubata | Jul 2003 | A1 |
| 20030144829 | Geatz et al. | Jul 2003 | A1 |
| 20030171921 | Manabe et al. | Sep 2003 | A1 |
| 20030184544 | Prudent | Oct 2003 | A1 |
| 20040010210 | Avinash et al. | Jan 2004 | A1 |
| 20040054273 | Finneran et al. | Mar 2004 | A1 |
| 20040068409 | Tanaka et al. | Apr 2004 | A1 |
| 20040073104 | Brun del Re et al. | Apr 2004 | A1 |
| 20040080499 | Lui | Apr 2004 | A1 |
| 20040092839 | Shin et al. | May 2004 | A1 |
| 20040194500 | Rapport | Oct 2004 | A1 |
| 20040210165 | Marmaropoulos et al. | Oct 2004 | A1 |
| 20040243342 | Rekimoto | Dec 2004 | A1 |
| 20040254617 | Hemmerling et al. | Dec 2004 | A1 |
| 20050005637 | Rapport | Jan 2005 | A1 |
| 20050012715 | Ford | Jan 2005 | A1 |
| 20050070227 | Shen et al. | Mar 2005 | A1 |
| 20050070791 | Edney et al. | Mar 2005 | A1 |
| 20050115561 | Stahmann et al. | Jun 2005 | A1 |
| 20050119701 | Lauter et al. | Jun 2005 | A1 |
| 20050177038 | Kolpin et al. | Aug 2005 | A1 |
| 20060037359 | Stinespring | Feb 2006 | A1 |
| 20060058699 | Vitiello et al. | Mar 2006 | A1 |
| 20060061544 | Min et al. | Mar 2006 | A1 |
| 20060121958 | Jung et al. | Jun 2006 | A1 |
| 20060129057 | Maekawa et al. | Jun 2006 | A1 |
| 20060149338 | Flaherty et al. | Jul 2006 | A1 |
| 20060211956 | Sankai | Sep 2006 | A1 |
| 20070009151 | Pittman et al. | Jan 2007 | A1 |
| 20070016265 | Davoodi et al. | Jan 2007 | A1 |
| 20070023662 | Brady et al. | Feb 2007 | A1 |
| 20070132785 | Ebersole, Jr. et al. | Jun 2007 | A1 |
| 20070148624 | Nativ | Jun 2007 | A1 |
| 20070172797 | Hada et al. | Jul 2007 | A1 |
| 20070177770 | Derchak et al. | Aug 2007 | A1 |
| 20070185697 | Tan et al. | Aug 2007 | A1 |
| 20070256494 | Nakamura et al. | Nov 2007 | A1 |
| 20070285399 | Lund | Dec 2007 | A1 |
| 20080001735 | Tran | Jan 2008 | A1 |
| 20080051673 | Kong et al. | Feb 2008 | A1 |
| 20080052643 | Ike et al. | Feb 2008 | A1 |
| 20080058668 | Seyed Momen et al. | Mar 2008 | A1 |
| 20080103639 | Troy et al. | May 2008 | A1 |
| 20080103769 | Schultz et al. | May 2008 | A1 |
| 20080136775 | Conant | Jun 2008 | A1 |
| 20080152217 | Greer | Jun 2008 | A1 |
| 20080163130 | Westerman | Jul 2008 | A1 |
| 20080214360 | Stirling et al. | Sep 2008 | A1 |
| 20080221487 | Zohar et al. | Sep 2008 | A1 |
| 20080262772 | Luinge et al. | Oct 2008 | A1 |
| 20080278497 | Jammes et al. | Nov 2008 | A1 |
| 20080285805 | Luinge et al. | Nov 2008 | A1 |
| 20090005700 | Joshi et al. | Jan 2009 | A1 |
| 20090007597 | Hanevold | Jan 2009 | A1 |
| 20090027337 | Hildreth | Jan 2009 | A1 |
| 20090031757 | Harding | Feb 2009 | A1 |
| 20090040016 | Ikeda | Feb 2009 | A1 |
| 20090051544 | Niknejad | Feb 2009 | A1 |
| 20090079607 | Denison et al. | Mar 2009 | A1 |
| 20090079813 | Hildreth | Mar 2009 | A1 |
| 20090082692 | Hale et al. | Mar 2009 | A1 |
| 20090082701 | Zohar et al. | Mar 2009 | A1 |
| 20090102580 | Uchaykin | Apr 2009 | A1 |
| 20090112080 | Matthews | Apr 2009 | A1 |
| 20090124881 | Rytky | May 2009 | A1 |
| 20090189864 | Walker et al. | Jul 2009 | A1 |
| 20090189867 | Krah et al. | Jul 2009 | A1 |
| 20090195497 | Fitzgerald et al. | Aug 2009 | A1 |
| 20090204031 | McNames et al. | Aug 2009 | A1 |
| 20090209878 | Sanger | Aug 2009 | A1 |
| 20090251407 | Flake et al. | Oct 2009 | A1 |
| 20090265671 | Sachs et al. | Oct 2009 | A1 |
| 20090318785 | Ishikawa et al. | Dec 2009 | A1 |
| 20090319230 | Case, Jr. et al. | Dec 2009 | A1 |
| 20090326406 | Tan et al. | Dec 2009 | A1 |
| 20090327171 | Tan | Dec 2009 | A1 |
| 20100030532 | Arora et al. | Feb 2010 | A1 |
| 20100041974 | Ting et al. | Feb 2010 | A1 |
| 20100063794 | Hernandez-Rebollar | Mar 2010 | A1 |
| 20100106044 | Linderman | Apr 2010 | A1 |
| 20100113910 | Brauers et al. | May 2010 | A1 |
| 20100228487 | Leuthardt et al. | Sep 2010 | A1 |
| 20100234696 | Li et al. | Sep 2010 | A1 |
| 20100240981 | Barboutis et al. | Sep 2010 | A1 |
| 20100249635 | Van Der Reijden | Sep 2010 | A1 |
| 20100280628 | Sankai | Nov 2010 | A1 |
| 20100292595 | Paul | Nov 2010 | A1 |
| 20100292606 | Prakash et al. | Nov 2010 | A1 |
| 20100292617 | Lei et al. | Nov 2010 | A1 |
| 20100293115 | Seyed Momen | Nov 2010 | A1 |
| 20100306713 | Geisner et al. | Dec 2010 | A1 |
| 20100315266 | Gunawardana et al. | Dec 2010 | A1 |
| 20100317958 | Beck et al. | Dec 2010 | A1 |
| 20110007035 | Shai | Jan 2011 | A1 |
| 20110018754 | Tojima et al. | Jan 2011 | A1 |
| 20110066381 | Garudadri et al. | Mar 2011 | A1 |
| 20110077484 | Van Slyke et al. | Mar 2011 | A1 |
| 20110082838 | Niemela | Apr 2011 | A1 |
| 20110092826 | Lee et al. | Apr 2011 | A1 |
| 20110119216 | Wigdor | May 2011 | A1 |
| 20110133934 | Tan et al. | Jun 2011 | A1 |
| 20110134026 | Kang et al. | Jun 2011 | A1 |
| 20110151974 | Deaguero | Jun 2011 | A1 |
| 20110166434 | Gargiulo | Jul 2011 | A1 |
| 20110172503 | Knepper et al. | Jul 2011 | A1 |
| 20110173204 | Murillo et al. | Jul 2011 | A1 |
| 20110173574 | Clavin et al. | Jul 2011 | A1 |
| 20110205242 | Friesen | Aug 2011 | A1 |
| 20110213278 | Horak et al. | Sep 2011 | A1 |
| 20110221672 | Osterhout et al. | Sep 2011 | A1 |
| 20110224556 | Moon et al. | Sep 2011 | A1 |
| 20110224564 | Moon et al. | Sep 2011 | A1 |
| 20110230782 | Bartol et al. | Sep 2011 | A1 |
| 20110248914 | Sherr | Oct 2011 | A1 |
| 20110262002 | Lee | Oct 2011 | A1 |
| 20110270135 | Dooley et al. | Nov 2011 | A1 |
| 20110295100 | Hegde et al. | Dec 2011 | A1 |
| 20110313762 | Ben-David et al. | Dec 2011 | A1 |
| 20120007821 | Zaliva | Jan 2012 | A1 |
| 20120029322 | Wartena et al. | Feb 2012 | A1 |
| 20120051005 | VanFleteren et al. | Mar 2012 | A1 |
| 20120066163 | Balls et al. | Mar 2012 | A1 |
| 20120071780 | Barachant et al. | Mar 2012 | A1 |
| 20120101357 | Hoskuldsson et al. | Apr 2012 | A1 |
| 20120117514 | Kim et al. | May 2012 | A1 |
| 20120157789 | Kangas et al. | Jun 2012 | A1 |
| 20120165695 | Kidmose et al. | Jun 2012 | A1 |
| 20120184838 | John | Jul 2012 | A1 |
| 20120188158 | Tan et al. | Jul 2012 | A1 |
| 20120203076 | Fatta et al. | Aug 2012 | A1 |
| 20120209134 | Morita et al. | Aug 2012 | A1 |
| 20120265090 | Fink et al. | Oct 2012 | A1 |
| 20120265480 | Oshima | Oct 2012 | A1 |
| 20120283526 | Gommesen et al. | Nov 2012 | A1 |
| 20120293548 | Perez et al. | Nov 2012 | A1 |
| 20120302858 | Kidmose et al. | Nov 2012 | A1 |
| 20120323521 | De Foras et al. | Dec 2012 | A1 |
| 20130004033 | Trugenberger | Jan 2013 | A1 |
| 20130005303 | Song et al. | Jan 2013 | A1 |
| 20130020948 | Han et al. | Jan 2013 | A1 |
| 20130027341 | Mastandrea | Jan 2013 | A1 |
| 20130038707 | Cunningham et al. | Feb 2013 | A1 |
| 20130077820 | Marais et al. | Mar 2013 | A1 |
| 20130080794 | Hsieh | Mar 2013 | A1 |
| 20130106686 | Bennett | May 2013 | A1 |
| 20130123656 | Heck | May 2013 | A1 |
| 20130127708 | Jung et al. | May 2013 | A1 |
| 20130131538 | Gaw et al. | May 2013 | A1 |
| 20130135223 | Shai | May 2013 | A1 |
| 20130141375 | Ludwig et al. | Jun 2013 | A1 |
| 20130144629 | Johnston et al. | Jun 2013 | A1 |
| 20130165813 | Chang et al. | Jun 2013 | A1 |
| 20130191741 | Dickinson et al. | Jul 2013 | A1 |
| 20130198694 | Rahman et al. | Aug 2013 | A1 |
| 20130207889 | Chang et al. | Aug 2013 | A1 |
| 20130217998 | Mahfouz et al. | Aug 2013 | A1 |
| 20130221996 | Poupyrev et al. | Aug 2013 | A1 |
| 20130232095 | Tan et al. | Sep 2013 | A1 |
| 20130259238 | Xiang et al. | Oct 2013 | A1 |
| 20130265229 | Forutanpour et al. | Oct 2013 | A1 |
| 20130265437 | Thorn et al. | Oct 2013 | A1 |
| 20130271292 | McDermott | Oct 2013 | A1 |
| 20130285913 | Griffin et al. | Oct 2013 | A1 |
| 20130293580 | Spivack | Nov 2013 | A1 |
| 20130310979 | Herr et al. | Nov 2013 | A1 |
| 20130312256 | Wesselmann et al. | Nov 2013 | A1 |
| 20130317382 | Le | Nov 2013 | A1 |
| 20130317648 | Assad | Nov 2013 | A1 |
| 20130332196 | Pinsker | Dec 2013 | A1 |
| 20140020945 | Hurwitz et al. | Jan 2014 | A1 |
| 20140028546 | Jeon et al. | Jan 2014 | A1 |
| 20140045547 | Singamsetty et al. | Feb 2014 | A1 |
| 20140049417 | Abdurrahman et al. | Feb 2014 | A1 |
| 20140052150 | Taylor et al. | Feb 2014 | A1 |
| 20140092009 | Yen et al. | Apr 2014 | A1 |
| 20140094675 | Luna et al. | Apr 2014 | A1 |
| 20140098018 | Kim et al. | Apr 2014 | A1 |
| 20140100432 | Golda et al. | Apr 2014 | A1 |
| 20140107493 | Yuen et al. | Apr 2014 | A1 |
| 20140121471 | Walker | May 2014 | A1 |
| 20140122958 | Greenebrg et al. | May 2014 | A1 |
| 20140142937 | Powledge et al. | May 2014 | A1 |
| 20140143064 | Tran | May 2014 | A1 |
| 20140147820 | Snow et al. | May 2014 | A1 |
| 20140194062 | Palin et al. | Jul 2014 | A1 |
| 20140196131 | Lee | Jul 2014 | A1 |
| 20140198034 | Bailey et al. | Jul 2014 | A1 |
| 20140198035 | Bailey et al. | Jul 2014 | A1 |
| 20140198944 | Forutanpour et al. | Jul 2014 | A1 |
| 20140200432 | Banerji et al. | Jul 2014 | A1 |
| 20140201666 | Bedikian et al. | Jul 2014 | A1 |
| 20140223462 | Aimone et al. | Aug 2014 | A1 |
| 20140236031 | Banet et al. | Aug 2014 | A1 |
| 20140240103 | Lake et al. | Aug 2014 | A1 |
| 20140240223 | Lake | Aug 2014 | A1 |
| 20140245200 | Holz | Aug 2014 | A1 |
| 20140249397 | Lake et al. | Sep 2014 | A1 |
| 20140257141 | Giuffrida et al. | Sep 2014 | A1 |
| 20140277622 | Raniere | Sep 2014 | A1 |
| 20140278139 | Hong et al. | Sep 2014 | A1 |
| 20140278441 | Ton et al. | Sep 2014 | A1 |
| 20140279860 | Pan et al. | Sep 2014 | A1 |
| 20140282282 | Holz | Sep 2014 | A1 |
| 20140285326 | Luna et al. | Sep 2014 | A1 |
| 20140297528 | Agrawal et al. | Oct 2014 | A1 |
| 20140299362 | Park et al. | Oct 2014 | A1 |
| 20140304665 | Holz | Oct 2014 | A1 |
| 20140310595 | Acharya et al. | Oct 2014 | A1 |
| 20140330404 | Abdelghani et al. | Nov 2014 | A1 |
| 20140334083 | Bailey | Nov 2014 | A1 |
| 20140334653 | Luna et al. | Nov 2014 | A1 |
| 20140337861 | Chang et al. | Nov 2014 | A1 |
| 20140340857 | Hsu et al. | Nov 2014 | A1 |
| 20140344731 | Holz | Nov 2014 | A1 |
| 20140349257 | Connor | Nov 2014 | A1 |
| 20140354528 | Laughlin et al. | Dec 2014 | A1 |
| 20140354529 | Laughlin et al. | Dec 2014 | A1 |
| 20140355825 | Kim et al. | Dec 2014 | A1 |
| 20140358024 | Nelson et al. | Dec 2014 | A1 |
| 20140358825 | Phillipps et al. | Dec 2014 | A1 |
| 20140359540 | Kelsey et al. | Dec 2014 | A1 |
| 20140361988 | Katz | Dec 2014 | A1 |
| 20140364703 | Kim et al. | Dec 2014 | A1 |
| 20140365163 | Jallon | Dec 2014 | A1 |
| 20140368428 | Pinault | Dec 2014 | A1 |
| 20140368474 | Kim | Dec 2014 | A1 |
| 20140375465 | Fenuccio et al. | Dec 2014 | A1 |
| 20140376773 | Holz | Dec 2014 | A1 |
| 20150006120 | Sett et al. | Jan 2015 | A1 |
| 20150010203 | Muninder et al. | Jan 2015 | A1 |
| 20150011857 | Henson et al. | Jan 2015 | A1 |
| 20150019135 | Kacyvenski et al. | Jan 2015 | A1 |
| 20150025355 | Bailey et al. | Jan 2015 | A1 |
| 20150029092 | Holz et al. | Jan 2015 | A1 |
| 20150035827 | Yamaoka et al. | Feb 2015 | A1 |
| 20150045689 | Barone | Feb 2015 | A1 |
| 20150045699 | Mokaya et al. | Feb 2015 | A1 |
| 20150051470 | Bailey et al. | Feb 2015 | A1 |
| 20150057506 | Luna et al. | Feb 2015 | A1 |
| 20150057770 | Bailey et al. | Feb 2015 | A1 |
| 20150065840 | Bailey | Mar 2015 | A1 |
| 20150070270 | Bailey et al. | Mar 2015 | A1 |
| 20150070274 | Morozov | Mar 2015 | A1 |
| 20150072326 | Mauri et al. | Mar 2015 | A1 |
| 20150084860 | Aleem et al. | Mar 2015 | A1 |
| 20150091790 | Forutanpour et al. | Apr 2015 | A1 |
| 20150094564 | Tashman et al. | Apr 2015 | A1 |
| 20150099946 | Sahin | Apr 2015 | A1 |
| 20150106052 | Balakrishnan et al. | Apr 2015 | A1 |
| 20150109202 | Ataee et al. | Apr 2015 | A1 |
| 20150124566 | Lake et al. | May 2015 | A1 |
| 20150128094 | Baldwin et al. | May 2015 | A1 |
| 20150141784 | Morun et al. | May 2015 | A1 |
| 20150148641 | Morun et al. | May 2015 | A1 |
| 20150157944 | Gottlieb | Jun 2015 | A1 |
| 20150160621 | Yilmaz | Jun 2015 | A1 |
| 20150169074 | Ataee et al. | Jun 2015 | A1 |
| 20150170421 | Mandella et al. | Jun 2015 | A1 |
| 20150177841 | Vanblon et al. | Jun 2015 | A1 |
| 20150182113 | Utter, II | Jul 2015 | A1 |
| 20150182130 | Utter, II | Jul 2015 | A1 |
| 20150182160 | Kim et al. | Jul 2015 | A1 |
| 20150182163 | Utter | Jul 2015 | A1 |
| 20150182164 | Utter, II | Jul 2015 | A1 |
| 20150182165 | Miller et al. | Jul 2015 | A1 |
| 20150186609 | Utter, II | Jul 2015 | A1 |
| 20150187355 | Parkinson et al. | Jul 2015 | A1 |
| 20150193949 | Katz et al. | Jul 2015 | A1 |
| 20150199025 | Holz | Jul 2015 | A1 |
| 20150213191 | Abdelghani et al. | Jul 2015 | A1 |
| 20150216475 | Luna et al. | Aug 2015 | A1 |
| 20150220152 | Tait et al. | Aug 2015 | A1 |
| 20150223716 | Korkala et al. | Aug 2015 | A1 |
| 20150230756 | Luna et al. | Aug 2015 | A1 |
| 20150234426 | Bailey et al. | Aug 2015 | A1 |
| 20150237716 | Su et al. | Aug 2015 | A1 |
| 20150242009 | Xiao et al. | Aug 2015 | A1 |
| 20150242575 | Abovitz et al. | Aug 2015 | A1 |
| 20150261306 | Lake | Sep 2015 | A1 |
| 20150261318 | Scavezze et al. | Sep 2015 | A1 |
| 20150272483 | Etemad et al. | Oct 2015 | A1 |
| 20150277575 | Ataee et al. | Oct 2015 | A1 |
| 20150288944 | Nistico et al. | Oct 2015 | A1 |
| 20150289995 | Wilkinson et al. | Oct 2015 | A1 |
| 20150296553 | DiFranco et al. | Oct 2015 | A1 |
| 20150302168 | De Sapio et al. | Oct 2015 | A1 |
| 20150305672 | Grey et al. | Oct 2015 | A1 |
| 20150309563 | Connor | Oct 2015 | A1 |
| 20150309582 | Gupta | Oct 2015 | A1 |
| 20150312175 | Langholz | Oct 2015 | A1 |
| 20150313496 | Connor | Nov 2015 | A1 |
| 20150323998 | Kudekar | Nov 2015 | A1 |
| 20150325202 | Lake et al. | Nov 2015 | A1 |
| 20150332013 | Lee et al. | Nov 2015 | A1 |
| 20150346701 | Gordon et al. | Dec 2015 | A1 |
| 20150351690 | Toth | Dec 2015 | A1 |
| 20150355716 | Balasubramanian et al. | Dec 2015 | A1 |
| 20150355718 | Slonneger | Dec 2015 | A1 |
| 20150366504 | Connor | Dec 2015 | A1 |
| 20150370326 | Chapeskie et al. | Dec 2015 | A1 |
| 20150370333 | Ataee et al. | Dec 2015 | A1 |
| 20150379770 | Haley, Jr. et al. | Dec 2015 | A1 |
| 20160011668 | Gilad-Bachrach et al. | Jan 2016 | A1 |
| 20160020500 | Matsuda | Jan 2016 | A1 |
| 20160026853 | Wexler et al. | Jan 2016 | A1 |
| 20160049073 | Lee | Feb 2016 | A1 |
| 20160050037 | Webb | Feb 2016 | A1 |
| 20160071319 | Fallon et al. | Mar 2016 | A1 |
| 20160092504 | Mitri et al. | Mar 2016 | A1 |
| 20160099010 | Sainath et al. | Apr 2016 | A1 |
| 20160107309 | Walsh et al. | Apr 2016 | A1 |
| 20160113587 | Kothe et al. | Apr 2016 | A1 |
| 20160144172 | Hsueh et al. | May 2016 | A1 |
| 20160150636 | Otsubo | May 2016 | A1 |
| 20160156762 | Bailey et al. | Jun 2016 | A1 |
| 20160162604 | Xioli et al. | Jun 2016 | A1 |
| 20160170710 | Kim et al. | Jun 2016 | A1 |
| 20160187992 | Yamamoto et al. | Jun 2016 | A1 |
| 20160195928 | Wagner et al. | Jul 2016 | A1 |
| 20160199699 | Klassen | Jul 2016 | A1 |
| 20160202081 | Debieuvre et al. | Jul 2016 | A1 |
| 20160206206 | Avila et al. | Jul 2016 | A1 |
| 20160207201 | Herr et al. | Jul 2016 | A1 |
| 20160217614 | Kraver et al. | Jul 2016 | A1 |
| 20160235323 | Tadi et al. | Aug 2016 | A1 |
| 20160239080 | Marcolina et al. | Aug 2016 | A1 |
| 20160242646 | Obma | Aug 2016 | A1 |
| 20160259407 | Schick | Sep 2016 | A1 |
| 20160262687 | Imperial | Sep 2016 | A1 |
| 20160263458 | Mather et al. | Sep 2016 | A1 |
| 20160274732 | Bang | Sep 2016 | A1 |
| 20160274758 | Bailey | Sep 2016 | A1 |
| 20160275726 | Mullins | Sep 2016 | A1 |
| 20160282947 | Schwarz et al. | Sep 2016 | A1 |
| 20160291768 | Cho et al. | Oct 2016 | A1 |
| 20160292497 | Kehtarnavaz et al. | Oct 2016 | A1 |
| 20160309249 | Wu et al. | Oct 2016 | A1 |
| 20160313798 | Connor | Oct 2016 | A1 |
| 20160313801 | Wagner et al. | Oct 2016 | A1 |
| 20160313890 | Walline | Oct 2016 | A1 |
| 20160313899 | Noel | Oct 2016 | A1 |
| 20160314623 | Coleman et al. | Oct 2016 | A1 |
| 20160342227 | Natzke et al. | Nov 2016 | A1 |
| 20160350973 | Shapira et al. | Dec 2016 | A1 |
| 20170031502 | Rosenberg et al. | Feb 2017 | A1 |
| 20170035313 | Hong et al. | Feb 2017 | A1 |
| 20170061817 | Mettler May | Mar 2017 | A1 |
| 20170068445 | Lee | Mar 2017 | A1 |
| 20170075426 | Camacho Perez et al. | Mar 2017 | A1 |
| 20170079828 | Pedtke et al. | Mar 2017 | A1 |
| 20170080346 | Abbas | Mar 2017 | A1 |
| 20170090604 | Barbier | Mar 2017 | A1 |
| 20170091567 | Wang et al. | Mar 2017 | A1 |
| 20170095178 | Schoen et al. | Apr 2017 | A1 |
| 20170119472 | Herrmann et al. | May 2017 | A1 |
| 20170123487 | Hazra et al. | May 2017 | A1 |
| 20170124474 | Kashyap | May 2017 | A1 |
| 20170124816 | Yang et al. | May 2017 | A1 |
| 20170147077 | Park et al. | May 2017 | A1 |
| 20170161635 | Oono et al. | Jun 2017 | A1 |
| 20170188878 | Lee | Jul 2017 | A1 |
| 20170188980 | Ash | Jul 2017 | A1 |
| 20170197142 | Stafford et al. | Jul 2017 | A1 |
| 20170209055 | Pantelopoulos et al. | Jul 2017 | A1 |
| 20170220923 | Bae et al. | Aug 2017 | A1 |
| 20170237789 | Harner et al. | Aug 2017 | A1 |
| 20170237901 | Lee et al. | Aug 2017 | A1 |
| 20170259167 | Cook et al. | Sep 2017 | A1 |
| 20170262064 | Ofir et al. | Sep 2017 | A1 |
| 20170277282 | Go | Sep 2017 | A1 |
| 20170285744 | Juliato | Oct 2017 | A1 |
| 20170285756 | Wang et al. | Oct 2017 | A1 |
| 20170285757 | Robertson et al. | Oct 2017 | A1 |
| 20170285848 | Rosenberg et al. | Oct 2017 | A1 |
| 20170296363 | Yetkin et al. | Oct 2017 | A1 |
| 20170301630 | Nguyen et al. | Oct 2017 | A1 |
| 20170308118 | Ito | Oct 2017 | A1 |
| 20170312614 | Tran et al. | Nov 2017 | A1 |
| 20170329392 | Keskin et al. | Nov 2017 | A1 |
| 20170329404 | Keskin et al. | Nov 2017 | A1 |
| 20170340506 | Zhang et al. | Nov 2017 | A1 |
| 20170344706 | Torres et al. | Nov 2017 | A1 |
| 20170347908 | Watanabe et al. | Dec 2017 | A1 |
| 20170371403 | Wetzler et al. | Dec 2017 | A1 |
| 20180000367 | Longinotti-Buitoni | Jan 2018 | A1 |
| 20180018825 | Kim et al. | Jan 2018 | A1 |
| 20180020285 | Zass | Jan 2018 | A1 |
| 20180020951 | Kaifosh et al. | Jan 2018 | A1 |
| 20180020978 | Kaifosh et al. | Jan 2018 | A1 |
| 20180020990 | Park et al. | Jan 2018 | A1 |
| 20180024634 | Kaifosh et al. | Jan 2018 | A1 |
| 20180024635 | Kaifosh et al. | Jan 2018 | A1 |
| 20180024641 | Mao et al. | Jan 2018 | A1 |
| 20180064363 | Morun et al. | Mar 2018 | A1 |
| 20180067553 | Morun et al. | Mar 2018 | A1 |
| 20180068489 | Kim et al. | Mar 2018 | A1 |
| 20180074332 | Li et al. | Mar 2018 | A1 |
| 20180081439 | Daniels | Mar 2018 | A1 |
| 20180088675 | Vogel et al. | Mar 2018 | A1 |
| 20180088765 | Bailey | Mar 2018 | A1 |
| 20180092599 | Kerth et al. | Apr 2018 | A1 |
| 20180093181 | Goslin et al. | Apr 2018 | A1 |
| 20180095542 | Mallinson | Apr 2018 | A1 |
| 20180095630 | Bailey | Apr 2018 | A1 |
| 20180101235 | Bodensteiner et al. | Apr 2018 | A1 |
| 20180101289 | Bailey | Apr 2018 | A1 |
| 20180107275 | Chen | Apr 2018 | A1 |
| 20180120948 | Aleem et al. | May 2018 | A1 |
| 20180133551 | Chang et al. | May 2018 | A1 |
| 20180140441 | Poirters | May 2018 | A1 |
| 20180150033 | Lake et al. | May 2018 | A1 |
| 20180153430 | Ang et al. | Jun 2018 | A1 |
| 20180153444 | Yang et al. | Jun 2018 | A1 |
| 20180154140 | Bouton et al. | Jun 2018 | A1 |
| 20180168905 | Goodall et al. | Jun 2018 | A1 |
| 20180178008 | Bouton et al. | Jun 2018 | A1 |
| 20180217249 | La Salla et al. | Aug 2018 | A1 |
| 20180239430 | Tadi et al. | Aug 2018 | A1 |
| 20180240459 | Weng et al. | Aug 2018 | A1 |
| 20180247443 | Briggs et al. | Aug 2018 | A1 |
| 20180279919 | Bansbach et al. | Oct 2018 | A1 |
| 20180301057 | Hargrove et al. | Oct 2018 | A1 |
| 20180307314 | Connor | Oct 2018 | A1 |
| 20180314879 | Khwaja et al. | Nov 2018 | A1 |
| 20180321745 | Morun et al. | Nov 2018 | A1 |
| 20180321746 | Morun et al. | Nov 2018 | A1 |
| 20180330549 | Brenton | Nov 2018 | A1 |
| 20180333575 | Bouton | Nov 2018 | A1 |
| 20180344195 | Morun et al. | Dec 2018 | A1 |
| 20180356890 | Zhang et al. | Dec 2018 | A1 |
| 20180360379 | Harrison et al. | Dec 2018 | A1 |
| 20190008453 | Spoof | Jan 2019 | A1 |
| 20190025919 | Tadi et al. | Jan 2019 | A1 |
| 20190027141 | Strong et al. | Jan 2019 | A1 |
| 20190033967 | Morun et al. | Jan 2019 | A1 |
| 20190033974 | Mu et al. | Jan 2019 | A1 |
| 20190038166 | Tavabi et al. | Feb 2019 | A1 |
| 20190056422 | Park et al. | Feb 2019 | A1 |
| 20190076716 | Chiou et al. | Mar 2019 | A1 |
| 20190089898 | Kim et al. | Mar 2019 | A1 |
| 20190113973 | Coleman et al. | Apr 2019 | A1 |
| 20190121305 | Kaifosh et al. | Apr 2019 | A1 |
| 20190121306 | Kaifosh et al. | Apr 2019 | A1 |
| 20190146809 | Lee et al. | May 2019 | A1 |
| 20190150777 | Guo et al. | May 2019 | A1 |
| 20190192037 | Morun et al. | Jun 2019 | A1 |
| 20190196585 | Laszlo et al. | Jun 2019 | A1 |
| 20190196586 | Laszlo et al. | Jun 2019 | A1 |
| 20190197778 | Sachdeva et al. | Jun 2019 | A1 |
| 20190209034 | Deno et al. | Jul 2019 | A1 |
| 20190212817 | Kaifosh et al. | Jul 2019 | A1 |
| 20190216619 | McDonnall et al. | Jul 2019 | A1 |
| 20190223748 | Al-natsheh et al. | Jul 2019 | A1 |
| 20190227627 | Kaifosh et al. | Jul 2019 | A1 |
| 20190228330 | Kaifosh et al. | Jul 2019 | A1 |
| 20190228533 | Giurgica-Tiron et al. | Jul 2019 | A1 |
| 20190228579 | Kaifosh et al. | Jul 2019 | A1 |
| 20190228590 | Kaifosh et al. | Jul 2019 | A1 |
| 20190228591 | Giurgica-Tiron et al. | Jul 2019 | A1 |
| 20190247650 | Tran | Aug 2019 | A1 |
| 20190279407 | McHugh et al. | Sep 2019 | A1 |
| 20190294243 | Laszlo et al. | Sep 2019 | A1 |
| 20190324549 | Araki et al. | Oct 2019 | A1 |
| 20190348026 | Berenzweig et al. | Nov 2019 | A1 |
| 20190348027 | Berenzweig et al. | Nov 2019 | A1 |
| 20190357787 | Barachant et al. | Nov 2019 | A1 |
| 20190362557 | Lacey et al. | Nov 2019 | A1 |
| 20200042089 | Ang et al. | Feb 2020 | A1 |
| 20200057661 | Bendfeldt | Feb 2020 | A1 |
| 20200065569 | Nduka et al. | Feb 2020 | A1 |
| 20200069210 | Berenzweig et al. | Mar 2020 | A1 |
| 20200069211 | Berenzweig et al. | Mar 2020 | A1 |
| 20200073483 | Berenzweig et al. | Mar 2020 | A1 |
| 20200077955 | Shui et al. | Mar 2020 | A1 |
| 20200097081 | Stone et al. | Mar 2020 | A1 |
| 20200111260 | Osborn et al. | Apr 2020 | A1 |
| 20200142490 | Xiong et al. | May 2020 | A1 |
| 20200143795 | Park et al. | May 2020 | A1 |
| 20200163562 | Neaves | May 2020 | A1 |
| 20200205932 | Zar et al. | Jul 2020 | A1 |
| 20200225320 | Belskikh et al. | Jul 2020 | A1 |
| 20200245873 | Frank et al. | Aug 2020 | A1 |
| 20200275895 | Barachant | Sep 2020 | A1 |
| 20200301509 | Liu et al. | Sep 2020 | A1 |
| 20200305795 | Floyd et al. | Oct 2020 | A1 |
| 20200320335 | Shamun et al. | Oct 2020 | A1 |
| 20210109598 | Zhang et al. | Apr 2021 | A1 |
| 20210117523 | Kim et al. | Apr 2021 | A1 |
| 20210290159 | Bruinsma et al. | Sep 2021 | A1 |
| 20220256706 | Xiong et al. | Aug 2022 | A1 |
| Number | Date | Country |
|---|---|---|
| 2902045 | Aug 2014 | CA |
| 2921954 | Feb 2015 | CA |
| 2939644 | Aug 2015 | CA |
| 1838933 | Sep 2006 | CN |
| 103777752 | May 2014 | CN |
| 105009031 | Oct 2015 | CN |
| 105190477 | Dec 2015 | CN |
| 105190578 | Dec 2015 | CN |
| 106102504 | Nov 2016 | CN |
| 110300542 | Oct 2019 | CN |
| 111902077 | Nov 2020 | CN |
| 112074225 | Dec 2020 | CN |
| 112469469 | Mar 2021 | CN |
| 112822992 | May 2021 | CN |
| 4412278 | Oct 1995 | DE |
| 0301790 | Feb 1989 | EP |
| 1345210 | Sep 2003 | EP |
| 2198521 | Jun 2012 | EP |
| 2541763 | Jan 2013 | EP |
| 2959394 | Dec 2015 | EP |
| 3104737 | Dec 2016 | EP |
| 3 200 051 | Aug 2017 | EP |
| 3487395 | May 2019 | EP |
| 2959394 | May 2021 | EP |
| H05-277080 | Oct 1993 | JP |
| 3103427 | Oct 2000 | JP |
| 2002287869 | Oct 2002 | JP |
| 2005-095561 | Apr 2005 | JP |
| 2009050679 | Mar 2009 | JP |
| 2010-520561 | Jun 2010 | JP |
| 2016-507851 | Mar 2016 | JP |
| 2017-509386 | Apr 2017 | JP |
| 2019023941 | Feb 2019 | JP |
| 2021072136 | May 2021 | JP |
| 20120094870 | Aug 2012 | KR |
| 20120097997 | Sep 2012 | KR |
| 2015-0123254 | Nov 2015 | KR |
| 2016-0121552 | Oct 2016 | KR |
| 10-2017-0067873 | Jun 2017 | KR |
| 20170107283 | Sep 2017 | KR |
| 10-1790147 | Oct 2017 | KR |
| 9527341 | Oct 1995 | WO |
| 2006086504 | Aug 2006 | WO |
| WO 2008109248 | Sep 2008 | WO |
| WO 2009042313 | Apr 2009 | WO |
| WO 2010104879 | Sep 2010 | WO |
| 2011070554 | Jun 2011 | WO |
| WO 2012155157 | Nov 2012 | WO |
| WO 2014130871 | Aug 2014 | WO |
| WO 2014186370 | Nov 2014 | WO |
| WO 2014194257 | Dec 2014 | WO |
| WO 2014197443 | Dec 2014 | WO |
| WO 2015027089 | Feb 2015 | WO |
| WO 2015073713 | May 2015 | WO |
| WO 2015081113 | Jun 2015 | WO |
| 2015100172 | Jul 2015 | WO |
| WO 2015123445 | Aug 2015 | WO |
| 2015184760 | Dec 2015 | WO |
| 2015192117 | Dec 2015 | WO |
| WO 2015199747 | Dec 2015 | WO |
| WO 2016041088 | Mar 2016 | WO |
| WO 2017062544 | Apr 2017 | WO |
| 2017075611 | May 2017 | WO |
| WO 2017092225 | Jun 2017 | WO |
| WO 2017120669 | Jul 2017 | WO |
| WO 2017172185 | Oct 2017 | WO |
| WO 2017208167 | Dec 2017 | WO |
| 2018022602 | Feb 2018 | WO |
| 2018098046 | May 2018 | WO |
| 2019099758 | May 2019 | WO |
| 2019147953 | Aug 2019 | WO |
| 2019147958 | Aug 2019 | WO |
| 2019147996 | Aug 2019 | WO |
| 2019217419 | Nov 2019 | WO |
| 2019226259 | Nov 2019 | WO |
| 2019231911 | Dec 2019 | WO |
| 2020061451 | Mar 2020 | WO |
| 2020047429 | Mar 2020 | WO |
| 2020061440 | Mar 2020 | WO |
| 2020072915 | Apr 2020 | WO |
| Entry |
|---|
| Extended European Search Report for European Application No. EP 17835111.0 dated Nov. 21, 2019. |
| Extended European Search Report for European Application No. EP 17835140.9 dated Nov. 26, 2019. |
| International Preliminary Report on Patentability for International Application No. PCT/US2017/043686 dated Feb. 7, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2017/043686 dated Oct. 6, 2017. |
| International Preliminary Report on Patentability for International Application No. PCT/US2017/043693 dated Feb. 7, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2017/043693 dated Oct. 6, 2017. |
| International Preliminary Report on Patentability for International Application No. PCT/US2017/043791 dated Feb. 7, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2017/043791 dated Oct. 5, 2017. |
| International Preliminary Report on Patentability for International Application No. PCT/US2017/043792 dated Feb. 7, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2017/043792 dated Oct. 5, 2017. |
| International Search Report and Written Opinion for International Application No. PCT/US2018/056768 dated Jan. 15, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2018/061409 dated Mar. 12, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2018/063215 dated Mar. 21, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/015134 dated May 15, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/015167 dated May 21, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/015174 dated May 21, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/015238 dated May 16, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/015183 dated May 3, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/015180 dated May 28, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/015244 dated May 16, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/037302 dated Oct. 11, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/028299 dated Aug. 9, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/034173 dated Sep. 18, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/042579 dated Oct. 31, 2019. |
| Invitation to Pay Additional Fees for International Application No. PCT/US2019/031114 dated Aug. 6, 2019. |
| Invitation to Pay Additional Fees for International Application No. PCT/US2019/049094 dated Oct. 24, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/052131 dated Dec. 6, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/046351 dated Nov. 7, 2019. |
| International Search Report and Written Opinion for International Application No. PCT/US19/20065 dated May 16, 2019. |
| Al-Mashhadany, Inverse Kinematics Problem (IKP) of 6-DOF Manipulator Bgy Locally Recurrent Neural Networks (LRNNs). Management and Service Science (MASS). 2010 International Conference ON, IEEE. Aug. 24, 2010. 5 pages. ISBN: 978-1-4244-5325-2. |
| Arkenbout et al., Robust Hand Motion Tracking through Data Fusion of 5DT Data Glove and Nimble VR Kinect Camera Measurements. Sensors. 2015;15:31644-71. |
| Benko et al., Enhancing Input On and Above the Interactive Surface with Muscle Sensing. The ACM International Conference on Interactive Tabletops and Surfaces. ITS '09. 2009:93-100. |
| Boyali et al., Spectral Collaborative Representation based Classification for hand gestures recognition on electromyography signals. Biomedical Signal Processing and Control. 2016;24:11-18. |
| Cheng et al., A Novel Phonology- and Radical-Coded Chinese Sign Language Recognition Framework Using Accelerometer and Surface Electromyography Sensors. Sensors. 2015;15:23303-24. |
| Csapo et al., Evaluation of Human-Myo Gesture Control Capabilities in Continuous Search and Select Operations. 7th IEEE International Conference on Cognitive Infocommunications. 2016;000415-20. |
| Davoodi et al., Development of a Physics-Based Target Shooting Game to Train Amputee Users of Multijoint Upper Limb Prostheses. Presence. Massachusetts Institute of Technology. 2012;21(1):85-95. |
| Delis et al., Development of a Myoelectric Controller Based on Knee Angle Estimation. Biodevices 2009. International Conference on Biomedical Electronics and Devices. Jan. 17, 2009. 7 pages. |
| Diener et al., Direct conversion from facial myoelectric signals to speech using Deep Neural Networks. 2015 International Joint Conference on Neural Networks (IJCNN). Oct. 1, 2015. 7 pages. |
| Ding et al., HMM with improved feature extraction-based feature parameters for identity recognition of gesture command operators by using a sensed Kinect-data stream. Neurocomputing. 2017;262:108-19. |
| Farina et al., Man/machine interface based on the discharge timings of spinal motor neurons after targeted muscle reinnervation. Nature. Biomedical Engineering. 2017;1:1-12. |
| Favorskaya et al., Localization and Recognition of Dynamic Hand Gestures Based on Hierarchy of Manifold Classifiers. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. 2015;XL-5/W6:1-8. |
| Gallina et al., Surface EMG Biofeedback. Surface Electromyography: Physiology, Engineering, and Applications. 2016:485-500. |
| Gopura et al., A Human Forearm and wrist motion assist exoskeleton robot with EMG-based fuzzy-neuro control. Proceedings of the 2nd IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics. Oct. 19-22, 2008. 6 pages. |
| Hauschild et al., A Virtual Reality Environment for Designing and Fitting Neural Prosthetic Limbs. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2007;15(1):9-15. |
| Jiang, Purdue University Graduate School Thesis/Dissertation Acceptance. Graduate School Form 30. Updated Jan. 15, 2015. 24 pages. |
| Kawaguchi et al., Estimation of Finger Joint Angles Based on Electromechanical Sensing of Wrist Shape. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2017;25(9):1409-18. |
| Kim et al., Real-Time Human Pose Estimation and Gesture Recognition from Depth Images Using Superpixels and SVM Classifier. Sensors. 2015;15:12410-27. |
| Kipke et al., Silicon-substrate Intracortical Microelectrode Arrays for Long-Term Recording of Neuronal Spike Activity in Cerebral Cortex. IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2003;11(2):151-155. |
| Koerner, Design and Characterization of the Exo-Skin Haptic Device: A Novel Tendon Actuated Textile Hand Exoskeleton. 2017. 5 pages. |
| Lee et al., Motion and Force Estimation System of Human Fingers. Journal of Institute of Control, Robotics and Systems. 2011;17(10):1014-1020. |
| Li et al., Motor Function Evaluation of Hemiplegic Upper-Extremities Using Data Fusion from Wearable Inertial and Surface EMG Sensors. Sensors. MDPI. 2017;17(582):1-17. |
| Lopes et al., Hand/arm gesture segmentation by motion using IMU and EMG sensing. ScienceDirect. Elsevier. Procedia Manufacturing. 2017;11:107-13. |
| Marcard et al., Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs. Eurographics. 2017;36(2). 12 pages. |
| Martin et al., A Novel Approach of Prosthetic Arm Control using Computer Vision, Biosignals, and Motion Capture. IEEE. 2014. 5 pages. |
| McIntee, A Task Model of Free-Space Movement-Based Gestures. Dissertation. Graduate Faculty of North Carolina State University. Computer Science. 2016. 129 pages. |
| Mendes et al., Sensor Fusion and Smart Sensor in Sports and Biomedical Applications. Sensors. 2016;16(1569):1-31. |
| Mohamed, Homogeneous cognitive based biometrics for static authentication. Dissertation submitted to University of Victoria, Canada. 2010. 149 pages. URL:http://hdl.handle.net/1828/3211 [last accessed Oct. 11, 2019]. |
| Naik et al., Source Separation and Identification issues in bio signals: A solution using Blind source separation. Intech. 2009. 23 pages. |
| Naik et al., Subtle Hand gesture identification for HCI using Temporal Decorrelation Source Separation BSS of surface EMG. Digital Image Computing Techniques and Applications. IEEE Computer Society. 2007;30-7. |
| Negro et al., Multi-channel intramuscular and surface EMG decomposition by convolutive blind source separation. Journal of Neural Engineering. 2016;13:1-17. |
| Saponas et al., Demonstrating the Feasibility of Using Forearm Electromyography for Muscle-Computer Interfaces. CHI 2008 Proceedings. Physiological Sensing for Input. 2008:515-24. |
| Saponas et al., Enabling Always-Available Input with Muscle-Computer Interfaces. UIST '09. 2009:167-76. |
| Saponas et al., Making Muscle-Computer Interfaces More Practical. CHI 2010: Brauns and Brawn. 2010:851-4. |
| Sartori et al., Neural Data-Driven Musculoskeletal Modeling for Personalized Neurorehabilitation Technologies. IEEE Transactions on Biomedical Engineering. 2016;63(5):879-93. |
| Sauras-Perez, et al., A Voice and Pointing Gesture Interaction System for Supporting Human Spontaneous Decisions in Autonomous Cars. Clemson University. All Dissertations. 2017. 174 pages. |
| Shen et al., I am a Smartwatch and I can Track my User's Arm. University of Illinois at Urbana-Champaign. MobiSys' 16. 12 pages. |
| Son et al., Evaluating the utility of two gestural discomfort evaluation methods. PLoS One. 2017. 21 pages. |
| Strbac et al., Microsoft Kinect-Based Artificial Perception System for Control of Functional Electrical Stimulation Assisted Grasping. Hindawi Publishing Corporation. BioMed Research International. 2014. 13 pages. |
| Torres, Myo Gesture Control Armband. PCMag. Https://www.pcmag.com/article2/0,2817,2485462,00.asp 2015. 9 pages. |
| Valero-Cuevas et al., Computational Models for Neuromuscular Function. NIH Public Access Author Manuscript. Jun. 16, 2011. 52 pages. |
| Wittevrongel et al., Spatiotemporal Beamforming: A Transparent and Unified Decoding Approach to Synchronous Visual Brain-Computer Interfacing. Frontiers in Neuroscience. 2017;11:1-12. |
| Wodzinski et al., Sequential Classification of Palm Gestures Based on A* Algorithm and MLP Neural Network for Quadrocopter Control. Metrol. Meas. Syst., 2017;24(2):265-76. |
| Xue et al., Multiple Sensors Based Hand Motion Recognition Using Adaptive Directed Acyclic Graph. Applied Sciences. MDPI. 2017;7(358):1-14. |
| Yang et al., Surface EMG based handgrip force predictions using gene expression programming. Neurocomputing. 2016;207:568-579. |
| Zacharaki et al., Spike pattern recognition by supervised classification in low dimensional embedding space. Brain Informatics. 2016;3:73-8. DOI: 10.1007/s40708-016-0044-4. |
| EP 17835111.0, Nov. 21, 2019, Extended European Search Report. |
| EP 17835140.9, Nov. 26, 2019, Extended European Search Report. |
| PCT/US19/20065, Oct. 6, 2017, International Search Report and Written Opinion. |
| PCT/US2017/043686, Oct. 6, 2017, International Search Report and Written Opinion. |
| PCT/US2017/043686, Feb. 7, 2019, International Preliminary Report on Patentability. |
| PCT/US2017/043693, Oct. 6, 2017, International Search Report and Written Opinion. |
| PCT/US2017/043693, Feb. 7, 2019, International Preliminary Report on Patentability. |
| PCT/US2017/043971, Oct. 5, 2017, International Search Report and Written Opinion. |
| PCT/US2017/043791, Feb. 7, 2019, International Preliminary Report on Patentability. |
| PCT/US2017/043792, Oct. 5, 2017, International Search Report and Written Opinion. |
| PCT/US2017/043792, Feb. 7, 2019, International Preliminary Report on Patentability. |
| PCT/US2018/056768, Jan. 15, 2019, International Search Report and Written Opinion. |
| PCT/US2018/061409, Mar. 12, 2019, International Search Report and Written Opinion. |
| PCT/US2018/063215, Mar. 21, 2019, International Search Report and Written Opinion. |
| PCT/US2019/015134, May 15, 2019, International Search Report and Written Opinion. |
| PCT/US2019/015167, May 21, 2019, International Search Report and Written Opinion. |
| PCT/US2019/015174, May 21, 2019, International Search Report and Written Opinion. |
| PCT/US2019/015180, May 28, 2019, International Search Report and Written Opinion. |
| PCT/US2019/015183, May 3, 2019, International Search Report and Written Opinion. |
| PCT/US2019/015238, May 16, 2019, International Search Report and Written Opinion. |
| PCT/US2019/015244, May 16, 2019, International Search Report and Written Opinion. |
| PCT/US2019/028299, Aug. 9, 2019, International Search Report and Written Opinion. |
| PCT/US2019/031114, Aug. 6, 2019, Invitation to Pay Additional Fees. |
| PCT/US2019/034173, Sep. 18, 2019, International Search Report and Written Opinion. |
| PCT/US2019/037302, Oct. 11, 2019, International Search Report and Written Opinion. |
| PCT/US2019/042579, Oct. 31, 2019, International Search Report and Written Opinion. |
| PCT/US2019/046351, Nov. 7, 2019, International Search Report and Written Opinion. |
| PCT/US2019/049094, Oct. 24, 2019, Invitation to Pay Additional Fees. |
| PCT/US2019/052131, Dec. 6, 2019, International Search Report and Written Opinion. |
| International Search Report and Written Opinion received for PCT Application Serial No. PCT/US2019/052151 dated Jan. 15, 2020, 10 pages. |
| Al-Jumaily A., et al., “Electromyogram(EMG) Driven System based Virtual Reality for Prosthetic and Rehabilitation Devices,” Proceedings of the 11th InternationalConference on Information Integration AndWeb-Based Applications & Services, Jan. 1, 2009, pp. 582-586. |
| European Search Report for European Application No. 19861903.3, dated Oct. 12, 2021, 2 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/052151, dated Apr. 1, 2021, 9 pages. |
| Al-Timemy A.H., et al., “Improving the Performance Against Force Variation of EMG Controlled Multifunctional Upper-Limb Prostheses for Transradial Amputees,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, Jun. 2016, vol. 24 (6), 12 Pages. |
| Berenzweig A., et al., “Wearable Devices and Methods for Improved Speech Recognition,” U.S. Appl. No. 16/785,680, filed Feb. 10, 2020, 67 pages. |
| Brownlee J., “Finite State Machines (FSM): Finite State Machines as a Control Technique in Artificial Intelligence (Al),” FSM, Jun. 2002, 12 pages. |
| Cannan J., et al., “A Wearable Sensor Fusion Armband for Simple Motion Control and Selection for Disabled and Non-Disabled Users,” Computer Science and Electronic Engineering Conference, IEEE, Sep. 12, 2012, pp. 216-219, XP032276745. |
| Communication Pursuant to Article 94(3) for European Patent Application No. 17835112.8, dated Dec. 14, 2020, 3 Pages. |
| Communication Pursuant to Rule 164(1) EPC, Partial Supplementary European Search Report for European Application No. 14753949.8, dated Sep. 30, 2016, 7 pages. |
| Co-pending U.S. Appl. No. 15/659,072, inventors Patrick; Kaifosh et al., filed Jul. 25, 2017. |
| Co-pending U.S. Appl. No. 15/816,435, inventors Ning; Guo et al., filed Nov. 17, 2017. |
| Co-pending U.S. Appl. No. 15/882,858, inventors Siephen; Lake et al., filed Jan. 29, 2018. |
| Co-pending U.S. Appl. No. 15/974,430, inventors Adam; Berenzweig et al., filed May 8, 2018. |
| Co-pending U.S. Appl. No. 16/353,998, inventors Patrick; Kaifosh et al., filed Mar. 14, 2019. |
| Co-pending U.S. Appl. No. 16/557,383, inventors Adam; Berenzweig et al., filed Aug. 30, 2019. |
| Co-Pending U.S. Appl. No. 16/557,427, inventors Adam; Berenzweig et al., filed Aug. 30, 2019. |
| Co-Pending U.S. Appl. No. 15/974,430, filed May 8, 2018, 44 Pages. |
| Co-Pending U.S. Appl. No. 16/353,998, filed Mar. 14, 2019, 43 pages. |
| Co-Pending U.S. Appl. No. 16/557,383, filed Aug. 30, 2019, 94 Pages. |
| Co-Pending U.S. Appl. No. 16/557,427, filed Aug. 30, 2019, 93 Pages. |
| Co-Pending U.S. Appl. No. 16/577,207, filed Sep. 20, 2019, 67 Pages. |
| Co-Pending U.S. Appl. No. 14/505,836, filed Oct. 3, 2014, 59 Pages. |
| Co-Pending U.S. Appl. No. 15/816,435, filed Nov. 17, 2017, 24 Pages. |
| Co-Pending U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 54 Pages. |
| Co-Pending U.S. Appl. No. 15/974,384, filed May 8, 2018, 44 Pages. |
| Co-Pending U.S. Appl. No. 15/974,454, filed May 8, 2018, 45 Pages. |
| Co-Pending U.S. Appl. No. 16/557,342, filed Aug. 30, 2019, 93 Pages. |
| Corazza S., et al.,“ A Markerless Motion Capture System to Study Musculoskeletal Biomechanics: Visual Hull and Simulated Annealing Approach,” Annals of Biomedical Engineering, Jul. 2006, vol. 34 (6), pp. 1019-1029, [Retrieved an Dec. 11, 2019], 11 pages, Retrieved from the Internet: URL: https://www.researchgate.net/publication/6999610_A_Markerless_Motion_Capture_System_to_Study_Musculoskeletal_Biomechanics_Visual_Hull_and_Simulated_Annealing_Approach. |
| Costanza E., et al., “EMG as a Subtle Input Interface for Mobile Computing,” Mobile HCI, LNCS 3160, 2004, pp. 126-130. |
| Costanza E., et al., “Toward Subtle Intimate Interfaces for Mobile Devices Using an EMG Controller,” CHI, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 2-7, 2005, pp. 481-489. |
| Cote-Allard U., et al., “Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, Jan. 26, 2019, vol. 27 (4), 11 Pages. |
| European Search Report for European Application No. 19863248.1, dated Oct. 19, 2021, 2 pages. |
| European Search Report for European Application No. 19868789.9, dated May 9, 2022, 9 pages. |
| Extended European Search Report for European Application No. 18879156.0, dated Mar. 12, 2021, 11 pages. |
| Extended European Search Report for European Application No. 19744404.5, dated Mar. 29, 2021, 11 pages. |
| Extended European Search Report for European Application No. 19799947.7, dated May 26, 2021, 10 pages. |
| Extended European Search Report for European Application No. 17835112.8, dated Feb. 5, 2020, 17 pages. |
| Extended European Search Report for European Application No. 19806723.3, dated Jul. 7, 2021, 13 pages. |
| Extended European Search Report for European Application No. 19850130.6, dated Sep. 1, 2021, 14 Pages. |
| Extended European Search Report for European Application No. 19855191.3, dated Dec. 6, 2021, 11 pages. |
| Extended European Search Report for European Application No. 19883839.3, dated Dec. 15, 2021, 7 pages. |
| Final Office Action dated Jun. 2, 2020 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 127 Pages. |
| Final Office Action dated Jun. 2, 2020 for U.S. Appl. No. 16/557,383, filed Aug. 30, 2019, 66 Pages. |
| Final Office Action dated Nov. 3, 2020 for U.S. Appl. No. 15/974,430, filed May 8, 2018, 27 Pages. |
| Final Office Action dated Feb. 4, 2020 for U.S. Appl. No. 16/577,207, filed Sep. 20, 2019, 76 Pages. |
| Final Office Action dated Feb. 4, 2021 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 42 Pages. |
| Final Office Action dated Jun. 5, 2020 for U.S. Appl. No. 16/557,427, filed Aug. 30, 2019, 95 Pages. |
| Final Office Action dated Oct. 8, 2020 for U.S. Appl. No. 16/557,342, filed Aug. 30, 2019, 73 Pages. |
| Final Office Action dated Apr. 9, 2020 for U.S. Appl. No. 15/974,454, filed May 8, 2018, 19 Pages. |
| Final Office Action dated Dec. 11, 2019 for U.S. Appl. No. 15/974,430, filed May 8, 2018, 30 Pages. |
| Final Office Action dated Jan. 13, 2021 for U.S. Appl. No. 16/577,207, filed Sep. 20, 2019, 91 Pages. |
| Final Office Action dated Dec. 18, 2019 for U.S. Appl. No. 16/258,279, filed Jan. 25, 2019, 45 Pages. |
| Final Office Action dated Feb. 19, 2021 for U.S. Appl. No. 16/258,279, filed Jan. 25, 2019, 58 Pages. |
| Final Office Action dated Sep. 23, 2020 for U.S. Appl. No. 15/816,435, filed Nov. 17, 2017, 70 Pages. |
| Final Office Action dated Jan. 28, 2020 for U.S. Appl. No. 16/557,342, filed Aug. 30, 2019, 15 Pages. |
| Final Office Action dated Jul. 28, 2017 for U.S. Appl. No. 14/505,836, filed Oct. 3, 2014, 52 Pages. |
| Final Office Action dated Jun. 28, 2021 for U.S. Appl. No. 16/557,342, filed Aug. 30, 2019, 11 Pages. |
| Final Office Action dated Nov. 29, 2019 for U.S. Appl. No. 15/659,072, filed Jul. 25, 2017, 36 Pages. |
| Final Office Action dated Nov. 29, 2019 for U.S. Appl. No. 16/353,998, filed Mar. 14, 2019, 33 Pages. |
| Fong H.C., et al., “PepperGram With Interactive Control,” 22nd International Conference Onvirtual System & Multimedia (VSMM), Oct. 17, 2016, 5 pages. |
| Ghasemzadeh H., et al., “A Body Sensor Network With Electromyogram and Inertial Sensors: Multimodal Interpretation of Muscular Activities,” IEEE Transactions on Information Technology in Biomedicine, Mar. 2010, vol. 14 (2), pp. 198-206. |
| Gourmelon L., et al., “Contactless Sensors for Surface Electromyography,” Proceedings of the 28th IEEE EMBS Annual International Conference, New York City, NY, Aug. 30-Sep. 3, 2006, pp. 2514-2517. |
| International Search Report and Written Opinion for International Application No. PCT/US2014/017799, dated May 16, 2014, 9 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2014/037863, dated Aug. 21, 2014, 10 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/031114, dated Nov. 19, 2020, 16 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/049094, dated Mar. 11, 2021, 24 Pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2014/017799, dated Sep. 3, 2015, 8 Pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2014/037863, dated Nov. 26, 2015, 8 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2014/052143, dated Mar. 3, 2016, 7 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2014/067443, dated Jun. 9, 2016, 7 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2015/015675, dated Aug. 25, 2016, 8 Pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2018/061409, dated May 28, 2020, 10 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/015174, dated Aug. 6, 2020, 7 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/028299, dated Dec. 10, 2020, 11 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/046351, dated Feb. 25, 2021, 8 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/052131, dated Apr. 1, 2021, 8 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/054716, dated Apr. 15, 2021, 10 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/061759, dated May 27, 2021, 12 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2020/049274, dated Mar. 17, 2022, 14 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2014/052143, dated Nov. 21, 2014, 8 Pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2014/067443, dated Feb. 27, 2015, 8 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2015/015675, dated May 27, 2015, 9 Pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/031114, dated Dec. 20, 2019, 18 Pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/049094, dated Jan. 9, 2020, 27 Pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/054716, dated Dec. 20, 2019, 11 Pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2019/061759, dated Jan. 29, 2020, 12 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2020/025735, dated Jun. 22, 2020, 10 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2020/025772, dated Aug. 3, 2020, 11 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2020/025797, dated Jul. 9, 2020, 10 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2020/049274, dated Feb. 1, 2021, 17 pages. |
| International Search Report and Written Opinion for International Application No. PCT/US2020/061392, dated Mar. 12, 2021, 12 pages. |
| Kainz et al., “Approach to Hand Tracking and Gesture Recognition Based on Depth-Sensing Cameras and EMG Monitoring,” Acta Informatica Pragensia, vol. 3, Jan. 1, 2014, pp. 104-112, Retrieved from the Internet URL: https://aip.vse.cz/pdfs/aip/2014/01/08.pdf. |
| Morris D., et al., “Emerging Input Technologies for Always-Available Mobile Interaction,” Foundations and Trends in Human-Computer Interaction, 2010, vol. 4 (4), pp. 245-316. |
| Naik G.R., et al., “Real-Time Hand Gesture Identification for Human Computer Interaction Based on ICA of Surface Electromyogram,” IADIS International Conference Interfaces and Human Computer Interaction, 2007, pp. 83-90. |
| Non-Final Office Action dated Mar. 2, 2021 for U.S. Appl. No. 15/974,430, filed May 8, 2018, 32 Pages. |
| Non-Final Office Action dated Sep. 2, 2020 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 66 Pages. |
| Non-Final Office Action dated Aug. 3, 2020 for U.S. Appl. No. 16/593,446, filed Oct. 4, 2019, 44 pages. |
| Non-Final Office Action dated Jun. 3, 2021 for U.S. Appl. No. 15/816,435, filed Nov. 17, 2017, 32 Pages. |
| Non-Final Office Action dated Jun. 5, 2020 for U.S. Appl. No. 15/659,072, filed Jul. 25, 2017, 59 Pages. |
| Non-Final Office Action dated Feb. 8, 2021 for U.S. Appl. No. 16/557,342, filed Aug. 30, 2019, 11 Pages. |
| Non-Final Office Action dated Oct. 8, 2020 for U.S. Appl. No. 16/577,207, filed Sep. 20, 2019, 51 Pages. |
| Non-Final Office Action dated Aug. 11, 2021 for U.S. Appl. No. 16/577,207, filed Sep. 20, 2019, 35 Pages. |
| Non-Final Office Action dated Jun. 13, 2019 for U.S. Appl. No. 16/258,279, filed Jan. 25, 2019, 38 Pages. |
| Non-Final Office Action dated Jun. 15, 2020 for U.S. Appl. No. 16/557,342, filed Aug. 30, 2019, 46 Pages. |
| Non-Final Office Action dated Jan. 16, 2020 for U.S. Appl. No. 16/389,419, filed Apr. 19, 2019, 26 Pages. |
| Non-Final Office Action dated May 16, 2019 for U.S. Appl. No. 15/974,384, filed May 8, 2018, 13 Pages. |
| Non-Final Office Action dated May 16, 2019 for U.S. Appl. No. 15/974,430, filed May 8, 2018, 12 Pages. |
| Non-Final Office Action dated Nov. 19, 2019 for U.S. Appl. No. 16/577,207, filed Sep. 20, 2019, 32 Pages. |
| Non-Final Office Action dated Aug. 20, 2020 for U.S. Appl. No. 15/974,454, filed May 8, 2018, 59 Pages. |
| Non-Final Office Action dated Dec. 20, 2019 for U.S. Appl. No. 15/974,454, filed May 8, 2018, 41 Pages. |
| Non-Final Office Action dated Jan. 22, 2020 for U.S. Appl. No. 15/816,435, filed Nov. 17, 2017, 35 Pages. |
| Non-Final Office Action dated Oct. 22, 2019 for U.S. Appl. No. 16/557,342, filed Aug. 30, 2019, 16 Pages. |
| Non-Final Office Action dated Dec. 23, 2019 for U.S. Appl. No. 16/557,383, filed Aug. 30, 2019, 53 Pages. |
| Non-Final Office Action dated Dec. 23, 2019 for U.S. Appl. No. 16/557,427, filed Aug. 30, 2019, 52 Pages. |
| Non-Final Office Action dated Feb. 23, 2017 for U.S. Appl. No. 14/505,836, filed Oct. 3, 2014, 54 Pages. |
| Non-Final Office Action dated Jul. 23, 2020 for U.S. Appl. No. 16/593,446, filed Oct. 4, 2019, 28 pages. |
| Non-Final Office Action dated May 24, 2019 for U.S. Appl. No. 16/353,998, filed Mar. 14, 2019, 20 Pages. |
| Non-Final Office Action dated May 26, 2020 for U.S. Appl. No. 16/353,998, filed Mar. 14, 2019, 60 Pages. |
| Non-Final Office Action dated Nov. 27, 2020 for U.S. Appl. No. 16/258,279, filed Jan. 25, 2019, 44 Pages. |
| Non-Final Office Action dated Apr. 30, 2019 for U.S. Appl. No. 15/659,072, filed Jul. 25, 2017, 99 Pages. |
| Non-Final Office Action dated Apr. 30, 2020 for U.S. Appl. No. 15/974,430, filed May 8, 2018, 57 Pages. |
| Non-Final Office Action dated Dec. 30, 2019 for U.S. Appl. No. 16/593,446, filed Oct. 4, 2019, 43 pages. |
| Non-Final Office Action dated Jun. 30, 2016 for U.S. Appl. No. 14/505,836, filed Oct. 3, 2014, 37 Pages. |
| Non-Final Office Action dated Oct. 30, 2019 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 22 Pages. |
| Notice of Allowance dated Nov. 2, 2020 for U.S. Appl. No. 15/974,454, filed May 8, 2018, 24 Pages. |
| Notice of Allowance dated Nov. 4, 2019 for U.S. Appl. No. 15/974,384, filed May 8, 2018, 39 Pages. |
| Notice of Allowance dated Feb. 9, 2022 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 9 pages. |
| Notice of Allowance dated Nov. 10, 2021 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 6 pages. |
| Notice of Allowance dated Jul. 15, 2021 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 2 pages. |
| Notice of Allowance dated Dec. 16, 2020 for U.S. Appl. No. 16/593,446, filed Oct. 4, 2019, 44 pages. |
| Notice of Allowance dated May 18, 2020 for U.S. Appl. No. 16/258,279, filed Jan. 25, 2019, 42 Pages. |
| Notice of Allowance dated Aug. 19, 2020 for U.S. Appl. No. 16/557,427, filed Aug. 30, 2019, 22 Pages. |
| Notice of Allowance dated May 20, 2020 for U.S. Appl. No. 16/389,419, filed Apr. 19, 2019, 28 Pages. |
| Notice of Allowance dated Oct. 22, 2021 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 8 pages. |
| Notice of Allowance dated Aug. 23, 2021 for U.S. Appl. No. 15/974,430, filed May 8, 2018, 12 pages. |
| Notice of Allowance dated Dec. 23, 2020 for U.S. Appl. No. 15/659,072, filed Jul. 25, 2017, 26 Pages. |
| Notice of Allowance dated Jun. 28, 2021 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 18 pages. |
| Office action for European Application No. 17835112.8, dated Feb. 11, 2022, 11 Pages. |
| Partial Supplementary European Search Report for European Application No. 18879156.0, dated Dec. 7, 2020, 9 pages. |
| Picard R.W., et al., “Affective Wearables,” Proceedings of the IEEE 1st International Symposium on Wearable Computers, ISWC, Cambridge, Ma, USA, Oct. 13-14, 1997, pp. 90-97. |
| Preinterview First Office Action dated Jun. 24, 2020 for U.S. Appl. No. 16/785,680, filed Feb. 10, 2020, 90 Pages. |
| Rekimoto J., “GestureWrist and GesturePad: Unobtrusive Wearable Interaction Devices,” ISWC Proceedings of the 5th IEEE International Symposium on Wearable Computers, 2001, 7 pages. |
| Sato M., et al., “Touche: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects,” CHI, Austin, Texas, May 5-10, 2012, 10 pages. |
| Ueno A., et al., “A Capacitive Sensor System for Measuring Laplacian Electromyogram through Cloth: A Pilot Study,” Proceedings of the 29th Annual International Conference of the IEEE EMBS, Cite Internationale, Lyon, France, Aug. 23-26, 2007, pp. 5731-5734. |
| Ueno A., et al., “Feasibility of Capacitive Sensing of Surface Electromyographic Potential through Cloth,” Sensors and Materials, 2012, vol. 24 (6), pp. 335-346. |
| Xiong A., et al., “A Novel HCI based on EMG and IMU,” Proceedings of the 2011 IEEE International Conference on Robotics and Biomimetics, Phuket, Thailand, Dec. 7-11, 2011, pp. 2653-2657. |
| Xu Z., et al., “Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors,” Proceedings of the 14th International Conference on Intelligent User Interfaces, D211 Sanibel Island, Florida, Feb. 8-11, 2009, pp. 401-406. |
| Zhang X., et al., “A Framework for Hand Gesture Recognition Based on Accelerometer and EMG Sensors,” IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, Nov. 2011, vol. 41 (6), pp. 1064-1076. |
| European Search Report for European Application No. 19890394.0, dated Apr. 29, 2022, 9 pages. |
| Extended European Search Report for European Application No. 19743717.1, dated Mar. 3, 2021, 12 pages. |
| Extended European Search Report for European Application No. 18869441.8, dated Nov. 17, 2020, 20 pages. |
| Extended European Search Report for European Application No. 19810524.9, dated Mar. 17, 2021, 11 pages. |
| Gargiulo G., et al., “Giga-Ohm High-Impedance FET Input Amplifiers for Dry Electrode Biosensor Circuits and Systems,” Integrated Microsystems: Electronics, Photonics, and Biotechnolgy, Dec. 19, 2017, 41 Pages, Retrieved from the Internet: URL: https://www.researchgate.net/profile/Aiistair_Mcewen/publication/255994293_Gigaohm_high_impedance_FETinput_amplifiers_for_dry_electrode_biosensor_circuits_and_systems. |
| International Preliminary Report on Patentability for International Application No. PCT/US2018/056768, dated Apr. 30, 2020, 7 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/015183, dated Aug. 6, 2020, 7 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/015238, dated Aug. 6, 2020, 7 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/034173, dated Dec. 10, 2020, 9 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/063587, dated Jun. 10, 2021, 13 pages. |
| International Preliminary Report on Patentability for International Application No. PCT/US2019/063587, dated Mar. 25, 2020, 16 pages. |
| Non-Final Office Action dated Sep. 6, 2019 for U.S. Appl. No. 16/424,144, filed May 28, 2019, 11 Pages. |
| Non-Final Office Action dated Apr. 9, 2019 for U.S. Appl. No. 16/258,409, filed Jan. 25, 2019, 71 Pages. |
| Non-Final Office Action dated Apr. 29, 2019 for U.S. Appl. No. 16/257,979, filed Jan. 25, 2019, 63 Pages. |
| Notice of Allowance dated Feb. 6, 2020 for U.S. Appl. No. 16/424,144, filed May 28, 2019, 28 Pages. |
| Notice of Allowance dated Jul. 19, 2019 for U.S. Appl. No. 16/258,409, filed Jan. 25, 2019, 36 Pages. |
| Notice of Allowance dated Jul. 31, 2019 for U.S. Appl. No. 16/257,979, filed Jan. 25, 2019, 22 Pages. |
| Office Action for European Patent Application No. 19743717.1, dated Apr. 11, 2022, 10 pages. |
| Notice of Allowance dated May 18, 2022 for U.S. Appl. No. 15/882,858, filed Jan. 29, 2018, 10 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20200097082 A1 | Mar 2020 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62734138 | Sep 2018 | US |
