Real-time processing of handstate representation model estimates

Information

  • Patent Grant
  • 11587242
  • Patent Number
    11,587,242
  • Date Filed
    Friday, August 20, 2021
    3 years ago
  • Date Issued
    Tuesday, February 21, 2023
    a year ago
Abstract
System and methods are provided for providing a dynamically-updated musculoskeletal representation of a hand. The system includes a plurality of neuromuscular sensors configured to continuously record a plurality of neuromuscular signals from a user, and at least one computer processor programmed to provide as input to a trained statistical model, the plurality of neuromuscular signals and temporally smooth in real-time an output of the trained statistical model. The system is also programmed to determine, based on the smoothed output of the trained statistical model, position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation, force information describing a force exerted by at least one segment of the musculoskeletal representation, and update the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.
Description
BACKGROUND

In some computer applications that generate musculoskeletal representations of the human body, it is desirable for the application to know the spatial positioning, orientation and movement of a user's body to provide a realistic representation of body movement. For example, in a virtual reality (VR) environment, tracking the spatial position of the user's hand enables the application to represent hand motion in the VR environment, which allows the user to interact with (e.g., by grasping or manipulating) virtual objects within the VR environment. Some existing techniques for tracking movements of a user's body using wearable sensors include using information obtained from multiple Inertial Measurement Units (IMUs) affixed to different parts of the user's body, and using external imaging devices (e.g., fixed-position cameras) to reconstruct the position and orientation of parts of the user's body.


SUMMARY

In some computer applications that generate musculoskeletal representations of the human body, it is appreciated that it is desirable for the application to provide a more realistic representation of body position, movement, and force. In an example in the VR environment, tracking the spatial position of the user's hand enables virtually rendering a hand, and rendering that realistically approximates natural kinematics and gestures may enhance immersion for the user in the virtual environment. Although some camera-based systems attempt to track movement of a user's body, it is appreciated that such interpretations may be improved by measuring and modeling human anatomy using wearable neuromuscular sensors.


Some embodiments are directed to predicting information about the positioning and movements of portions of a user's arm and/or hand represented as a multi-segment articulated rigid body system with joints connecting the multiple segments of the rigid body system. Signals recorded by wearable neuromuscular sensors placed at locations on the user's body are provided as input to a statistical model trained to predict estimates of the position (e.g., absolute position, relative position, orientation) and forces associated with a plurality of rigid segments in a computer-based musculoskeletal representation associated with a hand when a user performs one or more movements. The combination of position information and force information associated with segments of a musculoskeletal representation associated with a hand is colloquially referred to herein as a “handstate” of the musculoskeletal representation. As a user performs different movements, a trained statistical model interprets neuromuscular signals recorded by the wearable neuromuscular sensors into position and force estimates (handstate information) that are used to update the musculoskeletal representation. As the neuromuscular signals are continuously recorded, the musculoskeletal representation is updated in real time (or near real-time) and a visual representation of a hand (e.g., within a virtual reality environment) is optionally rendered based on the current handstate estimates.


In some aspects, a computerized system for providing a dynamically-updated musculoskeletal representation of a hand is provided. In some embodiment, the system comprises a plurality of neuromuscular sensors configured to continuously record a plurality of neuromuscular signals from a user, and at least one computer processor programmed to provide as input to a trained statistical model, the plurality of neuromuscular signals, temporally smoothing in real-time an output of the trained statistical model, determine, based on the smoothed output of the trained statistical model, position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation, force information describing a force exerted by at least one segment of the musculoskeletal representation, and update the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.


According to some embodiments, temporally smoothing the output of the trained statistical model comprises processing the output of the trained statistical model using at least one filter. According to some embodiments, the at least one filter comprises at least one filter from a group comprising an exponential filter, a Kalman filter, a non-linear Kalman filter, a particle filter and a Bayesian filter. According to some embodiments, the at least one computer processor is programmed to determine an accuracy of the trained statistical model, and temporally smooth, responsive to the determined accuracy of the trained statistical model, the output of the trained statistical model in real-time.


According to some embodiments, the at least one computer processor is programmed to determine at least one of an input accuracy and an output accuracy of the trained model.


According to some embodiments, temporally smoothing the output of the trained statistical model comprises including in the trained statistical model at least one temporal smoothing characteristic. According to some embodiments, the trained statistical model includes at least one temporal smoothing characteristic that comprises training the statistical model with a penalization term to promote smoothness in the model outputs.


According to some embodiments, the trained statistical model including at least one temporal smoothing characteristic comprises training the statistical model with an error function that includes an error in estimates of time derivatives of the output of the statistical model. According to some embodiments, the trained statistical model including at least one temporal smoothing characteristic comprises including in the trained statistical model a prior on the smoothness of the output of the statistical model.


According to some embodiments, temporally smoothing the output of the trained statistical model further comprises generating the output of the trained statistical model for a first timepoint based on first neuromuscular signals recorded at the first timepoint and second neuromuscular signals recorded at least one second timepoint after the first timepoint. According to some embodiments, the at least one computer processor is further programmed to determine a quality of the smoothed output of the trained statistical model at a first timepoint, adjust at least one smoothing parameter based on the determined quality, and temporally smooth output of the trained statistical model at a second timepoint based on the at least one adjusted smoothing parameter.


According to some embodiments, the at least one computer processor is further programmed to temporally smooth the output of the trained statistical model based, at least in part, on at least one characteristic of a motion of the user during recording of the plurality of neuromuscular signals. According to some embodiments, the system further comprises at least one inertial measurement unit (IMU) configured to continuously record a plurality of inertial measurement unit signals, wherein the at least one characteristic of a motion of the user is determined based, at least in part, on the plurality of inertial measurement unit signals.


According to some embodiments, the at least one characteristic of a motion of the user comprises a speed at which a user is moving a hand and/or arm of the user.


According to some embodiments, temporally smoothing the output of the trained statistical model comprises increasing an amount of temporal smoothing of the output of the trained statistical model when it is determined that the user is moving a hand and/or arm of the user at a speed greater than a threshold value. According to some embodiments, the at least one computer processor is further programmed to update a loss function of the trained statistical model based, at least in part, on the smoothed output of the trained statistical model.


According to some embodiments, the at least one computer processor is further programmed to execute an application that provides a virtual reality environment, render, based on the updated musculoskeletal model of a part of the user's body (e.g. the hand), a visual representation of a hand within the virtual reality environment, and modulate an amount of temporal smoothing of the output of the trained statistical model based, at least in part, on a spatial relationship between the visual representation of the hand and one or more objects within the virtual reality environment. According to some embodiments, modulating an amount of temporal smoothing of the output of the trained statistical model comprises at least one of decreasing and increasing the amount of temporal smoothing when the visual representation of the hand approaches an object in the virtual reality environment. According to some embodiments, the amount of temporal smoothing is either increased or decreased in a selected degree of freedom.


According to other aspects, a method is provided for dynamically-updating a musculoskeletal representation of a hand. In some embodiments, the method comprises receiving, from a plurality of neuromuscular sensors, a plurality of neuromuscular signals from a user, executing a trained statistical model representing handstate, providing, as input to the executed statistical model, the plurality of neuromuscular signals, temporally smoothing in real-time an output of the trained statistical model, determining, based on the smoothed output of the trained statistical model position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation, and force information describing a force exerted by at least one segment of the musculoskeletal representation, and updating the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.


According to other aspects, a computer-readable medium encoded with a plurality of instructions that, when executed by at least one computer processor, performs a method for providing a dynamically-updated musculoskeletal representation of a hand. In some embodiments, the method comprises receiving, from a plurality of neuromuscular sensors, a plurality of neuromuscular signals from a user, executing a trained statistical model representing handstate, providing, as input to the executed statistical model, the plurality of neuromuscular signals, temporally smoothing in real-time an output of the trained statistical model, determining, based on the smoothed output of the trained statistical model position information describing a spatial relationship between two or more connected segments of the musculoskeletal representation, and force information describing a force exerted by at least one segment of the musculoskeletal representation, and updating the musculoskeletal representation of the hand based, at least in part, on the position information and the force information.


It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein.





BRIEF DESCRIPTION OF DRAWINGS

Various non-limiting embodiments of the technology will be described with reference to the following figures. It should be appreciated that the figures are not necessarily drawn to scale.



FIG. 1 is a schematic diagram of a computer-based system for generating a musculoskeletal representation based on neuromuscular sensor data in accordance with some embodiments of the technology described herein;



FIG. 2 is a flowchart illustrating a computer-based process for processing position and force information according to some embodiments described herein;



FIG. 3 is a flowchart illustrating a computer-based process for smoothing an output of a statistical model in real-time according to some embodiments described herein;



FIG. 4 is a flowchart illustrating a computer-based process for temporally smoothing an output in real-time according to some embodiments described herein;



FIG. 5 is a flowchart illustrating a computer-based process for including a smoothing characteristic in a trained statistical model according to some embodiments described herein;



FIG. 6 is a flowchart of an illustrative process for generating a statistical model for predicting musculoskeletal position information using signals recorded from sensors, in accordance with some embodiments of the technology described herein;



FIG. 7A illustrates a wearable system with sixteen EMG sensors arranged circumferentially around an elastic band configured to be worn around a user's lower arm or wrist, in accordance with some embodiments of the technology described herein;



FIG. 7B is a cross-sectional view through one of the sixteen EMG sensors illustrated in FIG. 7A; and



FIGS. 8A and 8B schematically illustrate components of a computer-based system on which some embodiments are implemented. FIG. 8A illustrates a wearable portion of the computer-based system and FIG. 8B illustrates a dongle portion connected to a computer, wherein the dongle portion is configured to communicate with the wearable portion.





DETAILED DESCRIPTION

Although there are camera-based systems used to track human movement, they are lacking for realistically tracking (and, optionally, rendering) a user's body part (e.g. hand), at least because temporal smoothing in such systems do not account for realistic physics, human anatomy (including joint kinematics), and stereotyped and common gestures. Some embodiments described herein address limitations of existing systems for tracking and, optionally, rendering a part of a user's body (e.g. a hand) in order to more realistically render the position and/or movement of a part of a user's body. More realistic rendering of a part of a user's body such as a hand may enhance immersion in virtual environments.


All or portions of the human musculoskeletal system can be modeled as a multi-segment articulated rigid body system, with joints forming the interfaces between the different segments and joint angles defining the spatial relationships between connected segments in the model. Constraints on the movement at the joints are governed by the type of joint connecting the segments and the biological structures (e.g., muscles, tendons, ligaments) that restrict the range of movement at the joint. For example, the shoulder joint connecting the upper arm to the torso and the hip joint connecting the upper leg to the torso are ball and socket joints that permit extension and flexion movements as well as rotational movements. By contrast, the elbow joint connecting the upper arm and the forearm and the knee joint connecting the upper leg and the lower leg allow for a more limited range of motion. As described herein, a multi-segment articulated rigid body system is used to model portions of the human musculoskeletal system. However, it should be appreciated that some segments of the human musculoskeletal system (e.g., the forearm), though approximated as a rigid body in the articulated rigid body system, may include multiple rigid structures (e.g., the ulna and radius bones of the forearm) that provide for more complex movement within the segment that is not explicitly considered by the rigid body model. Accordingly, a model of an articulated rigid body system for use with some embodiments of the technology described herein may include segments that represent a combination of body parts that are not strictly rigid bodies.


In kinematics, rigid bodies are objects that exhibit various attributes of motion (e.g., position, orientation, angular velocity, acceleration). Knowing the motion attributes of one segment of the rigid body enables the motion attributes for other segments of the rigid body to be determined based on constraints in how the segments are connected. For example, the hand may be modeled as a multi-segment articulated body with the joints in the wrist and each finger forming the interfaces between the multiple segments in the model. In some embodiments, movements of the segments in the rigid body model can be simulated as an articulated rigid body system in which position (e.g., actual position, relative position, or orientation) information of a segment relative to other segments in the model are predicted using a trained statistical model, as described in more detail below.


The portion of the human body approximated by a musculoskeletal representation as described herein as one non-limiting example, is a hand or a combination of a hand with one or more arm segments and the information used to describe a current state of the positional relationships between segments and force relationships for individual segments or combinations of segments in the musculoskeletal representation is referred to herein as the handstate of the musculoskeletal representation. It should be appreciated, however, that the techniques described herein are also applicable to musculoskeletal representations of portions of the body other than the hand including, but not limited to, an arm, a leg, a foot, a torso, a neck, or any combination of the foregoing.


In addition to spatial (e.g., position/orientation) information, some embodiments are configured to predict force information associated with one or more segments of the musculoskeletal representation. For example, linear forces or rotational (torque) forces exerted by one or more segments may be estimated. Examples of linear forces include, but are not limited to, the force of a finger or hand pressing on a solid object such as a table, and a force exerted when two segments (e.g., two fingers) are pinched together. Examples of rotational forces include, but are not limited to, rotational forces created when segments in the wrist or fingers are twisted or flexed. In some embodiments, the force information determined as a portion of a current handstate estimate includes one or more of pinching force information, grasping force information, or information about co-contraction forces between muscles represented by the musculoskeletal representation.



FIG. 1 illustrates a system 100 in accordance with some embodiments. The system includes a plurality of sensors 102 configured to record signals resulting from the movement of portions of a human body. Sensors 102 may include autonomous sensors. As used herein, the term “autonomous sensors” refers to sensors configured to measure the movement of body segments without requiring the use of external devices. Examples of external devices used in non-autonomous sensors include, but are not limited to, wearable (e.g. body-mounted) cameras, global positioning systems, or laser scanning systems. In some embodiments, sensors 102 may also include non-autonomous sensors in combination with autonomous sensors. As used herein, the term “non-autonomous sensors” refers to sensors configured to measure the movement of body segments using external devices.


Autonomous sensors may include a plurality of neuromuscular sensors configured to record signals arising from neuromuscular activity in skeletal muscle of a human body. The term “neuromuscular activity” as used herein refers to neural activation of spinal motor neurons that innervate a muscle, muscle activation, muscle contraction, or any combination of the neural activation, muscle activation, and muscle contraction. Neuromuscular sensors may include one or more electromyography (EMG) sensors, one or more mechanomyography (MMG) sensors, one or more sonomyography (SMG) sensors, a combination of two or more types of EMG sensors, MMG sensors, and SMG sensors, and/or one or more sensors of any suitable type that are configured to detect neuromuscular signals. In some embodiments, the plurality of neuromuscular sensors may be used to sense muscular activity related to a movement of the part of the body controlled by muscles from which the neuromuscular sensors are arranged to sense the muscle activity. Spatial information (e.g., position and/or orientation information) and force information describing the movement may be predicted based on the sensed neuromuscular signals as the user moves over time.


Autonomous sensors may include one or more Inertial Measurement Units (IMUs), which measure a combination of physical aspects of motion, using, for example, an accelerometer, a gyroscope, a magnetometer, or any combination of one or more accelerometers, gyroscopes and magnetometers. In some embodiments, IMUs may be used to sense information about the movement of the part of the body on which the IMU is attached and information derived from the sensed data (e.g., position and/or orientation information) may be tracked as the user moves over time. For example, one or more IMUs may be used to track movements of portions of a user's body proximal to the user's torso relative to the sensor (e.g., arms, legs) as the user moves over time.


In embodiments that include at least one IMU and a plurality of neuromuscular sensors, the IMU(s) and neuromuscular sensors may be arranged to detect movement of different parts of the human body. For example, the IMU(s) may be arranged to detect movements of one or more body segments proximal to the torso (e.g., an upper arm), whereas the neuromuscular sensors may be arranged to detect movements of one or more body segments distal to the torso (e.g., a forearm or wrist). It should be appreciated, however, that autonomous sensors may be arranged in any suitable way, and embodiments of the technology described herein are not limited based on the particular sensor arrangement. For example, in some embodiments, at least one IMU and a plurality of neuromuscular sensors may be co-located on a body segment to track movements of body segment using different types of measurements. In one implementation described in more detail below, an IMU sensor and a plurality of EMG sensors are arranged on a wearable device configured to be worn around the lower arm or wrist of a user. In such an arrangement, the IMU sensor may be configured to track movement information (e.g., positioning and/or orientation over time) associated with one or more arm segments, to determine, for example whether the user has raised or lowered their arm, whereas the EMG sensors may be configured to determine movement information associated with wrist or hand segments to determine, for example, whether the user has an open or closed hand configuration.


Each of the autonomous sensors includes one or more sensing components configured to sense information about a user. In the case of IMUs, the sensing components may include one or more accelerometers, gyroscopes, magnetometers, or any combination thereof to measure characteristics of body motion, examples of which include, but are not limited to, acceleration, angular velocity, and sensed magnetic field around the body. In the case of neuromuscular sensors, the sensing components may include, but are not limited to, electrodes configured to detect electric potentials on the surface of the body (e.g., for EMG sensors) vibration sensors configured to measure skin surface vibrations (e.g., for MMG sensors), and acoustic sensing components configured to measure ultrasound signals (e.g., for SMG sensors) arising from muscle activity.


In some embodiments, the output of one or more of the sensing components may be processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and/or rectification). In other embodiments, at least some signal processing of the output of the sensing components may be performed in software. Thus, signal processing of autonomous signals recorded by the autonomous sensors may be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect.


In some embodiments, the recorded sensor data may be processed to compute additional derived measurements that are then provided as input to a statistical model, as described in more detail below. For example, recorded signals from an IMU sensor may be processed to derive an orientation signal that specifies the orientation of a rigid body segment over time. Autonomous sensors may implement signal processing using components integrated with the sensing components, or at least a portion of the signal processing may be performed by one or more components in communication with, but not directly integrated with the sensing components of the autonomous sensors.


In some embodiments, at least some of the plurality of autonomous sensors are arranged as a portion of a wearable device configured to be worn on or around part of a user's body. For example, in one non-limiting example, an IMU sensor and a plurality of neuromuscular sensors are arranged circumferentially around an adjustable and/or elastic band such as a wristband or armband configured to be worn around a user's wrist or arm. Alternatively, at least some of the autonomous sensors may be arranged on a wearable patch configured to be affixed to a portion of the user's body. In some embodiments, multiple wearable devices, each having one or more IMUs and/or neuromuscular sensors included thereon may be used to predict musculoskeletal position information for movements that involve multiple parts of the body.


In some embodiments, sensors 102 only includes a plurality of neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensors 102 includes a plurality of neuromuscular sensors and at least one “auxiliary” sensor configured to continuously record a plurality of auxiliary signals. Examples of auxiliary sensors include, but are not limited to, other autonomous sensors such as IMU sensors, and non-autonomous sensors such as an imaging device (e.g., a camera), a radiation-based sensor for use with a radiation-generation device (e.g., a laser-scanning device), or other types of sensors such as a heart-rate monitor.


System 100 also includes one or more computer processors (not shown in FIG. 1) programmed to communicate with sensors 102. For example, signals recorded by one or more of the sensors may be provided to the processor(s), which may be programmed to execute one or more machine learning techniques that process signals output by the sensors 102 to train one or more statistical models 104, and the trained (or retrained) statistical model(s) 104 may be stored for later use in generating a musculoskeletal representation 106, as described in more detail below. Non-limiting examples of statistical models that may be used in accordance with some embodiments to predict handstate information based on recorded signals from sensors 102 are discussed in more detail below.


System 100 also optionally includes a display controller configured to display a visual representation 108 (e.g., of a hand). As discussed in more detail below, one or more computer processors may implement one or more trained statistical models configured to predict handstate information based, at least in part, on signals recorded by sensors 102. The predicted handstate information is used to update the musculoskeletal representation 106, which is then optionally used to render a visual representation 108 based on the updated musculoskeletal representation incorporating the current handstate information. Real-time reconstruction of the current handstate and subsequent rendering of the visual representation reflecting the current handstate information in the musculoskeletal model may provide visual feedback to the user about the effectiveness of the trained statistical model to accurately represent an intended handstate. Not all embodiments of system 100 include components configured to render a visual representation. For example, in some embodiments, handstate estimates output from the trained statistical model and a corresponding updated musculoskeletal representation are used to determine a state of a user's hand (e.g., in a virtual reality environment) even though a visual representation based on the updated musculoskeletal representation is not rendered (e.g. for interacting with virtual objects in a virtual environment in the absence of a virtually-rendered hand).


In some embodiments, a computer application configured to simulate a virtual reality environment may be instructed to display a visual representation of the user's hand. Positioning, movement, and/or forces applied by portions of the hand within the virtual reality environment may be displayed based on the output of the trained statistical model(s). The visual representation may be dynamically updated based on current reconstructed handstate information as continuous signals are recorded by the sensors 102 and processed by the trained statistical model(s) 104 to provide an updated computer-generated representation of the user's movement and/or exerted force that is updated in real-time.


As discussed above, some embodiments are directed to using a statistical model for predicting musculoskeletal information based on signals recorded from wearable autonomous sensors. The statistical model may be used to predict the musculoskeletal position information without having to place sensors on each segment of the rigid body that is to be represented in the computer-generated musculoskeletal representation. As discussed briefly above, the types of joints between segments in a multi-segment articulated rigid body model constrain movement of the rigid body. Additionally, different individuals tend to move in characteristic ways when performing a task that can be captured in statistical patterns of individual user behavior. At least some of these constraints on human body movement may be explicitly incorporated into statistical models used for prediction in accordance with some embodiments.


Additionally or alternatively, the constraints may be learned by the statistical model though training based on ground truth data on the position and exerted forces of the hand and wrist in the context of recorded sensor data (e.g., EMG data). Constraints imposed in the construction of the statistical model are those set by anatomy and the physics of a user's body, while constraints derived from statistical patterns are those set by human behavior for one or more users from which sensor measurements are measured and used to train the statistical model. As is described in more detail below, the constraints may comprise part of the statistical model itself being represented by information (e.g., connection weights between nodes) in the model.


As discussed above, some embodiments are directed to using a statistical model for predicting handstate information to enable the generation and/or real-time update of a computer-based musculoskeletal representation. The statistical model may be used to predict the handstate information based on IMU signals, neuromuscular signals (e.g., EMG, MMG, and SMG signals), external device signals (e.g., camera or laser-scanning signals), or a combination of IMU signals, neuromuscular signals, and external device signals detected as a user performs one or more movements.


Although it is appreciated that statistical modeling can be used to predict handstate information, it should be appreciated that the states of other parts of the body may be modeled and predicted using statistical models. To this end, there may be other types of devices having one or more sensor types that can be used on other parts of the body to predict their position and any associated forces, movements, or accelerations using, for example, IMU, neuromuscular signals, or other external device signals.


According to one aspect, it is appreciated that the instantaneous best (e.g. lowest mean square estimate (MSE)) estimate of the actual joint angles and forces may fluctuate with an autocorrelation function that does not match that of behavior. This fluctuation results in poor renderings and sub-optimal interactive experiences. According to one aspect, a real-time smoothing function is provided for adjusting handstate (or other body part) model estimates. For example, model prediction outputs may be smoothed using one or more processing steps (e.g., a filter), introducing a smoothing function within the model, or providing other adjustments or controls that permit the output to more accurately represent handstate in a variety of movement conditions or permit the output to be perceived by the user (or a third-party in a multi-player virtual environment) as more natural and realistic (even if such rendering is not intended to maximize accuracy of rendering a particular instance of a movement by a user). Such smoothing may be performed in real-time, providing adjustment of a representation of the model state in a more accurate and responsive manner.



FIG. 2 illustrates a process 200 in accordance with some embodiments. At block 201, the system (e.g., such as that discussed above with reference to FIG. 1) inputs sensor data to one or more statistical models. As discussed above, a statistical model that represents handstate may be used to project estimates of position and forces in a computer-based musculoskeletal representation associated with a hand when a user performs one or more gestures. Neuromuscular signals provided by a wearable neuromuscular device is recorded and provided as an input to the statistical model.


At block 202, the system temporally smooths an output of the statistical models in real-time. For instance, the system may include one or more components that enforce a temporal smoothness of the statistical model outputs. In one implementation, the system may perform a smoothing operation on one or more outputs of the model prediction outputs.


At block 203, the system determines position information of a spatial relationship between connected segments of the musculoskeletal representation. Further, at block 204, the system determines forces exerted by the musculoskeletal representation. Collectively, the combination of position information and force information is referred to as a handstate of the musculoskeletal model. At block 205, the system updates the musculoskeletal representation based on the position information and determined force.



FIG. 3 illustrates a schematic diagram of a computer-based process 300 for smoothing an output of the statistical model in real-time according to some embodiments. At block 301, an output is provided from a trained statistical model of a hand. To that output is applied, at block 302, at least one filter, which then provides at block 303, a temporally smoothed output in real-time. The smoothed output may be used to update the musculoskeletal representation which can be then used to render, for example, a visual representation based on the updated musculoskeletal representation that incorporates the current handstate information or may otherwise be used as a control input for one or more systems. Such a visual representation may be shown, for example, within a virtual reality environment. In other embodiments, the output may be used to control one or more systems such as a robotic system that is responsive to handstate information. In such systems, it may be useful to provide a smoothing effect that restricts outputs responsive to situational outputs that provide inaccurate representations. Such smoothing may be performed in real-time so as to be useful within such virtual reality, control, or other environments.



FIG. 4 illustrates a schematic diagram of a computer-based process 400 for temporally smoothing an output in real-time according to some embodiments. According to one aspect, it is appreciated that the best estimate of the model at any particular time point may not be the correct or most accurate representation. Thus, in a system where sensor outputs are received in real-time, including noise, high variation outputs (e.g., highly variant movement data), or other signals that may drastically change the state of the statistical model, it may be beneficial to provide some type of smoothing function (e.g. a smoothing function correspond to a non-linear Kalmaan filter) that reduces errors and/or otherwise improves the reliability that the output response models the user's intended movement. In some cases, a smoothing function may be configured to generate an output response that adheres to realistic physics, anatomy of a user's body part (e.g. a hand), natural joint kinematics, and/or common gestures (i.e. a ‘thumbs up’ gesture) rather than optimizing for accuracy of an estimate of the movement, position, and/or force of a part of the user's body (e.g., a handstate). In some instances, preset movements may be stored, and it may be determined whether a user's movement corresponds to a particular preset movement (e.g., a ‘thumbs up’ gesture). In some example implementations, the system may be capable of rendering the preset movement rather than an interpretation of the actual movement (e.g., an estimated handstate). Also, in some cases, a tradeoff may be made between an increased time lag and accuracy in the modeled movement.


According to one implementation, the component that enforces a temporal smoothness of the statistical model may be used to modulate the output of the model, and/or may comprise a pre-processing of sensor outputs prior to being processed by the model. In one example, the system may average over a small amount of consecutive time points resulting in a smooth output over time. Such an output may be adjusted by a smoothing function that operates in real-time to produce an output of a modeling operation. The smoothing function may be turned on or off based on some input, such as during a point at which there is fast movement detected. For instance, the input may be modulated at the sensor prior to being input into the statistical model. For instance, there may exist EMG artifacts that exist on the sensor outputs that can be smoothed prior to updating the statistical model. In another example, an IMU output may be accelerating very quickly, and in response, the immediate responsiveness of the model may be adjusted (e.g., downward to be less responsive to the changing input). In addition to, or in combination with pre-filtering, output of the model may be smoothed prior to, for example, representing the model visually.


Such adjustments of a smoothing function may be performed responsive to one or more control inputs. For example, it is appreciated that the trained model may record its own measure of accuracy which can be used as control input for controlling a smoothing function. In one embodiment, at block 401, the system may receive an estimate of an accuracy of the statistical model which may be used as a control function for controlling the smoothing operation. For example, an accuracy of the statistical model may comprise a likelihood or confidence metric. Further, the system may determine, at block 402, quality metrics associated with incoming neuromuscular data received from neuromuscular sensors, IMUs, or other external sensors. In some embodiments, the system trades off smoothness and error—that is, the system may perform in an adjusted response mode that makes the model less susceptible to input changes yet may not necessarily accurately reflect the current state of the body part. Responsive to these estimates and/or determined quality metrics, the system may temporally smooth an output of the model. Such a smoothed output may be provided as a control input to another system and/or rendered such as by a visualization (e.g., in a VR environment).


In one example implementation, a simple non-linear Kalman filter may be used to smooth model prediction outputs. According to one aspect, performance of the Kalman filter may be modified in real-time responsive to how fast the output changes and where the system believes the model is at in real-time (e.g., modify loss function to the model). Although a nonlinear Kalman filter may be used, other filters may also be used to smooth an output. For instance, an exponential filter, a particle filter, Bayesian filter, unscented Kalman filter (UKF) or other Kalman filter type, or other appropriate filter type may be used. Also, it is appreciated that pipelining may be used to receive and process data within the modeling system. Post-processing on the pipeline may permit the system to pick out pieces of data as the data moves through the pipeline to perform a similar filtering effect.


Also, it should be appreciated that the model itself may be controlled and/or modified based on identification of certain inputs, estimations, or outputs of the model. FIG. 5 is a schematic diagram of a computer-based process 504 including a smoothing characteristic in the trained statistical model according to some embodiments. In particular, at block 501, it is determined by the system that a smoothing adaptation should be performed within the model. Generally, the system at block 502 may include a temporal smoothing characteristic in the trained statistical model (e.g., the model of handstate).


Many different methods may be used to modify the statistical model to enact the smoothing function. For instance, at block 503, the statistical model may be trained with the penalization term. In one implementation, the system may be programmed to train the model with a penalization term, such as ∥y_t−y_{t−1}∥{circumflex over ( )}2, to promote smoothness of the model outputs. In another example implementation at block 504, the system may be configured to train the statistical model with an error function. For instance, the system may train the model with an error function that includes the error in the estimates of the time derivatives of joint angles and forces. The error function may include an error in estimates of one or more derivatives of various orders, including first-order derivatives (velocity), second-order derivatives (acceleration), third-order derivatives (jerk), fourth-order derivatives (jounce or snap), or higher-order derivatives (crackle, pop, lock, drop, shot, put, etc.).


In another implementation at block 505, the system may include an evaluation of future information into a smoothing effect. For example, the system may include, in the model, a prior estimate on the smoothness of the output time series and may perform real-time estimates at a slight lag so as to get a smoothing effect by incorporating “future” information (e.g., EMG measurements received from after the time point for which the joint angles and forces are being estimated). In another example, the system may use online quality estimates of the model output to modulates smoothing and postprocessing. For instance, the system may use an output of the IMU to modulate EMG model postprocessing. It should be appreciated that other processing stages may be used to adjust inputs and/or outputs to adjust the model and provide one or more smoothing effects, alone or in combination with other effects.


In some cases, dynamic temporal smoothing may be implemented by varying the frequency of model outputs. For example, the length of a temporal window of input data (i.e. neuromuscular data) may be varied so that a longer time window is used when a quality metric calculated by the statistical model determines that the smoothness of a series of model outputs is below a threshold. By averaging across a longer time series of sensor data, variation present in the sensor data may be reduced, causing outputs of the model to exhibit increased smoothness and improved immersion for a user experiencing a virtual rendering of the model output.


In some embodiments, temporal smoothing is achieved by modulating (either continuously or dynamically) the dimensionality of the multi-articulated rigid body used to represent a part of the user's body (e.g. a hand). For example, the output of the statistical model may exhibit higher accuracy and improved temporal smoothness (e.g. better reflecting realistic physics, joint kinematics, human anatomy, etc.) by only estimating a subset of joint angles connecting the multiple segments of the multi-articulated rigid body. Reducing the dimensionality of the estimate of position and/or force of the part of the user's body (e.g. handstate) may be implemented by projecting onto a lower-dimensional manifold at model output or at an earlier stage of the data processing pipeline. In one example of reducing the dimensionality of the multi-articulated rigid body, the proximal and distal interphalangeal joints generally cannot move with full independence from each other, so these two joints may be modeled as one fused joint angle. In another example of reducing the dimensionality of the multi-articulated rigid body, the ring and pinky fingers often exhibit coupled flexion and extension, so the flexion and extension of these two fingers may be represented as a single output in a lower-dimensional statistical model.


Temporal smoothing may be implemented by incorporating an error function that causes the output of the statistical model to more closely adhere to natural positions, movements, and/or forces of a part of a user's body (e.g. hand). For example, an extensive training set acquired during various natural hand movements and comprising neuromuscular data and ground truth data (e.g. collected from a camera-based system) about the position of joints of a multi-articulated rigid body (e.g. a hand) may be used to construct a prior on dynamics that constrains the output of the statistical model to natural patterns of movement. For instance, the system may incorporate a function that restricts the output of the statistical model to one or a combination of stored natural patterns of movement, which can then be rendered in an interface, be provided as a control input, etc.


Symbolic hand gestures represent common patterns of movement, and gestures are salient for visual immersion due in part to their role in human communication. In an embodiment, the statistical model is configured to identify, based on a pattern of neuromuscular data and/or data from other sensors (e.g. an IMU), that a user is making a particular gesture. The system may be further configured to store, on a non-transitory computer-readable storage medium, an averaged or prototypical trajectory of joint angles of a hand for a particular gesture. When the system detects a gesture (i.e. based on a likelihood metric), a visual rendering of the movement may be rendered for the user based on the stored stereotyped gesture which is highly natural and smooth, instead of rendering the idiosyncratic instance of the gesture that would otherwise be estimated by the statistical model for that particular gestural movement. For example, the system may detect a ‘thumbs-up’ gesture and render a prototypical ‘thumbs-up’ that is pre-recorded. In a variation of this embodiment, a rich set of hand postures and movements are stored, on a non-transitory computer-readable storage medium, as averaged or prototypical positions and movements, and the statistical model is configured to estimate handstate within this limited set of states and achieve temporal smoothness by reducing the set of positions and/or movements.


Other functionality may be provided which allows the model to react dynamically to system inputs. For example, components may be provided that permit dynamic operations to be performed as an auxiliary task in the model. For example, such a component may perform batching methods that enforce prediction in time-scale, perform loss functions on time series, enforce some “sticky” dynamics such as by staying in a certain position (zero position or certain gesture) until some (e.g., a Bayesian) method determines otherwise, among other types of dynamic behavior.


In other embodiments, the model may be responsive to external inputs, such as situational inputs provided in a virtual reality (VR) environments. For instance, after a rendering of the model in the VR environment, a smoothing operation may be controlled responsive to environmental situation information within the VR environment, such as a spatial relationship of the modeled representation (e.g., of a hand) in relation to one or more objects within the virtual reality environment. For instance, a smoothing effect (or a change in a dynamic smoothing effect) can be enforced on the model as the visual representation of the hand approaches an object within the VR environment. For example, a user is more likely to recognize a movement as non-natural or non-smooth if their (virtual or real) body part (e.g. hand) is in proximity to an object (in a virtual or real environment), so dynamic smoothing based on virtual object proximity to a rendered part of the user's body (e.g. hand) may enhance the user's experience of immersive virtual reality.



FIG. 6 describes a process 600 for generating (sometimes termed “training” herein) a statistical model using signals recorded from sensors 102. Process 600 may be executed by any suitable computing device(s), as aspects of the technology described herein are not limited in this respect. For example, process 600 may be executed by one or more computer processors described with reference to FIGS. 8A and 8B. As another example, one or more acts of process 600 may be executed using one or more servers (e.g., servers included as a part of a cloud computing environment). For example, at least a portion of act 610 relating to training of a statistical model (e.g., a neural network) may be performed using a cloud computing environment.


Process 600 begins at act 602, where a plurality of sensor signals are obtained for one or multiple users performing one or more movements (e.g., typing on a keyboard). In some embodiments, the plurality of sensor signals may be recorded as part of process 600. In other embodiments, the plurality of sensor signals may have been recorded prior to the performance of process 600 and are accessed (rather than recorded) at act 602.


In some embodiments, the plurality of sensor signals may include sensor signals recorded for a single user performing a single movement or multiple movements. The user may be instructed to perform a sequence of movements for a particular task (e.g., opening a door) and sensor signals corresponding to the user's movements may be recorded as the user performs the task he/she was instructed to perform. The sensor signals may be recorded by any suitable number of sensors located in any suitable location(s) to detect the user's movements that are relevant to the task performed. For example, after a user is instructed to perform a task with the fingers of his/her right hand, the sensor signals may be recorded by multiple neuromuscular sensors circumferentially (or otherwise) arranged around the user's lower right arm to detect muscle activity in the lower right arm that give rise to the right hand movements and one or more IMU sensors arranged to predict the joint angle of the user's arm relative to the user's torso. As another example, after a user is instructed to perform a task with his/her leg (e.g., to kick an object), sensor signals may be recorded by multiple neuromuscular sensors circumferentially (or otherwise) arranged around the user's leg to detect muscle activity in the leg that give rise to the movements of the foot and one or more IMU sensors arranged to predict the joint angle of the user's leg relative to the user's torso.


In some embodiments, the sensor signals obtained in act 602 correspond to signals from one type of sensor (e.g., one or more IMU sensors or one or more neuromuscular sensors) and a statistical model may be trained based on the sensor signals recorded using the particular type of sensor, resulting in a sensor-type specific trained statistical model. For example, the obtained sensor signals may comprise a plurality of EMG sensor signals arranged around the lower arm or wrist of a user and the statistical model may be trained to predict musculoskeletal position information for movements of the wrist and/or hand during performance of a task such as grasping and twisting an object such as a doorknob.


In embodiments that provide predictions based on multiple types of sensors (e.g., IMU sensors, EMG sensors, MMG sensors, SMG sensors), a separate statistical model may be trained for each of the types of sensors and the outputs of the sensor-type specific models may be combined to generate a musculoskeletal representation of the user's body. In other embodiments, the sensor signals obtained in act 602 from two or more different types of sensors may be provided to a single statistical model that is trained based on the signals recorded from the different types of sensors. In one illustrative implementation, an IMU sensor and a plurality of EMG sensors are arranged on a wearable device configured to be worn around the forearm of a user, and signals recorded by the IMU and EMG sensors are collectively provided as inputs to a statistical model, as discussed in more detail below.


In some embodiments, the sensor signals obtained in act 602 are recorded at multiple time points as a user performs one or multiple movements. As a result, the recorded signal for each sensor may include data obtained at each of multiple time points. Assuming that n sensors are arranged to simultaneously measure the user's movement information during performance of a task, the recorded sensor signals for the user may comprise a time series of K n-dimensional vectors {xk|1≤k≤K} at time points t1, t2, . . . , tk during performance of the movements.


In some embodiments, a user may be instructed to perform a task multiple times and the sensor signals and position information may be recorded for each of multiple repetitions of the task by the user. In some embodiments, the plurality of sensor signals may include signals recorded for multiple users, each of the multiple users performing the same task one or more times. Each of the multiple users may be instructed to perform the task and sensor signals and position information corresponding to that user's movements may be recorded as the user performs (once or repeatedly) the task he/she was instructed to perform. When sensor signals are collected by multiple users which are combined to generate a statistical model, an assumption is that different users employ similar musculoskeletal positions to perform the same movements. Collecting sensor signals and position information from a single user performing the same task repeatedly and/or from multiple users performing the same task one or multiple times facilitates the collection of sufficient training data to generate a statistical model that can accurately predict musculoskeletal position information associated with performance of the task.


In some embodiments, a user-independent statistical model may be generated based on training data corresponding to the recorded signals from multiple users, and as the system is used by a user, the statistical model is trained based on recorded sensor data such that the statistical model learns the user-dependent characteristics to refine the prediction capabilities of the system for the particular user.


In some embodiments, the plurality of sensor signals may include signals recorded for a user (or each of multiple users) performing each of multiple tasks one or multiple times. For example, a user may be instructed to perform each of multiple tasks (e.g., grasping an object, pushing an object, and pulling open a door) and signals corresponding to the user's movements may be recorded as the user performs each of the multiple tasks he/she was instructed to perform. Collecting such data may facilitate developing a statistical model for predicting musculoskeletal position information associated with multiple different actions that may be taken by the user. For example, training data that incorporates musculoskeletal position information for multiple actions may facilitate generating a statistical model for predicting which of multiple possible movements a user may be performing.


As discussed above, the sensor data obtained at act 602 may be obtained by recording sensor signals as each of one or multiple users performs each of one or more tasks one or more multiple times. As the user(s) perform the task(s), position information describing the spatial position of different body segments during performance of the task(s) may be obtained in act 604. In some embodiments, the position information is obtained using one or more external devices or systems that track the position of different points on the body during performance of a task. For example, a motion capture system, a laser scanner, a device to measure mutual magnetic induction, or some other system configured to capture position information may be used. As one non-limiting example, a plurality of position sensors may be placed on segments of the fingers of the right hand and a motion capture system may be used to determine the spatial location of each of the position sensors as the user performs a task such as grasping an object. The sensor data obtained at act 602 may be recorded simultaneously with recording of the position information obtained in act 604. In this example, position information indicating the position of each finger segment over time as the grasping motion is performed is obtained.


Next, process 600 proceeds to act 606, where the sensor signals obtained in act 602 and/or the position information obtained in act 604 are optionally processed. For example, the sensor signals or the position information signals may be processed using amplification, filtering, rectification, or other types of signal processing.


Next, process 600 proceeds to act 608, where musculoskeletal position characteristics are determined based on the position information (as collected in act 604 or as processed in act 606). In some embodiments, rather than using recorded spatial (e.g., x, y, z) coordinates corresponding to the position sensors as training data to train the statistical model, a set of derived musculoskeletal position characteristic values are determined based on the recorded position information, and the derived values are used as training data for training the statistical model. For example, using information about the constraints between connected pairs of rigid segments in the articulated rigid body model, the position information may be used to determine joint angles that define angles between each connected pair of rigid segments at each of multiple time points during performance of a task. Accordingly, the position information obtained in act 604 may be represented by a vector of n joint angles at each of a plurality of time points, where n is the number of joints or connections between segments in the articulated rigid body model.


Next, process 600 proceeds to act 610, where the time series information obtained at acts 602 and 608 is combined to create training data used for training a statistical model at act 610. The obtained data may be combined in any suitable way. In some embodiments, each of the sensor signals obtained at act 602 may be associated with a task or movement within a task corresponding to the musculoskeletal position characteristics (e.g., joint angles) determined based on the positional information recorded in act 604 as the user performed the task or movement. In this way, the sensor signals may be associated with musculoskeletal position characteristics (e.g., joint angles) and the statistical model may be trained to predict that the musculoskeletal representation will be characterized by particular musculoskeletal position characteristics between different body segments when particular sensor signals are recorded during performance of a particular task.


In embodiments comprising sensors of different types (e.g., IMU sensors and neuromuscular sensors) configured to simultaneously record different types of movement information during performance of a task, the sensor data for the different types of sensors may be recorded using the same or different sampling rates. When the sensor data is recorded at different sampling rates, at least some of the sensor data may be resampled (e.g., up-sampled or down-sampled) such that all sensor data provided as input to the statistical model corresponds to time series data at the same time resolution. Resampling at least some of the sensor data may be performed in any suitable way including, but not limited to using interpolation for upsampling and using decimation for downsampling.


In addition to or as an alternative to resampling at least some of the sensor data when recorded at different sampling rates, some embodiments employ a statistical model configured to accept multiple inputs asynchronously. For example, the statistical model may be configured to model the distribution of the “missing” values in the input data having a lower sampling rate. Alternatively, the timing of training of the statistical model occur asynchronously as input from multiple sensor data measurements becomes available as training data.


Next, process 600 proceeds to act 612, where a statistical model for predicting musculoskeletal position information is trained using the training data generated at act 610. The statistical model being trained may take as input a sequence of data sets each of the data sets in the sequence comprising an n-dimensional vector of sensor data. The statistical model may provide output that indicates, for each of one or more tasks or movements that may be performed by a user, the likelihood that the musculoskeletal representation of the user's body will be characterized by a set of musculoskeletal position characteristics (e.g., a set of joint angles between segments in an articulated multi-segment body model). For example, the statistical model may take as input a sequence of vectors {xk|1≤k≤K} generated using measurements obtained at time points t1, t2, tK, where the ith component of vector xj is a value measured by the ith sensor at time tj and/or derived from the value measured by the ith sensor at time tj. In another non-limiting example, a derived value provided as input to the statistical model may comprise features extracted from the data from all or a subset of the sensors at and/or prior to time tj (e.g., a covariance matrix, a power spectrum, a combination thereof, or any other suitable derived representation). Based on such input, the statistical model may provide output indicating, a probability that a musculoskeletal representation of the user's body will be characterized by a set of musculoskeletal position characteristics. As one non-limiting example, the statistical model may be trained to predict a set of joint angles for segments in the fingers in the hand over time as a user grasps an object. In this example, the trained statistical model may output, a set of predicted joint angles for joints in the hand corresponding to the sensor input.


In some embodiments, the statistical model may be a neural network and, for example, may be a recurrent neural network. In some embodiments, the recurrent neural network may be a long short-term memory (LSTM) neural network. It should be appreciated, however, that the recurrent neural network is not limited to being an LSTM neural network and may have any other suitable architecture. For example, in some embodiments, the recurrent neural network may be a fully recurrent neural network, a recursive neural network, a variational autoencoder, a Hopfield neural network, an associative memory neural network, an Elman neural network, a Jordan neural network, an echo state neural network, a second order recurrent neural network, and/or any other suitable type of recurrent neural network. In other embodiments, neural networks that are not recurrent neural networks may be used. For example, deep neural networks, convolutional neural networks, and/or feedforward neural networks, may be used.


In some of the embodiments in which the statistical model is a neural network, the output layer of the neural network may provide a set of output values corresponding to a respective set of possible musculoskeletal position characteristics (e.g., joint angles). In this way, the neural network may operate as a non-linear regression model configured to predict musculoskeletal position characteristics from raw or pre-processed sensor measurements. It should be appreciated that, in some embodiments, any other suitable non-linear regression model may be used instead of a neural network, as aspects of the technology described herein are not limited in this respect.


In some embodiments, the neural network can be implemented based on a variety of topologies and/or architectures including deep neural networks with fully connected (dense) layers, Long Short-Term Memory (LSTM) layers, convolutional layers, Temporal Convolutional Layers (TCL), or other suitable type of deep neural network topology and/or architecture. The neural network can have different types of output layers including output layers with logistic sigmoid activation functions, hyperbolic tangent activation functions, linear units, rectified linear units, or other suitable type of nonlinear unit. Likewise, the neural network can be configured to represent the probability distribution over n different classes via, for example, a softmax function or include an output layer that provides a parameterized distribution e.g., mean and variance of a Gaussian distribution.


It should be appreciated that aspects of the technology described herein are not limited to using neural networks, as other types of statistical models may be employed in some embodiments. For example, in some embodiments, the statistical model may comprise a hidden Markov model, a Markov switching model with the switching allowing for toggling among different dynamic systems, dynamic Bayesian networks, and/or any other suitable graphical model having a temporal component. Any such statistical model may be trained at act 612 using the sensor data obtained at act 602.


As another example, in some embodiments, the statistical model may take as input, features derived from the sensor data obtained at act 602. In such embodiments, the statistical model may be trained at act 612 using features extracted from the sensor data obtained at act 602. The statistical model may be a support vector machine, a Gaussian mixture model, a regression based classifier, a decision tree classifier, a Bayesian classifier, and/or any other suitable classifier, as aspects of the technology described herein are not limited in this respect. Input features to be provided as training data to the statistical model may be derived from the sensor data obtained at act 602 in any suitable way. For example, the sensor data may be analyzed as time series data using wavelet analysis techniques (e.g., continuous wavelet transform, discrete-time wavelet transform, etc.), Fourier-analytic techniques (e.g., short-time Fourier transform, Fourier transform, etc.), and/or any other suitable type of time-frequency analysis technique. As one non-limiting example, the sensor data may be transformed using a wavelet transform and the resulting wavelet coefficients may be provided as inputs to the statistical model.


In some embodiments, at act 612, values for parameters of the statistical model may be estimated from the training data generated at act 610. For example, when the statistical model is a neural network, parameters of the neural network (e.g., weights) may be estimated from the training data. In some embodiments, parameters of the statistical model may be estimated using gradient descent, stochastic gradient descent, and/or any other suitable iterative optimization technique. In embodiments where the statistical model is a recurrent neural network (e.g., an LSTM), the statistical model may be trained using stochastic gradient descent and backpropagation through time. The training may employ a cross-entropy loss function and/or any other suitable loss function, as aspects of the technology described herein are not limited in this respect.


Next, process 600 proceeds to act 614, where the trained statistical model is stored (e.g., in datastore—not shown). The trained statistical model may be stored using any suitable format, as aspects of the technology described herein are not limited in this respect. In this way, the statistical model generated during execution of process 600 may be used at a later time, for example, to predict musculoskeletal position information (e.g., joint angles) for a given set of input sensor data, as described below.


In some embodiments, sensor signals are recorded from a plurality of sensors (e.g., arranged on or near the surface of a user's body) that record activity associated with movements of the body during performance of a task. The recorded signals may be optionally processed and provided as input to a statistical model trained using one or more techniques described above. In some embodiments that continuously record autonomous signals, the continuously recorded signals (raw or processed) may be continuously or periodically provided as input to the trained statistical model for prediction of musculoskeletal position information (e.g., joint angles) for the given set of input sensor data. As discussed above, in some embodiments, the trained statistical model is a user-independent model trained based on autonomous sensor and position information measurements from a plurality of users. In other embodiments, the trained model is a user-dependent model trained on data recorded from the individual user from which the data associated with the sensor signals is also acquired.


After the trained statistical model receives the sensor data as a set of input parameters, the predicted musculoskeletal position information is output from the trained statistical model. As discussed above, in some embodiments, the predicted musculoskeletal position information may comprise a set of musculoskeletal position information values (e.g., a set of joint angles) for a multi-segment articulated rigid body model representing at least a portion of the user's body. In other embodiments, the musculoskeletal position information may comprise a set of probabilities that the user is performing one or more movements from a set of possible movements.


In some embodiments, after musculoskeletal position information is predicted, a computer-based musculoskeletal representation of the user's body is generated based, at least in part, on the musculoskeletal position information output from the trained statistical model. The computer-based musculoskeletal representation may be generated in any suitable way. For example, a computer-based musculoskeletal model of the human body may include multiple rigid body segments, each of which corresponds to one or more skeletal structures in the body. For example, the upper arm may be represented by a first rigid body segment, the lower arm may be represented by a second rigid body segment the palm of the hand may be represented by a third rigid body segment, and each of the fingers on the hand may be represented by at least one rigid body segment (e.g., at least fourth-eighth rigid body segments). A set of joint angles between connected rigid body segments in the musculoskeletal model may define the orientation of each of the connected rigid body segments relative to each other and a reference frame, such as the torso of the body. As new sensor data is measured and processed by the statistical model to provide new predictions of the musculoskeletal position information (e.g., an updated set of joint angles), the computer-based musculoskeletal representation of the user's body may be updated based on the updated set of joint angles determined based on the output of the statistical model. In this way the computer-based musculoskeletal representation is dynamically updated in real-time as sensor data is continuously recorded.


The computer-based musculoskeletal representation may be represented and stored in any suitable way, as embodiments of the technology described herein are not limited with regard to the particular manner in which the representation is stored. Additionally, although referred to herein as a “musculoskeletal” representation, to reflect that muscle activity may be associated with the representation in some embodiments, as discussed in more detail below, it should be appreciated that some musculoskeletal representations used in accordance with some embodiments may correspond to skeletal structures, muscular structures or a combination of skeletal structures and muscular structures in the body.


In some embodiments, direct measurement of neuromuscular activity and/or muscle activity underlying the user's movements may be combined with the generated musculoskeletal representation. Measurements from a plurality of sensors placed at locations on a user's body may be used to create a unified representation of muscle recruitment by superimposing the measurements onto a dynamically-posed skeleton. In some embodiments, muscle activity sensed by neuromuscular sensors and/or information derived from the muscle activity (e.g., force information) may be combined with the computer-generated musculoskeletal representation in real time.



FIG. 7A illustrates a wearable system with sixteen neuromuscular sensors 710 (e.g., EMG sensors) arranged circumferentially around an elastic band 720 configured to be worn around a user's lower arm or wrist. As shown, EMG sensors 710 are arranged circumferentially around elastic band 720. It should be appreciated that any suitable number of neuromuscular sensors may be used. The number and arrangement of neuromuscular sensors may depend on the particular application for which the wearable device is used. For example, a wearable armband or wristband can be used to generate control information for controlling an augmented reality system, a robot, controlling a vehicle, scrolling through text, controlling a virtual avatar, or any other suitable control task.


In some embodiments, sensors 710 includes a set of neuromuscular sensors (e.g., EMG sensors). In other embodiments, sensors 710 can include a set of neuromuscular sensors and at least one “auxiliary” sensor configured to continuously record auxiliary signals. Examples of auxiliary sensors include, but are not limited to, other sensors such as IMU sensors, microphones, imaging sensors (e.g., a camera), radiation based sensors for use with a radiation-generation device (e.g., a laser-scanning device), or other types of sensors such as a heart-rate monitor. As shown the sensors 710 may be coupled together using flexible electronics 730 incorporated into the wearable device. FIG. 7B illustrates a cross-sectional view through one of the sensors 710 of the wearable device shown in FIG. 7A.


In some embodiments, the output of one or more of the sensing components can be optionally processed using hardware signal processing circuitry (e.g., to perform amplification, filtering, and/or rectification). In other embodiments, at least some signal processing of the output of the sensing components can be performed in software. Thus, signal processing of signals sampled by the sensors can be performed in hardware, software, or by any suitable combination of hardware and software, as aspects of the technology described herein are not limited in this respect. A non-limiting example of a signal processing chain used to process recorded data from sensors 710 are discussed in more detail below in connection with FIGS. 8A and 8B.



FIGS. 8A and 8B illustrate a schematic diagram with internal components of a wearable system with sixteen EMG sensors, in accordance with some embodiments of the technology described herein. As shown, the wearable system includes a wearable portion 810 (FIG. 8A) and a dongle portion 820 (FIG. 8B) in communication with the wearable portion 810 (e.g., via Bluetooth or another suitable short range wireless communication technology). As shown in FIG. 8A, the wearable portion 810 includes the sensors 710, examples of which are described in connection with FIGS. 7A and 7B. The output of the sensors 710 is provided to analog front end 830 configured to perform analog processing (e.g., noise reduction, filtering, etc.) on the recorded signals. The processed analog signals are then provided to analog-to-digital converter 832, which converts the analog signals to digital signals that can be processed by one or more computer processors. An example of a computer processor that may be used in accordance with some embodiments is microcontroller (MCU) 834 illustrated in FIG. 8A. As shown, MCU 834 may also include inputs from other sensors (e.g., IMU sensor 840), and power and battery module 842. The output of the processing performed by MCU may be provided to antenna 850 for transmission to dongle portion 820 shown in FIG. 8B.


Dongle portion 820 includes antenna 852 configured to communicate with antenna 850 included as part of wearable portion 810. Communication between antenna 850 and 852 may occur using any suitable wireless technology and protocol, non-limiting examples of which include radiofrequency signaling and Bluetooth. As shown, the signals received by antenna 852 of dongle portion 820 may be provided to a host computer for further processing, display, and/or for effecting control of a particular physical or virtual object or objects.


The above-described embodiments can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. It should be appreciated that any component or collection of components that perform the functions described above can be generically considered as one or more controllers that control the above-discussed functions. The one or more controllers can be implemented in numerous ways, such as with dedicated hardware or with one or more processors programmed using microcode or software to perform the functions recited above.


In this respect, it should be appreciated that one implementation of the embodiments of the present invention comprises at least one non-transitory computer-readable storage medium (e.g., a computer memory, a portable memory, a compact disk, etc.) encoded with a computer program (i.e., a plurality of instructions), which, when executed on a processor, performs the above-discussed functions of the embodiments of the present invention. The computer-readable storage medium can be transportable such that the program stored thereon can be loaded onto any computer resource to implement the aspects of the present invention discussed herein. In addition, it should be appreciated that the reference to a computer program which, when executed, performs the above-discussed functions, is not limited to an application program running on a host computer. Rather, the term computer program is used herein in a generic sense to reference any type of computer code (e.g., software or microcode) that can be employed to program a processor to implement the above-discussed aspects of the present invention.


Various aspects of the present invention may be used alone, in combination, or in a variety of arrangements not specifically discussed in the embodiments described in the foregoing and are therefore not limited in their application to the details and arrangement of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.


Also, embodiments of the invention may be implemented as one or more methods, of which an example has been provided. The acts performed as part of the method(s) may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.


Use of ordinal terms such as “first,” “second,” “third,” etc., in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed. Such terms are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term).


The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” “having,” “containing”, “involving”, and variations thereof, is meant to encompass the items listed thereafter and additional items.


Having described several embodiments of the invention in detail, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to be within the spirit and scope of the invention. Accordingly, the foregoing description is by way of example only, and is not intended as limiting. The invention is limited only as defined by the following claims and the equivalents thereto.

Claims
  • 1. A computer-implemented method comprising: receiving one or more neuromuscular signals from a plurality of neuromuscular sensors associated with a user; andproviding, as input to a trained statistical model, the one or more neuromuscular signals;temporally smoothing one or more outputs of the trained statistical model;predicting, based on the smoothed output of the trained statistical model, a probability that a musculoskeletal representation of the user's body is characterized by a specified set of position characteristics that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation;determining, based on the predicted probability, how the two or more connected segments of the user's body are to move in relation to each other; andupdating the musculoskeletal representation based, at least in part, on the determined movements.
  • 2. The computer-implemented method of claim 1, wherein the position characteristics that that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation comprise an indication of joint type.
  • 3. The computer-implemented method of claim 1, wherein the position characteristics that that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation comprise an indication of joint angle.
  • 4. The computer-implemented method of claim 1, wherein the position characteristics that that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation comprise an indication of which user is providing the received neuromuscular signals.
  • 5. The computer-implemented method of claim 1, wherein one or more specific ways in which the user moves their body are incorporated into the trained statistical model.
  • 6. The computer-implemented method of claim 1, wherein the trained statistical model provides an output that indicates the likelihood that the musculoskeletal representation of the user's body is characterized by the specified set of musculoskeletal position characteristics.
  • 7. The computer-implemented method of claim 1, further comprising: determining the accuracy of the trained statistical model; andtemporally smoothing, responsive to the determined accuracy of the trained statistical model, the output of the trained statistical model in real-time.
  • 8. The computer-implemented method of claim 7, further comprising determining at least one of an input accuracy and an output accuracy of the trained statistical model.
  • 9. The computer-implemented method of claim 1, wherein temporally smoothing the output of the trained statistical model comprises including in the trained statistical model at least one temporal smoothing characteristic.
  • 10. The computer-implemented method of claim 9, wherein the trained statistical model that includes the at least one temporal smoothing characteristic trains the statistical model with a penalization term to promote smoothness in the model outputs.
  • 11. The computer-implemented method of claim 9, wherein the trained statistical model that includes the at least one temporal smoothing characteristic trains the statistical model with an error function that includes an error in estimates of time derivatives of the output of the statistical model.
  • 12. The computer-implemented method of claim 9, wherein the trained statistical model that includes the at least one temporal smoothing characteristic includes in the trained statistical model a prior on the smoothness of the output of the statistical model.
  • 13. A system comprising: at least one physical processor; andphysical memory comprising computer-executable instructions that, when executed by the physical processor, cause the physical processor to: receive one or more neuromuscular signals from a plurality of neuromuscular sensors associated with a user; andprovide, as input to a trained statistical model, the one or more neuromuscular signals;temporally smooth one or more outputs of the trained statistical model;predict, based on the smoothed output of the trained statistical model, a probability that a musculoskeletal representation of the user's body is characterized by a specified set of position characteristics that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation;determine, based on the predicted probability, how the two or more connected segments of the user's body are to move in relation to each other; andupdate the musculoskeletal representation based, at least in part, on the determined movements.
  • 14. The system of claim 13, wherein temporally smoothing the outputs of the trained statistical model comprises processing the output of the trained statistical model using at least one filter.
  • 15. The system of claim 13, wherein temporally smoothing the output of the trained statistical model further comprises generating the output of the trained statistical model for a first timepoint based on first neuromuscular signals recorded at the first timepoint and second neuromuscular signals recorded at at least one second timepoint after the first timepoint.
  • 16. The system of claim 13, wherein the at least one physical processor is further programmed to: determine a quality of the smoothed output of the trained statistical model at a first timepoint; andadjust at least one smoothing parameter based on the determined quality; andtemporally smooth output of the trained statistical model at a second timepoint based on the at least one adjusted smoothing parameter.
  • 17. The system of claim 13, wherein the at least one physical processor is further programmed to temporally smooth the output of the trained statistical model based, at least in part, on at least one characteristic of a motion of the user during recording of the plurality of neuromuscular signals.
  • 18. The system of claim 13, further comprising: at least one inertial measurement unit configured to continuously record a plurality of inertial measurement unit signals, wherein the at least one characteristic of a motion of the user is determined based, at least in part, on the plurality of inertial measurement unit signals.
  • 19. The system of claim 18, wherein the at least one characteristic of a motion of the user comprises a speed at which a user is moving a hand and/or arm of the user.
  • 20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to: receive one or more neuromuscular signals from a plurality of neuromuscular sensors associated with a user; andprovide, as input to a trained statistical model, the one or more neuromuscular signals;temporally smooth one or more outputs of the trained statistical model;predict, based on the smoothed output of the trained statistical model, a probability that a musculoskeletal representation of the user's body is characterized by a specified set of position characteristics that describe how two or more connected segments of the user's body move in relation to each other within the musculoskeletal representation;determine, based on the predicted probability, how the two or more connected segments of the user's body are to move in relation to each other; andupdate the musculoskeletal representation based, at least in part, on the determined movements.
CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No. 16/589,539, filed 1 Oct. 2019, which is a continuation of U.S. application Ser. No. 16/258,232, filed 25 Jan. 2019, which claims priority to U.S. Provisional Application Ser. No. 62/621,741, entitled “Real-Time Processing of Handstate Representation Model Estimates,” filed on Jan. 25, 2018, each of which are incorporated by reference in their entirety herein.

US Referenced Citations (185)
Number Name Date Kind
4896120 Kamil Jan 1990 A
5625577 Kunii et al. Apr 1997 A
6005548 Latypov et al. Dec 1999 A
6009210 Kand Dec 1999 A
6244873 Hill et al. Jun 2001 B1
6658287 Litt et al. Dec 2003 B1
6720984 Jorgensen et al. Apr 2004 B1
6774885 Even-Zahar Aug 2004 B1
6942621 Avinash et al. Sep 2005 B2
7089148 Baclunann et al. Aug 2006 B1
7351975 Brady et al. Apr 2008 B2
7574253 Edney et al. Aug 2009 B2
7580742 Tan et al. Aug 2009 B2
7787946 Stahmann et al. Aug 2010 B2
7805386 Greer Sep 2010 B2
7901368 Flaherty et al. Mar 2011 B2
8170656 Tan et al. May 2012 B2
8190249 Gharieb et al. May 2012 B1
8311623 Sanger Nov 2012 B2
8351651 Lee Jan 2013 B2
8421634 Tan et al. Apr 2013 B2
8435191 Barboutis et al. May 2013 B2
8437844 Syed Momen et al. May 2013 B2
8447704 Tan et al. May 2013 B2
8484022 Vanhoucke Jul 2013 B1
8718980 Garudadri et al. May 2014 B2
8744543 Li et al. Jun 2014 B2
8754862 Zaliva Jun 2014 B2
8880163 Barachant et al. Nov 2014 B2
8890875 Jammes et al. Nov 2014 B2
8892479 Tan et al. Nov 2014 B2
9037530 Tan et al. May 2015 B2
9218574 Phillipps et al. Dec 2015 B2
9235934 Mandella et al. Jan 2016 B2
9240069 Li Jan 2016 B1
9278453 Assad Mar 2016 B2
9299248 Lake et al. Mar 2016 B2
9367139 Ataee et al. Jun 2016 B2
9372535 Bailey et al. Jun 2016 B2
9389694 Ataee et al. Jul 2016 B2
9408316 Bailey et al. Aug 2016 B2
9459697 Bedikian et al. Oct 2016 B2
9483123 Aleem et al. Nov 2016 B2
9597015 McNames et al. Mar 2017 B2
9600030 Bailey et al. Mar 2017 B2
9612661 Wagner et al. Apr 2017 B2
9613262 Holz Apr 2017 B2
9659403 Horowitz May 2017 B1
9687168 Jolm Jun 2017 B2
9696795 Marcolina et al. Jul 2017 B2
9720515 Wagner et al. Aug 2017 B2
9741169 Holz Aug 2017 B1
9766709 Holz Sep 2017 B2
9785247 Horowitz et al. Oct 2017 B1
9788789 Bailey Oct 2017 B2
9864431 Keskin et al. Jan 2018 B2
9867548 Le et al. Jan 2018 B2
9880632 Ataee et al. Jan 2018 B2
9891718 Connor Feb 2018 B2
10042422 Morun et al. Aug 2018 B2
10070799 Ang et al. Sep 2018 B2
10101809 Morun et al. Oct 2018 B2
10152082 Bailey Dec 2018 B2
10188309 Morun et al. Jan 2019 B2
10199008 Aleem et al. Feb 2019 B2
10203751 Keskin et al. Feb 2019 B2
10216274 Chapeskie et al. Feb 2019 B2
10460455 Giurgica-Tiron et al. Oct 2019 B2
20030144829 Geatz et al. Jul 2003 A1
20030171921 Manabe et al. Sep 2003 A1
20040092839 Shin et al. May 2004 A1
20070172797 Hada et al. Jul 2007 A1
20070256494 Nakamura et al. Nov 2007 A1
20070285399 Lund Dec 2007 A1
20080052643 Ike et al. Feb 2008 A1
20080214360 Stirling et al. Sep 2008 A1
20080221487 Zohar et al. Sep 2008 A1
20090112080 Matthews Apr 2009 A1
20090326406 Tan et al. Dec 2009 A1
20090327171 Tan et al. Dec 2009 A1
20100030532 Arora et al. Feb 2010 A1
20100063794 Hernandez-Rebollar Mar 2010 A1
20100106044 Linderman Apr 2010 A1
20100280628 Sankai Nov 2010 A1
20100292617 Lei et al. Nov 2010 A1
20100293115 SeyedMomen Nov 2010 A1
20100315266 Gunawardana Dec 2010 A1
20110077484 Van Slyke et al. Mar 2011 A1
20110092826 Lee et al. Apr 2011 A1
20120066163 Balls et al. Mar 2012 A1
20120188158 Tan et al. Jul 2012 A1
20120265480 Oshima Oct 2012 A1
20120283526 Gommesen et al. Nov 2012 A1
20130077820 Marais et al. Mar 2013 A1
20130141375 Ludwig et al. Jun 2013 A1
20130207889 Chang et al. Aug 2013 A1
20130217998 Mahfouz et al. Aug 2013 A1
20130232095 Tan et al. Sep 2013 A1
20130317382 Le Nov 2013 A1
20130317648 Assad Nov 2013 A1
20140052150 Taylor et al. Feb 2014 A1
20140092009 Yen et al. Apr 2014 A1
20140098018 Kim et al. Apr 2014 A1
20140196131 Lee Jul 2014 A1
20140198034 Bailey et al. Jul 2014 A1
20140198035 Bailey et al. Jul 2014 A1
20140240103 Lake et al. Aug 2014 A1
20140240223 Lake et al. Aug 2014 A1
20140245200 Holz Aug 2014 A1
20140249397 Lake et al. Sep 2014 A1
20140278441 Ton et al. Sep 2014 A1
20140304665 Holz Oct 2014 A1
20140334083 Bailey Nov 2014 A1
20140344731 Holz Nov 2014 A1
20140355825 Kim et al. Dec 2014 A1
20140365163 Jallon Dec 2014 A1
20140376773 Holz Dec 2014 A1
20150006120 Sett et al. Jan 2015 A1
20150010203 Muninder et al. Jan 2015 A1
20150025355 Bailey et al. Jan 2015 A1
20150029092 Holz et al. Jan 2015 A1
20150035827 Yamaoka et al. Feb 2015 A1
20150045699 Mokaya et al. Feb 2015 A1
20150051470 Bailey et al. Feb 2015 A1
20150057770 Bailey et al. Feb 2015 A1
20150070270 Bailey et al. Mar 2015 A1
20150084860 Aleem et al. Mar 2015 A1
20150109202 Ataee et al. Apr 2015 A1
20150124566 Lake et al. May 2015 A1
20150128094 Baldwin et al. May 2015 A1
20150141784 Morun et al. May 2015 A1
20150148641 Morun et al. May 2015 A1
20150157944 Gottlieb Jun 2015 A1
20150169074 Ataee et al. Jun 2015 A1
20150193949 Katz et al. Jul 2015 A1
20150234426 Bailey et al. Aug 2015 A1
20150261306 Lake Sep 2015 A1
20150261318 Scavezze et al. Sep 2015 A1
20150277575 Ataee et al. Oct 2015 A1
20150302168 De Sapia et al. Oct 2015 A1
20150309563 Connor Oct 2015 A1
20150309582 Gupta Oct 2015 A1
20150313496 Connor Nov 2015 A1
20150323998 Kudekar et al. Nov 2015 A1
20150325202 Lake et al. Nov 2015 A1
20150346701 Gordon et al. Dec 2015 A1
20150370326 Chapeskie Dec 2015 A1
20150370333 Ataee et al. Dec 2015 A1
20160011668 Gilad-Bachrach et al. Jan 2016 A1
20160187992 Yamamoto et al. Jun 2016 A1
20160235323 Tadi et al. Aug 2016 A1
20160262687 Imperial Sep 2016 A1
20160274758 Bailey Sep 2016 A1
20160292497 Kehtarnavaz et al. Oct 2016 A1
20160313801 Wagner et al. Oct 2016 A1
20160313899 Noel Oct 2016 A1
20170061817 Mettler May Mar 2017 A1
20170080346 Abbas Mar 2017 A1
20170090604 Barbier Mar 2017 A1
20170119472 Herrmann May 2017 A1
20170123487 Hazra May 2017 A1
20170124816 Yang et al. May 2017 A1
20170188980 Ash Jul 2017 A1
20170259167 Cook et al. Sep 2017 A1
20170308118 Ito Oct 2017 A1
20180020951 Kaifosh et al. Jan 2018 A1
20180020978 Kaifosh et al. Jan 2018 A1
20180024634 Kaifosh et al. Jan 2018 A1
20180024635 Kaifosh et al. Jan 2018 A1
20180064363 Morun et al. Mar 2018 A1
20180067553 Morun et al. Mar 2018 A1
20180088765 Bailey Mar 2018 A1
20180095630 Bailey Apr 2018 A1
20180101289 Bailey Apr 2018 A1
20180120948 Aleem et al. May 2018 A1
20180150033 Lake et al. May 2018 A1
20180153430 Ang et al. Jun 2018 A1
20180153444 Yang et al. Jun 2018 A1
20180154140 Bouton et al. Jun 2018 A1
20180307314 Connor Oct 2018 A1
20180344195 Morun et al. Dec 2018 A1
20180360379 Harrison et al. Dec 2018 A1
20190076716 Chiou et al. Mar 2019 A1
20190121305 Kaifosh et al. Apr 2019 A1
20190121306 Kaifosh et al. Apr 2019 A1
Foreign Referenced Citations (41)
Number Date Country
2 902 045 Aug 2014 CA
2 921 954 Feb 2015 CA
2 939 644 Aug 2015 CA
1838933 Sep 2006 CN
104665962 Jun 2015 CN
105190578 Dec 2015 CN
105578954 May 2016 CN
105830092 Aug 2016 CN
106102504 Nov 2016 CN
106227346 Dec 2016 CN
106293057 Jan 2017 CN
106648071 May 2017 CN
111902847 Nov 2020 CN
2 198 521 Jun 2012 EP
2 959 394 Dec 2015 EP
3 073 351 Sep 2016 EP
3 104 737 Dec 2016 EP
H05-277080 Oct 1993 JP
2005-095561 Apr 2005 JP
2010-520561 Jun 2010 JP
2016-507851 Mar 2016 JP
2017-509386 Apr 2017 JP
2015-0123254 Nov 2015 KR
2016-0121552 Oct 2016 KR
2008109248 Sep 2008 WO
2009042313 Apr 2009 WO
2014130871 Aug 2014 WO
2014186370 Nov 2014 WO
2014194257 Dec 2014 WO
2014197443 Dec 2014 WO
2015027089 Feb 2015 WO
2015073713 May 2015 WO
2015081113 Jun 2015 WO
2015123445 Aug 2015 WO
2015199747 Dec 2015 WO
2016041088 Mar 2016 WO
2017062544 Apr 2017 WO
2017092225 Jun 2017 WO
2017120669 Jul 2017 WO
2017172185 Oct 2017 WO
2019147949 Aug 2019 WO
Non-Patent Literature Citations (63)
Entry
International Search Report and Written Opinion for International Application No. PCT/US2017/043686 dated Oct. 6, 2017.
International Preliminary Report on Patentability for International Application No. PCT/US2017/043686 dated Feb. 7, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2017/043693 dated Oct. 6, 2017.
International Preliminary Report on Patentability for International Application No. PCT/US2017/043693 dated Feb. 7, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2017/043791 dated Oct. 5, 2017.
International Preliminary Reporton Patentability for International Application No. PCT/US2017/043791 dated Feb. 7, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2017/043792 dated Oct. 5, 2017.
International Preliminary Report on Patentability for International Application No. PCT/US2017/043792 dated Feb. 7, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2018/056768 dated Jan. 15, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2018/061409 dated Mar. 12, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2018/063215 dated Mar. 21, 2019.
Benko et al., Enhancing Input on and Above the Interactive Surface with Muscle Sensing. The ACM International Conference on Interactive Tabletops and Surfaces. ITS '09. 2009: 93-100.
Boy Ali et al., Spectral Collaborative Representation based Classification for hand gestures recognition on electromyography signals. Biomedical Signal Processing and Control. 2016;24:11-18.
Cheng et al., A Novel Phonology- and Radical-Coded Chinese Sign Language Recognition Framework Using Accelerometer and Surface Electromyography Sensors. Sensors. 2015; 15:23303-24.
Csapo et al., Evaluation of Human-Myo Gesture Control Capabilities in Continuous Search and Select Operations. 7th IEEE International Conference on Cognitive Infocommunications. 2016;000415-20.
Delis et al., Development of a Myoelectric Controller Based on Knee Angle Estimation. Biodevices 2009. International Conference on Biomedical Electronics and Devices. Jan. 17, 2009. 7 pages.
Diener et al., Direct conversion from facial myoelectric signals to speech using Deep Neural Networks. 2015 International Joint Conference on Neural Networks (IJCNN). Oct. 1, 2015 7 pages.
Ding et al., HMM with improved feature extraction-based feature parameters for identity recognition of gesture command operators by using a sensed Kinect-data stream Neurocomputing 2017;262:108-19.
Farin A et al., Man/machine interface based on the discharge timings of spinal motor neurons after targeted muscle reinnervation. Nature. Bio medical Engineering. 2017; 1: 1-12.
Gallina et al., Surface EMG Biofeedback. Surface Electromyography: Physiology, Engineering, and Applications. 2016:485-500.
Jiang, Purdue University Graduate School Thesis/Dissertation Acceptance. Graduate School Form 30. Updated Jan. 15, 2015. 24 pages.
Kawaguchi et al., Estimation of Finger Joint Angles Based on Electromechanical Sensing of Wrist Shape. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2017;25(9):1409-18.
Kim et al., Real-Time Human Pose Estimation and Gesture Recognition from Depth Images Using Superpixels and SVM Classifier. Sensors. 2015;15:12410-27.
Koerner, Design and Characterization of the Exo-SkinHaptic Device: A Novel Tendon Actuated Textile Hand Exoskeleton 2017. 5 pages.
Li et al., Motor Function Evaluation of Hemiplegic Upper-Extremities Using Data Fusion from Wearable Inertial and Surface EMG Sensors. Sensors. MDPI. 2017;17(582):1-17.
McIntee, A Task Model of Free-Space Movement-Based Geastures. Dissertation. Graduate Faculty of North Carolina State University. Computer Science. 2016. 129 pages.
Naik et al., Source Separation and Identification issues in bio signals: A solution using Blind source seperation. Intech. 2009. 23 pages.
Naik et al., Subtle Hand gesture identification for HCI using Temporal Decorrelation Source Separation BSS of surface EMG. Digital Image Computing Techniques and Applications. IEEE Computer Society 2007;30-7.
Negro et al., Multi-channel intramuscular and surface EMG decomposition by convolutive blind source separation. Journal of Neural Engineering. 2016;13:1-17.
Saponas et al., Demonstrating the Feasibility of Using Forearm Electromyography for Muscle-Computer Interfaces. CHI 2008 Proceedings. Physiological Sensing for Input. 2008:515-24.
Saponas et al., Enabling Always-Available Input with Muscle—Computer Interfaces. UIST '09. 2009:167-76.
Saponas et al., Making Muscle—Computer Interfaces More Practical. CHI 2010: Brauns and Brawn. 2010:851-4.
Sauras-Perez et al., A Voice and Pointing Gesture Interaction System for Supporting Human Spontaneous Decisions in Autonomous Cars. Clemson University. All Dissertations. 2017. 174 pages.
Shen et al., I am a Smartwatch and I can Track my User's Arm. University of Illinois at Urbana-Champaign. MobiSys'16.
Son et al., Evaluating the utility of two gestural discomfort evaluation methods. PLOS One. 2017. 21 pages.
Strbac et al., Microsoft Kinect-Based Artificial Perception System for Control of Functional Electrical Stimulation Assisted Grasping. Hindawi Publishing Corporation. BioMed Research International 2014. 13 pages.
Torres, Myo Gesture Control Armband. PCMag. Https://www .pcmag.com/article2/0,28 I 7 ,2485462,00 .asp 2015. 9 pages.
Wodzinski et al., Sequential Classification of Palm Gestures Based on A* Algorithm and MLP Neural Network for Quadrocopter Control. Metrol. Meas. Syst., 20 I 7;24(2):265-76.
Xue et al., Multiple Sensors Based Hand Motion Recognition Using Adaptive Directed Acyclic Graph. Applied Sciences. MDPI. 2017;7(358):1-14.
Non-Final Rejection received for U.S. Appl. No. 16/258,232 dated Apr. 30, 2019, 15 pages.
Notice of Allowance received for U.S. Appl. No. 16/258,232 dated Jun. 14, 2019, 18 pages.
International Search Report and Written Opinion for International Application No. PCT/US2019/015134 dated May 15, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015167 dated May 21, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015174 dated May 21, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015183 dated May 3, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015238 dated May 16, 2019.
International Search Report and Written Opinion for International Application No. PCT/US2019/015244 dated May 16, 2019.
International Search Report and Written Opinion for International Application No. PCT/US19/20065 dated May 16, 2019.
Arkenbout et al., Robust Hand Motion Tracking through Data Fusion of 5DT Data Glove and Nimble VR Kinect Camera Measurements Sensors. 2015;15:31644-71.
Da Voodi et al., Development of a Physics-Based Target Shooting Game to Train Amputee Users of Multijoint Upper Limb Protheses. Presence. Massachusetts Institute of Technology. 2012;21(1):85-95.
Office Action dated Feb. 17, 2022 for Chinese Application No. 201980022115.1, filed Jan. 25, 2019, 24 pages.
Fa Vorska Ya et al., Localization and Recognition of Dynamic Hand Gestures Based on Hierarchy of Manifold Classifiers. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. 2015; XL-5/W6:1-8.
Hauschild et al., A Virtual Reality Environment for Designing and Fitting Neural Prosthetic Limbs. IEEE Transactions on Neural Systems and Rehabilitation Engineering 2007;15(1):9-15.
Lee et al., Motion and Force Estimation System of Human Fingers. Journal of Institute of Control, Robotics and Systems. 2011;17(10):1014-1020.
Lopes et al., Hand/arm gesture segmentation by motion using IMU and EMG sensing. ScienceDirect. Elsevier. Procedia Manufacturing. 2017;1l:107-13.
Martin et al., A Novel Approach of Prosthetic Arm Control using Computer Vision, Biosignals, and Motion Capture. IEEE 2014. 5 pages.
Mendes et al., Sensor Fusion and Smart Sensor in Sports and Biomedical Applications. Sensors. 2016; 16(1569):1-31.
Sartori et al., Neural Data-Driven Musculoskeletal Modeling for Personalized Neurorehabilitation Technologies. IEEE Transactions on Biomedical Engineering. 2016;63(5):879- 93.
International Preliminary Report on Patentability received for PCT Patent Application Serial No. PCT/US2019/015167 dated Aug. 6, 2020, 6 pages.
Extended European Search Report received for EP Patent Application Serial No. 19744115.7 dated Feb. 25, 2021, 10 pages.
Non-Final Office Action received for U.S. Appl. No. 16/589,539 dated Mar. 8, 2021, 17 pages.
Notice of Allowance received for U.S. Appl. No. 16/589,539 dated May 27, 2021, 49 pages.
First Office Action received for Chinese Patent Application Serial No. 201980022115.1 dated Jul. 16, 2021, 22 pages. (Including English Translation).
Provisional Applications (1)
Number Date Country
62621741 Jan 2018 US
Continuations (2)
Number Date Country
Parent 16589539 Oct 2019 US
Child 17408339 US
Parent 16258232 Jan 2019 US
Child 16589539 US