The present application generally relates to a biomechanical assistive device, and particularly to activity recognition by the assistive device.
Exoskeletons are devices that can amplify a person's natural ability and improve their quality of life. In one or more examples, exoskeleton devices facilitate overcoming physical human limitations by amplifying human strength, endurance, and mobility potential. The exoskeleton devices are thus biomechanical assistive devices that may be worn by a user, for example worn in association with a joint in the body, to amplify or improve the functioning of that joint.
Exoskeleton devices can be classified as either passive or powered devices. A passive device typically cannot generate and deliver energy external to the user, rather a passive device helps the user employ his own muscle power more effectively. Passive devices can include springs, and can store potential energy and deliver it in addition to the human motion. One example of exoskeleton-based passive assist is passive gravity support where the exoskeleton supports part of the user's weight. However, the exoskeleton cannot contribute to raise the user's center of gravity, for example when getting up from a chair.
A powered exoskeleton device on the other hand generates and supplies energy to the user through external means (i.e. electrical, hydraulic, etc), in one or more examples, in a continuous way, to help the user to elevate the center of mass of the body at one point or another by generating torque, for example using one or more actuators. The biomechanical assistive devices that are described herein are powered exoskeleton devices.
For operation of the assistive devices, the devices have to provide the appropriate amount of torque to assist with the user's activity, one way of providing such assist is done by detecting the user's current activity (ex. walking, standing, sitting). Typically, the assistive devices require direct user input, or are very slow to recognize activities automatically. Accordingly, there is a need for the assistive devices to automatically recognize user activity within a predetermined duration threshold.
Technical solutions are described for providing torque assist at a human joint, such as the hip joint, using a personal biomechanical assistive device. The technical solutions described herein facilitate the assistive device to automatically detect an activity that a user is engaged in and provide corresponding amount of assistive torque.
Technical solutions described herein include a biomechanical assistive device that includes a motor control system that generates assistive torque based on a torque profile that is associated with a present activity being performed by a user. The device further includes an activity module that detects a new activity being initiated by the user, and computes a confidence index of the new activity. In response to the confidence index being above a threshold, the activity module switches the biomechanical assistive device to use a torque that is associated with the new activity.
According to one or more embodiments, a method for generating assistive torque by a biomechanical assistive device includes generating a first torque command for generating the assistive torque using a motor control system of the assistive device according to a present activity being performed by a user that is wearing the biomechanical assistive device. The method further includes detecting a new activity based on one or more sensor signals, and computing a confidence index of the new activity being performed. The method further includes, in response to the confidence index of the new activity being greater than or equal to a predetermined threshold, generating a second torque command for generating the assistive torque according to the new activity.
According to one or more embodiments, a computer program product for determining a present state of a device includes computer readable storage medium with computer executable instructions therein. The computer executable instructions cause a processing circuit to perform receiving a plurality of confidence indices, each confidence index is associated with a respective state of the device has been detected. The processing circuit further performs arbitrating the received confidence indices to determine a new state, the arbitrating performed based on a present state and the new state, and transitioning the device to the new state.
These and other advantages and features will become more apparent from the following description taken in conjunction with the drawings.
An exoskeleton, particularly, an active exoskeleton is a biomechanical assistive device that provides torque assist at a human joint, such as the hip joint. A technical challenge exists for the assistive device to recognize the user's current activity (walking, standing, sitting, etc.) so that appropriate amount of assist torque can be generated and provided to the joint/user. Typically, the assistive devices either require a direct user input, such as buttons, to indicate the activity that is about to be performed, or is being performed.
The technical solutions described herein address such technical challenges by automatically detecting the activity being performed, or that is about to be started. A technical challenge with automatic recognition of the user activity is the time taken to identify the activity. If the activity recognition takes longer than a predetermined duration the user is not comfortable using the assistive device. Typical solutions to mitigate the time taken for the activity recognition is using more sensors, often in conjunction with user interaction. However, such solutions drive up the product cost as well as weight and complexity of the assistive device, and even reduce intuitiveness of using the assistive device in case direct input identifying an activity is received/requested from the user. Further, because of the delay in activity recognition, the assistive devices include devices to provide feedback to the user that the activity has been recognized, for example haptic feedback devices. Such constant user input complicates and may slow down the device, and waiting for the device to detect modes and provide feedback is slow and feels unnatural.
The technical solutions described herein address such technical challenges by facilitating the assistive device to intuitively recognize the user activity using a minimal number of sensors to recognize the current user activity and within a predetermined duration, such as 1 second. Thus, the user has no substantial waiting time for the activity recognition, and no feedback is required. Further, the technical solutions described herein reduce cost and weight of the assistive device. In addition, the user-experience of the assistive device is improved by making it intuitive and practically immediate.
The biomechanical assistive device typically operates as a (or using a) finite state machine. Typically, each activity may be considered a ‘state’ of the state machine and determining when to transition from one activity (state) to another is defined by the state machine. A finite state machine is broadly defined as a system with a finite number of discrete states, where each state has criteria to transition to one or more other states of the state machine.
A technical challenge for a state machine with numerous states and numerous transitions, is that the design and implementation of the state machine becomes difficult to maintain. For example, a state-flow diagram becomes difficult to understand, and it is further difficult to map the design elements to the implementation of the state machine. Further, if there is substantial redundancy in the transition criteria, optimizing the implementation can become difficult.
The technical solutions described herein address such technical challenges with state machine implementation and optimization. The technical solutions described herein use a confidence-based state machine that separates the (redundant) transition criteria processing from the transition arbitration logic. In one or more examples, the confidence-based state machine uses a transition table. Additional features and advantages of the technical solutions are described herein. It should be noted that the technical solutions of implementing and optimizing a state machine are described herein in the context of a biomechanical assistive device, however, the technical solutions are applicable in any other context where a finite state machine is used.
The technical solutions described herein use embodiments directed to a hip-joint assistive device, however it will be appreciated that the technical solutions can be implemented in assistive devices used at any other joint, limb, or extremity in a body such as the ankle, knee, or hip joint of a leg or the wrist, elbow, or shoulder joint of an arm. Also, the user can be a human or an animal. Additionally, for ease of explanation, the term “limb” may be used to describe a limb segment (such as a lower leg or an upper arm) attached to a joint of a limb.
It should be noted that although the technical solutions described herein use embodiments in the context of particular biomechanical assistive devices, the technical solutions can be used in other devices that use a state machine, such as in an electric power steering (EPS) systems for signal arbitration (position, torque, speed, etc.), in an EPS for loss of assist mitigation and arbitration. The technical solutions described herein can also be used in an automotive for collision avoidance for autonomous and semi-autonomous vehicles, or for calculating a safest path to pass a vehicle in front. Alternatively, or in addition, the technical solutions described herein are applicable in an EPS, such as a Steer by wire system for initialization process (checking clutch, hand wheel, road wheel sensors etc.), or other diagnostics to be performed. The above is a non-limiting, exemplary list of applications for the technical solutions herein.
Referring to
The lumbar support apparatus 21 is configured as a torso brace that interfaces with the user 12. The lumbar support apparatus 21 is disposed about a user's waist proximate a user's hip region. The lumbar support apparatus 21 is configured to adjust overall human-exoskeleton interface stiffness through the use of various lumbar support types. The various lumbar support types permit the user 12 to adjust for comfort and load or torque transfer efficiency from the powered exoskeleton 10 to the user 12. The device 10 further includes a controller 200. It should be noted that the depicted exoskeleton 10 is an example and that the technical solutions described herein are applicable to other types of biomechanical assistive devices too.
The I/O devices 240, 245 may further include devices that communicate both inputs and outputs, for instance disk and tape storage, a network interface card (NIC) or modulator/demodulator (for accessing other files, devices, systems, or a network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, and the like.
The processor 205 is a hardware device for executing hardware instructions or software, particularly those stored in memory 210. The processor 205 may be a custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the system 200, a semiconductor based microprocessor (in the form of a microchip or chip set), a macroprocessor, or other device for executing instructions. The processor 205 includes a cache 270, which may include, but is not limited to, an instruction cache to speed up executable instruction fetch, a data cache to speed up data fetch and store, and a translation lookaside buffer (TLB) used to speed up virtual-to-physical address translation for both executable instructions and data. The cache 270 may be organized as a hierarchy of more cache levels (L1, L2, and so on.).
The memory 210 may include one or combinations of volatile memory elements (for example, random access memory, RAM, such as DRAM, SRAM, SDRAM) and nonvolatile memory elements (for example, ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), tape, compact disc read only memory (CD-ROM), disk, diskette, cartridge, cassette or the like). Moreover, the memory 210 may incorporate electronic, magnetic, optical, or other types of storage media. Note that the memory 210 may have a distributed architecture, where various components are situated remote from one another but may be accessed by the processor 205.
The instructions in memory 210 may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions. In the example of
Additional data, including, for example, instructions for the processor 205 or other retrievable information, may be stored in storage 220, which may be a storage device such as a hard disk drive or solid state drive. The stored instructions in memory 210 or in storage 220 may include those enabling the processor to execute one or more aspects of the systems and methods described herein.
The system 200 may further include a display controller 225 coupled to a user interface or display 230. In some embodiments, the display 230 may be an LCD screen. In other embodiments, the display 230 may include a plurality of LED status lights. In some embodiments, the system 200 may further include a network interface 260 for coupling to a network 265. The network 265 may be an IP-based network for communication between the system 200 and an external server, client and the like via a broadband connection. In an embodiment, the network 265 may be a satellite network. The network 265 transmits and receives data between the system 200 and external systems. In some embodiments, the network 265 may be a managed IP network administered by a service provider. The network 265 may be implemented in a wireless fashion, for example, using wireless protocols and technologies, such as WiFi, WiMax, satellite, or any other. The network 265 may also be a packet-switched network such as a local area network, wide area network, metropolitan area network, the Internet, or other similar type of network environment. The network 265 may be a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WAN) a personal area network (PAN), a virtual private network (VPN), intranet or other suitable network system and may include equipment for receiving and transmitting signals.
In one or more examples, using only two position sensors (one for each hip position), the technical solutions described herein facilitates the assistive device to recognize a new activity of a user with no additional user input and transition to a torque profile for the new activity within the predetermined duration. For example, the assistive device identifies different activities of the user such as sitting, standing, sit-to-stand, stand-to-sit, and walking, and other such activities, and facilitates near real-time transition from one activity (present activity) to another activity (new activity) that the user began without any explicit input from the user identifying the new activity. The technical solutions described herein thus facilitate an intuitive operation of the assistive device for the user, in turn improving the performance of the assistive device.
Each activity has separate criteria to be recognized, some of which may overlap. This includes thresholds and ranges for leg position and velocity, and their relationship to each other. For example, Sitting is detected when both legs are together and neither leg is moving, and Walking is detected when legs move in opposite directions. All activity criteria are evaluated in parallel; when an activity is recognized, a corresponding timer for that activity begins counting up. When the activity is no longer recognized, the corresponding timer resets. The values of these timers are passed to an arbitration unit 320 to determine the recognized activity as a final state. The recognized activity may be the new activity that the user is about to begin or the present activity that the user continues to perform.
The arbitration unit 320 determines the recognized activity based on the present activity and the time for each activity that the respective activity recognition modules 312 provide. Each potential transition has a timing threshold; that is, once an activity-recognition timer of an activity recognition module 312 passes a unique threshold corresponding to the present activity, the arbitration unit 320 outputs the activity corresponding to the activity recognition module 312 as the recognized activity. This results in a two-dimensional transition table implementation, which can expand as other activities are added.
The assistive device 10 associates a “confidence index” to each state (i.e. activity) that is part of the finite state machine in which the assistive device 10 operates. The confidence index of a state represents an entry criteria for the particular state, each state having its respective confidence index. In one or more examples, the confidence index can be realized with one or more parameters (e.g. time). In one or more examples, all confidence indices in the assistive device 10 have the same ‘meaning’ relative to each other although they can each be calculated in completely different ways. Typically, a confidence index increases as the criteria are more “fully” met, or over time as the criteria are continuously met.
The confidence indices are calculated independently from one another. This means that there can be overlap between the criteria (i.e. two Confidence Indices may increase in parallel), and changes in the criteria for one state may or may not impact any other state. In one or more examples, the arbitration module 320 uses only the confidence indices and the current state of the state machine (i.e. assistive device 10) for its operation to determine a state transition. The configuration of the arbitration module 320, in one or more examples, is represented in a transition table.
The technical solutions described herein, accordingly, split the implementation of the state machine of the assistive device 10 into two stages—a first stage where entry criteria for each state is defined, resulting in a plurality of confidence indices; and a second stage where an arbitration determines a next state (activity) that a user wants to transition into based on the previously detected state, and the continuously monitored confidence indices. In an example, the threshold confidence indices that are stored in the transition table 450 are dynamically updated.
As described herein, the arbitration module 320 accesses the values corresponding to a present state that the assistive device 10 is in and determines the next state to transition into based on the confidence index thresholds in the transition table 450. If a specific row-column pair does not have a valid confidence index threshold value, the assistive device 10 cannot transition from the first activity (row) to the second activity (column) corresponding to the pair. For example, an invalid entry may be an ‘infinity’ value stored in the entry in the transition table 450. Else, if a valid confidence index threshold value is stored in the entry, and once the state's confidence index meets or exceeds the table value, a state transition occurs.
The arbitration unit thus detects transitions (e.g. Standing to Walking), which are more probable, faster than less probable transitions (e.g. Sitting to Walking). By allowing the assistive device 10 to detect the less probable transitions increases the robustness of the assistive device, as any incorrectly detected states do not result in a permanent error condition.
Accordingly, the technical solutions described herein facilitate activity recognition using a minimal number of sensors, which can be expanded with further activities and transitions. The arbitration unit 320 facilitates fast, intuitive activity switching (e.g. less than 250 milliseconds for transitions).
The recognized activity is used to determine a torque command to be provided to the motor system 14. Alternatively, or in addition, the recognized activity is used to determine a motor velocity command to be provided to the motor system 14. The motor system 14 uses the input commands to operate the one or more motors of the assistive device 10 to generate a corresponding amount of torque and/or displacement of the motor to provide the assist to the user.
In one or more examples, the assistive device 10 uses a torque profile particular to the activity being performed by the user. For example, the assistive device 10 uses a walking torque profile to generate the assistive torque when the user is performing ‘walking’, a sitting torque profile when the user is performing ‘sitting’ and so on. Each torque profile may include one or more gain factors that are specific to the activity to generate different assistive torque for the specific activities. In such cases, the controller 200 selects the specific torque profile for generating the torque command/velocity command for the motor system 14 based on the recognized activity determined by the activity module 300. If the present activity is the same as the recognized activity the existing torque profile in use is continued, else the controller 200 switches the torque profile from that of the present activity to the torque profile of the recognized activity.
The comparator module 322 further receives as input 323 a subset of threshold times from the transition table, for example, a row of the transition table 400, based on the present activity for which the assistive device 10 is assisting. The input 323 includes the predetermined threshold times for determining a new activity based on the present activity. The selector module 321 identifies and selects the subset of threshold times as the input 323 to compare with the input 324 based on the present activity (block 620). In one or more examples, the present activity is input to the selector 322 as a delayed output of the arbitration module 320 itself using a delay unit 325. In one or more examples, the selected input 323 is one row of threshold times from the transition table 400, the row corresponding to the present activity. For example, referring to the exemplary transition table 400 depicted in
The comparator 322 compares the corresponding values from the input 323 and the input 324 with each other to determine the recognized activity (block 630). For example, the comparator 322 compares the activity-i time in the input 324 (from the activity recognition modules) with the activity-i threshold in the input 323 (from the transition table 400).
If the comparator 322 detects that the activity-i time has passed the activity threshold time corresponding to the present activity, the comparator indicates the activity-i as the recognized activity (blocks 640 and 650). For example, consider the example where the present activity is standing. In this case the activity recognition times are compared with the threshold times provided in the corresponding row in the transition table 400. If the activity recognition time for the activity ‘standing to sitting’ has crossed 50 ms, the arbitration unit 320 outputs ‘standing to sitting’ as the recognized activity, and the assistance device 10 generates torque command/switches torque profile accordingly. Alternatively, if the ‘standing to sitting’ has not yet crossed the threshold, and instead if the ‘sitting’ activity has been detected for at least the last 1000 ms (more than corresponding threshold), the arbitration unit 320 outputs ‘sitting’ as the recognized activity.
The assistive device 10 selects the torque profile according to the output recognized activity (block 660). Using the torque profile can include generating one or more torque/motor commands for the motor control system 14 to generate the assistive torque corresponding to the recognized activity. If the recognized activity is different than the present activity, the controller 200 switches the torque profile from that of the present activity to the recognized activity.
The method is executed continuously in a loop so that a new activity that the user is about to initiate can be intuitively recognized by the assistive device 10, without any direct input from the user to indicate what activity is being started. Thus, the method facilitates the assistive device 10 to recognize a new activity that the user is about to begin from the present activity that the user is already performing. The recognition is performed with minimal delay (less than 250 milliseconds) and using minimal sensors (2 position sensors), driving the cost and weight of the assistive device lower than devices using additional sensors.
The condition check module 710 determines if a predetermined condition for a specific activity is met and in such a case, activates the timer 720. The timer measures a duration for which the condition check module 710 determines that the predetermined condition is being met by the one or more input signals. If the condition stops being true, the condition check module 710 disables the timer 720. In one or more examples, disabling the timer 720 can also reset the timer 720. The timer 720 thus indicates an amount of time for which the activity was recognized according to the condition check module 710. The condition checked is different for different activities.
The activity recognition times from the one or more different activity recognition modules are further used for detecting the transition from one activity to another without any additional input from the user and without any additional sensors than the two position sensors used for detecting the activities. Based on the transition, the assistive device 10 can switch a torque profile and/or generate a torque command for generating assistive torque for the new activity that the user is performing. The transition is within a predetermined threshold intuitive as the user simply can move from one activity to another, without communication with the assistive device or waiting for the assistive device to switch from one mode to another. The technical solutions described herein thus improve the performance of an assistive device.
Further yet, the technical solutions use additional sensors and user input devices to capture additional measurements. In one or more examples, the technical solutions described herein facilitate the assistive device to capture user specific information, such as the user height, weight, and other body measurements in addition to sensor measurements. In one or more examples, the assistive device captures parameter measurements and data specific to one or more clinical functions. The data captured corresponding to the one or more clinical functions is further provided for further analysis and reporting via one or more communication channels. In one or more examples, the data may be provided to an external analysis system.
While the technical solutions has been described in detail in connection with only a limited number of embodiments, it should be readily understood that the technical solutions are not limited to such disclosed embodiments. Rather, the technical solutions can be modified to incorporate any number of variations, alterations, substitutions or equivalent arrangements not heretofore described, but which are commensurate with the spirit and scope of the technical solutions. Additionally, while various embodiments of the technical solutions have been described, it is to be understood that aspects of the technical solutions may include only some of the described embodiments. Accordingly, the technical solutions are not to be seen as limited by the foregoing description.
This patent application claims priority to U.S. Provisional Patent Application Ser. No. 62/588,645, filed Nov. 20, 2017, which is incorporated herein by reference in its entirety.
Number | Date | Country | |
---|---|---|---|
62588645 | Nov 2017 | US |