Patent Application
20200129109
The Faculty of Heath Sciences at Western University, London, Ontario, https://www.uso.ca, defines a person's ability to move or be moved as their mobility which forms a major outcome of human health and decreased mobility often occurs within the contexts of injury such as brain concussion or chronic diseases.
Bunn et al. U.S. Pat. No. 7,999,857 filed Jul. 25, 2003 revealed a computerized system including an intelligent camera and plurality of sensors which system analyzes video data viewing a scene of the movement of a subject to determine the subject's movement mobility and emotional stress to recognize the impairment level of the subject.
Bunn et al. USPTO application 20060190419 filed Feb. 22, 2005 revealed an intelligent video surveillance fuzzy logic neural network computerized camera system which system analyzes video data viewing the movement of a subject to determine the subjects facial and physical condition.
Bunn et al. U.S. Pat. No. 7,988,647 filed Mar. 16, 2009 revealed a computerized system including a video camera which system analyzes the video data of the movement of a subject to determine abnormalities in the subject's movement mobility from which comparison to known norms of mobility movement the system determines the subject's condition in relationship those norms. The application notes that results from comparison of the subject's condition to known abnormalities for diseases and illnesses those results may be used to generate treatment regimes.
Bunn et al. USPTO application 20140024971 filed Jul. 17, 2013 revealed a computerized system including a video camera which system using an active logic engine analyzes the video data of the movement of a subject to assess the mobility of the subject to determine mobility abnormalities, impairments, and there deterioration in the subject's mobility condition. The application notes that results from comparison of that subject's condition to known abnormalities and deterioration for conditions of brain concussion and diseases may be used to generate treatment regimes that lead to the restoration of the subject's health and thereby curing the condition.
The present invention relates to the objective computer video analysis computer code implemented by a computer coded logic engine, implementing a uniquely objective analysis computer code of the Mobility Assessment Tracking Tool (MATT) systems and objective computerized methods of determining and assessing the mobility of a subject performing movements and actions specified by Tinetti and SCAT-5 test protocols, through the administration of said code for computer vision objective analysis applied to 3-D video derived multi joint skeletal representation of the subjects' moving body parts and of the subject's foot pressure when performing on a force balance platform with pressure sensors. MATT fuzzy logic computer machine learning performs administration of complex bio-mechanical objective analysis assessments of the subject's static and dynamic balance and locomotion of said parts. The subject's kinematics of movement are measured, assessed and monitored by said objective analysis that derives mobility values from which the said code determines the level of the subject's assessed mobility functioning compared to normative values for said movement. Measurements and assessments of the mobility of the subject follow the kinesiology definitions established by the Tinetti test of gait and balance for risk of falling and the SCAT-5 concussion test. The Tinetti and SCAT-5 tests are originally subjective tests while here in revelled, the MATT is an objective computerize Tinetti test and SCAT-5 test. Boundary parameters for said measured kinematics may be adjusted as required, based on new and established best practices, for select populations including gender, age, athleticism, and disease or injury subgroups.
The present invention also relates to the objective computer video analysis computer code implemented by a computer coded logic engine, implementing a uniquely objective analysis computer code of the Mobility Assessment Tracking Tool (MATT) systems and objective computerized methods of determining and assessing the mobility of a subject through the administration of said code for computer vision objective analysis applied to 3-D video derived multi joint skeletal representation of the subjects' moving body parts and said code for a subject's foot pressure objective analysis applied to the pressure sensor data derived from a pressure force platform said sensors detecting the force of the subject's feet when performing movements on the platform. MATT fuzzy logic computer machine learning performs administration of complex bio-mechanical objective analysis assessments of the subject's static and dynamic balance and locomotion of said body parts and said foot pressure. The subject's kinematics of movement are measured, assessed and monitored by said objective analysis that derives mobility values from which the said code determines the level of the subject's assessed mobility functioning compared to normative values for said movement. Measurements and assessments of the mobility of the subject follow the kinesiology definitions established by the Tinetti test of gait and balance for risk of falling and by the SCAT-5 concussion test. The Tinetti and SCAT-5 tests are originally subjective tests while here in revelled, the MATT modifies these test to be objective, reproducible, computerize tests. Boundary parameters for said measured mobility assessment may be adjusted as required, based on new and established best practices, for select populations including gender, age, athleticism, and disease or injury subgroups.
The MATT is designed to save time for both administrators and health care professionals. Mobility assessments results provide gross overall values, scores and detailed results of a subject's performance of movements. Provided in a readily accessible test format, results can quickly and easily be recorded within computer database frameworks based on kinesiology practice and protocol, and output in printable standardized kinesiology report formats of the numerical and textural mobility assessment results and recommendations.
In general terms, the present invention provides a system, the Mobility Assessment Tracking Tool (MATT), for objective computerized analysis assessing the mobility of a subject, said system comprising: two or more motion sensors to observe movement of a subject performing 8 simple movements and to generate and record a 3-D video digital data stream representative of such movements. An active logic engine administering computer vision technologies including machine vision and machine learning functions apply MATT objective computer code implemented by a said logic engine analysis to determine and record from the video a multi-nodal skeleton representation of the physical joints of a subjects' body parts movement for each video frame to frame of the moving subject which representation is isolated from the stationary background. A set of fuzzy logic computerized code instructs said active logic engine to apply machine vision objective code such that for each measurement, each with one or more adjustable parameters, can be administered for interpreting the kinesiology defined kinematics of each body part movement within which the movement can be determined by the said engine administering machine learning logic of the objective code, the measure of level of function of the movement to be lying within or lying outside of normative range of values of specific features' values of the movements. The administration of further objective code by the engine to these features' values by which the said further objective code can determine the mobility assessment of the subject. Further additional objective code provides automated output of assessment results as a readily accessible text format for standardized reports in numerical and text interpretations of the assessment.
Through computerized automation the MATT provides consistent, reliable and reproducible mobility assessment results across testers administering the MATT tests of a subject's mobility. Every subject receives identical computerized verbal and video instructions each time they perform the assessment test thereby eliminating tester and intra-tester reliability as a source of error. Verbal instructions can be provided in a variety of languages to suit the subject being assessed.
The MATT is designed to save time for both administrators and health care professional. Assessment results provide both gross overall mobility scores and detailed results of the subject's performance of test movements. Here-to-fore such assessments have been made in a subjective assessment by kinesiology professionals. The MATT provides these assessments in a computerized objective, repeatable and reliable system utilizing objective analysis determining said assessments. Provided in a readily accessible text and numerical formats, the results can be recorded within internal system frameworks, based on practice setting. These formats provide to mobility practitioners, automation efficiency reducing their time to make and report a subject's mobility assessment while increasing the consistency of those assessments.
The MATT tool provides to mobility practitioners, a computer-automated, objective, reproducible and reliable assessment in keeping with kinesiology standards of measurement of the mobility of a subject and potential related medical conditions and remedial procedures, providing the critical information the practitioner needs for diagnosis of the subject's condition, injury, illness, or affliction and the treatments that may be needed.
In a further aspect, the invention provides a method of assessing mobility of a subject comprising the steps of recording motion of said subject, administering fuzzy logic machine vision and machine learning computer code applied with said active logic engine, on said motions to determine kinematic assessments of mobility and for determination of abnormalities of such movement, determining relationships of said abnormalities to known normative values, and determining whether said abnormalities are within a known norm or range of known norms.
In a further aspect, the invention provides methods and systems of administering an allocator on said active logic engine to determine if said abnormalities are within a known normative value or range of known normative values whereby to determine the possible existence of bio mechanical or neurological conditions or injuries of the subject and to determine at what stage are the said conditions or injuries. The invention further can infer from these determined conditions or injuries what are their relationships to known kinesiology rehabilitation procedures and treatments for such conditions or injuries, and can determine the potentially appropriate rehabilitation procedures recommended by the said methods and systems to relieve, repair or restore the subject's health and potentially cure the said conditions or injuries.
From the above, it will be clear the determination of mobility impairment will include the deterioration of the walking gait of a subject. It has been shown by extensive studies that the deterioration in mobility, including gait, of a subject has been directly correlated to neurological deterioration of the subject. Dr. Dean M. Wingerchuk at the Mayo Clinic in Rochester Minn. has reported “Gait analysis adds objective, reliable outcome measures sensitive to detecting neurological deterioration.” Neurological deterioration can also be caused by brain concussion for which the SCAT-5 test was specifically designed to detect. Dr. Wingerchuk states that “Gradual deterioration in ambulatory function is one of the major manifestations of progressive forms of Multiple Sclerosis”. At the Alzheimer's Association International Conference 2012 in Vancouver, Canada, three independent research studies each surveying more than 1,000 people, all confirmed mobility deterioration in gait of subjects directly reflected their neurological deterioration due to their Alzheimer's dementia. The studies were conducted by Dr. Stephanie A. Bridgenbaugh of the Basel Mobility Center in Basel, Switzerland; Dr. Mohammad Ikram at Erasmus MC Rotterdam, the Netherlands; and Dr. Rodolfo Savica of the Mayo Clinic Study of Aging, Rochester Minn.
From the above, it will be clear the assessment methods and system means described could be applied to the determination of mobility impairment including the deterioration of the walking gait of a subject to determine the potential existence of brain related illnesses including but not limited to Multiple Sclerosis and Alzheimer's dementia and brain concussion.
In this example, the expert system administers the said objective computer code by the active logic engine to the data available to identify that a mobility impairment condition exists in one or more movements in the current assessment and accesses a data base to determine relationships of this mobility impairment condition to a previous assessment for this subject, stored in the database component of this system, to determine if this mobility impairment condition was detected in a previous assessment. If the mobility impairment condition did so exist, the computer system, administering time derivative determinations, calculates the rate of change in the mobility impairment condition between successive assessments for this subject. The computer facility, using a predetermined baseline matrix of outcomes, then determines if a critical mobility impairment condition exists and, comparing to previous assessments, determines if a deterioration in the mobility impairment condition has occurred, and if so occurring computes the rate of change of this deterioration. This said objective computer code of the active logic engine of the MATT computer system can be applied to the assessment of brain concussions and conditions of the subject as is discussed herein.
Embodiments of the invention will now be described by way of example only with reference to the accompanying drawings in which:
Prior to describing the system and its function in assessing mobility of a subject from observing the subject performing 8 specific movements: sit still in a chair; arise from a chair; stand still; stand still with eyes closed; walk in a straight line path; turn 360 degrees walking in a circle; and turn 360 degrees walking on-the-spot. A number of the typical assessment environments will be described to provide context to the operation of the system.
Referring to
In
As an example, say the subject takes two attempts to rise from the chair (101, dotted lines). The cameras (103 &104) capture the movement of the subject (101) in a time dependant manner and the data are transferred to the computer (107). As will be described more fully below, the expert system administers the new and uniquely developed mobility assessment movement said objective computer code instruction to the said logic engine being revealed herein, which will be referred to as the mobility code to determine normality or abnormality of the movement according to kinesiology standards of movement as derived from the 13 specific features' and applies this information and additional input to provide the criteria required to apply standardized kinesiology test criteria and test parameters. In the example provided, the two attempts to rise are determined as an abnormal mobility condition and these determinations indicate that the subject has a significant limited level of compliance to the kinesiology standards for that movement and defining the subject's impairment condition for that movement.
Continuing with this
Eight kinesiology accepted movements have been selected that are used to observe and assess mobility, the occurrence of mobility impairment and conditions of a subject and the subject's potential of having related injury, illness, pain or disease for that subject being assessed.
Normal for a specific subject means movement that has been previously observed and recorded in databases for this subject and is accepted as a base level of compliance to the kinesiology standards for that movement. Standards for that movement can be defined as movements that have been observed and recorded in databases of typical movements for subjects of similar age, sex, health, and mobility and is accepted as a base level of compliance to the kinesiology standards for that movement for any similar subject.
The mobility codes revealed in this invention are administered by the active logic engine expert system, to the input video data streams from a multiplicity cameras to derive the skeleton nodal data streams and to derived specific features' values data streams, said additional mobility codes functioning as an administrator, to conduct detection determinations, and specific features' extraction from the nodal data stream administrations, from which to assess the likelihood of limited level of compliance to the kinesiology standards for that movement for a subject. This is accomplished by administering the mobility codes by the active logic engine to video data, to develop for each frame of the video data stream a computerized frame by frame skeleton nodal data stream representation of the subjects' body including multiple control joints such as: head, neck, shoulders, elbows, wrists, hands, torso, hips, knees, ankles and feet. Further mobility codes are administered to each skeleton nodal representation for each frame to determine a measurement of specific features' values of the movements of each joint relative to their location in the previous frame. Additional mobility codes are administered to each measurement to determine metric amount of that joint's movement where by the mobility codes can determine the bio mechanical movement of the subject's body at each joint. For example specific features' movements values such as for feet movements: step length, height of moving foot off the floor, separation between feet, step frequency can be determined. Another example for arm specific features” movements relative to: shoulder, elbow, wrist and torso joints, the angle of the upper arm and lower arm relative to the position of the torso can be determined from the angles formed by the wrist-elbow-shoulder joints. Not all such movement examples will be discussed here but it will be clear to any one informed in bio mechanics that with sufficient control joints, most bio mechanical body movements can be determined.
These specific features' values as determined by administration of the mobility codes described above, also produce electronic or mathematical signatures of said movements such that administration of additional mobility codes can derive from these movements, an allocator value to determine whether the values of said signatures are within known norms of the movement of personal, and/or, normal range level of compliance to the kinesiology standards for that movement and deviations there from for features' movement of normal subjects which provide features' signatures of movement are stored in the system in related databases. Then, deriving similar signatures of subjects to be assessed as to mobility performance of the movements, the active logic engine mobility codes determine the deviation of these signatures from the normal signatures to make the decisions as to infer limited level of compliance to the kinesiology standards for that movement. If limited level is interpreted, the mobility codes then determine whether the movement indicates a bio mechanical or neurological injury, pain, or illness and if so indicated, it informs the appropriate health care personnel or systems. Similarly, determinations of the deviation of subject's movements could result from medical emergencies such as heart attack, or seizure that such emergencies also require healthcare personnel assessment in responding to the subject in question for which appropriate medical actions can be taken.
The administration of the mobility codes of the system using the active logic engine, can implement unique determinations and subsequent reporting assessment results for mobility level of compliance to the kinesiology standards for that movement. These reports can be in readily accessible text format that can be cut and pasted into internal and external standardized reports based on kinesiology practice. Later, such observations of the subject will determine the changes in the subject's movement as it correlates to their earlier determinations and in real time determine any deviations that could relate to mobility reduced level of compliance to the kinesiology standards for that movement and possible existence of injury, pain or medical health condition as determined by the active logic engine mobility codes. However, if the mobility codes administration system through access to related databases has access to medical and health information and database of related mobility impairment signatures of the subject, the active logic engine processor may be able to determine if the subject being observed is in fact having a health problem such as heart attack, stroke, diabetic coma, epileptic seizure or brain related diseases such as Multiple Sclerosis, Parkinson's, Dementia, Cerebral Palsy, or brain concussion, and any of which could be needing immediate medical assistance and if so determined, can inform the proper health care providers.
In the case for that a subject is determined to have a reduced level of compliance to the kinesiology standards for a movement, for example as a stagger back shown in
Further, a significant foot placement specific features' test while walking is to request the subject to walk toe-to-heal such that the subject places each foot at each step so that the heal of the front foot touches the toe of the back foot. This is a more difficult and perhaps stressful walking task for the subject and the mobility assessment of the subject's movement can determine more subtle effects of and existence of bio mechanical or neurological problem. Further, an even more difficult walking task is to request the subject to walk either regular walk or toe-to-heal walk but with the moving foot to cross over the stationary foot such that the subject's feet when both are stationary are crossed at every step in the walk. Mobility assessment of the subject, under the stress in this task, can determine even more subtle effects of and existence of bio mechanical or neurological problems. It will be obvious to anyone verse in bio mechanics, that many more movements will be applicable for administration of the mobility codes revealed herein for mobility assessment, however for brevity are not detailed here.
The above examples relate to an assessment performed in a controlled environment by a medical practitioner, tester or operator. The MATT system incorporates computerized voice instructions for each movement the subject is requested to perform thereby providing consistent reproducible test procedures. The expert system may also be used in a normal non-clinical environment as a continuous, non-invasive mobility assessment tool, such as a mobile computer and cameras system implemented near an athletic playing field to provide quick on-sight assessment of athletes before, during or after play. Particularly if a player is suspected of having suffered a hit, shaking or injury to the body during play, a prompt mobility assessment at the time of such occurrence could be critical in assessment for potential bio mechanical or neurological problem and the expert system mobility codes could be administered to alter health providers and practitioners such that immediate action for medical attention can be taken as needed.
The Sport Concussion Assessment Tool, now in version #5, BJSM Online First Apr. 26, 2017-097506SCAT5, is an established, professional kinesiology subjective test for the assessment of the mobility of sports participants expected to have suffered such a hit to the body that may have produce the potential bio mechanical or neurological problem resulting from brain concussion. Herein we reveal a purely objective computerized implementation of SCAT5 using an instrumented force balance board. One such board is the WiiBoard balance board for the game console Wii Fit by Nintendo, Wii balance board referenced in Wikipedia. The MATT has integrated pressure data from the WiiBoard data into the MATT system for collection of pressure data from the pressure sensors located on the four corners of the board. Additional mobility codes of the system using the active logic engine are implemented for accessing the Wii data and integrating these data into the MATT system.
Further additional mobility codes of the expert system analyze these data to determine the balance and foot pressure on the board by the subject preforming the movements required for the SCAT-5 test. The MATT system expert system analysis mobility codes also records the video of the subject's movements on the Wii board. Additional mobility codes of the system using the active logic engine are implemented to integrate and analyze the Wii board balance and pressure data. The MATT expert system also records the video of the movements of the subject conducting the required SCAT-5 movements on the balance board including but not limited to: standing on both feet, standing on one foot, standing extending lifted leg, standing toe to heal.
The MATT system additional mobility codes using the active logic engine extracts and records the following four specific features from the analysis of the Wii board data. The sagittal sway which is defined as the sum of the weight ratio of adjacent left and adjacent right Wii board sensor plates which are separated by the sagittal plane. The coronal sway which is defined as the sum of the weight ratios of the anterior and posterior sensor plates which are separated by the coronal plane. The diagonal sway which is defined as the sum of the weight ration of the sensor plates on each diagonal of the Wii board. And the center of pressure, COP, which is defined as the center of the pressure distribution over all sensor plates on the Wii board.
The measured values from the Wii board sensor data for these four specific features are analyzed by the additional mobility codes using the active logic engine processing as illustrated in
Stepping and stumbling are analyzed from the Wii board data by computation of the rotational transformation about the center of pressure. A threshold value establishes the boundary values by MATT based on established clinical data to determine the boundary between stepping-stumbling and heel-to-toe lift as illustrated in
The center of pressure angle can be calculated as:
The criteria used for scoring in the balance testing related to the Wii board are:
Heel—Toe Lift: which we defined as a regime of angular positions from the patient stationary reference frame, which has the Wii board coordinate system as it's inertial reference (See
Step Stumble—which we defined as a regime of angular positions between Heel-Toe (See
Duration: The period of time for which the peak of signal which exceeds a sway or center of pressure threshold.
The values in conjunction with the video stream data are used to satisfy the objectives of the SCAT-5 Balance testing. Deductions from the Wii board data are assigned based on duration of loss of balance (stance), and type of balance loss; step, stumble, or heel-toe lift (See
For each of the stance positions (See
The three nominal test stances as defined by the SCAT-5 methodology (See
Double Leg stance uses the Sagittal, and coronal sway data to search for peaks, and the nature of score deduction is determined by observing the center of pressure angle during the interval for which there was a peak greater than the threshold values for balance loss;
Single Leg stance uses the Sagittal, and coronal sway data to search for peaks, and the nature of score deduction is determined by observing the center of pressure angle during the interval for which there was a peak greater than the threshold values for balance loss; and
Tandem stance uses the diagonal sway data to search for peaks, and the nature of score deduction is determined by observing the center of pressure angle during the interval for which there was a peak greater than the threshold values for balance loss. In this test, the patient reference coordinate frame is not parallel with the inertial reference frame.
MATT also incorporates the mobility assessment values from the video and skeleton data streams observations and mobility assessment of the subject's movements on the Wii board, to determine the nature of the SCAT-5 balance test score. MATT further establishes the boundary values based on the established clinical data to determine the boundary thresholds for the distance of the wrist from Iliac crest and for the angular raise of the wrist from established values for nominal stance as defined in the SCAT-5 test for deductions associated with hand lift from Iliac crest and hop abduction.
The MATT expert system mobility codes determining the subjects stance and balance for these video data mobility assessments further determines the SCAT-5 test scoring. Vocal response and vision response SCAT-5 tests are administered by additional expert system mobility codes from which the expert system determines the subjects scoring of cognitive SCAT-5 data tests.
The MATT expert system mobility codes integrate the scoring results from the video data assessments and from the cognitive data assessments from which to objectively determine the assessment of the potential that the subject suffered a brain concussion. The balance and foot pressure analyses are integrated with the MATT mobility video analyses, assessments for the participant subject, from which analyses MATT expert system mobility codes derive the unique determinations and subsequent assessments results for the subject's mobility level in compliance to the kinesiology standards for the SCAT-5 test.
The implementation of the expert system can be considered as having two main linked components: a basic mobility assessment system and an advanced mobility assessment system. The basic system permits an operator to control part or all of the assessment process and to input assessments of the mobility of the subject being assessed. The advanced system contains the mobility codes and computer facility active logic engine neural networks decision computations with which the expert system determines the assessment outcomes and recommendations according to established parameters, the mobility assessment total score number, and the differential determination of current assessment to previous assessments, and generates reports of remedial actions, possible aids and healthcare procedures, to the subject, or to the subject's employers or to the caregivers of the subject.
Further, the expert system may be administered using a limited number of skeleton nodal control points such as head, shoulders, trunk, elbows, wrists, hands for monitoring larger arm movements. Alternatively the expert system could also use a larger number of control points including the above plus thumbs, fingers, knuckles for refined higher resolution of movements such as for observing shaking of hands that could be typical of diseases such as Parkinson's.
The advanced system can compute a larger number of skeleton nodal control points and related selected specific features' values assessed than does the basic assessment system, for each video frame. Using known video skeleton nodal control point to create additional points, the advanced system can then derive additional specific extracted features' with which to detect the finer more precise subject's movement of each control point from frame to frame based on the displacement of each control point on a given frame relative to the same control point on the previous frame by differentiating between those two to determine the control points that are moving and those that are stationary on a frame to frame basis. Extracting said features applies to a subject's movements made while standing on the balance board and to the subject's movements made while not standing on the balance board.
Extracting said features may be performed by the MATT objective computerized analysis mobility codes determining a specific extracted feature such as whether a given skeleton nodal control point, E for example of an elbow movement, in the image frame, x, moves or is displaced by or more than say 3 video pixel spaces in any direction for this control point in its location in the next image frame, y. If so then this control point, E, in frame x is identified by the MATT objective computerized analysis mobility codes as moved and assigned pixel component location. If pixel E in frame x, moved less than 3 pixel spaces at its new location in frame y, then this control point, E, is identified as not moved and assigned the pixel components it had in frame x. By the MATT objective computerized analysis mobility codes computing the movement of all control points from frame x to their locations in frame y and assigning all those that move 3 or more spaces, with the new pixel locations where they appear in frame y and all those control points in frame x that move less than 3 pixel spaces to retain their pixel locations from frame x, a skeleton motion-rendition of the subject's movements wherein all movement of the subject can be observed and movement assessed by the MATT objective computerized analysis mobility codes. The number of pixels, for example here being 3 or more, is set by adjustable additional mobility codes pixel parameter by which the administration of the pixel movement MATT mobility codes determines the number of pixels moved. Additionally, administration of the MATT pixel movement mobility codes to the 3-D data stream components of the cameras, the MATT objective computerized analysis mobility codes can determine the physical distance of the skeleton nodal control joint movement from frame to frame where the distance of the movement is set by an adjustable additional mobility codes distance parameter input to the administration of the mobility codes.
The finer movement and measurements resulting from the higher number of skeleton nodal control points can be considered as a higher resolution detection skeleton nodal data stream and derived specific features' values representation which in this case is the subject being mobility level of compliance to the kinesiology standards for that movement assessed, and stores that skeleton nodal data steam and derived features' values representational data in a database. It is preferred that the mobility impairment detection mobility codes revealed herein are advances on and entirely new derivations of those stagger computer codes which only consider detection of the movement of the envelope shape of the entire body of a subject revealed in U.S. Pat. Nos. 7,988,647, and 7,999,857, and networking computer codes of U.S. patent application 20060190419 and determination of medical conditions by measuring mobility patent application 20100049095, and assessment and cure of brain concussion and medical conditions by determining mobility patent application 20140024971, the contents of which are incorporated herein by reference.
However the mobility codes revealed in this patent application are completely new. The mobility codes of this application objectively observe, measure and assess the movement of the individual body parts by measuring and tracking the movement of the skeleton nodes and assessing the pressure sensor observations of the subject performing movements while standing on and while not standing on a balance board, indicated earlier, for determinations of the assessment of mobility and mobility impairment and potential of brain concussion of the subject.
By using such techniques, it is possible to evaluate if a particular movement is indicative of a mobility level of compliance to the kinesiology standards for that movement and if an impairment condition exists from determining the movements of a subject. Each of these evaluations may be made from the specific extracted features' values derived from the skeleton nodal data stream of the motion by determining the average deviation of a set of specific features' data representing the body, for example determining the average location of the centreline of the subject relative to the normal path for that movement.
The mobility assessment mobility codes are administered to the real-time or recorded video data stream and to the associated skeleton nodal data for an objective determination of the 13 specific features' values measure of the movements by a subject according to the Tinetti mobility test requirements which are defined and accepted as following kinesiology standards and protocol: the Tinetti test defined the subjective assessments. The MATT system makes these assessments objective computerized measurements and assessments. The eight selected movements of the subject are: sit still in a chair, arise from sitting in a chair, stand still, stand still with eyes closed, sit down on a chair, walk in a straight path, turn 360 degrees walking in a circle and turn 360 degrees turning on-the-spot. With these eight, simple movements, the administration of the mobility assessment mobility codes of the MATT objectively extract 13 specific features' values measure of mobility parameters with which the mobility assessment determines if the measured numerical values of these features' are within the range of the thresholds set for each feature. Feature values lying outside these thresholds allow additional mobility codes to determine the mobility abnormalities these out-of-range features' and may further determine the possible conditions, illness, injury, pain, disease of the subject indicative of such abnormalities.
In an alternative embodiment images from multiple cameras may be used as shown schematically in
For data acquisition, the Kinect sensor samples at a frequency of approximately 30 Hz and video frames are captured both in color and depth. Using captured frames, the middleware of Kinect software SDK, on a frame basis segments the subject's human shape and imposes skeleton nodes on the shape providing in each frame the output of a human skeleton represented by 20 nodes, for the Kinect V-1 and 25 nodes for the V-2, as control points in the Kinect's own reference frame known as the skeleton space. Each node represents a specific joint with 3D position information in units of meters. The skeleton space uses a right-handed coordinate system: the Y axis lies in vertical direction of the image plane, the Z axis extends in depth perpendicularly from the sensor and the X axis is horizontal in the image plane and orthogonal to the Y and Z axes.
In pre-processing, the MATT subjective computerized analysis mobility codes compute the position and the speed of each joint node from frame to frame in the time sequence each of which are considered as one-dimensional signals. The MATT mobility codes apply two 2nd order low-pass Butterworth smoothing filters were used to reduce the noise in the signals. The MATT analysis mobility codes apply empirically-determined cut-off frequencies of 4 Hz and 1 Hz were used for the objectively determined position and speed signals of each joint, respectively.
To extract the features' of walking steps, it is necessary to accurately segment the steps, i.e. determine the start and the end of a step. The Z component (in depth) of foot speed is used because it showed good regularity in relation to the phases of the steps. The MATT mobility codes robustly segments the steps while ignoring the small peaks generated by the interference from parts of the body overlapping or the distance between the subject and the camera being too long.
MATT objective computerized analysis mobility codes determine the time series of the Z speeds of both feet during stepping. The most important features' are the start-, the mid- and the end-points. MATT objective computerized analysis mobility codes determine these as feature points and use them for analyzing the gait. The MATT mobility codes are insensitive to the tilt angle of the Kinect sensor since we they use the Z component (in depth) of foot speed for step segmentation.
MATT analysis mobility codes finding overlaps of the feet in the 360° Turn the analysis uses the same pre-processing step as the gait mobility codes. Since a subject is turning 360° on the spot, it is difficult to segment the steps using the method for the gait analysis. To measure the continuity of a turn, the MATT objective analysis mobility codes identify the skeleton frames in which the speeds of both feet are below a certain speed threshold. Specifically, the speed is defined by the MATT objective analysis mobility codes as the Euclidean norm of X and Z components of the speed of a foot. A group of consecutive skeleton frames below a certain speed threshold indicates that a subject may have paused during a 360° turn. The mobility codes identify pauses during the 360° turn based on a toe-off speed threshold of 0.2 m/s. The time interval of each pause is determined by the MATT objective analysis mobility codes as the difference in timestamp of the first skeleton frame and the last skeleton frame in a group.
Several trunk features' are determined by the MATT objective analysis mobility codes. The stability of the trunk of the body is monitored by two factors: the use of arms for balancing and the lean angle of the trunk in the coronal plane. Additionally, for MATT gait mobility codes, it is necessary for them to calculate the deviation of the base of the spine relative to the traveled path.
It is assumed that at the start of an assessment the subject is not using their arms for balancing and the wrists are placed at the sides of body as directed by the computerized voice instructions. In other words, the wrists are at their resting positions. The distance between wrists is defined by the MATT objective analysis mobility codes as the Euclidean norm of X and Y components of positions of two wrists. The Z component is ignored since the arms typically swing during walking.
During a walk or a 360° turn, when subjects use their arms for balancing or lean the trunk of their body, the distance between the wrists will increase. By MATT objective analysis mobility codes calculating the difference of the distance between the wrists at the resting positions and the distance of wrists during a walk or a 360° turn, the use of arms for balancing can be detected by the MATT objective analysis mobility codes. To illustrate the process, the algorithms mobility codes detect the changes of X distances of two wrists with respect to the origin of the Kinect to measure the interval in which a subject may use an arm for balancing.
The leaning angle of the trunk is defined by the MATT objective mobility codes as the angle between the vector of the trunk (between the center of shoulders and the spine base) and the gravitational vector in the coronal plane. The angle is obtained by the mobility codes calculating the mathematical dot product of these two vectors.
To measure the deviation from the path during a walk, a path vector P is calculated by the MATT objective analysis mobility codes using the position of spine base in the first frame and last frame in a walk. The instantaneous deviation from the path is defined by the MATT objective analysis mobility codes as the perpendicular distance between the position of the base of the spine and the straight line path along vector P.
There are 13 specific features' values extracted by the MATT objective mobility codes from the above measurements that further administration of mobility codes will determine for the Walk Gait assessments. For the gait mobility codes, of interest are the three feature points of each step: the start, the mid and the end. The MATT objective analysis mobility codes feature point contains the timestamp, the position and the speed of the moving foot. The gait specific features' values involved in the gait mobility codes are the following (with units in parentheses):
Some of these specific features' values associated with the movements of feet are illustrated in
The classification of normal and abnormal patterns of each gait feature of a subject is performed by setting thresholds for the features' values extracted from the recorded skeleton. To determine the thresholds of the features', data were captured from athletes with potential risk of concussion and a kinesiologist was asked to score the athletes by watching the pre-recorded videos using the software developed for the study. The scores given by the kinesiologist were used as the ground truth for determining the thresholds.
The mobility codes were designed using Matlab 2014a for data analysis and later were redesigned and coded in C++. Using Microsoft Visual Studio 2013, a desktop application was designed for performing experimental real-time assessments and further advancement of the designs has created the MATT as a tool for kinesiology professionals, practitioners and clinical testers to use as the new and validated mobility assessment tool. By way of example, the design methods will now be revealed herein.
In this example data were captured from 14 athlete subjects sample group by researchers in the department of Kinesiology at York University. Three athletes had a history of concussions, one had a suspected concussion and the rest were healthy controls. Informed consent was obtained from the participants in accordance with a protocol approved by the Human Participants Review Subcommittee at York University.
A Kinect V-1 sensor (camera) used was placed 0.84 m above the ground. For gait assessments, the athletes were asked to stand 3.8 m away from the camera, perform a straight line walk towards the camera and stop at 1.8 m away from the camera. For 360° turning assessments, the athletes were asked to stand at a position between 1.8 m and 3.8 m away from the camera and perform a 360° turn. To calibrate the system, the specific features' values to be extracted from the collected data were determined using the developed mobility codes, as shown in the table of
The approach taken to set the thresholds is to consider the 14-subject sample group representative of normal variation. Then, for each feature, the values limit selected is one that will enable all normal participants to pass the automated assessment since all 14 were passed by the kinesiologist's subjective assessment; the limit is normally the value that represents the worst case in a sub-assessment as shown in the table of
The primary task for any given subject sample group is to build a database that contains as many samples as possible from relevant clinical populations. When the number of samples is large enough and adequately covers normal and abnormal patterns of each gait feature, the accuracy of the determination and segmentation of normal and abnormal gait is improved and new thresholds and more advanced classification mobility codes can be determined. It will be clear to anyone with a kinesiology understanding that the MATT methods and mobility codes revealed in this patent disclosure, will allow the establishing of databases specialized for clinical populations having particular mobility issues such as related to specific injuries, illnesses, pain, diseases and conditions such as concussion, dementia, chronic pain, Parkinson's, and stroke. The database described by the above example was dealing with male and female subjects in age ranges of 18-25, who are athletic and who have a risk of suffering brain concussions. It will further be clear that due to the objectivity, reliability and reproducibility of the testing mobility of subjects with the MATT system and the mobility codes, that the results from repeated testing with the MATT of subjects will permit the tracking and monitoring over time, of a subject's particular condition and it's progression of improvement or lack of improvement during treatment being given the subject for that condition. The MATT could become as common and fundamental a medical professional tool as the blood pressure measuring tools found in almost every medical practitioner's office to track and monitor patient's heart and blood pressure cardio vascular condition.
In clinical tests of subjects with the MATT mobility assessment system conducted to date to test and validate the assessment methods and apparatus, it was found that the methods and apparatus were well received by the kinesiology professionals as functional and highly accepted as a unbiased, objective and reproducible tool providing valuable patient mobility information. For the linkage relationships determined between current and previous subject's assessments in evaluating the changes in mobility and mobility impairment and potential existence of concussion as well as and for illness, pain or disease curing, arresting or reversing effects of the illness, pain or disease the MATT was also recognized to be effective.
The system described above has the capability to determine relationships of a subject's present assessments to the subject's previous assessments whereby the expert system can determine and measure the changes in any of the actions and motions of the subject specifically tailored to the subject's individual conditions and health. The expert system not only has databases of information on what are considered normal movements and actions of persons depending on age, sex, health condition and drug use, but also has similar databases specific to the subject being assessed, and thus the expert system can also base-line calibrate its decision-making determinations to what are considered normal movements and actions of the subject being assessed. Determining the relationships to the subject's base-line the expert system can further determine if the present assessment is normal or if it indicates a mobility impairment condition and possible potential existence of injury such as concussion, illness, pain or disease. If the system determines that a mobility impairment condition exists, then the system can determine relationships of the present assessment to previous assessments for this subject to further determine changes in the mobility impairment conditions. Further, if video monitoring in areas where the subject moves about, such as in a residence, home, hospital, playing and sports fields, professional stadium and sports entertainment facilities or natural environments are implemented as the earlier discussion noted, the expert system can determine relationships of these data with which the system can determine the mobility impairment and changes in the subject's mobility in the subject's daily living environment from which the system can determine more comprehensive preventative and remedial practices, health and well-being programs, mobility aids, and monitoring programs for improved quality of life activities, work related activities, monitoring of rehabilitation programs and their success or failure or modifications specific for the subject.
In either real-time or post-recording, the MATT expert system can be the decision-making facility which permits the actual operation of the system and assessment to be done by regular staff of the subject's employer, or clinic, or athletic or sports facilities without the need for highly qualified and expensive professional personnel. This frees up the professional practitioners time by integrating the MATT results into the diagnosis of their patient's mobility and health condition. The apparatus and methods described above can also allow authorized personnel, such as professional physiotherapists, neurologists and concussion specialists to review this new source of mobility assessment data and the determinations made by the MATT system, and integrate this information into their diagnosis of their patients' conditions.
A new and unique embodiment of the expert system is revealed here, that for the first time provides a fully computerized automation implementation of the standard kinesiology mobility test fundamentals of the Tinetti test as a tool for the kinesiology professional which provides consistent, reproducible and reliable testing results across any and all testers. Every subject receives identical computer generated verbal and video instructions, each and every time the subject performs the assessment test. This eliminates inter-tester and intra-tester reliability errors. Instructions are in a variety of selectable languages suited to the subject's requirements.
Also, the MATT revealed herein, is designed to save time for both administrators and health care professionals. Assessment results are provided in both gross overall mobility scores and detailed results of the subject's performance of specified movements. Numerical and textual data are provided in readily accessible formats that can quickly and easily be stored and transferred within internal file format frameworks and exported to standardized spreadsheet and word processing formats based on kinesiology practice and setting.
Double blind clinical trails have been conducted at the York University Kinesiology and Health Department, Dr. Lauren Segio and Dr. Diana Gorbet. Testing during the trials was conducted on over 20 of the Canadian Women's and Men's handball team players from the 2015 Canada Summer Games several of whom were considered to likely have suffered injuries and some possible concussions. Also tested were over 70 university athletes considered as normal uninjured subjects. Tests included several standard kinesiology kinematic, cognitive, balance, gait, coordination, and vision tests to determine the physical and mental condition of the subjects.
The objective MATT gait and balance expert system's fully computerized tests described here in were administered to the video taken for each subject as they carry out the Tinetti gain and balance test movements. The expert system analyses of each video produced a Tinetti score for each subject. Also, three independent physiotherapists separately conducted the subjective Tinetti gait balance test scoring for each subject. The physiotherapists were required to make their personal subjective Tinetti test scoring assessments, and were only allowed to view the videos of the subjects movement but not allowed any access to the other double blind test data or results. Only the team of Drs. Sergio and Gorbet has access to all test data and results prior to their publication.
The early results from the clinical trials, indicates that the MATT Tinetti scoring and the independent physiotherapists Tinetti Scoring and the York University Kinesiology testing results are all in good agreement. Further the early results also indicate that the reliability, reproducibility, and consistency of the MATT assessments demonstrated that the variability in the physiotherapy personalized testing strongly supports the need for computerized expert gait and balance technology assessment of athletes, such as the MATT. Final results will be published 2018-19. Publication of results from the comparison of the York University Kinesiology subjective scoring of the SCAT-5 test and the MATT objective scoring of the SCAT-5 test will follow.
From the above it will be clear the assessment methods and apparatus of the MATT tool described could be applied to many environments, such as, hospitals, private homes, hotels, commercial establishments, doctor's offices, clinics, drugstores, mobility-aids stores, and in the broad sense anywhere people are moving about such as sports and athletic facilities, playing fields, gyms, employment facilities. Also it will be clear to anyone versed in the healthcare field that many different mobility codes, mobility codes test parameters, action scoring methods and determinations can be implemented, including, mobility impairment mobility codes, time derivative determinations and mobility testing, such as those we reveal as incorporated into the computer facility active logic engine neural networks decision determinations methods and apparatus with which we can assess mobility impairment and potential existence of injury, illness, pain or disease, the preventative outcomes and recommendations to reduce further mobility impairment and potential further injury, and for improved quality of life for assessed subjects. Further, it will also be clear that the methods and apparatus of the MATT tool, assessments and recommendations facilitated by the expert system can have application to any subject persons regardless of their age, health, sex, location or activity. Also, it will also be clear that the methods and apparatus, assessments and recommendations facilitated by the expert system can have application to assessment of and the tracking the progression of injury such as brain concussion, and the effects of treatments and rehabilitation regimes whether trials or long-term such as drugs, physiotherapy, nutrition, exercise, and success or failure of those treatments, and for other conditions such as diseases, illnesses, pains and injuries not limited to only those disclosed herein.
