Patent Application
20110054359
The invention relates generally to weight management devices, and more particularly to a footwear-based system for monitoring body weight, postural allocation, physical activity classification, and energy expenditure calculation, and providing feedback aimed at maintaining healthy levels of physical activity and weight management.
Many Americans and adults worldwide are overweight or obese. Obesity is due to a sustained positive energy balance, i.e., in which an individual's energy intake is greater than the individual's energy expenditure. Weight gain is typically coupled with low levels of physical activity and sedentary lifestyles. As a result, most weight management programs recommend regular monitoring of body weight and increased energy expenditure lifestyle alterations that increase physical activity levels.
While increasing energy expenditure can be achieved via exercise, there are other components of daily energy expenditure, such as non-exercise activity thermogenesis (NEAT), which also play an important role in weight management. NEAT includes the energy expenditure associated with posture allocation, for example, time spent lying, sitting and standing, as well as energy expenditure associated with non-exercise movement, for example, walking and other free-living movements. In fact, the energy expended each day via non-exercise movement and posture can be greater than that of a vigorous exercise session. Because the positive daily energy balance associated with weight gain may be relatively small, i.e., on the order of 25-100 kcal/day, relatively minor adjustments in daily activity patterns may promote weight loss and prevent weight gain.
While body weight can be monitored using an electronic scale, scales are not discreet and may not be used throughout the day by individuals who are sensitive about their weight. Devices that quantitatively monitor levels of physical activity, e.g., accelerometers have been shown to improve weight management outcomes. These devices are also able to more accurately estimate activity energy expenditure, as they can distinguish activities from stationary postures, which have different metabolic rates. However, current devices are incapable of differentiating between different postures, e.g., sitting and standing, and instead group these postures together. These devices further cannot differentiate between different activities, for example, cycling and ascending and descending stairs.
The information included in this Background section of the specification, including any references cited herein and any description or discussion thereof, is included for technical reference purposes only and is not to be regarded subject matter by which the scope of the invention is to be bound.
The wearable energy expenditure monitoring system disclosed herein assists users in losing weight and maintaining healthy level of physical activity by calculating body weight, allocating posture, classifying physical activity, and calculating energy expended by a user and providing feedback to the user based on the calculated energy expenditure. In one embodiment, the monitoring system may calculate the energy expended by the user based on a combination of acceleration data collected from an accelerometer and pressure data collected from a pressure sensor. The acceleration and pressure data may be transmitted to a processing device, which may provide periodic feedback to the user regarding his or her calculated energy expenditure. Accordingly, the wearable monitoring system may assist individuals in achieving and maintaining a healthy body weight though monitoring of physical activity and encouraging health-promoting lifestyle changes.
One embodiment may take the form of a footwear system for monitoring weight, posture allocation, physical activity classification, and energy expenditure calculation includes an accelerometer configured to obtain acceleration data indicative of movement of a user's foot or leg. The footwear system may also include a pressure sensing device mounted in an insole and configured to obtain pressure data indicative of pressure applied by a user's foot to the insole, as well as a transmitter communicatively coupled to both the accelerometer and the pressure sensing device and configured to transmit the acceleration and pressure data to a first processing device configured process the acceleration data and the pressure data to distinguish a first posture from a second posture different from the first posture and process the acceleration data and the pressure data to distinguish a first movement-based activity from a second movement-based activity different from the first movement-based activity.
Another embodiment may take the form of a method executed by a processing device for recognizing posture and activities using the processing device. The method may include receiving pressure data indicative of pressure applied by a user's foot to the insole, receiving acceleration data from an accelerometer indicative of movement of a user's foot or leg, and processing the pressure and acceleration data so as to distinguish a first posture from a second posture and to distinguish a first movement-based activity from a second movement-based activity.
Yet another embodiment may take the form of a method for deriving an energy expenditure value. The method may include obtaining pressure data using a capacitive pressure sensor indicative of pressure applied by a user's foot to the insole. The capacitive pressure sensor may include one or more conducting plates. The method may also include obtaining acceleration data using an accelerometer indicative of movement of a user's foot or leg and transmitting the pressure and acceleration data to a processing device configured to process the pressure and acceleration data and derive an energy expenditure value based on the pressure and acceleration data.
Another embodiment may take the form of a computer-readable medium having computer-executable instructions for performing a computer process for recognizing posture and activities. The instructions include causing the computer server to receive pressure data indicative of pressure applied by a user's foot to an insole, receive acceleration data from an accelerometer indicative of movement of a user's foot or leg, and process the pressure and acceleration data so as to distinguish a first posture from a second posture and to distinguish a first movement-based activity from a second movement-based activity.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. A more extensive presentation of features, details, utilities, and advantages of the present disclosure is provided in the following written description of various embodiments, illustrated in the accompanying drawings, and defined in the appended claims.
Embodiments disclosed herein include a wearable energy expenditure monitoring system for monitoring body weight, postural allocation, physical activity classification, and energy expenditure calculation. “Posture allocation,” as used herein, includes distinguishing between various postures that may be held by a user. Some examples of postures may include, but are not limited to, lying down, sitting, standing, and so on and so forth. “Physical activity classification,” as used herein, includes distinguishing between various movement-based activities performed by a user. For example, physical activities may include, but are not limited to, walking, jogging, running, cycling, climbing stairs, and so on and so forth.
In one embodiment, the wearable monitoring system may include an accelerometer and a pressure sensor that is integrated into an insole. An “insole,” as used herein, is a member that sits beneath a foot. For example, an insole may include the interior bottom of a shoe, a foot-bed, or a removable insert that may be positioned in a shoe or in a sock. Additionally, an “insole” may include a member that is integrated into a sock so that when the sock is worn, the insole is sits beneath the foot. The integration of the accelerometer and pressure sensor into conventional footwear requires minimal extra effort from the user to wear these devices, thus reducing the burden and conspicuousness associated with activity monitoring and facilitating everyday use.
The wearable monitoring system may be based on a combination of multiple sensor modalities, including acceleration and pressure readings from the accelerometer and pressure sensor. The combination of these two modalities may identify many metabolically significant postures and activities. For example, standing can be differentiated from sitting by observing a higher amount of pressure at low levels of acceleration, while walking and jogging may each produce unique patterns of pressure and acceleration at every phase of a gait cycle. Accordingly, the combination of pressure and acceleration data allows for differentiation between major classes of metabolically significant activities, including sitting, standing, walking, jogging, cycling, ascending stairs, descending stairs, household chores, and so on, in which an average person spends the majority of time.
The accelerometer and the pressure sensor may be communicatively connected to a portable or stationary processing device configured to receive and process the data from these devices. For example, the processing device may be configured to automatically determine the user's posture or activity based on the received data, compute energy expenditure based on the type and intensity of the activity, compute performance metrics for different exercise activities (e.g., number of steps, distance for walking or jogging), compute body weight estimates, and/or provide user feedback to maintain a higher metabolic rate. The processing device may be any electronic device having data processing capabilities, and may desirably be a portable device, including a smartphone, a personal digital assistant (PDA), an MP3 player, a laptop computer, table computer or other similar device.
In one embodiment, shown in
The accelerometer 101, 201 may be configured to measure the physical acceleration experienced by the user's feet. In some embodiments, the accelerometer 101, 201 may be a one-dimensional accelerometer, a two-dimensional accelerometer, or a three-dimensional accelerometer. One example of a two-dimensional accelerometer that may be used in conjunction with the disclosed embodiments is a two-dimensional MEMS accelerometer, which is configured to measure sagittal plane accelerations of the user's feet. An example of a three-dimensional accelerometer that may be used in conjunction with the disclosed embodiments is an LIS3L02AS4 MEMS accelerometer, which is configured to measure accelerations of the user's shoes 109, 209 in three or more orthogonal directions. It should be appreciated that other embodiments may use one or more accelerometers 101, 201 mounted to other portions of the user's shoes or body, and that the accelerometer may sense in other desired planes, such as the coronal plane.
The insole 103, 203 may include a pressure sensor configured to detect changes in the amount of pressure applied to the insole 103, 203 by the sole of the user's feet. The insole 103, 203 may be a flexible insole, and may be configured as a removable insert, incorporated into user's socks, e.g., using a polymer backing or a conductive thread, or attached to the user's shoe 109, 209. As will be further described with respect to
The gait cycle identification and loading profiles obtained from the pressure sensor may be used to classify the type of motion-based activity that the user is performing (e.g., walking vs. running), quantify the amount of body motion in static postures (e.g., shifts in body weight while standing), and distinguish between movement performed along a level surface from movement performed along an inclined (i.e., uphill or downhill) surface, such as a gradually inclined surface, stairs, etc. The gait cycle identification and loading profiles may also be used to detect asymmetries in the gait pattern indicating fatigue or potential development of injury. Additionally, data regarding key temporal and spatial gait parameters, including, but not limited to, cadence, stride length, and stance time, may be extracted from the pressure and/or acceleration data and used to characterize the user's movement-based activities and provide feedback to the user. For example, the feedback may include the number of steps taken by the user, distance walked, cadence, etc.) While the embodiments disclosed herein include capacitive and force-sensitive resistor-based sensing elements, other embodiments may use other pressure sensors to determine the plantar pressure exerted by a user.
Some embodiments of the monitoring system 100, 200 may further include one or more physiological sensors 121 that are also connected to the processing device 105. For example, the physiological sensor 121 may be a bioelectric sensor that is configured to detect electric currents that flow in a user's nerves and muscles, such as the user's heart. In other embodiments, the physiological sensor 121 may be a heart monitor, a piezoelectric pulse monitor, a reflectance optical oximeter configured to detect oxygenation and/or pulse, a respiration sensor, a galvanic skin response sensor, a skin temperature sensor, and so on and so forth. The physiological sensor 121 may be connected to any part of the user's body through either a wired or a wireless connection. For example, the physiological sensor 121 may be positioned directly on the user's skin, over the user's clothing, or in one or both of the user's shoes as an insertable insole, in the user's socks, etc. In one embodiment, the pressure 103, 203 and the physiological sensor 121 can be implemented as a single capacitive sensor. For example, a high impedance capacitive sensor may be used to measure both pressure under the user's feet, as well as bioelectric potential created by the user's heartbeat. These signals may later be separated by signal processing techniques such as frequency filtering, wavelet, or some other transform.
In one embodiment, the accelerometer 101, 201, pressure sensor 103, 203, battery 107, power switch 111, transmitter 115, and processor 120 may be coupled to the user's shoes 109, 209. For example, as shown in
Referring to
In one embodiment, the monitoring system 100, 200 may also include an activation mechanism configured to allow the user to activate and deactivate the monitoring system 100, 200. The activation mechanism may be a mechanism provided on the user's shoe 109, 209, such as a switch, button, lever, motion sensor, pressure sensor (resistive or capacitive), etc. or may be a device that is remotely connected to the monitoring system 100, 200, such as a remote control, a remote motion sensor, etc.
As discussed above, the pressure sensor 103, 203 may be configured to activate and deactivate the monitoring system 100, 200. For example, the pressure sensor 103, 203 may be configured to activate the monitoring system 100, 200 when the user is wearing the shoes 109, 209, i.e., when pressure is applied to the pressure sensor 103, 203, and deactivate the monitoring system 100, 200 when the user is not wearing the shoes 109, 209, i.e., when no pressure is applied to the pressure sensor 103, 203. In other embodiments, the pressure sensors 103, 203 may further be configured to place the monitoring system 100, 200 into a low-power, or “sleep” state when the sensors 103, 203 determine that the user is not wearing one or both shoes 109, 209. The “sleep” state may serve to prolong the battery life of the monitoring system, and may further serve to expedite the time required for activating the monitoring system 100, 200.
The transmitter 115 may be connected to the processor 120 of the monitoring system 109, 209, and may be configured to transmit sampled pressure and acceleration data collected by the accelerometer 101, 201, the pressure sensor 103, 203, and/or the physiological sensor 121 to a processing device 105 that is configured to process the received data. The data transmission may be through either a wired or a wireless transmission medium. In one embodiment, the transmitter 115 may be a wireless transmitter, and may use a wireless protocol for communicating with the processing device. For example, the transmitter 115 may use an a Bluetooth wireless protocol, an ANT protocol, or a ZigBee protocol. In one embodiment, the wireless protocol may be a low-power consumption protocol that preserves the battery life of the battery 107.
The processing device 105 may be a dedicated electronic device or a ubiquitous electronic device that is configured to perform other functions. Some examples of electronic devices that may be used in conjunction with the disclosed embodiments include, but are not limited to, a personal computer, such as a laptop, tablet PC or a handheld PC, a PDA, a mobile telephone, a media player, such as an MP3 player, or a television receiver. As will be further discussed below, the processing device 105 may run monitoring software configured to process the collected data and provide feedback to the user regarding the collected data. For example, the monitoring software may use the collected acceleration and pressure data to calculate the weight and energy expended by the user and provide this information to the user as feedback.
As discussed above,
As discussed above,
The accelerometer, battery, power switch, and/or transmitter may be more or less distributed in other embodiments. For example, the accelerometer and the transmitter may be integrated into the shoe, while the battery and the power switch maybe provided on a separate device.
As shown in
Referring to
is the permittivity of free space, A is the area of overlap between plates in m2, d is the distance between plates in m.
The capacitive sensor may provide data similar to that provided by a force sensitive resistor sensor. For example, the changes in capacitance of the sensors may be proportional to the pressure applied by user in static postures and dynamic activities. Thus, the changes in capacitance can be used to identify various parts of gait cycle, amount of body weight shifting (fidgeting) in static postures, and/or be used for weight measurement. As an example, the capacitive sensor may be configured to sense a significant change, i.e., increase, in capacitance when a user's heel strikes the ground, a decrease in capacitance during the middle of a stance, and an increase in capacitance during the end of stance.
As shown in
As an example, if the resistance R is approximately 1 Mohm, the discharge time may vary between 322 uS to 1.9 mS for sampling frequencies greater than 500 Hz. When in an input configuration, the MSP430 microcontroller may have a ±50 nA leakage port current that is negligible as compared to the discharge current through the resistor R (3 uA at 3V), and thus does not impact the accuracy of the readings. The ESR of the capacitive sensor may also be taken into consideration if necessary, i.e., if the ESR is high enough to influence the discharge time of the capacitive sensor. In one embodiment, a 16-bit timer may be clocked using a 16 MHz crystal, which may result in 5000 to 30400 counts per measurement. The resulting discretization of the capacitance is fine enough to capture even minute variations in pressure and/or weight, as applied to the capacitive sensor.
In one embodiment, the insert may include five total resistors 409 positioned under various points of contact with the user's foot, including the heel, heads of metatarsal bones and the big toe. While five sensors are illustrated, it should be appreciated that any number of force sensitive resistors 409 may be distributed in any configuration throughout the insole, so long as the pressure sensor 401 provides sufficient information to recognize and characterize postures, activities and/or measure weight. Additionally, the size of the resistors 409 may vary in other embodiments. For example, a single force sensitive resistor may be large enough to cover both a metatarsal bone and the toe.
The positioning of the force-sensitive resistors may allow for differentiation of various parts of the user's gait cycle. For example, a pressure sensor under the heel may serve to detect the initial contact of the foot with the ground (i.e. heel strike). Additionally, various amounts of pressure on the heel and metatarsal sensors, in combination with acceleration readings, may suggest a particular stance phase of the user's gait cycle, and so on and so forth.
The number of pressure sensors 401 may vary from embodiment to embodiment. For example, one embodiment may use a single pressure sensor covering the entire area under the foot or a portion of the area under the foot, while another embodiment may use 34 pressure sensors that are positioned at various locations under the user's foot. One particular sensor layout includes 3 pressure sensors: a pressure sensor that is positioned under the user's heel, a pressure sensor that is positioned under the user's metatarsal heads, and a pressure sensor that is positioned under the user's big toe. Another sensor layout includes a multitude of sensors, for example between 25 to 100 sensors, that are evenly distributed under the foot.
The layout of the sensors is not dependent on the sensor type, i.e., whether the sensor is a resistive or a capacitive pressure sensor, but may instead be selected based on the desired functionality and accuracy of the overall pressure sensor. For example, different layouts and/or numbers of sensors may have varying impacts on functionality, accuracy, and implementation costs.
The data signals from the pressure sensors, accelerometer, and physiological sensors may transmitted to different processing modules, which may apply signal processing, pattern recognition, and classification algorithms to the received data to measure weight, recognize postures and activities, estimate energy expenditure, and provide feedback to a user. As shown in
Additionally, the processing device may include a weight estimation module 450 that is configured to receive information about the user's posture and/or activity from the activity pattern recognition module 452, as well as acceleration and pressure data from the accelerometer and pressure sensor, to compute an estimate of the user's weight. In one embodiment, the user's weight can be measured as proportional to pressure when the device detects a standing posture, and acceleration data indicates that the user is in a quiet standing position, for example, if the acceleration data indicates that the acceleration of the user is below a fidgeting threshold.
The signal processing module 454 may be configured to receive physiological data and extract various metrics of interest, such as the user's heart rate. Additionally, the signal processing module 454 may be configured to receive and process acceleration data from the accelerometer to remove to remove signal artifacts that may be created by user's movements.
The processing device 105 may further include an energy expenditure estimation module 456 that is configured to receive acceleration data from the accelerometer 109, 209, as well as processed data from the signal processing module 454, and apply one or more predictor values to calculate the user's energy expenditure. The predictor values may include, but are not limited to, the user's weight, height, current posture or activity, features derived from pressure and/or acceleration data, the user's heart rate, and so on and so forth. In one embodiment, the energy expenditure estimation module may also be configured to monitor the time that a user is performing a particular activity or holding a particular posture, or if the user's energy expenditure level is below a set target for a predetermined period of time. Where such an event is detected, the device 105 may be configured to proactively alert the user and/or suggest corrective actions to boost the user's energy expenditure.
In another embodiment, the processing device may be configured to allow the user to retrieve both historical and current data on demand. It should be noted that other embodiments may include more or fewer software processing modules. For example, the weight estimation module 450 may be replaced by an application that is configured to allow users to enter their weight through a Graphical User Interface. As another example, the physiological sensor 121 may be absent from the system and heart rate may not used in the energy expenditure calculation. Other combinations of sensors and/or processing modules are also possible.
Additionally, the processing device may be connected to other processing devices running the monitoring software, for example, through a network. A “network,” as used herein, is a group of communication devices connected to one another and capable of passing data therebetween. As such, a network may be the Internet, a computer network, the public-switched telephone network, a wide-area network, a local area network, a cellular network, a global Telex network, a cable network or any other wired or wireless network.
In this embodiment, the monitoring software may be stored on a separate server 460 connected to the network, rather than on the processing devices, e.g., 105, themselves, and the processing devices may be configured to access the monitoring software through the server 460. In another embodiment, the server 460 may be configured to host a community website that users can access through the processing devices to compare their posture and movement-based activity data to statistics from other individual users and to the user population in general. The website may also be configured to host competition-based weight maintenance/loss/gain programs based on data collected from each user's monitoring system. These and other functions may be accessed and managed by a user via a graphical user interface (“GUI”). For example, the GUI of the monitoring software may permit users to add contacts, create web groups, schedule meetings, and so on and so forth.
In the operation of block 505, the processor may be configured to sample the data and form a feature vector from the collected data. In one embodiment, the processor may obtain multiple readings, compute derived features, and combine them into a feature vector that includes several lagged measurements of acceleration and/or pressure. As an example, the pressure and acceleration data may be sampled at 25 Hz by a 12-bit analog-to-digital converter. In one embodiment, data acquisition may be based on a Wireless Intelligent Sensor and Actuator Network (“WISAN”) that is configured for time-synchronous data acquisition. Accordingly, the WISAN may allow for data sampling at substantially the same time (with a difference of no more than 10 microseconds) from two shoes, as worn by the user. In another embodiment, data acquisition may be based on a circuit that combines a microcontroller equipped with an analog-to-digital converter (and/or an SPI or 12C interface for reading sensor signals) and a Bluetooth or a Bluetooth Low Energy module for wireless transmission of the data. In yet another embodiment, the microcontroller may be configured to transmit the sensor data through a standardized (for example, Zigbee, ANT, and so on) or custom (for example, based on an nRF24LE01 chip or a similar chip) wireless interface.
In the operation of block 507, pattern recognition is performed on the feature vectors to determine if the user is wearing the shoe. An example of pattern recognition algorithm may be a simple threshold classifier that is configured to determine that the user is wearing a shoe when the pressure reading from the pressure sensor exceeds a predefined threshold. Another example of a pattern recognition algorithm may determine that the user is wearing a shoe when the collected acceleration data indicates that the motion of the shoe exceeds a predefined threshold. In other embodiments, both acceleration and pressure data may be used to determine whether the user is wearing the shoe. For example, other algorithms may include artificial neural networks, support vector machines, and other classification algorithms.
If, in operation 509, the processor determines that the user is wearing the shoe, then in the operation of block 511, the processor determines whether the wireless transmitter needs to be turned on. If in operation of 511 the processor determines that the wireless link is off, then, in the operation of block 517, the wireless link is turned on.
If, in the operation of block 509, the processor determines that the user is not wearing the shoe, then, in the operation of block 513, the processor may turn off the wireless transmitter to save power. Additionally, the processor may further be configured to turn off any sensors and/or other electronic components of the monitoring system.
If, in the operation of block 511, the processor determines that the wireless transmitter is turned on, then, in the operation of block 515, the processor may determine whether the transmitter is connected to the processing device. This may include determining whether the receiver in the processing device is turned on and that the receiver is enabled to receive data from the transmitter. If, in the operation of block 511, the processor determines that the wireless transmitter is turned off, then, in the operation of block 517, the processor may turn on the wireless link, and, in operation 515, determine whether the transmitter is connected to the processing device.
If, in the operation of block 515, the processor determines that the transmitter is connected to the processing device, then, in the operation of block 519, the transmitter may transmit the pressure and acceleration data to the receiving processing device. In one embodiment, the pressure and acceleration data may be sampled at a higher rate than when the monitoring system is in an inactive mode. If, however, in the operation of block 515, the processor determines that the transmitter is not connected to the processor device, then, in the operation of block 521, the processor may store the data in a storage device for later transmission. For example, the processor may store the data until it determines that the transmitter is connected to the processor device, at which point it may retrieve the data from the storage device and transmit the data to the receiving processing device.
If, in the operation of block 605, the processing device determines that the wireless link was successfully established, then, in the operation of block 609, the processing device may receive data transmitted from the accelerometer and the pressure sensor. In the operation of block 611, the processing device may use signal processing techniques to condition the received data signals, as well as extract relevant features. Examples of signal processing techniques that may be used include normalization of data to a specified range of values, formation of lagged vectors representing a time slice of the signals, computation of derived metrics such as room-mean-square, entropy, spectral coefficients, and so on.
In the operation of block 613, the processed signals may be further processed to use pattern recognition to recognize various postures and/or movement-based activities of the user. For example, the processing device may be configured to determine whether the user is sitting, standing, walking, etc. by applying pattern recognition algorithms to the received data signals. Pattern recognition algorithms that may be used include artificial neural networks, for example, multi-layer perceptron, or other classification algorithms, such as support vector machines, Bayesian classifiers, etc. A feature vector may be presented to the pattern recognition algorithm, which may assign it to one of the classes ('sitting', ‘standing’, etc) based on previously learned examples. For example, low values of acceleration combined with a pressure reading that is less than the user's body weight indicate that the user is sitting, while low acceleration values combined with a pressure reading that is substantially equal to the user's body weight indicate that the user is standing. Walking may be characterized by horizontal and vertical acceleration patterns that exhibit low cycle-to-cycle variability, combined with pressure changes that alternate between high and low (stance/swing) and travel from heel to toe.
The posture or activity represented by the feature vector may not belong to the list classes known to the pattern recognition algorithm. For example, the pressure and/or acceleration readings may not match a posture or activity that is readily classifiable by the processing device. In such cases, the classification may be performed with a hard assignment in which an unknown posture or activity is assigned to the closest classifiable posture or activity, or, in other embodiments, the classification may be performed with a rejection in which the unknown posture or activity is classified as an unclassifiable posture or activity.
In the operation of block 615, the processing device may determine whether the posture is the first posture within a list of predetermined postures. For example, the processing device may determine whether the user is sitting or standing. If, in the operation of block 615, the processing device determines that the posture is the first posture, then, in the operation of block 617, the processing device may log the time spent in the first posture and compute the estimated energy expended by the user in the first posture. The computation of energy expenditure may be performed using a linear or non-linear regression utilizing one or more predictors, such as weight (either measured or entered by the user), height, age, body-mass-index, heart rate, and metrics derived from pressure and acceleration signals such as mean, root-mean-square, standard deviation, coefficient of variation, entropy, number of zero crossings, including metrics after logarithmic transform and metrics for combinations of signals (for example, as a sum or product). To improve the accuracy of the energy expenditure estimation, a dedicated regression model may be built for a specific posture class known to the classifier (e.g. ‘sitting’). The predictors and the regression equations may vary from posture to posture.
If, in the operation of block 615, the processing device determines that the posture is not the first posture in the predetermined list of postures, then, in the operation of block 619, the processing device may determine whether the posture performed by the user is another posture in the predetermined list of postures. If, in the operation of block 619, the processing device determines that the posture is the another posture in the predetermined list of postures, then, in the operation of block 621, the processing device may log the time that the user is performing the posture and calculate the energy expended by the user while in the posture.
If, however, in the operation of block 619, the processing device determines that the posture is not in the predetermined list, then, in the operation of block 623, the processing device may be configured to determine whether the user is performing a first movement-based activity within a list of predetermined movement-based activities. As discussed above, some activities may include, for example, walking, cycling, running, climbing stairs, and so on. If, in the operation of block 623, the processing device determines that the user is not performing the first activity, then, in the operation of block 627, the processing device may be configured to determine whether the user is performing another activity in the list. If, in the operation of block 623, the processing device determines that the user is performing the first activity, then, in the operation of block 625, the processing device may be configured to log the time that the user spends performing the activity and compute the energy expended by the user while performing the activity.
In one embodiment, energy expenditure may be computed by a linear or non-linear regression model that uses one or more predictors such as weight (either measured or entered by the user), height, age, body-mass-index, heart rate, and/or metrics derived from pressure and acceleration signals. To improve accuracy of energy expenditure estimation, a dedicated regression model may be built for a specific activity class known to the classifier (e.g. ‘walking’). The predictors and the regression equation may vary from activity to activity. For example, each activity may have an associated energy expenditure model. Additionally, each activity may include an associated “intensity” to more accurately estimate the energy expenditure of the user. For example, if the energy expenditure associated with walking at 2.5 mph is 4 kcal/min, then this value is used when the classifier identifies the activity as walking at an intensity of 2.5 mph. Additionally, the processing device may be configured to compute one or more characteristics of the activity, for example, the number of steps taken by the user, and use this information to compute the energy expended by the user in performing the activity.
If, in the operation of block 627, the processing device determines that the user is not performing a movement-based activity in the list, then, in the operation of block 631, the processing device will categorize the activity or posture as an unclassifiable or unrecognized activity or posture, and compute the energy expended by the user in performing the unrecognized activity or holding the unrecognized posture. For example, the acceleration and/or pressure data may be used to model the energy expenditure of the user, rather than classifying the activity or posture being performed by the user. In other embodiments, the processing device may assign a generic energy expenditure value that can be used to compute the energy expended by the user in performing the unrecognized activity or posture. These energy expenditure calculations may be performed by a regression model, as previously discussed above with respect to known activities and postures. Alternatively, the processing device may use the acceleration and/or pressure data to classify the activity or posture of the user into one of the known activities or postures. However, the regression model for the unclassifiable activity or posture may encompass a range of postures and activities and may therefore be less accurate than activity- or posture-specific models.
If, in the operation of block 627, the processing device determines that the user is performing another activity on the list, then, in the operation of block 633, the processing device may be configured to log the time that the user spends performing the recognized activity and compute the energy expended by the user while performing the activity. For example, the processing device may be configured to compute one or more characteristics of the sensor signals (such as metrics for the regression models described above) or characteristics of the activity being performed (for example, cadence and/or number of steps). Other characteristics may include the associated posture, intensity of the acceleration signal (magnitude and frequency) and/or the magnitude of the pressure readings.
In the operation of block 635, the processing device may add the calculated energy expenditures for each of the recognized and unrecognized postures and/or activities to the user's prior calculated energy expenditures to obtain a cumulative energy expenditure statistic for a predefined period of time. This may be done periodically, for example, every minute or each time that the system determines that the user is performing a new activity or holding a new posture. Additionally, the time period may vary from embodiment to embodiment, or may be user selected. For example, the cumulative period may be a day, an hour, a week, and so on.
In the operation of block 637, the processing device may determine whether the activity level or energy expenditure for the user is below a predefined threshold. The threshold may be calculated by the processing device, for example, based on the user's weight and a target weight, target energy expenditure for a person of certain anthropometric characteristics, input by the user, or obtained by some other means. If, in the operation of block 637, the processing device determines that the energy expenditure is below the threshold or the user has been assuming a static posture for too long, then, in the operation of block 639, the processing device may determine whether the alerts for notifying the user that his or her energy expenditure is below the threshold have been enabled. If, in the operation of block 637, the processing device determines that the energy expenditure of the user meets or exceeds the threshold, then, in the operation of block 641, the processing device may determine whether or not a visualization of the user's energy expenditure data should be generated and provided to the user. For example, the processing device may determine whether or not the user has prompted the processing device for a visualization of his or her energy expenditure data.
If, in the operation of block 639, the processing device determines that the alerts for notifying the user that his or her energy expenditure is below the threshold have been enabled, then, in the operation of block 643, the processing device may provide an audio, tactile, and/or visual alert to the user. For example, the processing device may generate a pop-up icon or sound that notifies the user as to his or her failure to meet the threshold. In addition, the processing device may offer a suggested corrective action. For example, the processing device may generate an alert advising the user to “take at least 100 steps more a day,” and so on.
If, in the operation of block 641, the processing device determines that a visualization of the user's energy expenditure should be generated, then, in operation 645, the processing device may generate a visual depiction of the user's cumulative energy expenditure, activity, and/or behavioral data. This may include any graphs and/or charts summarizing this information. If, in the operation of block 641, the processing device determines that a visualization of the user's energy expenditure should not be generated, then, in operation 647, the processing device may determine whether it should periodically send cumulative energy expenditure, activity, and/or behavioral data to a data storage device. This feature may be enabled by a user, for example, by manipulating settings through the graphical user interface of processing software running on the processing device. The data storage device may be a remote data server, or in other embodiments, may be a memory device within the processing device. If, in operation 647, the processing device determines that it should periodically send cumulative energy expenditure, activity, and/or behavioral data to a data storage device, then, in operation 649, the processing device may be configured to upload the data to the user's personal server account. If, in operation 647, the processing device determines that it should not periodically send cumulative energy expenditure, activity, and/or behavioral data to a data storage device, then, the method returns to operation 609.
A. Data Collection
1. Shoe-Based Wearable Sensor
The plantar pressure and heel acceleration data were collected by a wearable sensor system embedded into subjects' shoes. Each shoe incorporated five force-sensitive resistors (Interlink Inc.) integrated with a flexible insole and positioned under the critical points of contact: heel, heads of metatarsal bones and the big toe. Such positioning allowed for differentiation of the most critical parts of the gait cycle such as heel strike, stance phase and toe-off as well as accounting for differences in loading of anterior and posterior areas of the foot in ascending/descending stairs and cycling. In an alternative configuration, not used in these studies, a clip-on sensor device may be attached to a shoe. The motion information was provided by a 3-dimensional accelerometer (LIS3L02AS4) positioned on the back of the shoe. The goal of accelerometer was to detect orientation of the shoe with respect to gravity, to characterize the motion trajectory and to help characterize fidgeting in static postures as well as intensity of physical activity. The battery, power switch and wireless board were installed on a rigid circuit board glued to the back of the shoe. The tail of the flexible insole was fed through a narrow cut in the shoe and connected to the same circuit board. The sensor system was very lightweight and created no observable interference with motion patterns.
Pressure and acceleration data were sampled at 25 Hz by a 12-bit analog-to-digital converter and sent over a wireless link to the base computer. The wireless system used for data acquisition was based on Wireless Intelligent Sensor and Actuator Network (WISAN) developed specifically for time-synchronous data acquisition. Application of WISAN allowed for data sampling at exactly the same time (with a difference of no more than 10 microseconds) from both shoes. The sensor data were streamed to a portable computer with a Labview front end and stored on the hard drive for further processing.
2. Data Collection Protocol
Data collection was performed on a group of 16 human subjects, 8 males and 8 females. Institutional Review Board approval and each subject provided informed consent. The subjects were chosen to reflect a diverse adult population. Subject characteristics were: mean age of 25±6.5 years (range 18-44); mean weight was 76.9±20.6 kg (range 48.6-119.8), mean Body Mass Index (BMI) was 26.7±6.5 kg/m2 (range 18.1-39.4). The shoe sizes (US) ranged from 9.5 -11 for men and from 7-9 in women. Based on self-report, volunteers were weight stable (<2 kg weight fluctuation) over the previous 6 months. Individuals that were healthy, non-smokers who were sedentary to moderately active (<2-3 bouts of exercise/wk or participation in any sporting activities <3 hr/wk) were invited to participate in the study. Pregnant women and those who had impairments that prevented physical activity were excluded
Data collection for each subject was performed during a single 2.5-3 hour visit. The subjects wore the sensor-equipped shoes for the duration of the visit. The subjects also wore a portable metabolic system (Viasys Oxycon Mobile) to measure energy expenditure. The data collection protocol is shown in Table 1. A total of 20 hours 37 minutes of data were recorded for 6 major posture/activity classes: sitting motionless or with fidgeting (3 hr 9 min), standing motionless or with fidgeting (3 hr 5 min), walking/jogging at various speeds and grades (10 hr 33 min), ascending stairs (36 min), descending stairs (32 min) and cycling at 50 and 75 rpm (2 hr 34 min). Recognizing these 6 classes from the shoe sensor data was one of the major goals of this study. Subjects were not restricted in the way they assumed postures and or performed activities. Standing did not require any specialized equipment; a chair with a rigid back was used for sitting; walking/jogging was performed on Biodex Gait Trainer 1 treadmill; subjects used stairs between the ground and second floor for ascending and descending; cycling utilized Ergomedic 828E bicycle exerciser:
3. Integrity Review
The data collected during the study were manually reviewed for integrity through a software package written in Labview. The review revealed that in 7 subjects the solder connections on one or more pressure sensors failed due to the pressures under the feet. The common mode of failure was break in the trace on the flexible insole resulting in a flat line output for all activities. The remaining 9 subjects had no failures in the sensor data and are referred further as the ‘no failures’ group.
B. Methods
1. Preprocessing of the Data
Only minimal preprocessing consisting of feature vector forming and normalization was applied to the sensor data. No other features were extracted. The feature vectors were formed to represent a time period (epoch) of two seconds in duration. Time histories of pressure and acceleration from both shoes were used as follows. A single sample of data from a shoe is represented by vector S={AAP, AML, ASI, PH, PMO, PMM, PMI, PT}, where AAP is anterior-posterior acceleration, AML is medial-lateral acceleration, ASI is superior-inferior acceleration, PH is heel pressure, PMO, PMM, PMI are pressures from outer, middle and inner metatarsal sensors, respectively, and PT is pressure from the big toe sensor. The time series of data from both shoes can be represented as ft={SL, SR}t, i={1, . . . , M}, where Sl,SR are the data samples from the left and right shoe, respectively, and M is the length of time series. The feature vector for an epoch e was produced using a decimation factor d as Fe,d={fe*N+d*k+1}k∈{0, . . . ,[N-1/d]}, where N is the number of samples in an epoch at the original 25 Hz sampling frequency (N=50 equivalent to a 2 second epoch used in this study), and k is the selection index. Use of decimation is equivalent to resampling of the original signals to a lower frequency. For example, d=1 corresponds to a sampling frequency of 25 Hz, d=5 corresponds to 5 Hz, etc. allowing to study the effects of changes in the sampling frequency on recognition accuracy. Due to decimation, the size of the feature vectors with all sensors included may vary from 800 elements (d=1, 2 shoes×8 sensors×25 samples per second×2 seconds=800 samples) to 32 elements (d=25, 2 shoes×8 sensors ×1 samples per second×2 seconds=32 samples). The features vectors from all epochs in the experiment were combined in a feature matrix
2. Classification by SVM
The pairs of feature vectors and class labels {Fe,d,Le} were presented to a supervised classification algorithm for training and validation. The labels Le represented a distinct class {1-sitting, 2-standing, 3-walking, 4-ascending stairs, 5-descending stairs, 6-cycling}. The selected classifier was a variation of Support Vector Machine (SVM) implemented as a Matlab package (libSVM). The choice of the classifier was defined by the consideration of the generalization ability. The maximum margin classifier implemented by an SVM is less prone to overfitting compared to other available methods. For the target application of automatic classification of postures and activities the ability to generalize effectively is extremely important. As an example, motion of the lower extremities during ambulation is not perfectly repeatable. Similar variation in sensor data is expected from other postures and activities. In addition, some of the recorded data segments may contain transitions between similar postures and activities introducing the data, which cannot be perfectly labeled as one the classes. Thus the classifier is posed with a difficult task of learning a decision boundary, which should provide the best generalization from expectedly imperfect data.
The SVM classifier utilized Gaussian kernel (exp(−γ* (u−v)2). The best values of parameter C=10 (cost of misclassification) and y=0.0156 (width of Gaussian kernel) were found in grid search procedure varying C as C=10x, x={−1, . . . ,3} and y as y=2y, y={−8, . . . ,−2}.
3. Training, Validation and Calculations of Accuracy
A common training and validation procedure was deployed for all analyses. Specifically, a 4-fold cross validation was utilized where three quarters of all data were used as training set and the remaining quarter was used as validation set. The accuracy was reported as an average across 4 folds.
4. Six-Class Individual Models
The individual models are the best fit to the individual traits and thus represent the baseline accuracy for comparison. For the individual models, the folds were computed for each subject. All postures and activities were proportionally represented in each of the folds. All sensors were utilized in feature computation and dwas set to 1.
5. Six-Class Group Models
These models established group classification accuracies in the whole population of subjects as well as in the smaller ‘no failures’ group. The goal was comparison of recognition accuracies between these two group and the individual models, and evaluation of impact of solder connection failures. For these and other group models, the folds were organized by including the full dataset from each individual subject that belonged to a fold. All shoe sensors were utilized, and the decimation factor dwas set to 1.
6. Investigation of Best Sensor Configuration on the Group of 9 Subjects with No Sensor Failures
The goal of this analysis was to investigate the contribution of each individual sensor to recognition accuracy and determine the best sensor configuration. The study was performed using the backward selection procedure. First, the baseline population-average accuracy was established in configuration with all 8 sensors active. Next, sensors were excluded one at a time from the base configuration and accuracy of recognition was evaluated by a 4-fold cross-validation. On the next step, the configuration with highest accuracy from the previous step became the base configuration. The procedure was repeated until only one sensor was left. Since excluding a failed sensor could boost the recognition accuracy and lead to incorrect interpretation of sensor configuration, this procedure was performed only on the group of 9 subjects with no solder connection failures. The testing was performed with a decimation factor d=1.
7. Recognition Sensitivity to Sampling Frequency
Even momentary snapshots of pressure and acceleration are very unique to a given posture or activity. This analysis tested the hypothesis that using combination of pressure and acceleration performs well even at a lower sampling frequency. The accuracy of pattern classification was established on the group of 9 ‘no failures’ subjects using the best known sensor configuration and a range of decimations d={1,2,3,4,5,6,7,8,9,10,16,20,25} equivalent to the range of sampling frequencies of 25 to 1 Hz.
8. One Shoe vs. Two Shoes
The effect of wearing one sensor-equipped shoe vs. two shoes was investigated by changing the way the feature vectors were formed. Specifically, the feature vector was formed either as ft={SL}t, or ft={SR}t. Thus recognition accuracy from the data generated only by the left or right shoe can be compared to the data acquired from both shoes.
C. Results
Table 2 shown in
The results of the backward selection of various sensor configurations are shown in Table 3, which is presented in
The population-cumulative confusion matrix for recognition using the best sensor configuration is presented in Table 4:
The graph of the recognition accuracy as a function of the decimation factor is shown in Table 5, which is presented in
D. Discussion
The proposed device achieved the greater recognition rates (95%-98%) compared to previous experiments that used similar postures and activities. For example, demonstrated 88% percent accuracy with 6 postures and activities. The proposed shoe-based approach also matched or outperformed other single-location methodologies such as which reported a 95% accuracy across 8 postures and activities. The proposed device should be capable of maintaining the accuracy as other metabolically significant activities are classified (e.g., elliptical trainer)).
The device has shown excellent accuracy using group models, suggesting that individual calibration is not necessary. As Table 2 shows, the 98.6% average accuracy of the individual models is similar to the 98.1% accuracy of the group model for 9 ‘no failure’ subjects. A comparison to the full 16-subject group model shows a small 3% decrease in accuracy due to effects of sensor failures for some subjects. Sensor failures also explain the increased variance of the results. Subject number 15 suffered from multiple sensor failures at the very beginning of data collection and is an obvious outlier. This subject is a good example that demonstrates the effects of failed sensors: the individual model shows high recognition accuracy because of the redundancy in the pressure readings and the 16-subject group model shows a substantial reduction of accuracy because the sensors normally present in other subjects have failed in this subject.
As the confusion matrix in Table 3 shows, the shoe device achieves greater than 80% precision and recall in recognition of difficult activities such as ascending and descending stairs. The actual accuracy may actually be even higher as the subjects had to take several steps on a flat surface (which is correctly is classified as walking) when transitioning from one flight of steps to another.
The backward selection of the best sensor configuration shown in Table 2 clearly illustrates the redundancy in the sensor data. The best population-average accuracy of 98.1% is achieved by discarding signals from PMO and PMM. It is possible that the pressure patterns on those sensors may carry more individual traits than PMI. This figure also demonstrates the complementary nature of plantar pressure and heel acceleration patterns. While the population-average recognition accuracy using just AP acceleration is 83.9% and using just heel pressure is 84.4%, combining these two together provides an immediate increase of over 10% to 95.2%. High accuracy in configurations with only left and right shoe indicates that only one shoe can be equipped with sensors and still maintain high accuracy of classification.
Table 4 effectively demonstrates tolerance of the proposed combination of sensor modalities to lower sampling frequencies. While the highest accuracy of 98.1% is observed at 25 Hz, the relative decline for 5 Hz sampling is only 0.6% (accuracy of 97.5%). As example, a relative decline of 12% (from 85% to 75%) was reported while changing sampling from 25 Hz to 5 Hz for Y axis of an accelerometer. This useful property allows for lower data rates in a body network and a potential for extended battery life.
Similarly, the proposed methodology does not need signal processing and feature extraction beyond simple forming of vectors and normalization. This compares very favorably with extensively used frequency domain features that need substantial computing power and thus may present a heavy burden for a wearable computing platform.
Finally, it is possible to substantially increase durability of the pressure sensitive insole by changes in the manufacturing process. The reason for failures in 7 subjects was not the pressure sensors themselves but rather high pressure points created by solder connections. These points failed under substantial impact forces of walking and jogging. Eliminating the solder connections or encapsulating them into elastic buffer material should resolve this issue.
Research was focused on the design of a novel pressure-sensitive sensor that would utilize the sole of person's foot as one of the capacitor plates. Such a design offered much higher durability, cost savings and allow incorporation of truly novel functionality for weight measurement. Body weight is a metabolically-relevant physiological indicator that can further improve the accuracy of the device.
The research was carried out as a series of Tasks, each with its own deliverable:
To enable weight measurement the tested sensor spanned the whole area of the insole. Thus, changes in the distribution of weight on the sole of foot did not change the measurement. Use of a single pressure sensor was different from the existing shoe prototype, but research has shown that one pressure sensor is sufficient for highly accurate posture and activity recognition. A thin (8 mil) flexible insole had two isolated conductive plates (shown as green and blue areas) interleaved in a comb-like structure. The interleaving minimized Equivalent Series Resistance (ESR) in the biological tissue of the foot. Pressure applied to the top plate changed the gap d between the plates. Higher pressure resulted in a smaller gap and higher capacitance. The equivalent electrical circuit consisted of two variable capacitors in series.
To characterize the sensors, a series of experiments was conducted in which a realistic model of a foot was utilized. This eliminated the need for human subjects testing. The plastic hollow models of the foot was be acquired from an anatomy warehouse. These models represented high, normal arches and flat foot. To closely simulate properties of real living tissue, the models were be filled with a conductive gelatin solution which closely resembles electrical properties of the body. Weights were be added to the feet to apply a known amount of pressure to the sensor.
A. Task 1: Practically establish the range of capacitances for the sensor as a function of plate topology
The expected value of sensor capacitance was estimated based on the following considerations. The surface area of the insole varies approximately from 125 cm2 (women's US size 5) to 250 cm2 (men's US size 12). The capacitance in the simple plate model can be expressed as C=εγεo A/d, where εγ is the relative static permittivity (dielectric constant) of the material between the plates,
is the permittivity of free space, A is the area of overlap between plates in m2, and d is the distance between plates in meters. The estimate of C1 and C2 values can be obtained under following assumptions: 1) C1 and C2 represent approximately half of the surface area each 2) C1≈C2 3) εγ≈3.5 is that of rubber foam with approximately 50/50 air to rubber volume ratio, 4) typical thickness of the foam padding separating the foot and the capacitive plates is d=3 mm uncompressed and d=1 mm compressed. Then, for women's size 5, the minimal capacitance C↓1↑MIN=3.5*8.854E−12*0.0125/(3E−3)=129 pF. Similarly, the maximum capacitance in men shoe size 12 is C↓1↑MAX=3.5*8.854E−12*0.025/(1E−3)=774 pF. Thus, the expected range of capacitances for C1 and C2 is from 129 pF to 774 pF. Under ideal conditions, the capacitance of the sensor is equivalent to the capacitance of the series connection:
Thus expected values of capacitance were very close to those typically used in capacitive proximity and pressure sensors. However, the calculations above were based on a number of idealized assumptions. In practice, values of C1 and C2 depended on the complex interaction of the shape of the footprint and weight distribution which is hard to evaluate analytically. To identify the best possible plate topology on the flex insole, two plate configurations were tested: 1) uniformly distributed comb, and 2) comb following the pressure pattern in standing. The insoles with such patterns were fabricated using photo transfer and chemical etching process on flexible PCB material from LPKF. A copper electrode was inserted into the artificial foot and will act as the middle electrode. Capacitance of C1 and C2 will be practically measured in relation the middle electrode under pressures of 400-4000 Pa. The result was a configuration of plates that provides equivalent changes in C1 and C2 under load corresponding to standing (position in which the weight measurement will be taken).
B. Task 2: Establish the Plate Topology with the Lowest ESR
The plate topology was initially analyzed in Task 1 to ensure approximately equal values of C1 and C2. The second goal was to look at the effect of plate topology on the ESR. While the step of the comb structure spacing has no bearing on capacity (the area of overlap remains constant), it may have a significant effect on ESR. Indeed, the ESR was reduced as the length of the line separating two plates' increased (which in turn involves more tissue into equivalent electrical contact). A step size from the set {2 cm, 1 cm, 0.5 cm, 0.25 cm} was tested. The insoles with the varying comb structures were fabricated using photo transfer and chemical etching process. The ESR of the sensor was measured using AnaTek ESR meter under various loads. As the result of Tasks 1 and Task 2, the optimal plate topology was established.
C. Task 3: Characterize the Capacitive Sensor in Static Loading Tests
Sensitivity, non-linearity, repeatability and hysteresis were important parameters defining the basic accuracy of the sensor and were needed for additional numerical correction (e.g. non-linearity) or statistical processing of the measurements.
This experiment utilized an artificial prosthetic foot capable of carrying loads in excess of 100 kg, e.g. Flex-Foot Axia by Ossur. The loading characteristic of the sensor (capacitance vs. applied weight) was constructed using a set of weights in the range of 5-100 kg applied through the prosthetic. The weights were progressively loaded and unloaded from the foot. The resulting loading curve was used to calculate the following standard characteristics: sensitivity (pF/kg), repeatability (%), non-linearity (%), and hysteresis (%). These values determined the need for additional numerical correction of the sensor output for practical weight measurement.
D. Task 4: Demonstrate Continuous Capacitive Sensing by an Inexpensive Microcontroller-Based Circuit
This task had two goals: first, a design of software and hardware for continuous real-time (at least 25 Hz update rate) monitoring of sensor capacitance; second, proof of commercial viability of the proposed sensor which allow substantial saving to the cost of FSR.
The capacitive sensing was performed by a MSP430 microcontroller which was already incorporated into the shoe electronics. The principle of operation was be based on measuring discharge time of an RC circuit in which the capacitor is the pressure sensor. A general-purpose pin in output mode charged the capacitor to a known voltage. Then a timer was started and the pin was switched to input mode. The capacitors discharged though a known resistance R. When the voltage on the capacitor crossed the low threshold voltage of the input pin, an internal interrupt was generated which stopped counting of the internal timer. The captured number of Timer clicks (discharge time) was proportional to the capacitance C. The capacitance C was in the range of between 64.5-387pF. The discharge time in an RC circuit to near ground was approximately TDISCHARGE≈5t≈5RC. Choosing R value to be 1M, the discharge time varied between 322 uS to 1.9 mS, corresponding to sampling frequencies better than 500 Hz. In input configuration, the MSP430 microcontroller had a ±50 nA leakage port current which was negligible compared to the discharge current through resistance R (3 uA at 3V) and thus did not impact the accuracy. The value of the pressure sensor's ESR was taken into consideration if necessary (i.e. if it was high enough to influence discharge time). A 16-bit Timer A was clocked using 16 MHz crystal, which resulted in 5000 to 30400 counts per each measurement (from min capacitance to max capacitance). Resulting discretization of the capacitance was fine enough to capture even minute variations in the weight.
This Task was started in parallel with Task 1 as they were independent. The output of capacity measurement was visualized through a serial connection from the MSP430 development board to a personal computer. At the end of the design phase we were able to capture pressure readings in real time from a person wearing the shoes. The resulting design cost pennies compared to an expensive (several dollars) FSR used in the current design and was considerably more durable (durability will be tested after design-for-manufacture in Phase II). Ultimately this contributed to better affordability of the shoe monitor.
III. Third Experiment (Using Force Sensitive Resistor Pressure Sensors)
A. Shoe Design
Data for the pilot study was collected using a prototype pair of instrumented shoes. The insole of each shoe was equipped with 5 Force-Sensitive Resistors (FSRs). The FSRs were located under the heel, metatarsal bones and the toe. A three-dimensional MEMS accelerometer (LIS3D02AQ) was mounted on the heel of the shoe. Pressure and acceleration data were sampled at 25 Hz and sent over a wireless link to the base computer. The battery, power switch and the wireless board are all installed at the back of the shoe. The whole design was very lightweight and created no interference with normal motion patterns. It should also be noted that this design was very inexpensive (<$100 in mass quantities) and durable.
B. Data Flow in the System for Weight and Energy Expenditure Measurement
Acceleration sensors and/or pressure sensors were incorporated into the shoe or implemented as a shoe insert (insole). A physiological sensor may measure heart or respiration rate. Examples of the physiological sensor are: piezoelectric pulse monitor located on a wrist or an ankle or inside of the shoe system; reflectance optical oximeter detecting oxygenation and/or pulse located on a wrist or an ankle or inside the shoe system; respiration sensor (a plethysmographer) located around the chest. The physiological sensor may have a wired or a wireless connection. The physiological sensor is optional and may be used for higher accuracy of measuring metabolic activity.
Data from the sensors (acceleration, pressure, and optional physiological sensor) was delivered by a wired or a wireless connection to a data processing device. The data processing device may be a dedicated device (i.e. a wrist unit that could also be combined with a physiological sensor, or a personal computer) or a ubiquitous computing device such as a cell phone or PDA. The data processing device applied methods of signal processing such a filtering, normalization and others to condition the sensor signal for further processing. Then the continuous signals were split into short segments (epochs) for which prediction will be made and features of interest were extracted (see Table 5 for an example) such as time-lagged measurements of pressure and acceleration, and/or energy measures (RMS, etc.), and/or entropy measures and/or time-frequency decompositions (short-time Fourier transform, wavelets, etc). The features were representative of the posture/activity and intensity with which a posture/activity is performed. The features characteristic of posture and activity were fed into a classifier that performs recognition of the posture/activity. For example the classifier can be implemented as an artificial neural network such as LIRA (Appendix A), Multi-Layer perceptron or other network. Alternatively the classifier may be a machine learning algorithm such as a linear or non-linear discriminant, parametric or non-parametric model, etc. For example, classifications can be performed by Support Vector Machines or other methods. Features characteristic of intensity of posture/activity were fed into a regression model defined specifically for each posture/activity. The regression model took features and parameters (for example, weight of the person) as inputs and produces estimates of energy expenditure as the output. This output can be summarized in a number of ways (total calories burned, calories per posture/activity, calories above/below the target, daily trends, weeks/monthly trends, etc) and presented as biofeedback to the user. The device can also detect prolonged periods of low activity and cue the user on performing physical exercise.
C. Data Preprocessing
Captured sensor data was processed to form feature vectors for the classifier. Each 800-element feature vector represents pressure and acceleration histories from both shoes for the past two seconds (2 shoes×8 sensors×25 samples per second×2 seconds=800 samples). Thus, all predictions were made for non-overlapping 2-second epochs. Tables 6A-6E show a two-dimensional representations of the feature vectors for each posture/activity. The X-axis shows time progression and Y-axis shows color-coded reading from the sensors in ADC units. First 8 sensors (top half of each image) correspond to the left shoe and next 8 sensors (bottom half of each image) correspond to the right shoe. As Tables 6A-6E presented in
D. Classifier Training and Validation
Twenty-five percent (25%) of the collected dataset was used for training and 75% for validation (reporting of the results). Each posture/activity was represented in the same proportion both in training and validation sets.
Each feature vector was assigned a label representing a distinct class (1-sitting, 2-standing, 3-walking, 4-ascending stairs, 5-descending stairs). The feature vectors and corresponding labels from the training set were presented to a multi-class Support Vector Machine (SVM). SVM is known for robust theoretical foundation and generalization capabilities. Data from the training set were used to train a classifier that would assign a label (1-5) to a presented feature vector.
Finally, the data from the validation set were presented to the classifier. Predicted labels were compared against expected. Multiple experiments were conducted in which the content of training and validation sets was randomly selected from available data. The accuracy of prediction varied from 98% to 100% for multiple randomized trials. These results demonstrate that the proposed device is capable of accurate recognition of a variety of postures and activities.
The goals of this study were: 1) to show the improvement in the accuracy of energy expenditure prediction using shoe-based device over existing methods in the area; 2) to demonstrate the superiority of prediction performance of model using accelerometer and pressure sensors signals over the models that use only accelerometer signals obtained from the wearable shoe sensors; 3) to validate the branched modeling approach for prediction of energy expenditure using shoe-based device; 4) to establish the need of sensors to be embedded in both shoes.
A. Subjects
Sixteen subjects (8 males and 8 females, 18-44 yr, 48.6-119.8 kg, 61-72 in., 18.1-39.4 kg/m2) were included in the study. They were asked to perform the a variety of activities while wearing shoes with sensors. All subjects were healthy with a mean peak O2 uptake (Vo2) of 23.2 ml·min−1·kg−1 (range: 15.18-33.35 ml·min−1·kg−1). Informed, written consent was obtained from each participant before entering the study.
B. EE Measurement
Energy expenditure for each 1-min. activity was measured by indirect calorimetry. In indirect calorimetry the measurements of respiratory gases (oxygen uptake and carbon dioxide production) are used to predict the total amount of oxygen consumed, which in turn is used to estimate energy expenditure. Oxygen uptake and carbon dioxide production was measured during each activity, using a portable gas analysis system. Energy expenditure (kcal·min−1) was converted from predicted oxygen consumption using equivalence of 1 liter of consumed oxygen to 4.78 kcal of energy expended.
C. Movement and Foot Pressure Measurement.
The sensor data for this study was collected by a wearable sensor system embedded into shoes. Each shoe incorporated five force-sensitive sensors embedded in a flexible insole and positioned under the critical points of contact: heel, metatarsal bones and the toe. Such positioning allowed for differentiation of the most critical parts of the gait cycle such as heel strike, stance phase and toe-off. The information from the pressure sensors was supplemented by the data from a 3-dimensional accelerometer positioned on the back of the shoe. The goal of accelerometer was to detect orientation of the shoe in respect to gravity, to characterize the motion trajectory and to help characterize a specific posture or activity (for example, ambulation velocity). Pressure and acceleration data were sampled at 25 Hz and sent over a wireless link to the base computer. The wireless system used for data acquisition was based on Wireless Intelligent Sensor and Actuator Network (WISAN) developed specifically for time-synchronous monitoring applications. Application of WISAN allowed for data sampling at exactly the same time from both shoes, thus avoiding potential complications that could be created in systems with varying time delay between sensors. The battery, power switch and the WISAN board were installed at the back of the shoe. The sensor system was very lightweight and created no interference with motion patterns in subjects.
D. Study Protocol
Each subject was asked to perform a variety of 1-min. activities while wearing the gas mask and the appropriately sized shoe device with embedded sensors. There were 13 different activities from four groups (Sit, Stand, Walk and Cycle) performed by each subject, see Table 7.
The data consisted of 1-min experiments (1 for each activity, for the total of 13 activities) obtained from every subject. Thus, for the 16 subjects that originally participated in the study there were 208 1-min experiments.
The following data were available for every experiment:
Four models were constructed to predict EE in kcal·min−1 using this data. These consisted of two models branched by activity (“Sit”, “Stand”, “Walk”, “Cycle”): branched ACC-PS (included physical measurements, accelerometer and pressure sensors predictors) and branched ACC (included physical measurements and accelerometer predictors), and also two nonbranched models with sets of predictors corresponding to the branched versions: nonbranched ACC-PS and nonbranched ACC. The purpose of constructing the models was to investigate if the performance was improved by branching the model and also by including pressure sensor predictors.
Accelerometer and pressure sensors signals were preprocessed to extract meaningful metrics to be used as predictors for the model. For each sensor, the following metrics were tested for the inclusion into each model as predictors: coefficient of variation (cv); standard deviation (std); coefficient of variation (cv); frequency which is computed as the number of “zero crossings,” i.e. the number of times the signal crosses its median (frq) normalized by the signal's length; entropy Hof the distribution X of signal values (ent) computed as:
H(X)=−Σpk log pk.
For each model, the derived metrics were used as possible predictors for the ordinary least squares (OLS) linear regression. The transformed predictors (log, inverse and square root) and interactions (as products of 2 or more candidate predictors) were also considered as separate linear terms within regression.
In branched models, a separate branch model was constructed for each identified posture activity: “Sit”, “Stand”, “Walk” and “Cycle”. For each model (branch activity models and nonbranched models), selection of the most significant set of predictors was performed using the forward selection procedure. We used the “leave-one-out” approach for cross-validation when training and predicting the EE for each experiment for every subject. Within each model there were several activities performed by each subject, all of such experiments related to the same subject were removed from the training set. A model (coefficients) computed using the rest of the subjects was then used to predict the EE for all experiments of the left out subject. The best set of predictors had to provide the best fit (by producing the maximum adjusted coefficient of determination, R2adj and the minimum Akaike Information Criterion, AIC) in the training step and the best predictive performance (the minimum mean squared error, MSE and the minimum mean absolute error, MAE) in the verification step.
Originally, the study included 16 subjects. As a result of data quality analysis, it was detected that subjects 6, 8, 11, 12, 13 had pressure sensors failure on both shoes in at least one activity group experiments and, therefore, they were completely excluded from the analysis. Thus, the input for the model was the set of 1-min experiments from 11 subjects (4 males and 7 females). The summary statistics of the physical characteristics of the 11 subjects used for subsequent model construction are shown in Table 2. In the “walk” activity group, some subjects did not have an energy expenditure record or had no sensors recorded for some experiments within this group. These 4 experiments were dropped from each model's input. Thus, the sample size of the input data for each model was (11×13)−4=139 experiments.
Measured and predicted energy expenditure values in kcal·min−1 for each experiment were then converted to METs (kcal·kg−1·hour−1) for both branched ACC-PS and ACC models and their nonbranched versions.
One of the goals of the analysis was to establish the need of using sensors on both shoes. Preliminary analysis showed that metrics derived from the difference between the signals from left and right shoes exhibited good predictive power for the model. Namely, the frequency (i.e. the number of “zero crossings”) of the difference signal between left and right shoe sensors for pressure sensor 4 showed improvement if included into the “Walk” branch model. Several versions of the branched ACC-PS model (as a representative model) were constructed using accelerometer and pressure sensors data separately from each shoe and both shoes together.
E. Statistics
The following performance assessment measures were computed for each model predicting energy expenditure per experiment in kcal·min−1 or METs):
RMSEMET—the root mean squared error for energy expenditure prediction expressed in METs. This error is computed as the difference between model predicted EE and the measured EE for each experiment.
ARD—the Average Relative Difference (signed):
ARD=mean((predEE−EE)/EE)
AARD—the Average Absolute Relative Difference:
AARD=mean(|predEE−EE|/EE)
RMSE%—the RMSE expressed as the percentage of the mean measured energy expenditure (in METs)
Bias—the mean difference between predicted and measured energy expenditure in METs:
bias=mean(predEE−EE)
Interval of agreement—a prediction of energy expenditure in METs: (bias±2·SD(bias))
A Bland-Altman plot analysis was conducted to reveal any systematic pattern of the error (calculated as the difference between predicted and measured EE) across the range of measurements (as the mean of predicted and measured EE) and to assess the bias and interval of agreement for prediction of EE.
Passing-Bablok regressions (as a robust alternative to least squares regression) for all four models and for two units of prediction (kcal·min−1 and METs) were constructed. Passing-Bablok regression is best suited for method comparison because it allows measurement error in both variables, does not require normality of errors and is robust against outliers. In addition, Passing-Bablok regression procedure estimates systematic errors in form of fixed (by testing if 95% CI includes 0) and proportional bias (by testing if 95% CI includes 1).
F. Results
First, the effect of inclusion of predictors was investigated from both shoes into the model using branched ACC-PS model as an example.
Table 9 below shows comparative performance of the models that used the best selected set of predictors (cv, std, frq and ent, computed separately for each shoe) and the difference metrics derived from the difference between signal form left and right shoe. Each model's performance is reported as the aggregated results from four branch models (“Sit”, “Stand”, “Walk” and “Cycle”). “Mean” and “Max” models used respectively mean and maximum values of all predictors (accelerometer and pressure sensors), “Left” and “Right” models used only signals from left or right shoe. The “difference” model included the difference metric in addition to the previously selected set of predictors. As Table 3 indicates, there was almost no improvement provided by inclusion of the difference metrics when compared to the rest of the models. Overall, models based on metrics derived for both shoes (“Mean”, “Max” and “Difference”) performed slightly better than single shoe models. However, this improvement is not significant and for all practical purposes single shoe models can be successfully used instead. In addition, it should be noted that in the data set some subjects had pressure sensor failure on one of the shoes (resulting in derived metrics being close to 0). This loss of information can explain poor results for single shoe models reported in Table 9. On the other hand, such failures did not particularly affect either “Mean” or “Max” since it was accounted for during either averaging or, especially, maximum value selection.
The rest of the results reported here correspond to ACC-PS, ACC branched and nonbranched models constructed using mean of {left, right} accelerometer metrics and maximum of {left, right} pressure sensors metrics as an approximate “single shoe” model.
As a result of selection of the best set of predictors final branch models within branched ACC-PS and branched ACC models with included predictors and coefficients are reported in Table 10 and Table 11, respectively. Final nonbranched ACC-PS and nonbranched ACC models are given in Table 12. The coefficients for all models were obtained by averaging the coefficients of the 11 runs (one for each left out subject) of the OLS regression on the training sets. Almost all coefficients for all models were highly stable over all runs as given by low absolute values of coefficients of variation (CV). As can be expected, weight and BMI always explain part of the variability of each model, other physical characteristics were highly correlated to weight variable and didn't add to the fit or the prediction performance of either model.
Results shown in Table 12 include performance comparison of the proposed branched ACC-PS model, branched ACC model, nonbranched ACC-PS, nonbranched ACC and several existing models reported from recent studies on energy expenditure prediction. As described above, these results indicate performance by experiment where sample size is equal to the total number of experiments from all subjects in all activity groups.
As shown in Table 13, both branched models (ACC-PS and ACC) outperform existing models in several performance assessment measures. Branched models also exhibit significantly better prediction performance than nonbranched models in all of the reported characteristics. The same improvement in performance is shown when comparing branched or nonbranched ACC-PS models to ACC models. At the same time, both nonbranched models achieve almost the same level of performance as the existing models found in the literature, (as indicated by similar levels of RMSEMET, bias and 95% intervals of agreement).
Bland-Altman plots (constructed for both EE, kcal·min−1 and EE, METs prediction) for all four shoe-based models are shown in Tables 14A-14H in
Passing-Bablok regression analysis for the four shoe-based models was conducted using Matlab. Results of this regression analysis are depicted in the plots of Tables 15A-15H as shown in
Bias at mean, minimum and maximum measured EE values (as percentages of these values) was also evaluated using obtained Passing-Bablock regression equations (Tables 15A-15H). Both branched models (ACC-PS and ACC) showed significantly better accuracy of prediction than other reported models; bias at mean was 1.49% vs −5.84%. In addition, the ACC-PS model also showed improved results upon ACC model. (See Tables 16 and 17.) Nonbranched models revealed significant bias (10.48-16.52%) at the minimum measured EE values.
*p-value
#Bias is calculated from the Passing-Bablok regression line Ŷ = a + bX, where Y is the predicted value and X is the criterion. Reported bias values are computed as (Ŷ(Xi) − Xi)/Xi for the criterion energy expenditure Xi as mean, minimum, and maximum values observed in the sample.
The prediction of energy expenditure was estimated by subject as indicated in Table 18. Total energy expenditure (TEE) for each subject was computed as the sum of energy expenditures (in kcal·kg−1) over all 1-min activities/experiments extrapolated over 780 min. proportionally to the original time allocated to each activity (2:2:6:2). The 780 min. was chosen as the length of a hypothetical 13-hour active wake cycle. The value of TEE was then normalized by an individual's weight. Initially, walking experiments included 7 activities (walk 1.5, walk 2.5, walk 3.5, jog 4.5, uphill, downhill, loaded). Four subjects had missing data for the jogging experiment, and, thus, the jogging was dropped in calculation of the TEE.
As can be seen from Table 18, the branched ACC-PS model performed slightly better than models constructed using accelerometer and heart rate, achieving 9.35% SEE versus 9.89%. The difference in the performance can be attributed to the difference in study protocols, in particular, different distribution of activities. Nevertheless, a branched ACC-PS model achieves accuracy in prediction similar to that of a branched model.
Although nonbranched models showed low SEEs, they provided biased estimates (as indicated by the greater deviation of the mean predicted TEE from the mean measured TEE) than the branched models.
G. Discussion
An earlier developed wearable shoe-based device with embedded accelerometer and pressure sensors was used for prediction of energy expenditure. The signals obtained from the shoe-based device was previously used to classify postures and activities into four groups: sit, stand, walk and cycle. A model was proposed that branched according to these actual performed groups of activities to be able to later combine the activity classification algorithm with this energy prediction model.
The proposed branched model that used both accelerometer signals and pressure sensors signals (branched ACC-PS) significantly improved the accuracy of prediction upon the branched model based solely on accelerometer readings (branched ACC) achieving root mean squared error (RMSE) of 0.66 METs vs 0.73 METs. In particular, the improvement was most significant due to the decrease in error rate in Stand and Cycle branch models. Both branched model outperformed existing methods based on accelerometry, heart rate and branching. Introduction of pressure sensors provided valuable information which also made a positive impact on the prediction of nonbranched ACC-PS versus nonbranched ACC models.
Comparison of branched ACC-PS and ACC models to their nonbranched versions suggested that branching considerably improves the prediction by lowering systematic bias, error rates and the width of the interval of prediction. In addition, the results of performance of the nonbranched models showed that they achieve accuracy comparable to that of the existing studies.
The quality of prediction provided by the model that uses the data from both shoes is not significantly different from the model that used single shoe data. Therefore, for all practical purposes, the use of single shoe embedded sensors is validated.
Although the embodiments have been described with respect to particular apparatuses, configurations, components, systems and methods of operation, it will be appreciated by those of ordinary skill in the art upon reading this disclosure that certain changes or modifications to the embodiments and/or their operations, as described herein, may be made without departing from the spirit or scope of the present disclosure. Accordingly, the proper scope of the present disclosure is defined by the appended claims. The various embodiments, operations, components and configurations disclosed herein are generally provided as examples rather than limiting in scope.
The present application claims priority to U.S. Provisional Patent Application No. 61/208,196, filed 20 Feb. 2009, entitled, “Footwear-based System and Method for Monitoring Body Weight, Postural Allocation, and Energy Expenditure,” U.S. Provisional Patent Application No. 61/256,132 filed 29 Oct. 2009, entitled “Shoe-based Wearable Sensor for Monitoring Body Weight, Postural Allocation, and Energy Expenditure,” and U.S. Provisional Patent Application No. 61/266,319, filed 3 Dec. 2009, entitled “A novel wireless, wearable shoe-based system for weight and physical activity management”, each of which is incorporated by reference herein in its entirety.
This technology was developed in part with sponsorship by National Institutes of Health Grant No. 1R43DK083229-01A1 and the U.S. federal government may have certain rights to this technology.
| Number | Date | Country | |
|---|---|---|---|
| 61208196 | Feb 2009 | US | |
| 61256132 | Oct 2009 | US | |
| 61266319 | Dec 2009 | US |
