Patent Application
20170122769
Field
This application relates to mobile and wearable devices, specifically to methodologies to leverage user's gait characteristics.
Discussion of Related Art
Sensors within mobile and wearable devices allow monitoring of some physical activities. Current technologies and devices enable activity recognition with certain degree of accuracy, depending on the type and quality of the sensors and methodologies employed, among other factors. However, commonly available devices may suffer from important inaccuracies due to variations in movement attributes during the physical activity. Particularities in gait characteristics and other conditions may add to the difficulties that some devices face to accurately recognize activities. There is a need to efficiently leverage the sensors embedded in mobile and wearable devices to accurately determine gait characteristics and enable new applications.
Although the following text sets forth a detailed description of numerous different embodiments, it should be understood that the legal scope of the description is defined by the words of the claims set forth at the end of this disclosure. The detailed description is to be construed as exemplary only and does not describe every possible embodiment since describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims.
It should also be understood that, unless a term is expressly defined in this patent using the sentence “As used herein, the term ‘ ’ is hereby defined to mean . . . ” or a similar sentence, there is no intent to limit the meaning of that term, either expressly or by implication, beyond its plain or ordinary meaning, and such term should not be interpreted to be limited in scope based on any statement made in any section of this patent (other than the language of the claims). To the extent that any term recited in the claims at the end of this patent is referred to in this patent in a manner consistent with a single meaning, that is done for sake of clarity only so as to not confuse the reader, and it is not intended that such claim term be limited, by implication or otherwise, to that single meaning.
Some inventive functionality and inventive principles may be implemented with or in software programs or instructions and integrated circuits (ICs) such as application specific ICs. In the interest of brevity and minimization of any risk of obscuring the principles and concepts according to the present invention, discussion of such software and ICs, if any, is limited to the essentials with respect to the principles and concepts within some of the embodiments.
In some embodiments,
In a particular embodiment, any number of dashboard elements (120) and/or (130) may be displayed; in other embodiments, these and/or any other elements may not be displayed, while the presence of any or all of these elements may be temporary depending on a plurality of criteria, such as device power management, adaptation of screen space to particular conditions, user's choice, or any other reason. In some embodiments, the properties of any or all the elements displayed may be fixed, while in other embodiments, the properties of any or all the elements displayed may vary depending on a plurality of criteria, such as device power management, adaptation of screen space to particular conditions, user's choice, results of interaction, or any other reason. By way of example without limitation, properties in the elements that may be varied at any time and for any reason include, shape, color, update rate, purpose, ways of indication (needle, bar, line, etc.), location in the screen, transparency, capability of being interactive in any way, degree of interaction, or any other feature.
In some embodiments, element (110) may represent any means of transportation, including by way of example without limitation, any transportation means by ground (e.g. car, truck, van, train, horse pulled wagon, motorcycle, bicycle, skateboard, snowboard, sledge, etc.), any transportation means by air (e.g. airplane, military fighter, rocket, balloon, glider, etc.), any transportation means by space (e.g. spaceship, rocket, etc.), any transportation means by water (e.g. ship, boat, kayak, submarine, etc.), or any other tool, device or object of any nature capable of moving in any way through any means, including hybrids and/or mixtures and/or combinations thereof; in some embodiments, any aspect or property related to element (110) may be controlled by any characteristic of the user's movement or gait.
In other embodiments, element (110) may represent any kind of human being (including him/herself) or groups of human beings, or any kind of non-human beings or objects of any nature and form, including, by way of example without limitation, any type of animal or groups of animals, any type of robots, any type of virtual, imaginary or fantasized being or object, of any nature, shape, color, size or any other properties, and any mixtures and/or combinations thereof.
In some embodiments, one or more elements (110) may be displayed with the same or similar properties; in other embodiments with more than one element (110), they may have different properties; in other embodiments, the properties of one or more elements (110) may be varied at any time depending on a plurality of criteria, including, by way of example without limitation, user's choice, results of interaction, device power management, adaptation of screen space to particular conditions, or any other reason. In some embodiments, the properties that may be varied include, by way of example without limitation, shape, color, form, nature, purpose, ways of movement, different capabilities (including capability of being interactive in any way), location in the screen, transparency, degree of interaction, or any other feature or property.
In some embodiments, element (140) may represent a road along which element (110) moves, or any kind of entity of any nature capable of allowing any type of movement; in other embodiments, element (140) may represent any kind of means, including by way of example and not limitation, any kind of ground road, highway, path, snow covered field, ice platform, river, cross country fields, outer space, racing means and/or environments of any type and nature, any kind of means of any nature (e.g. solid, liquid, gas, plasma) where any element (110) may perform any type of movement, or any kind of real, imaginary or fantasized entity or substance, and any mixture and/or combinations thereof.
In some embodiments, element (150) may represent traffic lights or any kind of object or being of any nature capable of influencing, regulating, monitoring and/or controlling the movement of element (110) in any way; in other embodiments, element (150) may represent any type of object or being of any nature capable of catching the attention of individual (101) or (111); in other embodiments, element (150) may represent any type of object or being of any nature capable of interacting with element (110) and/or individual (101) or (111) in any way. By way of example and not limitation, in some embodiments element (150) may represent any type of element (similar or different to any type of element (110)) moving in the same environment and being able to interact with element (110) destructively (e.g. enemy car shooting at wheels, enemy zombie in the middle of the road, enemy truck spilling oil on the road, etc.) or constructively (e.g. safety truck for rescue operations, friendly car offering prizes, friendly gnome offering rewards, etc.). In some embodiments, interactions of some element (150) with individual (101) or (111) may include, by way of example without limitation, acoustic signals, visual messages, vibration of the device for a certain amount of time, etc. In other embodiments, element (150) may represent any type of object or being of any kind or nature, including, by way of example without limitation, obstacles, man waving flag indicating distance travelled, road signs, etc. In other embodiments, the number of elements (150) displayed on the screen may be any (including more than one), and they may have any nature (including real, virtual, fantasized, etc.) and/or properties, including mixtures and/or combinations thereof.
In other embodiments, the number of individuals participating in a networking environment may be any, each one of the individuals may have any number of devices of any type positioned/attached/located/worn on any place, and each one of the individuals may perform any type of walking, jogging, running, sprinting, or any other type of activity regardless of the type of device and/or their position. In some embodiments, the individuals (201), (204), (206), (208) and/or any other number of individuals participating in a networking environment, may all be physically located next to each other in approximately the same location. In other embodiments, one or more (or all) individuals may be physically located in different and/or distant locations. By way of example without limitation, each individual could be in a different city.
In some embodiments, communication between devices (202), (203), (207), (205) and/or others to enable the networking environment may leverage any means, including, by way of example without limitation, any wireless and/or any other type of communications technology, such as LTE, UMTS, GSM, WiFi, Bluetooth and/or any other kind and combinations thereof. In some embodiments, the means of communications and/or their properties may be varied at any time and depending on any reason. These changes and/or choices may depend on a plurality of factors, including, by way of example without limitation, network availability, physical proximity of individuals, power management, communication efficiency, specific usage plans, etc. In some embodiments where the devices are in close proximity, they may communicate directly with each other using any kind of short-range communications technology or any other type of means and/or technology without the need to relay on other communications networks elements such as cellular base stations or WiFi access points.
In some embodiments, the networking virtual environment illustrated in
In some embodiments, the networking environment in which individuals (201), (204), (206) and (208) participate may be intended to generate a representation of a virtual environment in the displays of their devices, which would be shared by all individuals (201), (204), (206) and (208) regardless of their physical location, type of device, positioning of the device, activity that the individuals are performing, or any other condition. In a particular embodiment represented in
In some embodiments, scene (250) may be composed of any or all of the elements described in
In some embodiments, the different individuals participating in the networking environment and/or elements representing them (e.g. the different individuals in
In some embodiments, all the individuals participating in the networking environment are assumed or expected to perform the same gait activity or activities, and their experiences in the virtual environment will be influenced by characteristics/metrics/variables of their gait activities. In other embodiments, the individuals participating in the networking environment may be carrying out any type of activity (different individuals may also perform different activities) and the type of activity performed by the individuals may influence their representation in scene (250) and/or their overall experience in the virtual environment. In other embodiments, the individuals participating in the networking environment may be carrying out any type of activity regardless of their experiences and/or representation in the virtual environment. Any variations and/or combinations are also possible.
In some embodiments, the individuals participating in the networking environment may be moving in their real environment in any direction and/or combinations of directions regardless of their experiences and/or representation in the virtual environment. In other embodiments, the real environment direction of movement of the individuals participating in the networking environment may influence their experiences and/or representation or any other aspect in the virtual environment. The same applies to any individuals from
In some embodiments, the individuals participating in the networking environment may be moving in their real environment with some measurable characteristics (including, by way of example without limitation, speed, stride length, cadence, acceleration) that may influence their experiences and/or representation in the virtual environment in any way; the same is applicable in cases where an individual is on his/her own interacting with any virtual environment as described by any or all of
In some embodiments, all devices participating in the networking environment will display the same representation of the virtual environment; in some embodiments, the representation of the virtual environment in all devices will be the same or similar, with possible adaptations to any specificity of the devices (by way of example without limitation: any adaptation due to availability (or lack of availability) of hardware in the device, any adaptation due to operating system or any software characteristics in the device, any scaling to adapt to the display size of the device, etc.); in some embodiments, any or all of the devices participating in the networking environment may have a different representation of the virtual environment, depending on different criteria (including, by way of example without limitation, display size, power management, user's choice, activity type, activity characteristics, device positioning, user location, or any other reason and/or combinations thereof).
In some embodiments, all the individuals participating in the networking environment have the same representation in the virtual environment (e.g. all individuals in the networking environment are represented in the virtual environment by the same type of cars, or by the same type of motorcycles, or by the same type of airplanes, etc.). In other embodiments, any or all of the individuals participating in the networking environment may be represented in the virtual environment by different types of elements (e.g. one individual may choose to be represented in the shared networking virtual environment by a car, while other individual may choose to be represented in the shared networking virtual environment by a motorcycle, other individual may choose to be represented in the shared networking virtual environment by a leopard, other individual may choose to be represented in the shared networking virtual environment by an athlete, etc.). In other embodiments, any combinations of the previously described possibilities of representation of individuals in any virtual environment are applicable to any type of environment, including, by way of example without limitation, the possibility of selecting and/or changing the representation of any or all of the individuals in any way, at any time and because of any reason.
In some embodiments, the presentation of any or all of the elements (and/or any information about them) in any environment may be scalable and/or scrollable and/or modifiable and/or sortable and/or organizable in any way depending on any reason. By way of example without limitation, a very large number of individuals participating in a networking environment may trigger a process to organize information about the individuals in a scrollable view.
In some embodiments, the properties of any or all the elements described or any other features of any embodiment may vary depending on a plurality of criteria, such as device power management, adaptation of screen space to particular conditions, user's choice, results of interaction, or any other reason. By way of example without limitation, the properties or features that may be varied at any time and for any reason include: shape, color, update rate, purpose, location in the screen, transparency, capability of being interactive in any way, degree of interaction, or any other and combinations thereof. Also, any descriptions, modifications and/or combinations thereof are applicable to any of the figures, embodiments and/or any of their elements.
In some embodiments, any contextual information may be displayed directly on the user's device display. By way of example and not limitation, the velocity of the user may be displayed in real time (typically, fractions of a second) on the mobile device display as shown in
In some embodiments, the representation of a moving needle (320) may be leveraged to indicate the real time velocity of the user. In other embodiments, other representations may be leveraged to indicate the real time velocity of the user, including but not limited to, the surface of a varying semicircle whose angle grows from 0 degrees to 180 degrees depending on the velocity. In other embodiments, semi-arcs or other types of geometries, shapes, sizes, figures, etc. may also be leveraged. In some embodiments, combinations geometries and/or color may also be leveraged to display the velocity information. In some embodiments, the presentation of information to the user or to any type of managing or monitoring entity may be performed personalized and in any of several ways including, by way of example, and not limitation, visual, acoustic, etc. For example, a button for sound (340) may be used to enable or disable the acoustic delivery of contextual information. This button may also be leveraged to enable or disable playing music or other encouraging sound in the background, or to trigger an out-loud-reader mechanism to read-out-loud contents on the display (e.g. text from a website, messages received from friends, etc.) when predetermined and/or selectable thresholds or levels on the user's velocity or general context are reached. Another button may be used to change the units of the velocity (350), for example, meters per second, kilometers per hour, etc. In some embodiments, automatic localization or other means may be leveraged to infer the country of the user and automatically adapt units, language, and other variables. Additional buttons (360) may also be employed for other purposes, including but not limited to, displaying a time evolution of the user velocity, dynamics, or general context over a selected or available period of time, allow personalized calibration, set preferences, etc.
In some embodiments, any element(s) described for any figure or embodiment may be optional, or any of them and any other additional element(s) with any features and/or combinations thereof, may also be included in any fashion in any figure or embodiment.
In some embodiments, element (410) is an optional label and may provide the name of the gait characteristic being displayed together with any form of any possible units in which it may be measured (if applicable), or any other type of information of any nature, including abbreviations; particular examples may include: mph, miles per hour, kmh, kilometer hour, m/s, meter per second, etc. In other embodiments, if this label is included, depending on a variety of circumstances/conditions/choices, it may present any type of statements (e.g. “velocity (kmh)”, “speed (mph)”, “speed (kmh)”, “stride length (m)”, “stride length (ft)”, etc.), forms, shapes, positions, nature (e.g. any picture, icon, multimedia element, etc. and combinations thereof), or any other property and/or combinations thereof. In some embodiments, element (420) is also optional and it represents chart axes or grid for clarity purposes (any of its components is also optional and may have any characteristic); in some embodiments, the vertical axis may be scaled in any way depending on the gait characteristic being displayed, and hold a set of representative figures together with any type of unit, statement, or any other element of any nature, form, or any other characteristic, and arranged/aligned/distributed in any way; in a particular embodiment, the vertical axis is scaled from 0 to 6 in consecutive numbers (representing units of velocity in miles per hour), and horizontal lines may cross the chart for each one of the presented numbers.
In some embodiments, the scale may additionally include a variety of features, such as the preferred walking velocity or others. These features may be average values or personalized values for each particular user. Other embodiments may use any other types of features and/or combinations thereof. In other embodiments, any or all of the horizontal bars and/or numbers along the vertical axis and/or any other element may be optional (e.g. it may not be displayed at any time for any reason) and if they are displayed, any or all of the referred elements or any other of any type that may also be added, may present any properties/features and combinations thereof.
Element (430) represents the measurement of the gait characteristic or attribute or any other related information being displayed. In a particular embodiment, it is a continuous line or curve (linking points ordered in time, each point corresponding to each measurement of the gait characteristic, e.g. velocity) freely following the measurements, having up to a predetermined threshold in the number of points, and accepting a new point to be displayed appended in a continuous form to the right edge of the curve every time a new measurement arrives. When the threshold in the number of points is reached, every time a new measurement arrives, the first point from the left edge (which in this embodiment represents the oldest measurement) is discarded, and the rest of points (except for the new one included at the right edge) are offset one position towards the left, thus giving the impression of a continuous flow of points following the arriving measurements. In some embodiments, the threshold in the maximum number of points in element (430) may be set to a fixed amount (e.g. a hundred or any other number), while in other embodiments it may be variable and depend on a variety of factors/circumstances/conditions, user's choices or any other reason. In some embodiment, any other type of indication and combinations thereof may be used instead of simple points or dots placed at the actual measurement value, such as, by way of example without limitation, any number of stars, squares, diamond shaped icons, any other type of polygon/icon/drawing/entity/element, any type of dotted lines/curves, any type of line/curve from any edge of the chart (or any other place) to the actual measurement value, any type of rectangle or any other polygon, icon, drawing, entity, element covering an area from any edge of the chart (or any other place) to the actual measurement value, or any other element(s) with any properties distributed and/or organized in any way, including any modifications and/or combinations thereof. In some embodiments, the indications may represent any type of information, including by way of example without limitation, the actual raw measurement of the gait characteristic being displayed, any value derived from the raw measurement or from any group of measurements (e.g. mean, standard deviation, etc.), or any other value, information, processed data or any other element in any way related with the gait characteristic and combinations thereof.
In some embodiments, the frequency at which a new point (or any indication of any type corresponding to measurements or any other data) is introduced in element (430) may be the frequency at which a new measurement is generated. In a particular embodiment presenting (by way of example) velocity, the use of (by way of example) methodology based on the application of the wavelet transform to the acceleration signal, would allow a new measurement every time a new acceleration value is available; consequently, the frequency at which a new measurement is generated may be equal to the accelerometer sampling frequency; in other words, the frequency at which the gait characteristic is updated may be equal to the accelerometer sampling rate, which in some embodiments may be higher than the user's step frequency. In some embodiments, other frequencies (lower or higher) may also be possible making use of different techniques, including by way of example without limitation, the use of any extra device, hardware, software, up sampling, down sampling, filtering, or any other techniques, tools and/or methodologies and any variations and/or combinations thereof. By way of example without limitation, in some embodiments the update frequency for the gait characteristic may be 60 Hz or 120 Hz depending on device hardware and other circumstances/conditions/choices, therefore achieving an enhanced real-time presentation of information (and user experience) in comparison with other methods with lower update rates; in some embodiments, when the user's step frequency is below 1 Hz (e.g. 0.5 Hz), the update rate may also be chosen just above the user's step frequency (e.g. 0.6 Hz), or above 1 Hz, or set as the accelerometer sampling rate (e.g. 60 Hz or 120 Hz) to enhance the real-time presentation of information (and user experience); other embodiments may choose any other update frequency or characteristic by modifying any settings, conditions, and/or choices of the referred and/or any other method. Other embodiments may employ any modification to any aspect previously mentioned, and/or combinations thereof.
In some embodiments, the presentation of information to the user or to any type of managing or monitoring entity may be performed personalized and in any of several ways including, by way of example, and not limitation, visual, acoustic, etc. For example, a button for sound (440) may be used to enable or disable the acoustic delivery of contextual or any other type of data/information (including by way of example without limitation, any kind of multimedia streaming and combinations thereof). This button may also be leveraged to enable or disable playing music or other encouraging sound in the background, or to trigger an out-loud-reader mechanism to read-out-loud contents on the display (e.g. text from a website, messages received from friends, etc.) when predetermined and/or selectable thresholds or levels on the user's velocity or general context are reached. Another button may be used to change the units of the gait characteristic being displayed (450), for example, velocity in meters per second, kilometers per hour, etc. In some embodiments, automatic localization or other means may be leveraged to infer the country of the user and automatically adapt units, language, and other variables. Additional buttons (460) may also be employed for other purposes, including but not limited to, displaying the gait characteristic in different format, or displaying different information, set preferences, modify any aspect or property of the presentation and/or any application, etc. and combinations thereof.
In a particular embodiment, the background of the display/screen in
In some embodiments, the gait/motion parameters or characteristics that may be determined/calculated/estimated/inferred include, by way of example without limitation, speed, stride length, cadence, total distance, pace, gait efficiency, energy, power, changes in acceleration, speed variability, strike time, steps, and any combination thereof. In some embodiments, any number of gait/motion parameters and/or any other information may be leveraged to determine additional gait/motion parameters in any way; by way of example without limitation, physics principles may be used to determine distance (e.g. stride length) from velocity, and other parameters or characteristics that may be obtained in this or other fashion include energy consumption, different types of costs, etc. In some embodiments, any variations of any said characteristics or parameters and/or combinations thereof may also be determined in any fashion, and any user's characteristic such as height, weight, gender, age, etc. may also be used to help in the determination of the motion or gait parameters.
Some embodiments may test if the user is performing any type of gait activity (520), leveraging any of the characteristics/data/methodologies herein mentioned, or through any other methodology; in some embodiments, the type of user's movement that the system tries to recognize in (520) may include any activity that may be classified as human gait, in other words, any gait activity, including, by way of example without limitation, any type of walking, jogging, running, sprinting, ascending or descending stairs, exercising on any apparatus such as stationary elliptical trainer or bicycle, and any variation and/or combination thereof regardless of forward/backward direction, flat/inclined surface, type of environment, etc. In some embodiments, any gesture or movement different from walking, jogging or running may not be considered as a gait activity. In other embodiments, the user's movement to be recognized by the system in (520) may include any type of movement and/or activity. By way of example without limitation, a particular embodiment may consider walking, jogging, or running as gait activity. Any other variation and/or combination may also be possible.
As a result of the test in (520), in case of affirmative answer, some embodiments may enable any application and the use of gait characteristics for control (540). By way of example without limitation, regarding (540), some embodiments may enable or proceed with the triggering, launching, initiation, continuation, pausing, displaying, controlling in any way, interrupting, terminating, or any other action or procedure or combinations thereof of any process, function, procedure, program, application, environment or any other entity or element and/or combinations thereof, while any or all of the user's gait characteristics may be leveraged to control any aspect, feature, condition, property or any other attribute of any said element(s) in any way. In a particular example of embodiment, once the mobile or wearable device recognizes that the user is walking with, for example, a determined velocity, cadence and/or stride length, the device may enable and/or display a virtual environment like the one represented in
In some embodiments, the determined user's gait parameters, variables, or characteristics may also control any type of avatar of any form (including human, animal, object, etc.), any virtual environment, any aspect or object or element of any virtual environment, etc. and combinations thereof. In some embodiments, a virtual environment may include, by way of example without limitation, a representation of any setting in which the user perceives himself/herself to be and within which interaction takes place; in some embodiments, the representation may be three-dimensional; in some embodiments, the representation may have any number of dimensions, use any type of technology, presentation and/or projection methodology/means and combinations thereof. In some embodiments, a virtual environment may also refer to any computer-based simulated environment allowing for any number of users, where the environment and/or their rules may draw from any reality and/or fantasy worlds and/or combinations thereof. In some embodiments, other examples of virtual environments may include, without limitation, any type of games, computer conferencing, chat-rooms, shared spaces, virtual reality, augmented reality, multi-user chat systems, mixed reality environments, multi-user games, multi-user games with persistent storage features, multi-user interactive environments, immersive environments, collaborative virtual environments, any other form of virtual habitats, etc. and/or combinations thereof. By way of example without limitation, in a particular embodiment, a game object may be controlled by characteristics of the user's gait in any fashion; for instance, a player may increase or decrease the speed of a car (e.g. element (110) in
In some embodiments, if the answer to the test in (520) is negative, the user will be prompted, or communicated in any way (e.g. text, visual message, acoustic message, mechanical vibration, etc. and combinations thereof) about the need to start a gait activity (530) in order to enable the application, proceed to the next level, continue a process, etc. as described for (540). The system may continue processing sensor data determining motion characteristics and activity, and keep communicating the user to start gait activity until a gait activity is recognized. In some embodiments, the system may employ a predetermined time as a threshold (a value which may be constant or variable depending on user's context, choice, etc.), after which, if the user has not initiated a gait activity, the whole process/application/procedure or some aspect of it may be terminated and/or some element may be disabled. In other embodiments, any condition, event, action, etc. may be used to end this loop and terminate the whole process, continue to the next step in some way, or any other possibility depending on any reasons/circumstances and combinations thereof.
Some embodiments may allow any variations, modifications, additions, eliminations, etc. and/or combinations thereof.
For the purpose of obtaining the dynamics of the user through the processing of sensor(s) signal(s), some embodiments may select an appropriate sampling frequency, which optimizes performance and attempts to minimize power consumption. In some embodiments, it may not be possible to set a desired sampling frequency (620). By way of example, and not limitation, some operating systems may allow the selection of predefined sampling frequency levels, which may work as indicators of the final sampling frequencies, but there is no guarantee of obtaining a specific frequency value. In fact, the final sampling frequency values may also be device and hardware specific. In conclusion, the algorithm in some embodiments will need to adapt to the available sampling frequencies in each particular device. In this sense, the sampling frequency may be selected (630) taking into account two criteria: first, performance optimization; second, power consumption minimization. In fact, optimum performance may depend on the sampling frequency among other factors. In some embodiments, the quality of the results obtained through the application of the wavelet transform to process the sensor(s) (e.g. accelerometer) signal(s) may depend on the sampling frequency. Once the desired or available sampling frequency has been selected, that frequency is set in the device (640). Some embodiments may use single axis sensor information to be processed (by way of example and not limitation, acceleration in x-axis, acceleration in y-axis, acceleration in z-axis). Some embodiments may use the signal vector module to be processed (by way of example and not limitation, the signal vector module of a tri-axial accelerometer). Some embodiments may use different configurations and/or combinations of sensors signals (including but not limited to sensor fusion information) to be processed. It must be noted that in some embodiments, the set frequency may still vary depending on a variety of factors, including but not limited to, device-specific behavior. Consequently, in some embodiments, a frequency resetting procedure may be necessary to maintain desired performance. Some embodiments may use dynamic selection of sampling frequency; by way of example and not limitation, when periods of inactivity are detected, the sampling frequency may be reduced in order to minimize power consumption, and once some activity is detected again, the sampling frequency may be increased again to deliver desired performance.
In some embodiments, the selection of the transformation parameters to process the sensor(s) signal(s) may take place after the sampling frequency is set (650). In some embodiments, the wavelet transform may be applied for processing sensor(s) signal(s). In other embodiments, other transformations may be applied, including but not limited to, short-time Fourier transform, other techniques leveraging Fourier analysis, application of filter banks, etc. In other embodiments different combinations of techniques, methodologies and transformations including wavelets maybe used. In some embodiments, the parameters of each transformation, which by way of example and not limitation, may comprise levels of decomposition, mother wavelet, processing time window parameters, etc. may be set appropriately/dynamically to optimize performance and minimize computation burden.
In some embodiments, the appropriate transformation coefficients may be obtained (660) and be leveraged in subsequent processes in combination with other parameters and metrics (670). In some embodiments, the application of metrics with the previously obtained information results in excellent correlations with the velocity of the user, and the activity of the user (e.g. walking, running, jumping, etc.), leading to a characterization of the user dynamics (680). In some embodiments, by way of example and not limitation, weighted (e.g. by levels, number of coefficients, etc.) energies of wavelet transform coefficients may provide an excellent indicator to directly choose the appropriate coefficients from which to obtain a reconstructed wave whose positive-to-negative transitions will mark each step of the user. In some embodiments, useful metrics may comprise the summations of the square of transformation coefficients, these summations scaled by some factor (including but not limited to the number of coefficients, the number of levels of decomposition, a constant, etc.), or any other type of combinations. In some embodiments, the summations of weighted energies of transformation coefficients adequately scaled by some factor (including but not limited to level of decomposition) may provide an excellent correlation with the kinetic energy of the user. In some embodiments, some of the coefficients may be avoided for the calculation of metrics, and appropriate combinations of summations of weighted energies may be leveraged to compute information comprising velocity. In some embodiments, criteria to avoid transformation coefficients in the calculation of metrics may comprise: selection of a threshold, frequency content, etc. Some embodiments may leverage statistics (including but not limited to, range, mean, skewness, standard deviation, etc.) of the energies of transformation coefficients, or any other features or combinations thereof to be combined with the previously mentioned computed kinematic information and obtain user dynamics information comprising activity. By way of example and not limitation, some embodiments may leverage as metrics the summations of descriptive statistics (or combinations of them) of energies of transformation coefficients of predetermined levels (choice criteria may comprise threshold, frequency content, etc.), in combination with other summations of descriptive statistics (or combinations of them) of energies of transformation coefficients of predetermined levels (choice criteria may again comprise threshold, frequency content, etc.), in combination velocity information.
By way of example without limitation, we have carried out tests to estimate spatio-temporal parameters in walking and running patterns, analyzing in real time through the wavelet transform the signals obtained from the embedded accelerometers from mobile devices (e.g. mobile phones Android, iOS) placed on different parts of the body. In some of the tests, measurements were taken for 9 different types of gait, classified according to 3 different speeds (fast, normal and slow) and 3 different step lengths (long, medium and short). Wavelet transform parameters during some of the tests were dynamically adapted following criteria including performance optimization and power minimization; examples of parameters included: sampling frequencies ranging from 5 Hz to 200 Hz, mother wavelets including several of the Biorthogonal, Daubechies, Coiflets, Haar, Meyer, Mexican Hat, Morlet, Symlet and other families, etc. Through the application of the wavelet transform to the acceleration signal, we focus on the approximation (ai) and detail (di) coefficients, together with their reconstruction signals and their energies defined as follows: Edi, representing the energy of the detail coefficients at level i, equals to the summation of the squares of the detail coefficients considered at level i, divided by the number of coefficients considered; and Eai, representing the energy of the approximation coefficients at level i, equals to the summation of the squares of the approximation coefficients considered at level i, divided by the number of coefficients considered; choice criteria for the coefficients may include thresholding and frequency content. Our experiments show that the signal reconstructed from the detail coefficients at specific levels offers precise information to clearly distinguish steps as the time elapsed between two consecutive negative-to-positive transitions in that reconstructed signal. We also observe that the energy of the detail coefficients may work as an indicator (the largest value dominates) to the previously referred specific levels. In the same sense, for particular scenarios such as the “slow-short” walking pattern, weighted energies for the detail coefficients obtained as Edi divided by the product of the square root of 2 multiplied by: (J−i) when i ranges from 1 to J−1 or 1 when i equals J, where J represents the number of levels of decomposition in the wavelet transform, may deliver a more accurate metric, which also works for the rest of types of gait. Next, focusing on the energies of the approximation coefficients, it is possible to find a good correlation with the kinetic energy of the mobile device user. Consequently, a proper calibration can deliver the speed of the user directly through the energy of the approximation coefficients from the wavelet transform. Leveraging this velocity, the step length can be obtained leveraging the previously obtained time duration of each step.
Further reviewing the wavelet transform decomposition of a signal into approximation and detail coefficients, it can be seen that we are integrating the signal, which in our case represents acceleration, weighted by the scaling and wavelet functions. Consequently, we are integrating weighted accelerations, therefore making sense that we are obtaining weighted velocities. Further analyzing the relationship between the weighted energies of the detail coefficients and the kinetic energy of the mobile device user, we can actually estimate the speed of the user as half the square root of the summation of the previously defined weighted energies of detail coefficients divided by their respective orders of decomposition level.
Further studying the relationship between the detail coefficients and the velocities of the different gait patterns, other variants of the previous computation can be proposed to obtain velocity, which allow us to consider the trade-offs between accuracy in the results and computational burden of the different variants. For instance, using discrete version of Meyer as mother wavelet and focusing on a gait pattern with dominant frequency of 1.256 Hz (scale 16 in cwt), and if we employ more than 5 decomposition levels, the different variants to obtain velocity may differ on the consideration of the wavelet transform detail coefficients at levels 1 and the last level, which may account for the trade-off between accuracy and computational costs of the results. In particular, detail coefficients at the last level have generally the lowest energy, thus providing small increases in the accuracy of the results, although for the slow walking patterns where the overall levels of detail coefficients have low energy, the consideration of the last level coefficients can improve the resolution of our approach. In a similar way, first level detail coefficients have usually low energies, and since the amount of these particular coefficients is the largest, avoiding them in the calculations can help to minimize the computational burden of our method without compromising the accuracy in the results.
Some embodiments may leverage the previously mentioned information about the user's steps in combination with other metrics to enhance user's dynamics information, comprising velocity and activity. Some embodiments may leverage the obtained information on user's steps in combination with the information on user's dynamics to determine stride length. Some embodiments may leverage the information on user's dynamics to compute distance. Some embodiments may enhance distance through the combination of user's dynamics information with localization information. Some embodiments may use different techniques, principles and/or methodologies to obtain all the previous information and metrics, including but not limited to machine learning. In some embodiments, all the computation, processing, information presentation, and other steps may be carried out within a single mobile device without the need of external resources. In some embodiments, the computation or some other step or combinations of steps may be performed external to the mobile device, or with the assistance of some external element, such as external sensor, server, database or any other element. In some embodiments, software may be stored on the mobile or wearable device, for instance, in its memory for execution by its processor or processors. Some embodiments may store data structures and code on computer readable storage medium, which by way of example, and not limitation, may comprise field-programmable gate arrays, application-specific integrated circuits, magnetic and/or optical storage devices, etc.
In some embodiments, the sensor portion of the device or the device itself or any other device containing a sensor and with the capability to communicate in any fashion with the user's device, or any other type of device or accessory may be positioned or attached to any part of the user, including by way of example without limitation, the wrist, arm, hand, face, head, waist, chest, pocket, hat, shoe, any type of clothing, accessories and any combinations thereof and in any way. In some embodiments, the system may be trained to recognize and/or learn activity, motion type, attachment position of the device, movement characteristic, etc. In some embodiments, analysis of acceleration signature may help determine activity, motion type, attachment position of the device, movement/gait characteristic, etc. By way of example without limitation, the acceleration signal may be processed to identify maximums, minimums, mean, standard deviation, frequency components, period, orientation, distribution of peaks, patterns, etc. and/or combinations thereof in order to help determine activity, motion type, attachment position of the device, movement/gait characteristic, etc. In some embodiments, Fourier analysis, any kind of filtering, peak counting, determination of frequency components leveraging the wavelet transform or any other method and combinations thereof may also be utilized to determine user's gait activity, characteristics, etc. In some embodiments, any type of prompt to the user may also be leveraged to request information about his/her activity, motion type, attachment position of the device, movement/gait characteristic, etc. In some embodiments, activity, motion type, attachment position, movement/gait characteristic, etc. may be determined through correlation of any type of sensor values or any type of parameter or metric generated with them, based on any type of model that has been calibrated in any fashion for a particular activity, motion type, attachment position, movement characteristic, etc. In some embodiments, any other sources, means, methods and/or configurations may be leveraged to determine activity, motion type, attachment position, movement/gait characteristic, etc., including by way of example without limitation, the use of sensors and/or signals obtained independently of the sensed acceleration (e.g. GPS), the use of statistics and/or any other empirical information, algorithms, databases or other information stored anywhere and in any fashion, combinations thereof, etc. In some embodiments, the referred methods, configurations, systems, etc. may be modified, updated and/or calibrated in any way, periodically or continuously over any time interval.
Some embodiments may include any external sources to obtain any parameter or information about movement, environment, context, etc. including by way of example without limitation, speed and/or distance monitors, any number of portable electronic devices (e.g. GPS receivers, any kind of computing and/or communications device, etc.), databases and/or networks. In some embodiments, other types of inputs may also be utilized, including by way of example without limitation, buttons, keys, keyboards, keypads, touchpads, joysticks, etc., which may be used in any fashion. Any type of satellite based navigation systems, cellular communications networks and other systems/networks may also be used to obtain speed in some embodiments (and/or provide feedback to help correct errors) under certain conditions.
In some embodiments, additional inputs may include traces from touch-sensitive screens, button presses, gesture recognition, voice commands, switches, and/or any other type of technological, physical or any nature means that allow the user to interact, and combinations thereof. In some embodiments, in addition to using gait characteristic for control, further control may be performed through any additional movements that the user may perform with the device, such as any type of tilting or any kind of gestures, including by way of example without limitation, any kind of raise, swing, twist, touch, press, swipe, drag, double touch, pinch, etc., and combinations thereof, regardless of performing them with or without direct contact to the device screen or any other element (e.g. the user may perform the pinch gesture touching a screen or in the air without touching a solid element). In some embodiments, any type of method may be employed to distinguish between different types of gestures, swings, twists, etc. that the user makes while he/she performs a pedestrian activity (e.g. walk, jog, run, etc.); by way of example without limitation, frequency analysis, filtering, acceleration thresholding, analysis of projection of gravity vector, feedback from other sensors, or any other technique/method and combinations thereof may be employed.
In some embodiments, the acceleration sensor may be an electrostatic or capacitance-coupling type, or any other technology (e.g. piezoelectric or piezoresistance type) now existing or later developed, and may be configured to deliver three-axis, two-axis, or one-axis acceleration. In some embodiments, in addition to accelerometers, any other type of technologies and/or sensors such as gyroscopes, magnetometers, pressure sensors, cameras, GPS, etc. may be used in any way to enhance accuracy or for any other purposes. In some embodiments, the user may have any number of any type of sensors, sensor units, devices, or accessories located anywhere in any fashion to determine the characteristics of his/her movement and/or for control or any other purposes.
In some embodiments, any processing, detection, recognition, or any other actions or operations may be performed regardless of the mode, state or any other condition of the device, application or any other entity, process or element. In other embodiments, any number of conditions and/or criteria of any type must be satisfied before proceeding with any of said actions or operations.
Any of the embodiments herein described may be implemented in numerous ways, including as a method, an apparatus, a device, a system, a computer readable medium, etc., and also be applicable in any environment, application (game, non-game, etc.), condition, etc. regardless of number of users, physical proximity, communication means, device, or any other factor.
Other configurations are also possible. By way of example, and not limitation, in some embodiments, all or part of the processes may be performed by chip-level systems, third-party applications, operating system kernel, firmware, or any other combination of hardware and/or software. In some embodiments, the software may be delivered in a variety of forms, including but not limited to, as stand-alone application, as library, as application programming interface, etc. In general, the functions of particular embodiments may be achieved by any means as is known in the art. Some embodiments may use distributed, networked sensors and/or systems, components, servers, databases, and/or circuits, and/or any combination of additional hardware and/or software and/or processing techniques and methodologies. Some embodiments may use any other type of sensor and/or system.
In some embodiments, sensors may be any of several types including, by way of example, and not limitation, any type of device, transducer or any other type of apparatus which may measure some quantity; in some embodiments, sensors may be implemented in any size, with any type of technique and technology, including but not limited to electronic, microelectronic, nanoelectronic, etc. By way of example, and not limitation, sensors may comprise any type of accelerometer, magnetometer, gyroscope, pressure sensor, proximity sensor, etc. and any other type of device sensitive to radio-frequency, sound, ultrasound, light, etc. including but not limited to, GPS antennas and/or their sensitive elements, WiFi antennas and/or their sensitive elements, and any other type of radio-frequency technology antennas and/or their sensitive elements. In some embodiments, sensors are integrated within the mobile or wearable device. In some embodiments, sensors or other mobile or wearable devices may be distributed outside the main mobile or wearable device, and they may communicate with the main mobile or wearable device by any means. Communication or transfer of data may be wired, wireless, or by any other means. In some embodiments, the user or other entity may rearrange characteristics of the components, or other features or elements of the system and the system may automatically adjust to new settings or arrangements.
In some embodiments, a method for enhancing a user's dynamics and localization information may be used as shown in
In some embodiments, recognizable places may be set as landmarks from which to extract very precise features regarding their location and general context (730). By way of example and not limitation, Radio Frequency Identification, Bluetooth, Zigbee and/or other technologies and/or combinations of them may be leveraged using a variety of techniques to identify landmarks with a very high resolution. Leveraging the information on the user's dynamics, some embodiments may obtain accurate inertial navigation information (740). In some embodiments with basic functionality where the device may not be equipped with gyroscope and/or magnetometer, a variety of mechanisms to identify straight-line trajectories may be leveraged to adapt the inertial navigation solution. When a new identifiable landmark is reached, location and general context features are extracted (750). By way of example and not limitation, some embodiments may use GPS outdoors, or radio beacons indoors detected as peaks in signal strength within a radio-fingerprinting localization system, to identify landmarks. In other embodiments, the use of other types of beacons or landmarks, derived from a variety of technologies, that may use a variety of principles to obtain the required information, is also possible. This information may be leveraged using a variety of possible techniques and methodologies to correct possible errors on the user's dynamics and enhance the localization solution (760). Some embodiments may use manual calibration by the user introducing required calibration parameters in ways he/she may choose from a variety of techniques, technologies and methodologies. Other embodiments may use automatic calibration. In some embodiments, the calibration may be successfully applied to enhance both the information on localization and the user's dynamics and contextual information.
Some embodiments may use all the available information to identify the position (and transitions between positions) of the mobile device within the user's body; by way of example and not limitation, the position information may comprise: held in front in reading position, held in hand while walking, held in pocket while walking, etc. Some embodiments may use external elements comprising user's input to identify positions; in other embodiments, positions will be recognized internally by the mobile device leveraging sensors information.
Some embodiments may use any type of smartphones, mobile devices, wearable devices and/or sensors, or any other types of devices or combinations of them, including but not limited to, personal digital assistants, personal navigation systems, portable electronic devices, tablets, laptops, computers, and their peripheral devices. In some embodiments, the definition of mobile device may comprise any type of mobile phone, smartphone, wearable device and/or sensor, or any other types of device or wearable or combinations of them.
Some embodiments may use combinations of strategies and techniques, including, by way of example, and not limitation, machine learning techniques, probabilistic models, sensor fusion techniques, extraction of statistics, employment of filter banks, application of dimensionality reduction techniques, a variety of approaches for classification, etc. Details are omitted to improve the clarity of the description. In addition, some embodiments may use a variety of programming languages and methodologies in combination with varied hardware configurations and execution strategies.
Some embodiments may leverage context information and provide supplemental information, which may be obtained through any means and sources, including but not limited to, social networks. Particular embodiments may also be used for targeted advertising or targeted information based on context, enable shopping of any type of product or service which may or may not be related to the contextual information, etc.
In some embodiments, various applications may use the obtained information as a trigger for activation. Alternatively, a user may be able to set preferences for different applications depending on the obtained information. By way of example, and not limitation, a user may set the font size and other features of the content (also obtainable through internet or any other means) in his/her mobile device display according to his/her dynamics to improve the reading experience. By way of example, and not limitation, the user may or may not have ear-speakers or head-phones or any other appropriate hardware connected to his/her device and he/she may opt for triggering an out-loud-reader or other type of application to read-out-loud or in some other way adapt the presentation of the content in the device display when his/her dynamic information stays within some preselected threshold levels. By way of example, and not limitation, application(s) and/or service(s) may request, trigger or in some way enable advertising from a commercial ad server or any other type of server or entity using either velocity information, user dynamics, key words, or other criteria as advertising keys. In some embodiments, the user's velocity and other information, including advertisements, may be presented on the mobile and/or wearable device for consideration by the user. Again, depending on preferences and personal privacy policies, information and lists of acquaintances, either desired or undesired, may be presented to the user or to desired friends or other people.
Some embodiments may be used to enhance the location information and to identify points of maximum wireless signal strength, or points with no signal strength, enabling applications or services that effectively leverage that information. Applications of some embodiments may include route searching, planning and optimization, precise geo-tagging of photos, etc. By way of example and not limitation, personalized routing in maps using pedestrian velocity, may enhance features such as travel time estimation, places of interest, navigation, context-based search, etc. For example, a pedestrian walking from home to University may be more interested in sandwich shops rather than gas stations.
Applications of some embodiments may comprise monitoring a variety of information of people in a variety of circumstances or contexts, including but not limited to, health-care, army, sports, etc. Some embodiments may perform the monitoring in a remote way and/or extend the monitoring to animals, robots, machines, etc. In some embodiments, services may be provided through subscription. Some embodiments may be applied for the estimation of calories consumption, or the diagnosis of diseases, such as Parkinson's or other neurodegenerative diseases. Some embodiments may be applied for the identification and/or treatment of disorders, such as gait disorders, associated with a wide variety of conditions, including but not limited to neurologic and orthopedic conditions. Some embodiments may obtain a wide variety of user's information, including but not limited to velocity, activity, stride length, cadence, step count, gait patterns, distance, etc. Some embodiments may apply the information to help in the prevention of falls, accidents or any other undesirable events. Applications of some embodiments may also include contextual interactions, interactive games, augmented reality, and other types of services. By way of example, and not limitation, in games, the attacking and/or crashing strength or power of a user may be set proportional to his/her velocity and certain events or communications may be triggered based on context.
In some embodiments, the obtained information may be used for social networking applications, such as finding and/or establishing communication and/or sharing information with friends and/or other people and/or groups of people whose contextual information might or might not in some way be related. By way of example, and not limitation, in some embodiments, users may be able to share and see the real-time and/or historical contextual information of their friends, edit contextual information on maps, etc. In some embodiments, the observation of two or more mobile and/or wearable devices following similar contextual patterns, may lead to infer a friendship.
Some embodiments may also be applied to infer information from a wide range of biological or other types of sensors/signals, either from humans, animals, mechanical entities such as robots or other machines, etc. Other embodiments may also be applied to monitor and optimize a variety of processes, including but not limited to, industrial and managerial processes. Other embodiments may also have many more applications.
Although the foregoing text sets forth a detailed description of numerous different embodiments of the invention, it should be understood that the scope of the invention is defined by the words of the claims set forth at the end of this patent. The detailed description is to be construed as exemplary only and does not describe every possibly embodiment of the invention because describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims defining the invention.
Thus, many modifications and variations may be made in the techniques and structures described and illustrated herein without departing from the spirit and scope of the present invention. Accordingly, it should be understood that the methods and apparatus described herein are illustrative only and are not limiting upon the scope of the invention.
This application claims priority from U.S. Provisional Patent Application Ser. No. 62/249,371, by David Martin, filed on Nov. 2 2015, entitled “Mobile device control leveraging user kinematics”.
| Number | Date | Country | |
|---|---|---|---|
| 62249371 | Nov 2015 | US |
