The present invention relates to localization and mapping.
There is an increasing need to provide localization and mapping in environments where global positioning systems (GPS) cannot be reliably used, such as in indoor environments where GPS signals are not typically able to register. Localization and mapping of these indoor environments is desirable for location-based services, targeted advertising, network deployment optimization for pervasive computing and the like. However, known localization and mapping systems for GPS-deprived environments lack accuracy or require too resource-intensive computations and/or specialized hardware to be widely employed.
According to an embodiment, a system for providing multi-floor activity classification for a mobile device within a multi-floor environment includes an activity recognition module receiving inertial readings from an inertial measurement unit of the mobile device and pressure readings from a barometer. The activity recognition module classifies activities for the mobile device from the inertial readings and the pressure readings.
According to an embodiment, the inertial readings include acceleration readings from an accelerometer and gyroscopic readings from a gyroscope.
According to an embodiment, the activity recognition module includes a feature extractor that extracts pose invariant features from the accelerometer and gyroscope readings, the activity classification being based on the pose invariant features.
According to an embodiment, the pose invariant features are represented by a vector of features from an autocorrelation matrix of the accelerometer and gyroscope readings.
According to an embodiment, the feature extractor extracts statistical data from the pressure readings, the activity classification being based on the statistical data.
According to an embodiment, the system also includes a post-processing module receiving the activity classifications from the activity recognition module and the pressure readings from a barometer of the mobile device. The post-processing module determines an activity label and floor label for the mobile device based on the activity classifications and the pressure readings.
According to an embodiment, the post-processing module includes a Hidden Markov Model.
According to an embodiment, the system also includes a geolocalization system including known locations for at least one of a staircase, an elevator or an escalator within the multi-floor environment. The geolocalization system is adapted to locate the mobile device within the multi-floor environment based on the known locations and activity labels including at least one of taking stairs, elevators or escalators.
According to an embodiment, the geolocalization system is adapted to reconstruct a trajectory for the mobile device within the multi-floor environment based on the activity label, the floor label and position estimates from the inertial measurement unit of the mobile device.
According to an embodiment, the geolocalization system includes pre-defined activities indicative of known locations within the multi-floor environment. The geolocalization system is adapted to locate the mobile device within the multi-floor environment based on the known locations when the pre-defined activities are detected from the activity labels.
According to an embodiment, a computerized localization method includes receiving, at an activity recognition module executing on a processor, inertial readings from an inertial measurement unit of a mobile device and pressure readings from a barometer. The computerized method further includes classifying, by the activity recognition module executing on the processor, activities for the mobile device from the inertial readings and the pressure readings.
According to an embodiment, the inertial readings include acceleration readings from an accelerometer and gyroscopic readings from a gyroscope.
According to an embodiment, the computerized method additionally includes extracting, by a feature extractor of the activity recognition module, pose invariant features from the accelerometer and gyroscope readings, and classifying the activities based on the pose invariant features.
According to an embodiment, extracting the pose invariant features includes determining a vector of features from an autocorrelation matrix of the accelerometer and gyroscope readings.
According to an embodiment, the computerized method additionally includes extracting, by the feature extractor of the activity recognition module, statistical data from the pressure readings, and classifying the activities based on the statistical data.
According to an embodiment, the computerized method additionally includes post-processing, by a post-processing module executing on the processor, the activity classifications from the activity recognition module and the pressure readings from the barometer in a Hidden Markov Model to determine an activity label and a floor label for the mobile device based on the activity classifications and the pressure readings.
According to an embodiment, a non-transitory, tangible computer-readable medium storing instructions adapted to be executed by at least one processor of a mobile device to perform a method may comprise the steps of receiving, at an activity recognition module executing on the at least one processor, inertial readings from an inertial measurement unit of a mobile device and pressure readings from a barometer, and classifying, by the activity recognition module executing on the at least one processor, activities for the mobile device from the inertial readings and the pressure readings.
According to an embodiment, the method may also include extracting, by a feature extractor of the activity recognition module, pose invariant features from the inertial readings, and classifying the activities based on the pose invariant features.
According to an embodiment, the method may additionally include extracting, by the feature extractor of the activity recognition module, statistical data from the pressure readings, and classifying the activities based on the statistical data.
According to an embodiment, the method may additionally include post-processing, by a post-processing module executing on the at least one processor, the activity classifications from the activity recognition module and the pressure readings from the barometer in a Hidden Markov Model to determine an activity label and a floor label for the mobile device based on the activity classifications and the pressure readings.
These and other embodiments will become apparent in light of the following detailed description herein, with reference to the accompanying drawings.
Before the various embodiments are described in further detail, it is to be understood that the invention is not limited to the particular embodiments described. It will be understood by one of ordinary skill in the art that the systems and methods described herein may be adapted and modified as is appropriate for the application being addressed and that the systems and methods described herein may be employed in other suitable applications, and that such other additions and modifications will not depart from the scope thereof.
In the drawings, like reference numerals refer to like features of the systems and methods of the present application. Accordingly, although certain descriptions may refer only to certain Figures and reference numerals, it should be understood that such descriptions might be equally applicable to like reference numerals in other Figures.
Referring to
As should be understood by those skilled in the art, the mobile device 22 may also include one or more radio frequency (“RF”) transmitters/receivers 32 available on modern smart phones for sending/receiving RF transmissions including WiFi, 4G Long Term Evolution (LTE) and/or Bluetooth transmissions and may include one or more of a camera 34, global positioning system (GPS) 36 and/or near field communication (NFC) chip 38, all of which are available on modern smart phones. These additional features may be used in conjunction with the system 10 to provide pose-invariant multi-floor localization and for the reconstruction of indoor radio frequency signal maps, as discussed below.
The system 10 includes the necessary electronics, software, memory, storage, databases, firmware, logic/state machines, microprocessors, communication links, and any other input/output interfaces to perform the functions described herein and/or to achieve the results described herein. For example, the system 10 may include one or more processors 40 and memory 41, which may include system memory, including random access memory (RAM) and read-only memory (ROM). Suitable computer program code may be provided to the system 10 for executing numerous functions, including those discussed in connection with the feature extractor 13, Support Vector Machine module 14 and Hidden Markov Model 16.
For example, in embodiments, feature extractor 13 and the Support Vector Machine module 14 may be stored in memory 41 on the mobile device 22 and may be executed by at least one processor 40 of the mobile device 22, while the Hidden Markov Model 16 may be stored on and executed by a remote computing device, such as a server in communication with the mobile device 22 over a network, as should be understood by those skilled in the art. In embodiments, the feature extractor 13, Support Vector Machine module 14 and Hidden Markov Model 16 may all be stored in memory 41 on the mobile device 22 and may be executed by the at least one processor 40 of the mobile device 22. In yet further embodiments, both the Support Vector Machine module 14 and the Hidden Markov Model 16 may be stored on and executed by one or more remote computing devices in communication with the mobile device 22, as should be understood by those skilled in the art.
The one or more processors 40 may include one or more conventional microprocessors and one or more supplementary co-processors such as math co-processors or the like. The one or more processors 40 may communicate with other networks and/or devices such as servers, other processors, computers, smart phones, cellular telephones, tablets and the like.
The one or more processors 40 may be in communication with memory 41, which may comprise an appropriate combination of magnetic, optical and/or semiconductor memory, and may include, for example, RAM, ROM, flash drive, an optical disc such as a compact disc and/or a hard disk or drive. The one or more processors 40 and the memory 41 may be, for example, located entirely within a single computer or other device; or connected to each other by a communication medium, such as a USB port, serial port cable, a coaxial cable, an Ethernet type cable, a telephone line, a radio frequency transceiver or other similar wireless or wired medium or combination of the foregoing.
The memory 41 may store inertial measurements taken by the IMU 18 and any other information required by the feature extractor 13, Support Vector Machine module 14 and/or the Hidden Markov Model 16, an operating system, such as for the mobile device 22, and/or one or more other programs (e.g., computer program code and/or a computer program product) adapted to direct the feature extractor 13, Support Vector Machine module 14 and Hidden Markov Model 16 to perform according to the various embodiments discussed herein. The feature extractor 13, Support Vector Machine module 14, Hidden Markov Model 16 and/or other programs discussed herein may be stored, for example, in a compressed, an uncompiled and/or an encrypted format, and may include computer program code executable by the one or more processors 40. The instructions of the computer program code may be read into a main memory of the one or more processors 40 from the memory 41 or a computer-readable medium other than the memory 41. While execution of sequences of instructions in the program causes the one or more processors 40 to perform the process steps described herein, hard-wired circuitry may be used in place of, or in combination with, software instructions for implementation of the processes of the present invention. Thus, embodiments of the present invention are not limited to any specific combination of hardware and software.
The methods and programs discussed herein may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices or the like. Programs may also be implemented in software for execution by various types of computer processors. A program of executable code may, for instance, comprise one or more physical or logical blocks of computer instructions, which may, for instance, be organized as an object, procedure, process or function. Nevertheless, the executables of an identified program need not be physically located together, but may comprise separate instructions stored in different locations which, when joined logically together, comprise the program and achieve the stated purpose for the programs such as providing localization activity recognition. In an embodiment, an application of executable code may be a compilation of many instructions, and may even be distributed over several different code partitions or segments, among different programs, and across several devices.
The term “computer-readable medium” as used herein refers to any medium that provides or participates in providing instructions and/or data to the one or more processors of the system 10 (or any other processor of a device described herein) for execution. Such a medium may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical, magnetic, or opto-magnetic disks, such as memory. Volatile media include dynamic random access memory (DRAM), which typically constitutes the main memory. Common forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, a RAM, a PROM, an EPROM or EEPROM (electronically erasable programmable read-only memory), a FLASH-EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.
Various forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to the one or more processors (or any other processor of a device described herein) for execution. For example, the instructions may initially be stored on a magnetic disk of a remote computer (not shown). The remote computer can load the instructions into its dynamic memory and send the instructions over an Ethernet connection, cable line, telephone line using a modem, wirelessly or over another suitable connection. A communications device local to a computing device can receive the data on the respective communications line and place the data on a system bus for the one or more processors. The system bus carries the data to the main memory, from which the one or more processors 40 retrieve and execute the instructions. The instructions received by main memory may optionally be stored in memory 41 either before or after execution by the one or more processors 40. In addition, instructions may be received via a communication port as electrical, electromagnetic or optical signals, which are exemplary forms of wireless communications or data streams that carry various types of information.
Referring to
To extract the pose-invariant features 24, the feature extractor 13 uses a subset s(t) of the raw data from the accelerometer 28 and gyroscope 30. For example, the feature extractor 13 may use data for a predetermined number of time points (e.g. 64 time points 0.02 seconds apart) and may use a sliding window for every time data is collected (e.g. a sliding window of step size 15 time points). The feature extractor 13 then calculates the autocorrelation matrix A in the frequency domain for the subset s(t) using the following equations:
f(ω)=∫exp(−iωt)s(t)dtεC3, (1)
F=[f(ω1),f(ω2), . . . ,f(ωn)]εC3×n, (2)
A=F*F. (3)
Specifically, equation (1) converts the subset s(t) to the frequency domain. Signal f(ω) is calculated by Fast Fourier transform on s(t) using equation (1). Autocorrelation matrix A, in the frequency domain, is acquired in equation (3) by multiplying conjugate transpose of the Fourier transformed signal F. The autocorrelation matrix A provides a pose invariant property from the raw acceleration and gyroscope data and is a n×n symmetric matrix where n is the number of frequencies. Since the autocorrelation matrix A is symmetric, the feature extractor 13 uses only the upper triangle of the autocorrelation matrix A to provide the pose invariant features 24 as a vector of pose invariant features.
The feature extractor 13 also extracts the statistical data 26 from the barometer 20 in each sliding window of time points. For example, the statistical features may include:
where b(t) is the measured barometric pressure at a given time point t. The statistical data 26 provides for more robust observations than observations based on the raw barometric pressure data alone since the barometer readings typically fluctuate, even when the barometer 20 remains in the same position.
The activity recognition module 11, shown in
Although described in connection with the exemplary activities above for simplicity, it should be understood by those skilled in the art that additional activities recognizable from inertial and barometric data, that are either naturally occurring or specially designed, may be added to the list of activities identifiable by the Support Vector Machine module 14.
The activity classification decision of the Support Vector Machine module 14 is provided to the Hidden Markov Model 16 of the post-processing module 12, shown in
Referring to
The Hidden Markov Model 16 takes a sequence of activity class probability estimates p(sact
p(sact
p(sfloor
T is the length of the sequence; and
S is the number of states (e.g. the number of possible activities multiplied by the number of floors).
Once the observation probabilities p(y|si) are calculated, the post-processing module 12, shown in
Referring back to
For instance, referring to
Referring to
Referring to
Referring to
The optimized trajectory may be used for a variety of purposes, such as to provide mapping of the environment of concern on the mobile device 22, shown in
Thus, the system 10 advantageously provides the geolocalization system with organic landmarks 78 corresponding to staircases and elevators and provides floor identification, which allows the geolocalization system to precisely track the trajectory of pedestrians inside environments of concern, such as multi-story buildings, solely using the mobile device 22 and its various sensors with the mobile device 22 placed in the pedestrian's pocket. Accordingly, with the system 10, there is no need for the pedestrian to take the mobile device 22 out of the pocket to acquire additional landmark information from, for example, NFC tags or QR code that may provide information about specific locations in the environment. Alternatively, if there are insufficient organic landmarks 78 in the environment of concern, the system 10 may define landmarks as special gestures (such as stopping and walking following a pre-defined pattern) that are easily recognizable as activities by the system 10. These special gestures may be pre-defined and associated with specific points in the environment or building and, therefore, may serve as artificial landmarks 78 without the need to introduce another sensing system like the NFC tags or QR codes discussed above.
The system 10 may advantageously be implemented with geolocalization systems, as discussed above, to provide pedestrian tracking that requires no effort on the part of the pedestrian, since the mobile device 22 may be in any orientation and may remain in the pedestrian's pocket. This pedestrian tracking may advantageously be used for the reconstruction of indoor radio RF signal maps, as discussed in U.S. patent application Ser. No. 14/063,735, without prior knowledge of the sources of RF signals. Thus, the maps required for RF indoor localization may be constructed without any effort from the builder aside from walking in the environment of concern, simply by exploiting all of the sensor signals recorded on one or more mobile devices 22.
The system 10 for localization activity classification may advantageously be employed in indoor geolocalization products and services for institutions such as airports, shopping malls, museums, campuses, or the like to provide location services, advertisements, product recommendations, security-related services or other similar services inside buildings. The system 10 for localization activity classification may also be used for planning and optimization of the deployment of telecommunication networks such as LTE/4G small cells and the like.
The system 10 advantageously works entirely from the pocket with the mobile device 22 placed in any pose, unlike previous inventions that require mobile devices to be held in specific poses in specific locations. Thus, the system 10 simplifies data acquisition processes for RF map building. Additionally, the system 10 advantageously integrates activity detection with floor identification so that both activity and floor changes may be detected, thereby providing a more accurate and robust localization system.
Since the system 10 provides for pose-invariant activity detection, the system 10 advantageously allows the mobile device 22 to be placed in the pocket or the like of the pedestrian and does not require the mobile device 22 to be oriented or positioned in any particular manner (e.g., held vertically in the hand or the like).
Although this invention has been shown and described with respect to the detailed embodiments thereof, it will be understood by those skilled in the art that various changes in form and detail thereof may be made without departing from the spirit and the scope of the invention. For example, although the system 10, shown in