Patent Application
20090262974
The subject matter of the present application relates to obtaining georeferenced mapping data for a target structure or premises in absolute geographical coordinates, and in particular although not limited to, an aided-inertial based mapping system for mapping any region or structure where GPS signals are unavailable or insufficient for an accurate determination of position and location. An indoor mapping instrument is capable of generating indoor maps, for example, that are highly accurate and can be produced quickly by using the instrument while simply walking through the interior areas of the building.
Maps enhance the value of positioning by effectively converting position information of natural and man-made objects, persons, vehicles and structures to location information. Outdoor mapping such as street mapping capability has been announced by companies Navteq and Tele-Atlas. These outdoor location services are GPS-based in that they acquire and use GPS signals to obtain precise position and location information for positioning and mapping. One example is discussed in U.S. Pat. No. 6,711,475. This patent, as well the other patents identified or described herein, are incorporated herein by reference.
Where GPS signals are not available or not dependable (such as indoors) attempts have been made to determine position or location. U.S. Pat. No. 5,959,575 describes the use of a plurality of ground transceivers which transmit pseudo-random signals to be used by a mobile GPS receiver indoors.
In mining operations where GPS signals are not available, U.S. Pat. No. 6,009,359 describes the use of an Inertial Navigation System (INS) to determine position, and obtaining image frames which are tiled together to get a picture of inside the mine. U.S. Pat. No. 6,349,249 describes a system for obtaining mine Tunnel Outline Plan views (TOPES) using an inertial measurement unit (IMU). U.S. Pat. No. 6,608,913 describes a system for obtaining point cloud data of the interior of a mine using an INS, to thereafter locate a position of a mining vehicle in the mine.
In indoor facilities such as buildings, U.S. Pat. No. 7,302,359 describes the use of an IMU and rangefinder to obtain a two-dimensional map of the building interior, such as wall and door locations. U.S. Pat. No. 6,917,893 describes another indoor mapping system for obtaining two-dimensional or three-dimensional data using an IMU, laser rangefinder and camera.
None of these patents appear to disclose obtaining three-dimensional data in a GPS-denied zone such as indoors, wherein the data includes not only three-dimensional position information, but also characteristic image data information, such as color, brightness, reflectivity and texture of the target surfaces to enable an image display of a virtual tour of an interior region as if the person were actually inside the premises.
Sensor technologies that will not only operate indoors but will do it without relying on building infrastructure provide highly desirable advantages for public safety crews, such as firefighters, law enforcement including SWAT teams, and the military. The need for such indoor mapping has increased due to the ever increasing concern to protect the public from terrorist activity especially since terrorist attacks on public, non-military targets where citizens work and live. In addition to terrorist activity, hostage activity and shootings involving student campuses, schools, banks, government buildings, as well as criminal activity such as burglaries and other crimes against people and property have increased the need for such indoor mapping capability and the resulting creation of displayable information that provides avirtual travel through interior regions of a building structure.
What is needed is a system and method for accurate three dimensional mapping of regions, especially those regions where GPS signal information is not available or is unreliable such as within a building structure, and for showing the location and boundaries of interior objects and structures, as well as characteristic image data such as color, reflectivity, brightness, texture, lighting, shading and other features of such structures, whereby such data may be processed and displayed to enable a virtual tour of the mapped region. In particular, a mobile system and method are needed capable of generating indoor maps that are highly accurate and can be produced quickly by simply walking through the interior areas of a building structure to obtain the data needed to create the maps without the use of support from any external infrastructure or the need to exit the indoor space for additional data collection. In addition, a system and method are needed for providing such indoor location information based upon the operator's floor, room and last door walked through, which information can be provided by combining position information with an indoor building map. Moreover, a mobile mapping system and method are need by which high-rate, high-accuracy sensor, position and orientation data are used to geo-reference data from mobile platforms. A benefit from geo-referencing data from a mobile platform is increased productivity since large amounts of map data may be collected over a short period of time.
A system and method for acquiring spatial mapping information of surface data points defining a region unable to receive effective GPS signals, such as the interior of a building structure, includes an IMU for dynamically determining geographical positions relative to at least one fixed reference point, a LIDAR or camera for determining a range of the IMU to each surface data point, and a processor to determine position data for each surface data point relative to the at least one reference point. A digital camera obtains characteristic image data, including color data, of each surface data point, and the processor correlates the position data and image data for the surface data points to create an image of the region. Aerial or ground-vehicle based views of the exterior of a building structure containing the region are seamlessly combined to provide indoor and outdoor views.
A system and method are disclosed for acquiring geospatial data information, comprising a positioning device for determining the position of surface data points of a structure in three-dimensions in a region unable to receive adequate GPS signals, an image capture device for obtaining characteristic image data of the surface data points, and a data store device for storing information representing the position and characteristic image data of the surface data points, and for correlating the position and image data for the data points.
A system and method are disclosed for acquiring spatial mapping information, comprising an indoor mapping system (IMS) for determining the position of surface data points of building structure in three-dimensions in a region unable to receive adequate GPS signals. The IMS comprises an IMU for determining position data relative to at least one reference point, and a light detection and ranging (LIDAR) sensor for determining the distance between the IMU and a plurality of surface data points on the building structure, an image capture device for obtaining characteristic image data of the surface data points, a data processor including a data store device for storing information representing the positions of the surface data points and the characteristic image data of the surface data points, and for correlating the position data and image data for the surface data points.
A system and method is disclosed for acquiring spatial mapping information comprising an IMS device for determining the position of surface data points of building structure in three-dimensions in a region unable to receive adequate GPS signals, the IMS device comprising an IMU for determining position data relative to at least one reference point, and a LIDAR sensor for determining the distance between the IMU and surface data points on the building structure. A GPS receiver may be used in a GPS active area for obtaining the position of at least one initial reference point which may be used as a starting reference point by the IMU. The IMS further includes a digital camera for obtaining characteristic image data of the surface data points, the image data including color data, and a processor and data store device by which digital information representing the positions of surface data points and the characteristic image data of the surface data points is stored and correlated. The processor recreates for display an image of the building structure using the position data and image data.
In an embodiment, an IMS is based on a navigation-grade IMU aided by zero-velocity updates. The IMU is combined with a scanning laser and a digital camera. The system is small and lightweight and can be backpack portable. The aided-inertial system measures the IMS position as well as pitch, roll, heading and the laser measures the distance between the IMS and the laser data points. Combining these measurements provides a detailed map of the details of the surveyed regions of the building. This can be further visually enhanced by combining digital cameral imagery with the laser data points. The resulting photomaps are geo-referenced digital imagery of the surveyed regions, and can be detailed at sub-meter accuracies.
By providing information to enable a virtual tour of the interior premises, a roving person such as a law enforcement officer or military person can be equipped with a display device, which may be near the eyes, such as a head-up display or a stereo display device, and can walk through the premises and have a virtual tour even if there is no light or if the premises is filled with smoke or the like. The person can be directed by other personnel outside the premises who can be equipped with the same display of the same images observed by the rover to enable such personnel to communicate with and guide the person inside the premises. This can minimize the number of personnel at risk. Alternatively, a robot can be used, guided by outside personnel, which could be maneuvered throughout a desired region of the premises without placing a person at risk.
For a further understanding of the subject matter described herein, reference may be had to the accompanying drawings in which:
As used herein, the term “geospatial data” means image and position data for points in space.
As used herein, the term “georeferencing” means the assigning of geographical coordinates to one or more points in space.
As used herein, the term “mobile mapping” means the collection of georeferenced data from a mobile platform, such as a person, or a land vehicle.
As used herein, the term “image data” means information which characterizes the visual attributes of a structure or object, other than location or position, such as color, reflectivity, brightness, texture, lighting and/or shading for example.
As used herein, the term “building structure” means walls, partitions, or other structure which define the interior space a building, such as a commercial building, residence building or the like.
As used herein, the term “position” means the geographical coordinates of longitude, latitude and altitude of an object or thing, such as a point.
As used herein, the term “location” means the relative position of an object or thing, such as a point, as defined by its surroundings, such as the floor and room in an indoor structure.
With reference to
The sensor platform 11 may also include an IMU 11B for determining positions within the GPS inactive region relative to at least one reference point. The IMU 11B is functionally integrated with the LIDAR 11A and the camera 12 for enabling the determination of the position of each of a plurality of surface data points on the target structure relative to the reference point. The LIDAR 11A, the IMU 11B and the image capture device 12 may be mounted on a common frame backpack type of frame 14. As depicted in
With reference to
In some circumstances, the sensor platform may further include a GPS receiver forming part of a smart antenna 17, shown in dotted lines in
The system processor 13 receives ranging, imaging and position data from the LIDAR 11A, the camera 12 and the IMU 11B, respectively. A data store retains position data and image data for use by the processor to correlate the stored position data and image data for each of the surface data points. This is accomplished by assigning the geographical coordinates to geospatial data so that the image date is correlated with position data. In this way the processor 13 is able to create an image of the target structure or region from a perspective different from the location of the positioning capability. As an example, when a target region is the interior of a building structure, the processor may create on a display 19 (
The positioning data and digital image data can be used to create photomaps of all visible surfaces or objects and structures in an interior building space. The in-building photomaps are accurately georeferenced. This means that every image pixel in the collected imagery has accurate geographical coordinates assigned to it. The resulting photomaps are georeferenced digital imagery of a building's interior detail at decimeter-level accuracies. This level of accuracy may be necessary in order to determine the exact location of operators within the building and, as an example, quickly and effectively guide rescue missions in law enforcement or military operations.
Outside photomaps of the building can be collected from a land vehicle and/or aircraft or helicopter. The collection of outdoor photomaps may be done by integrating GPS position information with data obtained from LIDAR sensors and digital cameras, as described above. When GPS is available, it is not necessary to employ navigation-grade IMU sensors to establish positions, as is necessary for indoor mapping operations. A seamless blending of indoor building photomaps with other indoor photomaps, as well as with outdoor photomaps, enables the creation of a complete inside-outside view of an entire building.
With reference to
The IMU at block 21 represents a highly precise, navigation-grade IMU having various components, including three gyroscope and three accelerometer sensors that provide incremental linear and angular motion measurements to the Inertial Navigator. The IMU may be high-performance, navigation-grade, using gyroscopes with 0.01 deg/hr performance or better, such as the Honeywell HG9900, HG2120 or micro IRS. The Inertial Navigator, using sensor error estimates provided by a Kalman filter at block 23, corrects these initial measurements and transforms them to estimates of the x, y, z position, and orientation data including pitch, roll and heading data for the backpack or cart, at a selected navigation frame. When GPS signals are available, a GPS receiver, shown at block 24 in dotted lines, provides GPS data to the Kalman Filter for the initial alignment of the IMU only. The alignment process based upon GPS position information may be static or dynamic. If static, it occurs at a fixed and known position with known coordinates. It may also be accomplished on a moving vehicle using GPS to aid in obtaining correct position information from the IMU.
For continued operation in an interior region of a building subsequent navigation is performed in the complete absence of GPS. In such a case, when the GPS signal is lost, the IMU takes over and acquires the position data. The Kalman filter at block 23 provides processed measurement information subject to errors to an error controller at block 26, which keeps track of the accumulated errors in estimated measurements over time. When the Kalman Filter's estimated measurement errors grow above a threshold, usually over a period of from 1 to 2 minutes, the system requests a zero velocity update (ZUP), indicated at block 27, from the operator through an audio notification. The sensor platform 11, either a backpack or cart, is then motionless for 10-15 sec to permit the Kalman filter to perform error corrections for the then existing position of the sensor platform. The mapping operation is resumed after each roughly 15 second delay period. In this situation, the IMU can operate without any GPS aiding for hours, using only ZUP as an aid to correction of the IMU's sensor errors. In this way, the Inertial Navigator obtains updated correct position information every few minutes, a technique that avoids the otherwise regular degradation in accuracy for IMU position measurements over time.
The upper left section of
All data, including the LIDAR and image data, as well as the IMU incremental x, y, z position and pitch, roll and heading information are stored on a mass storage device at block 31, depicted in the upper right section of
The data is retrieved post-mission through a post processing suite at block 32 which combines the aided-inertial system's position and orientation measurements with the LIDAR's range measurements. Post-mission software performs two-functions. One function is to combine pitch/roll/heading with the range measurements to build a three dimensional geo-referenced point cloud of the traversed space. The lower right section of
With reference to
The next step “Walk” involves any walking speed or movement of the data acquisition/collection apparatus through the premises being mapped. The person has a LIDAR and digital camera to acquire depth and image data, as described above.
The next step “ZUP” involves obtaining a zero-velocity update of position by, for example, stopping every 1-2 minutes and standing motionless for 10-15 seconds in order to permit correction of the measured position information. The step “Walk” is then continued until the next ZUP period. The steps of Walk and ZUP are repeated until mapping of the target region is complete.
With reference to
The first step in the process is to determine the vector rsM. In outdoor environments this can be accomplished by using GPS or a GPS-aided inertial system. In an indoor environment this can be accomplished by using a ZUP-aided IMU. The next step is to determine the vector rps by determining the polar coordinates of the sensor platform S (attitude angles: roll, pitch, heading) and the distance of the sensor platform S from the point P. The angles may be determined using gyroscopes and a ZUP-aided IMU. In an embodiment, the ZUP-aided IMU is a navigation-grade IMU. The distance from the position sensor to the point P may be determined using a laser scanning device such as the LIDAR described above, or by using a stereo camera pair and triangulating. A single camera may also be used for obtaining sequentially spaced images of the target point from which distance from the position sensor to the target point P may be derived. As indicated above, the camera also provides characteristic image data for each target point P on the surface to be mapped. The information available from the foregoing vectors enables the computation of the coordinates of the target point P.
When the image data is correlated with the stored point position data, a data base exists by which the processor can reconstruct an image of a mapped interior surface area of the premises by selecting a vantage point, and selecting an azimuth and direction from that vantage point from which to display an image defined by the stored three dimensional positions for each mapped point on the surface area being mapped. These may be visualized using a suite such as the one from Object Raku. The processor will recreate or reconstruct an image representing the actual interior of the premises as though the viewer were actually inside the premises looking through an image capture device. The image seen can be continuously changed by selecting different vantage points as though the viewer was traveling through the premises, and the azimuth and direction may also be changed, either when the vantage point is constant or changing. The processor may also create stereo images, with an image provided separately to each eye of a viewer, to provide a three dimensional image. The images may be displayed on left and right displays worn as eyewear. Such an arrangement provides a virtual reality tour of the inside of the premises without actually being present inside the premises. The image or images viewed may be panned horizontally or vertically, or zoomed in or out.
While various exemplary embodiments of a georeferencing system and method have been shown and described, the described embodiments do not limit scope of protection afforded by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the appended claims, which alone constitute the sole measure of the scope of protection for the subject matter shown, described and claimed herein.
The present application is based upon and hereby claims the benefit of the filing date of prior-filed U.S. provisional application No. 61/124,722, filed Apr. 18, 2008.
| Number | Date | Country | |
|---|---|---|---|
| 61124722 | Apr 2008 | US |
