Patent Application
20090237396
The present invention relates to the field of imaging and computer graphics, and more particularly, this invention relates to a system and method for correlating and synchronizing a three-dimensional site model and two-dimensional imagery.
Some advanced imaging systems and commercially available software applications display two-dimensional imagery, for example, building interiors, floor plan layouts and similar two-dimensional images, and also display three-dimensional site model structures to provide spatial contextural information in an integrated environment. There are some drawbacks to such commercially available systems, however. For example, a majority of photogrametrically produced three-dimensional models have no interior details. A familiarization with building interiors while viewing a three-dimensional model would be useful to many users of such applications, for example, for security and similar applications. Some software imaging applications display interior images that give detail without site reconstruction and are becoming more readily available, but even these type of software imaging applications are difficult to manage and view in a spatially accurate context. A number of these applications do not have the imagery geospatially referenced to each other and it is difficult to identify what a user is looking at when viewing the images. For example, it is difficult for the user to determine which room or rooms are contained in any given image, especially when there are many similar images that make it difficult for a user to correlate and synchronize between the various images, especially when the user pans or rotates an image view.
Some interior imaging systems, for example, those having power to display 360-degree panoramic images, can capture interior details It is difficult, however, even in these software applications, to comprehend what any given portion of an image references, for example, which room is displayed or which hallway is displayed within the image or which room is next to which room, and what is behind a given wall. This becomes even more difficult when the rooms and hallways in a building look similar such that a user has no bearing or common reference to use for orientation relative to the different hallways and rooms within the building. It is possible to label portions of the image with references so the user understands better what they are looking at, but this does not sufficiently solve this problem, which is further magnified when there are dozens of similar images.
For example,
Some imaging applications provide a two-dimensional layout of a building floor plan with pop-ups that show where additional information is available, or provide an image captured at a specific location within a building, but provide no orientation as to the layout of the building. For example, a system may display a map of a site and contain markers, which a user could query or click-on to obtain a pop-up that shows an interior image of that respective area. Simply querying or clicking-on a marker in a serial manner, however, does not give the user the context of this information concerning the location the user is referenced at that site. Furthermore, it is difficult to comprehend the contents of an image that contains many rooms or unique perspectives. Sometimes images may be marked-up to provide some orientation, but any ancillary markers or indicia often clutters the image. Even with markers, these images still would not show how components within the image relate to each other.
One proposal as set forth in U.S. Patent Publication No. 2004/0103431 includes a browser that displays a building image and icon hyperlinks that display ancillary data. It does not use a three-dimensional model where images and plans are geospatially correlated. As disclosed, the system is directed to emergency planning and management in which a plurality of hyperlinks are integrated with an electronic plan of the facility. A plurality of electronic capture-and-display media provide visual representations of respective locations at the facility. One of the electronic capture-and-display media is retrieved and played in a viewer, after a hyperlink associated with the retrieved media is selected. The retrieved media includes a focused view of a point of particular interest, from an expert point of view.
An imaging system includes a 3D database for storing data relating to three-dimensional site model images having a vantage point position and orientation when displayed. A 2D database stores data relating to a two-dimensional image that corresponds to the vantage point position and orientation for the three-dimensional site model image. Both the three-dimensional site model image and two-dimensional image are displayed typically on a common display. A processor operative with the two-dimensional and three-dimensional databases and display will create and display the three-dimensional site model image and two-dimensional image from data retrieved from the 2D and 3D databases and correlates and synchronizes the three-dimensional site model image and two-dimensional image to establish and maintain a spatial orientation between the images as a user interacts with an image.
The imaging system includes a graphical user interface in which the three-dimensional site model and two-dimensional images are displayed. The three-dimensional site model image could be synchronized with a panoramic view obtained at an image collection point within a building interior. The two-dimensional images include a floor plan image centered on the collection point within the building interior. The processor can be operative for rotating the panoramic image and updating the floor plan image with a current orientation of the panoramic image.
A dynamic heading indicator can be displayed and synchronized to a rotation of the three-dimensional site model image. The processor can update at least one of the 2D and 3D databases based upon additional information obtained while a user interacts with an image. The 2D database can be formed of rasterized vector data and the 3D database can include data for a local space rectangular or world geocentric coordinate systems. Both the 2D and 3D databases can store ancillary data to the 2D database and 3D database and provide additional data that enhances an image during user interaction with an image.
An imaging method is also set forth.
Other objects, features and advantages of the present invention will become apparent from the detailed description of the invention which follows, when considered in light of the accompanying drawings in which:
Different embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments are shown. Many different forms can be set forth and described embodiments should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope to those skilled in the art. Like numbers refer to like elements throughout.
In accordance with a non-limiting example of the present invention, the system and method correlates and synchronizes a three-dimensional site model and two-dimensional imagery with real or derived positional metadata, for example, floor plans, panoramic images, video and similar images to establish and maintain a spatial orientation between the images, such as formed from disparate data sets. For example, a two-dimensional floor plan image could be displayed as centered on a collection point of a three-dimensional site model image as a panoramic image and the collection point marked on the three-dimensional site model image. As the panoramic image is rotated, the floor plan is updated with a current orientation. This process can associate ancillary information to components within the image such as the room identification, attributes and relative proximities.
Correlation, of course, can refer to the correspondence between the two different images such that the reference point as a collection point for a panoramic image, for example, will correspond or correlate to the spatial location on the two-dimensional image such as a floor plan. As a user may rotate a panoramic image and the two-dimensional image will be synchronized such that the orientation may change in the two-dimensional image, for example, a line or indicator pointing in the direction of the rotation or other similar marker. The speed and image changes would be synchronized as a user interacts with a two-dimensional image and the three-dimensional image changes or when the user interacts with the other image.
Interior images can be located on the three-dimensional site model image at the point the imagery was originally captured at a collection point. From within the immersive three-dimensional environment, at these identified collection points, the user can view the image at the same perspective and spatial orientation of the three-dimensional site model image. Each image can have information associated with it, such as its spatial position and the collection azimuth angle. This information is used to synchronize it with one or other two-dimensional images and to correlate all the images to the three-dimensional model. For example, a portion of a floor plan correlated to where a panoramic image was taken as a collection point with the floor plan can have a dynamic heading indicator synchronized to the rotation of the panoramic image. Information correlated in this manner makes it more intuitive from a user's point of view to recognize from the two-dimensional images what portion of the three-dimensional site model is being explored, as well as those portions that are adjacent, orthogonal or hidden from the current viewing position. The system and method in accordance with a non-limiting example of the present invention accurately augments the data providing the three-dimensional site model and provides a greater spatial awareness to the images. It is possible to view a three-dimensional site model image, panoramic image, and two-dimensional image. These images are correlated and synchronized.
Another block 54 shows the two-dimensional imagery as a database or data set that could be available in different forms, including rasterized vector data, including floor plans, and interior images, panoramic images, and video sequences. This data set is correlated and synchronized such that any reorientation or interaction with any of the two-dimensional image content prompts the system to synchronize and update any other two-dimensional and three-dimensional orientation information.
The associated databases 56 can represent ancillary data or information to the two-dimensional and three-dimensional data sets and can supply auxiliary/support data that can be used to enhance either environment. The associated databases 56 can be updated based upon different user interactions, including any added notations supplied by the user and additional image associations provided by the system or by the user, as well as corresponding similar items.
Typically with rasterized vector data, the raster representation divides an image into arrays of cells or pixels and assigns attributes to the cells. A vector based system, on the other hand, displays and defines features on the basis of two-dimensional Cartesian coordinate pairs (such as X and Y) and computes algorithms of the coordinates. Raster images have various advantages, including a more simple data structure and a data set that is compatible with remotely sensed or scanned data. It also uses a more simple spatial analysis procedure. Vector data has the advantage that it requires less disk storage space. Topological relationships are also readily maintained. The graphical output with vector based images more closely resembles hand-drawn maps.
As shown in the flowchart, while a user is in the three-dimensional environment and spanning or rotating an image or otherwise maintaining position (block 52), for example, as shown in one of the images of
If that three-dimensional position corresponds to a registered location of the two-dimensional imagery, the system retrieves and calculates the orientation of parameters of all two-dimensional imagery at this position (block 60). The system displays and updates any two-dimensional images at this position reflecting the orientation of the image relative to any viewing parameters (block 62) At this point, the user interacts with the two-dimensional imagery and moves along the two-dimensional image, changing views or adding new information. The viewing parameters could be specified by the user and/or the system during or after image initialization. The user interacts with the two-dimensional imagery and can change, view, exit, or add new information to a database and perform other similar processes (block 64).
At this time, the system determines if the user desires to exit from the two-dimensional imagery environment (block 66), and if yes, then the two-dimensional image views are closed depending on the specific location of the user relative to the two-dimensional image (block 68). The orientation in the three-dimensional environment is then adjusted, for example, relative to where the user might be positioned on the two-dimensional image (block 70). An example is explained later with reference to
Referring now again to block 64 where the user has both two-dimensional and three-dimensional screen images as shown in
Referring now to
As illustrated, the graphical user interface can be displayed on a video screen or other monitor 130 that is part of a personal computer 132, which includes a processor 134 operative with the 2D database 136 and 3D database 138. The processor is also operative with the associated database 140 as illustrated in the block components shown with the monitor 130.
The system could generate shells from modeling based upon satellite/aerial imagery and include building interior details. The system and method geospatially correlates two-dimensional imagery with three-dimensional site models and offers a data product that allows a user to identify quickly portions of a scene contained in interior imagery as it relates to a three-dimensional orientation. Typically, C++ code is used with different libraries and classes that represent different entities, such as a panoramic image or display with a built-in mechanism to maintain a three-dimensional position. The code is developed to synchronize and correlate images once the system enters the two-dimensional view and matches and reorients any two-dimensional images and three-dimensional site model images. A graphics library similar to Open GL can be used. Other three-dimensional graphics packages can be used.
The system can be augmented with the use of a three-dimensional packages such as the InReality™ application from Harris Corporation, including use of a system and method for determining line-of-sight volume for a specified point, such as disclosed in commonly assigned U.S. Pat. No. 7,098,915, the disclosure which is hereby incorporated by reference in its entirety or the RealSitet™ site modeling application also from Harris Corporation.
There now follows a more detailed description of the RealSite™ application that can be used as a complement to the correlation and synchronization as described above. It should be understood that this description of RealSite™ is set forth as an example of a type of application that can be used in accordance with a non-limiting example of the present invention.
A feature extraction program and geographic image database, such as the RealSite™ image modeling software developed by Harris Corporation of Melbourne, Fla., can be used for determining different geometry files. This program can be operative with the InReality™ software program also developed by Harris Corporation of Melbourne, Fla. Using this application with the RealSite™ generated site models, it is possible for a user to designate a point in three-dimensional space and find the initial shape of the volume to be displayed, for example a full sphere, upper hemisphere or lower hemisphere and define the resolution at which the volume is to be displayed, for example, 2°, 5° or 10° increments. It is also possible to define the radius of the volume to be calculated from the specified point The InRealty™ viewer system can generate a process used for calculating the volume and automatically load it into the InRealty™ viewer once the calculations are complete. A Line-of-Sight volume can be calculated by applying the intersection calculations and volume creation algorithms from a user selected point with display parameters and scene geometry as developed by RealSite™ and InRealty™, as one non-limiting example. This solution would provide a situation planner immediate information as to what locations in a three-dimensional space have a Line-of-Sight to a specific location within a three-dimensional model of an area of interest. Thus, it would be possible for a user to move to any point in the scene and determine the Line-of-Sight to the point. By using the InReality™ viewer program, the system goes beyond providing basic mensuration and displaying capabilities. The Line-of-Sight volumes can detail, in the three-dimensional site model, how areas are obscured in the synchronized two-dimensional imagery.
It is possible to use modified ray tracing for three-dimensional computer graphic generation and rendering an image. For purposes of description, the location, i.e., the latitude and longitude of any object that would effect the Line-of-Sight can be located and determined via a look-up table of feature extraction from the geographic image database associated with RealSite™ program. This geographic database could include data relating to the natural and man-made features in a specific area, including data about buildings and natural land formations, such as hills, which all would effect the Line-of-Sight calculations.
For example, a database could include information about a specific area, such as a tall building or water tower. A look-up table could have similar data and a system processor would interrogate and determine from the look-up table the type of buildings or natural features to determine the geometric features.
For purposes of illustration, a brief description of an example of a feature extraction program that could be used, such as the described RealSite™, is now set forth. The database could also be used with two-dimensional or three-dimensional feature imaging as described before. Optical reflectivity can be used for finding building plane surfaces and building edges.
Further details of a texture mapping system used for creating three-dimensional urban models is disclosed in commonly assigned U.S. Pat. No. 6,744,442, the disclosure which is hereby incorporated by reference in its entirety. For purposes of description, a high level review of feature extraction using RealSite™ is first set forth. This type of feature extraction software can be used to model natural and man-made objects. These objects validate the viewing perspectives of the two dimensional imagery and Line-of-Sight calculations, and can be used in two-dimensional and three-dimensional image modes.
RealSite™ allows the creation of three-dimensional models in texture mapping systems and extends the technology used for terrain texturing to building texture by applying clip mapping technology to urban scenes, It can be used to determine optical reflectivity values and even radio frequency reflectivity.
It is possible to construct a single image of a building from many images that are required to paint all the sites. Building site images can fit into a composite image of minimum dimension, including rotations and intelligent arrangements. Any associated building vertex texture coordinates can be scaled and translated to match new composite images. The building images can be arranged in a large “clip map” image, preserving the horizontal relationships of the buildings. If the horizontal relationships cannot be accurately preserved, a “clip grid” middle layer can be constructed, which can be used by the display software to accurately determine the clip map center.
At its highest level, the system creates a packed rectangle of textures for each of a plurality of three-dimensional objects corresponding to buildings to be modeled for a geographic site. The system spatially arranges the packed rectangle of textures in a correct position within a site model clip map image. The texture mapping system can be used with a computer graphics program run on a host or client computer having an OpenGL application programming interface. The location of a clip center with respect to a particular x,y location for the site model clip map image can be determined by looking up values within a look-up table, which can be built by interrogating the vertices of all building polygon faces for corresponding texture coordinates. Each texture coordinate can be inserted into the look-up table based on the corresponding polygon face vertex coordinate.
In these types of systems, the graphics hardware architecture could be hidden by a graphics API (Application Programming Interface). Although different programming interfaces could be used, a preferred application programming interface is an industry standard API such as OpenGL, which provides a common interface to graphics functionality on a variety of hardware platforms. It also provides a uniform interface to the texture mapping capability supported by the system architecture.
OpenGL allows a texture map to be represented as a rectangular pixel array with power-of-two dimensions, i.e. , 2m×2n. To increase rendering speed, some graphics accelerators use pre-computed reduced resolution versions of the texture map to speed up the interpolation between sampled pixels. The reduced resolution image pyramid layers are referred to as MIPmaps by those skilled in the art. MIPmaps increase the amount of storage each texture occupies by 33%.
OpenGL can automatically compute the MIPmaps for a texture, or they can be supplied by the application. When a textured polygon is rendered, OpenGL loads the texture and its MIPmap pyramid into the texture cache. This can be very inefficient if the polygon has a large texture, but happens to be far away in the current view such that it only occupies a few pixels on the screen. This is especially applicable when there are many such polygons.
Further details of OpenGL programming are found in Neider, Davis and Woo, OpenGL Programming Guide, Addison-Wesley, Reading, Mass., 1993, Chapter 9, the Guide disclosure which is hereby incorporated by reference in its entirety.
Clip texturing can also be used, which improves rendering performance by reducing the demands on any limited texture cache. Clip texturing can avoid the size limitations that limit normal MIPmaps by clipping the size of each level of a MIPmap texture to a fixed area clip region.
Further details for programming and using clip texturing can be found in Silicon Graphics, IRIS Performer Programmer's Guide, Silicon Graphics, Chapter 10: Clip Textures, the Programmer's Guide, which is hereby incorporated by reference in its entirety.
IRIS Performer is a three-dimensional graphics and visual simulation application programming interface that lies on top of OpenGL. It provides support for clip texturing that explicitly manipulates the underlying OpenGL texture mapping mechanism to achieve optimization. It also takes advantage of special hardware extensions on some platforms. Typically, the extensions are accessible through OpenGL as platform specific (non-portable) features.
In particular, IRIS Performer allows an application to specify the size of the clip region, and move the clip region center. IRIS Performer also efficiently manages any multi-level paging of texture data from slower secondary storage to system RAM to the texture cache as the application adjusts the clip center.
Preparing a clip texture for a terrain surface (DEM) and applying it can be a straightforward software routine in texture mapping applications, as known to those skilled in the art. An image or an image mosaic is orthorectified and projected onto the terrain elevation surface. This single, potentially very large, texture is contiguous and maps monotonically onto the elevation surface with a simple vertical projection.
Clip texturing an urban model, however, is less straightforward of a software application. orthorectified imagery does not always map onto vertical building faces properly. There is no projection direction that will map all the building faces. The building textures comprise a set of non-contiguous images that cannot easily be combined into a monotonic contiguous mosaic. This problem is especially apparent in an urban model having a number of three-dimensional objects, typically representing buildings and similar vertical structures. It has been found that it is not necessary to combine contiguous images into a monotonic contiguous mosaic. It has been found that sufficient results are achieved by arranging the individual face textures so that spatial locality is maintained.
Referring now to
Once the individual images are pre-processed, image sizes for each contributing image are loaded into memory (block 1160). These dimensions are sorted by area and image length (block 1180). A new image size having the smallest area, with the smallest perimeter, is calculated, which will contain all the building's individual textures (block 1200). The individual building textures are efficiently packed into the new image by tiling them alternately from left to right and vice versa, such that the unused space in the square is minimized (block 1220).
A site model clip map image is next created. Because each composite building texture map (CBTM) is as small as possible, placing each one spatially correct in a large clip map is realizable. Initially, each composite building texture map is placed in its correct spatial position in a large site model clip map (block 1240). A scale parameter is used to initially space buildings at further distances from each other while maintaining relative spatial relations (block 1260). Then each composite building texture map is checked for overlap against the other composite building texture maps in the site model clip map (block 1280). The site model clip map is expanded from top right to bottom left until no overlap remains (block 1300). For models with tall buildings, a larger positive scale parameter may be used to allow for the increased likelihood of overlap. All texture map coordinates are scaled and translated to their new positions in the site model clip map image.
Referring now to
A clip center or point in the clip map is used to define the location of the highest resolution imagery within the clip map (block 1540). Determining this center for a terrain surface clip map is actually achievable with little system complexity because a single clip texture maps contiguously onto the terrain elevation surface, so the camera coordinates are appropriate. The site model clip map has a clip center of its own and is processed according to its relative size and position on the terrain surface (block 1560). The site model clip map, however, does introduce some locality limitations resulting from tall buildings or closely organized buildings. This necessitates the use of an additional look-up table to compensate for the site model clip map's lack of complete spatial coherence. The purpose of the clip grid is to map three-dimensional spatial coordinates to clip center locations in the spatially incoherent clip map.
The clip grid look-up table indices are calculated using a x,y scene location (the camera position) (block 1580). If the terrain clip map and site model clip map are different sizes, a scale factor is introduced to normalize x,y scene location for the site model clip map (block 1600). It has been found that with sufficient design and advances in the development of the spatial correctness of the building clip map, the need for the clip grid look-up table can be eliminated in up to 95% of the cases.
It is also possible to extend the algorithm and use multiple site model clip maps. Using many smaller clip maps rather than one large clip map may prove to be a useful approach if clip maps of various resolutions are desired or if the paging in and out of clip maps from process space is achievable. However, it requires the maintenance of multiple clip centers and the overhead of multiple clip map pyramids.
The RealSite™ image modeling software has advantages over traditional methods because models can be very large (many km2) and can be created in days versus weeks and months of other programs. Features can be geodetically preserved and can include annotations and be geospatially accurate, for example, one meter or two meter relative. Textures can be accurate and photorealistic and chosen from the best available source imagery and are not generic or repeating textures. The InReality™ program can provide mensuration where a user can interactively measure between any two points and obtain an instant read-out on the screen of a current distance and location. It is possible to find the height of a building, the distance of a stretch of highway, or the distance between two rooftops along with Line-of-Sight information in accordance with the present invention. There are built-in intuitive navigation controls with motion model cameras that “fly” to a desired point of view. The InReality™ viewer can be supported under two main platforms and operating systems: (1) the SGI Onyx2 Infinite Reality2™ visualization supercomputer running IRIX 6.5.7 or later and an X86-based PC running either Microsoft WindowsNT 4.0 or Windows 98 or more advanced systems. The IRIX version of the InReality™ viewer can take full advantage of high-end graphics capabilities provided by Onyx2 such as MIPMapping in the form of clip textures, multi-processor multi-threading, and semi-immersive stereo visualization that could use Crystal Eyes by Stereo Graphics. InReality™ for Windows allows great flexibility and scalability and can be run on different systems.
Crystal Eyes produced by Stereo Graphics Corporation can be used for stereo 3D visualization. Crystal Eyes is an industry standard for engineers and scientists who can develop, view and manipulate 3D computer graphic models. It includes liquid crystal shutter eyewear for stereo 3D imaging.
Another graphics application that could be used is disclosed in commonly assigned U.S. Pat. No. 6,346,938, the disclosure which is hereby incorporated by reference in its entirety.
Many modifications and other embodiments of the invention will come to the mind of one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is understood that the invention is not to be limited to the specific embodiments disclosed, and that modifications and embodiments are intended to be included within the scope of the appended claims.
