1. Statement of the Technical Field
The present invention is directed to the field of visualization of point cloud data, and more particularly for visualization of point cloud data based on scene content.
2. Description of the Related Art
Three-dimensional (3D) type sensing systems are commonly used to generate 3D images of a location for use in various applications. For example, such 3D images are used creating a safe training or planning environment for military operations or civilian activities, for generating topographical maps, or for surveillance of a location. Such sensing systems typically operate by capturing elevation data associated with the location. One example of a 3D type sensing system is a Light Detection And Ranging (LIDAR) system. LIDAR type 3D sensing systems generate data by recording multiple range echoes from a single pulse of laser light to generate a frame sometimes called image frame. Accordingly, each image frame of LIDAR data will be comprised of a collection of points in three dimensions (3D point cloud) which correspond to the multiple range echoes within sensor aperture. These points can be organized into “voxels” which represent values on a regular grid in a three dimensional space. Voxels used in 3D imaging are analogous to pixels used in the context of 2D imaging devices. These frames can be processed to reconstruct a 3D image of the location. In this regard, it should be understood that each point in the 3D point cloud has an individual x, y and z value, representing the actual surface within the scene in 3D.
To further assist interpretation of the 3D point cloud, colormaps have been used to enhance visualization of the point cloud data. That is, for each point in a 3D point cloud, a color is selected in accordance with a predefined variable, such as altitude. Accordingly, the variations in color are generally used to identify points at different heights or at altitudes above ground level. Notwithstanding the use of such conventional colormaps, 3D point cloud data has remained difficult to interpret.
Embodiments of the present invention provide systems and method for visualization of spatial or point cloud data using colormaps based on scene content. In a first embodiment of the present invention, a method for improving visualization and interpretation of spatial data of a location. The method includes selecting a first scene tag from a plurality of scene tags for a first portion of a radiometric image data of the location and selecting a first portion of the spatial data, where the spatial data includes a plurality of three-dimensional (3D) data points associated with the first portion of the radiometric image data. The method also includes selecting a first color space function for the first portion of the spatial data from a plurality of color space functions, the selecting based on the first scene tag, and each of the plurality of color space functions defining hue, saturation, and intensity (HSI) values as a function of an altitude coordinate of the plurality of 3D data points. The method further includes displaying the first portion of the spatial data using the HSI values selected from the first color space function using the plurality of 3D data points associated with the first portion of the spatial data. In the method, the plurality of scene tags are associated with a plurality of classifications, where each of the plurality of color space functions represents a different pre-defined variation in the HSI values associated one of the plurality of classifications.
In a second embodiment of the present invention, a system for improving visualization and interpretation of spatial data of a location is provided. The system includes a storage element for receiving the spatial data and radiometric image data associated with the location and a processing element communicatively coupled to the storage element. In the system, the processing element is configured for selecting a first scene tag from a plurality of scene tags for a first portion of a radiometric image data of the location and selecting a first portion of the spatial data, the first portion of the spatial data includes a plurality of three-dimensional (3D) data points associated with the first portion of the radiometric image data. The processing element is also configured for selecting a first color space function for the first portion of the spatial data from a plurality of color space functions, the selecting based on the first scene tag, and each of the plurality of color space functions defining hue, saturation, and intensity (HSI) values as a function of an altitude coordinate of the plurality of 3D data points. The system is further configured for displaying the first portion of the spatial data using the HSI values selected from the first color space function using the plurality of 3D data points associated with the first portion of the spatial data. In the system, the plurality of scene tags are associated with a plurality of classifications, where each of the plurality of color space functions represents a different pre-defined variation in the HSI values associated one of the plurality of classifications.
In a third embodiment of the present invention, a computer-readable medium, having stored thereon a computer program for improving visualization and interpretation of spatial data of a location is provided. The computer program includes a plurality of code sections, the plurality of code sections executable by a computer. The computer program includes code sections for selecting a first scene tag from a plurality of scene tags for a first portion of a radiometric image data of the location and selecting a first portion of the spatial data, the spatial data includes a plurality of three-dimensional (3D) data points associated with the first portion of the radiometric image data. The computer program also includes code sections for selecting a first color space function for the first portion of the spatial data from a plurality of color space functions, the selecting based on the first scene tag, and each of the plurality of color space functions defining hue, saturation, and intensity (HSI) values as a function of an altitude coordinate of the plurality of 3D data points. The computer program further includes code sections for displaying the first portion of the spatial data using the HSI values selected from the first color space function using the plurality of 3D data points associated with the first portion of the spatial data. In the computer program, the plurality of scene tags are associated with a plurality of classifications, where each of the plurality of color space functions represents a different pre-defined variation in the HSI values associated one of the plurality of classifications.
The present invention is described with reference to the attached figures, wherein like reference numerals are used throughout the figures to designate similar or equivalent elements. The figures are not drawn to scale and they are provided merely to illustrate some embodiments of the present invention. Several aspects of the invention are described below with reference to example applications for illustration. It should be understood that numerous specific details, relationships, and methods are set forth to provide a full understanding of the invention. One having ordinary skill in the relevant art, however, will readily recognize that the invention can be practiced without one or more of the specific details or with other methods. In other instances, well-known structures or operations are not shown in detail to avoid obscuring the invention. The present invention is not limited by the illustrated ordering of acts or events, as some acts may occur in different orders and/or concurrently with other acts or events. Furthermore, not all illustrated acts or events are required to implement a methodology in accordance with the present invention.
A 3D imaging system generates one or more frames of 3D point cloud data. One example of such a 3D imaging system is a conventional LIDAR imaging system. In general, such LIDAR systems use a high-energy laser, optical detector, and timing circuitry to determine the distance to a target. In a conventional LIDAR system one or more laser pulses is used to illuminate a scene. Each pulse triggers a timing circuit that operates in conjunction with the detector array. In general, the system measures the time for each pixel of a pulse of light to transit a round-trip path from the laser to the target and back to the detector array. The reflected light from a target is detected in the detector array and its round-trip travel time is measured to determine the distance to a point on the target. The calculated range or distance information is obtained for a multitude of points comprising the target, thereby creating a 3D point cloud. The 3D point cloud can be used to render the 3-D shape of an object.
In general, interpreting 3D point cloud data to identify objects in a scene can be difficult. Since the 3D point cloud specifies only spatial information with respect to a reference location, at best only height and shape of objects in a scene is provided. Some conventional systems also provide an intensity image along with the 3D point cloud data to assist the observer in ascertaining height differences. However, the human visual cortex typically interprets objects being observed based on a combination of information about the scene, including the shape, the size, and the color of different objects in the scene. Accordingly, a conventional 3D point cloud, even if associated with an intensity image, generally provides insufficient information for the visual cortex to properly identify many objects imaged by the 3D point cloud. In general, the human visual cortex operates by identifying observed objects in a scene based on previously observed objects and previously observed scenes. As a result, proper identification of objects in a scene by the visual cortex relies on not only on identifying properties of an object, but also identifying known associations between different types of objects in a scene.
To overcome the limitations of conventional 3D point cloud display systems and to facilitate the interpretation of 3D point cloud data by the human visual cortex, embodiments of the present invention provide systems and methods for applying different colormaps to different areas of the 3D point cloud data based on a radiometric image. In particular, different colormaps, associated with different terrain types, are associated with the 3D point cloud data according to tagging or classification of associated areas in an radiometric image. For example, if an area of the radiometric image shows an area of man-made terrain (e.g., an area where the terrain is dominated by artificial or man-made features such as buildings, roadways, vehicles), a colormap associated with a range of colors typically observed in such areas is applied to a corresponding area of the 3D point cloud. In contrast, if an area of the radiometric image shows an area of natural terrain (e.g., an area dominated by vegetation or other natural features such as water, trees, desert), colormaps associated with a range of colors typically observed in these types of areas is applied to a corresponding area of the 3D point cloud. As a result, by applying different colormaps to different portions of the 3D point cloud, colors that are more likely associated with the shapes of objects in the different portions of the 3D point cloud are presented to the observer and are more easily recognizable by the human visual cortex.
The term “radiometric image”, as used herein, refers to an two-dimensional representation (an image) of a location obtained by using one or more sensors or detectors operating on one or more electromagnetic wavelengths.
An exemplary data collection system 100 for collecting 3D point cloud data and associated image data according to an embodiment of the present invention is shown in
In the various embodiments of the inventions, the physical volume 108 is imaged using a variety of different sensors. As shown in
In some instances, the line of sight between sensors 102-i and 102-j and an object 104 may be partly obscured by another object (occluding object) 106. In the case of a LIDAR system, the occluding object 106 can comprise natural materials, such as foliage from trees, or man made materials, such as camouflage netting. It should be appreciated that in many instances, the occluding object 106 will be somewhat porous in nature. Consequently, the sensors 102-i, 102-j will be able to detect fragments of object 104 which are visible through the porous areas of the occluding object 106. The fragments of the object 104 that are visible through such porous areas will vary depending on the particular location of the sensor.
By collecting data from several poses, such as at sensors 102-i and 102-j, an aggregation of 3D point cloud data can be obtained. Typically, aggregation of the data occurs by means of a registration process. The registration process combines the data from two or more frames by correcting for variations between frames with regard to sensor rotation and position so that the data can be combined in a meaningful way. As will be appreciated by those skilled in the art, there are several different techniques that can be used to register this data. Subsequent to such registration, the aggregated 3D point cloud data from two or more frames can be analyzed to improve identification of an object 104 obscured by an occluding object 106. However, the embodiments of the present invention are not limited solely to aggregated data. That is, the 3D point cloud data can be generated using multiple image frames or a single image frame.
In the various embodiments of the present invention, the radiometric image data collected by sensors 103-j and 103-j can include intensity data for an image acquired from various radiometric sensors, each associated with a particular range of wavelengths (i.e., a spectral band). Therefore, in the various embodiments of the present invention, the radiometric image data can include multi-spectral (˜4 bands), hyper-spectral (>100 bands), and/or panchromatic (single band) image data. Additionally, these bands can include wavelengths that are visible or invisible to the human eye.
In the various embodiments of the present invention, aggregation of 3D point cloud data or fusion of multi-band radiometric images can be performed using any type of aggregation or fusion techniques. The aggregation or fusion can be based on registration or alignment of the data to be combined based on meta-data associated with the 3D point cloud data and the radiometric image data. The meta-data can include information suitable for facilitating the registration process, including any additional information regarding the sensor or the location being imaged. By way of example and not limitation, the meta-data includes information identifying a date and/or a time of image acquisition, information identifying the geographic location being imaged, or information specifying a location of the sensor. For example, information indentifying the geographic location being image can include geographic coordinates for the four corners of a rectangular image can be provided in the meta-data.
Although, the various embodiments of the present invention will generally be described in terms of one set of 3D point cloud data for a location being combined with a corresponding set of one radiometric image data set associated with the same location, the present invention is not limited in this regard. In the various embodiments of the present invention, any number of sets of 3D point cloud data and any number of radiometric image data sets can be combined. For example, mosaics of 3D point cloud data and/or radiometric image data can be used in the various embodiments of the present invention.
In the various embodiments of the present invention, 3D point cloud data is color coded for improved visualization. For example, a display color of each point of 3D point cloud data is selected in accordance with an altitude or z-axis location of each point. In order to determine which specific colors are displayed for points at various z-axis coordinate locations, a colormap can be used. For example, a red color could be used for all points located at a height of less than 3 meters, a green color could be used for all points located a heights between 3 meters and 5 meters, and a blue color could be used for all points located above 5 meters. A more detailed colormap can use a wider range of colors which vary in accordance with smaller increments along the z axis. Although the use of a colormap can be of some help in visualizing structure that is represented by 3D point cloud data, applying a single conventional colormap to all points in the 3D point cloud data is generally not effective for purposes of improving visualization. First of all, providing a range of colors that is too wide, such as in a conventional red, green, blue (RGB) colormap, provides a variation the color coding for the 3D point cloud that is incongruent with color variation typically observed in objects. Second, providing a single conventional colormap provides incorrect coloring for some types of scenes. Accordingly, embodiments of the present invention instead provide improved 3D point cloud visualization that uses multiple colormaps for multiple types of terrain in an imaged location, where the multiple colormaps can be tuned for different types of features (i.e. buildings, trees, roads, water) typically associated with the terrain. Such a configuration allows different areas of the 3D point cloud data to be color coded using colors for each area that are related to the type of objects in the areas, allowing improved interpretation of the 3D point cloud data by the human visual cortex.
Although any types of different colormaps can be used, in some embodiments of the present invention non-linear colormaps defined in accordance with hue, saturation and intensity (HSI color space) can be used for each type of scene. As used herein, “hue” refers to pure color, “saturation” refers to the degree or color contrast, and “intensity” refers to color brightness. Thus, a particular color in HSI color space is uniquely represented by a set of HSI values (h, s, i) called triples. The value of h can normally range from zero to 360° (0°≦h≦360°. The values of s and i normally range from zero to one (0≦s, ≦1), (0≦i≦1). For convenience, the value of h as discussed herein shall sometimes be represented as a normalized value which is computed as h/360.
Significantly, HSI color space is modeled on the way that humans generally perceive color and can therefore be helpful when creating different colormaps for visualizing 3D point cloud data for different scenes. Furthermore, HSI triples can easily be transformed to other colors space definitions such as the well known RGB color space system in which the combination of red, green, and blue “primaries” are used to represent all other colors. Accordingly, colors represented in HSI color space can easily be converted to RGB values for use in an RGB based device. Conversely, colors that are represented in RGB color space can be mathematically transformed to HSI color space. An example of this relationship is set forth in the table below:
As described above, one of the difficulties in interpreting 3D point cloud data is that the human visual cortex generally expects a particular range of colors to be associated with a particular type of terrain being observed. This is conceptually illustrated with respect to
Therefore in the various embodiments of the present invention, the colormaps applied to different areas of the imaged location are selected to be appropriate for the types of objects in the location. For example,
Similarly,
In some embodiments of the present invention, some types of objects can be located in several types of areas, such as ground-based vehicles. In general, a ground-based vehicle will generally have a height within a predetermined target height range 406. That is, the structure of such objects will extend from a ground level 405 to some upper height limit 408. The actual upper height limit will depend on the particular types of vehicles. For example a typical height of a truck, bus, or military vehicle is generally around 3.5 meters. A typical height of a passenger car is generally around 1.5 meters. Accordingly, in both the rural and urban colormaps, the data points at such heights can be provided a different color to allow easier identification of such objects, regardless of the type of scene being observed. For example, a color that is not typically encountered in the various scenes can be used to highlight the location of such objects to the observer.
Referring now to
The normalized curves representing saturation and intensity also have a local peak value at the upper height limit 408 of the target range. However, the normalized curves 504 and 506 for saturation and intensity are non-monotonic, meaning that they do not steadily increase or decrease in value with increasing elevation (altitude). According to an embodiment of the invention, each of these curves can first decrease in value within a predetermined range of altitudes above the target height range 408, and then increases in value. For example, it can be observed in
Notably, the peak in the normalized curves 504, 506 for saturation and intensity causes a spotlighting effect when viewing the 3D point cloud data. Stated differently, the data points that are located at the approximate upper height limit of the target height range will have a peak saturation and intensity. The visual effect is much like shining a light on the tops of the target, thereby facilitating identification of the presence and type of target. The second peak in the saturation curve 504 at treetop level has a similar visual effect when viewing the 3D point cloud data. However, in this case, rather than a spotlight effect, the peak in saturation values at treetop level creates a visual effect that is much like that of sunlight shining on the tops of the trees. The intensity curve 506 shows a localized peak as it approaches the treetop level. The combined effect helps greatly in the visualization and interpretation of the 3D point cloud data, giving the data a more natural look.
Referring now to
Referring now to
Referring again to the exemplary colormap for natural locations in
The colormap in
Referring again to the exemplary colormap for natural locations in
The colormap in
By selecting the colormap for natural areas or locations in
Referring once again to the exemplary colormap for natural locations in
Notably, the second peak in saturation and intensity curves 504, 506 occurs at treetop level 410. As shown in the exemplary color map for natural locations in
A similar coloration effect is shown in
Referring again to the exemplary colormap for urban locations in
The colormap in
Referring again to exemplary colormap for urban areas in
The exemplary colormap in
By selecting the exemplary colormap for urban areas in
Referring once again to the exemplary colormap for urban areas in
As described above, prior to applying the various colormaps to different portions of the imaged location, a scene tag or classification is obtained for each portion of the imaged location. This process is conceptually described with respect to
Although feature detection for
However, embodiments of the present invention are not limited solely to geometric methods. In some embodiments of the present invention, analysis of the radiometric data itself can be used for scene tagging or classification. For example, a spectral analysis can be performed to find areas of vegetation using the near (˜750-900 nm) and/or mid (˜1550-1750 nm) infrared (IR) band and red (R) band (˜600-700 nm) from a multi-spectral image. In such embodiments, calculation of the normalized difference vegetation index (NDVI=(IR−R)/(IR+R)) can be used to identify regions of healthy vegetation. In such an analysis, areas can be tagged according to the amount of healthy vegetation (e.g., <0.1 no vegetation, 0.2-0.3 shrubs or grasslands, 0.6-0.8 temperate and/or tropical rainforest). However, the various embodiments of the present invention are not limited to identifying features using any specific bands. In the various embodiments of the present invention, any number and types of spectral bands can be evaluated to identify features and to provide tagging or classification of features or areas.
In the various embodiments of the present invention, feature detection is not limited to one methods. Rather in the various embodiments of the present invention, any number of feature detection methods can be used. For example, a combination of geometric and radiometric analysis methods can be used to identify features in the radiometric image 800.
Once the features of interest (for classification or tagging purposes) are detected in the radiometric image 800, the radiometric image 800 can be divided into a plurality of regions 804 to form a grid 806, for example, as shown in
Although an radiometric image can be assigned into gridded into regions, in some embodiments of the present invention, the radiometric image can be divided into regions based on the locations of features (i.e., markings 802). For example, the regions 804 can be selected by first identifying locations within the radiometric image 800 with large numbers of identified features and centering the grid 806 to provide minimum number of regions for such areas. The position of the first ones of regions 804 is selected such that a minimum number is used for such locations. The designation of other ones regions 804 can then proceed from this initial placement. After a colormap is selected for each portion of the radiometric image, the 3D point cloud data can be registered or aligned with the radiometric image. Such registration can be based on meta-data associated with the radiometric image and the 3D point cloud data, as described above. Alternatively, in embodiments, where a spectral analysis method is used, each pixel of the radiometric image could be considered a separate region. As a result, the colormap can vary from pixel to pixel in the radiometric image.
Although only one exemplary embodiment of a grid is illustrated in
After the 3D point cloud data and the radiometric image are registered, the colormap for each of regions 804 is then used to add color data to the 3D point cloud data. A set of exemplary results of this process is shown in
Although classification of portions of a 3D point cloud has been described with respect to exemplary urban or natural scene tags and corresponding colormaps, embodiments of the present invention are not limited solely to these two types of scene tags. In the various embodiments of the present invention, any number and types of scene tags can be used. For example, one classification scheme can include tagging for agricultural or semi-agricultural areas (and corresponding colormaps) in addition to natural and urban area tagging. Furthermore, for each of these area, subclasses of these area can also be tagged and have different colormaps. For example, agricultural and semi-agricultural areas can be tagged according to crop or vegetation type, as well as use type. Urban areas can be tagged according to use as well (e.g., residential, industrial, commercial, etc.). Similarly, natural areas can be tagged according to vegetation type or water features present. However, the various embodiments of the present invention are not limited solely to any single type of classification scheme and any type of classification scheme can be used with the various embodiments of the present invention.
Furthermore, as previously described, each pixel of the radiometric image can be considered to be a different area of the radiometric image. Consequently, spectral analysis methods can be further utilized to identify specific types of objects in radiometric images. An exemplary result of such a spectral analysis is shown in
For example, a normalized difference vegetation index (NDVI) values, as previously described, can be used to identify vegetation and other features in a radiometric image and apply a corresponding colormap to associated points in the 3D point cloud data. For example, an exemplary result of such feature tagging is shown in
Although the various embodiments of the present invention have been discussed in terms of substantially constant elevation ground level, in many cases the ground level elevation can vary. If not accounted for, such elevation variations in the ground level within a scene represented by 3D point cloud data can make scene and object visualization difficult.
In some embodiments of the present invention, in order to account for variations in terrain elevation when applying the colormaps to the 3D data, the volume of a scene which is represented by the 3D point cloud data can be divided into a plurality of sub-volumes. This is conceptually illustrated with respect to
Each sub-volume 1002 can be aligned with a particular portion of the surface of the terrain represented by the 3D point cloud data. According to an embodiment of the invention, a ground level 405 can be defined for each sub-volume. The ground level 405 can be determined as the lowest altitude 3D point cloud data point within the sub-volume. For example, in the case of a LIDAR type ranging device, this will be the last return received by the ranging device within the sub-volume. By establishing a ground reference level for each sub-volume, it is possible to ensure that the colormaps used for the various portions of the 3D point cloud will be properly referenced to a true ground level for that portion of the scene.
In light of the foregoing description of the invention, it should be recognized that the present invention can be realized in hardware, software, or a combination of hardware and software. A method in accordance with the inventive arrangements can be realized in a centralized fashion in one processing system, or in a distributed fashion where different elements are spread across several interconnected systems. Any kind of computer system, or other apparatus adapted for carrying out the methods described herein, is suited. A typical combination of hardware and software could be a general purpose computer processor or digital signal processor with a computer program that, when being loaded and executed, controls the computer system such that it carries out the methods described herein.
The present invention can also be embedded in a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which, when loaded in a computer system, is able to carry out these methods. Computer program or application in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following a) conversion to another language, code or notation; b) reproduction in a different material form.
While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation. Numerous changes to the disclosed embodiments can be made in accordance with the disclosure herein without departing from the spirit or scope of the invention. Thus, the breadth and scope of the present invention should not be limited by any of the above described embodiments. Rather, the scope of the invention should be defined in accordance with the following claims and their equivalents.
Although the invention has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art upon the reading and understanding of this specification and the annexed drawings. In addition, while a particular feature of the invention may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, to the extent that the terms “including”, “includes”, “having”, “has”, “with”, or variants thereof are used in either the detailed description and/or the claims, such terms are intended to be inclusive in a manner similar to the term “comprising.”
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. I t will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.