The present invention generally relates to an aerial surveying method, and more particularly, the invention relates to an aerial surveying method that combines overlapping video imagery into an overall mosaic using modified 3-D ray tracing and graphics methodologies.
Unmanned Aerial Vehicles (UAV) or other manned aircraft can fly over areas of interest and make video images of those areas. Such an aerial surveillance has both military and civilian applications in the areas of reconnaissance, security, land management and natural disaster assessment to name a few. The heavily overlapped video imagery produced by the surveillance typically may be transmitted to a ground station where the images can be viewed. However, this heavily overlapped imagery shows only small pieces of a larger area of interest and in that respect is similar to the pieces of a jigsaw puzzle. Until all the pieces are put together in context, the meaning of any individual piece may be misunderstood or unclear. Therefore, a mosaic of the overlapping imagery data is needed. Prior art uses various approaches to merge such overlapping imagery data into an overall mosaic for further use and analysis.
Photogrammetrists mosaic images, but these images are typically orthorectified before the final mosaic is produced. Orthorectification, the process of converting oblique pairs of images into a single corrected top down view, requires stereo images or at least two images taken from different angles using a very high-resolution camera under strictly controlled conditions. Moreover, photogrammetry can be a labor-intensive task, requiring a human operator to place control points in the images prior to further processing. Conversely, NASA has provided image mosaics of planets, the moon and solar systems. Some newer techniques involve wavelet decomposition with equidistant measurement; whereas, older systems refer to more classical photogrammetry approaches to image mosaicking. All of the aforementioned prior art approaches are computation intensive and require extensive data collection.
Others in the field generate what are commonly known as, “Waterfall Displays” with each new image pasted at the end of a strip of the past several images. Old images roll off one end as new images are pasted on the other end. These images are not integrated in any way but are somewhat geo-referenced because one frame tends to be adjacent to the next frame in space. Nevertheless, a Waterfall Display is not a mosaic; it is just a collection of adjacent frames of video. Still others attempt to combine the images using the Optical Flow technique, which combines images based on detected image content but does not utilize geo-referencing information. Still others attempt to merge the pictures without proper integration; instead, the latest image is glued on top of whatever images came before, thereby, losing all the information from the previous images. In this case, the adjacent image edges are not blended resulting in a crude paste up appearance.
These traditional mosaicking techniques as discussed above do not provide a method for rapidly merging heavily overlapped imagery data utilizing geo-referencing information, without extensive data collection or computation.
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files and records, but otherwise reserves all copyrights whatsoever.
In illustrative embodiments, a method merges overlapping imagery data collected during an aerial survey into an overall mosaic of the images to provide useful, integrated information to an image viewer rapidly and without extensive data collection or computation.
To those ends, in various embodiments of the invention, aerial reconnaissance either manned or unmanned may collect the image data over an area of interest along with the supporting numerical Global Positioning System (GPS), Inertial Navigation System (INS) and camera angle data. The mosaics of smaller images provide the viewer with a comprehensive, integrated look at a larger area. A decision maker can review the mosaic level images for current conditions or changes from previous mosaicked images. Automated image processing software could also compare two mosaicked images for differences. Some embodiments have applications in both military and civilian fields, such as military reconnaissance, security, natural disaster assessment, and land management such as fire fighting and drought assessment.
In accordance with one embodiment of the invention, a method of taking an aerial survey maps boundaries of a first image and a second image from a first plane to a second plane to determine boundaries of an output image in the second plane; and for a plurality of pixels in the output image determines a corresponding pixel of either the first image or second image in the first plane.
In accordance with another embodiment of the invention, an aerial survey method maps boundaries of a plurality of images in a first plane to a second plane to determine the boundaries of an output image in the second plane, the plurality of images in the first and second planes and the output image having a plurality of pixels; and for the plurality of pixels in the output image, this embodiment determines a corresponding pixel of the plurality of images in the first plane.
In accordance with yet another embodiment of the invention, an aerial survey method defines an image plane with a plurality of image portions with a resolution; receives one of a plurality of pictures of at least a part of a ground area; divides the ground area part based on the resolution of the image plane to form a plurality of ground portions; and uses ray tracing mathematics to map the plurality of ground portions to the plurality of image portions.
a is a flowchart showing the overlapping pixel selection process.
b is a flowchart showing the Best Pixel Rule.
c is a flowchart showing the Overlapping Pixel Tolerance Rule.
Embodiments of the invention involve a two-pass process for combining individual video images, or still frames, with GPS, INS and camera angle data provided per frame, into one larger, oblique mosaic image. Such embodiments do not require stereo imagery, multiple sets of video on the same area to create stereo imagery, or ortho-rectification of the imagery prior to creating the mosaic. Unless the context otherwise requires, the two-pass process used to create the output image or mosaic is further defined as follows:
The 3-D ray trace intersection of a ray with an oblate spheroid is derived using the WGS84 ellipsoid whose parameters are specified for the GPS coordinate model. This equation is not a standard ray tracing equation as presented in the standard ray tracing literature, which mostly deals with simple geometric shapes like spheres, boxes, cones, planes and tori. The geo-location intersection calculations utilizing the ray trace intersection with the oblate spheroid as shown below.
In an alternative embodiment, multiple output images could be created should a single image be too large for practical usage due to memory limitations. Overall, a single virtual output image can be mapped with the final output of that image tiled in a simple way to an actual output image. These multiple, adjacent, mosaicked images could then be panned to create the same impact of a single mosaicked image.
The input images are heavily overlapped in terms of the ground area. Due to variations in altitude, camera pointing direction and camera angles the pixels on each still input image may be at a different scale in terms of pixel per meters covered on the ground area. Adjustments can be made for each image to map the input pixels to the output pixels based on pixels per meter so that the output image is scaled appropriately. Multiple input pixels may map to the same output pixel.
Furthermore, the geo-referenced coordinate of each pixel is approximate. The GPS coordinate of the camera is known within an error tolerance of the GPS device estimating the location. The INS device provides the camera orientation angles within an error tolerance. The geo-referenced coordinates of the pixels in each image is an estimate based on the GPS location of the camera and the INS data. The estimated geo-referenced coordinate of each pixel may be off slightly from the actual location on the ground. The geo-referenced estimates of pixels closer to the center of the image may be more accurate than the estimates for the pixels on the edge of the image. The error in pixel geo-referencing estimates increases the difficulty in getting a perfect image registration. Image registration refers to aligning one or more images so that the corresponding pixels in those images are on top of each other. With multiple input pixels mapping to the same output pixel, rules must be used to decide which pixel(s) to keep and which to discard.
Several rules are introduced to decide which input pixels to use or to combine to create the final output image pixel. If the pixels could be perfectly registered, then a simple averaging of all pixels that map to one location might suffice. However, the pixels are not perfectly registered with pixels at the edges of the images likely to contain the most positional error. A 3-D graphics, Z-buffer like rule (aka “The Best Pixel” rule, as shown in
The GPS coordinates of the camera, the horizontal and vertical camera field of view angles (FOV) as well as the INS orientation angles are used in the calculations to mathematically determine the area of the earth covered. The INS data is provided in the form of pitch, roll and yaw angles, which can be used to generate both a pointing (heading) vector and a rotation matrix. The rotation matrix is used to orient the camera's image plane and the ground area covered by the image.
The sampling rates of GPS measurements and INS measurements may not precisely match the time that a given image was taken. In this case, simple numeric interpolation can be used to estimate intermediate values between two sets of sampled values for GPS and INS data.
A pyramid is constructed using the camera as the apex of the pyramid. The four corners of the pyramid extend from the camera position, through the four corners of the camera image plane and those edges are then extended until they intersect the ground. The corner intersection rays run along the edges of the pyramid as shown in the detailed camera pyramid diagram in
Each input image must be scaled, rotated and translated to position the input image to map to the output image.
Utilizing this two-pass process, some embodiments rapidly map multiple overlapping, oblique aerial video images or still images into a single output mosaic utilizing 3-D ray tracing and graphics methodologies to geo-reference the pixels in the original video images to the final output image.
Further details, including specific code instructions of the two-pass process, for computing the mosaic are outlined and further explained below. These details illustrate one of a variety of different embodiments.
Pass One—Positioning and Calculating the Output Image Size:
Step 1: Synchronize Image and Numeric Support Data
Step 2: Pointing Vector
Step 3: Rotation Matrix
Step 4: Image Plane Pyramid
Step 5: Earth Plane Pyramid
Step 6: The Ray-Oblate Spheroid Intersection is defined as follows:
Step C: Standard Cartesian (Geocentric) to GPS (Geographic) Coordinate Conversion
Step 6: Miscellaneous Image Range Calculations
Compute Output Pixel Position
Step 7: Calculating the Output Size
Pass Two—Mapping the Input Images to the Output Image:
Step 1: Pass One Numeric Data
Step 2: Extract Image
Step 3: Compute Shortest Distance To Camera for Frame
Step 4: Pixel Mapping
Step 4a: Compute Output Image Corner Indices for Current Input Image:
Step 4b: Compute Start and Stop Output Image Indices
Step 4c: Compute Rectified Corner Output Image Indices
Step 4d: Reverse Map Pixels to Output Image
Compute Start and Stop Indices on Trapezoid
CalculateIntersectionPoint
Step 4e: Ray Triangle Intersection
Step 4: Compute Normalized Image Plane Coordinate from Cartesian Intersection
Step 5: Scale Output Image and Output Data
Other variations obvious to one of ordinary skill in the art include:
Various embodiments of the invention may be implemented at least in part in any conventional computer programming language. For example, some embodiments may be implemented in a procedural programming language (e.g., “C”), or in an object oriented programming language (e.g., “C++”). Other embodiments of the invention may be implemented as preprogrammed hardware elements (e.g., application specific integrated circuits, FPGAs, and digital signal processors), or other related components.
In an alternative embodiment, the disclosed apparatus and methods (e.g., see the various flow charts described above) may be implemented as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium. The medium may be a tangible medium (e.g., optical or analog communications lines). The series of computer instructions can embody all or part of the functionality previously described herein with respect to the system.
Those skilled in the art should appreciate that such computer instructions can be written in a number of programming languages for use with many computer architectures or operating systems. Furthermore, such instructions may be stored in any memory device, such as semiconductor, magnetic, optical or other memory devices, and may be transmitted using any communications technology, such as optical, infrared, microwave, or other transmission technologies.
Among other ways, such a computer program product may be distributed as a removable medium with accompanying printed or electronic documentation (e.g., shrink-wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the network (e.g., the Internet or World Wide Web). Of course, some embodiments of the invention may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other embodiments of the invention are implemented as entirely hardware, or entirely software.
Although the above discussion discloses various exemplary embodiments of the invention, it should be apparent that those skilled in the art could make various modifications that will achieve some of the advantages of the invention without departing from the true scope of the invention.
This patent application claims priority from U.S. Provisional Patent Application Ser. No. 60/987,883, entitled, “Method and Apparatus of Taking Aerial Surveys,” filed on Nov. 14, 2007, and naming Elaine S. Acree as inventor, the disclosure of which is incorporated herein, in its entirety, by reference.
Number | Name | Date | Kind |
---|---|---|---|
4525858 | Cline et al. | Jun 1985 | A |
4945478 | Merickel et al. | Jul 1990 | A |
5684935 | Demesa, III et al. | Nov 1997 | A |
5911035 | Tsao | Jun 1999 | A |
5987164 | Szeliski et al. | Nov 1999 | A |
6009190 | Szeliski et al. | Dec 1999 | A |
6044181 | Szeliski et al. | Mar 2000 | A |
6078701 | Hsu et al. | Jun 2000 | A |
6157747 | Szeliski et al. | Dec 2000 | A |
6219462 | Anandan et al. | Apr 2001 | B1 |
6256036 | Matsumoto | Jul 2001 | B1 |
6377278 | Curtright et al. | Apr 2002 | B1 |
6393163 | Burt et al. | May 2002 | B1 |
6522787 | Kumar et al. | Feb 2003 | B1 |
6597818 | Kumar et al. | Jul 2003 | B2 |
6757445 | Knopp | Jun 2004 | B1 |
6819318 | Geng | Nov 2004 | B1 |
6947591 | Risson | Sep 2005 | B2 |
20030235344 | Kang et al. | Dec 2003 | A1 |
20050111753 | Peleg et al. | May 2005 | A1 |
Number | Date | Country |
---|---|---|
WO 2005013001 | Feb 2005 | WO |
WO 2007031248 | Mar 2007 | WO |
Number | Date | Country | |
---|---|---|---|
20100283853 A1 | Nov 2010 | US |
Number | Date | Country | |
---|---|---|---|
60987883 | Nov 2007 | US |