Patent Grant
8717351
The invention generally relates to a computer readable medium encoded with computer readable instructions for reading an image.
It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not to be viewed as being restrictive of the invention, as claimed. Further advantages of this invention will be apparent after a review of the following detailed description of the disclosed embodiments, which are illustrated schematically in the accompanying drawings and in the appended claims.
Embodiments may be implemented as an apparatus or article of manufacture using programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof. The term computer-readable medium (devices, carriers, or media), includes, but is not limited to, a magnetic storage media, “floppy disk”, CD-ROM, RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, a File server providing access to the programs via a network transmission line, holographic unit, etc. Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope of the present invention.
Embodiments of the invention include a computer readable medium encoded with a data structure. The data structure includes an image File format including a PFI File Header that defines information needed to display a Precision Fires Image (PFI). The PFI is an overhead satellite image of an object with a geo-referenced, three dimensional (3-D) template of the object. The geo-referenced three dimensional (3-D) template provides a translation from image coordinates to ground coordinates. The image File format further includes an image segment pertaining to a “displayable image”, and an image segment pertaining to a “non-displayable” template image.
A Precision Fires Image (PFI) is an image from which a user may designate a point that is converted to a precision targeting coordinate. The PFI is a working National Imagery Transmission Format (NITF) Version 2.1 File extension that incorporates the Digital Point Positioning Data Base (DPPDB) structure as the basis for development. The PH provides a user with the ability to precisely designate items of interest within their field of view and area of influence by simply positioning a single marker, a cursor, on the desired item, a target. Precision targeting coordinates reduce non-combatant casualties, increase combatant casualties, reduce collateral damage, use munitions effectively and lower delivery costs while providing immediate detailed information regarding local terrain.
A PFI electronic File (“PFI File”) is generated using a National Imagery Transmission Format (NITF) File that consists of a single overhead satellite image, also known as a surveillance image, and a geo-referenced, three dimensional (3-D) template derived from a stereo referenced image (also referred to as “non-displayable” image). Several types of stereo referenced imagery are available and they include, the Digital Point Positioning Database (DPPDB), the Controlled Image Base (DB), Digital Terrain Elevation Data (DTED) and vector maps including VMAP or its commercial equivalents. Regardless of the type of stereo reference imagery used, the user is then forced to select one of two processing paths.
One path uses the stereo referenced image and a surveillance image provided from either a surveillance satellite or aircraft and invokes portions of the Digital Precision Strike Suite—Scene Matching (DPSS-SM) processing. DPSS-SM is an example of a path when the stereo referenced imagery and a surveillance image are both available. This is due to the timeliness and relevancy of the information included within the tactical image since a current satellite image or other current tactical image may present road movable targets.
A second path is selected in the absence of a surveillance image. The PFI software application is used to generate a PFI directly from the stereo referenced imagery when only the stereo referenced imagery is available. Regardless of the image source used to generate the PFI, the PFI enabled hand held is then used to accept a point designation from the user that is converted to a precision targeting coordinate and passed to the guided munitions.
During PFI File generation, the three dimensional (3-D) template and the stereo-referenced image are correlated, geo-registering a visible image with World Geodetic System 1984 (WGS-84) coordinate data, creating a single image File. To obtain the WGS-84 coordinates with height and elevation errors, an operator can perform a “single click” on the image. The PFI creation process occurs when the DPPDB stereo reference imagery pair is loaded into a PFI generating application. The user defines a region of interest by selecting a section from the left and right reference pair as the framework for an overlying three dimensional (3-D) template. One of the image pairs or a single tactical NITF image is used as the base image of the overall PFI File. Finally, the three dimensional (3-D) template and base image are correlated using an offset scheme to map the image pixel positions; and the final PFI product is created. Each stereo reference image undergoes extensive edge extraction, using edge extraction algorithm (in this case we used the Sobel algorithm). Once the imagery has completed edge erosion, it undergoes the correlation process where pixels are matched and template geographic locations are implemented.
A PFI is intended to be used on systems that lack the resources to run either the Precision Strike Suite for Special Operations (PSS-SOF) or Digital Precision Strike Suite—Scene Matching (DPSS-SM) applications. A typical File size nominally covers an area of 4×4 kilometers, allowing the image to be transmitted and viewed on low capacity computing devices with minimal storage and bandwidth constraints. Because of its size, multiple PFI image sets can be deployed at a given time, per area of focus. PFIs deployed on handheld devices also provide a platform for imagery and targeting capabilities that are beneficial to lower level echelon mission requirements.
The Digital Precision Strike Suite—Scene Matching (DPSS-SM) or Precision Fires Image Generator (PFIG) applications can be used to create an individual PFI product. DPSS-SM is the preferred source when overhead satellite imagery is directly available, due to the timeliness and relevancy of this data; PFIG, however, can be used to generate a PFI directly from the reference imagery if overhead imagery is not available. Both applications use an NGA validated algorithm to extract data from DPPDB stereo reference imagery in order to generate individual PFI Files.
An individual PFI is chipped from the DPPDB sizing up to 4096×4096 pixels. Support data, including latitude, longitude, elevation, CE, and LE, are implemented into the PFI structure and are essential in target designation and accuracy. PFIG performs PFI generation solely using DPPDB imagery for the correlation of a three dimensional (3-D) template and a single NITF image as the base for image pixilation. The DPSS-SM application uses the same correlation process but includes the use of up-to-date tactical imagery in the correlation process.
PFI supporting applications provide users with the capability to perform parallel operations in image exploitation and management. On a desktop platform, operators are expected to generate PFI images (using PFIG and DPSS-SM applications), and perform image categorization and File maintenance to include the exploitation of imagery. Operators then establish handheld synchronization for PFI management (using the Handheld Sync application), deploy Files onto handheld PFI viewing applications, and perform minimal exploitation and coordinate generation (while in a dismounted state).
In some embodiments, PFI Files are stored as raw uncompressed images. In other embodiments, PFI Files are stored using Joint Photographic Experts Group (JPEG) or JPEG 2000 compression; implementations of binary values, however, are not compressed within the PFI File structure.
The PFI File structure and content conforms to the NITFS, NITF Version 2.1 format specification in order to maintain interoperability between systems and differing agencies.
In some embodiments, compression, including, for example JPEG and JPEG 2000, and Gridded Reference Graphic (GRG) Text enhancements are included to allow for better compression, easier transmission, and more capability. Compression will be specified in the Image Header “IC” field.
Some embodiments of the invention include a scale integer in the “Image Segment Template” (4 in
The PFI File format configuration consists of multiple correlated Files. This configuration correlates a single “displayable” image segment with a “non-displayable” image segment. The “non-displayable” image segment includes correlated DPPDB stereo reference data and extended binary data allocated for PFI implementation.
The “displayable” image is the base image segment that conforms to the NITF 2.1 standard File Header structure; it is the underlying image that provides visible geographic features. The displayable image can also be stored as raw uncompressed pixel values or JPEG compressed.
The “non-displayable” template image will perform as the non-destructive overlay to the “displayable” image, with offsets to the X and Y values on the coordinate plane. The following syntax identifies the data structure for the “non-displayable” template within the PFI File Header:
struct. PFIData
{
Smallint row;
Smallint col;
float TemplateX;
float TemplateY;
float TemplateZ;
float TemplateDir;
};
struct PFIHeader
{
float Version;
bool IsSAR;
float CEP;
float CEM;
float CE90;
float LEP;
float LEM;
float LE90;
float SigmaX2;
float SigmaY2;
float SigmaXY;
double AimpointX;
double AimpointY;
double AimpointZ;
int scale; <--PFI 3.0 Version Implementation-->
};
The PFI structure implements a Tagged Record Extension (“TRE”) segment. This segment supports coordinate accuracy of the geo-location output for the four corner points. The TRE segment implementation will improve accuracy approximately one tenths of the standard NITF coordinate output. The TRE extension adheres to the standard TRE outlined in the “The Compendium of Controlled Extensions for the NITF 2.1.
GRGs—a product with an overprinted grid that may be either a non rectified image with a common reference grid or a rectified image with a precise metric grid—and text labels are user-defined labels implemented in the PFI File during PFI creation using the PFI Generator software application. GRG and text labels designate specified X and Y pixels on a PFI image.
The number of template point values is computed by subtracting the “Number of Columns” from the “Size of the PFI Header” divided by the “Size of the PFI Data.”
Number of Template Points=(Number of Columns)−(Size of PFI Header/Size of PFI Data)
The PFI binary code was developed using an Intel/AMD X86 processor on a Windows operating system. For development purposes, PFI byte ordering uses the “little endian” sequencing of data, which is determined at the machine processing level. For interoperability on systems other than the Windows platform (i.e., Macintosh/Sun Systems), the sequence of data must be parsed in the reverse order.
In order to maintain the integrity of PFI functionality on target based systems, the mathematical model and error terms will not be included in this document. As a result, a Dynamic Link Library (DLL) is available in order to provide third party applications with the capability to read PFI embedded information. DLL implementation is available for Windows operating systems and includes the code, data, and resources required for executing PFI generation.
Table A-1 identifies PFI File Headers. The table identifies PFI File Headers using a field ID, name (and associated information/function), size, and value range; however, the field ID, name, size and value range can vary without departing from the invention.
PFI Image Header.
Text File Header information. The Text Header specifies Gridded Reference
Graphics (GRG) labels for buildings and intersections. In some embodiments, numerical values indicate buildings and alphabetical values indicate intersections. However, in other embodiments, alphabetical values indicate buildings and numerical values indicate intersections. Table A-3 specifies Text File Header fields used in an embodiment of the invention that includes Text File Header Information. The table identifies Text File Headers using a field ID, name (and associated information/function), size, and value range; however, the field ID, name, size and value range can vary without departing from the invention.
Buildings: Numeric values are positioned on an X Pixel and Y Pixel and is divided into categories of “macro” and “micro” labels. A “macro” label is the beginning number to a series of “micro” numbers. In Table A-3, for example, numbers 10 and 20 start off the series to number sets 11-15 and 21-25. These number sets are defined as the “micro” labels. The number of “micro” labels assigned depends on mission requirements. Table A-3 provides an example of the GRG labeling system in order to identify the hierarchy of GRG text labels.
Intersections: Alphabetical values have been added to the “Text File Data 0” field to mark intersections or other physical features on the image.
Some embodiments include a tagged record extension (“TRE”) segment that supports coordinate accuracy of a geo-location output for the four corner points of the PFI. Table A-4 provides a table of Header fields for embodiments including a controlled TRE including additional geo-location identifiers intended to improve accuracy in coordinate output. The table identifies Text File Headers using a field ID, name (and associated information/function), size, and value range; however, the field ID, name, size and value range can vary without departing from the invention. The TRE is located in Image Header 0 and includes data associated with the Image Geographic Location (IGEOLO). The TRE provides enhanced precision accuracy of the four corner points.
Embodiments of a PFI Reader
Embodiments of a PFI Reader apparatus is/are described with reference to the block diagram 10 in
One function request by a host application that is received and identified by the electronic processor in some embodiments of the invention is a request to open a File 2. Once received and identified by the electronic processor, the computer readable instructions stored on the electronic processor performs the process identified in
Another function request by a host application that is received and identified by the electronic processor in some embodiments of the invention is a request for Image Corners 26. Once received and identified by the electronic processor, the computer readable instructions (in this case ‘image corner computer readable instructions’) stored on the electronic processor cause the electronic processor to access and process a saved “displayable” Image Header including four geographical corner points of the “displayable” image 28, with the corner points being defined (when defined properly) by a location identified by latitude and longitude coordinates. The computer readable instructions cause the electronic processor to read the corners 28 and determine whether the corners are valid. The computer readable instructions cause the electronic processor to return the read corners and indicate an error code when the corners are not defined using a pre-determined system of location identification 30.
Another function request by a host application that is received and identified by the electronic processor in some embodiments of the invention is a request for Image Security 32. Once received and identified by the electronic processor, the computer readable instructions (in this case ‘security status computer readable instructions’) stored on the electronic processor cause the electronic processor to access a saved File Header and displayable Image Header including the security classification of the “displayable” image 34. The computer readable instructions cause the electronic processor to determine whether the read Security classification is valid. The computer readable instructions cause the electronic processor to return Image Security strings and return an error code when the read security classification is not defined using a pre-determined system of security classification 36.
A further function request by a host application that is received and identified by the electronic processor in some embodiments of the invention is a request for Image Data 38. Once received and identified by the electronic processor, the computer readable instructions (in this case ‘displayable image computer readable instructions’) stored on the electronic processor cause the electronic processor to access and process at least one image block within the “displayable” image and, when requested by a user, reconstruct an image including a plurality of image blocks included in the “displayable” image 40. The computer readable instructions cause the electronic processor to return image data, image dimensions, and an error code when the “displayable” image File is corrupted 42.
A further function request by a host application that is received and identified by the electronic processor in some embodiments of the invention is a request for the 3-D Template (“non-displayable” image) data and error code 44. Once received and identified by the electronic processor, the computer readable instructions (in this case ‘3-D Template computer readable instructions’) stored on the electronic processor cause the electronic processor to read the 3-D template image by accessing and processing the non-displayable (3-D template) image Headers and information therein 46. The computer readable instructions cause the electronic processor to return an error code when the 3-D template image File is corrupted 50.
Another function request by a host application that is received and identified by the electronic processor in some embodiments of the invention is a request for coordinate(s) 52. Once received and identified by the electronic processor, the computer readable instructions (in this case ‘coordinate computer readable instructions’) stored on the electronic processor cause the electronic processor to read the 3-D template data and location by accessing and processing the non-displayable image Headers and the information included therein 54, finding the template point(s) closest to the image pixel for which a coordinate is requested 56, and interpolating the template points to compute a “displayable” image coordinate with latitude, longitude, and elevation 58. The computer readable instructions cause the electronic processor to return the “displayable” image coordinates 60. The computer readable instructions cause the electronic processor to calculate and output an estimated accuracy of the outputted coordinate 62. The computer readable instructions also cause the electronic processor to output an error code when a “displayable” image coordinate cannot be generated; the error code will correspond to at least one error code in a set of error codes that describe why the “displayable” image coordinate could not be generated 64.
While the invention has been described, disclosed, illustrated and shown in various terms of certain embodiments or modifications which it has presumed in practice, the scope of the invention is not intended to be, nor should it be deemed to be, limited thereby and such other modifications or embodiments as may be suggested by the teachings herein are particularly reserved especially as they fall within the breadth and scope of the claims here appended.
This application is a continuation in part of U.S. Patent Application having Navy Case number 100218. The invention described herein may be manufactured and used by or for the government of the United States of America for governmental purposes without the payment of any royalties thereon or therefor.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6507660 | Wirtz et al. | Jan 2003 | B1 |
| 6988049 | Wirtz et al. | Jan 2006 | B1 |
| 7440610 | Wirtz et al. | Oct 2008 | B1 |
| 8064640 | Wirtz et al. | Nov 2011 | B2 |
| 20080181454 | Wirtz et al. | Jul 2008 | A1 |
| 20080303832 | Kim et al. | Dec 2008 | A1 |
| 20100180191 | Taber | Jul 2010 | A1 |
| Entry |
|---|
| U.S. Appl. No. 12/587,322, Michael Mathew Wirtz, et al. |
| U.S. Appl. No. 12/924,801, Michael M. Wirtz, et al. |
| U.S. Appl. No. 13/374,622, Michael M. Wirtz, et al. |
| U.S. Appl. No. 13/374,618, Michael M. Wirtz, et al. |
| Blackburn, Christopher, Amendment Under 37 CFR 1.111, U.S. Appl. No. 12/924,801, filed Jan. 27, 2014. |
| Shin, Andrew, Notice of Allowance and Fees Due, U.S. Appl. No. 12/894,801 to Michael Matthew Wirtz, et al, mailed Mar. 3, 2014. |
| Shin, Andrew, Notice of Allowability, U.S. Appl. No. 12/894,801 to Michael Matthew Wirtz, et al, mailed Mar. 3, 2014. |
| Wirtz, Michael M, U.S. Appl. No. 12/894,801, filed Sep. 28, 2010. |
