Patent Application
20180349747
NA
The invention relates generally to the field of image processing. More specifically, the invention relates to a processing unit and methods for simultaneous real time processing of sensor data streams of still imagery, video imagery, thermal imagery and, hyperspectral imagery. Attributes in the scene are identified by running a plurality of processing algorithms on the image data which are in the form of convolutions on spatial, temporal, and color contents of the images which emulate the image processing of the human visual path consisting of eye, retina, and cortex. Additional simultaneous processing of hyperspectral data streams detect the presence of specific materials based on matching their spectral templates against imbedded catalogues of material properties and by detection of pixels whose spectral content are significantly different from the spectral content of the surrounding pixels. The invention produces object detections, object tracks, object classifications, and activity recognition and interpretations by means of cross-modal correlation of the different sensor analysis results. The negligible latency with which these results are produced enable the various sensor operating modes to be optimized during sensing operations.
Current hyperspectral sensor systems generally consist of some form of sensor providing scene imaging which may be in the visible or infrared spectral regions and a hyperspectral imaging sensors which produces images of scenes in very many narrow spectral bands. Very large amounts of data are generated by these sensor systems and its exploitation is very time consuming and human interpreter intensive. Access to important scene content data is often significantly delayed. In addition the operations of the sensor suites are prearranged and do not respond in a short time to the actual observations being taken. Further, the spectral response of the hyperspectral sensors is not modifiable in short periods of time.
What is needed is a signal processing appliance that can analyze the content of the visible or infrared imagers and determine salient scene content. Simultaneous analysis of the hyperspectral scene data can identify materials of salient interest. The correlation of the salient image data with the salient hyperspectral data can improve the quality of information extraction from such sensors. If the processing can be performed with very low latency, in near real-time, then the timely extraction of actionable intelligence is greatly improved. If such data can be extracted in near real-time and on-board the sensors bearing platform's such as Unmanned Aerial Vehicles (UAVs), then a newly available detector technology that allows the detector spectral response to be changed in real-time can be incorporated into the hyperspectral sensors and their operations spectrally optimized based on real-time understanding of scene content and its implications for mission operations.
The Hy-ALERT sensor processor concept combines cognitive-inspired image processing to determine salient object content of visible of thermal images of scenes. Anomaly detection and spectral signature matching processing identifies salient material content of observed scenes. Cross-correlation analysis of the salient contents of the imagers and hyperspectral sensors enables the near real-time interpretation of complex object/material combinations. The data extraction of highly salient information content of the data sets then will enable the optimization of the spectral sensing characteristics and priorities of the hyperspectral sensor. The intensive processing required by the Hy-ALERT processing paradigm is hosted on state of the art FPGA or GPU and CPU processor units who design and architectures are optimized for efficient and timely processing to support real-time exploitation if it adaptive capabilities.
These and various additional aspects, embodiments and advantages of the present invention will become immediately apparent to those of ordinary skill in the art upon review of the Detailed Description and any claims to follow.
While the claimed apparatus and method herein has or will be described for the sake of grammatical fluidity with functional explanations, it is to be understood that the claims, unless expressly formulated under 35 USC 112, are not to be construed as necessarily limited in any way by the construction of “means” or “steps” limitations, but are to be accorded the full scope of the meaning and equivalents of the definition provided by the claims under the judicial doctrine of equivalents, and in the case where the claims are expressly formulated under 35 USC 112, are to be accorded full statutory equivalents under 35 USC 112.
The invention and its various embodiments can now be better understood by turning to the following description of the preferred embodiments which are presented as illustrated examples of the invention in any subsequent claims in any application claiming priority to this application. It is expressly understood that the invention as defined by such claims may be broader than the illustrated embodiments described below.
Military and commercial users have been developing airborne Hyper-Spectral Imaging or “HSI” sensor suites which contain hyperspectral imagers and combinations of visible and thermal imagers for the last twenty years as a means to recognize targets based upon their unique spectral signatures and associated scene content. However, an unanticipated problem resulted from this development, that is, HSI sensor suites are extremely high-data output sensors that are capable of quickly overwhelming the capacity of current air-to-ground communications links. Prior art attempts have partially solved this problem through on-board processing and reporting on a limited set of spectral signatures and recording all data for later post-mission analysis.
The assignee of the instant application, Irvine Sensors Corporation, discloses herein a sensor data processor for using in an HSI sensor suite that significantly increases the timeliness and effectiveness of the processing, exploitation, dissemination (PED) data.
Further, the invention permits the real-time optimization of the HSI sensors while they are in operation by a method illustrated in
Further, the invention permits the optimization and operational deployment of a processor utilizing cognitive image processing principals which analyzes sensor outputs (imagery and hyper-spectral data cubes) and annotates regions of potential threat or having a pre-determined characteristic equally at the same rate as the sensor is producing data. The invention permits the performance of analysts by significantly reducing the time required for assessment and distribution of results and improving the probability of potential threat detection and prioritization.
The invention overcomes the prior art deficiencies by emulating how the human visual path processes large data volumes and identifies regions or targets areas of salient interest. The invention's saliency processing approach relies on characterizing the spatial content (size, shape, orientation) and color content of the imagery from multiple spectra and characterizing the hyper-spectral data by determining locations where the spectral content matches that of known objects of interest and where locations show anomalous spectral signatures when compared to adjacent locations. This processing is accomplished in parallel and at an accelerated rate.
A preferred processing architecture is illustrated in
The exploitation of cognitive processing to rapidly search large data streams and data bases, determine regions of high priority interest, and alert operator/analysts can lead to significant enhancement by providing improved support for intelligence, surveillance and reconnaissance or “ISR”
Many alterations and modifications may be made by those having ordinary skill in the art without departing from the spirit and scope of the invention. Therefore, it must be understood that the illustrated embodiment has been set forth only for the purposes of example and that it should not be taken as limiting the invention as defined by the following claims. For example, notwithstanding the fact that the elements of a claim are set forth below in a certain combination, it must be expressly understood that the invention includes other combinations of fewer, more or different elements, which are disclosed above even when not initially claimed in such combinations.
The words used in this specification to describe the invention and its various embodiments are to be understood not only in the sense of their commonly defined meanings, but to include by special definition in this specification structure, material or acts beyond the scope of the commonly defined meanings. Thus if an element can be understood in the context of this specification as including more than one meaning, then its use in a claim must be understood as being generic to all possible meanings supported by the specification and by the word itself.
The definitions of the words or elements of the following claims are, therefore, defined in this specification to include not only the combination of elements which are literally set forth, but all equivalent structure, material or acts for performing substantially the same function in substantially the same way to obtain substantially the same result. In this sense it is therefore contemplated that an equivalent substitution of two or more elements may be made for any one of the elements in the claims below or that a single element may be substituted for two or more elements in a claim. Although elements may be described above as acting in certain combinations and even initially claimed as such, it is to be expressly understood that one or more elements from a claimed combination can in some cases be excised from the combination and that the claimed combination may be directed to a subcombination or variation of a sub combination.
Insubstantial changes from the claimed subject matter as viewed by a person with ordinary skill in the art, now known or later devised, are expressly contemplated as being equivalently within the scope of the claims. Therefore, obvious substitutions now or later known to one with ordinary skill in the art are defined to be within the scope of the defined elements.
The claims are thus to be understood to include what is specifically illustrated and described above, what is conceptually equivalent, what can be obviously substituted and also what essentially incorporates the essential idea of the invention.
This application claims the benefit of U.S. Provisional Patent Application No. 62/306,711, filed on Mar. 11, 2016 entitled “A Real Time Signal Processor for Analyzing, Labeling, and Exploiting Data in Real time from Hyperspectral Sensor Suites (Hy-ALERT)” pursuant to 35 USC 119, which application is incorporated fully herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62306711 | Mar 2016 | US |
