Patent Grant
8330647
1. Field of the Invention
The method and apparatus described herein are motivated by the need for surveillance and security of the land borders of the United States, including property and facility borders such as military bases and nuclear/petroleum facilities, respectively. The border surveillance system will detect, track, and/or classify people or vehicles automatically and in real-time that are approaching, that may attempt to cross, or have crossed a border. The surveillance method is comprised of (1) a radar for wide-area surveillance to detect, track, and perform 1st stage classification of potential targets that cues (2) one or more optical sensors, which may be comprised of one or more infrared (IR) and electro-optical (EO) sensor systems, and/or coherent radars, for classification of these targets, and (3) a supercomputer such as an IBM® Cell Broadband Engine® supercomputer to process the collected data in real-time. The preferred embodiment is a non-coherent radar cueing a long wavelength IR (LWIR) sensor system, where the non-coherent radar can also be operated as a stand-alone system.
The detection, tracking, and classification will be facilitated by advanced signal processing techniques that allow the use of less expensive and off-the-shelf radars and sensor systems, and at the same time achieve better performance, increased range, and automatic decision making than these off-the-shelf systems. The sensor systems can be implemented on one or more tower-like structures spaced appropriately to cover the border sector of interest, or a mobile vehicle designed to move the surveillance system to a particular location of interest, where it can be used alone or in combination with the tower system. Detection, tracking, and classification are done automatically in real time, and suspected targets are confirmed by visual inspection of the sensor output images by an operator before resources are committed for apprehension. A low-cost tracker UAV can be launched to track the designated target until apprehension. The method includes the sensor suite, the operational strategy, the signal processing approach, the signal processing hardware, and the signal process algorithms. The method has homeland, military, energy, property, building, and facility security applications.
2. Brief Description of Prior Art
The method and apparatus of the present invention is motivated by the need to secure the borders of the United States of America from illegal entry in a cost effective, reliable manner. The invention proposed herein is further motivated by the need to rapidly, accurately, automatically, and cost effectively detect, track, and classify people and vehicles in real time with a fully integrated system. The invention proposed herein is a smart sensor system that can be used to address this surveillance problem using one or more of the system implementation architectures described herein. The same system or subsystems (e.g., the non-coherent radar acting as a stand-alone system) can be used to address a wide range of surveillance and security applications, including military, industrial, and commercial one. For example, a smart sensor system can be used in addition to country borders, for example, to determine all people and vehicles (including aircraft) approach a secure military base, a nuclear power plant, a crucial commercial asset (e.g., water supply), a U.S. Department of Energy (DOE) facility, or a petroleum facility/pipeline.
The problem and the significance of the border surveillance problem for the United States are clearly illustrated in a report issued by the Department of Homeland Security (DHS) in December 2005 [1]. As an example, the Southern Border between Mexico and the United States is 2,000 miles long. The DHS estimates that there may be 3,000 illegal aliens per day crossing the Southern Border, and the report indicates that the border patrol agents are responding to an unwieldy number of possible crossings, one every 44 s [1]. The existing sensor system used at the borders is comprised of seismic and magnetic sensors, and the remote video surveillance (RVS) cameras, which are not cued to the sensors. The report suggests that the probability of false alarm of the system is very high (between 34% and 96% of the sensor alerts) and the probability of detection is very low (between 1% and 57%) [source: Table 1, page 22 of [1]]. Of the seven recommendations made to improve border surveillance, four were technologically driven. They included better integration of assets, standardization of the data collection, processing and reporting, development and application of performance measures, and identify and deploy the use of non-permanent or mobile surveillance platforms. The method and apparatus of the present invention addresses all of these recommendations.
The inadequacy of the current surveillance sensor algorithm and automatic false alarm mitigation technology, the excessively large number of illegal aliens attempting to cross the border, and the lack of sufficient agents to verify possible crossings and to apprehend the illegal aliens makes this a very difficult problem. The invention proposed herein provides the needed surveillance tracking and false alarm mitigation technology that is real-time, integrated, automatic, and has the range to allow the border patrol agents to avoid wasting time on false alarms, enables operators to quickly validate real-targets and determine in advance if they are safety threats, and projects real target locations to future apprehension points. This can be accomplished by the right combination of commercially available off-the-shelf (COTS) sensors and advanced signal processing algorithms.
The report further states that the existing RVS cameras do not automatically steer and cover the area where a seismic or magnetic sensor alarm occurs and do not have the ability to detect movement automatically without operator input. Thus, with existing systems being used at the U.S. borders, it is possible for illegal activity to go unnoticed if an agent is not manning the video terminals or misses some activity on the screen. This can be addressed by used of signal processing algorithms that automatically detect, track, and discriminate human activity without the need for operator involvement on the screen.
The report also states that with existing systems, weather conditions and power outages can have a significant impact on the performance of the systems. While the ice and snow can impair visual acuity along the Northern Borders, the hot and humid conditions can impact the performance of the RVS cameras and the resolution of the IR systems. Other sensors, like an X-band tracking Radar that, unlike higher frequency systems that are impacted by weather, has demonstrated reliable performance in high humidity, rain, and ice/snow conditions. In addition, overlapping Radar coverage is needed such that coverage integrity is maintained with the loss of adjacent sensors, and is enhanced with diverse Radar look directions in difficult terrain conditions.
The method and apparatus of the present invention address some of the most significant deficiencies in the current border operations. The poor performance of the present technology-based systems is due partially to the lack of integration between the sensors and the RVS cameras, the high number of false alarms of the sensors, and the lack of automation for the present camera systems. The 24/7 requirement for visual inspection of the camera images by an operator, and the requirement to deal with a sensor alert every 44 s is too people-intensive to work over an extended period of time. Automation can reduce the workload and take better advantage of an operator's inherent ability to discriminate threats from false alarms. It is clear that better coverage, better integration (cueing of sensor and RVS camera resources), the use of advanced signal processing algorithms for automatic detection, and improved operator discrimination can improve the performance of the system.
The method and apparatus of the present invention will increase the PD, reduce the PFA, reduce the PFT (i.e., number of false agent prosecutions after final operator screening), reduce the total number of alerts to deal with, and allow operators to be more effective. The proposed method enables more effective sensor assets because of better integration, automation, and better coverage. The method and apparatus of the proposed system architecture can be integrated with the existing underground seismic and magnetic sensors, the RVS cameras, and other assets presently being used by DHS.
Detection of small cross-section targets such as people or groups of people that can potentially cross a border, especially one that is large, is a challenging technical problem if it is to be accomplished in a cost effective method. The problem is made even more difficult when the measurements need to be made at ranges up 10 km or more for detection, tracking, and classification. People, with very cross sections (less than 1 m2) are very difficult to detect. The Department of Homeland Security has been using unattended ground sensors, EO/IR cameras mounted on towers and in UAVs for visual identification and classification of people and vehicles (at short ranges) by an operator. However, as described below, this approach has experienced problems with an unacceptably high false alarm rate and a poorer than desired probability of detection. The former results in an expensive misuse of limited resources, and the later does not provide the desired border security.
The invention proposed herein whereby a radar is used for wide area surveillance to detect and track potential targets in an area of interest and then cues an optical system such as a long wavelength IR sensor for classification has never been used for border surveillance. The use of the radar allows targets to be detected at sufficient ranges so that potential border crossings can be anticipated in sufficient time for an optical system to identify the target before an illegal crossing is attempted. The invention proposed herein also describes a two-zone implementation, one for wide area surveillance and one for classification and continued tracking until the target is dismissed as a potential problem or through apprehension. The towers or vehicles presently used by DHS do not include radars and consequently have not used such radars to cue the EO/IR sensors. For this border surveillance architecture to be effect requires that the radar have a wide area surveillance capability with frequent updates of the potential targets moving towards the border; this requires many millions to tens of millions of radar samples be collected over the target area. For complete surveillance, the radar should be preferably implemented with 360-degree, 24-h-per-day coverage.
Most modern-day radars developed by the military are coherent radars, which detect moving objects such as people, vehicles, aircraft, ships, and missiles using range-Doppler processing. However, these radars are very expensive and to realize acceptable coverage for border surveillance applications would generally not be cost effective. The less expensive non-coherent radars have not been used for border surveillance applications because of the lack of capability to detect very small cross-section targets like people. Incoherent radars have mainly been used to detect threats much larger than people like aircraft and ships.
Besides low cost, these radars have sufficient range and an effective wide-area surveillance capability to cover and provide a large number of samples on the target area of interest. The advantages of such a non-coherent radar is that it is very inexpensive, can put a lot of energy on the target, and can collect a lot of samples on the target. This type of non-coherent radar is the key element of the preferred embodiment of the present invention proposed herein because the advantages of this type of radar, when applying advanced signal processing, can be converted into a radar to detect targets with very small cross sections (i.e., people). The invention proposed herein also describes the use of a radar propagation model, which includes the effects of multi-path, to determine signal-to-noise ratio (SNR) of for each resolution cell in the radar target area of interest, where the terrain (in three dimensions), the radar location and elevation, and the target size and height is input to the model. This model can be used (1) in optimally locating the position and elevation of the sensor measurement systems (towers and mobile assets) for maximum coverage and performance, and (2) in developing and maintain a target track even when the target passes through resolutions cells without sufficient SNR for detection.
Our search of the patent literature does not indicate the use of non-coherent for wide area surveillance of people and other targets with small cross sections. Nor does our search indicate the disclosure of the system architecture whereby a radar cues an optical system, the use two-zone method for wide area surveillance in the outer zone and classification in an inner zone. Finally, our search did not indicate the use of a radar propagation model for locating towers or mobile vehicle for optimal coverage, or for use in developing more robust tracks of the targets of interest.
While the use of advanced signal process algorithms and supercomputers automated, real-time measurements with remote sensor systems is not in and of itself novel, it is novel for application to inexpensive non-coherent radar system for detection and tracking people or groups of people.
It is the object of this invention to provide a method and an apparatus for surveillance of a border, property, or facility for people or vehicles approaching, crossing, or leaving the border.
It is the object of this invention to provide a method and an apparatus for surveillance of a border, property, or facility for people or vehicles where entry or access is limited or prohibited without permission.
It is the object of this invention to provide a method and an apparatus for automatic, real-time detection, tracking, or classification of small targets such as people or vehicles with a smart sensor system.
It is the object of this invention to provide a method and apparatus for detection, tracking, and 1st stage classification of people, vehicles, or targets with small cross sections with a low-cost, non-coherent radar.
It is the object of this invention to provide a method and an apparatus for detection, tracking, and classification of people, vehicles, or targets with small cross sections with a smart sensor system comprised of a radar, which will detect, track, and perform 1st stage classification of all targets in a target area of interest, to cue an optical system for classification of the targets.
It is also the object of this invention to provide a method and an apparatus for a surveillance system comprised of a non-coherent radar cueing a long wavelength IR camera system for detection, tracking and classification of people and vehicles.
It is also the object of this invention to provide a method and an apparatus for a tower-based surveillance system.
It is also the object of this invention to provide a method and an apparatus for a mobile-based surveillance system.
The method and apparatus of the present invention is a two-zone system architecture of a real-time, integrated, automated and reliable border surveillance system using off-the-shelf sensor systems and a suite of novel signal processing algorithms to detect, track, and/or discriminate illegal border crossing of people and/or vehicles (i.e., small, slow-moving targets in a high-clutter environment). The final decision about whether or not an illegal crossing will, is, or has occurred can be verified by visual inspection by a border patrol agent using an image or movie clip output from the automated system.
The preferred method of the present invention is a two-zone, system architecture comprising a wide-area, all-weather, non-coherent, X-band radar surveillance system for detection and tracking of people and vehicles in the outer zone, which cues, one or more high dynamic range, long-wave infrared (IR) and electro-optical (EO) surveillance system for classification (discrimination of actual targets from false targets) and continue tracking in the inner zone. The non-coherent radar can be used for 1st stage classification in the outer zone. In addition, a coherent radar can also be used for classification in either zone. The signal processing algorithms enable the use of COTS radars and optical sensors to automatically detect, track, and classify illegal aliens and vehicles in real-time, including passing continuous real-time visual “movies” of the suspected target to an operator for visual confirmation that the target is real before committing resources for apprehension. The use of sophisticated signal processing algorithms enables the use of low-cost, COTS sensors by shifting the surveillance burden from sensors to processing algorithms and implementing the real-time software in one of the latest supercomputer technology (e.g., an IBM® commercially available Cell Broadband Engine® processor). The system architecture concept described herein uses towers and/or mobile-vehicles for implementation of the sensor systems.
a. Analytical estimate of target distance as a function of SNR and tower height for a 25-kW radar and a person (0 dBsm target at 5 ft height).
b. Analytical estimate of target visibility out to 6 km using a terrain model assuming a target height of 5 ft.
c. Analytical estimate of target SNR for a 25-kW radar and a person (0 dBsm target at 5 ft) with the effects of terrain and multipath included.
It is important to point out that the essential element of the radar is that a non-coherent radar is used. The preferred embodiment uses a K-band radar, but other frequencies of operation (C-band, Ku-band) could also be used. At the present time, an X-band, Furuno® radar is being used; this radar is a marine navigation radar, and this radar/type of radar has been in use for many tens of years. Also, a coherent radar could be used, but the cost of using one with a wide area surveillance capability is very expensive, and especially expensive when there is a need to instrument a large number of towers and mobile vehicles (e.g., a 1,000 towers are planned along the Southern Border of the U.S.). The distances will vary according to range, elevation of the radar/sensor, and the capabilities of these and are used to represent the type of distance for which the system may be operated, and the exact distances are not essential to the embodiment. A LWIR sensor is used in the preferred embodiment. While desired, another IR system (e.g., a mid-wave IR system) or an optical system that operates in the visible band (i.e., a camera) could also be used. In general, multiple EO/IR sensors, especially an EO sensor system (i.e., in the visible frequency band) might be included in addition to the LWIR sensor system for classification. In addition, a coherent radar might also be used for classification when the line of sight of an EO/IR sensor is obstructed by terrain or environmental conditions. This smart sensor system approach is possible because of automated, real-time, advanced signal processing algorithms and the advent of supercomputers like the IBM® Cell Broadband Engine® supercomputer (
The targets currently of interest to DHS are walking people, groups of walking people, horses moving with intent, and moving vehicles. A high level of performance, in terms of probability of detection (PD) and probability of false alarm (PFA), is achieved with this sensor suite because of the use of advanced signal processing algorithms such as the data-intense, Bayesian-based, signal processing and fusion algorithms. Performance goals for the smart sensor system were developed to provide a PD>99% for detecting and tracking all targets of interest with less than 1 false target every two days for a ten-tower coverage sector (i.e., a PFT<0.001% for targets incorrectly classified by an operator). Because of the high performance of the radar and the fact that the LWIR sensor is cued to one or more potential targets of interest, the LWIR sensor can be operated with a automated, real-time processor with achievable performance parameters (PCLASS>99% and a PFA<0.25%).
The collage of photographs in
A description of the new generation of small, cooled, higher performance, and less expensive long wave infrared (LWIR) sensors is illustrated in
Previous tests indicate a strong preference for the use of a Long Wave IR (LWIR) sensor system (i.e., camera system), because it has greater than a 10 dB performance advantage over the mid-wave IR band under normal conditions and has considerably greater advantages in obscure conditions such as the dusty environment (see
In addition to greater contrast in the long wave band, automated smart sensor system algorithms require a consistent phenomenology response across diurnal cycles for effective 24-7 classification performance. Previous tests indicate that long wave IR can provide a consistent emissive response, whereas mid-wave IR performance is highly bi-modal, i.e., predominantly reflective in daytime conditions and predominantly emissive in night-time conditions. This bi-modal mid-wave IR response requires serious consideration of the need to implement different smart sensor system algorithms as a function of time-of-day, which drives system cost and complexity. Of greater concern is the response cross-over points where periods of several hours of poor to unusable mid-wave band performance during dawn and dusk conditions. During these two daily phenomenology transition points the received signal energy is changing to increasing reflective energy content from the decreasing emissive signal energy content, and vice versa.
In addition, tests on previous programs indicate that these conflicting daytime and night-time mid-wave IR signals are not necessarily consistent, thereby causing dead periods for effective border target classification. All of these long wave IR and mid-wave IR factors strongly suggest that long wave IR is the strongly preferred IR band for effective 24-7 classification.
The advantages of a low-cost, non-coherent, X-band radar for surveillance over the other types of radars are discussed below. An analysis of the requirements for a 360° tower search out to 12 km ranges with a surveillance radar indicated that over 10 million full bandwidth range-azimuth samples/s are required. The use of a non-coherent Furuno® radar generates over 12 million full bandwidth samples/s. Other low-power, solid state radars that generate approximately a 1,000 samples/s would generally be inadequate for high performance surveillance in terms of probability of detection and probability of false alarm for these large search areas. The signal processing approach being used insures a high probability of detection, because over 500 radar contacts per scan can be obtained with the radar, which allow a track-before-detection signal processing approach to be used. Through the application of its smart sensor algorithms, the radar reduces raw sensor imagery, clutter, and false alarms by over 1,000,000 to 1.
The performance for the sensor suite and the signal processing algorithms for detection, tracking, and classification of people and vehicles are expressed in terms of PD, PFA or false alarm rate (FAR), and probability of false target (PFT) or false target rate (FTR). An important distinction is made between false alarms and false targets. A false target is a target falsely identified by the operator after inspection of the visual images and passed to a border patrol agent or some other asset as a potential target. A false alarm, on the other hand, is a target falsely identified by the sensor suite and automatic signal processing that is passed to the operator as a target. A false alarm can be mitigated or confirmed as a target by the operator. Thus, the PFA is based upon the automated output of the sensor suite, while the PFT is based upon both the sensor suite and the operator's visual assessment. To be effective, the number of false alarms produced by the automatic sensor suite needs to be acceptably small.
Both false alarms and false targets need to be infrequent so that operator and border agent resources are not unduly taxed. The requirements for the PD and PFA of the sensor suite to achieve an overall performance are summarized below.
Estimating performance of the proposed smart sensor system is a very challenging task, especially because of the requirements for a very high PD and a very low PFA and PFT. Normally, performance is estimated using the complete sensor suite and signal processing algorithms deployed in the operational environment of interest for a sufficient time to assess the output of the sensor system over the environmental and operational conditions of interest. This is typically accomplished using pre-production and full-production systems that are or soon will become commercially available. Our initial assessment of performance involves three components:
The proposed sensor suite and signal processing algorithms have the following advantages:
Two-Zone Implementation of the Smart Sensor System. One or more of the system architectures for the smart sensor system are described next. The following set of guiding principles was used to develop the architectures.
The radar will be used to detect and track any moving object in the Outer Zone at a range that is sufficiently far away from a border that no border crossings are immediately possible. In addition, tracking will continue, or if the need arises, to initiate a detection and tracking, in the Inner Zone. The objective of acquiring the Outer Zone detections and tracks is to cue a narrow field-of-view LWIR sensor to pick up the targets detected and being tracked by the radar at the boundary of the Inner and Outer Zones, and to classify and then to continue to track the targets in the Inner Zone by the radar and the LWIR sensor until the LWIR sensor is steered to another possible target for classification. Video clips from LWIR (and EO) cameras will be sent or will be made be accessible to an operator for visual interpretation and be used to verify potential targets that might potentially cross the border. The operator can also be cued by the smart sensor system to review potential targets that might cross the border. With some modifications, the same LWIR algorithms used to detect, track, and classify targets with the narrow field-of-view LWIR sensors can be implemented for the wide field-of-view IR cameras directly and with some further adaptation, the EO cameras as well.
The detection, tracking, and classification problem will be specified in terms of the targets of Interest (TOI) and the sources of interference (i.e., both false targets, such as animals, and system noise or ambient clutter), both of which can result in false alarms and/or false targets. While spurious system noise and ambient clutter can produce false alarms, they should be small in number and fairly easily to mitigate over time because of Bayesian, track-before-detect signal processing algorithms. The targets of interest are mainly people or groups of people, vehicles or groups of vehicles moving very slowly to very fast. In some instances, stationary people and vehicles are of interest. It is anticipated that the tracks of these targets may not be continuous as the targets go in and out of regions of high clutter, move behind main-made and natural obstructions such as buildings and natural terrain features, or change speeds, stop to rest or to take evasive actions. The targets may also move by vehicle, horseback, or bicycle/motorcycle. False targets will mainly be animals or groups of animals such as dogs, coyotes, cows, and horses. While initial detections and tracks may not be able to differentiate a horse from a person on horseback, the speed and direction of the track will ultimately lead to a correct classification that differentiates horses or cows grazing in a field and a person on a horse attempting to illegally cross the border.
The sources of interference will depend upon the type of sensor system being used. The system noise can be defined for each type of sensor in a relatively straightforward manner and is generally smaller than the ambient noise or clutter (and the TOIs). Given that the system noise characteristics are small and understood, the main limitation of each sensor will be a function of the environment, as well as the range and resolution of the sensor. The ambient noise or clutter will mainly depend on the atmospheric, topographic, and terrain (both natural and man-made) features. There will be some sources of ambient noise/clutter for which each type of system may not operate effectively or operate with degraded performance. However, by combining two or more sensor systems, which are affected differently by the ambient noise/clutter, will help mitigate such problems. These problems might also be mitigated by different looks of the same system. As an example, the performance of the non-coherent X-band radar being proposed will not be affected by rain or moisture whereas an IR system might be. Therefore, the radar will be able to detect and track the target even though the IR system may not. As another example, a terrain feature that masks detection of a target from one look angle can be mitigated by using a second sensor system mounted on an adjacent tower with another look angle. The use of the preferred embodiment from multiple towers is to maximize coverage and enhance performance. As another example, a target track might be temporarily lost due to high ambient noise and regained again once the target moves through that region of high noise. The radar propagation model, which can be used to predict such location, can also be used in the tracking algorithm to regain any temporary losses of the track. For some conditions, however, the combination of sensors might not work and the Command, Control, Communications, and Computers (C4I) display software will identify a potential lost target and wait for it to re-appear. In this later instance, other sensors and sensor platforms might be required such as an EO camera or a UAV or a mobile radar system.
In the Outer Zone, the objective is to detect and track any target present with the radar. Since the resolution cell within a radar dwell is generally larger than people and about the same size as a vehicle, generally, it will not be possible to determine the number or type of targets and whether they are real (e.g., a person or vehicle) or false (e.g., an animal). Some classification with the radar will be possible, however, and will be useful in supporting classification by the LWIR sensor system. Such parameters such as the speed or size of the target will be useful in a 1st stage classification. Clearly, a target moving at 40 mph is not likely to be a person, but rather a vehicle. A target path at 40 mph for an extended period of time that is aligned with a road is more likely to be a vehicle than an animal.
The main source of interference for the radar is mainly ground clutter, which is affected by vegetation, manmade structures, topography, etc. In addition to the ground clutter, terrain and manmade features may hide or partially mask the targets of interest. Weather, however, is generally not a problem for the radar being used in the preferred embodiment, but may impact the performance of the LWIR sensor.
Classification occurs when the LWIR camera with coverage in the Inner Zone is cued to the detection and track of a target provided by the radar. The LWIR system will pick up the detection, classify it to verify that it is a real target vice a false target, and then track it until directed to another target of interest. The radar will continue to track the target of interest, and in addition, might also acquire new or re-acquire old targets.
The people and the vehicles that intend to cross the border may be attempting to confuse the sensors by crawling, moving in random patterns, stopping and starting, or hiding behind man-made or topographical features. Many of these activities are also possible for people and vehicles in their normal activities. In most instances, an estimate of the number of people in a group can be made as demonstrated in recent field tests. The processing method will be designed initially to distinguish small groups from large groups. A more detailed count can be made as the targets get close to the border and are sensed with more resolution.
At any time after a target has been classified, a movie clip or single images can be turned over to an operator for review and evaluation. If the operator believes that the target is real, the operator will have enough track and discrimination information (1) to immediately dispatch border patrol agents for apprehension at a pre-computed intercept point for the radar track. The operator may also decide (2) to continue tracking the target until an agent becomes available to intercept and apprehend, or (3) to dispatch an unmanned asset (a) to provide another sensor input or (b) to provide continued outer region tracking and discrimination until intercept, or (4) to dispatch an agent and an unmanned asset. Any mistakes made by the operator, which are ultimately determined by border patrol agents as not a person or vehicle will be classified as false targets. The decision by the operator to launch an unmanned tracker asset and/or to dispatch agents to apprehend the people or vehicle(s) will be made before the person(s) or vehicle(s) leaves the Inner Zone (Red Zone) and enters the Outer Zone (Green Zone). This approach allows a human operator to make the final decision about a target, but does not require and tire the operator to review around-the-clock sensor data. The approach presents information, not data, for decision making. In summary, the radar will detect and track people and vehicles in both the Outer and Inner Zones (Blue, Green and Red Zones) and the LWIR/EO sensors will detect, classify and track people in the Inner Zone (Red Zone). This system can be configured to also inform operators of staging individuals or groups of people before they get to the border allowing border agents to pre-position themselves for deterrence and/or apprehension.
During tower-sensor placement the topography will be mapped to ensure optimal sensor placement and line-of-sight coverage. A tracker display screen will map the position of moving targets onto the standard Commercial/Joint Mapping Tool Kit (C/JMTK) for operator situational awareness using a commercial internet-like user environment modified to include a radar propagation model, which includes the effects of multipath. While the primary intent of the border system is to automatically detect, track and classify moving targets in real-time, additional analyses can be conducted in real-time to improve the detection, tracking or classificiation capabilities. For example, a source-sink analysis of radar tracks will determine the probable location of tunnels or hidden pathways across the border and automatically recommend locations for additional unattended ground sensors (UGS). The final surveillance system will also provide the fusion of radar data from up to five adjacent towers to more accurately track targets and better locate them in regions shadowed by terrain. As part of the final surveillance system concept, a supercomputer technology such as the IBM® Cell Broadband Engine® supercomputing technology, will be used. This technology will be a key element in performing local processing of high performance algorithms at each tower.
A flat earth analysis is being assumed initially in all of the signal processing performed on the recent field tests, but actual topography will be included as required to enhance performance. If it were not, elevation would adversely impact the detection performance (aside from hiding the target) if there are significant elevation differences in the field of view, because the range to the target is not well known.
Some practical scenarios for detection, tracking and classification of people and vehicles in the Outer Zone and the Inner Zone, which includes the border, were considered in the system development and are described below. While the list is not exhaustive, it is representative of many of the detection, tracking, and classification challenges.
A number of scenarios for the Outer Zone are presented below.
Performance Requirements. In the course of developing the smart sensor system and system architecture, a set of performance goals for detection, tracking, and classification were developed in terms of PD, PFA, and PFT for specific targets and target environments.
As a goal, we set the PD to be PD≧99% and the PFT to be PFT≦1 false target every two days (for a 10-tower system), where a false target is a target incorrectly identified as a real target by an operator after assessing the targets visually in the LWIR video image clip provided by the automatic processor. The requirements for the “cued” LWIR sensor were developed assuming that the operator would be presented with no more than 5 false alarms per hour over a 10-tower sector (i.e., no more than 1 false alarm every 12 min). The analysis further assumed 200 targets per hour being classified by the automated algorithms. This results in a PFA≦0.25% and a PD≧99%. Thus, the PET≦0.50% for the operator based on the visual assessments presented. Overall, the smart sensors and the operator would achieve a probability of rejecting false targets of (PFT*PFT)≦0.00125%. The analysis is summarized in Table 1.
The performance for the radar, the cued LWIR sensor, and the operator are summarized below and in Table 2. TBR means that the capabilities are within scope but needs “To Be Resolved” for each specific area through additional work.
The Southern Border. The Southern Border of the United States is comprised of a wide range of terrain, topography, and environmental conditions in which the smart sensor system will need to operate. (For example, the general conditions are not very different than might be obtained when using the system for military and DOE facilities.)
Concept of Operations. As presented in
Components comprising the smart sensor system architecture are described briefly below.
Smart Sensor Field Test Results. A set of field tests were conducted in Salinas, Calif., to demonstrate various aspects of the preferred embodiment of the smart sensor system. The analysis of the radar and LWIR data collected, when combined with some radar and IR performance modeling, support the high level of performance required for effective management of the border. For both the LWIR and non-coherent radar tests, people and vehicle targets were detected and tracked at ranges of up to 2.1 km; in addition, people and vehicle targets were properly classified with the LWIR sensor at these ranges. These short ranges are mainly due to the low elevation of the radar and LWIR sensors used during these tests (i.e., 2 m). The analysis of the LWIR demonstrates that the targets of interest (people and vehicles) can be detected tracked, and classified automatically. The LWIR field test results indicated that all 44 of the people walking alone or in groups at distances up to 1.8 km were detected, tracked, and properly classified when the number of pixels on the target were 15 or greater. These tests were designed to be a demonstration of method effectiveness. Greater radar and LWIR ranges will be achieved with (1) the deployment of the radar and LWIR sensor at greater heights and (2) the deployment of the latest version of LWIR sensors.
The preliminary performance estimates made from field test are based upon a non-coherent, X-band radar (Furuno USA, Inc.), a narrow field-of-view LWIR sensor (DRS Technologies, Inc.), and several signal processing methods. For the field tests conducted, the LWIR sensor system performance is independent of and does not take advantage of the performance gains of being cued by the radar.
Tables 3 and 4 summarize the characteristics of the Furuno® radar and the LWIR sensor system used in the Salinas, Calif. field tests. The LWIR sensor used for data collection is an AN/TAS 4(x) FLIR. As indicated above in
The most accurate and reliable method of determining the performance of a sensor detection system in terms of PD and PFA is to collect enough actual data with the sensor detection system (including data fusion and all operator inputs) with all known sources of interference (system and ambient noise/clutter) and with and without the targets of interest in the coverage of a detection system to estimate histograms of the noise and the signal-plus-noise. Different estimates of performance may be made to properly reflect the ambient noise/clutter conditions of interest for a particular operating area as a function of range, movement, weather, terrain/topography, and man-made features. Obviously, a comprehensive performance estimate can be a very large undertaking and is usually best made once the actual sensor system is deployed and operated. However, before deployment of a sensor system, a reasonable estimate of the performance of the sensor system needs to be made. The PD requires than many measurements be made on each type of target in all of the same noise conditions used to define the PFA. Simplistically, if a PD of 99% of detecting a single person in a certain amount of time is desired, then data on a single person over the full range of conditions used to define the background noise/clutter conditions and movement conditions needs to be acquired.
This approach evaluates the sensor system, including the signal processing, and its operational implementation. Any changes to these parts of the system, especially the signal processing, will change the performance of the system. The field tests were implemented using a non-real-time system. The performance of the system will improve as more data and operating environments are acquired. The results of the field tests demonstrate that an automated algorithm can perform the visual pattern recognition previously accomplished by visual inspection by a trained person. The algorithm automatically processes video clips previously acquired and outputs continuous displays of detections and tracks of people and vehicles that are properly classified. This accomplishment is significant, because when combined with the operator's visual inspection on targets suspected of illegally crossing a border (ultimately the final step before a decision to apprehend a border crossing), a very robust system performance can be achieved.
Our approach for estimating performance is comprised of two steps. First, a combination of modeling and some limited field data will be combined to make one or more preliminary estimates of the performance for a specified set of conditions. Second, one or more border regions that typify the overall surveillance conditions will be selected for a field estimate of performance using the sensor system suite. The performance estimate presented herein is based on the former; more field testing will be performed to develop performance estimates based on the latter.
A discussion of the radar, the basic processing approach, a discussion of the results obtained from processing some of the images that were obtained from detection and tracking movies, and an estimate of performance based on model estimates of ROC performance curves are presented below. A similar discussion is presented for the LWIR sensor. In addition, sufficient target ground truth was available in the LWIR movies to make a very preliminary estimate of the probability of detection and the probability of false alarm.
Radar Modeling and Field Test Results. An illustration of the type of performance expected for the non-coherent radar smart sensor is presented below. First, model estimates of radar range versus tower height as a function of signal-to-noise ratio (SNR) are made using a radar propagation model. Second, a high level view of the signal processing is presented. Next, a few illustrations of the output of the signal processing for detection and tracking of people and vehicles are presented. Individual images obtained from several movies are used to illustrate the capability. Fourth, preliminary results of the receiver operating curves (ROCs) are presented for the detect-before-track Bayesian algorithms.
Radar Detection Range on a Walking Human. The performance of a given radar is ultimately dependent on the effects of multipath, the clutter environment, and the signal processing strategy implemented to address these effects. The effects of multipath can be modeled with assumed or measured terrain and atmospheric conditions. The clutter is highly inhomogeneous and generally well above the receiver noise level. In most instances, the clutter level, not the noise level of the radar, limits the performance of the radar. The noise floor, however, is the performance limiting factor at long ranges.
The maximum detection range that can be achieved for a single walking human or a moving vehicle is dependent upon the site-specific terrain and the sensor height above the ground. To develop design requirements for the tower height and spacing, an analytical estimate of the signal-to-noise (SNR) of a radar was made as a function of range and tower height for a 25 kW radar and target cross sections of 0 dBsm at a height of 5 ft (where 0 dBsm corresponds to the return of a target with radar cross section (RCS) of 1 m2), which is readily scaled to the actual RCS range of human and vehicle targets. The analytical estimate included the effects of the constructive and destructive multi-path interference with the ground and surface roughness. The reference calculation is shown in
To conservatively estimate the detection range for a person, and therefore, tower height and tower spacing, it was assumed that the radar would yield high performance if SNRs of 10 to 20 dB were maintained. For tower heights between 40 and 100 ft, the radar range for detecting walking humans can be estimated by following the appropriate signal-to-clutter contour in
This radar model was extended to include the effects of terrain and was used to make an analytical estimate for the line-of-sight visibility and the radar propagation for the Salinas, Calif., site used in the radar and LWIR field tests described below.
Radar Detection Range on a Moving Vehicle. The radar range that is achieved for a moving vehicle is also dependent upon the site-specific terrain and sensor height above the ground. A similar set of curves can be generated for a vehicle with an assumed cross section of 10 dBsm. Ranges of over 10 km are achieved at a tower height of 40 ft assuming a 10 dB SNR (when the vehicle is not blocked by terrain or foliage).
Radar Performance. The algorithms implemented as part of the present invention include Space-Time Adaptive Processing (STAP), data mapping to ground-fixed coordinates, data linearization and filtering, peak detection, association and tracking, track management, likelihood accumulation, track filtering, declaration, and display. Distinguishing characteristics of the clutter reduction algorithms include replacement of simple CFAR with likelihood-based scaling. The algorithms use advanced 3- and 4-dimensional (2-D and 3-D spatial plus time) Wiener filters to remove clutter based on complex spatial and temporal structure inherent in the clutter. Filters are adaptive to the actual data environment being observed and system performance gains are dependent on sensor characteristics and clutter statistics.
Both theoretically-based and empirically-driven signal processing methods have been used to detect the targets of interest (e.g., people). Two theoretically-based detection processing methods have been implemented and tested. The first method is to develop probability density functions (based on theoretical statistical models or empirical histograms) for the noise/clutter and the signal-plus-noise/clutter and to set a threshold to achieve an acceptable probability of false alarm (or false alarm rate) and a certain probability of detection against each type of target of interest. The analysis of the radar data used to evaluate this method suggests that better performance can be achieved because the noise and clutter characteristics are different for each resolution cell or groups of cells, sometimes dominated by noise and sometimes dominated by clutter. Improved performance can be achieved by determining the noise/clutter characteristics for each resolution cell, and once this is accomplished, set different thresholds based on the probability density distributions for each type of noise and clutter characteristics. Thus, the resolution cells can separated into clutter-dominated cell and clutter-absent cell (i.e., system noise). The class clutter-dominated cells can then be further partitioned into low-medium- and high-variance cells. The data collected to date suggests that a majority of the resolution cells are dominated by system noise, which is lower and easier to characterize than clutter, and results in very high performance. This is manifest, because the radar is operated at low low-grazing angles. This method can produce very high probability detections with very low probability of false alarms.
If the data behavior cannot be defined, another approach is use a totally empirically-based approach. For example, a “CFAR” approach can be used, wherein a threshold is applied for each resolution cell for a given false-alarm level and some ad hoc method (m-of-n, for example) is used to declare valid tracks and reject false ones. Such a data-driven detector is robust, can adjust to differences in site topographies and other characteristics, and automatically adjusts to unmodeled effects. This approach does not require a detailed understanding of every source of interference. The main significant disadvantage is that the training process must estimate an entire distribution function rather than merely some parameters conditioning it, and furthermore, what substitutes for an analytical log-likelihood ratio (LLR) is a table-lookup function. The actual approximation, however, as compared to the optimal one is good.
The second method is a Track-before-Detect method. A Bayesian, Log-Likelihood-Based, Track-Before-Detect signal processing method for processing the radar data was implemented and tested. The field test results illustrated below using this processing method. The track-before-detect processing method is used to suppress clutter and find targets in radar data without any detailed knowledge of the noise or the clutter. The operation of the track-before-detect system calculates a likelihood ratio and bases track confirmation on the growth of this likelihood ratio to a declarable threshold. The likelihood ratio is computed by using Bayes' Rule, which allows sequential updates of likelihood as each new observation is acquired.
Two thresholds are used. The first threshold is set very low to allow a large number of contacts, mostly non-targets, to be detected. The statistics from some pre-determined fraction of all radar cells are processed as possible contacts on every 360° sweep, whether those cells truly contain targets or not. Contacts (Ncontacts) above the first threshold, which are most likely to be associated with current tracks, are retained from sweep-to-sweep; all other contacts initiate new tracks. This first threshold insures that “almost” all of the targets of interest present in the radar field of view will be detected. This very high false contact rate is mitigated by the track-before-detect algorithm, because the return from non-targets will be random and will be eliminated because such events will not generate sufficient likelihood to be declared as true targets. Only the tracks that build up (likelihood) in excess of a second threshold become “detections” (hence the term “track-before-detect”). Many thousands of false contacts may be allowed. It is important to note that these initial detections or contacts will be further processed for classification and ultimate action. The track-before-detect signal processing algorithm, if properly implemented, can achieve the best possible performance given the available data.
As stated above, the probability of detection of the radar (PDradar) is controlled by setting the threshold of the detection statistics (i.e., the first threshold threshold) low enough to capture real targets, but without encountering an overwhelming number of candidate contacts. Achieving a satisfactory value for PDradar requires that target SNR being sufficiently large (or at least comparable) to fluctuations in the clutter return. The second threshold is used to actually declare the presence of a target; the probability of detection and the probability of false alarm of the radar are functions of both thresholds.
Each data frame has an associated intensity field that has potential target contributions and background (non-target) contributions. System noise sets a fundamental uniform limitation on target detection. The non-uniform and environmentally sensitive structure of non-target energy in the data field is more critical. System performance depends on instrument sensitivity, noise, resolution, and overall instrument quality. One figure of merit is the minimum detectable signal in a resolved cell. A detectable target at a specified distance from the source must deliver at least that much energy. It is assumed that any pre-processing integration necessary to achieve the minimum detectable signal requirement has been performed prior to data capture.
Registration, Data Collection and Segmentation. All data processing operations depend on accurate frame-to-frame identification of persistent features. Both instrumental and environmental factors can cause artificial displacements of the features that appear as image motion. The detection algorithm must reduce these errors to a prescribed level to meet performance objectives. Surveyed landmarks in the scene, either natural or manmade, will be used as required.
Once data frames are registered, system specific operations that facilitate subsequent algorithmic operations are performed. These include constant background removal, pre-filtering, and noise background estimation. These operations provide small but important performance enhancements. Data conditioning might also include system performance monitoring and calibration operations.
Segmentation is the formal process by which registered and conditioned data frames are translated into detection fields. The detection field intensity, S, consists of uniform noise, N, a non-uniform background I, and target energy T. Often I is the dominant component. The signal to interference ratio SINR0=S/(S+I) can be very small where non-target energy dominates. The purpose of segmentation is to identify the background, which is ideally characterized by the ratio ESINR0=S′/N. To the extent that S′=T+N, one has a uniform detection field in the absence of targets. The problem is that segmentation invariably involves some target energy loss and some background leakage. Performance evaluation requires estimation of segmentation signal losses and threshold inflation required to accommodate leakage in all the environments that the system must operate.
Contact Selection, Peak Formation, and Tracking. Once a detection field has been generated and converted to target likelihood units, a threshold can be set to capture all peaks that have sufficiently large single-frame likelihoods to be considered as target candidates. In most situations, the peaks that exceed a threshold have contiguous neighbors that form clusters. Clusters are usually assigned a single location that is passed to the tracker as a “contact.” As consecutive frames are processed, the tracker assigns new contacts to existing tracks or initiates new tracks. The formal procedure uses a Kalman filter and a motion model to predict where the next contact in an active track should occur. The metric distance between the nearest contact and its expected location provides the basis for cumulative likelihood computation. A matched filter can also be applied at this stage to maximize the signal to background noise ratio.
Target Detection and 1st Stage Classification. A target is detected once a high-confidence track has been developed. The first stage classifier for the radar distinguishes humans from say animals or coincidentally moving small objects and vehicles of various types. This information is potentially contained in the shape of any resolved targets, their size, and how the targets move relative to the surface. Such features can be converted into metrics that have some discrimination power. More subtle features include shadows and radar sidelobes. These first-stage classification decisions are useful in supporting the prioritization of LWIR sensor cueing and LWIR sensor resource planning by greatly reducing the burden of looking at false targets that can be readily identified from radar-only returns (e.g. vehicles). Pass-through of data (from radar processing to the cued LWIR processing) such as the precise range to targets, accurate estimates of the in-range velocity components, and general indications of radar cross-section substantially improves the performance of the second stage EO/IR classifier.
Radar Field Test Results from Salinas, Calif. The performance of the smart sensor system (non-coherent, X-band surveillance radar cuing a narrow field-of-view LWIR sensor) was demonstrated with data obtained in a field test in Salinas, Calif., with the radar and the LWIR camera operated on a river levee. The tests were conducted with mountains, farm lands, levees, a river, and many vehicles and people and vehicles moving on paved and dirt roads and in the fields.
Two separate tests were conducted, one with the narrow-FOV LWIR sensor and one with the Furuno® radar. The purpose of both the LWIR and radar tests was to determine how well each sensor system could detect, track, and classify known targets of interest. It is important to note that the performance achieved in these tests for the LWIR sensor was independent of the radar cueing the LWIR sensor.
The radar detection and tracking movie depicts registered radar returns for the Furuno® radar mounted on top of a SUV parked on a Salinas River Levee (elevation of 6 meters above the fields). The number of ground-truth targets in the radar tests were limited to one SUV equipped with a GPS unit, three radar reflectors of known cross section (0, 2.8, and 5.3 dBsm), one field person, and several field hands identified during the tests.
Humans Walking. An analysis of the environmental conditions indicated that the tests were conducted under low, medium, and high clutter environments for both the radar and the LWIR that were comparable to those that would be encountered at the Southern Border.
The radar tests were performed with reflectors (0, 2.8, and 5.3 dBsm) that were indicative of people and vehicles. The tests included people walking in the vicinity of the 0 dBsm reflector that were detected and tracked. By design, the radar tests were only conducted with calibrated targets out to 2 km.
High Clutter Radar Measurements. The circled “arc” center portion of the left image obtained from the radar classification movie shown in
An analysis was performed to determine the level of clutter present during the Salinas field tests.
Radar Model Performance Estimates. A preliminary estimate of the system-level performance of the radar in terms of a receiver operating curve (ROC) was made. It was assumed that each scan of the radar is independent, and for the purposes of this preliminary estimate, it was further assumed that the data is Gaussian. The ROC presents the probability that a target will be declared as confirmed after a track-before-detect system has had N opportunities to collect radar contacts from it versus the mean number of false-track declarations that will occur per update cycle (one scan). Inputs are the target SNR, the first threshold (for accepting contacts for passage to the radar), and the number of radar update cycles to permit before a decision of confirm or not-to-confirm is required. In this presentation, we make no attempt to document the assumptions, approximations, or methodology of the model.
Four cases are presented in
As shown in
Finally, if the same SNR and first threshold values is retained as in the last case, but only 5 updates are allowed beyond the initial contacts, the result is shown in
Radar Detection Range on a Moving Vehicle. A vehicle has a cross-section that is 10 times larger than a person and is moving through radar resolution cells at 10 to 20 times the speed of a person. In comparison to a person or a group of people, a vehicle is much easier to detect and classify. The movies of the radar test in Salinas illustrate that detection of vehicles should be robust.
LWIR Modeling and Field Test Results. A preliminary estimate of performance is presented below for the LWIR smart sensor. First, a high level view of the signal processing is presented. Next, a few illustrations of the output of the signal processing for detection, classification and tracking of people and vehicles is presented. Images obtained from several movies are then used to illustrate the capability. Third, the results of the probability of recognizing a person or a vehicle visually by an operator, which is determined from NVTherm™ thermal imaging model calculations, are then presented. The NVTherm™ thermal imaging model was developed by the Night Vision Laboratory of the U.S. Army. Finally, output target statistics based on the field measurements for over 50 known targets are summarized.
Overview of a high-resolution optical surveillance system. The method used by the optical sensors is philosophically the same as the radar algorithm, except there is significant a priori information available to the optical sensors to make detection and tracking more robust.
Two sets of algorithms comprise the method of the present invention. They exploit the high-resolution electro-optical and long-wave infrared optical sensors to identify and classify subjects of interest within the border region. Two sets of LWIR/EO sensors will be employed. The first set will have a narrow angle of view (<3°) optimized for classification of objects in the outer ranges of the Inner Zone. The LWIR camera employed will be high resolution and cooled. The second set of cameras will have a wide angle of view (25 to 50°) and be used for wide-area surveillance at longer ranges as well as classification at shorter ranges. In addition, a wide angle IR sensor can employed in conjunction with the EO camera using an uncooled micro-bolometer.
There are two optical detection, tracking, and classification operational scenarios to consider. The first and primary mode of operation of the optical sensors is to exploit the wide-area search capabilities of the radar using its track information (i.e., a priori information) to cue the optical sensors. For this implementation, the optical sensor picks up the track presented by the radar by first optically detecting the target and then developing a track. The radar track and the optical track should be closely correlated. Once the track is picked up, the optical system classifies the target to determine if the target is real, how many targets are present, and what the target is (e.g. human or vehicle).
The second mode of operation is to independently detect, track, and classify without the benefit of a radar track. This might be needed for very slow moving targets, targets initiated in the Inner Zone, or unusual border crossing tactics designed to foil the sensor system. In this instance, the detection is accomplished without a priori radar tracking information. Acquiring the target, while very similar from a processing view point, will require more data than needed when the radar track is available. In this mode, the EO/IR system will be scanned across the Inner Zone.
Two spectral bands (IR and visual) are exploited to enhance the ability of the system to function during any portion of the diurnal cycle. The signal processing algorithms that underly the optical system is primarily responsible for classification (i.e. discrimination of real targets from false targets) but also needs to detect and track a target. The goal is to identify humans (and vehicles) that might be trying to cross the border and to distinguish humans (and vehicles) from animals and other objects not of interest. These algorithms rely on “features” that various moving objects. These features include:
The optical system will classify the target to determine if the target is real, how many targets are present, and what the target is (e.g., human or vehicle). A variety of a priori sources of information that aid in the detection, tracking and classification process include: weather conditions such as temperature, wind speed, and humidity; ambient light level; known human, vehicle and wildlife activity patterns; and terrain information such as roads, trails and topography. The statistics of these sources will be developed for input to the signal processing and may be known a priori, acquired from site surveys, or “learned” over a period of time. The algorithm uses a unified approach wherein each component of the system exploits these differentiating features.
Detection. The detection approach will exploit many of the unique facets of the sensor and the sensing environment. These facets include the sensor point spread function, fixed imaging geometry, the statistics of the background (including terrain, weather, and time of day), the movement of the subjects of interest and the known confusors (non-human targets such as animals, tumbleweed, swaying vegetation), and various combinations of each. The algorithmic approaches applied to the LWIR and the EO data are similar, but the distinct LWIR emissions of humans and warm vehicles provide a particularly powerful means for detection. The detector will incorporate (1) change detection between a new scene and a fully characterized background and (2) target motion and location in the scene. The detector will exploit both a prior and/or adaptive statistical model of the background and the movement of the target. Further, it will also be selective of size, velocity, and object temperature. The detector will exploit the cued data provided by the wide-area surveillance radar system and will also be able to independently detect, track, and classify without the benefit of the radar track.
Before the data can be exploited they must be registered to a common frame of reference. Accordingly, the first optical subsystem registers the outputs of both cameras and the terrain information to this common frame. Surveyed landmarks in the scene will be used in this process and will be carefully implemented to ensure the effectiveness of subsequent processing.
Exploitation of the benign nature of the background is achieved by developing spatial-temporal statistical models of the subject-free scene. This is accomplished by scanning the background with the camera to develop a database of background statistics. In instances for which a target-free calibration pass is not possible, adaptive statistical methods will be used to develop an estimate of the background dynamically. A two-step detection process is used. The first step exploits the stationary sensor geometry and the ability to carefully measure the target-free background. The second step applies a movement-based detector to those regions detected by the first stage background detector. While detection of movement using image flow or similar techniques can be computationally intensive, the fact that only a small subset of the data need be processed permits real-time operation. Velocities exceeding a given threshold are detected while those below the threshold are rejected. As a final step in the detection process, nearly-contiguous regions are conjoined using morphological techniques whose parameters depend on geodesic positioning data available from the camera pointing subsystem. The results are then passed onto the tracker algorithm for detection validation and tracking.
Alternatively, or in conjunction with this processing method, a track-before-detect method similar to the one that can be used for the radar can be implemented for the LWIR sensor.
Tracking. The tracking algorithm associates temporally-localized detections into longer term object trajectories. This is accomplished by developing a valid track of the target over a sufficient period of time to determine that movement is occurring. Once objects are detected, groups of pixels deemed to pertain to distinct targets are grouped and presented to the tracking algorithm. The algorithm to be used by the optical sensors is philosophically the same as the radar algorithm, except there is significant a priori information available to the optical sensors to make detection and tracking more robust. The tracker also plays a pivotal role in distinguishing targets of interest from confusors. The latter is accomplished by making the tracker selective to achievable velocity/acceleration as well as to the geographical regions that it is possible to traverse. In this way, detections that do not form feasible tracks are eliminated without having to undergo consideration from the classifier. A maximum likelihood tracker will be used. Additionally, the track developed using the EO and IR sensor data will be fused with the radar track to enhance performance.
Classification. The purpose of the classifier is to make the final decision pertaining to whether a given track needs to be further investigated by a border control agent. Along with this decision, the operator is presented with other information that was used to form the classification decision; for example, “thumbnail” movies showing the activities of targets over the observation intervals. The operator then uses this information to make a final decision as to whether the target is of interest or not.
The following features will be exploited for classification: (1) temperature and magnitude from the detection process; (2) velocity, acceleration, and trajectory geometry from the tracking process; (3) special features such as frequency-wavenumber signature, gait, as well as intensity variations. All of these will be used to identify people and vehicles illegally attempting to cross the border and to distinguish them from confusers like wildlife, terrain features, and other people and vehicles not attempting to cross the border. The different day-night outputs of the EO and the IR sensors will also be exploited.
The performance of the smart sensor system will be specified in terms of the probability of detection and the probability of false alarm. The signal (i.e., people or groups of people, vehicles or groups of vehicles moving very slowly to very fast, including people crawling or standing still for prolonged periods), and the sources of interference or noise, which can result in false alarms and false target detection, including sensor system and sensor background noise, terrain, and environmental conditions (e.g., weather, animals, vegetation such as rolling sagebrush, etc.). As part of this task, the various scenarios for detection, tracking and classification of people and vehicles in the Outer Zone, the Inner Zone, and after a border crossing will be developed and specified. The signal processing algorithms will be adapted to address each of these scenarios.
The method of the present invention will be used to detect a wide range of target signals. The variables of importance include number, size, speed, trajectory, and crossing tactics. The signal may be a single person crossing the border, where this person may be crawling, walking, running, walking and hiding behind terrain features, or other evasive tactics in each of the zones. The signal may also consist of a group of individuals or a person and an animal, where one or more of the group will scatter or attempt to cross the border. For high performance, each type of interference masking detection, tracking and classification or resulting in false alarms must be compensated for addressed. These interferences include (1) the important sources of background and noise influences, (2) the various categories of terrain and how terrain will influence performance, and (3) the various categories of environmental effects and their impact on performance. The specific design of the system will depend, may need to refine the design of the sensor system suite in terms of sensor and sensor system requirements, spacing, height, precision, resolution, and fusion.
LWIR Signal Processing.
Registration Processing. All data processing operations depend on accurate frame-to-frame identification of persistent features. Both instrumental and environmental factors can cause artificial displacements of the features that appear as image motion. The detection algorithm must reduce these errors to a prescribed level to meet performance objectives.
Registration requires two key processing steps: (1) camera alignment (not presently implemented) and (2) geospatial coordination. Camera alignment provides stabilization against frame-to-frame jitter and drift in the LOS. Geospatial coordination registers the image to a specific mapping on the earth's surface using in-scene fiducial points or a model based on digital terrain elevation data (DTED). This step is critical to successful cueing by the radar required for accurate position reporting of TOI locations.
Background Removal Processing. Once data frames are registered, system specific operations that facilitate subsequent algorithmic operations are performed. One of these is to determine the background of a scene so that it can be removed as part of the segmentation step. Gaussian statistical tests are used to flag pixels that are unlikely to belong to the scene as being foreground objects. This process can impact PD, because TOI's lost in the background will be removed and may result in missed detections.
Segmentation Processing. As discussed for the radar processing, segmentation is the formal process by which registered and conditioned data frames are translated into detection fields. Segmentation processing is required to perform first-order object detection. This process aggregates individual foreground pixels into target candidates, as shown in
Bayesian Tracker Processing. The tracker is required for final object declaration, and is the key to the function of the subsequent modules. The tracker leverages Bayesian tracking capabilities, eliminates spurious inputs due to clutter motion (as well as background removal and segmentation failures), rejoins separate single-object sections produced by occlusion or segmentation failure, produces time-history of object position, and handles obscuration. The results presented in this report were achieved without the rejoining and obscuration algorithms, which were not implemented at the time of the measurements. The Bayesian methods optimally match inputs with prior state. Decision statistics rely on space-time histories of objects, and model for dynamics of common objects. Tracker outputs include declared object positions and extent (within the image), and ground track (time history of position within the scene). In the final version of the software, the result will be presented in ground coordinates.
Classifier Processing. The system classifies target types based on model-derived and/or data-derived probability distributions as shown in
The system must support an open-ended configuration that allows runtime specification of class definitions, down-selection of type discriminator set, as well as decision rule logic. It must also accumulate results over the entire sequence of frames for a given track, greatly improving accuracy and robustness. Conventional Bayesian inference and Dempster-Shafer belief functions are supported, including the ability to:
As an example of this approach, statistical distributions of imaged-based features corresponding to type classes currently supported by the classifier can be derived from polygon representations of class type instances such as those shown in
Multiple target instances can also be projected onto the focal plane simultaneously to simulate groups of targets. Since the technique simulates the camera optics, class distributions can be tuned to a particular range or aspect. Because the features are extracted by the EO/IR processing code back-end itself, any responses introduced during EO/IR processing are fully comprehended. Sensitivities to noise and error sources can also be accounted for systematically.
A final cumulative classification decision is rendered and reported to the operator only after a consistency check has been applied to an accumulated sequence of frame-by-frame classification results. The consistency check is based on an assumption that agreement with the majority of class type decisions accumulated from previous frames constitutes “success” and any other decision constitutes “failure”. The relative proportion of successful and failed trials is compared to a binomial distribution where the assumed probability for success is held at a high confidence level (nominally 0.98). A threshold is applied to a log-likelihood ratio based on the probability that the classification sequence belongs to the assumed distribution versus the probability it does not.
4.2
LWIR Field Test Results from Salinas, Calif. The performance of the LWIR sensor for detecting, tracking, and classifying people and vehicles was tabulated from 14 different LWIR target tests conducted in Salinas. The results are shown below in Tables 5-7. Over the applicable range of the sensor, the analyses showed that all 44 of the people targets at ranges up to 1.8 km were detected, tracked, and classified. In this period, the analysis also showed that the classifier could also distinguish people as individuals or groups of people. The LWIR results indicated that people could be accurately and reliably detected, tracked, and classified at ranges of over 2 km with as few as 15 pixels on target. With the new higher resolution cameras, our analyses indicate that the current performance can be extended out to distances of 4 to 6 km. Some of the images obtained from the video clips are presented and discussed below.
LWIR Performance. An estimate of the performance of an IR sensor for detection, recognition, and identification of a target by visual inspection of an image by a trained operator can be made with the latest version of NVTherm™ thermal imaging model. The output of the model is expressed in terms of probability of detection, probability of recognition (P Recog) and probability of identification of a person or vehicle at a given range and for a given set of environmental conditions. The distinction between detection, recognition, and identification is straightforward. Detection means that a target has been declared even though the person may not know what it is. Recognition means that the type of target can be determined (e.g., a person can be distinguished from a vehicle or a group of people). Identification means that the type of target can be determined (e.g., what type of vehicle is it, a SUV or a station wagon). For the border application, DHS is mainly concerned, for example, about differentiating animals from humans, single individuals from groups of humans, and people from vehicles. Furthermore, for the purposes of this discussion, the probability of classification and the probability of recognition will be treated as referring to the same metric.
A set of NVTherm™ thermal imaging model predictions were made to determine the impact of the following parameters on the Precog for a LWIR sensor:
These results are summarized in
As illustrated in
The impact of turbulence on performance is illustrated in
Performance Classification Statistics for LWIR Data Collections at Baseline Ranges, Near Ranges, and Far Ranges. The results of the LWIR tests in the Salinas farmlands were tabulated to begin developing some statistics on the number of people and vehicles that could be detected, tracked, and classified. While not all of the signal processing algorithms are currently implemented (e.g., rejoining and obscuration), the results were excellent and suggest that the PD, PFA, and PFT desired by DHS can be achieved.
Baseline Classification Ranges. Table 5 summarizes the number of people, groups of people, and vehicles detected, tracked, and classified with the LWIR sensor system at ranges between 400 m and 1.8 km. The performance of the sensor system that is illustrated in this table is independent and was not cued by the radar. The results are excellent. All 44 of the 44 people targets were detected, tracked and properly classified. The numbers in the parentheses indicates if
groups of people were targets. Thus, at 1,460 m, the four people targets “(1+3)” consisted of “1” human walking alone, and one group of “3” humans walking together. The number of pixels on target was less than 35 at a range of 1.8 km. One missed track and several false alarms occurred for the vehicles, mainly because of obscuration, which is not implemented yet in the algorithms. The reasons for the errors are summarized in the footnotes on the table.
The results in Table 5 were used to estimate the performance at greater ranges when the next version of LWIR sensors are used. The baseline classification range for the test configuration camera (a LWIR camera with 320×240 pixels, a 40μ focal plane pitch, and 302 mm lens focal length) that was tested during Phase I was measured to be 400 m to 1.8 km. For a total azimuth field-of-view of at least 1.0 degree and a sufficiently stable tower structure, the baseline range can be scaled directly with the number of pixels on target. Therefore, the baseline range scales inversely with detector pitch and scales directly with focal length of the camera. A baseline range multiplication factor of five (5) is used to estimate a production baseline performance ranges over those used in the Phase I tests based on the following formula: (40μ/15μ)×(750 mm/302 mm) ˜5. Therefore, the baseline range for the early production configuration cameras (320×240 pixels Long Wave IR @ 15μ focal plane pitch and 750 mm lens focal length) that will be tested in Phase II is estimated to be 2,000 m to 9,000 m. It is important to note that the estimated performance impact in production for atmospheric scintillation and turbulence at longer ranges have not yet been measured. However, our model output above gives an indication of the impact of these effects on performance.
Near Classification Ranges. The results of some LWIR measurements at ranges less than 400 m were also made and are presented in Table 6. All 18 of the people targets were accurately detected and tracked. However, some of the people targets were not classified, because a large number of the farmhands observed in the video were picking crops and did not move in sustained translational motion. These targets are clearly visible and are detected as foreground objects, but
do not result in the formation of sustained tracks as very little or no displacement in target position is evident—a precondition for the current implementation of the tracker algorithm to declare an object as a target. Future versions of the software will include augmented detection and tracking algorithms to better address this type of target.
Far Classification Ranges. Table 7 presents the results of our analysis for greater ranges than 1.8 km. The probability of detection, tracking, and classification has dramatically fallen off at these ranges, mainly because of the decreased number of pixels on target and the foreground obscuring (reducing pixels on target) the targets. Based on a similar analysis as presented above, the performance results achieved at these ranges would scale to greater than 9,000 m using the next generation of LWIR camera.
Summary. In summary, the smart sensor system, which can be implemented either from a tower, a mobile vehicle, or both, is comprised of a low-cost, non-coherent X-band radar, a narrow field-of view LWIR sensor, and a real-time, signal processor in a supercomputer. The radar provides early detection, tracking, and 1st stage classification. When the targets of interest enter the field of view of the LWIR sensor, the radar cues that sensor to the targets, whereupon they are tracked and classified. Finally, the classified targets are forwarded to a control center for final confirmation by an operator, who upon validation of the target, designates an agent to apprehend the target. The field tests indicate that this system will achieve a high level of performance against the targets of interest (i.e., people walking, singly or in groups, horses moving with intent, and moving vehicles) for detection, tracking and classification.
While certain representative embodiments and details have been shown for purposes of illustrating the invention, it will be apparent to those skilled in the art that various changes in the methods and apparatus disclosed herein may be made without departing from the scope of the invention which is defined in the appended claims.
This application claims the benefit of U.S. Provisional Patent Application Ser. No. 60/804,251 filed Jun. 8, 2006.
| Number | Name | Date | Kind |
|---|---|---|---|
| 2940072 | Floresheim | Jun 1960 | A |
| 2965894 | Sweeney | Dec 1960 | A |
| 2985373 | Helgeson et al. | May 1961 | A |
| 3076961 | Bibbero | Feb 1963 | A |
| 3119582 | Kaufmann | Jan 1964 | A |
| 3141161 | King | Jul 1964 | A |
| 3153230 | Krevelen et al. | Oct 1964 | A |
| 3206745 | Waters | Sep 1965 | A |
| 3209352 | Buck et al. | Sep 1965 | A |
| 3209353 | Wiener et al. | Sep 1965 | A |
| 3213447 | Burrows et al. | Oct 1965 | A |
| 3241141 | Wall | Mar 1966 | A |
| 3287724 | Zupanick et al. | Nov 1966 | A |
| 3315255 | Sullivan et al. | Apr 1967 | A |
| 3325807 | Burns et al. | Jun 1967 | A |
| 3328795 | Hallmark | Jun 1967 | A |
| 3333263 | Kazakevicius et al. | Jul 1967 | A |
| 3643257 | Thomson et al. | Feb 1972 | A |
| 3898654 | Vehrs, Jr. | Aug 1975 | A |
| 3903525 | Mullins et al. | Sep 1975 | A |
| 4001826 | Moulton | Jan 1977 | A |
| 4015257 | Fetter | Mar 1977 | A |
| 4035800 | Lewis et al. | Jul 1977 | A |
| 4041491 | Taylor, Jr. et al. | Aug 1977 | A |
| 4075703 | Dillard | Feb 1978 | A |
| 4089001 | Donahue | May 1978 | A |
| 4104633 | Donahue et al. | Aug 1978 | A |
| 4137533 | Briechle et al. | Jan 1979 | A |
| 4555703 | Cantrell | Nov 1985 | A |
| 4618864 | Martin et al. | Oct 1986 | A |
| 4707697 | Coulter et al. | Nov 1987 | A |
| 4894658 | Hecht-Nielsen et al. | Jan 1990 | A |
| 4970660 | Marchant | Nov 1990 | A |
| 5469168 | Anderson | Nov 1995 | A |
| 5594451 | Krikorian et al. | Jan 1997 | A |
| 5703592 | Watts | Dec 1997 | A |
| 5963653 | McNary et al. | Oct 1999 | A |
| 6042050 | Sims et al. | Mar 2000 | A |
| 6211815 | Richey et al. | Apr 2001 | B1 |
| 6212471 | Stiles et al. | Apr 2001 | B1 |
| 6542739 | Garner | Apr 2003 | B1 |
| 6707419 | Woodington et al. | Mar 2004 | B2 |
| 6717545 | Dizaji et al. | Apr 2004 | B2 |
| 6765654 | Asaka et al. | Jul 2004 | B2 |
| 6771207 | Lang | Aug 2004 | B1 |
| 6828929 | Barbella et al. | Dec 2004 | B2 |
| 6844841 | Masciulli | Jan 2005 | B1 |
| 6850186 | Hellsten | Feb 2005 | B2 |
| 6853328 | Guice et al. | Feb 2005 | B1 |
| 6856272 | Levitan et al. | Feb 2005 | B2 |
| 6884841 | Nava | Apr 2005 | B2 |
| 6903676 | Frady et al. | Jun 2005 | B1 |
| 6967612 | Gorman et al. | Nov 2005 | B1 |
| 7049998 | Frady et al. | May 2006 | B1 |
| 7142150 | Thackray | Nov 2006 | B2 |
| 7236123 | Siegel | Jun 2007 | B2 |
| 7286079 | Blunt et al. | Oct 2007 | B2 |
| 7295106 | Jackson et al. | Nov 2007 | B1 |
| 7298316 | Tsai et al. | Nov 2007 | B2 |
| 7376247 | Ohta et al. | May 2008 | B2 |
| 7583815 | Zhang et al. | Sep 2009 | B2 |
| 7646329 | Britton et al. | Jan 2010 | B2 |
| 7800527 | Douglass et al. | Sep 2010 | B2 |
| 20030030582 | Vickers | Feb 2003 | A1 |
| 20030179129 | Tamatsu et al. | Sep 2003 | A1 |
| 20040150552 | Barbella et al. | Aug 2004 | A1 |
| 20050046584 | Breed | Mar 2005 | A1 |
| 20060049978 | Siegel | Mar 2006 | A1 |
| 20060077255 | Cheng | Apr 2006 | A1 |
| 20060125679 | Horibe | Jun 2006 | A1 |
| 20060125680 | Thackray | Jun 2006 | A1 |
| 20070075892 | Horibe | Apr 2007 | A1 |
| 20090015460 | Fox et al. | Jan 2009 | A1 |
| 20090128399 | Root, Jr. | May 2009 | A1 |
| 20100013917 | Hanna et al. | Jan 2010 | A1 |
| 20100283662 | Fox et al. | Nov 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 03094186 | Apr 1991 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20110001657 A1 | Jan 2011 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60804251 | Jun 2006 | US |
