The current disclosure relates to a system and method for estimating the range of an object in video data using anthropometric measures.
Many applications, such as applications that determine and execute target strikes, require range measurements from a sensor to a target of interest, velocity measurements of interest, and a view of the target. For example, the range to the target is used for estimation of bullet drop in a target strike, and the target velocity is used to predict the anticipated target location when the bullet strikes. Numerous optical flows in video data have been developed, and velocities are often computed in the image plane and therefore capture only two dimensions of the velocity vector. Moreover, such range and velocity computations are very demanding and not suitable for portable devices wherein computation resources are scarce. Consequently, for a portable device, it can be critical that the complexity of the designed algorithms draw only minimal power.
Managing visual acuity and cognitive load when using firearms such as during combat operations are important to safely accomplishing mission objectives. While expert shooters have the ability to shift from visual attention to a somatosensory one during shooting, this ability is rarely encountered in a typical soldier. Developing technology that alleviates the need for visual attention by providing information extracted from the visual cues captured by a sensor will help every soldier be a marksman. The advanced video analytics methodology of the present disclosure facilitates and automates the interpretation of visual cues and enhances a soldier's or other rifleman's marksmanship.
Shooting a moving target is difficult since the target center is constantly shifting. Usually, a marksman cognitively estimates the target range and target velocity, then fires at the anticipated target location based on the estimated travel distance. In an embodiment, video analytics, rather than a marksman's cognitive estimations, are used to estimate target range and velocity, and to predict the expected target location. After the estimation process, the result can be displayed to the marksman, for example on a firearm scope, and this result including the position at which to aim enables a soldier to acquire and engage moving targets in the battlefield.
A firearm scope system can include a suite of sensors and an onboard processor, and as explained above, these features can augment the capabilities of a soldier or other marksman. These capabilities normally include visual information which is the primary cue sought by a soldier when using a scope. Enhancing the ability of a soldier in estimating the range to a target and the target velocity, and in compensating for bullet drop, will significantly improve the soldier's marksmanship, and therefore increase the safety of the soldier in the battlefield. Such advanced video analytics will improve the marksmanship of every soldier by reducing cognitive load on the solider and improving the assessment of ambient conditions. The video analytics of the current disclosure provides useful information to aid a soldier in striking the targets under situations ranging from close quarters combat (CQB) to long range targeting.
Numerous optic flow approaches have been developed to compute object velocity. However, these velocities are often computed in the image plane, and therefore capture only two dimensions of the velocity vector. Moreover, such computations are very intensive, and therefore are not suitable for a firearm application where computation resources are scarce. To address this issue, in an embodiment, the knowledge of the target range is leveraged, and an algorithm separately computes the angular (i.e., image plane velocity) and the radial components of the target velocity.
In an embodiment, the range to a target is estimated based on an anthropometric measurement such as a shape-based measure of virtually any object (such as the outline of a person's head and shoulders, or the outline of a car or tank). After determining the range, then a velocity estimate separately computes the radial and angular velocities.
For a target ranging aspect of an embodiment, anthropometric a priori knowledge is used. The knowledge of range to target can be important for precision target strikes. Many marksman requirements, such as bullet drop compensation, wind compensation, and moving target location prediction, depend primarily on the range to the target. It is known that expert shooters visually analyze anthropometric measurements such as the distance of the top of head to shoulders and shoulder to shoulder distance to estimate the target range. In an embodiment, a video analytics algorithm automates this ranging method, which reduces a soldier's cognitive load.
A three step process is used in computing the range to target. The first step automatically detects and extracts the anthropometric structure from the image. Many computer vision studies have shown that the head-and-shoulder contour has a consistent omega shape. Once the omega shape model is accurately fitted to the image, anthropometric measurements can be estimated directly from the template. The last step computes the range to the target using the extracted head-and-shoulder contour and length, pixel resolution, and a priori head-and-shoulder length derived from anthropometric statistics.
Computation of the target velocity is broken down into separate computations of its radial and angular components. The radial velocity leverages on the target range and the angular velocity (i.e., image plane velocity), and uses an approach requiring fewer computations than an optic flow approach. The radial velocity is computed based on the changes in object range between two frames and the time difference between the two frames. The range is estimated as described above. Angular velocity is computed using spatial-temporal derivatives based on a motion constraint equation. While instantaneous measurements of the three-dimensional velocity of the target are naturally prone to errors in computing gradient and ranging estimates, one or more embodiments are augmented by implementing filtering methods that leverage the high frame rate of a sensor to increase velocity estimation accuracy by combining instantaneous measurements and accumulations over a sliding window. An embodiment is best implemented in low power hardware, such as field programmable gate arrays (FPGA) or digital signal processors (DSP), with interfaces including memory storage and input image frame acquisition such as a frame grabber.
The first step in determining the range to the target consists of automatically detecting and extracting the anthropometric structure from the image. As noted above, numerous computer vision studies have shown that the often observed head-and-shoulder contour has a consistent omega (Ω) shape, and algorithms have been developed to detect such contour. In one approach, a processor detects the edges of a head-and-shoulder contour in an image, and the edges are matched with a nominal set of head-and-shoulder models using a gradient descent approach. Examples of several such models are illustrated in
Once the Ω-shape model is accurately fitted to the image, anthropometric measurements can be estimated directly from the template. In an embodiment, attention is focused primarily on the distance from the top of the head to the shoulder. See
The last step computes the range to the target, R, from the extracted head-to-shoulder length (LHS in pixels), pixel resolution (PΘ in pixels per radian, which depends on the scope magnification), and the a priori head-to-shoulder length, LA, derived from anthropometric statistics.
R=LA/θ=LA/LHS·Pθ
The range calculation is illustrated in
The head-to-shoulder length of each person differs slightly and the head-to-shoulder distance in pixels may not be precise, thus resulting in an error in range computation. However, an error of 10 m at 600 m distance is tolerable and will not negatively impact a soldier or other marksman. An error in range estimates can be quantified and a variance can be provided on the estimated range that reflects the accuracy of the Ω-shape model fitting. Results can always be analyzed to ensure that the range accuracy is within the tolerance.
The normal flow, which corresponds to the motion along the normal to edges of the head-to-shoulder contour, represents a good estimation of the lateral or angular target velocity, vn. The normal flow is computed from the spatial-temporal derivatives of the target structure as follows:
where It, Ix, Iy, are the temporal derivative and the intensity derivatives along the x and y directions respectively. ∇I is the spatial intensity gradient.
The radial velocity of the object is computed based on the changes in the range to the target, which is equivalent to changes in sizes of the object in the images. By registering the ranges of the object at time τ and τ−δτ, the radial velocity is readily computed by the following formula:
vr=(Rτ−δτ−Rτ)/δτ
These instantaneous measurements of the three dimensional velocity of the target are naturally prone to errors in computing gradient and ranging estimates. However, the method is augmented by implementing filtering techniques that leverage the high frame rate of the sensor to increase velocity estimation accuracy by combining instantaneous measurements and accumulations over a sliding window.
The calculated target range and target velocity can be used to compensate for moving targets. In order to strike a moving target, a rifleman predicts the location where the target is anticipated to move to before shooting. Using the algorithm, once the angular and radial target velocities are computed, the anticipated target location (xτ+δτ, yτ+δτ) is estimated from the current target location (xτ, yτ), the target velocity (vn, yr), and the flight time of the bullet, which is computed from the known bullet exit velocity, v0, and the range to target, R, as determined above. After these computations and estimations, the target location of a moving subject is accurately predicted and displayed on a scope. While this is a very challenging task as many factors have to be taken into account such as wind, ambient air density, drag coefficient, and elevation of the shot for an accurate estimation, a large number of resources are available to model bullet trajectory and therefore infer the time of flight. In combination with these additional resources, the physical parameters of range and velocity are derived from the currently disclosed video analytics system and method, variations are assessed, and efficient approximations are made that assist the soldier in engaging and shooting moving targets.
Referring now to
At 630, the object comprises a humanoid object. At 635, a contour of the object is detected using a model of the object. At 640, the model comprises an anthropometric model. At 645, the anthropometric model comprises an omega-shaped model to detect a head and shoulder contour from the image data. At 650, instantaneous measurements of the range and the three-dimensional velocity are calculated, and the instantaneous measurements of range and three-dimensional velocity are accumulated over a sliding window. At 655, the image data comprises video image data. At 660, the range calculation and the velocity calculation are used to display on a firearm scope a predicted location of a moving target.
Moreover, those skilled in the art will appreciate that the invention may be practiced with other computer system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCS, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computer environments where tasks are performed by I/0 remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
In the embodiment shown in
It should be understood that there exist implementations of other variations and modifications of the invention and its various aspects, as may be readily apparent, for example, to those of ordinary skill in the art, and that the invention is not limited by specific embodiments described herein. Features and embodiments described above may be combined with each other in different combinations. It is therefore contemplated to cover any and all modifications, variations, combinations or equivalents that fall within the scope of the present invention.
The Abstract is provided to comply with 37 C.F.R. §1.72(b) and will allow the reader to quickly ascertain the nature and gist of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.
Number | Name | Date | Kind |
---|---|---|---|
4370914 | Voles | Feb 1983 | A |
4922801 | Jaquard et al. | May 1990 | A |
5764786 | Kuwashima et al. | Jun 1998 | A |
5966859 | Samuels | Oct 1999 | A |
5991043 | Andersson et al. | Nov 1999 | A |
6174288 | Samuels | Jan 2001 | B1 |
6237462 | Hawkes et al. | May 2001 | B1 |
6871439 | Edwards | Mar 2005 | B1 |
6986302 | LaFata | Jan 2006 | B2 |
6997716 | Skala et al. | Feb 2006 | B2 |
7404268 | Page | Jul 2008 | B1 |
8051597 | D'Souza et al. | Nov 2011 | B1 |
8094934 | Morimitsu | Jan 2012 | B2 |
8325065 | Maris | Dec 2012 | B2 |
8336776 | Horvath et al. | Dec 2012 | B2 |
20040101162 | Higaki et al. | May 2004 | A1 |
20060005447 | Lenner et al. | Jan 2006 | A1 |
20070274566 | Fujimoto | Nov 2007 | A1 |
20080273750 | Fujimoto | Nov 2008 | A1 |
20090213219 | Eggert et al. | Aug 2009 | A1 |
20130028486 | Backlund et al. | Jan 2013 | A1 |
Number | Date | Country |
---|---|---|
4408864 | Sep 1995 | DE |
10141913 | May 1998 | JP |
Entry |
---|
“United Kingdom Application Serial No. 1122152.0, Combined Search and Examination Report mailed Apr. 23, 2012”, 5 pgs. |
Number | Date | Country | |
---|---|---|---|
20120170815 A1 | Jul 2012 | US |