The present invention relates generally to digital image processing. More specifically, but not by way of limitation, the present invention relates to methods and systems for tracking moving objects in a scene.
One problem that arises in the field of digital image processing is that of tracking moving objects in a scene. In some applications, there is a need to track objects that pass through the scene relatively quickly. In other applications, there is a need to track relatively slow-moving objects that linger in the scene for a longer period.
One example of the latter type of application above is the tracking of insects such as honeybees in the detection of explosive materials that are used in making bombs. Properly trained honeybees can detect the vapor plumes given off by such explosive materials. The honeybees can be conditioned by adding trace amounts of the major chemical components of the explosive materials to a feeder. The honeybees are thus conditioned to associate the chemical smell with food. When the honeybees are released at a particular location (e.g., near a vehicle checkpoint), they will pause over an object (e.g., a vehicle) containing explosive materials as they forage for food.
Tracking objects such as foraging honeybees presents some challenges. First, insects such as honeybees are very small objects when viewed from a distance and may occupy only a few pixels in a captured digital video frame. Second, the scene often includes noise due to rustling vegetation or ephemeral glints of sunlight. Of course, there is also some low-level noise inherent in any digital imaging device used to capture the scene. Finally, the scene may at times also include moving objects not of interest (e.g., birds or airplanes) that pass through the scene relatively quickly.
It is thus apparent that there is a need in the art for an improved method and system for tracking moving objects in a scene.
Illustrative embodiments of the present invention that are shown in the drawings are summarized below. These and other embodiments are more fully described in the Detailed Description section. It is to be understood, however, that there is no intention to limit the invention to the forms described in this Summary of the Invention or in the Detailed Description. One skilled in the art can recognize that there are numerous modifications, equivalents, and alternative constructions that fall within the spirit and scope of the invention as expressed in the claims.
The present invention can provide a method and system for tracking moving objects in a scene. One illustrative embodiment is a method for tracking moving objects in a scene, the method comprising acquiring a digital video signal corresponding to the scene; identifying in the digital video signal one or more candidate moving objects in the scene; locating at least one candidate moving object in the digital video signal subsequent to identification of the at least one candidate moving object; tracking candidate moving objects that, for at least a predetermined period after they have been identified, continue to be located in the digital video signal; assigning a score to each tracked candidate moving object in accordance with how long after passage of the predetermined period the tracked candidate moving object has continued to be located in the digital video signal; combining the respective scores of the tracked candidate moving objects to obtain an overall score for the scene; and indicating to a user whether the overall score satisfies a predetermined criterion.
Another illustrative embodiment of the invention is a system for tracking moving objects in a scene, the system comprising at least one processor and a memory containing a plurality of program instructions configured to cause the at least one processor to acquire a digital video signal corresponding to the scene; identify in the digital video signal one or more candidate moving objects in the scene; locate at least one candidate moving object in the digital video signal subsequent to identification of the at least one candidate moving object; track candidate moving objects that, for at least a predetermined period after they have been identified, continue to be located in the digital video signal; assign a score to each tracked candidate moving object in accordance with how long after passage of the predetermined period the tracked candidate moving object has continued to be located in the digital video signal; combine the respective scores of the tracked candidate moving objects to obtain an overall score for the scene; and indicate to a user whether the overall score satisfies a predetermined criterion.
These and other embodiments are described in further detail herein.
Various objects and advantages and a more complete understanding of the present invention are apparent and more readily appreciated by reference to the following Detailed Description and to the appended claims when taken in conjunction with the accompanying Drawings, wherein:
In various illustrative embodiments of the invention, the difficulties of tracking a small moving object such as a honeybee in the presence of noise due to digital-imaging-device imperfections, rustling vegetation, and ephemeral glints of sunlight are overcome through appropriate filtering of a digital video signal corresponding to the scene. The digital video signal may be one captured in real time, or it may be a prerecorded digital video signal that is analyzed after the fact.
In one illustrative embodiment, a system for tracking moving objects in a scene identifies one or more candidate moving objects in the digital video signal. To separate objects of interest from noise, the system requires that a candidate moving object, once it has been identified, continue to be observed in the digital video signal for at least a minimum period (e.g., a for predetermined number of frames). For each candidate moving object that has been observed in the digital video signal for at least the minimum period, the system tracks that candidate moving object and assigns a score to it in accordance with how long after passage of the minimum period it continues to be observed in the digital video signal.
In this illustrative embodiment, the system combines the respective scores of the tracked candidate moving objects to obtain an overall score for the scene. The system indicates to a user whether the overall score satisfies a predetermined criterion. In some embodiments, the digital video signal is analyzed in real time to provide a real-time indication of whether the overall score satisfies the predetermined criterion.
In various illustrative embodiments of the invention, the filtering techniques applied to the digital video signal exploit known properties of the moving objects that are of interest. For example, in an application in which honeybees are tracked to detect explosive materials, assigning a score to tracked moving objects that increases with time as the tracked moving object continues to be located in the digital video signal favors objects such as foraging honeybees that tend to dwell within the scene for relatively long periods and disfavors objects such as airplanes or birds that may pass relatively quickly through the scene.
Referring now to the drawings, where like or similar elements are designated with identical reference numerals throughout the several views, and referring in particular to
In one illustrative embodiment that has been field tested, digital imaging device 105 is a PANASONIC AG-DVX100B digital camcorder interfaced via FIREWIRE with a DELL INSPIRON E1505 laptop computer. In this particular embodiment, the digital camcorder is configured for 30 full (non-interlaced) frames of video per second. The camcorder's autofocus feature is generally turned off during the tracking of small moving objects such as honeybees.
In other embodiments, system 100 does not include a dedicated digital imaging device 105 as shown in
Versatile Disc (DVD), or a data network.
One advantage of the embodiment shown in
Though digital imaging device 105 and computing device 110 are shown as separate devices in
In the embodiment shown in
Memory 225 may include, e.g., random-access memory (RAM), read-only memory (ROM), flash memory, or a combination or sub-combination thereof. In the embodiment shown in
For convenience in this Detailed Description, application 240 has been divided into a set of functional modules: (1) data I/O module 245, (2) analysis engine 250, and (3) user interface 255.
Data I/O module 245 can be configured to handle both incoming and outgoing data. For example, data I/O module 245 can manage digital-image data received from a digital imaging device 105 or from a storage device containing a prerecorded digital video file. Data I/O module 245 can also control display 220 and perform other output-data-related functions.
Analysis engine 250 is configured to analyze a digital video signal corresponding to a scene of interest to identify and track moving objects in the scene. In doing so, analysis engine 250 employs various digital image processing techniques that will be described below.
User interface 255 manages interactions with a user of computing device 110. User interface 255 receives input commands from the user (e.g., via a keyboard or mouse) and, in conjunction with data I/O module 245, communicates appropriate prompts or other guidance to the user during the operation of computing device 110.
In other embodiments, the functional modules of application 240 described above may be subdivided or combined in ways other than that indicated in
In identifying and tracking moving objects in a digital video signal, it is helpful to separate moving foreground objects from the mostly stationary background. One technique for performing this separation is generating a differential video signal by computing the difference between frames of the digital video signal. This technique may be termed “frame subtraction,” and a frame of the resulting differential video signal may be termed a “difference frame.” Regions of a difference frame that are black (zero or close to zero in intensity) indicate background regions that do not change from frame to frame. Bright areas in a difference frame indicate possible moving objects or noise (e.g., due to ephemeral glints of sunlight or rustling vegetation). Distinguishing the former from the latter involves further processing that will be described below.
In the embodiment shown in
In the embodiment shown in
In the embodiment shown in
In the embodiment shown in
Those skilled in the art will recognize that the various illustrative embodiments discussed above in connection with
Once analysis engine 250 has produced a differential video signal from the digital video signal acquired via data I/O module 245, analysis engine 250 identifies and tracks candidate moving objects in the digital video signal. As discussed in further detail below, this may be accomplished by applying what may be termed a “proximity filter” to the differential video signal.
In the example of
The technique illustrated in
When analysis engine 250 has successfully located a particular candidate moving object 410 for a sufficiently long period to warrant a conclusion that it is not a false positive (e.g., speckle or noise), analysis engine 250 begins tracking and scoring the candidate moving object 410. For example, in one embodiment, analysis engine 250 begins tracking and scoring a candidate moving object 410 after it has been located in ten successive difference frames 405. For each difference frame 405 thereafter in which the particular candidate moving object 410 is observed, a numerical score associated with that candidate moving object 410 is incremented. Thus, the longer a given candidate moving object 410 continues to be observed in the digital video signal, the higher its score becomes. Such a scoring system favors relatively slow-moving objects that dwell for a relatively long time in the scene (e.g., foraging honeybees).
The above-described characteristic can be accentuated by increasing the score of a tracked candidate moving object 410 non-linearly with time. For example, in one embodiment, the score of a tracked candidate moving object 410 is incremented by one for each of the first ten difference frames 405 in which it is found once analysis engine 250 has begun tracking the candidate moving object 410, by four for each of the next ten difference frames 405 in which it is found, by eight for each of the next ten difference frames 405 in which it is found, etc. The particular values by which the score is incremented and the division of time into periods including ten difference frames 405 mentioned above are merely illustrative. In general, in such an embodiment, the rate at which the score of a tracked candidate moving object 410 is incremented is increased with time as the tracked candidate moving object 410 continues to be located in the differential video signal.
As mentioned above, the individual scores of the tracked candidate moving objects 410 can be combined into an overall score for the scene, and analysis engine 250 can be configured to test the overall score against a predetermined criterion. In one illustrative embodiment, analysis engine 250 compares the overall score with a predetermined threshold, and the predetermined threshold's being exceeded indicates that there is likely to be at least one foraging honeybee in the scene. User interface 255 may report to the user whether the predetermined criterion has been satisfied (e.g., whether or not the predetermined threshold has been exceeded). In reporting the status to the user, user interface 255 may employ a visual indicator such as text and/or graphics on display 220 or a light; an audible indicator such as a beep, tone, or recorded spoken message; or both.
In embodiments in which the satisfying of the predetermined criterion is the exceeding of a threshold, the threshold may be chosen in accordance with the particular application and environment and may be user adjustable. For example, on a given day, the threshold may be adjusted relative to a measured background score of the scene at the particular location where the system is to be used. Tracked candidate moving objects 410 corresponding to foraging honeybees or other moving objects of interest in the scene will then “stand out” from the established background score by exceeding an appropriately chosen threshold above the “floor” background score.
At 515, analysis engine 250 locates at least one candidate moving object 410 in the digital video signal subsequent to its initial identification at 510. At 520, analysis engine 250 tracks any candidate moving objects 410 that, for at least a predetermined period after their initial identification, continue to be observed in the digital video signal. At 525, analysis engine 250 assigns a score to each tracked candidate moving object 410 in accordance with how long the tracked candidate moving object 410 has continued to be located in the digital video signal beyond passage of the predetermined period.
At 530, analysis engine 250 combines (e.g., totals) the respective scores of the tracked candidate moving objects 410 to obtain an overall score for the scene. At 535, user interface 255 indicates to a user whether the overall score satisfies a predetermined criterion. The process terminates at 540.
In one embodiment, computing device 110 outputs the differential video signal produced by analysis engine 250 to display 220 in approximately real time (depending on how many video frames 305 are required for the computation of each difference frame 405, there is some latency in the differential video signal relative to the digital video signal due to buffering of video frames 305). Displaying the differential video signal can aid the user in aiming or repositioning digital imaging device 105. To further aid the user in aiming or repositioning the digital imaging device 105, a low-intensity version of the overall scene can be combined with the differential video signal in a single view. For example, each pixel of the digital video signal corresponding to the scene can be divided by a predetermined non-zero value greater than unity, and this reduced-intensity view of the scene and the differential video signal can be superimposed on display 220.
In one particular application alluded to above, a system based on the principles of the invention is used to track foraging honeybees at a vehicle checkpoint to detect vehicles containing explosive devices. In such an application, the number of foraging honeybees tracked at any one time may be as few as one to three. Many other applications of the principles of the invention are possible. In general, the techniques described above are particularly effective for tracking relatively slow-moving objects that remain within a scene for a relatively long period. Examples of such moving objects other than foraging honeybees include other kinds of insects, circling birds, weather balloons, etc.
In some applications, it may be advantageous to deploy a system based on the principles of the instant invention separately but simultaneously with a system based on the techniques described in commonly owned and assigned U.S. application Ser. No. 11/338,806, Attorney Docket No. MONT-031/01US, filed on Jan. 25, 2006.
In conclusion, the present invention provides, among other things, a method and system for tracking moving objects in a scene. Those skilled in the art can readily recognize that numerous variations and substitutions may be made in the invention, its use, and its configuration to achieve substantially the same results as achieved by the embodiments described herein. Accordingly, there is no intention to limit the invention to the disclosed illustrative forms. Many variations, modifications, and alternative constructions fall within the scope and spirit of the disclosed invention as expressed in the claims.
Number | Date | Country | Kind |
---|---|---|---|
PCT/US2005/016170 | Nov 2005 | US | national |
Some of the subject matter of the instant application is related to commonly owned and assigned U.S. application Ser. No. 11/338,806, Attorney Docket No. MONT-031/01US, entitled “Optical Detection of Oscillating Targets Using Modulation of Scattered Laser Light.”
This invention was made with government support under contract W909MY-06-C-0037 awarded by the U.S. Army. The U.S. government has certain rights in the invention.