Patent Grant
11576367
This application is related to and claims priority from earlier filed U.S. patent application Ser. No. 16/682,268, filed Nov. 13, 2019.
Embodiments of the invention relate to monitoring wildlife movement patterns, in particular bird movement patterns, and a system and method for automating bird deterrent systems to prevent birds from occupying certain areas of interest, such as airfields and crop fields.
Controlling wildlife is a difficult matter. Prior art in the area of bird control describes devices for automated bird repelling using laser technology which is pre-programmed to repeatedly move along set patterns or scan areas of concern (see WO2015093938 the entire contents of which are incorporated herein by reference). Although these systems have been proven to be effective, they have drawbacks.
Of particular concern in connection with the present invention is that the prior art systems are without any wildlife or bird detection capabilities. Accordingly, there is no means to do an objective assessment of the bird problem or determine the true effectiveness of the deterrent system. Moreover, without detection capability, the existing systems are not able to adapt to the behavior of the birds. Rather, the prior art system is pre-programmed during installation of the device and does not deviate from the programmed pattern or timing.
Another concern is that the described device also projects the laser when there are no birds present at all, or when birds are positioned at locations other than those at which the laser is pointing. Altogether, this increases the risk of habituation. It also results in unnecessary power consumption and more importantly it increases the risk of shortening the life time of the device.
In summary, the prior art lacks any objective means to measure the extent of the existing bird problem or to measure the effectiveness of the repelling system. Additionally, there is no feedback mechanism implemented to improve the repelling effectiveness, i.e. to adapt the motion of the laser to changing bird occupation habits. And finally, there is no way to efficiently decrease the run time of the laser to improve the durability and lifetime of the laser and other components.
Detecting any type of wildlife poses a great challenge in itself. Many sensors have a limited range or a limited field of view. An example of a limited range/field of view device is a so-called animal trap camera, which uses motion detection to take pictures. The limited range decreases the extent to which such a camera can be used for broader scope wildlife monitoring and control. On the other hand, sensors that can detect at great distances are much more expensive and are not easy to deploy. An example is a long-range radar system to track bird migrations around airports. The mentioned shortcomings of such a bulky system significantly decrease the usability of the detection device for different purposes. In summary, there is no wildlife detection system that has the flexibility to be implemented in different situations, while still being able to detect at larger distances as needed for wildlife measurements and feedback to a repelling device.
The present disclosure describes a system which is able to detect and recognize wildlife, and in particular birds, using camera images. The present solution is comprised of integrated hardware devices, algorithms and software which cooperate to provide an effective and reliable detection and monitoring system. Properly equipped, the system can be made to be portable and can be set up at any location for different wildlife detection and repelling purposes.
An automated wildlife monitoring and control system generally comprises a main processing module which includes a CPU/GPU, user interface, storage/memory and wired/wireless communication, a sensor module for acquiring a stream of sequential images and a control module for controlling a laser or other deterrent device. The sensor(s) gather data (images), which are first processed and then fed to the detection algorithms. The algorithms turn this data stream into object and environment information. Based on this, a high-level state machine (AI software running on the CPU or GPU) will select different machine behaviors for the purpose of monitoring or repelling. Subsequently, relevant commands are sent to the control algorithms and a controller towards the repelling devices in the system.
To adjust the system behavior to different environments and different goals, a system configuration method or protocol is implemented. To perform extensive analysis, a dedicated logging structure is put in place by storing historical images. Lastly, a user interface is provided to adjust a subset of the loaded configuration. This can for example be used for fine tuning of different parameters.
Accordingly, it is an object of the invention to provide an automated wildlife detection and monitoring system which is capable of objective wildlife assessment, autonomous monitoring and repelling of the wildlife, adaptability to changing wildlife patterns, feedback on repelling effectiveness, changing repelling behavior based on monitored wildlife patterns, and creating new or custom repelling behaviors.
It is a further object of the invention to provide a system which is suitable for different environments, which reduces unnecessary power consumption, reduces use with same or better efficiency, and increases device longevity.
While the specification concludes with claims particularly pointing out and distinctly claiming particular embodiments of the present invention, various embodiments of the invention can be more readily understood and appreciated by one of ordinary skill in the art from the following descriptions of various embodiments of the invention when read in conjunction with the accompanying drawings in which:
Certain exemplary embodiments will now be described to provide an overall understanding of the principles of the structure, function, manufacture, and use of the device and methods disclosed herein. One or more examples of these embodiments are illustrated in the accompanying drawings. Those skilled in the art will understand that the devices and methods specifically described herein and illustrated in the accompanying drawings are non-limiting exemplary embodiments and that the scope of the present invention is defined solely by the claims. The features illustrated or described in connection with one exemplary embodiment may be combined with the features of other embodiments. Such modifications and variations are intended to be included within the scope of the present disclosure. Further, in the present disclosure, like-numbered components of the embodiments generally have similar features, and thus within a particular embodiment each feature of each like-numbered component is not necessarily fully elaborated upon. Additionally, to the extent that linear or circular dimensions are used in the description of the disclosed systems, devices, and methods, such dimensions are not intended to limit the types of shapes that can be used in conjunction with such systems, devices, and methods. A person skilled in the art will recognize that an equivalent to such linear and circular dimensions can easily be determined for any geometric shape. Further, to the extent that directional terms like top, bottom, up, or down are used, they are not intended to limit the systems, devices, and methods disclosed herein. A person skilled in the art will recognize that these terms are merely relative to the system and device being discussed and are not universal.
The present disclosure provides an exemplary system which is able to detect and recognize wildlife, and in particular birds, using camera images. The exemplary embodiments are comprised of integrated hardware devices, peripherals, and processing devices which run software algorithms capable of processing images, detecting wildlife within the images, determining areas or targets of interest within the images, and creating or adapting repelling patterns or activities based on the detected wildlife. Properly equipped, the system can be made to be portable and can be set up at any location for different wildlife detection and repelling purposes.
Referring to
To adjust the system behavior to different environments and different goals, a system configuration method or protocol 134 is implemented. To perform extensive analysis, a dedicated logging structure 136 is put in place by storing historical images. Lastly, user interface data is provided to adjust a subset of the loaded configuration. This can for example be used for fine tuning of different parameters.
Referring back to
The sensor module 116 generally includes an optical lens (not shown), an electrooptical image sensor (camera) 140, and an image signal processor 142. Electrooptical image sensors 140 can include CCD, CMOS, Infra-Red (IR), thermal, etc. The lens projects the surroundings onto the image sensor 140 to create a digital image representation of the environment. It is further processed on the image signal processor 142 to improve the overall quality of the digital representation. A camera 140 is chosen for the sensor module 116 because it allows the system 100 to be adapted to different situations at different locations (customers) by swapping specific components. By choosing a thermal imaging sensor, for example, viewing capabilities during nighttime are enhanced. By selecting the proper lens and sensor resolution, detection at various distances is easily customized. Additionally, these types of imaging sensors can be easily mounted on an actuator such as a Pan and Tilt Unit to be able to cover all the surroundings and not be limited by the natural field of view of the lens system. In other embodiments, the imaging sensors may also utilize the input of a moving mirror which reflects the image of the surroundings onto the sensor.
For optimal performance, each situation needs to be individually assessed and the most suitable sensor and lens should be selected based on this assessment. This assessment includes the environment, as well as the type and size of wildlife intended to be detected and monitored. By optimizing optical and digital zoom functionality, the system can focus on selected targets such that the animal covers enough pixels on the image sensor to be detected as an animal instead of a blur or a dot. Additionally, the dynamic range of the image needs to be sufficient to be able to obtain a high level of detail on the image even during low lighting conditions. The processing unit 142 should always allow for (automatically) adjusting the gain, exposure time and white balance. Also, the processing unit 142 should be capable of selecting a Region of Interest (ROI) within the image to exclude regions of the image sensor during detection and to scan different regions of the image sequentially.
The following decision strategy is adopted to adapt the sensor module 116 to the use case.
1. Monitoring in real time?
Yes: global shutter (image sensor which scans the entire area of the image simultaneously.
No: rolling shutter. (image sensor which scans the image sequentially from one side of the sensor (usually top) to the other, line by line)
2. Monitor complete surroundings?
Yes: mount detection system on moving base.
No: mount detection system on fixed base.
3. Monitor during night time?
Yes: use a LWIR type image sensor (thermal vision).
No: use a CMOS/CCD type image sensor (regular vision).
4. Monitor during low light conditions?
Yes: maximize dynamic range.
No: no additional dynamic range requirements need to be added.
5. Monitor during low light conditions?
Yes: Monochrome sensor
No: Color Sensor
6. Must the interface be compatible with the default detection system hardware?
Yes: USB2/USB3/Gigabit Ethernet/CSI-2 should be used.
No: FireWire/CoaXPress/Camera Link are optional.
7. Does the monitoring system have a fixed based or rotating base? Is the monitoring area far away or nearby?
Fixed base and nearby: lower resolution and higher angle of view.
Fixed base and far away: higher resolution and higher angle of view.
Rotating base and nearby: lower resolution and lower angle of view.
Rotating base and far away: higher resolution and lower angle of view.
8. Do the implementation have a stage one or stage two pipeline?
Stage one: min 20 fps framerate
Stage two: min 5 fps framerate
Still referring to
The processing module 102 uses the data from the sensor module 116 to detect and recognize objects (birds 144) on the images 122. It can be configured to run several detection algorithms 125, to process data from different types of sensor modules 116 and to demonstrate different types of monitoring and repelling behavior. It contains a logging structure to save detection results and it includes a user interface to adjust the parameters. Lastly, it implements the wildlife detection software, but it can also implement the monitoring behavior, since it is available not only in hardware (normal situation) but also as software plugins which can be loaded if the configuration so requires.
Wildlife control software is implemented on an external agent 146 with which the detection system 102 communicates. The external agent 146 comprises a separate hardware device having its own CPU(s)/GPU(s) 148 running its own software and algorithms for creating new and/or adapting existing repelling and control schemes. The external control agent 146 has its own user input 150, display 152, wired and wireless communication interfaces 154,156 for communicating with the detection and monitoring system 102. See
The hardware of the processing module 102 comprises a Printed Circuit Board (PCB) containing a single or multi-core CPU (or even a multiple CPUs) and as well as one or more GPUs (104) to execute different algorithms 125 for detection (singular, sequentially or in parallel). It has various wired/wireless interfaces 112, 114 for connecting the camera 140, including Gigabit Ethernet, USB3 and MIPI CSI-2. This allows the processing module 102 to be generic and independent of the type of camera 140 that is chosen for the particular situation, as described earlier. For external communication, there is a Universal Asynchronous Receiver/Transmitter (UART) and several General Purpose Input/Outputs (GPIOs) available to support the different modes of operation as will be discussed later. There is an on-board Real Time Clock (RTC) with backup battery for time keeping purposes. This enables the module 102 to keep track of time, even when the power is lost. The correct and up to date time enables the use of scheduled tasks and keeps the timestamp used in all logs correct and in sequence. This enables true and more precise logs for analytics and debugging purposes. Lastly, an HDMI and USB2 interfaces are present for, input, debugging, display and demonstration purposes, enabling human machine interaction (HMI).
Two optional extensions are also possible. Firstly, a wireless connectivity chip can be connected to provide the system with WiFi, 3G or 4G connectivity. Moreover, the Gigabit Ethernet connection is available for connectivity if, for example, a camera was selected using a USB3 interface. This provides the detection system 100 with the possibility to be configured and send logs over a larger distance. Secondly, a storage media (hard drive, SSD storage) 108 is provided for extensive logging and data collection.
The control module 118 generally includes a control processor 158, motor controller(s) or actuator 160, and a repelling device, such as a laser module 120 or audio device (See
Referring to
These modules represent different sections of the system, and representations of these can seen in the system overviews (See
Wildlife Detection Software
The wildlife detection software has multiple goals. Firstly, it is used to count the number of animals and species within an area or within a certain time frame. This can be used for counting as its sole purpose and for the benefit of animal tracking by (government) organizations. Based on this, an objective measure can be made of the animal activity and the potential hazard or nuisance caused by these animals. Secondly, it is used to perform real-time wildlife control, by providing detected animals as a feedback mechanism for wildlife control devices. Thirdly, the detection software can be used to measure the effectiveness of such a wildlife control solution. Based on this, the solution can be optimized to achieve the desired effect. This all is an integrated part of the detection system and each, or multiple, of the goals can be activated during configuration depending on the need at that time, determining the end behavior of the system.
For the first goal, in order to determine the situation at a potential (client) location the detection system can be deployed by itself to paint a picture. This picture can be used as input to further analysis and predictions like estimated savings when using the repellent device or in general give the client a quantified description of the (potential) problem. To be able to provide this, a logging structure is in place, as will be elaborated upon later.
The second goal is the most complicated application and requires communication between the detection software and the wildlife control software (See
For the third goal, the same measurements and logging can be used as for the first goal, but implemented during the wildlife control period. By using a storage media and connectivity extensions, the end user, or an installation consultant (manufacturer), can monitor operation and results and act as a supervisor of the system. It allows the end user and/or consultant to actively review system effectiveness and change the high-level parameters of the wildlife control strategy, based on the measurements, and tune them until the desired effect is achieved. These changes can be made on-the-fly or after retrieval of logs, data and processing thereof. The wildlife detection system acts as a quantified feedback in this case.
Monitoring Strategy
The detection software achieves these goals by implementing a monitoring strategy, that determines how and when to survey different sections of the environment. A monitoring strategy will, depending on the situation, consist out of 1 or more or a combination of the following three survey methods:
1. General survey
2. Waypoint survey
3. Active scan survey
A general survey provides an overview of the environment by capturing one image that covers a part of the surroundings. This strategy is often used for situations where the system either is mounted on a fixed base, or for small areas where one camera can capture everything of importance.
The waypoint strategy uses specific waypoints, where each waypoint provides an overview of a small section of the surroundings. The system stays at each waypoint for a certain timeframe and if nothing of interest is detected or if a certain timeframe has passed the systems moves on to the next waypoint. Effectively scanning and monitoring a larger area, one part of it at a time.
The final survey strategy is the active scan survey, where the surroundings are surveyed by constantly scanning through the environment. How the surroundings are scanned can be configured to best fit the situations. Additionally, the active scan survey can also be used to scan a subsection of the environment, which provides the possibility for the system to create priorities for certain areas that would be classified as hotspots. Additionally, this allows the system to monitor while the platform is rotating, and therefore this survey method can be used while a pattern is displayed by the system. One way of implementing this is by using the waypoint strategy but upon reaching the waypoint immediately move on to the next one.
These surveys are used independently to perform wildlife counting (the first goal as previously mentioned). But note that, when performing wildlife control as well (the second and third goal), this monitoring strategy is mixed with a repelling strategy. These three different surveys can then be used to define strategies that allow the system to determine the most effect repellent strategy. An example strategy would be to use the active scan survey to survey the area. When the system detects an object of interest, the system switches to a general survey and stays in the orientation it was when the object as detected, until the object is deterred. When an object is detected, the current orientation of the system can be stored as a waypoint that the system can return to determine if the object of interest did not return and thus determine that the repellent actions was successful.
Object Detection
The goal of the detection system setup is to be flexible. It should be able to serve the multiple goals that were defined previously, but it should also be able to work in different environments with different cameras and other sensors and at different distances. Therefore, a configuration protocol is provided. This protocol executes upon start-up and configures the detection pipeline, including the models and parameters that are to be used.
Two major configurations exist that can be individually tweaked. The first aims to be deployed in (partly) unknown environments and primarily makes use of unsupervised learning. The second aims for a higher accuracy in known environments and uses supervised learning. Very often, the second follows the first, by simply performing a configuration update once the environment is better understood.
An unknown environment is defined as follows: a type of environment where no previously recorded relevant footage is present in the database. This means a deep neural network is not able to be trained for that particular situation since the required training data has not yet been gathered. It is partly unknown if the expected animals and objects to be encountered are already known. On the contrary, a known environment is one where appropriate footage has previously been obtained for training and has been annotated. This means that the first detection pipeline is often used to efficiently gather the required footage to set up the second detection pipeline.
This configurable two-stage approach is the essence of the whole detection system. It allows us to be flexible and provide value even when the environment is unknown and no model is available of the animals and environment on site. However, thanks to the storage media and connectivity extensions, it is still possible to adapt to the situation and achieve a greater performance in the future.
Detection Pipeline; First Stage
The stage one detection pipeline consists of multiple steps:
1. Background subtraction.
2. Morphological image processing.
3. Object detection.
4. Object suppression.
5. Classification (optional).
6. Flock detection (optional).
The first step is an unsupervised learning method called background subtraction. Using a sequence of successive images, the background subtraction method is able to separate the image in a foreground and a background mask, under the assumption that the background in the images is invariable over a certain amount of time. Specifically, a model is built for each pixel representing the probability of that pixel's value over a certain period of time. If the new value of that pixel differs significantly from the current model, it is signified as belonging to the foreground mask. It is assigned to the background mask otherwise. Specifically, a custom implementation of the Vu Meter background subtraction algorithm is implemented using a non-parametric background model. In this regard, a mask is provided that tells the algorithm where to operate and where not to operate, in order to ignore certain regions of the image. There are several benefits of using this algorithm for wildlife detection.
It is able to detect small changes, but it still produces much less noise than simple frame differencing. This is ideal for long-distant detection, as far away objects appear small in the image.
It is also computationally not very complex and particularly suitable for parallelization. It is therefore implemented in parallel on a GPU using CUDA (CUDA is a trademark of NVIDIA Corp.) and executed using an asynchronous kernel launch. CUDA is a parallel computing platform that lets developers execute code in parallel on a GPU via an API in different languages. Kernels are functions that are executed in parallel on the GPU (the device) and launched via an instruction from the CPU (the host). When launched asynchronously, control returns to the CPU immediately. Because of this implementation, the algorithm can process much larger resolutions in the same time as similar background subtraction algorithms. It allows the system to perform real-time detection on large resolutions. This is ideal for wildlife monitoring, because larger resolutions allow permit the system see more, as discussed previously.
The second step consists of (a) erosion and (b) dilation, two morphological image processing steps in order to decrease the amount of noise. These can be tuned for a particular situation. For example, when noise is expected due to for example moving vegetation, larger kernels are used to filter out most undesired movement in the environment. Otherwise, when less noise is expected because the environment is more static, a smaller kernel is used and can pick up more distant objects. Note that in this context a kernel refers a matrix that determines how the system shrinks (erosion) or grows (dilation) shapes in an image.
The third step is object detection. This is performed via (a) a contour detection algorithm, after which (b) the contours are converted to bounding boxes.
The fourth step is object suppression, consisting of several configurable steps. First, (a) non-maximum suppression merges overlapping bounding boxes. Then, (b) a limit on the object size is posed and (c) on the total number of objects. The latter two steps can be tuned similar to the morphological operations, to allow for more or less distant objects to be detected, in exchange for less or more noise.
For the fifth step, (a) a local search area is generated around the detected objects within which the system (b) performs a sliding window detection scheme. In particular, a Support Vector Machine is used with a Gaussian kernel. It is a binary classification, with the sole purpose to distinguish the expected animal (bird, boar, etcetera) from surroundings (vegetation, clouds, etcetera). Here, the kernel refers to a function that is used to express the similarity between two of its inputs. This step is optional. Only if the environment is partly unknown, a model is trained using data gathered from (online) database in order to enable classification. And only if the expected object distances are not too large, classification is used to separate objects of interest from its surroundings.
The sixth step, firstly (a) performs k-means clustering to find k clusters of detections. Then, (b), a model is selected and fit to a probability distribution around the clusters. This model can be a uniform or a Gaussian distribution. Based on this model, (c) the distance from the cluster center k is determined for each detection. Using the selected model, this distance gives the probability of a detection belonging to the particular cluster. If above a set threshold, the detection is added to the group. Then, (d) the detection is classified as a group if at least n objects belong to the group. This procedure is repeated for different numbers of k. This step is also optional. Only if the expected distances are large, it is used to separate groups of animals from background noise. It allows the system to see more distant objects by relying on the fact that animals often appear in groups.
Referring to
Detection Pipeline; Second Stage
The stage two detection pipeline consists of a single Convolutional Neural Network (CNN) for object detection. It uses a standard network architecture for object detection and it is trained using transfer learning. Specifically, state-of-the-art You Only Look Once (YOLO) object detection architecture, implemented using the darknet framework for neural networks, is utilized.
For transfer learning, at least 1000 images are used for each class that is desired to be detected. Images are obtained from the intended imaging hardware or a better camera set-up at the intended or similar environment. Pre-trained model weights are taken for most of the convolutional layers, and the remaining output layers are trained. The size of the layers is adjusted based on the number of classes that the system is intended to predict.
Ground truth labelling is performed on the level of individual objects, no matter how small. As long as a trained human eye can reasonably well determine the animal species, it is provided with a label. Distinct animal species, and not animal family names are used, based on an available database of species. The human “trainer” is allowed to determine the type of species based on indirect evidence. For example, suppose an unknown species of gull appears in an image. Then, if a certain species of gulls is more common on a geographical location, the gull can be labelled as such.
Additionally, flocks of wildlife can be detected in a similar fashion as during the first detection pipeline. To do so, a sixth step of the stage one detection pipeline is applied on the human labelled data for each species. This allows us to automatically generate labels for groups (flocks) of animal species. This is particularly powerful if, as explained, labelling is performed on even the smallest objects that are able to be distinguished based on direct and indirect evidence. The species' group will then become a separate class in the object detector.
Object Localization
Object localization consists of a three-step approach. The output of each of the steps is used for different wildlife control actions as shown later.
1. Object position in image coordinates.
2. Euclidean distance from camera.
3. Object position in world coordinates.
The first result is obtained from the output of the detection pipeline by taking the center coordinates of the bounding box in the image. If the first detection pipeline is used, the Euclidean distance is determined based on the object size and geometrical properties of the camera and the surroundings. In case of the second detection pipeline, the Euclidean distance is computed using nonlinear regression, conditional on the object class. It takes the bounding box size and the object class as inputs and predicts the Euclidean distance. The third result is obtained via forward kinematics, using the known current orientation of the camera.
Communication
The information that is passed from the detection software to the control module is based on results from the algorithms. When an algorithm identifies an object of interest, different information about that object is gathered. The system can be configured such that each type of information can result in a trigger towards an external agent. The triggers have a certain priority over others triggers. Based on these priorities, the final implemented strategy is determined by the wildlife control software. The detection software can provide the following triggers:
Object detected (yes/no)
Object class (species or other object details)
Object number (number of objects)
Object location on image (pixel location)
Object location on image (object distance)
Object location on the monitoring area with respect to monitoring system (x,y)
Object is moving/stationary
Object direction of movement
Object predicted trajectory
Object past path/trajectory
Event based on previous triggers, combination of triggers or recurring triggers.
Wildlife Control Software
For wildlife control, a strategy needs to be implemented based on the environment the system operates in. The prior art WO2015093938 describes an approach where the system is manually taught a sequence of waypoints that constitute a pattern which is executed during pre-programmed (set) times of the day.
For this detection system a control strategy can be implemented either manually or automatically. The manual implementation is still based on prior art, while the automated strategy adds to this by adapting the manually implemented waypoints or implementing a more complex method.
Adapting Manual Strategies
Overall, a large number of strategies can be defined by combining different actions to different triggers. This can be done manually upfront. However, thanks to the logging and analysis approach, as will be explained in the next section, selecting the strategy is particularly effective to do periodically. It allows for a large degree of customization, tailored for the particular customer. Below are several examples.
Playing a laser pattern in a hotspot area more often than in other areas.
Playing a more intense laser pattern with more waypoints in the hot spot areas.
Playing only laser patterns in the hot spot areas and disregarding other areas.
Playing the most efficient pattern at night, or in situation with bad visibility when the camera cannot detect, based on bird presence hotspots recorded during the daytime.
The automated system can adapt the manual installed pattern of waypoints using triggers from the detection software in addition to data from the logging analysis in one of the following ways:
Adapting Waypoints
Determine the pattern that covers the location of the object of interest and play it
Adapt the waypoints of a pattern to the situation based on recurring locations.
Rearrange the order of the waypoints.
Delete unnecessary waypoints.
Add new waypoints.
Adapting timeslots
Add new timeslot to existing pattern.
Remove time slot from existing pattern.
Change time slot duration.
Custom Automated Strategies
In addition to the manually installed control strategy the automated wildlife control software is also capable of generating more complicated strategies.
1. Generate pattern based on one object location.
2. Generate pattern based on multiple object locations.
3. Generate pattern conditional on object class (species, for example).
4. Target objects directly, one at a time until object has been deterred.
5. Target objects by swiping from different directions.
6. Vary laser intensity.
7. Vary projection speed, color, and strobe pattern.
Note that the object location can be expressed either in pixel coordinates, distance or world coordinates.
Each of these strategies basically uses one parameter of either a pattern, object information or other system parameters. Additionally, the generated patterns look very different from the manually implemented patterns. The manual patterns aim to most effectively cover the environment due to the lack of feedback, however, the automated patterns have a different goal. The goal for the automated patterns is to most effectively deter the object of interest. Strategies 1 through 4 generate custom patterns and can look very different each time a pattern is generated. They can even consist of several smaller patterns that converge around a location (example 1), or a pattern that covers an identified hotspot and generates patterns based on the identified class of the object (example 2)
The detection pipeline provides a location trigger and the system 100 is configured to target an object 144 directly and create any number of custom swipes 170 from any direction towards the location of the detected object 144.
Because birds respond to the approaching laser projection, it should be clearly visible, and it should be clearly approaching. These patterns are parametrized as shown in
The detection pipeline provides a class trigger (as an example a seagull 144) on a location identified as a hotspot 172 and the system 100 was configured to create a custom pattern with different laser intensity, due to it being a hotspot.
Due to the class being a seagull it is necessary to increase the laser intensity. Additionally, a custom pattern that covers the hotspot will be played, the hotspot area is bounded by the outer box in the image below.
For the lines 1-12 the laser intensity will be increased each time it moves on to the next line. This will cause the intensity to be highest when the laser reaches the location of the detected object.
Other Implementations
Because of the comprehensive set of triggers, it is also possible to determine repelling strategies based on a specific target, such that each individual bird will be targeted by the laser repelling device with a strategy that is known to be most effective. The pattern which is most effective depends on the species, time of day, season and geographical location and is left by a bird deterrent expert to decide and configure. The device is then able to verify if the dispersal strategy was effective and continue targeting the bird until it has been dispersed. With direct feedback of the camera images, the system can determine the effectiveness of many different dispersal strategies. The system can give objective measurements of animal activity, allowing verification of different strategies for different species. Even environmental factors or seasonal factors can be taken into account, for example bird migration season or mating season or time of the day.
This means it is also possible to target only specific species or avoid specific species. For example, on a poultry farm it is very unwelcome to have migrating birds nearby, because they can carry all kinds of unwanted disease like avian influenza. This potentially means the whole farm can get infected, with the consequence of having to kill and destroy all the poultry at the farm. Obviously, this has a huge impact on the farming operation, therefore most farmers put a lot of effort in preventing their poultry from getting infected. If the system is installed to be targeting all birds except for the chickens, this would have a huge benefit for the poultry farmer.
Once an object is detected, it is important to take the correct action. Once an animal that is on the “protected” list is recognized, the system cannot activate the laser deterrent. Recognizing a bird or other animal that is on the “wanted” list results in the laser deterrent activating and going into repelling mode. If a situation occurs where both are recognized (an object on the “protected” list and an object on the “wanted” list), the system will look at the priorities of the recognitions. These priorities are always in favor of protecting whatever is on the “protected” list, and thus the laser deterrent will not activate.
Logging and Analysis
Because the many captured images together take up a lot of digital storage space, it is important to have an efficient data handling and storage system. The stored files then need to be processed to improve the detection system. This allows for a feedback loop towards the detection and repelling strategies.
Logging Approach
The primary goal of the logging structure is to get a measure of the animal activity in the environment. This requires data about which type of objects are detected at what time and on which location. Furthermore, for efficiently building a database using the first detection pipeline, storage is required for the recorded images. As a secondary goal, the detection performance can also be evaluated.
To achieve both goals, the logging is divided in two states: a state when an object of interest is detected (indicated in lighter gray) 174 and a state when no object of interest is detected (indicated in darker gray) 176 (See
Object Detected:
a. Sample data at random with sampling rate S1.
b. Sample images conditionally, according to S1*S2.
No object detected:
a. Sample data at random with sampling rate S3.
b. Sample images conditionally, according to S3*S4.
Where for example S1=1/100, S3=1/10000, and S2=S4=1. This allows the sample rate of images to be set relatively high during the first detection pipeline when the database is being built for the tailor-made model, and turn it down afterwards.
Analysis Approach
The data and image logs are post-processed externally, after retrieving data from the hard drive extension or after receiving the data via the connectivity extension. Note that, if the logs come from the first detection pipeline, the second pipeline needs to first be trained and then used to process the logs. Then, because the number of objects is known as well as their type and their location, two main components can be generated for analysis.
1. Wildlife activity timelines.
2. Wildlife activity heat maps.
Wildlife Activity Timelines
Wildlife activity timelines express the number of birds averaged over a time interval. The time interval can any time interval which suits the reporting purpose. Additionally, several different time intervals can be compared to one another to determine the effect of a certain strategy or change in strategy. The timelines also provide a tool to represent the wildlife activity around the system for human interpretation.
Wildlife Activity Heatmap
Wildlife activity heat maps visualize the amount of wildlife active on different locations, averaged over a time interval.
Illustrated in
As can be understood from the above, this sampled way of logging creates a very flexible way of generating graphs over different time intervals and for creating databases for supervised learning. It thus has several benefits.
1. Provide feedback about the wildlife control effectiveness.
2. Provide feedback towards the customer about the wildlife problem.
3. Provide feedback about the detection algorithm performance.
Feedback can be used to improve the wildlife control (1) as has been thoroughly discussed. This can include teaching new patterns and setting new time slots manually or adjusting triggers and related strategies for automated wildlife control. With this data from the device it is also possible to create a very specific repelling strategy, which may also include other dispersal methods, like a net, bird spikes, chemical deterrents or sound canons. The detection system can be of great benefit for making a good bird dispersal plan for the customer.
Regarding feedback towards the customer (2), basically every customer is interested in the effectiveness of the device. Currently, there are no methods to determine repelling effectiveness apart from measuring the change in bird damage, which is an indirect way of measuring repelling effectiveness as damage reduction could also be the result of other causes (e.g. reduced crop losses could also be the result of improved harvesting techniques, better climatic conditions for crop growing than previous season or less birds were present in the area to begin with) and is always after the fact when it's too late for adjustments. With a detection system on the laser or light based repelling device, it is possible to count the birds before, during and after laser projection. With this data, the customer knows exactly how effective the device is. At the same time this can be uploaded to a manufacturer database system in order to improve future installations, as well as gain knowledge about the system as a whole which can facilitate improvement in future products. It is also possible to filter the birds on species, so it can be verified if the device is effective on a specific type of bird. The device can also generate a hot spot image to show the effectiveness for a certain area.
The feedback about the algorithm performance (3) is purely used for internal evaluation. By sampling both times when wildlife is detected and when no wildlife is detected, an objective overview of the predicted positives and predicted negatives is obtained. By giving a subset of this data ground truth labels, a confusion matrix for evaluating the machine learning algorithms for performance improvement can be created.
System Configurations
The detection system can consist of several different configurations to make sure that it is suitable for different situations. These configurations can be hardware wise, as will be discussed below, or can be implemented in its software configuration. As laid out above, wildlife can be detected with this system but this does not mean it is limited to only wildlife. Humans, for example, can also be detected if the models are trained to do so. Actually, any kind of object can be detected when properly trained and set up for that purpose. This means that for example a human walking in the field or driving a tractor, truck or car, even a bicycle can be detected. While this could be used for its own sake, this also opens up an interesting opportunity in combination with a repelling strategy. For example, the laser can be temporarily disabled while there is an object in the view (human, car, other object) but keep moving as normal. Or, when an object on the ‘forbidden list’ is detected, the laser can be disabled if it is in close proximity (˜1 meter) from the object. This is a kind of ‘obstacle avoidance’. Another class of object here can be reflecting surfaces. For example, an installation on a roof or an area with water. This would enable a repelling strategy without taking into account those objects, making and installation easier, and using the detection system to adapt to the situation at hand. This is especially useful in an environment where the surroundings have changed, for example there was a lot of rain and a puddle of water or even a flooded area arose.
Integrated Detection System
Referring now to
Separate Detection System
The detection module 102 can also be included into a separate housing on its own (rotating or fixed) base. This provides the system 100 with access to a different, more convenient view point compared to the rest of the installation. In this case, the detection system 102 implements the detection software and logging, as well as the monitoring strategy when on a rotating base (See unbounded systems in
Independent System (No Control Processing System)
Referring back to
Two Separate Systems
Referring back to
Multiple Detection Systems
The separate housing configuration also provides the possibility to connect several imaging modules 116 to one system 100. The system 100 can then determine what is the most important image and based on this adjust its machine behavior accordingly. Or, they can be independent systems with the sole purpose of creating an assessment of the wildlife status in the current environment creating the possibility of a continuous full coverage with fixed based systems.
System Implementation
Because of the flexibility of the system 100, a process is required for implementing it at a customer location. The most important aspect of the detection system is to be able to handle both known and unknown environments, by selecting one of two detection pipelines. In case of an unknown environment, the system can be put into operation under certain conditions, after which it can be upgraded once enough logs have been gathered. The steps for selecting components and installing a single detection system are discussed below.
1. Is automated wildlife control needed?
Yes: assemble an integrated detection system, or connect a separate detection system to the main system, go to 2.
No: assemble a separate detection system, got to 2.
2. Do the complete surroundings need to be monitored?
Yes: assemble on a rotating base, go to 3.
No: assemble on a fixed base, go to 3.
3. Does a dataset of annotated images exist of a similar environment?
Yes: configure the second detection pipeline, go to 7.
No: configure the first detection pipeline, go to 4.
4. Is the environment largely static?
Yes: go to step 5.
No: install the system at the customer location. Optimize system parameters for logging more objects more frequently, with the risk of detecting more noise, go to 6.
5. Is automated wildlife control already needed?
Yes: install the system at the customer location. Optimize system parameters for logging more frequently, go to 6.
No: install the system at the customer location. Optimize system parameters for logging more objects more frequently, with the risk of detecting more noise, go to 6.
6. Install the detection system for a determined amount of time to gather a database. Retrieve the logs via the storage extension or via the connectivity extension. Annotate the data and perform transfer learning according to the training method described. Push the updated model and reconfigure the system with the second detection pipeline, either physically via a second installation or via the connectivity extension. Go to step 7.
7. Install the detection system for operation.
8. Update periodically based on the system logs.
Step 6 is an essential step. This procedure allows the system to be upgraded from the first stage to the second stage detection pipeline. During the training period, the installer/end user can in parallel analyze the first batch of gathered log data. When pushing the new configuration, the repelling strategies can be additionally updated manually if so desired. See adapting manual strategies for details.
Having thus described certain particular embodiments of the invention, it is understood that the invention defined by the appended claims is not to be limited by particular details set forth in the above description, as many apparent variations thereof are contemplated. Rather, the invention is limited only be the appended claims which include within their scope all equivalent devices or methods which operate according to the principles of the invention as described.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7656300 | Ronnau | Feb 2010 | B2 |
| 8164462 | Bose et al. | Apr 2012 | B1 |
| 9519962 | Normand | Dec 2016 | B1 |
| 9563852 | Wiles et al. | Feb 2017 | B1 |
| 10152035 | Reid et al. | Dec 2018 | B2 |
| 10282852 | Buibas et al. | May 2019 | B1 |
| 10452980 | Kim et al. | Oct 2019 | B1 |
| 10614736 | Von Droogenbroeck et al. | Apr 2020 | B2 |
| 10694737 | Tillotson | Jun 2020 | B2 |
| 10796141 | Shepherd, Jr. | Oct 2020 | B1 |
| 20040141635 | Liang | Jul 2004 | A1 |
| 20060126918 | Oohashi et al. | Jun 2006 | A1 |
| 20090066817 | Sakamaki | Mar 2009 | A1 |
| 20110054691 | Lee et al. | Mar 2011 | A1 |
| 20110116682 | Wang et al. | May 2011 | A1 |
| 20110144829 | Kim et al. | Jun 2011 | A1 |
| 20120066640 | Kwak et al. | Mar 2012 | A1 |
| 20120093362 | Liu | Apr 2012 | A1 |
| 20120195471 | Newcombe et al. | Aug 2012 | A1 |
| 20130064432 | Banhazi et al. | Mar 2013 | A1 |
| 20130128034 | Carr | May 2013 | A1 |
| 20130249218 | Vassilev et al. | Sep 2013 | A1 |
| 20140144390 | Duncan et al. | May 2014 | A1 |
| 20140270378 | Aimura | Sep 2014 | A1 |
| 20140285667 | Aimura | Sep 2014 | A1 |
| 20150049941 | Hall | Feb 2015 | A1 |
| 20150379740 | Yang et al. | Dec 2015 | A1 |
| 20160055400 | Jorquera et al. | Feb 2016 | A1 |
| 20170172136 | LaRue | Jun 2017 | A1 |
| 20180107904 | Yang Mao et al. | Apr 2018 | A1 |
| 20180114068 | Balasundaram et al. | Apr 2018 | A1 |
| 20180125058 | Liu | May 2018 | A1 |
| 20180220641 | Read | Aug 2018 | A1 |
| 20190110444 | Boehm | Apr 2019 | A1 |
| 20190152595 | Mitchell et al. | May 2019 | A1 |
| 20190236614 | Burgin et al. | Aug 2019 | A1 |
| 20200323193 | King | Oct 2020 | A1 |
| 20200380678 | Sugaya | Dec 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 102004924 | Apr 2011 | CN |
| 106875398 | Jun 2017 | CN |
| 107886086 | Apr 2018 | CN |
| 109997834 | Jul 2019 | CN |
| 110063326 | Jul 2019 | CN |
| 2555836 | May 2018 | GB |
| 2014185780 | Nov 2014 | WO |
| 2015093938 | Jun 2015 | WO |
| Entry |
|---|
| Chernbumroong, S., et al., Measuring of Background Modeling and Subtraction Algorithms on Moving Object Detection in Video Sequences in Chiangmai, ARPN Journal of Engineering and Applied Sciences, vol. 12, No. 11, Jun. 2017. |
| Number | Date | Country | |
|---|---|---|---|
| 20210368776 A1 | Dec 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16682268 | Nov 2019 | US |
| Child | 17404187 | US |
