Patent Grant
12307746
This application claims priority of Chinese Patent Application No. 202410099370.9, filed on Jan. 24, 2024, the entire contents of which are incorporated herein by reference.
The disclosure relates to the technical field of computer vision object tracking, and in particular to a nighttime unmanned aerial vehicle (UAV) object tracking method fusing a hybrid attention mechanism.
In recent years, with small size, flexible action, simple operation and other advantages, UAV is increasingly applied in many fields including civil, military, and science. For instance, UAV has important research significance in disaster rescue and relief under extreme weather, enemy information collection in battlefield, intelligent monitoring in agriculture, and object recognition and tracking from a UAV perspective. UAV nighttime object tracking is classified into a branch of general object tracking, with its task involving that, an object, after being designated in the first frame of a video sequence, is continuously tracked in subsequent frames. Abounding box (typically represented by a rectangular box) is utilized to target the object, achieving positioning and scale estimation of the object.
Object tracking methods are mainly classified into two categories: one category is based on correlation filtering method and another category is based on deep learning method.
Correlation filtering method was first applied in the field of computer vision in the 1980s, which was introduced into the field of object tracking by Bolme et al. in 2010 for the first time, utilizing the Fast Fourier Transform (FFT) and the convolution theorem to propose the Minimum Output Sum of Squared Error (MOSSE) algorithm, which determines the position of the object by calculating the maximum response value in the candidate regions. However, relying on manual design, this method is greatly limited when dealing with object whose appearance deforms.
With the rapid advancement of deep learning, deep neural networks have been widely applied in the field of computer vision for learning image features from large-scale data. Siamese networks have become a hotspot in object tracking due to their advantages of fewer parameters and faster computation. In Siamese fully convolutional network (SiamFC), Siamese networks are firstly introduced into the field of object tracking. Inspired by Faster region-based convolutional neural networks (Faster RCNN), Siam region proposal network (SiamRPN) is obtained by incorporating RPN into the Siamese network framework, improving the robustness of tracker. Deep residual networks (ResNet) is introduced into SiamRPN++, significantly enhancing tracking performance. To avoid the complex subsequent calculations involved in anchor box design, some researchers have introduced the anchor-free concept, and anchor-free methods are proposed in HiFT and SiamAPN++ to distinguish between objects and backgrounds. However, most of these algorithms are designed for daytime tracking scenarios, and the different distribution of nighttime image features significantly impairs the nighttime UAV tracking performance.
In response to the above deficiencies existing in prior art, an objective of the disclosure is to provide a nighttime UAV object tracking method fusing a hybrid attention mechanism, aiming to improve the positioning accuracy of nighttime UAV object tracking and reduce the failure probability of object tracking by UAV in nighttime scenarios.
To solve the above technical problems, the disclosure adopts the following technical solutions.
A nighttime UAV object tracking method fusing a hybrid attention mechanism includes: acquiring a night vision video sequence from a UAV, inputting a nighttime image frame of the night vision video sequence into a pre-trained night vision image enhancement model, to obtain a corresponding enhanced nighttime image frame, and performing image object tracking and recognition on the enhanced nighttime image frame, to obtain an object tracking and recognition result;
In a preferred solution, the encoder module includes four convolutional layers and four downsampling layers connected in series, for performing convolutional processing and pooling processing on the inputted nighttime image frame in sequence, to obtain the initial convolutional feature map of the nighttime image frame after convolutional processing;
In a preferred solution, the spatial hybrid attention module includes a first normalization layer (LayerNorm1), a spatial attention module unit (SAM), a first window multi-head self-attention unit (W-MSA1), a second normalization layer (LayerNorm2), and a first residual feedforward network layer (ResFFN1); an input feature map of the spatial hybrid attention module is transmitted to the LayerNorm1, and an output of the LayerNorm1 is connected to an input of the SAM and an input of the W-MSA1, respectively; an output of the SAM and an output of the W-MSA1 are added to an input feature of the spatial hybrid attention module, to form an intermediate feature map; and then an intermediate feature, after being sequentially processed by the LayerNorm2 and the ResFFN1, is added to the intermediate feature map, to obtain the spatial attention feature map, which serves as an output of the spatial hybrid attention module;
In a preferred solution, the SAM is configured to perform a convolution operation of channel compression and a Gaussian error linear unit (GELU) activation function operation on an input feature map FN, followed by a convolution operation of channel restoration, and then the input feature map FN after those operations is inputted into a spatial attention layer; and in the spatial attention layer, the input feature map FN is subjected to max pooling and average pooling operations, respectively, followed by a Concat operation to aggregate vectors, and finally a two-dimensional (2D) convolution operation and a sigmoid activation function operation are performed to obtain a result, which is then multiplied by the input feature map FN, to obtain an output result; a processing procedure of the SAM being expressed as:
FO=ConvU(GELU[ConvD(FN)]);
SAM(FN)=sigmoid(Conv([AvgPool(FO);MaxPool(FO)]))×FO;
In a preferred solution, the channel hybrid attention module includes a third normalization layer (LayerNorm3), a channel attention block unit (CAB), a second window multi-head self-attention unit (W-MSA2), a fourth normalization layer (LayerNorm4), and a second residual feedforward network layer (ResFFN2); an input feature map of a hybrid attention block (HAB) is transmitted to the LayerNorm3, and an output of the LayerNorm3 is connected to an input of the CAB and an input of the W-MSA2, respectively; an output of the CAB and an output of the W-MSA2 are added to the input feature map of the HAB, to form an intermediate feature map; and an intermediate feature, after being sequentially processed by the LayerNorm4 and the ResFFN2, is added to the intermediate feature map, to obtain a channel attention feature map, which serves as an output of the channel hybrid attention module;
In a preferred solution, the CAB is configured to perform a convolution operation of channel compression and a GELU activation function operation on the input feature map, followed by a convolution operation of channel restoration, and then the input feature map after those operations is inputted into the channel attention layer; and in the channel attention layer, the input feature map is subjected to adaptive average pooling, followed by channel compression processing and ReLU activation function operation, and then channel restoration processing and sigmoid function operation are performed to obtain a result, which is then multiplied by the feature map inputted into the channel attention layer, to obtain an output result; a processing procedure of the CAB being expressed as:
FB=ConvU(GELU[ConvD(FP)]);
CAB(FP)=HGP(FB)·sigmoid[WU(δ(WD(FB)))]×FB;
In a preferred solution, the decoder module includes four convolutional layers and four upsampling layers connected in series, for performing convolutional processing and upsampling deconvolution processing on the inputted hybrid attention feature map in sequence, followed by hyperbolic tangent conversion processing, to obtain the curve estimation parameter map; a processing procedure of the decoder module being expressed as:
Fde1=Up(Conv(FCHA)de1)de1;
Fde2=Up(Conv(Fde1)de2)de2;
Fde3=Up(Conv(Fde2)de3)de3;
Fde4=Up(Conv(Fde3)de4)de4;
Fde=tanh(Conv(Fde4)de);
In a preferred solution, the curve projection module is configured to perform pixel-by-pixel multiplication using the curve estimation parameter map and the original nighttime image frame, and a result is mapped to the nighttime image frame through curve projection, forming the intermediate feature image;
In a preferred solution, the denoising processing module includes 19 convolutional activation units connected in series and a hyperbolic tangent branch unit, with each convolutional activation unit including a 2D convolutional layer and a ReLU activation function layer; a processing procedure of the denoising processing module being expressed as:
Xout=X′+tanh(W(X′)19);
In a preferred solution, the night vision image enhancement model can be trained as follows:
Compared with the prior art, the disclosure has the following beneficial effects:
1. According to the nighttime UAV object tracking method fusing a hybrid attention mechanism proposed by the disclosure, the night vision image enhancement model is utilized to enhance various night image frames in the night vision video sequence of UAV before performing image object tracking and recognition, to improve the accuracy of object tracking for nighttime images. Specifically, the enhancement processing on night image frames in the night vision video sequence by the night vision image enhancement model is regarded as a curve projection task, and the red, green, and blue channel values are adjusted to improve image quality. With the aid of the spatial channel hybrid attention mechanism, the model can adaptively learn spatial channel features, enhancing generalization ability, and allowing the network model to focus on more effective features. Additionally, the denoising processing module is introduced to remove noise hidden in nighttime images, thereby improving the object positioning accuracy and tracking performance of UAV in nighttime scenarios, and reducing the failure probability of object tracking by UAV in nighttime scenarios.
2. According to the nighttime UAV object tracking method fusing a hybrid attention mechanism proposed by the disclosure, after enhancing night image frames of the night vision video sequence using the night vision image enhancement model, various object tracking algorithms can be broadly combined and applied to execute nighttime UAV object tracking tasks, significantly enhancing UAV tracking performance in nighttime scenarios, and showing great value for technical application and promotion.
For clearer objective, technical solutions and advantages of the disclosure, the technical solutions of the examples in the present resourceful will be described clearly and completely by reference to the accompanying drawings of the examples in the disclosure below. Obviously, the examples described are only some, rather than all examples of the disclosure. The components of examples of the disclosure generally described and illustrated in the accompanying drawings herein may be arranged and designed in a variety of different configurations. Therefore, the detailed description of the examples of the disclosure provided in the accompanying drawings is not intended to limit the scope claimed by the disclosure, but merely to represent selected examples of the disclosure. On the basis of the examples of the present resourceful, all other examples obtained by those ordinary skilled in the art without creative efforts fall within the scope of protection of the disclosure.
The specific embodiment of the disclosure is described further with reference to the specific examples.
In response to the issue of poor positioning accuracy and performance of UAV executing object tracking tasks in nighttime scenarios in prior art, the disclosure provides a nighttime UAV object tracking method fusing a hybrid attention mechanism. The overall flow of this method is shown in
The night vision image enhancement model of the disclosure employs a deep learning network model that integrates a hybrid attention mechanism, and can be abbreviated as SCHAB-Net, with the processing flow shown in
Through
The night vision image enhancement model in the solution of the disclosure is described in detail below.
Night Vision Image Enhancement Model SCHAB-Net
The night vision image enhancement model SCHAB-Net adopted in the disclosure is a deep learning network model fusing a hybrid attention mechanism, which is primarily configured to enhance nighttime image frames from UAV night vision video sequences, thereby helping to improve the object tracking performance of UAV in nighttime scenarios, and improving the image quality of the input nighttime image after undergoing processing through the deep learning network. The network structure of the night vision image enhancement model SCHAB-Net is shown in
Encoder Module
The encoder module includes four convolutional layers and four downsampling layers connected in series, for performing convolutional processing and pooling processing on the inputted nighttime image frame in sequence, to obtain the initial convolutional feature map of the nighttime image frame after convolutional processing. In specific implementations, four 2D convolutional units and four pooling layers can be selected as the four convolutional layers and four downsampling layers in the encoder module, labeled as Encoder layers 1 to 4 in
A processing procedure for extracting the initial convolutional feature map of the nighttime image frame by the encoder module is expressed as:
Fen1=Pool(Conv(X)en1)en1;
Fen2=Pool(Conv(Fen1)en2)en2;
Fen3=Pool(Conv(Fen2)en3)en3;
Fen=Pool(Conv(Fen3)en4)en4;
The spatial hybrid attention module is labeled as a spatial hybrid attention layer in
A processing procedure for extracting the spatial attention feature map of the nighttime image frame by the spatial hybrid attention layer is expressed as:
FN=LN1(Fen);
FMΔ1SAM(FN)+W-MSA1(FN)+Fen;
FSHA=ResFFN1(LN2(FM))+FM;
The ResFFN1 is a residual feedforward network, which is a network unit commonly used in deep learning networks, with a brief introduction provided herein. As shown in
SAM
In the spatial attention hybrid module, the SAM includes a set of Conv-GELU-Conv layers and a spatial attention layer, as shown in
A processing procedure of SAM is expressed as:
FO=ConvU(GELU[ConvD(FN)]);
SAM(FN)=sigmoid(Conv([AvgPool(FO);MaxPool(FO)]))×FO;
The W-MSA is a network unit commonly applied in deep learning networks, with a brief introduction provided herein. The W-MSA directly obtains the input for each single-head attention head by evenly slicing the input feature map. After each single-head attention head is processed, Concat aggregation operation is performed.
Channel Hybrid Attention Module
The channel hybrid attention module is labeled as channel hybrid attention layer in
A processing procedure for extracting the hybrid attention feature map of the nighttime image frame by the channel hybrid attention layer is expressed as:
F=LN3(FSHA);
FQ=Δ2CAB(FP)+W-MSA2(FP)+FSHA;
FCHA=ResFFN2(LN4(FQ))+FQ;
In the channel attention hybrid module, the CAB includes a set of Conv-GELU-Conv layers and a channel attention layer, as shown in
A processing procedure of the CAB is expressed as:
FB=ConvU(GELU[ConvD(FP)]);
CAB(FP)=HGP(FB)·sigmoid[WU(δ(WD(FB)))]×FB;
The decoder module includes four convolutional layers and four upsampling layers connected in series, labeled as Layers 5 to 8 in
A processing procedure of the decoder module is expressed as:
Fde1=Up(Conv(FCHA)de1)de1;
Fde2=Up(Conv(Fde1)de2)de2;
Fde3=Up(Conv(Fde2)de3)de3;
Fde4=Up(Conv(Fde3)de4)de4;
Fde=tanh(Conv(Fde4)de);
The curve projection module is configured to perform pixel-by-pixel multiplication using the curve estimation parameter map and the original nighttime image frame, and a result is mapped to the nighttime image frame in a curve projection manner, forming an intermediate feature image.
A processing procedure of the curve projection module is expressed as:
X′=X+Fde⊙(1−X);
The Denoise leverages residual learning to accelerate the training process and enhance denoising performance. The structure of Denoise shown in
A processing procedure of the denoising processing module is expressed as:
Xout=X′+tanh(W(X′)19);
After the night vision image enhancement model enhances the nighttime image frames in the night vision video sequence, the object tracking algorithm currently employed by UAV for image object tracking tasks can be utilized to perform tracking and recognition of nighttime image objects, obtaining the object tracking and recognition results. During nighttime aerial photography with a UAV, any object can be designated for continuous tracking, and a bounding box can be used to mark the identified tracking object.
A procedure of the image object tracking task can be expressed as:
B=ϕ(Z;SCTHAB(X))
From the above introduction, it is evident that in the nighttime UAV object tracking method fusing a hybrid attention mechanism proposed in the disclosure, the night vision image enhancement model is employed to enhance various nighttime image frames in the night vision video sequence of the UAV, followed by performing image object tracking and recognition, to improve the accuracy of object tracking in nighttime images. Specifically, the enhancement processing on night image frames in the night vision video sequence by the night vision image enhancement model is regarded as a curve projection task, and the red, green, and blue channel values are adjusted to improve image quality. With the aid of the spatial channel hybrid attention mechanism, the model can adaptively learn spatial channel features, enhancing generalization ability, and allowing the network model to focus on more effective features. Additionally, the denoising processing module is introduced to remove noise hidden in nighttime images, thereby improving the object positioning accuracy and tracking performance of UAV in nighttime scenarios, and reducing the failure probability of object tracking in nighttime scenarios.
Next, the training of the night vision image enhancement model in the disclosure is described.
Training of Night Vision Image Enhancement Model
Specifically, the night vision image enhancement model can be trained as follows.
Pre-completed night vision image and daytime images are taken as paired training samples, to constitute a training sample set, which serves as an input of the night vision image enhancement model. A training loss is calculated according to the obtained enhanced night vision image and the corresponding daytime image of the training sample. Parameters of the night vision image enhancement model are optimized according to the training loss until the night vision image enhancement model converges, thereby obtaining a trained night vision image enhancement model.
During the training of the night vision image enhancement model, the training loss is calculated using the following formula:
In the disclosure, the training loss is calculated though the above formula, which is evaluated by comparing the differences between the feature maps extracted from the 3rd to 5th layers of the AlexNet network for the night image frames enhanced by the model and the daytime image frames in the paired training samples. This can effectively ensure the training gradient and avoid overly complex training flow and comparison operation flow, thereby favoring better robustness and efficiency in model training.
Solutions of the disclosure are described further in detail by reference to the example below.
In this example, an SCHAB-Net night vision image enhancement model is trained using 485 normal/low-light image pairs from an LOL dataset, and with the public test UAVDarkl35 as the evaluation benchmark, the performance of the algorithm is assessed through Success and Precision. In this example, the night vision image enhancement model SCTHAB-Net in the disclosure is combined with the SiamAPN++ and SiamRPN UAV tracking algorithms, with the comparison results of algorithms shown in Table 1. SiamAPN++ is selected as a UAV nighttime tracker, and in terms of object tracking and recognition, the night vision image enhancement model SCTHAB-Net of the disclosure is compared with the SCT algorithm and DCE++ night enhancement algorithm in prior art, with the object tracking and recognition results shown in Table 2, and the UAV object tracking results shown in
The above experimental results show that, in this example, the nighttime UAV object tracking method of the disclosure employing the night vision image enhancement model shows better object positioning accuracy and tracking performance, and the failure probability of object tracking is significantly reduced.
Finally, it is to be noted that: the example described above is merely used to illustrate rather than limiting the technical solutions of the disclosure. It is to be understood by those ordinary skilled in the art that: modifications or equivalent replacements made to the technical solutions of the disclosure without departing from the principle and scope of the technical solutions of the disclosure are included in the scope of claims of the disclosure.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202410099370.9 | Jan 2024 | CN | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 11810366 | Wang et al. | Nov 2023 | B1 |
| Number | Date | Country |
|---|---|---|
| 115456884 | Dec 2022 | CN |
| 116543021 | Aug 2023 | CN |
| 117011194 | Nov 2023 | CN |
| 117115616 | Nov 2023 | CN |
| Entry |
|---|
| Ye et al, Tracker Meets Night: A Transformer Enhancer for UAV Tracking, 2022, IEEE Robotics and Automation Letters, 2(7): 3866-3873. (Year: 2022). |
| Xi et al, Detection-Driven Exposure-Correction Network for Nighttime Drone-View Object Detection, 2024, IEEE Transactions on Geoscience and Remote Sensing, 62 (2024): 1-14. (Year: 2024). |
| He et al, REUT: A Retinex-inspired Low-light Image Enhancer for UAV Tracking at Night, 2022, IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles, pp. 1-7. (Year: 2022). |
