The present disclosure belongs to the technical field of recognition of a time sequence image of a concrete dam defect, and particularly relates to a defect recognition method based on a time sequence difference and a self-attention mechanism.
In the field of constructional engineering, inspection items or inspection points that do not satisfy requirements of engineering construction quality are defined as defects. With long-term operation of hydropower station dams, material aging, environmental impact and other factors have led to formation of defects to varying degrees. If a degree of defects is low, corresponding measures can be taken to deal with in time to meet the bearing requirements of the structures. Upon the defects are not treated and repaired in time, operational safety of dams will be threatened seriously. Therefore, utilizing an automatic patrol and inspection device to detect and eliminate defects timely can maintain the structural safety of dams effectively.
In an actual patrol and inspection scene of hydropower dams, data acquired by unmanned aerial vehicles, mobile cameras and other devices are all composed of videos. In the process of acquiring and transmitting, the videos will be compressed and encoded in order to save cost, resulting in that the model cannot directly process the video data. The videos thereby need to be converted into a time-dimensional image sequence, and the defects are located and recognized by extracting feature information from the time sequence image. While an actual acquired time sequence image often contains a large number of background frames independent of the defects, so it is difficult to recognize the whole image sequence directly.
An objective of the present disclosure: according to the above analysis of the prior art, a model needs to pay attention to a contextual feature relation of an image sequence so as to ensure integrity of defect extraction and recognition accuracy in a task of defect recognition of a time sequence image. In order to rapidly recognize the time sequence image of a dam defect collected by an automatic device, the present disclosure provides an intelligent recognition method for a time sequence image of a concrete dam defect.
Technical solutions: an intelligent recognition method for a time sequence image of a concrete dam defect includes extracting a feature sequence of a time sequence image containing a dam defect by using a two-stream network, and adding a time-dimensional self-attention mechanism so as to obtain a global feature relation; matching, in a training process of the model, a located defect with a real defect by using an objective function based on a distance intersection-over-union, and computing a time sequence position relation of a defect so as to accelerate a model convergence rate; adding a loss term based on a tight sensing intersection-over-union to a loss function of the model, so as to pay attention to integrity of a defect sequence and improve accuracy; and after completing defect location, extracting a defect feature by using a convolutional neural network based on a two dimensional (2D) time sequence difference and recognizing a defect type.
The specific steps are as follows:
(1) designing a defect location model for a time sequence image characteristic containing the dam defect; extracting, by the model, a time sequence feature by using a two-stream network and a Transformer network, specifically, extracting the image feature by using the two-stream network, and adding a time-dimensional self-attention mechanism to an image frame through the Transformer network; and obtaining a global feature relation, so as to locate the defect;
(2) matching, in a training process of the defect location model, a located defect with a real defect by using an objective function based on a distance intersection-over-union; computing a defect position relation so as to accelerate a model convergence rate; adding a loss term based on a tight sensing intersection-over-union to a loss function; and paying attention to integrity of a defect sequence so as to improve accuracy of defect location; and
(3) after locating the defect sequence, performing frame sampling by using a convolutional network based on a 2D time sequence difference; and extracting visual and displacement information from a defect image frame so as to recognize a defect type.
The extracting a time sequence feature by using a two-stream network and a Transformer network specifically includes:
(1.1) inputting an original time sequence image, denoted by X={xn}n=1l, where the sequence contains an image frame, and xi, represents an Nth frame of the sequence X;
(1.2) converting the original time sequence image into Sn=(xt
I
τ(u,v,2k−1)=dτ+k−1x(u,v),
I
τ(u,v,2k)=dτ+k−1x(u,v),u=[1;w],v=[1;h],k=[1;L],
(1.3) denoting a feature sequence of a time sequence image extracted by the two-stream network by
and constituting a boundary assessment network by three convolutional layers; computing probabilities
of each frame as a start frame and an end frame of the defect sequence; and multiplying and combining an input feature of the time sequence image and the predicted probabilities, corresponding to each time sequence position, of start and end of the defect, so as to obtain the feature sequence:
{circumflex over (x)}
t
=x
t
×p
s
t
⊗x
t
×p
e
t
;
(1.4) after obtaining the feature sequence {circumflex over (x)}t
MultHeadAtten({circumflex over (x)}t
(1.5) regarding an image sequence of the feature sequence of the defect image containing the attention weight as an input by using multi-layer perceptron, and predicting and outputting positions of the start frame and the end frame.
The matching, in a training process of the defect location model, a located defect with a real defect by using an objective function based on a distance intersection-over-union; computing a defect position relation so as to accelerate a model convergence rate; adding a loss term based on a tight sensing intersection-over-union to a loss function; and paying attention to integrity of a defect sequence so as to improve accuracy of defect location specifically includes:
(2.1) matching, in the training process of the model, the located defect {circumflex over (φ)}j with the real defect φj pairwise, and then computing an error between a model location interval and a real defect interval (an interval error for short) as a loss value so as to optimize the model; and computing, in a matching process, an optimal match by maximizing the objective function, where the objective function is as follows:
π=arg max Σn=1N(DIoU(φn,{circumflex over (φ)}n)−l1(φn,{circumflex over (φ)}n)),
(2.2) in order to a training rate of the model and ensure that the model may converge when the located defect and real defect are not overlapped, defining the objective function based on the distance intersection-over-union as:
(2.3) finally defining a loss function of a defect location task as:
L
bou=({circumflex over (t)}s−ts)2+({circumflex over (t)}e−te)2,
The performing frame sampling by using a convolutional network based on a 2D time sequence difference; and extracting visual and displacement information from a defect image frame so as to recognize a defect type specifically includes:
(3.1) dividing the image sequence {circumflex over (x)}t
(3.2) representing motion information by the sampling frame of {circumflex over (F)}t=Ft+H (xt), where in the feature {circumflex over (F)}t, the sampling frame Ft contributes visual image information and the feature stacking H(xt) contributes local motion information, which are obtained by extracting and stacking the features of each frame before and after the sampling frame by an average pooling layer; and
(3.3) decoding a sampled feature image sequence {circumflex over (F)}t by using a multi-layer perceptron and a softmax function, so as to obtain the defect type.
A computer device includes a memory, a processor and a computer program stored in the memory and runnable on the processor; and when the processor executes the computer program, the intelligent recognition method for a time sequence image of a concrete dam defect is implemented.
A computer-readable storage medium stores a computer program configured to execute the intelligent recognition method for a time sequence image of a concrete dam defect.
Beneficial effects: compared with the prior art, the present disclosure has the following advantages:
(1) the time-dimensional self-attention mechanism is added to the image sequence by using the Transformer network, such that the model can pay attention to the global feature relation so as to improve the accuracy of the defect location;
(2) the objective function based on the distance intersection-over-union and the loss term based on the tight sensing intersection-over-union are added in the training process of the model, and the position information and integrity of the defect are paid attention to, so that the model convergence rate is accelerated, and the location accuracy is improved; and
(3) in the process of recognizing the defect, the sampling frame may contain the image feature and displacement information of the defect simultaneously by using the convolutional network based on the 2D time sequence difference, and the recognition accuracy is improved while computation resources are saved.
The present disclosure will be further described in detail below with reference to specific examples. It should be understood that the following examples are merely illustrative of the present disclosure and not the limitation of the scope of the present disclosure. Upon reading the present disclosure, various equivalent modifications of the present disclosure made by those skilled in the art fall within the scope defined by the attached claims of the present application.
There are known time sequence images of patrol and inspection defects in a dam project of a power station, and each image may contain four kinds of defects, namely cracking, precipitation of alkaline substances, water seepage and scaling of concrete, as shown in
(1) Based on problems of a long time sequence of an image sequence for the dam defect and a large number of background frames independent of the defect, a defect location model is designed; a feature sequence of a time sequence image is extracted by using a two-stream network, and a time-dimensional self-attention mechanism is added to obtain a global feature relation so as to locate the defect, as shown in
(1.1) An original time sequence image is input and denoted by X={xn}n=1l, where the sequence includes an image frame, and xn represents an Nth frame of the sequence X;
(1.2) An original image sequence is converted into Sn=(xt
I
τ(u,v,2k−1)=dτ+k−1x(u,v),
I
τ(u,v,2k)=dτ+k−1x(u,v),u=[1;w],v=[1;h],k=[1;L],
(1.3) The feature sequence of the time sequence image extracted by the two-stream network is denoted by
and a boundary assessment network is constituted by using three convolutional layers; and probabilities
of each frame as the start and end frames of the defect sequence are computed. The convolutional layer is denoted by Cony (cf, ck, f), where the parameters of cf, ck and f are the number of convolution kernels, the number of channels, and an activation function, respectively, and a structure of the above boundary assessment network may be simply expressed as Cony (512, 3, Relu)→Conv (512, 3, Relu)→Conv (3, 1, sigmoid). The three convolutional layers has the same step length of 1. Finally an input feature of the time sequence image and the predicted probabilities of the start and end of the defect corresponding to each time sequence position are multiplied and combined, so as to obtain the feature sequence:
{circumflex over (x)}
t
=x
t
×p
s
t
⊗x
t
×p
e
t
.
(1.4) A position code is added to mark the time sequence position for each frame, and a global self-attention weight is computed for each frame by using a Transformer network:
MultHeadAtten({circumflex over (x)}t
(1.5) The feature sequence is regarded as an input of a multi-layer perceptron with three layers and 512 dimensions, and positions of the start and end frames are predicted and output.
(2) In a training process of the defect location model, to solve the problem that the model cannot pay attention to the position and integrity of the defect, a located defect is matched with a real defect by using an objective function based on a distance intersection-over-union; the time sequence position relation of the defect is computed to accelerate model convergence; and a loss term based on a tight sensing intersection-over-union is added to a loss function so as to improve accuracy of the model.
(2.1) In the training process of the model, the located defect {circumflex over (φ)}j is matched with the real defect φj pairwise, and then an interval error is computed as a loss value so as to optimize the model. An optimal match is computed by maximizing the objective function, and the objective function is as follows:
π=arg max Σn=1N(DIoU(φn,{circumflex over (φ)}n)−l1(φn,{circumflex over (φ)}n)),
(2.2) In order to accelerate a training rate of the model and ensure that the model may converge when the located defect and real defect are not overlapped, the objective function based on the distance intersection-over-union is defined as:
(2.3) Finally the loss function for the defect location task is defined as:
L
bou=({circumflex over (t)}s−ts)2+({circumflex over (t)}e−te)2,
(3) After the defect sequence is located, frame sampling is performed by using a convolutional network based on a 2D time sequence difference, and visual and displacement information is extracted from a defect image frame so as to recognize a defect type, and the recognition accuracy is ensured while recognition of the model is accelerated, as shown in
(3.1) The image sequence {circumflex over (x)}t
(3.2) Motion information is represented by the sampling frame of {circumflex over (F)}t=Ft+H(xt); and in the feature {circumflex over (F)}t, the sampling frame Ft contributes visual image information and the feature stacking H(xt) contributes local motion information, and the motion information of n frames before and after the sampling frame is obtained by extracting and stacking the features of each frame before and after the sampling frame by an average pooling layer. The network structure is shown in
(3.3) A sampled feature image sequence is decoded to obtain the defect type {circumflex over (F)}t by using the multi-layer perceptron with three layers and 512 dimensions and the softmax function.
Obviously, those skilled in the art should understand that the steps of the defect recognition method for a time sequence image of a dam defect described above can be implemented by a general-purpose computing apparatus. The steps can be concentrated on a single computing apparatus or distributed in a network composed of a plurality of computing apparatuses. Alternatively, the steps can be implemented by a procedure code executable by a computing apparatus, so that the steps can be stored in a storage apparatus and executed by the computing apparatus. In some cases, the steps can be implemented in the following ways: the steps shown or described can be performed in a different order from the order described herein, or can be separately made into individual integrated circuit modules, or a plurality of modules or steps therein can be made into a single integrated circuit. Therefore, the examples of the present disclosure are not limited to any particular combination of hardware and software.
Number | Date | Country | Kind |
---|---|---|---|
202210515193.9 | May 2022 | CN | national |
Number | Date | Country | |
---|---|---|---|
Parent | PCT/CN23/82484 | Mar 2023 | US |
Child | 18322605 | US |