Patent Application
20220230324
The invention belongs to the scene segmentation methods in the computer vision field, and the specific realization result is the segmentation of image content, particularly related to a segmentation method of camouflaged object in real environment images.
Two-dimensional image segmentation refers to distinguishing the pixels belonging to different objects in the environment image and determining their size, shape, and position. It is a key step from image processing to image analysis and has great application value. In recent years, segmentation-related tasks, such as object detection, depth estimation, salient region detection, and shadow detection, have achieved significant performance improvements.
Quite a few creatures in nature have evolved superb camouflaged skills and can disguise themselves to integrate into the surrounding environment. The appearance of such creatures in images will greatly affect the scene understanding, which affects the accuracy of various computer vision tasks and applications. The object's camouflaged ability makes the area of the object in images show the appearance of the surrounding environment, and most of the time, the camouflaged object is also a challenge to the human visual system. The diversity of the scene, the diversity of camouflaged objects in the scene, and the diversity of the camouflaged object's shape determine the difficulty of such image processing. Therefore, segmenting the camouflaged object is a fundamentally challenging task due to the fact that the camouflage strategy works by deceiving the visual perceptual system of the observer and thus a significant amount of visual perception knowledge is required to eliminate the ambiguities caused by the high intrinsic similarities between the target object and the background. Since there is no effective method to distinguish camouflaged objects from the image background, the camouflaged object's existence in the scene image will seriously affect the current computer vision system.
Methods related to camouflaged object segmentation mainly include generic object detection, salient object detection, and specific region segmentation.
Generic Object Detection (GOD) seeks to locate object instances from several predefined generic categories in natural images, which is one of the most fundamental and challenging problems in computer vision and forms the basis for solving complex or high-level vision tasks such as segmentation, scene understanding, and object tracking. The generic objects in a scene can be either conspicuous or camouflaged, and the camouflaged ones can be seen as hard cases. Therefore, directly applying GOD methods to segment camouflaged objects may not get the desired results.
Salient Object Detection (SOD) aims to identify and segment the most attention-grabbing object(s) in an input image. The traditional SOD method relies on artificially set low-level features (such as color and contrast); the SOD method based on deep learning uses bottom-up and top-down saliency inference, combined with the saliency estimation of local pixels and the search of global targets to detect the salient object. However, applying the above SOD approaches for camouflaged object segmentation may not appropriate as the term “salient” is essentially the opposite of “camouflaged”, i.e., standout versus immersion.
Specific Region Segmentation (SRS) we defined here refers to segmenting the specific region such as shadow, mirror, glass and water region in the scene. Such regions are special and has a critical impact on the vision systems. For the water, shadow and mirror region, there typically exists intensity or content discontinuities between the foreground and background. Instead, both the intensity and content are similar between the camouflaged objects and the background, leading to a great challenge of camouflaged object segmentation. Besides, the camouflaged objects are typically with more complex structures, compared with the glass region, and thus increasing the difficulty of accurate segmentation.
The present invention is based on the investigation and analysis of the existing scene segmentation technology and inspired by the process of prey capture by predators in nature. It proposes a segmentation method based on distraction mining, determines and removes distraction information, and forms a positioning and focus network for camouflaged object segmentation, namely PFNet. We have observed that in the process of predation in nature, predators will first locate potential prey from a global perspective and then focus on potential prey, gradually improve observation and judgment, and finally, determine prey and implement capture. The main difficulty for recognition or segmentation is the distraction information in the image, so PFNet's work is based on distraction mining. The input of PFNet is the RGB image containing camouflaged objects, and the output is the mask image. The value of the camouflaged object area in the mask image is 1, and the area without camouflaged object is 0. Experiments have proved that this method based on distraction mining can effectively segment different types of camouflaged objects and help improve the accuracy of scene understanding.
The purpose of the present invention is to creatively use the distraction mining method for the RGB image to realize the camouflaged object segmentation. PFNet successively includes a multi-level feature extractor, a positioning module, and three focus modules. The multi-level feature extractor uses a traditional feature extraction network to obtain contextual features with different levels; the positioning module is designed to mimic the detection process in predation for positioning the potential target objects from a global perspective and the focus module is then used to perform the identification process in predation for refining the initial segmentation results by focusing on the ambiguous regions. The present invention method is suitable for segmenting camouflaged objects with different types, positions, and sizes.
A method for camouflaged object segmentation with distraction mining, the method includes the following steps:
Step 1 Building PFNet
The camouflaged object segmentation network PFNet mainly consists of a multi-level feature extractor, a positioning module, and three focus modules.
The multi-level feature extractor is implemented based on the network ResNet-50 with feature extraction capabilities. The input of the multi-level feature extractor is an RGB image with several camouflaged objects. The extracted multi-level features are sequentially sent to a positioning module and three focus modules.
The positioning module aims to generate the initial location of camouflaged object. The positioning module consists of a channel attention block and a spatial attention block. The channel attention block and spatial attention block are implemented in a non-local way, to capture long-range dependencies in terms of channel and spatial position, for enhancing the semantic representation of the highest-level features from a global perspective. Specifically, given the input feature F ∈ RC×H×W, where C, H, and W represent the channel number, height, and width, respectively, we first reshape F to get the query Q, key K, and value V, respectively, where {Q, K,V}∈RC×N and N=H×W is the number of pixels. Then we perform a matrix multiplication between Q and the transpose of K, and apply a softmax layer to calculate the channel attention map X ∈ RC×C:
where, xij measures the jth channel's impact on the ith channel. Finally, to enhance the fault-tolerant ability, we multiply the result by a learnable scale parameter γ and perform an identify mapping operation to obtain the channel attention block's output F′ ∈ RC×H×W:
where γ gradually learns a weight from an initial value of 1. The feature F′ models the long-range semantic dependencies between the channels of feature maps and thus is more discriminative than the input feature F. Then, we feed the output features of channel attention block into the spatial attention block as the input. We first employ three 1×1 convolution layers on the features F′ and reshape the convolution results to generate three new feature maps Q′, K′ and V′ respectively, where K ∈ RC1×N}, and C1=C/8, and V′ ∈ RC×N. After that we perform a matrix multiplication between the transpose of Q ‘ and K′, and use the softmax normalization to generate the spatial attention map X′ ∈ RN×N:
where, x′ij denotes the jth position's impact on the ith position. Meanwhile, we conduct a matrix multiplication between V′ and the transpose of X′ and reshape the result to RC×H×W. Similar to the channel attention block, we multiply the result by a learnable scale parameter γ′ and add a skip-connection to obtain the spatial attention block's output F″ ∈RC×H×W:
where γ′ is also initialized as 1. F″ further gains the semantic correlations between all positions and thus enhancing the semantic representation of the feature. Finally, we can get the initial location map of the potential targets by applying a 7×7 convolution with the padding 3 on F″.
The focus module is designed to first discover and then remove these false predictions. The present invention first upsample the higher-level prediction and normalize it with a sigmoid layer. Then we use this normalized map and its reverse version to multiply the current-level features Fc, to generate the foreground-attentive features Fƒa and the background-attention features Fba, respectively. Finally, we feed these two types of features into two parallel context exploration (CE) blocks to perform contextual reasoning for discovering the false-positive distractions Fƒpd and the false-negative distractions Fƒnd respectively. The CE block consists of four context exploration branches and each branch includes a 3× 3 convolution channel reduction, a ki× ki convolution for local feature extraction, and a 3×3 dilated convolution with a dilation rate of ri for context perceiving. We set ki,i ∈ {1, 2, 3, 4} to 1, 3, 5, 7, and set ri, i ∈ {1, 2, 3, 4} to 1, 2, 4, 8, respectively. The output of the ith, i ∈ {1, 2, 3} CE branch will be fed into (i+1)th branch to be further processed in a larger receptive field. The outputs of all four branches are then concatenated and fused via a 3×3 convolution. By such design, the CE block gains the capability of perceiving abundant contexts over a wide range of scales and thus could be used for context reasoning and distraction discovery. After distraction discovery, we can perform distraction removal in the following way:
F
up
=U(CBR(Fh)),
F
r
=BR(Fup−αFƒpd),
F
r
′=BR(Fr+βFƒnd),
where Fh and Fr′ denote the input higher-level features and the output refined features, respectively; CBR presents the combination of convolution, batch normalization (BN) and ReLU; U is the bilinear upsampling; and α and β are the learnable scale parameters. Here we use the element-wise subtraction operation to suppress the ambiguous backgrounds (i.e., false-positive distractions) and the element-wise addition operation to augment the missing foregrounds (i.e., false-negative distractions). Finally, a more accurate prediction map can be obtained by applying a convolution layer on the refined feature Fr′;
Step 2 Training Process
During training, the training data is first fed into the multi-level feature extractor of the network, and the extracted results are fed into a positioning module and three focus modules successively. Then, the positioning module uses the channel attention block and the spatial attention block to initially locate the camouflaged object in the image. The focus module repeatedly mines the camouflaged object information and removes environmental distraction information, and finally obtains fine prediction results. We use the ground truth map to supervise the generated map, to force the Fr′ into a purer representation than Fh, i.e., the distraction removed features. This would further guide the CE block to discover the specific form of distractions and make the whole focus module works on distraction discovery and removal in an implicit way. For the positioning module, we impose binary cross-entropy (BCE) loss lbce and IoU loss liou on its output, i.e., Lpm=lbce+liou, to guide the positioning module to explore the initial location of the target object. For the focus module, we hope it could focus more on the distraction region. Such region is typically located at the object's boundaries, elongated areas, or holes. Thus we combine the weighted BCE loss lwbce and the weighted IoU loss lwiou, i.e., Lƒm=lwbce+lwiou to force the focus module pay more attention to the possible distraction region. Finally the overall loss function is:
Where Lƒmi denotes the loss for the prediction of the focus module at i-th level of the PFNet.
The beneficial effects of the present invention:
(1) Method innovation
The present invention introduces the concept of distraction mining into the field of the camouflaged object segmentation and develops a new distraction discovery and distraction removal strategy to help the camouflaged object segmentation. We propose the positioning and focus network with a well-designed focus module to first discover and then remove these distractions. To the best of our knowledge, we are the first to mine distractions for camouflaged object segmentation and we believe that the proposed strategy of distraction mining could provide insights to other vision tasks.
(2) Results and efficiency
As the first method of using distraction mining in camouflaged object segmentation, the present invention helps the visual system to distinguish the camouflaged object from the environment. In the comparative experiment, PFNet has obtained the superior performance on the three commonly used indicators (Sα,EΦad and Fβω), and the advantages are obvious. The segmentation result of PFNet is excellent, and the degree of fineness at the boundary of the camouflaged object is also satisfactory.
(3) Wider Applicability
The present invention is suitable for the segmentation of various types of camouflaged objects. PFNet has obvious performance advantages on the following types of images: a) images with small-volume camouflaged objects; b) images with large-volume camouflaged objects; c) images with camouflaged objects' area content similar to the surrounding environment or unclear borders; d) images with occluded camouflaged objects. All of the above proves the effectiveness and practicality of PFNet.
The specific embodiments of the present invention will be further described below in conjunction with the drawings.
We evaluate our method on three benchmark datasets: CHAMELEON (76 images), CAMO (1250 images), and COD10K (5066 images). We follow previous work to use the training set of CAMO and COD10K as the training set (4,040 images) and others as testing sets. For testing, the image is first resized to 416×416 for network inference and then the output map is resized back to the original size of the input image. Both the resizing processes use bilinear interpolation. The parameters of the feature extraction network are initialized by the pre-trained ResNet-50 network, and other parameters are initialized randomly.
In PFNet, images with the camouflaged object are fed into the multi-level feature extractor, and the result is fed into the positioning module and the focus module. The positioning module consists of a channel attention block and α spatial attention block, which captures the long-range semantic dependence of the channel and spatial position to infer the approximate position of the target object from a global perspective. The focus module performs multi-scale context exploration based on the foreground attention (background attention) function to find false positive (false negative) distraction, and then remove the distraction to obtain a purer segmentation of the target object. We implement our model with the PyTorch toolbox. An eight-core PC with an Intel Core i7-9700K 3.6 GHz CPU (with 64 GB RAM) and an NVIDIA GeForce RTX 2080Ti GPU (with 11 GB memory) is used for both training and testing.
| Number | Date | Country | Kind |
|---|---|---|---|
| 202110078735. 6 | Jan 2021 | CN | national |
