Patent Application
20210342594
The present disclosure relates to deep learning technology, and more particularly, to a method for sampling training data, an apparatus for sampling training data and a computer server.
In recent years, deep neural networks have achieved tremendous success in many fields including computer vision, natural language processing, etc. However, a deep neural network needs to meet the following three conditions in order to be applied in actual applications: 1) it meets speed requirements of the actual applications; 2) it is trained over a large amount of actual scenario data; and 3) it can be updated iteratively and quickly depending on actual requirements.
When a neural network is applied in practice, when a researcher or developer finds that processing of a video by a neural network is not optimal, he/she may need to sample some images from the video as images to be annotated, annotate the images to be annotated, add the annotated images to a training set and retrain the neural network based on the training set, so as to improve the performance of the neural network.
According to some embodiments of the present disclosure, a method for sampling training data is provided. The method includes: inputting a video to a target detection model to obtain a detection result for each frame of image; inputting the detection results for all frames of images in the video to a target tracking model, to obtain a tracking result for each frame of image; and for each frame of image in the video: matching the detection result and the tracking result for the frame of image, and when the detection result and the tracking result for the frame of image are inconsistent with each other, determining the frame of image as a sample image to be annotated, for which processing by the target detection model is not optimal.
According to some embodiments of the present disclosure, an apparatus for sampling training data is provided. The apparatus includes: a target detecting unit configured to input a video to a target detection model to obtain a detection result for each frame of image; a target tracking unit configured to input the detection results for all frames of images in the video to a target tracking model, to obtain a tracking result for each frame of image; and a determining unit configured to, for each frame of image in the video: match the detection result and the tracking result for the frame of image, and when the detection result and the tracking result for the frame of image are inconsistent with each other, determine the frame of image as a sample image to be annotated, for which processing by the target detection model is not optimal.
According to some embodiments of the present disclosure, a computer server is provided. The computer server includes a memory and one or more processors communicatively connected to the memory. The memory stores instructions executable by the one or more processors. The instructions, when executed by the one or more processors, cause the one or more processors to perform the method for sampling training data according to the above first aspect.
The figures are provided for facilitating further understanding of the present disclosure. The figures constitute a portion of the description and can be used in combination with the embodiments of the present disclosure to interpret, rather than limiting, the present disclosure. In the figures:
In the following, the solutions according to the embodiments of the present disclosure will be described clearly and completely with reference to the figures, such that the solutions can be better understood by those skilled in the art. Obviously, the embodiments described below are only some, rather than all, of the embodiments of the present disclosure. All other embodiments that can be obtained by those skilled in the art based on the embodiments described in the present disclosure without any inventive efforts are to be encompassed by the scope of the present disclosure.
In the related art, the images can be sampled from the video as the images to be annotated typically by sampling the image using a simple temporal interval sampling method to obtain the images to be annotated. However, the images sampled in this way may be images that have been processed well by the neural network, and thus the performance of the neural network may not be improved effectively by retraining the neural network based on these images. Hence, how to sample images whereby the neural network may be effectively improved has become a problem to be solved by those skilled in the art.
In view of the above problem, the present disclosure provides a method and an apparatus for sampling training data and a computer server, capable of sampling images for which processing by a target detection model is not optimal from a video as images to be annotated, such that the performance of the target detection model may be effectively improved by retraining the target detection model based on the images to be annotated.
Referring to
At step 101, a video is inputted to a target detection model to obtain a detection result for each frame of image.
At step 102, the detection results for all frames of images in the video are inputted to a target tracking model to obtain a tracking result for each frame of image.
At step 103, for each frame of image in the video, the detection result and the tracking result for the frame of image are matched, and when the detection result and the tracking result for the frame of image are inconsistent with each other, the frame of image is determined as a sample image to be annotated, for which processing by the target detection model is not optimal.
In some embodiments, the above step 103 of matching the detection result and the tracking result for each frame of image may be, but not limited to be, implemented by matching the detection result and the tracking result for the frame of image by using a bipartite graph matching algorithm. The bipartite graph matching algorithm may be e.g., Hungarian matching algorithm.
Alternatively, it can be appreciated by those skilled in the art that, for each frame of image, each detection target and each tracking target may be matched in a traversal manner. For example, when a detection result of a certain frame of image includes two detection targets, A1 and A2, and the tracking result of the frame of image includes two tracking targets, B1 and B2, four pairs, A1 and B1, A1 and B2, A2 and B1, and A2 and B2, may be matched respectively. If the detection result and tracking result in each pair match each other, it is determined that the frame of image is not a sample image to be annotated, for which processing by the target detection model is not optimal; otherwise the frame of image is determined as a sample image to be annotated, for which processing by the target detection model is not optimal.
In some embodiments, the sample images determined in the step 103 may contain a plurality of neighboring frames of images which are substantially the same. In order to avoid redundancy and reduce the annotating workload, in some embodiments of the present disclosure, the method according to each of the above embodiments may further include step 104 after the matching for all the frames of images in the video has completed in the above step 103, as shown in
At step 104, the sample images determined in the step 103 are sampled by using a temporal non-maximum suppression algorithm.
For example, it is assumed that the frames of images in the video are represented as t1, t2, t3, . . . , tn in chronological order, ten sample images are obtained in the step 103, which are t1, t2, t6, t7, t8, t20, t21, t22, t50 and t100, respectively. When the non-maximum value is set to 3, sample images in three successive frames will be sampled to obtain one frame, i.e., the ten sample images may be sampled to obtain: t1, t7, t21, t50 and t100.
It can be appreciated by those skilled in the art that other schemes, e.g., a simple interval sampling algorithm, can be used to sample the sample images in the step 104.
With the embodiments of the present disclosure, for each frame of image in a video, a detection result and a tracking result for the frame of image are matched, and when the detection result and the tracking result for the frame of image are inconsistent with each other, the frame of image is determined as a sample image to be annotated, for which processing by the target detection model is not optimal. In this way, each sample image as determined is an image for which processing by the target detection model is not optimal. A tracking result for a frame of image is typically obtained from a detection result for a previous frame of image. When the detection result and the tracking result for a frame of image are different from each other, the target detection model may have some problems such as missed detection or false detection. Typically, images for which processing by the target detection model is not optimal may better reflect defects of the target detection model. Therefore, the performance of the target detection model may be further improved by annotating the images for which processing by the target detection model is not optimal and retraining the target detection model based on the annotated images.
Based on the same concept as the method for sampling training data according to the above Embodiment 1, in Embodiment 2 of the present disclosure, an apparatus for sampling training data is also provided. The apparatus has a structure shown in
a target detecting unit 1 configured to input a video to a target detection model to obtain a detection result for each frame of image;
a target tracking unit 2 configured to input the detection results for all frames of images in the video to a target tracking model, to obtain a tracking result for each frame of image; and
a determining unit 3 configured to, for each frame of image in the video: match the detection result and the tracking result for the frame of image, and when the detection result and the tracking result for the frame of image are inconsistent with each other, determine the frame of image as a sample image to be annotated, for which processing by the target detection model is not optimal.
In some embodiments, the determining unit 3 being configured to match the detection result and the tracking result for the frame of image may include the determining unit 3 being configured to: match the detection result and the tracking result for the frame of image by using a bipartite graph matching algorithm.
In some embodiments, the bipartite graph matching algorithm may be Hungarian matching algorithm.
Further, the apparatus according to the above embodiments may include a sampling unit 4, as shown in
With the embodiments of the present disclosure, for each frame of image in a video, a detection result and a tracking result for the frame of image are matched, and when the detection result and the tracking result for the frame of image are inconsistent with each other, the frame of image is determined as a sample image to be annotated, for which processing by the target detection model is not optimal. In this way, each sample image as determined is an image for which processing by the target detection model is not optimal. A tracking result for a frame of image is typically obtained from a detection result for a previous frame of image. When the detection result and the tracking result for a frame of image are different from each other, the target detection model may have some problems such as missed detection or false detection. Typically, images for which processing by the target detection model is not optimal may better reflect defects of the target detection model. Therefore, the performance of the target detection model may be further improved by annotating the images for which processing by the target detection model is not optimal and retraining the target detection model based on the annotated images.
According to Embodiment 3 of the present disclosure, a computer server is provided. As shown in
The memory stores instructions executable by the one or more processors. The instructions, when executed by the one or more processors, cause the one or more processors to perform any method for sampling training data according to the above Embodiment 1.
In Embodiment 3 of the present disclosure, the computer server may be a hardware device such as a Personal Computer (PC), a notebook computer, a tablet, a Field Programmable Gate Array (FPGA), an industrial computer or a smart phone.
With the embodiments of the present disclosure, for each frame of image in a video, a detection result and a tracking result for the frame of image are matched, and when the detection result and the tracking result for the frame of image are inconsistent with each other, the frame of image is determined as a sample image to be annotated, for which processing by the target detection model is not optimal. In this way, each sample image as determined is an image for which processing by the target detection model is not optimal. A tracking result for a frame of image is typically obtained from a detection result for a previous frame of image. When the detection result and the tracking result for a frame of image are different from each other, the target detection model may have some problems such as missed detection or false detection. Typically, images for which processing by the target detection model is not optimal may better reflect defects of the target detection model. Therefore, the performance of the target detection model may be further improved by annotating the images for which processing by the target detection model is not optimal and retraining the target detection model based on the annotated images.
The basic principles of the present disclosure have been described above with reference to the embodiments. However, it can be appreciated by those skilled in the art that all or any of the steps or components of the method or apparatus according to the present disclosure can be implemented in hardware, firmware, software or any combination thereof in any computing device (including a processor, a storage medium, etc.) or a network of computing devices. This can be achieved by those skilled in the art using their basic programing skills based on the description of the present disclosure.
It can be appreciated by those skilled in the art that all or part of the steps in the method according to the above embodiment can be implemented in hardware following instructions of a program. The program can be stored in a computer readable storage medium. The program, when executed, may include one or any combination of the steps in the method according to the above embodiment.
Further, the functional units in the embodiments of the present disclosure can be integrated into one processing module or can be physically separate, or two or more units can be integrated into one module. Such integrated module can be implemented in hardware or software functional units. When implemented in software functional units and sold or used as a standalone product, the integrated module can be stored in a computer readable storage medium.
It can be appreciated by those skilled in the art that the embodiments of the present disclosure can be implemented as a method, a system or a computer program product. The present disclosure may include pure hardware embodiments, pure software embodiments and any combination thereof. Also, the present disclosure may include a computer program product implemented on one or more computer readable storage mediums (including, but not limited to, magnetic disk storage and optical storage) containing computer readable program codes.
The present disclosure has been described with reference to the flowcharts and/or block diagrams of the method, device (system) and computer program product according to the embodiments of the present disclosure. It can be appreciated that each process and/or block in the flowcharts and/or block diagrams, or any combination thereof, can be implemented by computer program instructions. Such computer program instructions can be provided to a general computer, a dedicated computer, an embedded processor or a processor of any other programmable data processing device to constitute a machine, such that the instructions executed by a processor of a computer or any other programmable data processing device can constitute means for implementing the functions specified by one or more processes in the flowcharts and/or one or more blocks in the block diagrams.
These computer program instructions can also be stored in a computer readable memory that can direct a computer or any other programmable data processing device to operate in a particular way. Thus, the instructions stored in the computer readable memory constitute a manufacture including instruction means for implementing the functions specified by one or more processes in the flowcharts and/or one or more blocks in the block diagrams.
These computer program instructions can also be loaded onto a computer or any other programmable data processing device, such that the computer or the programmable data processing device can perform a series of operations/steps to achieve a computer-implemented process. In this way, the instructions executed on the computer or the programmable data processing device can provide steps for implementing the functions specified by one or more processes in the flowcharts and/or one or more blocks in the block diagrams.
While the embodiments of the present disclosure have described above, further alternatives and modifications can be made to these embodiments by those skilled in the art in light of the basic inventive concept of the present disclosure. The claims as attached are intended to cover the above embodiments and all these alternatives and modifications that fall within the scope of the present disclosure.
Obviously, various modifications and variants can be made to the present disclosure by those skilled in the art without departing from the spirit and scope of the present disclosure. Therefore, these modifications and variants are to be encompassed by the present disclosure if they fall within the scope of the present disclosure as defined by the claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 201810498800.9 | May 2018 | CN | national |
This application is a continuation of U.S. patent application Ser. No. 16/421,320, filed May 23, 3019, titled “Method and Apparatus for Sampling Training Data and Computer Server,” published as US 2019-0384982 on Dec. 19, 2019, which in turn claims priority to and benefits of Chinese Application Number 201810498800.9 of the same title and content that was filed on May 23, 2018. The disclosures of the aforementioned patent applications are hereby incorporated by reference as part of the disclosure of this patent document.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16421320 | May 2019 | US |
| Child | 17378054 | US |
