Patent Grant
11769322
This application claims the benefit of priority to Japanese Patent Application Number 2020-063281 filed on Mar. 31, 2020. The entire contents of the above-identified application are hereby incorporated by reference.
The present disclosure relates to a program creation device, an object detection system, an anchor setting method, and an anchor setting program.
As a system for detecting an object from an acquired image, there is a system for detecting an object by using a learned program in which deep learning (machine learning) is performed on a large number of images. In object detection using general deep learning, first, convolution processing using a specific filter coefficient is performed on an image, which is used as an input, to extract features. Next, rectangular regions (bounding boxes) called anchors are arranged in feature spaces having different resolutions obtained in the process of the convolution processing, and a score indicating the likelihood of an object is calculated from features in the region for each anchor. Using the calculated score, the size of the anchor whose score is equal to or greater than the threshold value is adjusted by regression processing, and the anchor is output as a detection result.
In deep learning, a plurality of types of anchor shapes are set, and detection of an object is performed using anchors having different shapes, thereby improving the accuracy of detection of an object. However, as the number of anchors increases, the amount of computation processing increases. Therefore, the number of usable anchors is limited according to the processing capacity and the length of time until the computation result is obtained. As described above, it is desired to improve the accuracy of detection of an object while suppressing an increase in the number of anchors to be set.
At least one embodiment of the present disclosure, in order to solve the above problem, provides a program creation device, an object detection system, an anchor setting method, and an anchor setting program, which are capable of appropriately setting a shape of an anchor and detecting an object with high accuracy.
The present disclosure provides a program creation device configured to create an object detection program for detecting whether an object is included in an image. The program creation device includes, training data including a plurality of image data including area information of the object, a setting unit configured to set an anchor that is information of a frame specifying a region for each cell for detecting the presence or absence of the object from the image, and a learning unit configured to execute machine learning of the training data based on the information of the setting unit and to create a learned program for extracting the object from the image. The setting unit acquires information on a target region of the training data and on aspect ratios of the anchor, calculates a degree of matching between the anchor and the target region for each aspect ratio while changing a size of the anchor, and calculates an adoption rate of the target region whose degree of matching is a proportion no less than a threshold, and determines, based on a result of the calculation, the size of the anchor used in the learned program.
Additionally, the present disclosure provides an object detection system which includes, the program creation device described above, and an object detection device including, a computation unit configured to execute a learned program created by the program creation device, a camera unit configured to acquire an image, and a notification unit configured to notify an operator. The computation unit analyzes, by the learned program, an image acquired by the camera unit, and the notification unit notifies when it is detected that the object is included in the image.
Further, the present disclosure provides an anchor setting method of setting an anchor used in an object detection program for detecting whether an object is included in an image. The anchor setting method includes the steps of, acquiring training data including a plurality of image data including area information of the object, acquiring anchor information that is information of a frame specifying a region for each cell for detecting the presence or absence of the object from the image, acquiring information on a target region of the training data and on aspect ratios of the anchor, calculating a degree of matching between the anchor and the target region for each aspect ratio while changing the size of the anchor, and calculating an adoption rate of the target region whose degree of matching is a proportion no less than a threshold, and determining, based on a result of the calculation, the size of the anchor used in the learned program.
In addition, the present disclosure provides an anchor setting program for causing a processing of setting an anchor to be executed, where the anchor is used in an object detection program for detecting whether an object is included in an image. The anchor setting program causes the steps of; acquiring training data including a plurality of image data including area information of the object, acquiring anchor information that is information of a frame specifying a region for each cell for detecting the presence or absence of the object from the image, acquiring information on a target region of the training data and on aspect ratios of the anchor, calculating a degree of matching between the anchor and the target region for each aspect ratio while changing the size of the anchor, and calculating an adoption rate of the target region whose degree of matching is a proportion no less than a threshold, and determining, based on a result of the calculation, the size of the anchor used in the learned program, to be executed.
With the above-described configuration, it is possible to appropriately set the shape of the anchor and to detect the object with high accuracy.
The disclosure will be described with reference to the accompanying drawings, wherein like numbers reference like elements.
Hereinafter, embodiments according to the present disclosure will be described in detail with reference to the drawings. Note that the invention is not limited to the embodiments. In addition, components in the following embodiments include components that can be easily replaced by those skilled in the art or substantially the same components. Furthermore, the components described below can be appropriately combined, and when there are a plurality of embodiments, each embodiment can be combined.
Object Detection System
The program creation device 10 includes an input unit 12, an output unit 14, a computation unit 16, and a storage unit 18. The input unit 12 includes an input device such as a keyboard and a mouse, a touch panel, or a microphone that collects speech from an operator, and outputs a signal corresponding to an operation performed by the operator on the input device to the computation unit 16. The output unit 14 includes a display device such as a display, and displays a screen including various kinds of information such as a processing result and an image to be processed, based on a display signal output from the computation unit 16. The output unit 14 may include a recording device that outputs data on a recording medium. The program creation device 10 may include a communication unit that transmits data using a communication interface as the input unit 12 and the output unit 14. The communication unit transmits various data and programs acquired by communicating with an external device to the storage unit 18 and stores the data and programs. The communication unit may be connected to an external device via a wired communication line or a wireless communication line.
The computation unit 16 includes an integrated circuit (processor) such as a central processing unit (CPU) or a graphics processing unit (GPU), and a memory serving as a work area, and executes various processes by executing various programs using these hardware resources. Specifically, the computation unit 16 reads a program stored in the storage unit 18, deploys the program into the memory, and causes the processor to execute instructions included in the program deployed into the memory thereby executing various processing. The computation unit 16 includes a training data creation unit (an example of a data creation unit) 30, an anchor setting unit (an example of a setting unit) 32, a learning unit 34, and an object detection processing unit (an example of a processing unit) 36. Before describing each unit of the computation unit 16, the storage unit 18 will be described.
The storage unit 18 includes a non-volatile storage device such as a magnetic storage device or a semiconductor storage device, and stores various programs and data. The storage unit 18 includes image data 40, configuration data 42, a learning execution program 44, an anchor setting program 46, an object detection program 48, and a learned program 50.
The data stored in the storage unit 18 includes the image data 40 and the configuration data 42. The image data 40 includes training data used for learning. The training data is data which associates data of an image and a region (bounding box) where an object is displayed, when the object is included in the image. The image of the training data may include data used for learning and data for evaluating the accuracy of the program after learning. In addition, the image data may include image data in which an object needs to be detected. The configuration data 42 includes anchor setting information described later, information on conditions for executing the learned program, or the like.
The programs stored in the storage unit 18 include a learning execution program 44, an anchor setting program 46, an object detection program 48, and a learned program 50.
The learning execution program 44 performs deep learning processing on the training data included in the image data 40 based on the setting of the configuration data 42, and creates a learned program 50. As a deep learning model, it is possible to use a deep learning model such as Regions with Convolutional Neural Networks (R-CNN), You Only Look Once (YOLO), or Single Shot multibox Detector (SSD), in which a bounding box called an anchor is set for an image and a feature in the anchor based on the setting is processed to detect whether an object is included in the image.
The anchor setting program 46 executes, with the learning execution program 44 and the learned program 50, processing of setting an anchor to be used in performing image processing using the deep learning model. The anchor setting program 46 executes processing for setting the size of the anchor. Further, the anchor setting program 46 preferably executes processing for setting the aspect ratio of the anchor and the number of anchors to be used. The information set by the anchor setting program 46 is stored in the configuration data 42.
The object detection program 48 is a program that executes an object detection processing using the learned program 50. The object detection program 48 is also set with acquiring processing for an image and output processing of a determination result. The object detection program 48 may set processing for processing image data. The learned program 50 is a program created by executing the learning execution program 44. The object detection program 48 can calculate the feature (score) of the learned determination criterion by executing the learned program 50 by a computation unit that performs image processing, and can execute processing of detecting an object based on the feature.
In the storage unit 18, the learning execution program 44, the anchor setting program 46, and the object detection program 48 may be installed by reading the learning execution program 44, the anchor setting program 46, and the object detection program 48, which are recorded in the recording medium, or alternatively, the learning execution program 44, the anchor setting program 46, and the object detection program 48 may be installed by reading the learning execution program 44, the anchor setting program 46, and the object detection program 48, which are provided on the network.
The function of each unit of the computation unit 16 will be described. Each unit of the computation unit 16 can be executed by executing a program stored in the storage unit 18. When there is an object in the image data, the training data creation unit 30 associates the image data with frame information (bounding box) indicating a region of the object. The frame to be set is rectangular. The training data creation unit 30 sets frame information from an operation input to the input unit 12 in a state where, for example, an image is displayed on the output unit 14. The operation input to the input unit 12 is an operation in which the operator inputs information of a position surrounding the position of the frame (object) while viewing the image. Further, the training data creation unit 30 may acquire the result of the image extraction processing executed by the object detection processing unit 36. In this case, an operation performed by the operator to determine whether the position of the extracted frame is correct data of the training data may be detected, and data in which the position of the frame is determined to be correct by the operator may be acquired as the training data.
The anchor setting unit 32 executes the processing of the anchor setting program 46 and sets information of an anchor used in the image processing of the deep learning model executed by the learning unit 34 and the object detection processing unit 36. The processing of the anchor setting unit 32 will be described later.
The learning unit 34 executes the processing of the learning execution program 44 using the anchor setting set by the anchor setting unit 32, performs deep learning as training data of the image data 40, and creates a learned program. The processing of the learning unit 34 will be described later.
The object detection processing unit 36 processes the learned program 50 by using the object detection program 48, and executes processing of determining whether an object is included in the acquired image, that is, an object detection processing. The processing of the object detection processing unit 36 will be described later.
In the present embodiment, although the program creation device 10 includes the training data creation unit 30 and the object detection processing unit 36. However, the program creation device 10 may not include the training data creation unit 30 and the object detection processing unit 36. That is, the training data may be created by another device. The object detection processing unit 36 that executes the processing of detecting an object from an image may be provided only in the object detection device 102.
The object detection device 102 is installed in a moving body or a building as described above. The object detection device 102 may be capable of communicating with the program creation device 10, but may not have a communication function. Various processing conditions are set in advance in the object detection device 102 that does not have a communication function, and the object detection processing is executed based on the set conditions. The object detection device 102 may output the detection result to a control device that controls an installed mechanism. Thus, for example, in the case of a moving body, when an object is detected, a processing of stopping the moving body, a processing of avoiding the object, or the like can be performed.
The object detection device 102 includes a camera unit 112, a computation unit 114, a storage unit 116, and a notification unit 118. The camera unit 112 acquires an image of the target field of view. The camera unit 112 may continuously acquire images at a predetermined frame rate or may acquire images triggered by a predetermined operation.
The computation unit 114 includes an integrated circuit (processor) such as a CPU or a GPU and a memory serving as a work area, and executes various processing by executing various programs using these hardware resources. Specifically, the computation unit 114 reads a program stored in the storage unit 116, deploys the program into the memory, and causes the processor to execute instructions included in the program deployed into the memory, thereby executing various processing. The computation unit 114 executes a program stored in the storage unit 116 to execute processing of detecting an object from an image.
The storage unit 116 includes a non-volatile storage device such as a magnetic storage device or a semiconductor storage device, and stores various programs and data. The storage unit 116 stores an object detection program 120 and a learned program 122.
The notification unit 118 notifies the operator. The notification unit 118 includes a speaker, a light emitting device, a display, or the like. When the computation unit 114 executes processing and detects an object included in an image, the notification unit 118 notifies the operator that there is an object. When the object is a person, the person who is the detection target may be notified.
The object detection processing unit 36 of the present embodiment determines whether an object is included in an image by performing image processing by the set deep learning. The learning unit 34 performs machine learning, for example, deep learning, using the training data generated by the training data creation unit 30 to generate a learned program to be executed by the object detection processing unit 36.
Here, in the deep learning of the present embodiment, convolution processing is performed on a target image, and a feature obtained from a plurality of pixels is set as one piece of cell information. Then, as illustrated in
That is, as illustrated in
Here, in the setting of the deep learning, in order to calculate the evaluation of one cell, the anchor 212 is set as the information of the frame for acquiring the information. The anchor 212 of the present embodiment is set such that the center thereof coincides with the center of the cell to be evaluated. The size of the anchor 212 with respect to the cell is set, and the larger the cell, the larger the anchor 212. In addition, a plurality of anchors 212 for performing processing in deep learning are set.
Next, the training data will be described.
The object detection system 100 creates a learned model capable of extracting an object by performing a learning processing of a plurality of images, which includes image 220 containing information on frames 224 and 226 of the image data, while adapting the anchors, which are set for the teacher data, to each cell.
In the case of the image 220 illustrated in
In this manner, the object detection system 100 applies the anchor to each cell of the feature map and processes the image in the anchor by deep learning to detect whether the object is included in the image data.
Here, when a state of degree of matching, that is, the ratio of overlap between the region where the object is displayed, which is the bounding box in case of training data, and the anchor is high can be maintained, the accuracy of learning also increases, and the object can be detected with high accuracy. Here, the degree of matching is evaluated by Intersection over Union (IoU). Specifically, it is a percentage of (matching portion between the bounding box and the anchor)/(union of the bounding box and the anchor). On the other hand, when the region where the object is displayed, which is the bounding box in case of the training data, is located at the boundary of the anchor, the degree of matching is low in any anchor at the time of deep learning, the amount of learning in deep learning does not increase, and the object may not be detected by the learned program. In addition, even at the time of actual detection, an anchor having a high degree of matching with the region including the object is not generated, and detection may not be performed. On the other hand, the object detection system 100 executes the following processing.
Anchor Setting Method
Next, an anchor setting method will be described with reference to
In the example illustrated in
In deep learning, by making the shape of the anchor 212 a plurality of shapes, it is possible to provide an anchor close to the shape of the bounding boxes 280 and 280a, which are regions set to include an object. For example, in the case of
The anchor setting unit 32 according to the present embodiment sets the anchor based on the bounding box of the training data, thereby increasing the detection accuracy of the object. Anchor setting processing will be described with reference to
The anchor setting unit 32 acquires training data including region information of an object, that is, information of a bounding box (step S12). The anchor setting unit 32 reads the setting of the anchor (step S14). Here, the setting of the anchor is information on the aspect ratio of the anchor and the reference size with respect to the cell for all anchors used in learning.
As illustrated in
The anchor setting unit 32 calculates the IoU at each position based on the region information of the object and the information of the anchor (step S16).
The anchor setting unit 32 determines whether the evaluation of the sizes of all the anchors is completed (step S18). When the anchor setting unit 32 determines that the evaluation of the sizes of all the anchors is not completed (No in step S18), the anchor setting unit 32 changes the sizes of the anchors (step S20) and returns to the process of step S16. That is, the anchor is changed to an unevaluated size, and IoU is calculated for the changed size.
When the anchor setting unit 32 determines that the evaluations of the sizes of all the anchors have been completed (Yes in step S18), the anchor setting unit 32 determines the dimensions (sizes) of the anchors based on the results of the evaluations (step S22).
The anchor setting unit 32 can detect the size of the anchor with which the region information of the object included in the training data can be extracted by comparing the size of the anchor with the region information of the object and evaluating the adoption rate, in a case where the sizes of the anchors are different from each other.
As shown in
Learned Program Creation Method
The learning unit 34 acquires training data including region information of an object (step S30). The learning unit 34 reads the setting of the anchor (step S32). That is, the learning unit 34 reads the size information and aspect ratio information of the anchors set by the anchor setting unit 32. The learning unit 34 performs deep learning based on the training data and the anchor information (step S34). The learning unit 34 sets a deep learning model for the image of the training data based on the information of the anchor, and performs learning of the image of the training data using the set model. Accordingly, the learning unit 34 generates a learned program in which learning using the training data is executed.
The learning unit 34 evaluates the learning result using an image for evaluation (step S36). Here, the image for evaluation is a data set including both an image including the object and an image not including the object. The image for evaluation is associated with information indicating whether the object is included or not. The learning unit 34 detects the object in the evaluation image using the learned program at the evaluation time point, thereby evaluating whether the object in the evaluation image including the object has been detected, whether the evaluation image not including the object has been falsely detected as including the object, or the like. The learning unit 34 calculates a detection rate, a false detection rate, or the like, as the evaluation.
After calculating the evaluation, the learning unit 34 determines whether to end the learning (step S38). The evaluation criterion for the end of learning can be arbitrarily set. For example, the number of times of learning or the amount of computation may be used as a criterion, and the process may be ended when the detection rate or the false detection rate satisfies the set performance.
When it is determined that the learning is not ended (No in step S38), the learning unit 34 adjusts the condition of the deep learning (step S40) and returns to step S34. Thus, the learning processing is executed again. Here, the conditions of deep learning are not particularly limited, but as a learning program at the start of learning in step S34, there are setting of a current learning program, or partial replacement of an image of training data, or the like. When the learning unit 34 determines that the learning is ended (Yes in step S38), the learning unit 34 sets the program of the learning result as the learned program (step S42) and ends the processing.
As described above, the learning unit 34 executes the deep learning processing using the anchor set by the anchor setting unit 32 and creates a learned program.
Object Detection Method
Next, an object detection method using a learned program will be described with reference to
The object detection device 102 reads the learned program (step S50). The object detection device 102 acquires the learned program created by the program creation device 10. The object detection device 102 acquires an image data (step S52). Specifically, the object detection device 102 acquires an image by the camera unit 112.
The object detection device 102 analyzes the image data based on the learned program (step S54). In the object detection device 102, the computation unit 114 detects whether an object is included in the image data by using the learned program created by performing deep learning under the condition of the anchors that are set by the anchor setting unit 32.
The object detection device 102 determines whether there is an object from the analysis result of the step S54 (step S56). When it is determined that there is an object (Yes in step S56), the object detection device 102 notifies the detection of the object from the notification unit 118 (step S58). When it is determined that there is no object (No in step S56), or after the processing of step S58 is executed, the object detection device 102 determines whether the process is ended (step S60). When it is determined that the processing is not ended (No in step S60), the object detection device 102 returns to step S52, acquires the next image data, and performs object detection processing. When the object detection device 102 determines that processing is ended (Yes in step S60), the object detection device ends the present processing.
In the present embodiment, as described above, in various cases in which the size (scale) of the anchor is changed using the region information of the object of the training data, comparison between the regions is performed to calculate the degree of matching, the size of the anchor in which the number of matching anchors increases at a rate equal to or greater than the threshold is calculated, and the deep learning is performed using the calculated size of the anchor. Thus, the detection accuracy of the object can be further improved. In addition, by determining the size of the anchor in the processing of performing the comparison between the regions, it is possible to determine the size of the anchor with which the detection accuracy of the object can be further improved with significantly less calculation than in the case of performing deep learning for various combinations of anchors and finding an optimum condition. Further, by using the information of the training data in which the region information is set, it is possible to execute the processing without creating new data.
Another Example of Anchor Setting Method
Here, the anchor setting unit 32 may determine the aspect ratio of the anchor based on information of the frame of the target region of the training data.
The anchor setting unit 32 acquires the training data (step S70). The anchor setting unit 32 extracts the distribution of the aspect ratio of the object region (step S72). The anchor setting unit 32 detects the aspect ratio of the set region for all images of the training data. When the information of the aspect ratio is set in advance, the anchor setting unit 32 may read the information of the set aspect ratio.
The anchor setting unit 32 calculates a distribution of aspect ratios (step S74). As a result, as shown in
The anchor setting unit 32 determines a plurality of aspect ratios of the anchor based on the distribution of the aspect ratios (step S76). Specifically, based on the distribution of aspect ratios in the training data, for example, the aspect ratio positions at 2% and 98% of the percentage of the distribution and the aspect ratio position dividing evenly between those two positions are used as the aspect ratios of the anchor to be used. Alternatively, the aspect ratio of the anchor at the peak position of the distribution may be used.
In this manner, by setting the aspect ratio of the anchor based on the distribution of the aspect ratios of the training data, the aspect ratio of the anchor can be determined based on the aspect ratio of the object to be detected on the image. Accordingly, it is possible to further improve the detection accuracy of the object by the anchor.
Another Example of Anchor Setting Method
After determining the size of the anchor, the anchor setting unit 32 may evaluate the anchor to be used and reduce the number of anchors. That is, a combination of aspect ratios of anchors to be used may be evaluated, and anchors having aspect ratios that have a small influence on the detection rate may not be used.
The anchor setting unit 32 executes the present processing after determining the size of the anchor illustrated in
For example, in the example illustrated in
The anchor setting unit 32 compares the adoption rates of the respective anchor combinations (step S86). As illustrated in
In this manner, the anchor setting unit 32 evaluates the combination of anchors and considers the combination of reducing the number of anchors to be used while suppressing a reduction in the adoption rate, and thus it is possible to reduce the amount of calculation while suppressing a reduction in the accuracy of detection of an object. In addition, by performing evaluation using training data, it is possible to obtain a combination of anchors suitable for detection of an object.
While preferred embodiments of the invention have been described as above, it is to be understood that variations and modifications will be apparent to those skilled in the art without departing from the scope and spirit of the invention. The scope of the invention, therefore, is to be determined solely by the following claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-063281 | Mar 2020 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20180039853 | Liu et al. | Feb 2018 | A1 |
| 20190108411 | Liu | Apr 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 5172749 | Mar 2013 | JP |
| 2018-22484 | Feb 2018 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20210303823 A1 | Sep 2021 | US |
