The present invention relates to a technology for recognizing and learning a target such as an object and a person.
In the object recognition/learning technology, there are being studied a number of methods for generating a pseudo sample when learning samples for recognizing the object are insufficient. For example, there is a conversion method in which information that cannot be observed from learning input data to which a label is attached is artificially generated, as disclosed in PTL 1. In addition, there is being studied a method of extracting components and patterns forming data from the learning data and adding the number of samples by synthesizing the pseudo data, or a method of generating data from other domain input data (for example, characters) in a case where the actual learning real data (for example, voice) does not exist.
On the other hand, as disclosed in PTL 2, a method of covering an insufficient learning sample by updating a learning model online while adding a learning sample is being studied.
PTL 1: JP 2016-186515 A
PTL 2: JP 2008-204102 A
In order to improve the recognition accuracy when the number of learning samples is insufficient, information is generated in a pseudo manner with respect to the input learning data in PTL 1, so that only generated information for the input data can be used. In addition, in the method of adding the number of samples by extracting the components and patterns forming the data from the learning data and synthesizing the pseudo data, the components and patterns forming the data are extracted and the pseudo data is synthesized. Therefore, it is difficult to synthesize an abstract pattern in a complex data structure into a data sample that is close to real. In addition, in PTL 1, the number of samples and information is simply increased. Therefore, the generated samples and information do not always contribute to learning. For a recognition model, learning on a sample that is difficult to recognize or a sample with high uncertainty in recognition greatly contributes to improvement of the recognition performance of the model. In addition, in a case where the generated sample is similar to the existing learning information, there is a possibility that the contribution to the learning is small or not. Therefore, increasing the number of samples by a simple generation process has a problem that efficient and effective learning cannot be performed. Similarly, PTL 2 also has the above-described problem because a newly acquired learning sample is uniformly learned for an additional learning sample when a model is updated online.
The invention has been made in view of the above background art and problems, and an example thereof is an online recognition apparatus which includes a feature amount extraction unit that extracts a feature amount of input data, an identification result prediction unit that predicts an identification result based on the extracted feature amount, a prediction result evaluation unit that determines necessity of labeling from the predicted identification result unit, a correct answer assigning unit that assigns a correct answer to input data online from the determination result, a generator update unit that updates a parameter of a generator based on the input data with the correct answer, a pseudo-learning data generation unit that establishes a generator based on the parameter of the updated generator and generates pseudo-learning data, and an identifier update unit that online updates a parameter of an identifier prepared in advance based on the input data with the correct answer and the pseudo-learning data. The updated identifier is updated as a new identification result prediction unit.
According to the invention, it is possible to provide an online recognition apparatus, an online recognition method, and a setting screen used for the same, which can improve recognition accuracy even when learning samples are insufficient.
Hereinafter, embodiments of the invention will be described using the drawings.
As input data 101, image data obtained from input units such as a visible camera, a stereo camera, an IR camera, and a radiation (X-ray) irradiation device may be used. The identifier may be configured using a support vector machine (SVM), a neural network, or the like. The generator may be configured using a hidden Markov model (HMM), a neural network, or the like. The pseudo-learning data is data belonging to the same domain as the input data.
The identifier feature amount extraction unit 102 extracts an identifier feature amount from the received input data. The identifier feature amount is extracted by the identifier feature amount extraction unit 102, and is extracted as information capable of identifying the content of the target data.
The identification result prediction unit 103 identifies the content of an object based on the identifier feature amount received from the identifier feature amount extraction unit 102. The identification information is output using the identification result as a vector. The output identification information includes an identification label to be identified (hereinafter, class information), a probability indicating the degree of belonging of the class information to which the object belongs, entropy (a measure of uncertainty), and the like.
The prediction result evaluation unit 104 calculates the identification uncertainty of the input data based on the identification information received from the identification result prediction unit 103, and comprehensively evaluates whether the input data should be added as a learning sample based on the result. The input data satisfying the condition to be added as a learning sample is labeled with a correct answer class by the correct answer assigning unit 105.
The generator instruction variable generation unit 106 converts the class information received from the correct answer assigning unit 105 into an instruction variable. The converted variables include one hot vector. The generator update unit 107 updates the parameters of the generator using the assigned correct answer and the corresponding input data. The pseudo-learning data generation unit 108 generates samples of pseudo-learning data having the class information output from the correct answer assigning unit 105 as many as the number set from the generation sample number setting unit 109 based on the generator received from the generator update unit 107 and the instruction variable received from the generator instruction variable generation unit 106, and stores likelihood information between the pseudo-learning data generation sample and the real sample in the generation sample storage unit 110.
The pseudo-learning data selection unit 111 selects samples whose likelihood of the generated samples accumulated in the generation sample storage unit 110 is up to the ranking k, and outputs a list of selected samples. k is manually input by the user, but a method of estimating from past data may be used. The identifier update unit 112 updates the parameters of the identifier using the pseudo-learning data received from the pseudo-learning data selection unit 111 and the input data to which the correct answer class has been assigned, and outputs the updated parameters to the identification result prediction unit 103.
An information density calculation unit 202 calculates the similarity with the learning sample, and outputs the average as information density information. An average likelihood calculation unit 203 calculates a distance from the received identification information to the predicted distribution of a recognizer, and outputs the value of the distance as an average likelihood.
In a case where the identification information satisfies the conditions set by the uncertainty evaluation unit 201, the ensemble evaluation unit 204 performs the evaluation again based on the information density received from the information density calculation unit 202 and the average likelihood received from the average likelihood calculation unit 203, and determines whether the input data having the finally input identification information should be assigned a correct answer, for example, by comparing with a prepared threshold.
Next, in Step S407, a generator update process for updating the parameters of the generator is performed using the correct answer assigned by the generator update unit 107 and the input data corresponding thereto. Then, in Step S408, as a data list generation process, an instruction variable is generated by the generator instruction variable generation unit 106, and the pseudo-learning data generation unit 108 generates samples of pseudo-learning data having the class information output from the correct answer assigning unit 105 as many as the number set from the generation sample number setting unit 109 based on the generator received from the generator update unit 107 and the instruction variable, stores likelihood information between the pseudo-learning data generation sample and the real sample in the generation sample storage unit 110, and makes a list.
Next, in Step S409, a pseudo-learning data selection process is performed to select a sample whose likelihood of the generated sample accumulated in the generation sample storage unit 110 by the pseudo-learning data selection unit 111 is up to the ranking k. In Step S410, the correct answer information and the input data assigned with the correct answer class are acquired. In Step S411, an identifier update process is performed to update the parameters of the identifier using the pseudo-learning data received from the pseudo-learning data selection unit 111 and the input data assigned with the correct answer class.
The above is the processing for one input data, and this processing is repeated as long as there is input data.
A generator instruction variable generation unit 508 generates an instruction variable having a vector structure based on the correct answer class assigned to the learning data. A data generation unit 509 generates pseudo data having the same class as the learning data up to a predetermined target number of generation samples based on the generator received from the generator instruction variable generation unit 508 and the generator storage unit 507, and stores the data in the generated data storage unit 510.
An identifier reconstruction unit 511 updates the parameters of the identifier based on the generated data with the correct answer class from the stored generated data storage unit 510, and reconstructs the identifier. An identifier evaluation unit 512 evaluates the performance of the identifier updated from the identifier reconstruction unit 511, and in a case where the difference from the identifier performance stored in the identifier storage unit 505 is larger than a threshold, the identifier is updated to one with high performance by the replacement to the identifier in the identifier storage unit.
Next, in Step S705, the generator construction unit 506 learns the parameters of the generator using the learning data and the identifier received from the identifier storage unit 505, and performs a generator construction process of initializing the generator. Then, in Step S706, an identifier acquisition process of storing the initialized generator in the generator storage unit 507 is performed.
Next, in Step S707, the generator instruction variable generation unit 508 performs a generation instruction variable generation process of generating an instruction variable having a vector structure based on the correct answer class assigned to the learning data. Then, in Step S708, the data generation unit 509 performs a data generation process in which pseudo data having the same class as the learning data is generated up to a predetermined number of generation samples based on the received generator, and the pseudo data is stored in the generated data storage unit 510.
Next, in Step S709, the identifier reconstruction unit 511 performs an identifier reconstruction process in which the parameters of the identifier is updated based on the generated data with the correct answer class from the stored generated data storage unit 510, and the identifier is reconstructed. Then, in Step S710, the identifier evaluation unit 512 evaluates the performance of the identifier updated from the identifier reconstruction unit 511, and in a case where the difference from the identifier performance stored in the identifier storage unit 505 is larger than the threshold, replaces the updated identifier with an identifier stored in the identifier storage unit.
The identifier feature amount extraction process, the identification result prediction process, the prediction result evaluation process, the generation instruction variable generation process, the generator update process, the pseudo-learning data generation process, the pseudo-learning data selection process, and the identifier update process of the arithmetic device 802 include respectively the processes corresponding to the identifier feature amount extraction unit 102, the identification result prediction unit 103, the prediction result evaluation unit 104, the generator instruction variable generation unit 106, the generator update unit 107, the pseudo-learning data generation unit 108, the pseudo-learning data selection unit 111, and the identifier update unit 112 of
In addition, the correct answer assigning process, the generation instruction variable setting process, and the generation sample number setting process of the user interface 804 perform respectively the processes corresponding to the correct answer assigning unit 105, the generator instruction variable generation unit 106, and the generation sample number setting unit 109 of
The generation instruction variable setting process is a process in a case where the attributes of a generated sample are adjusted manually in order for the generator to generate a pseudo sample. From the prediction result evaluation process in the arithmetic device 802, the necessity of having a correct answer is determined based on the prediction result evaluation information stored in the recording device 803, and if necessary, the correct answer assigning process in the user interface 804 is performed. The correct answer information is stored in the recording device 803 as correct answer class information. In addition, information set in the generation instruction variable setting process and the generation sample number setting process is stored as a generation instruction variable of the recording device 803. The pseudo-learning data generation process is called, and pseudo-learning data is generated and stored in the recording device 803.
In the pseudo-learning data selection process, data having a high likelihood (reality of data) with real samples is selected manually or automatically from the pseudo-learning data, and the selected pseudo-learning data is stored in the recording device 803. The identifier update process calls the selected pseudo-learning data, learns the parameters of the identifier, and stores the parameters in the recording device 803 as identifier parameter information.
As described above, in this embodiment, the generator that generates a learning sample is configured so that the recognizer generates a sample with high difficulty or uncertainty in recognition, or generates data of a pattern that is not included in the learning sample. Thus, efficient and effective learning can be performed even in an environment where learning data is insufficient. In addition, the performance of the recognizer is effectively improved by screening hard-to-identify unlabeled samples from the test environment and updating the generators with efficient and accurate labeling online both the generator and the recognizer improves. It is also possible to generate samples from the updated generator to suit the test environment.
This makes it possible to provide an online recognition apparatus and an online recognition method that can improve recognition accuracy even when the number of learning samples is insufficient.
In
A learning data generation unit 904 generates pseudo-learning data using the attached correct answer information and the input image. The learning data generating unit 904 is configured as the generator update unit 107, the generator instruction variable generation unit 106, and the pseudo-learning data generation unit 108 in
In a case where it is unnecessary to assign a correct answer by the labeler, the result from the person detection/identification unit 902 is assigned to the input image as correct answer information, and the identifier update unit 112 updates the identifier using the correct answer information and the input image. The updated identifier is used for detecting and discriminating the next input data.
Thereby, the attribute configuration and position information of the person not captured from the camera, and an image that the identifier cannot identify well is complemented by the generated pseudo-learning data, and more robust detection and recognition can be realized by updating the identifier using these data.
In
In addition, a learning data generating unit 1004 is configured as the generator update unit 107, the generator instruction variable generation unit 106, and the pseudo-learning data generation unit 108 in
In
Using the correct answer information and the input data from the correct answer assigning unit 105, the learning data generation unit 1004 generates an image having different materials, shapes, and positions in the image for objects of the same category. This is because, in
In addition, in a case where it is unnecessary to assign a correct answer by the labeler, the result from the object detection/identification unit 1002 is assigned to the input image as correct answer information, and the identifier update unit 112 updates the identifier using the correct answer information and the input image.
In
The generation result display unit 1108 displays the results generated for each category in an order of the likelihood representing the reality of the image calculated at the time of generating the data by the pseudo-learning data generation unit 108 of the first embodiment. The pseudo-learning data input to the identifier update unit 112 in the first embodiment selects generated images listed in an order of likelihood up to a value set by a selection threshold. The threshold may be a likelihood or a number in the order of likelihood. In addition, in a case where the pseudo-learning data is selected manually to be updated to the identifier, an image which the check box displayed on the generation result display unit 1108 is checked is also used for learning. This example is an example, and the number of generation instruction variables that can be set may be increased or decreased as appropriate, or the current state may be displayed as texts instead of the input data display unit 1107 with a correct answer, or may be omitted. In addition, instead of the determination button 1106, a function of automatically transmitting the setting content at a certain timing may be provided. In addition, the pseudo-learning data generation setting is not limited to the method that can be freely input as in this example, and may be set by selecting from several candidates.
As described above, in this embodiment, the pseudo-learning data of the object reflected in the input image is generated and used for learning of the identifier, and the identifier can become more robust more quickly by updating the input data that cannot be detected online.
Number | Date | Country | Kind |
---|---|---|---|
2017-200825 | Oct 2017 | JP | national |
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/JP2018/029882 | 8/9/2018 | WO | 00 |