Patent Application
20230351264
The present invention relates to a storage medium, an accuracy calculation method, and an information processing device.
As a machine learning model generated through machine learning, a white box type model of which determination basis is easily explained or a black box type model of which determination basis is hard to be explained are known. For example, as a machine learning algorithm used for the white box type model, a decision tree, a random forest, logistic regression, and the like are known, and as a machine learning algorithm used for the black box type model, a neural network, deep learning, and the like are known.
In a case where such a machine learning algorithm is used, it is general to select a method from among all methods in a case where only accuracy is important. It is general to use a white box type method in a case where explanatory is particularly important. On the other hand, by knowing in advance whether or not a useful effect can be obtained with an optimal method or with which machine learning algorithm when performing machine learning using data in a tabular format or the like, efficiency, accuracy, or the like of machine learning are enhanced. In recent years, various tools that automatically perform model calculation or the like have been known.
Patent Document 1: Japanese Laid-open Patent Publication No. 2017-4509, Patent Document 2: Japanese Laid-open Patent Publication No. 2020-46888.
According to an aspect of the embodiments, a non-transitory computer-readable storage medium storing an accuracy calculation program that causes at least one computer to execute a process, the process includes acquiring data item that corresponds to a generation condition based on an input dataset that is a set of input data that includes at least one data item and the generation condition regarding a correspondence relationship between a combination of feature and label information, as selection target rules; acquiring accuracy of each of a plurality of machine learning algorithms, by using a dataset that holds one data item used for a selection rule that is a selection target rule that satisfies a certain condition among the selection target rules; and outputting the accuracy.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention.
In a case where a machine learning algorithm is selected using the tool described above or the like, it is difficult to determine whether or not model comparison is appropriate, and reliability of the selection result is low.
For example, since a data type or a machine learning algorithm supported by the tool differs, there is a case where an optimal machine learning algorithm is not included in selection candidates. Furthermore, although it is considered to perform selection by executing the plurality of tools, there is a case where different machine learning algorithms are determined to be the best by a narrow margin. However, there is a case where determination basis of the narrow margin is unknown and determination results of the respective tools are not compatible.
In one aspect, an object is to provide an accuracy calculation program, an accuracy calculation method, and an information processing device that can optimize accuracy comparison of a plurality of machine learning algorithms.
According to one embodiment, it is possible to optimize accuracy comparison of a plurality of machine learning algorithms.
Hereinafter, embodiments of an accuracy calculation program, an accuracy calculation method, and an information processing device according to the present invention will be described in detail with reference to the drawings. Note that the embodiments do not limit the present invention. Furthermore, the embodiments may be appropriately combined with each other in a range without contradiction.
[Description of Information Processing Device]
In recent years, various tools that automatically perform model calculation have been known. Each of such automated tools shortens an execution time using a unique method without trying every hyperparameter of every machine learning algorithm, further limits the execution time, and randomly searches for a parameter. Therefore, it is difficult for the automated tool to optimize accuracy comparison of the plurality of machine learning algorithms.
Therefore, the information processing device 10 according to the first embodiment can perform sufficient comparison verification of each model, by covering a possibility of parameter search at the time of model calculation using the machine learning algorithm within a certain time period to the maximum.
Specifically, as illustrated in
For example, the information processing device 10 extracts a combination of columns (rule) of data items corresponding to each hypothesis, about each hypothesis that is a set of rules describing that prediction is performed according to an objective variable. At this time, each rule (combination of columns) can be generated, for example, using the technology in Japanese Laid-open Patent Publication No. 2020-46888 and can include an importance (weight) corresponding to each hypothesis. Note that the weight can be calculated through logistic regression using each rule as training data.
Then, the information processing device 10 selects a combination of a plurality of columns corresponding to a preset specific condition, as the selection data, from among the plurality of extracted combinations of columns. Thereafter, the information processing device 10 can perform parameter search and evaluation of each machine learning algorithm using the selected combination of columns and outputs an execution result.
In this way, the information processing device 10 can search for a machine learning model with high accuracy under the same condition in many hypotheses and present the result to a user.
[Functional Configuration]
The communication unit 11 is a processing unit that controls communication with another device and, for example, is implemented by a communication interface or the like. For example, the communication unit 11 receives various instructions or the like from an administrator's terminal or the like and transmits an execution result or the like to the administrator's terminal or the like.
The output unit 12 is a processing unit that outputs various types of information and is implemented, for example, by a display, a touch panel, or the like. For example, the output unit 12 displays and outputs an evaluation result generated by the control unit 20 to be described later.
The storage unit 13 is an example of a storage device that stores various types of data, programs, or the like and is implemented, for example, by a memory, a hard disk, or the like. The storage unit 13 stores an input data DB 14 and an evaluation result DB 15.
The input data DB 14 is an example of a dataset used to evaluate a machine learning algorithm.
In this way, an input dataset stored in the input data DB 14 is data in a tabular format in which each data item such as the data name, the feather, or the label is set as a column and input data including each data item (set value or the like) is set as a row. Note that, in the present embodiment, as illustrated in the lower diagram in
The evaluation result DB 15 is a database that stores an evaluation result of each machine learning algorithm, executed by the control unit 20 to be described later. The evaluation result DB 15 stores the evaluation result in association with an executed time stamp, a user name who has requested to execute evaluation, information about input data used for evaluation, and a machine learning algorithm to be evaluated. Note that, since the stored information will be described later, detailed description thereof is omitted here.
The control unit 20 is a processing unit that performs overall control of the information processing device 10 and, for example, is implemented by a processor or the like. The control unit 20 includes a generation unit 21, a selection unit 22, an evaluation execution unit 23, and a display control unit 24. Note that the generation unit 21, the selection unit 22, the evaluation execution unit 23, and the display control unit 24 can be implemented, for example, as an electronic circuit included in a processor or a process to be executed by the processor.
The generation unit 21 is a processing unit that extracts a data item corresponding to a generation condition (extraction condition) or a combination of data items, as selection target data (rule), from the input data DB 14, according to the generation condition based on a correspondence relationship between a combination of one or a plurality of features and label information. Here, it can be said that the extraction condition is a condition used to generate a rule.
Specifically, the generation unit 21 executes preprocessing for discretizing input data, such as one-hot expression for a category column and discretization using minimum description length principle (MDLP) for a numerical value column. Subsequently, the generation unit 21 extracts (generate) a rule corresponding to a hypothesis from the preprocessed input data. Here, the hypothesis is a set of the plurality of rules, and each rule is a combination of columns. Furthermore, since the rule to be extracted (combination of columns) differs depending on setting of a minimum value of a sample ratio (hereinafter, referred to as column extraction parameter or simply referred to as parameter), a plurality of rules is set between “0.0 to 1.0” and extracted.
In the example in
For example, “A, 0.5” of the hypothesis 1 indicates that a column “item A” is extracted as data belonging to the hypothesis 1 and a weight in the hypothesis 1 is “0.5”. Furthermore, “C & E, 0.5” of the hypothesis 3 indicates that an AND condition of a column “item C” and a column “item E” is extracted as data belonging to the hypothesis 3 and a weight in the hypothesis 3 is “0.5”. Here, the “item” is not a value of data itself and is an item that is a superordinate concept of the data. That is, referring to the example in
Then, the generation unit 21 outputs each hypothesis generated in this way and the “rule, weight” belonging to each hypothesis to the selection unit 22. Note that, here, an example has been described where a hypothesis is generated and a rule corresponding to each hypothesis is extracted. However, the present invention is not limited to this. For example, the “rule, weight” can be generated in advance by an administrator or the like, and it is possible to extract a “rule” used for machine learning in the past and calculate and use a weight (importance) from a use ratio of the “rule”.
Returning to
Subsequently, the selection unit 22 extracts data (selection data) corresponding to the selected rule, from the input data. For example, as illustrated in
The evaluation execution unit 23 is a processing unit that evaluates a plurality of machine learning algorithms to be evaluated, using each dataset selected by the selection unit 22. Specifically, the evaluation execution unit 23 performs machine learning using data belonging to each selected rule with all the machine learning algorithms (machine learning method), searches for all parameters, and performs evaluation without performing optimization. Note that, for the parameter search performed here, general machine learning can be adopted.
Similarly, the evaluation execution unit 23 performs machine learning using each machine learning algorithm using “item A, item F, label” corresponding to two that is the number of columns as training data and acquires the accuracy “0.8” of the machine learning model X, the accuracy “0.8” of the machine learning model Y, and the accuracy “0.75” of the machine learning model Z.
Furthermore, the evaluation execution unit 23 performs machine learning using each machine learning algorithm using “item B, item F, item G, label” corresponding to three that is the number of columns as training data and acquires the accuracy “0.5” of the machine learning model X, the accuracy “0.6” of the machine learning model Y, and the accuracy “0.68” of the machine learning model Z.
Furthermore, the evaluation execution unit 23 performs machine learning using each machine learning algorithm using “item C, item D, item G, label” corresponding to three that is the number of columns as training data and acquires the accuracy “0.75” of the machine learning model X, the accuracy “0.5” of the machine learning model Y, and the accuracy “0.6” of the machine learning model Z.
Thereafter, the evaluation execution unit 23 calculates pseudo accuracy based on the result of machine learning and the weight of the rule and searches for a machine learning algorithm with high accuracy in many hypotheses.
Similarly, the evaluation execution unit 23 calculates each pseudo accuracy “(accuracy×weight of A & F)+(accuracy×weight of C & E)”, using the respective weights of the rules “A & F” and “C & E” selected from among the rules of the hypothesis 3. In other words, the evaluation execution unit 23 calculates “0.8×0.5+0.5×0.25=0.525” as “pseudo accuracy of machine learning algorithm X”, calculates “0.8×0.5+0.6×0.25=0.55” as “pseudo accuracy of machine learning algorithm Y”, and calculates “0.75×0.5+0.7×0.25=0.55” as “pseudo accuracy of machine learning algorithm Z”.
Similarly, the evaluation execution unit 23 calculates each pseudo accuracy “(accuracy×weight of C & D & F)+(accuracy×weight of C & E)+(accuracy×weight of B & F & G)”, using the respective weights of the rules “C & D & F”, “C & E”, and “B & F & G” selected from among the rules of the hypothesis 4. In other words, the evaluation execution unit 23 calculates “0.75×0.5+0.5×0.4+0.5×0.25=0.7” as “pseudo accuracy of machine learning algorithm X”, calculates “0.5×0.5+0.6×0.4+0.6×0.25=0.64” as “pseudo accuracy of machine learning algorithm Y”, and calculates “0.6×0.5+0.7×0.4+0.68×0.25=0.75” as “pseudo accuracy of machine learning algorithm Z”.
Note that the evaluation execution unit 23 determines that the hypothesis 2 is a dataset that can be sufficiently classified with a single column (with no difference depending on model).
As a result of these, the evaluation execution unit 23 determines that a model with high accuracy in many hypotheses as an appropriate machine learning model. In other words, the evaluation execution unit 23 evaluates the “machine learning algorithm Z” for the hypothesis 1, the “machine learning algorithm Y” and the “machine learning algorithm Z” for the hypothesis 3, and the “machine learning algorithm Z” for the hypothesis 4 as appropriate models and outputs the models to the display control unit 24.
The display control unit 24 is a processing unit that displays and outputs an execution result by the evaluation execution unit 23. Specifically, the display control unit 24 displays and outputs each hypothesis and the evaluation result by the evaluation execution unit 23 to the output unit 12, stores each hypothesis and the evaluation result in the storage unit 13, and transmits each hypothesis and the evaluation result to the administrator's terminal.
For example, in the “optimal model”, the “machine learning model Z” determined by the evaluation execution unit 23 and content such as “the machine learning model Z achieves the highest pseudo accuracy in combinations (1) to (4) of columns correlated with a label column of data” or the like, as a basis are described. Furthermore, (1) to (4) respectively correspond to the hypotheses 1 to 4 described above, and the calculation result of the pseudo accuracy described above is written under (1) or the like.
[Flow of Processing]
Then, the selection unit 22 selects the rule based on the specific condition (S104). Thereafter, the evaluation execution unit 23 performs machine learning with each machine learning model using the selected selection data (selection rule) (S105) and performs accuracy prediction (pseudo accuracy) of each machine learning model using a machine learning result (S106). Then, the display control unit 24 outputs a result of the accuracy prediction (S107).
As described above, the information processing device 10 can perform sufficient comparison verification of each model in divided data by sufficiently searching only data that is meaningful for classification for a parameter. Furthermore, by presenting the result, the information processing device 10 can enhance user's satisfaction for the presented model.
For example, in a case where the tool described above or the like is used, it is almost impossible to search all pieces of data for all parameters, and a calculation time can be reduced by using only a part of original data. However, the number of combinations of selected parts of the original data exponentially increases as the number of columns increases, and this is not realistic. For example, if the original data includes 200 columns, the number of combinations obtained by selecting only three columns from the original data is “(200×199×198)/(3×2×1)=1313400”, which is not realistic.
On the other hand, the information processing device 10 according to the first embodiment can perform all search (do not optimize) in parameter adjustment, by narrowing the input data, not by optimizing the parameter adjustment, that is, not executing only a part of all the parameters. As a result, the information processing device 10 can calculate and evaluate the pseudo accuracy that is highly reliable although being pseudo. Therefore, it is possible to evaluate the machine learning model at high speed and with high accuracy.
However, in the first embodiment, an example has been described where the set of the rules corresponding to the specific condition “the number of columns is equal to or more than two” is extracted from the rules of the hypothesis (combination of columns). However, the specific condition is not limited to this. This specific condition may be a condition for extracting a rule included in a certain number of hypotheses, from among hypotheses that is described to belong to a certain class at a certain sample ratio or higher. Therefore, in the second embodiment, as another example of the specific condition, a condition “extracting a rule included in 70% or more of all hypotheses” will be described. Note that, it is assumed that the functional configuration or the like be similar to those in the first embodiment.
First, a generation unit 21 executes preprocessing on input data and extracts a rule corresponding to a hypothesis from the preprocessed input data.
In the example in
Next, a selection unit 22 selects selection data that satisfies the specific condition “extracting a rule included in 70% or more of all hypotheses” from among the rules of each hypothesis generated by the generation unit 21.
Then, the selection unit 22 extracts data (selection data) corresponding to the selected rule, from the input data. For example, as illustrated in
Next, an evaluation execution unit 23 evaluates a plurality of machine learning algorithms to be evaluated, using each dataset selected by the selection unit 22.
Similarly, the evaluation execution unit 23 performs machine learning using each machine learning algorithm using “item C, item E, label” corresponding to two that is the number of columns as training data and acquires the accuracy “0.7” of the machine learning model X, the accuracy “0.2” of the machine learning model Y, and the accuracy “0.75” of the machine learning model Z.
Next, the evaluation execution unit 23 calculates pseudo accuracy based on the result of machine learning and the weight of the rule and searches for a machine learning algorithm with high accuracy in many hypotheses.
Similarly, the evaluation execution unit 23 calculates each pseudo accuracy “(accuracy×weight of B)”, using the weight of the rule “B” selected from among the rules of the hypothesis 2. In other words, the evaluation execution unit 23 calculates “0.15×0.4=0.06” as “pseudo accuracy of machine learning algorithm X”, calculates “0.3×0.4=0.12” as “pseudo accuracy of machine learning algorithm Y”, and calculates “0.28×0.4=0.112” as “pseudo accuracy of machine learning algorithm Z”.
Furthermore, the evaluation execution unit 23 calculates pseudo accuracy “(accuracy×weight of B)+(accuracy×weight of C & E)” using the accuracy of each machine learning algorithm and each weight of the rules “B” and “C & E” selected from among the rules of the hypothesis 3. In other words, the evaluation execution unit 23 calculates “0.15×0.4+0.7×0.25=0.235” as “pseudo accuracy of machine learning algorithm X”, calculates “0.3×0.4+0.2×0.25=0.17” as “pseudo accuracy of machine learning algorithm Y”, and calculates “0.28×0.4+0.75×0.25=0.2995” as “pseudo accuracy of machine learning algorithm Z”.
Furthermore, the evaluation execution unit 23 calculates each pseudo accuracy “(accuracy×weight of C & E)”, using the weight of the rule “C & E” selected from among the rules of the hypothesis 4. In other words, the evaluation execution unit 23 calculates “0.7×0.25=0.175” as “pseudo accuracy of machine learning algorithm X”, calculates “0.2×0.25=0.05” as “pseudo accuracy of machine learning algorithm Y”, and calculates “0.75×0.25=0.1875” as “pseudo accuracy of machine learning algorithm Z”.
As a result of these, the evaluation execution unit 23 determines that a model with high accuracy in many hypotheses as an appropriate machine learning model. In other words, the evaluation execution unit 23 evaluates the “machine learning algorithm Z” for the hypotheses 1, 3, and 4 and the “machine learning algorithm Y” for the hypothesis 2 as appropriate models, and outputs the models to a display control unit 24.
Finally, the display control unit 24 displays and outputs the execution result by the evaluation execution unit 23.
For example, in the “optimal model”, the “machine learning model Z” and content such as “the machine learning model Z achieves the highest pseudo accuracy in combinations (1) to (4) of columns correlated with a label column of data” or the like, as a basis are described. Furthermore, (1) to (4) respectively correspond to the hypotheses 1 to 4 described above, and the calculation result of the pseudo accuracy described above is written under (1) or the like.
Note that, since various conditions other than the specific condition “extracting a rule included in 70% or more of all hypotheses” described in the second embodiment can be set, an example of the conditions will be described. For example, as the specific condition, “top two hypotheses having a high ratio to be included, among all hypotheses” can be set.
In such a state, the selection unit 22 selects a rule corresponding to the specific condition “top two hypotheses having a high ratio to be included, among all hypotheses”, from among the rules included in each of the four hypotheses.
In other words, since a rule “A” is included in the hypotheses 1 and 2 of the four hypotheses, the selection unit 22 calculates “(2/4)×100=50%”, and since a rule “B” is included in the hypotheses 1 to 3 of the four hypotheses, the selection unit 22 calculates “(3/4)×100=75%”. Since a rule “C & E” is included in the hypotheses 1 to 3 of the four hypotheses, the selection unit 22 calculates “(3/4)×100=75%”, and since a rule “G” is included in the hypotheses 1 and 2 of the four hypotheses, the selection unit 22 calculates “(2/4)×100=50%”.
Furthermore, since a rule “E” is included in the hypothesis 2 of the four hypotheses, the selection unit 22 calculates “(1/4)×100=25%”, and since a rule “A & F” is included in the hypothesis 3 of the four hypotheses, the selection unit 22 calculates “(1/4)×100=25%”. Since a rule “C & D & F” is included in the hypothesis 4 of the four hypotheses, the selection unit 22 calculates “(1/4)×100=25%”, and since a rule “B & F & G” is included in the hypothesis 4 of the four hypotheses, the selection unit 22 calculates “(1/4)×100=25%”.
As a result of these, the selection unit 22 selects the rules “B” and “C & E” as the rule corresponding to the specific condition “top two hypotheses having a high ratio to be included, among all hypotheses”. Note that, since subsequent processing is similar to that in the first embodiment and the second embodiment, description of the processing will be omitted.
As described above, even in a case where a large number of rules included in the hypothesis are generated, the information processing device 10 can appropriately narrow rules for accuracy evaluation, by designating the specific condition. Furthermore, since the information processing device 10 can arbitrarily set the specific condition, it is possible to provide accuracy evaluation environment suitable for needs of each user such as a user who wants to use many rules with emphasis on accuracy or a user who has used rules narrowed with emphasis on speed.
Incidentally, while the embodiments of the present invention have been described above, the present invention may be implemented in a variety of different modes in addition to the embodiments described above.
[Numerical Values, etc.]
The number of hypotheses, the number of rules, the number of machine learning algorithms, a hypothesis generation technique, a rule generation technique, various numerical value examples, various thresholds, information described on the output screen, or the like described in the embodiments above are merely examples and can be arbitrarily changed. Note that, as the weight of the rule, for example, in addition to a weight calculated through logistic regression, various values obtained when a rule is extracted can be adopted. If merely an example is exemplified, “chi-square”, “normalized mutual information”, or the like can be adopted. Note that the “chi-square” is a value obtained by calculating a frequency of a positive example or a negative example in the entire sample as an expected value (theoretical value), it is evaluated whether or not a rule and a positive example or a negative example are independent, using a frequency of the positive example or the negative example in a sample that hits the rule as an observation value. The “normalized mutual information” indicates how many percent of an information amount of a label is obtained, by knowing information indicating whether or not a rule is hit.
[System]
Pieces of information including the processing procedure, control procedure, specific name, various types of data and parameters described above or illustrated in the drawings may be arbitrarily modified unless otherwise noted.
In addition, each component of each device illustrated in the drawings is functionally conceptual and does not necessarily have to be physically configured as illustrated in the drawings. In other words, specific forms of distribution and integration of the devices are not limited to those illustrated in the drawings. That is, all or a part of the devices may be configured by being functionally or physically distributed or integrated in optional units according to various loads, use situations, or the like. Note that the generation unit 21 is an example of a generation unit, and the selection unit 22 and the evaluation execution unit 23 are examples of a calculation unit. Furthermore, the hypothesis is an example of a generation condition, and the selection target rule is an example of a rule.
Moreover, all or any part of individual processing functions performed in each device may be implemented by a central processing unit (CPU) and a program analyzed and executed by the CPU, or may be implemented as hardware by wired logic.
[Hardware]
Next, a hardware configuration example of each device described in the embodiments above will be described. Note that, since each device have a similar hardware configuration, here, the information processing device 10 will be described as an example.
The communication device 10a is a network interface card or the like, and communicates with another server. The HDD 10b stores a program that activates the functions illustrated in
The processor 10d reads a program that executes processing similar to the processing of each processing unit illustrated in
As described above, the information processing device 10 operates as an information processing device that executes various processing methods by reading and executing the program. Furthermore, the information processing device 10 may also implement functions similar to the functions of the above-described examples by reading the above-described program from a recording medium by a medium reading device and executing the above-described read program. Note that other programs referred to in the embodiments are not limited to being executed by the information processing device 10. For example, the present invention can be similarly applied also to a case where another computer or server executes the program, or a case where such computer and server cooperatively execute the program.
This program can be distributed via a network such as the Internet. Furthermore, this program may be recorded in a computer-readable recording medium such as a hard disk, a flexible disk (FD), a compact disc read only memory (CD-ROM), a magneto-optical disk (MO), or a digital versatile disc (DVD), and may be executed by being read from the recording medium by a computer.
All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
This application is a continuation application of International Application PCT/JP2021/004189 filed on Feb. 4, 2021 and designated the U.S., the entire contents of which are incorporated herein by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2021/004189 | Feb 2021 | US |
| Child | 18347681 | US |
