This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2021-086149, filed on May 21, 2021, the entire contents of which are incorporated herein by reference.
The embodiments discussed herein are related to a storage medium, a machine learning method, and an information processing device.
Pruning techniques are known as techniques for speeding up processing of an information processing device using a neural network. The pruning techniques can reduce a calculation amount during inference and speed up the processing while maintaining inference accuracy by deleting nodes, channels, layers, and the like that have small impact on the inference accuracy using a neural network.
International Publication Pamphlet No. WO 2020/149178, Japanese National Publication of International Patent Application No. 2019-522850, and U.S. Patent Application Publication No. 2019/0340493 are disclosed as related art.
According to an aspect of the embodiments, a non-transitory computer-readable storage medium storing a machine learning program that causes at least one computer to execute a process, the process includes acquiring a calculation amount of each partial network of a plurality of partial networks that is included in a neural network; determining a target channel based on the calculation amount of the each partial network and a scaling coefficient of each channel in a batch normalization layer included in the each partial network; and deleting the target channel.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention.
With the above-described techniques, the effect of pruning may not be sufficiently obtained depending on a configuration of the neural network or the like. For example, in a case of applying the technique to a complex neural network having a plurality of components, the accuracy is significantly reduced and the effect of speeding up is small due to the pruning.
In one aspect, an object is to provide a machine learning program, a machine learning method, and an information processing device capable of suppressing a decrease in inference accuracy and speeding up processing.
According to one embodiment, it is possible to suppress a decrease in inference accuracy and speed up processing.
Hereinafter, embodiments of a machine learning program, a machine learning method, and an information processing device disclosed in the present application will be described with reference to the drawings. Note that the embodiments are not limited to the present disclosure.
Here, a reference technique for pruning the above-described neural network will be described. In the reference technique, a channel to be pruned is determined using a scaling coefficient γ applied to an output of a batch normalization (BN) layer of a component. Note that, in a case where the component does not have a BN layer, the BN layer may be inserted into the component and the pruning may be performed, and a value output by the BN layer may be deleted as a reference value.
Moreover, scaling processing of calculating an output zout of each channel is executed by applying a scaling coefficient γ to the output z′ that is the normalized distribution, adding a bias 13, and performing shifting, using the following equation (2).
z
out
=Yz′+β [Math. 2]
Here, training by L1 regularization (Lasso regression) is applied to the scaling coefficient γ, and iterative training is executed. In the L1 regularization, a loss function L is calculated by the following equation (3). In the equation (3), the first term is an original loss function and the second term is an L1 regularization term. In the L1 regularization, g(r)=|γ| is used.
Therefore, in the present embodiment, an information processing device capable of improving a calculation speed without sacrificing the accuracy by, in pruning a neural network, dividing the neural network into components by a difference in function, and determining channels for reduction on the basis of a value of a scaling coefficient and a calculation amount of each component will be described.
The communication unit 11 executes communication with another device. For example, the communication unit 11 receives input data I. Furthermore, the communication unit 11 may also receive the input data I via an Internet line. The communication unit 11 may also cause the storage unit 12 to store the input data I as a training data database (DB) 13.
The storage unit 12 is a storage device that stores various types of data, programs executed by the control unit 20, and the like, and is implemented by, for example, a memory, a hard disk, or the like. For example, the storage unit 12 stores the training data DB 13, a machine learning model 14, and the like.
The training data DB 13 is a database that stores a plurality of training data used for training a machine learning model. For example, the training data DB 13 stores a moving image obtained by capturing person's movement, and is used by the information processing device 10 for recognizing movement of person's joints captured in the moving image for various purposes.
The machine learning model 14 is a model generated by training. For example, as the machine learning model 14, a model of a deep neural network (DNN), a convolution neural network (CNN), or the like can be adopted.
The control unit 20 is a processing unit that controls the entire information processing device 10 and is implemented by a processor or the like. For example, the control unit 20 has a division unit 21, a calculation amount calculation unit 22, a ratio calculation unit 23, a determination unit 24, and an execution unit 25.
The division unit 21 divides the neural network into partial networks. The neural network is configured by, for example, a plurality of components C1 to C9 illustrated in
The calculation amount calculation unit 22 calculates the calculation amount of each component divided by the division unit 21. For example, the calculation amount calculation unit 22 calculates the calculation amount of the component determined according to the function for each function. The calculation amount calculation unit 22 calculates a calculation amount CA of the function A by the following equation (4) and calculates a calculation amount CB of the function B by the following equation (5).
Cconv_i in the equations (4) and (5) represents the calculation amount of a convolution layer and can be calculated by the following equation (6). Note that k1 is the number of convolution layers in the function A, and k2 is the number of convolution layers in the function B.
[Math. 6]
C
conv_i=(kernel_size)2×input_channel×output_channel×(input_size/stride)2 (6)
The ratio calculation unit 23 calculates a ratio of the calculation amount obtained by dividing the calculated calculation amount for each function by a sum of the calculation amounts each calculated for each function. The ratio calculation unit 23 calculates the ratio CA/Ctotal of the calculation amount of the function A and the ratio CB/Ctotal of the calculation amount of the function B. Note that Ctotal is the following equation (7).
[Math. 7]
C
total
=C
A
+C
B (7)
The determination unit 24 determines a channel to be deleted on the basis of the calculated calculation amount for each function and the scaling coefficient of each channel in the BN layer included in the component. For example, the determination unit 24 determines a channel having a small sum of the ratio of the calculation amount and the scaling coefficient of each channel in the BN layer included in the component as the channel to be deleted. The determination unit 24 calculates an index γ′m by the equation (8) using a scaling coefficient γm of a channel m in the function A, and calculates an index γ′n by the equation (9) using a scaling coefficient γn of a channel n in the function B.
[Math.8]
γ′m=γm+α(CA/Ctotal) (8)
[Math.9]
γ′n=γn+α(CB/Ctotal) (9)
Moreover, the determination unit 24 determines the number of channels to be deleted according to the pruning rate. Note that, in a case where the component does not have a BN layer, the BN layer may be inserted into the component.
The execution unit 25 executes pruning based on the target channel. The pruning may also be processing of deleting the target channel or may also be processing of setting a weight of the target channel to be small or to zero. Furthermore, the execution unit 25 applies training by L1 regularization to the scaling coefficient. As a result, it is possible to execute the pruning as illustrated in
Next, the division unit 21 classifies the components by function (step S2). The division unit 21 classifies the component C1 having the function to extract features of an image into the function A, and classifies the components C2 to C9 having the functions to separate features of Heatmap (cmap) representing joint point-likeness and paf representing a connection relationship between joint points and improve the accuracy into the function B.
Thereafter, the calculation amount calculation unit 22 calculates the calculation amount of the component for each function, and the ratio calculation unit 23 calculates the ratio of the calculation amount for each function (step S3).
Moreover, the determination unit 24 calculates the index γ′ (γ′m, γ′n) of each channel (step S4). Note that α=0.12 is set here, but the value of a is not particularly limited.
Next, the determination unit 24 sets the pruning rate (step S5). For example, the determination unit 24 sets the pruning rate to each of the predetermined values (0%, 10%, 20%, and 30%).
Moreover, the determination unit 24 sorts the indexes γ′ (γ′m, γ′n) of the respective channels (arranges the indexes in descending order of absolute values) and determines the number of channels of target j to be deleted according to the pruning rate (step S6). For example, in a case where the number of channels is N and the pruning rate is 10%, the determination unit 24 determines N×10/100 channels as the channels to be deleted.
Then, the execution unit 25 executes pruning to delete the target channels of the determined number from the channel with the smallest absolute value among the indexes γ′ (γ′m, γ′n) of the respective sorted channels (step S7).
Thereafter, the execution unit 25 evaluates the accuracy (loss) and the speed (speedup ratio) (step S8).
Then, the execution unit 25 evaluates the accuracy (loss) and the speed (speedup ratio), selects an appropriate pruning rate, and executes training using the pruned neural network, thereby to generate the machine learning model 14 having high-speed processing with high accuracy (step S9). For example, the execution unit 25 selects the appropriate pruning rate, deletes the channels, executes training with the channels deleted, and generates the machine learning model 14, thereby suppressing deterioration of the inference accuracy by the generated machine learning model 14 and speeding up the processing. For example, by applying this machine learning model 14 to trt-pose using Jetson nano, it is possible to recognize the movement of person's joints accurately at high speed.
As described above, according to the embodiment, by determining the channels to be pruned using the index γ′ obtained by adding the value related to the calculation amount to the scaling coefficient γ, the loss can be reduced and the processing can be speeded up as compared with the reference technique.
The data examples, numerical examples, component types and numbers, function types and numbers, specific examples, and the like used in the above embodiment are merely examples and can be arbitrarily changed. For example, the division unit 21 may also classify the components into two or more functions.
Pieces of information including a processing procedure, a control procedure, a specific name, various sorts of data, and parameters described above or illustrated in the drawings may be optionally changed unless otherwise noted.
Furthermore, each component of each device illustrated in the drawings is functionally conceptual and does not necessarily have to be physically configured as illustrated in the drawings. For example, specific forms of distribution and integration of each device are not limited to those illustrated in the drawings. For example, the whole or a part of the device may be configured by being functionally or physically distributed or integrated in optional units according to various loads, usage situations, or the like.
Moreover, all or any part of individual processing functions performed in each device may be implemented by a central processing unit (CPU) and a program analyzed and executed by the corresponding CPU, or may be implemented as hardware by wired logic.
The communication device 10a is a network interface card or the like and communicates with another device. The HDD 10b stores a program that activates the functions illustrated in
The processor 10d reads a program that executes processing similar to the processing of each processing unit illustrated in
As described above, the information processing device 10 operates as an information processing device that executes a machine learning method by reading and executing a program. Furthermore, the information processing device 10 may also implement functions similar to the functions of the above-described embodiments by reading the program described above from a recording medium by a medium reading device and executing the read program described above. Note that the program referred to in other embodiments is not limited to being executed by the information processing device 10. For example, the embodiments may be similarly applied to a case where another computer or server executes the program, or a case where these cooperatively execute the program.
This program may be distributed via a network such as the Internet. Furthermore, this program may be recorded in a computer-readable recording medium such as a hard disk, flexible disk (FD), compact disc read only memory (CD-ROM), magneto-optical disk (MO), or digital versatile disc (DVD), and may be executed by being read from the recording medium by a computer.
All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
2021-086149 | May 2021 | JP | national |