APPARATUS AND METHOD FOR OPTIMIZING QUANTIZED MACHINE-LEARNING ALGORITHM

Abstract

Disclosed herein are an apparatus and method for optimizing a quantized machine-learning algorithm. The apparatus includes one or more processors and executable memory for storing at least one program executed by the one or more processors. The at least one program sets the learning rate of the quantized machine-learning algorithm using at least one of an Armijo rule and golden search methods, calculates a quantized orthogonal compensation search vector from the search direction vector of the quantized machine-learning algorithm, compensates for the search performance of the quantized machine-learning algorithm using the quantized orthogonal compensation search vector, and calculates an optimized quantized machine-learning algorithm using the learning rate and the quantized machine-learning algorithm, the search performance of which is compensated for.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Korean Patent Applications No. 10-2019-0138793, filed Nov. 1, 2019, and No. 10-2020-0046724, filed Apr. 17, 2020, which are hereby incorporated by reference in their entireties into this application.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates generally to machine-learning and signal-processing technology, and more particularly to technology for optimizing a quantized machine-learning algorithm.

2. Description of the Related Art

In conventional machine-learning and nonlinear-signal-processing technology, a signal-processing operation is performed based on floating-point operations. However, the conventional machine-learning and nonlinear-signal-processing technology is not suitable for fields in which small and lightweight hardware is required because such technology uses multiple operation modules in order to provide real-time operation and because the size and complexity of a computation module provided for floating-point operations are greater than the size and complexity of a computation module provided for integer operations.

Accordingly, research on quantization of processing data is underway in various engineering fields. Quantization of processing data may confer many advantages, such as a decrease in the number of bits, calculation speed improvement, and availability improvement, when an engineering solution is implemented. For example, because a learning equation of a quantized domain is implemented by updating the least significant bit of a parameter, it is the same as applying a fixed learning speed to an update parameter. However, a general stochastic steepest-descent algorithm having a fixed step size cannot avoid performance degradation because of convergence on the optimum point in weak topology, such as convergence with first-order distribution.

Meanwhile, Korean Patent Application Publication No. 10-2018-0043154, titled “Method and apparatus for neural network quantization” relates to a neural network quantization method, and discloses a method for quantizing the parameters of a neural network, which includes determining the diagonals of a second-order partial derivative matrix (Hessian matrix) of the loss function of the network parameters of the neural network and assigning Hessian weights to the network parameters using the determined diagonals as part of the step of quantizing the network parameters.

SUMMARY OF THE INVENTION

An object of the present invention is to implement an optimization algorithm capable of minimizing a quantization error in machine-learning and nonlinear-signal-processing fields using quantization and exhibiting excellent performance on lightweight hardware.

Another object of the present invention is to implement a machine-learning algorithm capable of providing sufficient optimization performance even on low-performance hardware.

In order to accomplish the above objects, an apparatus for optimizing a quantized machine-learning algorithm according to an embodiment of the present invention may include one or more processors and executable memory for storing at least one program executed by the one or more processors. The at least one program may set the learning rate of the quantized machine-learning algorithm using at least one of an Armijo rule and golden search methods, calculate a quantized orthogonal compensation search vector from the search direction vector of the quantized machine-learning algorithm, compensate for the search performance of the quantized machine-learning algorithm using the quantized orthogonal compensation search vector, and calculate an optimized quantized machine-learning algorithm using the learning rate and the quantized machine-learning algorithm, the search performance of which is compensated for.

Here, the at least one program may set the learning rate through a learning-rate-setting function predefined by the Armijo rule using the gradient vector of the objective function of the search direction vector.

Here, the at least one program may set any one of a first candidate value, acquired by increasing the minimum candidate value of the learning rate by a golden ratio, and a second candidate value, acquired by decreasing the maximum candidate value of the learning rate by the golden ratio, as the learning rate.

Here, the at least one program may set any one of the first candidate value and the second candidate value as the learning rate when the difference value between the first candidate value and the second candidate value is equal to or less than a preset value.

Here, the at least one program may select a vector in the direction orthogonal to the direction opposite the largest component vector of the search direction vector and quantize the selected vector, thereby calculating the quantized orthogonal compensation search vector.

Here, when the solution of the quantized machine-learning algorithm is not able to escape from a local minimum point, the at least one program may make the solution escape from the local minimum point using the quantized orthogonal compensation search vector.

Also, in order to accomplish the above objects, a method for optimizing a quantized machine-learning algorithm, performed by an apparatus for optimizing the quantized machine-learning algorithm, according to an embodiment of the present invention may include setting the learning rate of the quantized machine-learning algorithm using at least one of an Armijo rule and golden search methods; calculating a quantized orthogonal compensation search vector from the search direction vector of the quantized machine-learning algorithm and compensating for the search performance of the quantized machine-learning algorithm using the quantized orthogonal compensation search vector; and calculating an optimized quantized machine-learning algorithm using the learning rate and the quantized machine-learning algorithm, the search performance of which is compensated for.

Here, setting the learning rate may be configured to set the learning rate through a learning-rate-setting function predefined by the Armijo rule using the gradient vector of the objective function of the search direction vector.

Here, setting the learning rate may be configured to set any one of a first candidate value, acquired by increasing the minimum candidate value of the learning rate by a golden ratio, and a second candidate value, acquired by decreasing the maximum candidate value of the learning rate by the golden ratio, as the learning rate.

Here, setting the learning rate may be configured to set any one of the first candidate value and the second candidate value as the learning rate when the difference value between the first candidate value and the second candidate value is equal to or less than a preset value.

Here, compensating for the search performance may be configured to select a vector in the direction orthogonal to the direction opposite the largest component vector of the search direction vector and to quantize the selected vector, thereby calculating the quantized orthogonal compensation search vector.

Here, compensating for the search performance may be configured such that, when the solution of the quantized machine-learning algorithm is not able to escape from a local minimum point, the solution is made to escape from the local minimum point using the quantized orthogonal compensation search vector.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the present invention will be more clearly understood from the following detailed description, taken in conjunction with the accompanying drawings, in which:

FIG. 1 is a flowchart illustrating a method for optimizing a quantized machine-learning algorithm according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating an optimization algorithm using a quantized Armijo rule according to an embodiment of the present invention;

FIG. 3 is a flowchart illustrating an optimization algorithm using a golden search according to an embodiment of the present invention;

FIG. 4 is a flowchart illustrating an optimization algorithm using a compensated search vector according to an embodiment of the present invention; and

FIG. 5 is a view illustrating a computer system according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will be described in detail below with reference to the accompanying drawings. Repeated descriptions and descriptions of known functions and configurations that have been deemed to unnecessarily obscure the gist of the present invention will be omitted below. The embodiments of the present invention are intended to fully describe the present invention to a person having ordinary knowledge in the art to which the present invention pertains. Accordingly, the shapes, sizes, etc. of components in the drawings may be exaggerated in order to make the description clearer.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a flowchart illustrating a method for optimizing a quantized machine-learning algorithm according to an embodiment of the present invention.

Table 1 shows mathematical symbols used for explaining a quantized machine-learning algorithm according to an embodiment of the present invention.

TABLE 1
mathematical
symbol  description
R       real number space
Z       integer space
N       natural number space, 0 is not included
Rn      n-dimensional vector space of real numbers
Zn      n-dimensional vector space of integers
R(a, b] {x|∀x ∈ R, a < x ≤ b}
R[a, b) {x|∀x ∈ R, a ≤ x < b}
Z[a, b] {x|∀x ∈ Z, a ≤ x ≤ b}
Z(a, b) {x|∀x ∈ Z, a < x < b}
δ(x)    Dirac-Delta function, if x = 0, δ(x) = 1, and if x ≠ 0,
        δ(x) = 0
Qpx     average value of x based on probability distribution for
        quantization error caused by quantization coefficient Qp

The definition of learning quantization and main quantization operations according to an embodiment of the present invention may be described as follows.

First, in order to define quantization of a variable x∈R, a round-off operation for obtaining an integer may be defined as shown in Equation (1).

x≡└x┘+ϵ (ϵ∈R[0,1))   (1)

In Equation (1), the symbol └x┘∈Z indicates a round-off operation, which may be defined as the integer that is nearest to x, among integers less than x. Using this, the Gauss symbol may be defined as shown in Equation (2).

[x]+└x+0.5┘=x+0.5−∇x+ϵ   (2)

When the operation of rounding x to the nearest number is defined as [x] using Equation (2), the round-off error, ϵ, may be defined as ϵ∈R(−0.5, 0.5┘. Therefore, when an arbitrary integer n E Z is given, the relationships of addition and multiplication may be represented as shown in Equation (3).

[x+n]=└x+n+0.5┘=└x+0.5+n┘=└x+0.5┘+n=[x]+n[n·[x]]=└n−└x+0.5┘+0.5┘=n·└x+0.5┘=n·[x], [n·x]≠n[x]   (3)

Also, when an arbitrary real-number sequence {x_k} (∀k∈N, x_k∈R) is given, if the round-off operation of each x_k is represented using ϵ_k as in Equation (1), Equation (4) may be obtained.

$\begin{array}{cc}\lfloor \sum _{k=1}^n{x}_k\rfloor =\lfloor \sum _{k=1}^n\left(\lfloor x_k\rfloor +{ϵ}_k\right)\rfloor =\sum _{k=1}^n\lfloor x_k\rfloor +\lfloor \sum _{k=1}^n{ϵ}_k\rfloor & \left(4\right)\end{array}$

Using Equations (1) and (2), a quantization operation may be defined as shown in Equation (5).

$\begin{array}{cc}{x}^Q\stackrel{\Delta }{=}\frac{1}{Q_p}\lfloor Q_p·\left(x+0.5\right)\rfloor & \left(5\right)\end{array}$

In Equation (5), Q_p denotes a quantization coefficient, which may determine the level of quantization. For example, when quantization to a fixed-point number at the level of 10⁻³ is attempted, Q_p=10³ may be set. For convenience, the quantization coefficient may be set to a positive integer (Q_p∈Z, Q_p>0). In order to represent quantization of a specific number without the Gauss symbol, a round-off error is used, whereby Equation (5) may be replaced with Equation (6).

$\begin{array}{cc}{x}^Q=\frac{1}{Q_p}\left[Q_p·x\right]=\frac{1}{Q_p}\lfloor \mathrm{Qp}·\left(x+0.5\right)\rfloor =\frac{1}{\mathrm{Qp}}\left(\mathrm{Qp}·\left(x+\varepsilon \right)\right)=x+\varepsilon & \left(6\right)\end{array}$

In Equation (6), x^Q satisfies x^Q∈R, but Q_px^Q=[Q_p·x]∈Z. That is, in the case of x^Q, quantization may be performed to obtain a fixed-point number format, rather than simply an integer.

In Equation (6), the range of the round-off error e may be represented as ε∈R(−0.5Q_p⁻¹,0.5Q_p⁻¹]=R(−5·(10Q_p)⁻¹,5·(10Q_p)⁻¹]. If Q_p is large enough that the average value for the distribution of the round-off error depending on Q_p satisfies E_Qpε=0, the relationship shown in Equation (7) may be satisfied for x_k, which is the sample data of x.

$\begin{array}{cc}\begin{array}{c}{𝔼}_{Q_p}{x}^Q\stackrel{\Delta }{=}\underset{N\to \infty }{\lim }\frac{1}{N}\sum _{k=0}^{N-1}{x}_k^Q=\underset{N\to \infty }{\lim }\frac{1}{N}\sum _{k=0}^{N-1}\left(x_k+{\varepsilon }_k\right)\\ ={𝔼}_{Q_p}x+{𝔼}_{Q_p}\varepsilon ={𝔼}_{Q_p}x.\end{array}\hspace{1em}& \left(7\right)\end{array}$

The addition and multiplication of quantized values have the characteristics shown in Equations (4) and (5). However, because the result of division may not be represented as a quantized value, the following method is applied. That is, when division of two integers x and a (x, a∈z) is represented using a quotient and a remainder, division of x by a may be represented as shown in Equation (8).

$\begin{array}{cc}\frac{x}{a}=\lfloor \frac{x}{a}\rfloor +\frac{1}{a}\left(x-a\lfloor \frac{x}{a}\rfloor \right)& \left(8\right)\end{array}$

In Equation (8), the quotient may be

$\lfloor \frac{x}{a}\rfloor ,$

and the remainder may be

$x-a\lfloor \frac{x}{a}\rfloor .$

When the Gaussian operation is applied to Equation in order to prove the quotient and the remainder, Equation (9) may be obtained.

$\begin{array}{cc}\begin{array}{c}\lfloor \frac{x}{a}\rfloor =\lfloor \lfloor \frac{x}{a}\rfloor +\frac{1}{a}\left(x-a\lfloor \frac{x}{a}\rfloor \right)\rfloor \\ =\lfloor \frac{x}{a}\rfloor +\lfloor \frac{1}{a}\left(x-a\lfloor \frac{x}{a}\rfloor \right)\rfloor \end{array}\hspace{1em}& \left(9\right)\end{array}$

Equation (9) shows that

$\left(x-a\lfloor \frac{x}{a}\rfloor \right)<a$

is satisfied.

Meanwhile, when the rounding operation is applied to Equation (8), Equation (10) may be obtained.

$\begin{array}{cc}\begin{array}{c}\left[\frac{x}{a}\right]=\lfloor \frac{x}{a}+0.5\rfloor \\ =\lfloor \lfloor \frac{x}{a}\rfloor +\frac{1}{a}\left(x-a\lfloor \frac{x}{a}\rfloor \right)+\frac{a}{2a}\rfloor \\ =\lfloor \frac{x}{a}\rfloor +\lfloor \frac{1}{a}\left(x-a\left(\lfloor \frac{x}{a}\rfloor -\frac{1}{2}\right)\right)\rfloor \end{array}\hspace{1em}& \left(10\right)\end{array}$

As a result the conditional expression shown in Equation 11 ma be obtained.

$\begin{array}{cc}\left[\frac{x}{a}\right]=\{\begin{array}{cc}\lfloor \frac{x}{a}\rfloor & x<a(\lfloor \frac{x}{a}\rfloor +\frac{1}{2}\rfloor )\Rightarrow a·\varepsilon >0\\ \lfloor \frac{x}{a}\rfloor +1& x\ge a(\lfloor \frac{x}{a}\rfloor +\frac{1}{2}\rfloor )\Rightarrow a·\varepsilon \le 0\end{array}& \left(11\right)\end{array}$

In Equation (11), the condition that the number added to

$\lfloor \frac{x}{a}\rfloor$

becomes 1 or 0 may be represented as shown in Equation (12). In order for the number to become 1 based on the definition of the Gauss symbol, the condition shown in Equation (12) should be satisfied.

$\begin{array}{cc}x-a\left(\lfloor \frac{x}{a}\rfloor -0.5\right)\ge a& \left(12\right)\\ \Rightarrow x\ge a·\lfloor \frac{x}{a}\rfloor +0.5a& \\ \Rightarrow x\ge a·\left(\frac{x}{a}-ϵ\right)+0.5a\because \mathrm{by}\left(\mathrm{eq}01:{\sec }_{01}\right)x=\lfloor x\rfloor +ϵ& \\ \Rightarrow x\ge x-0.5a+ϵ+0.5a\because \mathrm{by}\left(\mathrm{eq}02:{\sec }_{01}\right)\varepsilon =0.5-ϵ& \\ \Rightarrow a\varepsilon \le 0& \end{array}$

Particularly, when

$\frac{x}{a}<1$

is satisfied because |x|<|a| is satisfied, this may be expanded to

$Q_p\frac{x}{a}<Q_p,$

in which case the operation may be represented as shown in Equation (13).

$\begin{array}{cc}{Q}_p·\frac{x}{a}=\lfloor \frac{Q_{p}x}{a}\rfloor +\frac{1}{a}\left(Q_{p}x-a\lfloor \frac{Q_{p}x}{a}\rfloor \right)& \left(13\right)\end{array}$

Through Equations (5) and (13), quantization of division may be derived as Equation (14). First, the function, g(x,a)∈{0,1}, may be defined as shown in Equation (14).

$\begin{array}{cc}g\left(x,a\right)=\lfloor \frac{1}{a}\left(Q_{p}x-a\left(\lfloor \frac{Q_{p}x}{a}\rfloor -\frac{1}{2}\right)\right)\rfloor =\{\begin{array}{cc}1& a·\varepsilon \le 0\\ 0& a·\varepsilon >0\end{array}& \left(14\right)\end{array}$

When quantization for division is interpreted using Equation (14), this may be represented as shown in Equation (15):

$\begin{array}{cc}\begin{array}{c}\frac{1}{Q_p}\left[Q_p·\frac{x}{a}\right]=\frac{1}{Q_p}\lfloor \lfloor \frac{Q_{p}x}{a}\rfloor +\frac{1}{a}\left(Q_{p}x-a\left(\lfloor \frac{Q_{p}x}{a}\rfloor -\frac{1}{2}\right)\right)\rfloor \\ =\frac{1}{Q_p}\lfloor \frac{Q_{p}x}{a}\rfloor +\frac{1}{Q_p}\lfloor \frac{1}{a}\left(Q_{p}x-a\left(\lfloor \frac{Q_{p}x}{a}\rfloor -\frac{1}{2}\right)\right)\rfloor \\ =\frac{1}{Q_p}\left(\lfloor \frac{Q_{p}x}{a}\rfloor +g\left(x,a\right)\right)\end{array}\hspace{1em}& \left(15\right)\end{array}$

Based on the definition of learning quantization and main quantization operations according to an embodiment of the present invention, it may be understood that, when quantization based on the distribution of a remainder value is performed, the smallest value is 1 or 0.

Referring to FIG. 1, in the method for optimizing a quantized machine-learning algorithm according to an embodiment of the present invention, a learning rate may be set at step S10.

When the above-described quantization method is expanded to a vector, this may be defined in such a way that quantization is applied to the elements of each vector. At step S10, using the method of applying quantization to the elements of each vector, the quantization characteristics of a learning equation may be analyzed.

For example, it may be assumed that the learning equation shown in Equation (16) is given for x(t)∈Rⁿ.

x _t+1 =x _t−λ_th_t   (16)

In Equation (16), h_t∈Rⁿ is a search direction vector, λ_t∈R(0,1] is a learning rate, and t∈R is a parameter related to time. In Equation (16), when quantization of x_t is defined as x_t^Q≡[x]_t, the quantized learning equation shown in Equation (17) may be defined.

x _t+1 ^Q=(x_t−λ_th_t)^Q   (17)

In Equation (17), λ_t denotes a learning rate. Using h_t, x_t or x_t^Q may be updated to x_t+1^Q through the update term of Equation (18).

λ_th_t=(λ_th_t)^Q   (18)

Additionally, it is assumed that the following basic condition for the learning rate is satisfied.

λ_i=arg min ƒ(x_i−λh_i)   (19)

Here, when Equation (17), which is a quantized learning equation, is rewritten by substituting the quantized x_t^Q for x_t and by applying Equation (18), which is a quantized update term, Equation (20) may be obtained.

x _t+1 ^Q=(x_t^Q−(λ_th_t)^Q)^Q=x_t^Q−(λ_th_t)^Q   (20)

In Equation (20), because the learning rate λ_t∈R and the search direction vector h_t∈Rⁿ are a scalar and a vector, respectively, it is relatively difficult to implement quantization satisfying λ_th_t=(λ_th_t)^Q. Therefore, even though the optimized learning rate is found through a line search algorithm, it is required to recalculate it so as to enable quantization.

The most intuitive quantization of the update term is to set λ_t=1/Q_p. As the result thereof,

${\lambda }_t{h}_t=\frac{1}{Q_p}{h}_t$

is satisfied, and the quantized update term may be represented as shown in Equation (21).

$\begin{array}{cc}{\left({\lambda }_t{h}_t\right)}^Q=\frac{1}{Q_p}\left(Q_p{\lambda }_t{h}_t+Q_p\varepsilon \right)=\frac{1}{Q_p}{h}_t+\varepsilon ={\left(\frac{1}{Q_p}{h}_t\right)}^Q& \left(21\right)\end{array}$

Based on this, the function z(k_t)∈Z, z(k_t)>0, which outputs an arbitrary positive integer for the internal iteration index k_t in the t-th iteration, is defined, and using this function, the learning rate λ_t is set to λ_t=z(k_t)Q_p⁻¹. Accordingly, the update term may be represented as shown in Equation (22).

$\begin{array}{cc}{\left({\lambda }_t{h}_t\right)}^Q=\frac{z\left(k_t\right)}{Q_p}{h}_t+\varepsilon ={\left(\frac{z\left(k_t\right)}{Q_p}{h}_t\right)}^Q& \left(22\right)\end{array}$

If quantization is defined well so as to satisfy

$\frac{1}{Q_p}{h}_t\in Z$

by appropriately defining Equation (22) (or if quantization can be defined through the internal characteristics of a calculator), because z(k_t)∈Z is satisfied, Equation (23) may be obtained.

$\begin{array}{cc}{\left(\frac{z\left(k_t\right)}{Q_p}{h}_t\right)}^Q=z\left(k_t\right){\left(\frac{1}{Q_p}{h}_t\right)}^Q& \left(23\right)\end{array}$

In Equation (23), z(k_t)∈Z(0,Q_p) is satisfied. When the well-defined

$\frac{1}{Q_p}{h}_t\in Z$

is assumed to be a basic quantization search direction vector, Equation (24) may be defined.

$\begin{array}{cc}{h}_t^Q\stackrel{\Delta }{=}{\left(\frac{1}{Q_p}{h}_t\right)}^Q,h_t^Q\in Z& \left(24\right)\end{array}$

When Equation (22) is rewritten using Equation (24), Equation (25) may be obtained.

(λ_th_t)^Q=z(k_t)h_t^Q   (25)

That is, at step S10, the learning rate of the quantized machine-learning algorithm may be set using at least one of an Armijo rule and golden search methods.

Here, at step S10, using the gradient vector of the objective function of the search direction vector, the learning rate may be set based on a learning-rate-setting function predefined by the Armijo rule.

Here, at step S10, any one of a first candidate value, which is acquired by increasing the minimum candidate value of the learning rate by a golden ratio, and a second candidate value, which is acquired by decreasing the maximum candidate value of the learning rate by the golden ratio, may be set as the learning rate.

Here, at step S10, when the difference value between the first candidate value and the second candidate value is equal to or less than a preset value, any one of the first candidate value and the second candidate value may be set as the learning rate.

Also, in the method for optimizing a quantized machine-learning algorithm according to an embodiment of the present invention, quantization search performance may be compensated for at step S20.

That is, at step S20, a quantized orthogonal compensation search vector may be calculated from the search direction vector of the quantized machine-learning algorithm, and the search performance of the quantized machine-learning algorithm may be compensated for using the quantized orthogonal compensation search vector.

Here, at step S20, a vector in the direction orthogonal to the direction opposite the largest component vector of the search direction vector is selected, and the selected vector is quantized, whereby the quantized orthogonal compensation search vector may be calculated.

Here, at step S20, when the solution of the quantized machine-learning algorithm is not able to escape from a local minimum point, the solution may be made to escape from the local minimum point using the quantized orthogonal compensation search vector.

Also, in the method for optimizing a quantized machine-learning algorithm according to an embodiment of the present invention, an optimized learning algorithm may be calculated at step S30.

That is, at step S30, the optimized quantized machine-learning algorithm may be calculated using the learning rate and the quantized machine-learning algorithm, the search performance of which is compensated for.

FIG. 2 is a flowchart illustrating an optimization algorithm using a quantized Armijo rule according to an embodiment of the present invention.

Referring to FIG. 2, the process of setting the learning rate of a quantized machine-learning algorithm to which the Armijo rule is applied at step S10 is specifically illustrated.

First, at step S110, a quantization parameter may be set.

Also, at step S120, a learning-rate-setting parameter may be set.

Also, at step S130, a search direction vector may be calculated.

The search direction vector h_t is the gradient vector of an objective function ƒ(x) and is represented as h_t=∇ƒ(x_t), and the function for setting the learning rate may be represented as shown in Equation (26).

ϕ(β^k)=β^kα∥h_t∥²   (26)

Also, at step S140, when the error of the search direction vector is equal to or less than a preset value, the optimization of the quantized machine-learning algorithm is terminated, but when the error is greater than the preset value, the learning rate may be set at step S150.

At step S150, the learning rate may be set using Equation (27) to which the Armijo rule is applied.

$\begin{array}{cc}{\lambda }_t\stackrel{\Delta }{=}\underset{k\in N}{\mathrm{argmax}}[{\beta }^kf\left(x_t+{\beta }^k{h}_t\right)-f\left(x_t\right)\le -{\beta }^k\alpha {h_t}^2=\varphi \left({\beta }^k\right)]\alpha ,\beta \in R\left(0,1\right)& \left(27\right)\end{array}$

Finally, the quantization parameter may be configured with positive integers, as shown in Equation (28).

Q _p=η·ρⁿ η,ρ, n∈Z⁺⁺   (28)

Because λ_t=ϕ(β^k) is satisfied in the definition of the learning rate, the Armijo rule may be modified so as to satisfy the characteristics of the quantized learning rate in Equation (23). First, assuming that the search direction vector is the basic quantization search direction vector and is represented as h_t(h_t)^Q=Q_ph_t^Q, Equation (29) may be obtained.

$\begin{array}{cc}\begin{array}{c}\varphi \left({\beta }^k\right)=-{\beta }^k\alpha {h_t}^2\\ =-{\beta }^k\alpha {{\left(h_t\right)}^Q}^2\\ =-{\beta }^k\alpha {Q_p{h}_t^Q}^2\\ =-{\beta }^k{\mathrm{\alpha Q}}_p^2{h_t^Q}^2\because Q_p\in Z,Q_p>0\\ =-\left({\beta }^k{Q}_p\right)·\left({\mathrm{\alpha Q}}_p\right){h_t^Q}^2\\ =-\left({\beta }^k{\mathrm{\eta \rho }}^n\right)·\left(\alpha {\mathrm{\eta \rho }}^n\right){h_t^Q}^2\\ =-\left({\beta }^k{\rho }^n\right)·\left(\alpha {\eta }^2{\rho }^n\right){h_t^Q}^2\end{array}\hspace{1em}& \left(29\right)\end{array}$

Equation (29) may be solved as shown in Equation (30).

$\begin{array}{cc}\varphi \left({\stackrel{\_}{\beta }}^{\stackrel{\_}{k}}\right)\equiv -\left({\stackrel{\_}{\beta }}^{\stackrel{\_}{k}}\right)·\stackrel{\_}{\alpha }{h_t^Q}^2,0<{\stackrel{\_}{\beta }}^{\max \stackrel{\_}{k}}\le \frac{Q_p}{\eta },0<\stackrel{\_}{\alpha }\le \eta ·Q_p& \left(30\right)\end{array}$

Because α and β are arbitrary values satisfying α,β∈R(0,1), an integer that is equal to or less than η is taken (that is, ζ≤η) for α in Equation (30), whereby the condition for a may be satisfied as shown in Equation (31).

$\begin{array}{cc}\stackrel{\_}{\alpha }=\zeta ·Q_p\in Z,\eta <\stackrel{\_}{\alpha }\le \eta ·Q_p\because \alpha =\frac{\zeta }{\eta }& \left(31\right)\end{array}$

Also, when β=ρ⁻¹ is set, β may be represented as shown in Equation (32) for the fixed value n.

$\begin{array}{cc}{\stackrel{\_}{\beta }}^{\stackrel{\_}{k}}={\beta }^k{\rho }^n={\rho }^{-k}{\rho }^n=p^{n-k},0<\stackrel{\_}{k}\le n& \left(32\right)\end{array}$

Therefore, β and k are defined as β=ρ, k=n−k, and because max k=n is satisfied, Equation (33) may be obtained.

$\begin{array}{cc}{\stackrel{\_}{\beta }}^{\max \stackrel{\_}{k}}={\rho }^n=\frac{Q_p}{\eta }& \left(33\right)\end{array}$

Therefore, when the objective function ƒ:Rⁿ→R is configured so as to satisfy ƒ:Zⁿ→Z for zⁿ and when h₀ is configured so as to satisfy h₀=−∇ƒ(x₀)∈Z, the learning rate λ_t^Q based on the Armijo rule may be calculated using only the integer operations shown in Equation (34), and may then be applied to the quantized machine-learning algorithm at step S150.

$\begin{array}{cc}{\lambda }_t^Q\stackrel{\Delta }{=}\underset{\stackrel{\_}{k}\in N}{\mathrm{argmax}}\{{\stackrel{\_}{\beta }}^{\stackrel{\_}{k}}f\left(x_t+{\stackrel{\_}{\beta }}^{\stackrel{\_}{k}}{h}_t\right)-f\left(x_t\right)\le -{\stackrel{\_}{\beta }}^{\stackrel{\_}{k}}\stackrel{\_}{\alpha }{h_t^Q}^2=\varphi \left({\stackrel{\_}{\beta }}^{\stackrel{\_}{k}}\right)\}& \left(34\right)\end{array}$

Also, at step S160, the quantized learning equation may be updated using the set learning rate.

FIG. 3 is a flowchart illustrating an optimization algorithm using a golden search according to an embodiment of the present invention.

Referring to FIG. 3, the process of setting the learning rate of a quantized machine-learning algorithm to which a golden search method is applied at step S10 is specifically illustrated.

The golden search method is a method for finding the optimized learning rate, like the Armijo rule and a learning rate satisfying Equation 35 may be set.

$\begin{array}{cc}\underset{\lambda \ge 0}{\min }\left\{f\left(x_i+\lambda h_i\right)\right\}\Rightarrow \mathrm{Let}\varphi \left(\lambda \right)=f\left(x_i+\lambda h_i\right)-f\left(x_i\right)\Rightarrow \underset{\lambda }{\min }\varphi \left(\lambda \right)& \left(35\right)\end{array}$

First, at step S210, a quantization parameter may be set.

Also, at step S220, a search range for applying the golden search method may be set.

Also, at step S230, an initial condition for applying the golden search method may be set.

Here, at step S230, when quantization is not applied, a₀=0 and b₀=1 may be set and h_t=Q_ph_t^Q may be set.

Also, at step S240, the golden search method may be applied.

Here, at step S240, the i-th search range 1, may be defined as shown in Equation (36).

l _i b _i −a _i   (36)

Here, a_i and b_i may be the minimum candidate value and the maximum candidate value for the learning rate A.

Here, at step S240, when the value of the search range 1, is made to approach 0 by increasing the minimum candidate value by a golden ratio and decreasing the maximum candidate value by the golden ratio using the golden search method, one of the minimum candidate value and the maximum candidate value may be set as the learning rate.

The golden ratio used for the golden search may be F₀=0.618.

Also, at steps S250 to S280, the minimum candidate value and the maximum candidate value may be updated.

Here, the update of the minimum candidate value and the maximum candidate value may be represented as shown in Equation (37).

a′ _i =a _i+(1−F₀)l_i, b′_i=b_i−(1−F₀)l_i   (37)

Because a_i and b_i are set to a_i,b_i∈R(0,1) in Equation (37), quantization thereof is required.

Using Equations (24) and (25), the quantized learning equation shown in Equation (38) may be calculated.

x _t+1 ^Q =x _t ^Q−(λ_th_t)^Q=x_t^Q−(λ_tQ_ph_t^Q)^Q=x_t^Q−(λ_tQ_p)^Qh_t^Q   (38)

In Equation (38), because (λ_tQ_p)^Q is a value that falls within the range of z(0,Q_p), when λ_t^Q(λ_tQ_p) is set, Equation (39) may be obtained.

a _i ^Q≡(a_iQ_p)^Q, b_i≡(b_iQ_p)^Q, a_i,b_i∈Z(0,Q_p)   (39)

Equation (39) may be solved as a quantized operation using the golden search method. To this end, F=1−F₀ is set first, and the equation for the minimum candidate in Equation (37) is multiplied by Q_p, whereby Equation (40) may be obtained.

a′ _i Q _p =a _i Q _p +Q _p(1−F₀)l_i

a′ _i Q _p =a _i Q _p +Q _p Fl _i

(a′_iQ_p)^Q=(a_iQ_p+Q_pFl_i)^Q   (40)

The error resulting from quantization is taken into consideration, and because l_i=b_i^Q−a_i^Q∈Z is satisfied as the result of quantization, Equation (41) may be obtained.

a′ _i ^Q=(a_iQ_p)^Q+(Q_pF)^Ql_i+O(ε)   (41)

Because this is the operation for setting the learning rate, when the quantization error is ignored, Equations (39) and (41) may be solved as shown in Equation (42).

a′ _i ^Q =a _i ^Q+(Q_pF)^Ql_i, b′_i^Q=b_i^Q−(Q_pF)^Ql_i ∀i∈N, a_i^Q,b_i^Q∈Z(0,Q_p)   (42)

Because the learning rate set through the above process satisfies λ_t^QL∈Z(0,Q_p), when this is applied to the quantization unit search vector h_t^Q, the following quantized learning equation may be obtained.

x _t+1 ^Q =x _t ^Q−λ_t^QLh_t^Q   (43)

FIG. 4 is a flowchart illustrating an optimization algorithm using a compensated search vector according to an embodiment of the present invention.

Referring to FIG. 4, the process of compensating for the search performance of a quantized machine-learning algorithm at step S20 is specifically illustrated.

Here, at step S20, a point that makes an objective function smaller in a different direction other than a search direction vector is searched for, whereby optimization performance degradation may be overcome.

First, at step S310, a quantization parameter may be set.

Also, at step S320, an initial parameter may be set.

Also, at step S330, a search direction vector may be calculated.

Also, at step S340, when the error of the search direction vector is equal to or less than a preset value, the optimization of the quantized machine-learning algorithm is terminated, but when the error is greater than the preset value, a learning rate may be set at step S350.

At step S350, the learning rate may be set using the Armijo rule and the golden search method, as described with reference to FIG. 2 and FIG. 3.

Also, at step S360, whether the quantized learning equation using the set learning rate reaches a local minimum point is determined. When the quantized learning equation is determined not to reach the local minimum point, the search direction vector may be compensated for at step S370.

Here, at step S360, with regard to performance degradation, it may be checked whether the optimization parameter x, cannot converge near the local minimum point because the appropriate learning rate enabling arrival at the local minimum point cannot be applied due to quantization or whether a better minimum point cannot be found.

For the search direction vector h_t∈Rⁿ, the quantized vector (h_t)^Q=Q_ph_t^Q∈Rⁿ may be represented as shown in Equation (44).

$\begin{array}{cc}{\left(h_t\right)}^Q=\sum _{i=0}^{n-1}{v}_i{e}_i,\forall i\in Z\left[0,n-1\right],v_i\in Z& \left(44\right)\end{array}$

Here, at step S370, the different direction is the direction orthogonal to the search direction. Because a vector orthogonal to the search direction vector may have various vector directions, the vector in the direction orthogonal to the direction opposite the largest component vector of the search direction vector may be selected.

In Equation (44), e_i is the unit orthogonal vector of Euclidian space Rⁿ and satisfies ∀i,j∈Z[0,n), ∥e_i∥=1, e_i^Te_j=0 i≠j. Assume that the largest component, among the components {v_i} of the quantized vector (h_t)^Q, is v_m=max{∥v_i∥}, and that the index thereof is m=argmax_i{∥v_i∥}. Here, when the vector acquired by setting v_m=0 in the quantized vector (h_t)^Q is v and when the vector in which all of the components excluding v_m are 0 is 0, v and {circumflex over (v)} may be defined as shown in Equation (45).

$\begin{array}{cc}\stackrel{\_}{v}=\sum _{i=0}^{n-1}\left(1-\delta \left(i-m\right)\right)v_i{e}_i,\hat{v}=\sum _{i=0}^{n-1}\delta \left(i-m\right)v_i{e}_i& \left(45\right)\end{array}$

Therefore, the search direction vector may be divided into two orthogonal vectors v and {circumflex over (v)}, as shown in Equation (46).

$\begin{array}{cc}{\left(h_t\right)}^Q=\stackrel{\_}{v}+\hat{v}=\sum _{i=0}^{n-1}\left(1-\delta \left(i-m\right)+\delta \left(i-m\right)\right)v_i{e}_i=\sum _{i=0}^{n-1}{v}_i{e}_i& \left(46\right)\end{array}$

At step S370, vector z may be calculated as shown in Equation (47) in order to obtain the vector in the direction orthogonal to the largest component in the existing search direction vector (h_t)^Q.

z _t =v+r·{circumflex over (v)}, r∈R   (47)

In Equation (47), r∈R is a proportional constant for {circumflex over (v)}, and through this value, the orthogonal vector z may be calculated. Using the orthogonality between the vector z and the vector (h_t)^Q, r may be calculated as shown in Equation (48).

$\begin{array}{cc}\begin{array}{c}0=\langle {\left(h_t\right)}^Q,z\rangle =\langle \stackrel{\_}{v}+\hat{v},\stackrel{\_}{v}+r\hat{v}\rangle \\ ={\stackrel{\_}{v}}^2+\left(r+1\right)\langle \hat{v},\stackrel{\_}{v}\rangle +r{\hat{v}}^2\because \langle \hat{v},\stackrel{\_}{v}\rangle =0\\ ={\stackrel{\_}{v}}^2+r{\hat{v}}^2\end{array}& \left(48\right)\\ \therefore r=-\frac{{\stackrel{\_}{v}}^2}{{\hat{v}}^2}=-\frac{{{\left(h_t\right)}^Q}^2-v_m^2}{v_m^2}=1-\frac{{{\left(h_t\right)}^Q}^2}{v_m^2}& \end{array}$

However, because |r|<1 is satisfied, when this value is applied to the learning equation without change, the operation is not performed using an integer value.

Accordingly, the compensated search vector may become a general real number vector, rather than a vector configured with quantized values. Therefore, at step S370, the compensated search vector may be calculated in consideration of quantization of the proportional constant.

In Equation (47), because (h_t)^Q=Q_ph_t^Q is satisfied, when the equation is solved using v_m=Q_pv_m^Q, Equation (49) may be obtained.

$\begin{array}{cc}\hat{v}=\sum _{i=0}^{n-1}\delta \left(i-m\right)v_i{e}_i=Q_p·v_m^Q{e}_m& \left(49\right)\end{array}$

In Equation (49), when v_m^Qe_m{circumflex over (v)}^Q is set, Equation (50) may be obtained.

$\begin{array}{cc}\begin{array}{c}{z}_t=\stackrel{\_}{v}+r·\hat{v}+r·Q_p{\hat{v}}^Q=\stackrel{\_}{v}+Q_{p}r·{\hat{v}}^Q\\ =\stackrel{\_}{v}+Q_p{\hat{v}}^Q-Q_p{\hat{v}}^Q+Q_{p}r·{\hat{v}}^Q\\ =\left(\stackrel{\_}{v}+\hat{v}\right)+Q_p\left(r-1\right){\hat{v}}^Q\because \hat{v}=Q_p{\hat{v}}^Q\end{array}\hspace{1em}& \left(50\right)\end{array}$

Based on Equations (46) and (48) and (h_t)^Q=Q_ph_t^Q, Equation (51) may be obtained.

$\begin{array}{cc}{z}_t=Q_p{h}_t^Q-Q_p\frac{{{\left(h_t\right)}^Q}^2}{v_m^2}{\hat{v}}^Q& \left(51\right)\end{array}$

Accordingly, when the coefficient of {circumflex over (v)}^Q is solved using Equations (11) and (25), Equation (52) may be obtained.

$\begin{array}{cc}{Q}_p·\frac{{{\left(h_t\right)}^Q}^2}{v_m^2}=Q_p·\frac{\sum _{i=0}^{n-1}{v}_i^2}{v_m^2}=\sum _{i=0}^{n-1}\frac{Q_p{v}_i^2}{v_m^2}& \left(52\right)\end{array}$

Accordingly, when the compensated search vector z_t is quantized using Equations (51) and (52), Equation (53) may be obtained.

$\begin{array}{cc}\begin{array}{c}{\left(z_t\right)}^Q={\left(Q_p{h}_t^Q-Q_p\frac{{{\left(h_t\right)}^Q}^2}{v_m^2}{\hat{v}}^Q\right)}^Q\\ =Q_p{h}_t^Q-{\left(\sum _{i=0}^{n-1}\frac{Q_p{v}_i^2}{v_m^2}\right)}^Q{\hat{v}}^Q\\ =Q_p{h}_t^Q-\lfloor \sum _{i=0}^{n-1}\frac{Q_p{v}_i^2}{v_m^2}+0.5\rfloor {\hat{v}}^Q\\ =Q_p{h}_t^Q-\lfloor \sum _{i=0}^{n-1}\left\{\lfloor \frac{Q_p{v}_i^2}{v_m^2}\rfloor +\frac{1}{v_m^2}\left(Q_p{v}_t^2-v_m^2\lfloor \frac{Q_p{v}_i^2}{v_m^2}\rfloor \right)\right\}+0.5\rfloor {\hat{v}}^Q\end{array}\hspace{1em}& \left(53\right)\end{array}$

Therefore, Equation (53) may be solved to Equation (54) using Equation (4).

$\begin{array}{cc}{\left(z_t\right)}^Q=Q_p{h}_t^Q-(\sum _{i=0}^{n-1}\lfloor \frac{Q_p{v}_i^2}{v_m^2}\rfloor +\lfloor \sum _{i=0}^{n-1}\frac{1}{v_m^2}(Q_p{v}_t^2-v_m^2\lfloor \frac{Q_p{v}_i^2}{v_m^2}\rfloor +0.5\rfloor ){\hat{v}}^Q& \left(54\right)\end{array}$

In Equation (54),

$\lfloor \frac{Q_p{v}_i^2}{v_m^2}\rfloor$

and

$Q_p{v}_i^2-v_m^2\lfloor \frac{Q_p{v}_i^2}{v_m^2}\rfloor$

are the quotient and remainder of

$\frac{Q_p{v}_i^2}{v_m^2}.$

The part corresponding to the remainder may be simplified as shown in Equation (55).

$\begin{array}{cc}\mathrm{Rem}\left(\frac{Q_p{v}_i^2}{v_m^2}\right)=\frac{1}{v_m^2}\left(Q_p{v}_i^2-v_m^2\lfloor \frac{Q_p{v}_i^2}{v_m^2}\rfloor \right)& \left(55\right)\end{array}$

The quantized orthogonal compensation search vector may be represented as shown in Equation (56).

$\begin{array}{cc}{\left(z_t\right)}^Q=Q_p{h}_t^Q-\left(\sum _{i=0}^{n-1}\lfloor \frac{Q_p{v}_i^2}{v_m^2}\rfloor +\left[\mathrm{Rem}\left(\frac{Q_p{v}_i^2}{v_m^2}\right)\right]\right){\hat{v}}^Q& \left(56\right)\end{array}$

That is, at step S370, the quantized compensation search vector z_t, which is orthogonal to the search direction vector, may be calculated.

Here, at step S370, when the quantized learning equation does not escape from a local minimum point using the quantized orthogonal compensation search vector, the quantized learning equation may be made to escape from the local minimum point using the quantized compensation search vector.

Because the quantized compensation search vector has a scale of a quantization coefficient Q_p, this may be calculated as the basic quantization compensation search vector, as shown in Equation (57).

$\begin{array}{cc}{z}_t^Q\stackrel{\Delta }{=}\frac{1}{Q_p}{\left(z_t\right)}^Q& \left(57\right)\end{array}$

Also, at step S380, the quantized learning equation may be calculated.

That is, at step S380, based on the quantized compensation search vector defined in Equation (57), the quantized compensation search vector is multiplied by the quantized learning rate calculated using the quantized Armijo rule or the quantized line search algorithm, whereby the quantized learning equation shown in Equation (58) may be calculated.

The quantized learning equation shown in Equation (58) may be easily combined with the existing machine-learning or nonlinear algorithm.

x _t+1 ^Q =x _t ^Q−λ_t^Qh_t^Q, h_t^Q=z_t^Q   (58)

FIG. 5 is a view illustrating a computer system according to an embodiment of the present invention.

Referring to FIG. 5, the apparatus for optimizing a quantized machine-learning algorithm according to an embodiment of the present invention may be implemented in a computer system 1100 including a computer-readable recording medium. As illustrated in FIG. 5, the computer system 1100 may include one or more processors 1110, memory 1130, a user-interface input device 1140, a user-interface output device 1150, and storage 1160, which communicate with each other via a bus 1120. Also, the computer system 1100 may further include a network interface 1170 connected to a network 1180. The processor 1110 may be a central processing unit or a semiconductor device for executing processing instructions stored in the memory 1130 or the storage 1160. The memory 1130 and the storage 1160 may be any of various types of volatile or nonvolatile storage media. For example, the memory may include ROM 1131 or RAM 1132.

The apparatus for optimizing a quantized machine-learning algorithm according to an embodiment of the present invention includes one or more processors 1110 and executable memory 1130 for storing at least one program executed by the one or more processors 1110. The at least one program may set the learning rate of the quantized machine-learning algorithm using at least one of an Armijo rule and golden search methods, calculate a quantized orthogonal compensation search vector from the search direction vector of the quantized machine-learning algorithm, compensate for the search performance of the quantized machine-learning algorithm using the quantized orthogonal compensation search vector, and calculate an optimized quantized machine-learning algorithm using the learning rate and the quantized machine-learning algorithm, the search performance of which is compensated for.

Here, the at least one program may set the learning rate through a learning-rate-setting function predefined by the Armijo rule using the gradient vector of the objective function of the search direction vector.

Here, the at least one program may set any one of a first candidate value, which is acquired by increasing the minimum candidate value of the learning rate by a golden ratio, and a second candidate value, which is acquired by decreasing the maximum candidate value of the learning rate by the golden ratio, as the learning rate.

Here, when the difference value between the first candidate value and the second candidate value is equal to or less than a preset value, the at least one program may set any one of the first candidate value and the second candidate value as the learning rate.

Here, the at least one program may select a vector in a direction orthogonal to the direction opposite the largest component vector of the search direction vector and quantize the selected vector, thereby calculating the quantized orthogonal compensation search vector.

Here, when the solution of the quantized machine-learning algorithm is not able to escape from a local minimum point, the at least one program may make the solution escape from the local minimum point using the quantized orthogonal compensation search vector.

The present invention may implement an optimization algorithm capable of minimizing a quantization error in machine-learning and nonlinear-signal-processing fields using quantization and exhibiting excellent performance on lightweight hardware.

Also, the present invention may implement a machine-learning algorithm capable of providing sufficient optimization performance even on low-performance hardware.

As described above, the apparatus and method for optimizing a quantized machine-learning algorithm according to the present invention are not limitedly applied to the configurations and operations of the above-described embodiments, but all or some of the embodiments may be selectively combined and configured, so that the embodiments may be modified in various ways.

Claims

1. An apparatus for optimizing a quantized machine-learning algorithm, comprising: one or more processors; andexecutable memory for storing at least one program executed by the one or more processors,wherein the at least one program sets a learning rate of the quantized machine-learning algorithm using at least one of an Armijo rule and golden search methods, calculates a quantized orthogonal compensation search vector from a search direction vector of the quantized machine-learning algorithm, compensates for search performance of the quantized machine-learning algorithm using the quantized orthogonal compensation search vector, and calculates an optimized quantized machine-learning algorithm using the learning rate and the quantized machine-learning algorithm, the search performance of which is compensated for.

2. The apparatus of claim 1, wherein the at least one program sets the learning rate through a learning-rate-setting function predefined by the Armijo rule using a gradient vector of an objective function of the search direction vector.

3. The apparatus of claim 1, wherein the at least one program sets any one of a first candidate value, acquired by increasing a minimum candidate value of the learning rate by a golden ratio, and a second candidate value, acquired by decreasing a maximum candidate value of the learning rate by the golden ratio, as the learning rate.

4. The apparatus of claim 2, wherein the at least one program sets any one of the first candidate value and the second candidate value as the learning rate when a difference value between the first candidate value and the second candidate value is equal to or less than a preset value.

5. The apparatus of claim 1, wherein the at least one program selects a vector in a direction orthogonal to a direction opposite a largest component vector of the search direction vector and quantizes the selected vector, thereby calculating the quantized orthogonal compensation search vector.

6. The apparatus of claim 5, wherein, when a solution of the quantized machine-learning algorithm is not able to escape from a local minimum point, the at least one program makes the solution escape from the local minimum point using the quantized orthogonal compensation search vector.

7. A method for optimizing a quantized machine-learning algorithm, performed by an apparatus for optimizing the quantized machine-learning algorithm, comprising: setting a learning rate of the quantized machine-learning algorithm using at least one of an Armijo rule and golden search methods;calculating a quantized orthogonal compensation search vector from a search direction vector of the quantized machine-learning algorithm and compensating for search performance of the quantized machine-learning algorithm using the quantized orthogonal compensation search vector; andcalculating an optimized quantized machine-learning algorithm using the learning rate and the quantized machine-learning algorithm, the search performance of which is compensated for.

8. The method of claim 7, wherein setting the learning rate is configured to set the learning rate through a learning-rate-setting function predefined by the Armijo rule using a gradient vector of an objective function of the search direction vector.

9. The method of claim 7, wherein setting the learning rate is configured to set any one of a first candidate value, acquired by increasing a minimum candidate value of the learning rate by a golden ratio, and a second candidate value, acquired by decreasing a maximum candidate value of the learning rate by the golden ratio, as the learning rate.

10. The method of claim 9, wherein setting the learning rate is configured to set any one of the first candidate value and the second candidate value as the learning rate when a difference value between the first candidate value and the second candidate value is equal to or less than a preset value.

11. The method of claim 7, wherein compensating for the search performance is configured to select a vector in a direction orthogonal to a direction opposite a largest component vector of the search direction vector and to quantize the selected vector, thereby calculating the quantized orthogonal compensation search vector.

12. The method of claim 11, wherein compensating for the search performance is configured such that, when a solution of the quantized machine-learning algorithm is not able to escape from a local minimum point, the solution is made to escape from the local minimum point using the quantized orthogonal compensation search vector.