The present invention relates to a kernel-based machine learning approach, and in particular to an interpretable and efficient method and system of kernel learning.
Machine learning approaches have been widely applied in data science for building predictive models. To train a predictive model, a set of data examples with known labels is used as the input of a learning algorithm. After training, the fitted model is utilized to predict the labels of data examples that have not been seen before.
The representation of data is one of the essential factors that affect prediction accuracy. Usually, each data example is preprocessed and represented by a feature vector in a feature space. Kernel-based methods are a family of powerful machine learning approaches in terms of prediction accuracy, owing to the capability of mapping each data example to a high-dimensional (possibly infinite) feature space. The representation of data in this feature space is able to capture nonlinearity in data, e.g., infinite-order interactions among features can be represented in cases of the Gaussian Radial basis function (RBF) kernel. Moreover, the feature map in kernel-based methods is implicitly built, and the corresponding inner product can be directly computed via a kernel function. This is known as the “kernel trick”.
Nevertheless, the implicit feature map in a standard kernel function is difficult to interpret by humans, e.g., different effects of the original features on prediction cannot be clearly explained. This makes standard kernel-based methods unattractive in application domains such as marketing and healthcare, where model interpretability is highly required.
Multiple kernel learning (MKL) is designed for the problems that involve multiple heterogeneous data sources. Additionally, MKL can also provide interpretability for the resulting model, as discussed by Non Patent Literature 1. Specifically, the kernel function is considered as a convex combination of multiple sub-kernels in MKL, where each sub-kernel is evaluated on a feature representation, e.g., a subset of the original features. By optimizing the combination coefficients, it is possible to explain the effects of different feature representations on prediction. Patent Literature 1 discloses a machine learning for object identification. Patent Literature 1 describes, as the machine learning approach, an example of MKL using a Support Vector Machine (SVM) as a known technique.
Unfortunately, standard kernel-based methods suffer from the scalability issue, due to the storage and computation costs of the dense kernel matrix (generally quadratic in the number of data examples). This is even worse when using multiple kernels, because multiple kernel matrices have to be stored and computed.
Recently, several techniques have been developed for addressing the scalability issue of kernel methods. One of them is called random Fourier features (RFF), described by Non Patent Literature 2. The key idea of RFF is to directly approximate the kernel function using explicit randomized feature maps. Since the feature maps are explicitly built, large-scale problems can be solved by exploiting efficient linear algorithms without computing kernel matrices. Patent Literature 2 discloses, as one example of hash functions, a hash function based on Shift-Invariant Kernels that projects to a hash value using RFF.
As a remedy for the scalability issue, RFF is able to reduce the complexity of standard MKL from quadratic to linear in the number of data examples. However, it is still not computationally efficient when the number of sub-kernels is large, which is the usual case in MKL.
Alternating direction method of multipliers (ADMM) is a popular algorithm for distributed convex optimization. ADMM is particularly attractive for large-scale problems, because it can break the problem at hand into sub-problems that are easier to solve in parallel if the original problem can be transformed into an ADMM form. ADMM is thoroughly surveyed by Non Patent Literature 3. Patent Literature 3 discloses a ranking function learning apparatus in which an optimization problem is solved using an optimization scheme called ADMM.
The objective of this invention is to address the interpretability issue of standard kernel learning via an efficient distributed optimization approach and system.
In standard kernel learning, a kernel function is defined as the inner product of implicit feature maps. However, it is difficult to interpret different effects of features because all of them are packed into the kernel function in a nontransparent way. In multiple kernel learning (MKL), the kernel function is considered as a convex combination of sub-kernels, with each sub-kernel evaluated on a certain feature representation. To interpret the effects of different feature representations, an optimization problem is solved to obtain the optimal combination of sub-kernels. Unfortunately, this optimization process usually involves computing multiple kernel matrices, which is computationally expensive (generally quadratic in the number of data examples). Random Fourier features (RFF) is a popular technique of kernel approximation. In RFF, the feature map is explicitly built so that efficient linear algorithms can be exploited to avoid computing kernel matrices. RFF alleviates the computational issue of standard kernel-based methods when the number of data examples is large, that is, reducing the computation complexity from quadratic to linear in the number of data examples. Nevertheless, more efficient computational mechanisms are required if the effects of a large number of feature representations need to be interpreted.
A mode of the present invention comprises several components and steps: preprocessing and representing each data example as a collection of feature representations that need to be interpreted; designing a kernel function with an explicit feature map to embed the feature representations of data into a nonlinear feature space and to produce the explicit feature map for the designed kernel function to train a predictive model; formulating a non-convex problem for training the predictive model into a convex optimization problem based on the explicit feature map; and solving the convex optimization problem to obtain a globally optimal solution for training an interpretable predictive model.
An exemplary effect of the present invention is that interpretable yet efficient kernel learning can be conducted for training predictive models in a distributed way.
The present invention provides an approach and system for interpretable and efficient kernel learning.
The data preprocessing component 102 extracts features from data examples 101 and represent them as feature vectors. Let
{xi}i=1N [Math. 1]
xi=(xil, . . . ,xiD)∈ [Math. 2]
{xi(k)}k=1K [Math. 3]
xi(k) [Math. 4]
includes a subset of the original D features with the size Dt. Let γi be the corresponding prediction target for the i-th example. If the task at hand is regression, then
i∈ [Math. 5]
i∈{−1,1} [Math. 6]
For example, in the context of housing value prediction, users may have features such as income of residents, number of rooms, latitude and longitude of house. Users may be interested in the effect of intersection between latitude and longitude as well as that a single feature like income of residents. In this case, users may specify a feature representation only including latitude and longitude, and its effect on prediction may be captured in the trained model 110.
The explicit feature mapping component 103 embeds the feature representations into a nonlinear feature space produced by the kernel function designed in this example embodiment. Specifically, this kernel function is defined as:
{circumflex over (k)}GAU(x(k),z(k)) [Math. 8]
is a sub-kernel evaluated on the k-th feature representation, and
β=(β(1),β(2), . . . ,β(K))∈ [Math. 9]
with
∥β∥1=1 [Math. 10]
{circumflex over (k)}GAU [Math. 11]
is an approximation of the Gaussian kernel via random Fourier features (RFF), with the explicit feature map as
{circumflex over (ϕ)}:→, [Math. 12]
In standard kernel learning, the feature map is implicit and the kernel matrix has to be computed via the kernel function for the optimization process. In contrast, the designed kernel function in Equation (1) is not directly used; instead, the corresponding feature map is explicitly built so that efficient linear algorithms may be exploited in the optimization process. According to Equation (1) and Equation (2), the explicit feature map for the designed kernel function may be written as
ϕβ:→, [Math. 14]
[Math. 15]
ϕβ(x)=√{square root over (β(1))}{circumflex over (ϕ)}(x(1)),√{square root over (β(2))}{circumflex over (ϕ)}(x(2)), . . . ,√{square root over (β(K))}{circumflex over (ϕ)}(x(K))), (3)
so that
With this explicit feature map in Equation (3), efficient linear algorithms may be exploited to train a predictive model
where
w(k)∈ [Math. 18]
w∈ [Math. 19]
The convex problem formulating component 104 casts the problem of training a predictive model in Equation (4) as a convex optimization problem, where a globally optimal solution is to be obtained.
A predictive model in Equation (4) may be trained by solving the optimization problem
L(·) [Math. 21]
L(·) [Math. 22]
2-regularizer [Math. 23]
is imposed for w, and λ>0 is its parameter. β is constrained due to the definition of the designed kernel function in Equation (1). That is, the optimization problem (5) formulates a one-shot problem instead of two-phase.
However, Problem (5) is non-convex in the current form, meaning that a globally optimal solution may be difficult to obtain. For illustration, the upper panel of
To make the problem convex, let
{tilde over (w)}(k):=√{square root over (β(k))}w(k) for k=1, . . . ,K. [Math. 24]
Then the following convex optimization problem may be solved equivalently to obtain a globally optimal solution.
where
{tilde over (w)}(k)∈ [Math. 26]
is a sub-vector of
{tilde over (w)}∈ [Math. 27]
As described above, the convex problem formulating component 104 is configured to formulate a non-convex problem for training the predictive model into the convex optimization problem based on the explicit feature map by using a variable substitution trick.
The ADMM transforming component 105 transforms the convex optimization problem in Problem (6) into an ADMM form, and then the model training component 106 distributes the computation for training a predictive model among a group of computing nodes to perform ADMM iterations.
To efficiently solve Problem (6), it is convenient to alternatively minimize the objective function
w·r·t·{tilde over (w)} [Math. 28]
and w·r·t·β. First, the minimization
w·r·t·{tilde over (w)} [Math. 29]
is considered with a fixed feasible β, and Problem (6) is written in a compact form as
where the k-th block of embedded data is
{tilde over (Φ)}(k)∈ [Math. 31]
with the i-th row as
{circumflex over (Φ)}(xi(k)) [Math. 32]
and the vector of prediction target is
γ∈ [Math. 33]
with the i-th element as γi.
In Problem (7),
{tilde over (w)} [Math. 34]
is separated into sub-vectors
{tilde over (w)}(k) [Math. 35]
in the same way in the loss function and regularization terms. Hence, it can be expressed in an ADMM form as
with auxiliary variables
v(k)∈ [Math. 37]
as sub-vectors of
v∈ [Math. 38]
The variables
{tilde over (w)} [Math. 39]
now are referred to as primal variables in ADMM.
Since the optimization problem now admits an ADMM form as in Problem (8), it may be solved via the ADMM algorithm. The augmented Lagrangian with scaled dual variables
ũ(k) [Math. 40]
for Problem (8) is formed as
Then the following ADMM iterations may be performed until a stopping criterion for convergence is satisfied:
where the matrix of the entire embedded data is
Φ=[Φ(1)Φ(2) . . . Φ(K)]∈ [Math. 43]
It is observed that the
{tilde over (w)}-update [Math. 44]
step in Equation (9) and the
ũ-update [Math. 45]
step in Equation (11) may be carried out in parallel. In this parallelized case, the ADMM iterations are written as
The ADMM iterations may be further simplified by introducing an additional variable
Then the simplified ADMM iterations are derived as
where
The
{tilde over (w)}-update [Math. 50]
step in Equation (15) essentially involves K independent ridge regression problems that can be solved in parallel. The solution of the
step in Equation (16) depends on the loss function
L(·). [Math. 52]
For example, in cases of the square loss, the solution admits a simple closed form; in cases of the hinge loss, the solution may be analytically obtained using the soft-thresholding technique. In the straightforward u-update step, the vectors of dual variables
ũ(k) [Math. 53]
are replaced by a single one u because all of them are equal.
The above ADMM algorithm gives a solution of
{tilde over (w)}. [Math. 54]
With this
{tilde over (w)} [Math. 55]
fixed, the solution of β can be obtained by solving the following convex problem
which has a closed form solution
This β-update step may be done either inside or outside the ADMM iterations, termed as “inner update” and “outer update” respectively.
As described above, a combination of the ADMM transforming component 105 and the model training component 106 serves as an optimal solution solving component configured to solve the convex optimization problem to obtain the globally optimal solution for training the interpretable predictive model.
Φ[Φ(1)Φ(2) . . . Φ(K)]∈ [Math. 58]
according to feature representations, and distribute them to computing nodes 107. The global node 108 initializes sub-kernel coefficients β and ADMM variables: primal variables
{tilde over (w)}, [Math. 59]
auxiliary variables
and dual variables
ũ. [Math. 61]
In the broadcast step 204, the global node 108 communicates with local nodes 109 and shares the information of sub-kernel coefficients and ADMM variables. The step 205 is performed in parallel among local nodes, computing the solutions to update primal variables according to Equation (15). In the gather step 206, the global node 108 collects all of updated primal variables and compute the solution of sub-kernel coefficients as in Equation (18). Then the global node 108 checks whether an optimal β is obtained in the step 208 according to a certain criterion: if not, the process goes back to the step 204; otherwise, it proceeds to the step 209 to update auxiliary and dual variables on the global node as in Equation (16) and Equation (17). In the step 210, the global node checks whether a stopping criterion of ADMM is satisfied: if not, the process goes back to the step 204; otherwise, it proceeds to the end step 211 to output the trained model 110 with the final solutions of sub-kernel coefficients and ADMM variables.
The main difference between the process 200 and the process 300 is when the sub-kernel coefficients β are updated. In the process 200, the β-update step is inside ADMM iterations. This requires several times of communication between the global node 108 and local nodes 109 when alternatively updating primal variables
{tilde over (w)} [Math. 62]
and sub-kernel coefficient β. On the other hand, the β-update step is outside ADMM iterations in the process 300. However, whenever a new but not optimal β is obtained in the step 309, a new epoch of ADMM iterations have to be restarted from the step 304. While in the process 200, there is only one epoch of ADMM iterations.
The respective components of the kernel learning apparatus 100 may be realized using a combination of hardware and software. In a mode where the hardware and the software are combined with each other, the respective components of the kernel learning apparatus 100 are realized as respective various means by developing a kernel learning program in an RAM (random access memory) and by causing the hardware such as a control unit (CPU: central processing unit) and so on to operate based on the kernel learning program. In addition, the kernel learning program may be distributed with it recoded in a recording medium. The kernel learning program recorded in the recording medium is read out to a memory via a wire, a radio, or the recording medium itself to cause the control unit and so on to operate. As the recording medium, an optical disc, a magnetic disk, a semiconductor memory device, a hard disk or the like is exemplified.
If the above-mentioned example embodiment is explained by a different expression, the example embodiment may be realized by causing a computer serving as the kernel learning apparatus 100 to operate, based on the kernel learning program developed in the RAM, as the data preprocessing component 102, the explicit feature mapping component 103, the convex problem formulating component 104, and the optimal solution solving component (the ADMM transforming component 105 and the model training component 106).
Now, description will proceed to an example of the present invention with reference to drawings. In the example being illustrated, the example is an example of prediction task for predicting, as a prediction target γ, a house value based on, for example, California Hosing Dataset. It is assumed that the California Hosing Dataset has, as the D features, first through eighth features x1 to x8 as described in the following Table 1. That is, in the example being illustrated, D is equal to eight.
When the California Hosing Dataset is supplied to the trained model 110, the trained model 110 produces the degree of importance for the features in the prediction task, as being illustrated in
Furthermore, the trained model 110 further produces two drawings as illustrated in
Specifically,
Moreover, the trained model 110 further produces an explanation view indicative of a visualized example of the partial dependence for the features representing an interaction effect as shown in
With this configuration, a user can use, as decision making, a predicted selling value and the dependence. For example, the user can determine, based on outputs of the trained model 110, an optimal sales strategy of the house value.
While the invention has been particularly shown and described with reference to an example embodiment thereof, the invention is not limited to the embodiment. It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the sprit and scope of the present invention as defined by the claim. For example, the optimal solution solving component may be implemented by any one selected from other solving components although the optimal solution solving component comprises the combination of the ADMM transforming component 105 and the model training component 106 in the above-mentioned example embodiment. More specifically, the ADMM transforming component 105 may be omitted. In this event, the optical solution solving component is implemented by only the model training component except for the ADMM.
The present application is a continuation application of U.S. patent application Ser. No. 17/041,733 filed on Sep. 25, 2020, which is a National Stage Entry of International Application PCT/JP2018/012159 filed Mar. 26, 2018, the disclosures of which is incorporated herein in its entirety by reference.
Number | Date | Country | |
---|---|---|---|
Parent | 17041733 | Sep 2020 | US |
Child | 18240221 | US |