The present disclosure relates to a machine learning device and an environment adjusting apparatus including the same.
International Publication No. 2007/007632 discloses a configuration that infers the comfort of a subject by performing chaos analysis on time-series data of biological information of the subject and controls an environment adjusting apparatus on the basis of the inferred result.
A machine learning device according to a first aspect is configured to learn a thermal sensation of a subject. The machine learning device includes a first acquisition unit, a second acquisition unit, and a learning unit. The first acquisition unit is configured to acquire a first variable including a parameter related to biological information of the subject. The second acquisition unit is configured to acquire a second variable including a thermal sensation of the subject. The learning unit is configured to learn the first variable and the second variable in association with each other.
An environment adjusting apparatus 10 according to a first embodiment will be described with reference to the drawings. The environment adjusting apparatus 10 is an apparatus that adjusts an environment in a target space. In the first embodiment, the environment adjusting apparatus 10 is an air-conditioning control apparatus.
The environment adjusting apparatus 10 predicts a thermal sensation of a subject 20 in the target space by using biological information of the subject 20. On the basis of a predicted value of the thermal sensation of the subject 20, the environment adjusting apparatus 10 grasps the comfort of the subject 20 and implements air-conditioning control for achieving the comfort. The thermal sensation is an index representing the comfort of the subject 20 in the target space. For example, PMV (Predicted Mean Vote) is used as the index of the thermal sensation.
The environment adjusting apparatus 10 includes a machine learning device 100 that learns the thermal sensation of the subject 20 by using a machine learning technique. The machine learning device 100 is constituted by one or a plurality of computers. In the case where the machine learning device 100 is constituted by a plurality of computers, the plurality of computers may be connected to each other via a network.
The state variable acquisition unit 101 acquires a state variable (first variable) including at least one parameter related to biological information of the subject 20.
The control amount acquisition unit 102 acquires a control amount (second variable) including a thermal sensation of the subject 20.
As illustrated in
The function updating unit 104 calculates the reward on the basis of the control amount acquired by the control amount acquisition unit 102 and a predicted value of the control amount. Specifically, the function updating unit 104 calculates a higher reward as the thermal sensation of the subject 20 included in the control amount is closer to the predicted value of the thermal sensation of the subject 20. That is, the reward calculated by the function updating unit 104 increases as a difference between the actual value of the thermal sensation of the subject 20 and the predicted value of the thermal sensation of the subject 20 decreases.
As illustrated in
The state variable acquired by the state variable acquisition unit 101 includes at least one of parameters correlated to a brain wave, a skin blood flow rate, a skin temperature, an amount of sweat, and a heartbeat of the subject 20. The parameter correlated to a brain wave is at least one of the amplitude of the brain wave, the maximum value of the wave height of the brain wave, and the maximum Lyapunov exponent. The parameter correlated to a skin temperature is at least one of a skin temperature of a specific body portion of the subject 20 and a difference in skin temperature between two specific body portions of the subject 20. The parameter correlated to a heartbeat is, for example, an R-R interval.
The control amount acquisition unit 102 acquires the control amount including the thermal sensation of the subject 20 on the basis of at least one of a value related to the thermal sensation input by the subject 20 and an operation situation of the environment adjusting apparatus 10. The value related to the thermal sensation input by the subject 20 is a thermal sensation based on a subjective vote of the subject 20. For example, the value related to the thermal sensation input by the subject 20 is a thermal sensation input by the subject 20 based on a subjective sensation of the subject 20 and is a thermal sensation calculated from an answer from the subject 20 to a question related to the thermal sensation. The operation situation of the environment adjusting apparatus 10 refers to, for example, a parameter correlated to the brain wave of the subject 20 at the time of the operation of the environment adjusting apparatus 10.
The machine learning device 100 acquires the predicted value of the thermal sensation of the subject 20 by using biological information of the subject 20 which is an objective index. Thus, inclusion of the machine learning device 100 allows the environment adjusting apparatus 10 to acquire the predicted value of the thermal sensation of the subject 20 with a high accuracy. Therefore, the environment adjusting apparatus 10 can implement air-conditioning control for achieving the comfort of the subject 20 on the basis of the predicted value of the thermal sensation of the subject 20.
An environment adjusting apparatus 10 according to a second embodiment will be described with reference to the drawings. The environment adjusting apparatus 10 according to the first embodiment and the environment adjusting apparatus 10 according to the second embodiment have a common basic configuration. Differences between the first embodiment and the second embodiment will be mainly described below.
The operation amount candidate output unit 106 outputs candidates for an environmental parameter (third variable) for use in adjusting an environment in a target space. The environmental parameter includes a temperature in the target space. The operation amount candidate output unit 106 outputs candidates for the environmental parameter from a predetermined environmental parameter list, for example. As illustrated in
The operation amount determining unit 107 determines the environmental parameter such that the predicted value of the thermal sensation of the subject 20 satisfies a predetermined condition. Specifically, the operation amount determining unit 107 determines the environmental parameter such that a difference between a target value of the thermal sensation of the subject 20 and the predicted value inferred by the inference unit 105 decreases. As illustrated in
In the second embodiment, from among the candidates for the environmental parameter, the operation amount determining unit 107 can determine the environmental parameter suitable for creating a trained model capable of acquiring the predicted value of the thermal sensation of the subject 20 with a high accuracy. Therefore, the environment adjusting apparatus 10 can acquire the predicted value of the thermal sensation of the subject 20 with a high accuracy and implement air-conditioning control for achieving the comfort of the subject 20 on the basis of the predicted value of the thermal sensation of the subject 20.
An environment adjusting apparatus 10 according to a third embodiment will be described with reference to the drawings. The environment adjusting apparatus 10 is an apparatus that adjusts an environment in a target space. In the third embodiment, the environment adjusting apparatus 10 is an air-conditioning control apparatus.
The environment adjusting apparatus 10 predicts a thermal sensation of a subject 20 in the target space by using biological information of the subject 20. On the basis of a predicted value of the thermal sensation of the subject 20, the environment adjusting apparatus 10 grasps the comfort of the subject 20 and implements air-conditioning control for achieving the comfort.
The environment adjusting apparatus 10 includes a machine learning device 200 that learns a control parameter of the environment adjusting apparatus 10. The machine learning device 200 is constituted by one or a plurality of computers. In the case where the machine learning device 200 is constituted by a plurality of computers, the plurality of computers may be connected to each other via a network.
The state variable acquisition unit 201 acquires a state variable (first variable) including at least one parameter related to biological information of the subject 20 in the target space.
The control amount acquisition unit 202 acquires, as a control amount, a control parameter of the environment adjusting apparatus 10.
The evaluation data acquisition unit 205 acquires evaluation data for evaluating a control result of the environment adjusting apparatus 10.
The function updating unit 204 updates a learning state of the learning unit 203 by using the evaluation data acquired by the evaluation data acquisition unit 205.
As illustrated in
The learning unit 203 performs learning in accordance with an output of the function updating unit 204. In the third embodiment, the learning unit 203 performs reinforcement learning in which learning is performed by using a reward. The function updating unit 204 calculates the reward on the basis of the evaluation data acquired by the evaluation data acquisition unit 205. Specifically, the function updating unit 204 calculates a higher reward as the thermal sensation of the subject 20 is closer to neutral.
As illustrated in
The evaluation data acquisition unit 205 inputs predetermined to-be-assessed data to a predetermined evaluation function, and acquires an output value of the evaluation function as the evaluation data. That is, the evaluation function receives the to-be-assessed data as an input value from the evaluation data acquisition unit 205, and outputs the evaluation data. The to-be-assessed data is at least one of the value related to the thermal sensation input by the subject 20 and the operation situation of the environment adjusting apparatus 10. The value related to the thermal sensation input by the subject 20 is a thermal sensation based on a subjective vote of the subject 20. For example, the value related to the thermal sensation input by the subject 20 is a thermal sensation input by the subject 20 based on a subjective sensation of the subject 20 and is a thermal sensation calculated from an answer from the subject 20 to a question related to the thermal sensation. The operation situation of the environment adjusting apparatus 10 refers to, for example, a parameter correlated to the brain wave of the subject 20 at the time of the operation of the environment adjusting apparatus 10.
The evaluation data acquired by the evaluation data acquisition unit 205 includes at least the thermal sensation of the subject 20. The evaluation data is, for example, a predicted value of the thermal sensation of the subject 20. The predicted value of the thermal sensation of the subject 20 is acquired from at least one of the value related to the thermal sensation input by the subject 20 and the operation situation of the environment adjusting apparatus 10. The evaluation data may be a difference between the predicted value of the thermal sensation of the subject 20 and a neutral value of a thermal sensation. In this case, the function updating unit 204 calculates a higher reward as the difference, which is the evaluation data acquired by the evaluation data acquisition unit 205, is closer to zero.
The state variable acquired by the state variable acquisition unit 201 includes at least one of parameters correlated to a brain wave, a skin blood flow rate, a skin temperature, and an amount of sweat of the subject 20. The parameter correlated to a brain wave is at least one of the amplitude of the brain wave, the maximum value of the wave height of the brain wave, and the maximum Lyapunov exponent. The parameter correlated to a skin temperature is at least one of a skin temperature of a specific body portion of the subject 20 and a difference in skin temperature between two specific body portions of the subject 20.
The machine learning device 200 acquires the thermal sensation of the subject 20 on the basis of biological information of the subject 20 which is an objective index, and determines the control parameter of the environment adjusting apparatus 10 on the basis of the thermal sensation of the subject 20. Thus, inclusion of the machine learning device 200 allows the environment adjusting apparatus 10 to acquire the control parameter in which the biological information of the subject 20 is directly reflected. Therefore, the environment adjusting apparatus 10 can implement air-conditioning control for achieving the comfort of the subject 20 on the basis of the thermal sensation of the subject 20.
At least some modifications of the embodiments will be described below.
In the third embodiment, the learning unit 203 performs reinforcement learning in which learning is performed by using a reward. However, instead of reinforcement learning, the learning unit 203 may perform supervised learning in which learning is performed on the basis of training data.
An environment adjusting apparatus 10 according to a modification A will be described with reference to the drawings. The environment adjusting apparatus 10 according to the third embodiment and the environment adjusting apparatus 10 according to the modification A have a common basic configuration. Differences between the third embodiment and the modification A will be mainly described below.
The function updating unit 204 includes a training data accumulation unit 204a and an assessment unit 204b. By using the evaluation data acquired by the evaluation data acquisition unit 205, the assessment unit 204b outputs an assessment result of the evaluation data. In accordance with the assessment result obtained by the assessment unit 204b, the training data accumulation unit 204a accumulates training data based on the state variable acquired by the state variable acquisition unit 201 and the control parameter acquired by the control amount acquisition unit 202.
The learning unit 203 slightly alters a parameter of a discriminant function in accordance with the output of the function altering unit 207. The learning unit 203 alters the parameter of the discriminant function a plurality of times and outputs, for each discriminant function whose parameter has been altered, the control parameter from the state variable. The discriminant function refers to a mapping from the state variable included in training data to the control parameter. Specifically, the discriminant function is a function whose input variable is the state variable and whose output variable is the control parameter. The function altering unit 207 outputs the parameter of the discriminant function. If it is determined that the evaluation data obtained as a result of control of the environment adjusting apparatus 10 on the basis of the control parameter output by the learning unit 203 from the state variable is appropriate, the function updating unit 204 accumulates, as training data, the state variable and the control parameter output by the learning unit 203 from the state variable.
The learning unit 203 performs learning on the basis of the training data accumulated in the training data accumulation unit 204a. The purpose of learning performed by the learning unit 203 is to adjust the parameter of the discriminant function by using the training data as learning data so that correct or appropriate evaluation data can be obtained from a new state variable. The learning unit 203 uses, as the learning data, pairs of the state variable acquired in advance by the state variable acquisition unit 201 and the control parameter acquired by the control amount acquisition unit 202. The discriminant function whose parameter is sufficiently adjusted by the learning unit 203 corresponds to the trained model.
The control amount determining unit 206 determines the control parameter from a new state variable on the basis of the trained model obtained as a result of learning performed by the learning unit 203.
As described next, the learning unit 203 performs supervised learning based on online learning or batch learning.
In supervised learning based on online learning, the learning unit 203 generates a trained model in advance by using data (state variable) acquired in a test operation or the like performed before shipment or installation of the environment adjusting apparatus 10. At the time of the start of the initial operation of the environment adjusting apparatus 10, the control amount determining unit 206 determines the control parameter on the basis of the trained model generated in advance by the learning unit 203. The learning unit 203 then updates the trained model by using data (state variable) newly acquired during the operation of the environment adjusting apparatus 10. The control amount determining unit 206 determines the control parameter on the basis of the trained model updated by the learning unit 203. As described above, in the online learning, the trained model is regularly updated, and the control amount determining unit 206 determines the control parameter on the basis of the latest trained mode.
In supervised learning based on batch learning, the learning unit 203 generates a trained model in advance by using data (state variable) acquired in a test operation or the like performed before shipment or installation of the environment adjusting apparatus 10. At the time of the operation of the environment adjusting apparatus 10, the control amount determining unit 206 determines the control parameter on the basis of the trained model generated in advance by the learning unit 203. This trained model is not updated after being generated in advance by the learning unit 203. That is, the control amount determining unit 206 determines the control parameter by using the same trained model.
Note that a server connected to the environment adjusting apparatus 10 via a computer network such as the Internet may generate the trained model, or the trained model may be generated by using a cloud computing service.
In the first and second embodiments, the learning unit 103 performs reinforcement learning in which learning is performed by using a reward. However, instead of reinforcement learning, the learning unit 103 may perform supervised learning in which learning is performed on the basis of training data, as described in the modification A. In this case, the learning unit 103 may perform learning by using training data obtained from the state variable acquired by the state variable acquisition unit 101 and the control amount (the thermal sensation of the subject 20) acquired by the control amount acquisition unit 102.
In the modifications A and B, in the case where the learning units 103 and 203 perform supervised learning in which training data is used, the learning units 103 and 203 may use part of the training data as learning data to adjust the parameter of the discriminant function and may use the rest of the training data as test data. The test data is data that is not used in learning and is mainly used for evaluation of the performance of the trained model. The use of the test data enables the accuracy of the evaluation data obtained from a new state variable to be predicted in a form of an error probability for the test data. As techniques for splitting pieces of data acquired in advance into learning data and test data, hold-out, cross-validation, leave-one-out (jackknife), bootstrapping, and the like are used.
Supervised learning that is a machine learning technique used by the learning units 103 and 203 in the modifications A to C will be described. Supervised learning is a technique for generating an output corresponding to unseen input data by using training data. In supervised learning, learning data and a discriminant function are used. The learning data is a set of pairs of input data and training data corresponding to the input data. The input data is, for example, a feature vector in a feature space. The training data is, for example, parameters regarding discrimination, classification, and evaluation of the input data. The discriminant function represents a mapping from input data to an output corresponding to the input data. Supervised learning is a technique of adjusting a parameter of the discriminant function by using learning data given in advance such that a difference between an output of the discriminant function and training data decreases. Models or algorithms used in supervised learning include a regression analysis, a time-series analysis, a decision tree, a support vector machine, a neural network, ensemble learning, etc.
The regression analysis is, for example, a linear regression analysis, a multiple regression analysis, or a logistic regression analysis. The regression analysis is a technique for applying a model between input data (explanatory variable) and training data (objective variable) by using the least squares method or the like. The dimension of the explanatory variable is 1 in the linear regression Analysis and 2 or higher in the multiple regression analysis. In the logistic regression analysis, a logistic function (sigmoid function) is used as the model.
The time-series analysis refers to, for example, an AR model (autoregressive model), an MA model (moving average model), an ARMA model (autoregressive moving average model), an ARIMA model (autoregressive integrated moving average model), an SARIMA model (seasonal autoregressive integrated moving average model), or a VAR model (vector autoregressive model). The AR, MA, ARMA, and VAR models represent a stationary process. The ARIMA and SARIMA models represent a non-stationary process. The AR model is a model in which a value regularly changes as time passes. The MA model is a model in which a fluctuation in a certain period is constant. For example, in the MA model, a value at a certain time point is determined by a moving average before the time point. The ARMA model is a combined model of the AR model and the MA model. The ARIMA model is a model in which the ARMA model is applied to a difference between preceding and following values in consideration of a middle-term or long-term trend (increasing or decreasing trend). The SARIMA model is a model in which the ARIMA model is applied in consideration of a middle-term or long-term seasonal fluctuation. The VAR model is a model in which the AR model is expanded to support multiple variables.
The decision tree is a model for generating complex discrimination boundaries by combining a plurality of discriminators. Details of the decision tree will be described later.
The support vector machine is an algorithm for generating a two-class linear discriminant function. Details of the support vector machine will be described later.
The neural network is obtained by modeling a network that is formed by connecting neurons of the human cranial nervous system by synapses. The neural network means a multi-layer perceptron that uses error backpropagation in a narrow sense. The typical neural networks include a convolutional neural network (CNN) and a recurrent neural network (RNN). The CNN is a type of a non-fully-connected (coarsely-connected) forward-propagation neural network. The RNN is a type of the neural network having a directed cycle. The CNN and the RNN are used in audio/image/moving image recognition and natural language processing.
The ensemble learning is a technique for improving the discrimination performance by combining a plurality of models. The technique used in the ensemble learning is, for example, bagging, boosting, or a random forest. Bagging is a technique for training a plurality of models by using bootstrap sampling of learning data and determining evaluation for new input data by a majority vote of the plurality of models. Boosting is a technique for weighting learning data in accordance with a bagging-based learning result, so that incorrectly discriminated learning data is learned in a more concentrated manner than correctly discriminated learning data. The random forest is a technique for generating a decision tree group (random forest) constituted by a plurality of decision trees having a low correlation in the case where the decision tree is used as the model. Details of the random forest will be described later.
The neural network, the support vector machine, the decision tree, and the random forest, which will be described next, are used as preferable models or algorithms of supervised learning used by the learning units 103 and 203.
y=φ(Σi=1nxiwi−θ) (1)
In Expression (1), all of the inputs x, the output y, and the weights w are vectors, θ denotes a bias, and φ denotes an activation function. The activation function is a non-linear function and is, for example, a step function (formal neuron), a simple perceptron, a sigmoid function, or a ReLU (ramp function).
In the three-layer neural network illustrated in
In the first layer L1, the input vectors x1, x2, and x3 are multiplied by corresponding weights and are input to each of three neurons N11, N12, and N13. In
In the second layer L2, the feature vectors z11, z12, and z13 are multiplied by corresponding weights and are input to each of two neurons N21 and N22. In
In the third layer L3, the feature vectors z21 and z22 are multiplied by corresponding weights and are input to each of three neurons N31, N32, and N33. In
There are a learning mode and a prediction mode in terms of operation of the neural network. In the learning mode, the neural network learns the weights W1, W2, and W3 by using a learning dataset. In the prediction mode, the neural network performs prediction such as discrimination by using the parameters of the learned weights W1, W2, and W3.
The weights W1, W2, and W3 can be learned through error backpropagation (backpropagation), for example. In this case, information regarding the error is transferred from the output side toward the input side, that is, from the right side toward the left side in
The neural network can be configured to have more than three layers. A machine learning technique using a neural network having four or more layers is known as deep learning.
The support vector machine (SVM) is an algorithm that determines a two-class linear discriminant function that implements the maximum margin.
In
D
L={(ti,xi)} (i=1, . . . ,N) (2)
The learning dataset DL is a set of pairs of learning data (feature vector) xi and training data ti={−1, +1}. The number of elements of the learning dataset DL is N. The training data ti indicates which of the classes C1 and C2 the learning data xi belongs to. The class C1 is a class denoted by ti=−1, and the class C2 is a class denoted by ti=+1.
A normalized linear discriminant function that holds for all the pieces of learning data xi in
If ti=+1, wTxi+b≥+1 (3-1)
If ti=−1, wTxi+b≤−1 (3-2)
These two Expressions are represented by one Expression (4) below.
t
i(wTxi+b)≥1 (4)
In the case where each of the discrimination hyperplanes P1 and P2 is represented by Expression (5) below, the margin d thereof is represented by Expression (6).
In Expression (6), ρ(w) denotes the minimum value of a difference between lengths obtained by projecting the learning data xi of the class C1 and the learning data xi of the class C2 onto a normal vector w of the discrimination hyperplanes P1 and P2. The terms “min” and “max” in Expression (6) indicate points denoted by reference signs “min” and “max” in
t
i(wTxi+b)−1+ξi≥0 (7)
The slack variable ξi is used only at the time of learning and takes a value of 0 or greater.
In the case where the slack variable ξi is equal to 0, Expression (7) is equivalent to Expression (4). At this time, as indicated by blank circles or squares in
In the case where the slack variable ξi is greater than 0 and less than or equal to 1, as indicated by a hatched circle or square in
In the case where the slack variable ξi is greater than 1, as indicated by black circles or squares in
The use of Expression (7) in which the slack variable ξi is introduced enables the learning data xi to be discriminated in this manner also in the case where pieces of learning data of two classes are linearly inseparable.
From the description above, the sum of the slack variable ξi for all the pieces of learning data xi indicates the upper limit of the number of pieces of learning data xi incorrectly recognized. Here, an evaluation function Lp is defined by Expression (8) below.
L
p(w,ξ)=½wTw+CΣi=1Nξi (8)
The learning units 103 and 203 find a solution (w, ξ) that minimizes an output value of the evaluation function Lp. In Expression (8), a parameter C of the second term denotes a strength of a penalty for incorrect recognition. As the parameter C increases, a solution for prioritizing a reduction in the number of incorrect recognition (second term) over a norm (first term) of w is determined.
The decision tree is a model for obtaining a complex discrimination boundary (such as a non-linear discriminant function) by combining a plurality of discriminators. A discriminator is, for example, a rule regarding a magnitude relationship between a value on a certain feature axis and a threshold. Examples of a method for creating a decision tree from learning data include a divide and conquer algorithm for repeatedly finding a rule (discriminator) for dividing a feature space into two.
In the process of creating an appropriate decision tree by using the divide and conquer algorithm, it is necessary to consider three points (a) to (c) below.
(a) Selection of a feature axis and a threshold for configuring a discriminator.
(b) Determination of a terminal node. For example, the number of classes to which the learning data included in one terminal node belongs. Alternatively, selection of how far decision tree pruning (obtaining subtrees having the same root node) is to be performed.
(c) Assignment of a class to a terminal node by a majority vote.
In a decision-tree-based learning method, for example, CART, ID3, and C4.5 are used. CART is a technique for generating a binary tree as a decision tree by dividing, for each feature axis, a feature space into two at each of nodes other than terminal nodes as illustrated in
In learning using a decision tree, to improve the learning data discrimination performance, it is important to divide the feature space at an appropriate division candidate point at a non-terminal node. An evaluation function called a diversity index may be used as a parameter for evaluating the division candidate point of the feature space. As a function I(t) representing the diversity index of a node t, for example, parameters represented by Expressions (9-1) to (9-3) below are used. K denotes the number of classes.
(a) Error Rate at Node t
(b) Cross-Entropy (Deviance)
I(t)=−Σi=1KP(Ci|t)ln P(Ci|t) (9-2)
(c) Gini Coefficient
I(t)=Σi-1KΣj≠iP(Ci|t)P(Cj|t)=Σi-1KP(Ci|t)(1−P(Ci|t)) (9-3)
In Expressions above, a probability P(Ci|t) is a posterior probability of a class Ci at the node t, that is, a probability of data of the class Ci being selected at the node t. In the second part of Expression (9-3), a probability P(Cj|t) is a probability of data of the class Ci being incorrectly discriminated to be in a j-th (≠i-th) class. Thus, the second part represents an error rate at the node t. The third part of Expression (9-3) represents a sum of variances of the probability P(Ci|t) for all the classes.
In the case of dividing a node by using the diversity index as the evaluation function, for example, a technique of pruning the decision tree up to an allowable range that is determined by an error rate at the node and by the complexity of the decision tree is used.
The random forest is a type of ensemble learning and is a technique for enhancing the discrimination performance by combining a plurality of decision trees. In learning using the random forest, a group (random forest) of a plurality of decision trees having a low correlation is generated. The following algorithm is used in generation of the random forest and discrimination using the random forest.
(A) The following is repeated while m=1 to M.
(B) A random forest constituted by the m decision trees is output.
(C) A discrimination result of each decision tree of the random forest for input data is obtained. A discrimination result of the random forest is determined by a majority vote of the discrimination results of the respective decision trees.
In learning using the random forest, a correlation between decision trees can be made low by randomly selecting a predetermined number of features for use in discrimination at individual non-terminal nodes of the decision trees.
Reinforcement learning that is a machine learning technique used by the learning units 103 and 203 in the first to third embodiments will be described. Reinforcement learning is a technique for learning a policy that maximizes a reward which is a result of a series of actions. Models or algorithms used in reinforcement learning include Q-learning or the like. Q-learning is a technique for learning a Q-value that represents a value of selecting an action a in a state s. In Q-learning, an action a with the highest Q-value is selected as an optimum action. To determine a high Q-value, an entity (agent) of the action a is rewarded for the action a selected in the state s. In Q-learning, the Q-value is updated by using Expression (10) below every time the agent takes an action.
In Expression (10), Q(st, at) is the Q-value that represents a value of the agent in a state st selecting an action at. Q(st, at) is a function (action-value function) having a state s and an action a as parameters. st denotes a state of the agent at a time t. at denotes an action of the agent at the time t. α denotes a learning coefficient. α is set such that the Q-value converges to an optimum value in accordance with Expression (10). rt+1 denotes a reward obtained when the agent transitions to a state st+1. γ denotes a discount factor γ is a constant that is greater than or equal to 0 and less than or equal to 1. The term including max is a product obtained by multiplying by γ the Q-value in the case of selecting the action a with the highest Q-value in the state st+1. The Q-value determined by using the action-value function is an expected value of the reward to be obtained by the agent.
In the third embodiment, the machine learning device 200 includes the control amount acquisition unit 202. However, the machine learning device 200 need not include the control amount acquisition unit 202. In this case, the learning unit 203 of the machine learning device 200 may use, as the learning data, the control parameter determined by the control amount determining unit 206.
In the embodiments and modifications described above, the machine learning devices 100 and 200 use supervised learning or reinforcement learning. However, the machine learning devices 100 and 200 may use a combination technique of supervised learning and reinforcement learning.
In the embodiments and modifications described above, the learning units 103 and 203 may use various machine learning techniques. Machine learning techniques that may be used by the learning units 103 and 203 include unsupervised learning, semi-supervised learning, transductive learning, multi-task learning, transfer learning, etc. in addition to supervised learning and reinforcement learning already described. The learning units 103 and 203 may use these techniques in combination.
Unsupervised learning is a technique of grouping (clustering) input data on the basis of a predetermined statistical property without using training data. Models or algorithms used in unsupervised learning include k-means clustering, the Ward's method, the principal component analysis, etc. The k-means clustering is a technique in which a process of randomly assigning a cluster to each piece of input data, calculating the center of each cluster, and re-assigning each piece of input data to a cluster having the nearest center is repeated. The Ward's method is a technique in which a process of assigning each piece of input data to a cluster is repeated to minimize a distance from each piece of input data of a cluster to the mass center of the cluster. The principal component analysis is a technique of a multivariate analysis that generates variables called principal components having the lowest correlation from among a plurality of correlated variables.
The semi-supervised learning is a technique of performing learning by using both input data (unlabeled data) not assigned corresponding training data and input data (labeled data) assigned corresponding training data.
The transductive learning is a technique of generating an output corresponding to unlabeled data for use in learning and not generating an output corresponding to unseen input data in semi-supervised learning.
The multi-task learning is a technique of sharing information among a plurality of related tasks and causing these tasks to simultaneously perform learning to obtain a factor that is common to the tasks and increase the prediction accuracy.
The transfer learning is a technique of applying a model trained in advance in a certain domain to another domain to increase the prediction accuracy.
While the embodiments of the present disclosure have been described above, it should be understood that various modifications can be made on the configurations and details without departing from the gist and the scope of the present disclosure that are described in the claims.
The machine learning device can acquire a predicted value of the thermal sensation of a subject with a high accuracy.
Number | Date | Country | Kind |
---|---|---|---|
2019-213364 | Nov 2019 | JP | national |
This is a continuation of International Application No. PCT/JP2020/044112 filed on Nov. 26, 2020, which claims priority to Japanese Patent Application No. 2019-213364, filed on Nov. 26, 2019. The entire disclosures of these applications are incorporated by reference herein.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2020/044112 | Nov 2020 | US |
Child | 17824503 | US |