The present disclosure is generally directed to apparatus and data management, and more specifically, through optimization of deep learning network architectures for uncertainty estimation.
In the related art, equipment uptime has become increasingly important across difference industries which seek for new ways of increasing equipment availability. From the use of predictive maintenance, one can increase equipment availability, improve the safety of operators, and reduce the environmental incidents. Detecting faults in the system by condition based maintenance (CBM) may be insufficient, because at the time of fault occurrence, the spare parts may be unavailable or the needed resources (e.g., maintainers) may be busy.
Therefore algorithmic failure prediction and remaining useful life estimators have been developed. The predictors/estimators model the degradation process and predict failure time of the component or the time when component performance is below operational requirements. The degradation process of components in a complex system can be affected by many factors, such as undefined fault modes, operational conditions, environmental conditions, and so on. In some cases, such factors are not recorded and thus are considered unknown.
Prediction of failures or estimates of the remaining useful life are inherently uncertain. There can be various sources of uncertainty such as measurement noise, choice of predictive models and their complexity, and so on. Understanding uncertainty can be needed for understanding the utility of the data or a model. For example, based on the estimated uncertainty it is possible to provide confidence bounds on prediction values, Depending on predicted values and uncertainty (or confidence bounds) a decision maker can be more (large confidence bounds) or less careful (low confidence bounds) when taking the predictions into the decision making process.
In the case of high uncertainty, the decision maker can decide if more diverse data is necessary or if new machine learning models are needed for prediction. For example, uncertainty estimation is important when estimating remaining useful life (RUL) of critical equipment such as jet engines. The reliability team would likely schedule maintenance of the plane to meet the time when the lower bound of RUL confidence interval occurs. By doing this, catastrophic failure during operations can be avoided. Another example of importance of uncertainty estimates is operational planning, which involves multiple pieces of equipment performing different activities. Knowing the uncertainty along with predictions for durations of different activities could lead to more confident planning in terms of the final production outcome compared to the approach of taking just predictions alone. Quantifying uncertainty also facilitates better cost optimizations.
In related art implementations for failure prediction, and RUL estimation, time sequence information is taken into consideration through sliding windows, recurrent neural networks, Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) networks. Thus, information processing ranges from independent time windows for sliding window implementations, to sequence dependent time windows for recurrent neural network and LSTM implementations. However, none of these approaches provide any uncertainty estimate for the predictions made.
Generally in deep learning, uncertainty quantification has been an area of active research in the related art. A particular related art technique called “dropout” is regularly used in deep learning during the training phase as a model regularization technique. Related art implementations have demonstrated that dropout can be useful to provide uncertainty information at inference phase using a technique called Monte Carlo (MC) dropout. The amount of dropout is critical to accuracy and uncertainty estimates and selection of optimum dropout and network architecture is an important step. Currently, the dropout that gives best accuracy result in the validation phase is considered as the optimum for the inference phase.
However, related art implementations of robust optimization have shown that when uncertainty and accuracy both are considered, then it is a trade-off problem and thus both should be optimized simultaneously. In the related art implementations in MC dropout, focus is only on the accuracy and dropout is used to provide uncertainty information, and accuracy and uncertainty are not optimized simultaneously.
Example implementations described herein involve a mechanism with foundation in deep learning for tuning parameters of a deep learning network to optimize for accuracy and uncertainty simultaneously. The optimized network will provide prediction values as well as associated uncertainty in the prediction. Based on the foundations of MC dropout, deep learning and optimization in general, example implementations involve a technique along with a fitness function to be optimized that focuses on prediction accuracy and uncertainty simultaneously.
Example implementations are directed to addressing the problem of accurately predicting failures or Remaining Useful Life (RUL) while providing accurate information related to uncertainty in the prediction using time sequence sensor data, failure data and operational data. Such example implementations involve optimizing network parameters for accuracy and uncertainty simultaneously.
Example implementations directed herein involve a dynamic network creation of deep learning architecture. In example implementations, the base architecture stack-up defines the network layer types which are included in the model and the relationship between them. The base architecture stack-up is problem specific and it is assumed to be specified by the user. The base architecture can involve layer types such as Convolutional Neural Network (CNN), Long Short Term Memory (LSTM), and multi-layer fully connected neural network (NN).
In example implementations, the network architecture is created based on input network architecture parameters and base architecture stack-up. Network architecture parameters include, but are not limited to, the number of convolutional layers, number of convolutional filters in each layer, number of LSTM layers, number of LSTM units in each layer, number of fully connected layers, number of hidden units in each fully connected hidden layer, dropout rate (CNN layers, LSTM layers, fully connected layers), training optimization algorithm, training optimization algorithm learning rate, objective function for training, and so on.
Example implementations can also involve a fitness function to evaluate prediction accuracy and uncertainty simultaneously of the network architecture and related parameters under consideration. In example implementations, the fitness function is evaluated on validation dataset and it is used by optimization algorithm to find the optimum network architecture.
Example implementations can also involve an automated optimum network architecture selection and network training, which is a coupling dynamic network architecture creation with an optimization algorithm. The coupling of the dynamic network architecture with an optimization algorithm finds optimum network architecture parameters through dynamically creating and training a deep learning network, using trained deep learning network to evaluate fitness function on validation dataset, optimizing for the fitness function with respect to network architecture parameters, and conducting RUL prediction along with uncertainty through using multiple components.
Aspects of the present disclosure can include a method, which involves a) initializing deep learning architecture parameters for a pre-defined base architecture; b) conducting model training based on the deep learning architecture parameters to generate a trained model; c) obtaining predictions and uncertainties through iteratively applying Monte Carlo (MC) dropout trained model to evaluate the fitness function, wherein the fitness function is configured to evaluate accuracy and uncertainty of the predictions of the training model; d) for the fitness function indicative of the trained model not being optimized, updating the deep learning architecture parameters and repeating the method from step b); and e) for the fitness function indicative of the trained model being optimized, providing the trained model for prediction.
Aspects of the present disclosure can include an apparatus, which involves a processor configured to a) initialize deep learning architecture parameters for a pre-defined base architecture; b) conduct model training based on the deep learning architecture parameters to generate a trained model; c) obtain predictions and uncertainties through iteratively applying Monte Carlo (MC) dropout to the generated trained model to evaluate the fitness function, wherein the fitness function is configured to evaluate accuracy and uncertainty of the predictions of the training model; d) for the fitness function indicative of the trained model not being optimized, update the deep learning architecture parameters and repeating the method from step b); and e) for the fitness function indicative of the trained model being optimized, provide the trained model for prediction.
Aspects of the present disclosure can include a computer program, which involves instructions for a) initializing deep learning architecture parameters for a pre-defined base architecture; b) conducting model training based on the deep learning architecture parameters to generate a trained model; c) obtaining predictions and uncertainties through iteratively applying Monte Carlo (MC) dropout to the generated trained model to evaluate the fitness function, wherein the fitness function is configured to evaluate accuracy and uncertainty of the predictions of the training model; d) for the fitness function indicative of the trained model not being optimized, updating the deep learning architecture parameters and repeat the process from step b); and e) for the fitness function indicative of the trained model being optimized, providing the trained model for prediction. The computer program can be in the form of instructions stored on a non-transitory computer readable medium and executable by one or more processors.
The following detailed description provides further details of the figures and example implementations of the present application. Reference numerals and descriptions of redundant elements between figures are omitted for clarity. Terms used throughout the description are provided as examples and are not intended to be limiting. For example, the use of the term “automatic” may involve fully automatic or semi-automatic implementations involving user or administrator control over certain aspects of the implementation, depending on the desired implementation of one of ordinary skill in the art practicing implementations of the present application. Selection can be conducted by a user through a user interface or other input means, or can be implemented through a desired algorithm. “Uncertainty level” and “confidence level” may also be utilized interchangeably. Example implementations as described herein can be utilized either singularly or in combination and the functionality of the example implementations can be implemented through any means according to the desired implementations.
In an example implementation to demonstrate the working of the methodology, an example of a Remaining Useful Life (RUL) estimation of a system is utilized. RUL is the time remaining before the component of a system reaches end-of-life. Estimating RUL involves incorporating information from time sequence sensor data, event data, failure data and operational data. To accurately estimate RUL and along with the uncertainty using our methodology the following steps should be executed in the sequence presented.
Data preparation 101 involves data input for predictions. The data input for predictions is a set of time sequences generated from sensor measurements, sequence of operations and the events generated during the operations of the component/system which may be relevant to the problem. Several steps are necessary to perform before the data is used as an input to a deep learning algorithm. Example implementations leverage the data processing as previous contributions to the field of failure prediction, RUL estimation, and so on, based on sensors.
In an example, steps are conducted for data preparation 101 to determine RUL, such as outlier removal, and component based sequence construction to make data to be in a format consumable by deep learning. During sequence construction, it can be necessary to transform conventional time-scale to component time-scale as RUL should be expressed in operational time (e.g., it does not include non-operational time such as equipment downtimes, lunch breaks, and so on).
Additional steps for data preparation 101 can involve sensor data compression (feature extraction) by applying windows over the data using (a) predefined functions over the windows such as minimum, maximum, percentiles, average, FFT (Fast Fourier Transform), and so on, and (b) applying on Convolutional Neural Networks (CNNs) for automatic feature extraction. If automatic feature extraction using CNN is applied, then the same should be defined in the base architecture stack-up as described herein.
Additional steps for data preparation 101 can also involve the creation of labels for each element of created sequence with corresponding RUL value, depending on the desired implementation. This can be necessary for learning model parameters in the training phase.
Once the data is prepared, the data is further divided into a training set and a validation set at 102. The training set is used during the model training phase to learn model parameters, while the validation set is used for evaluating a fitness function as described below in the present disclosure.
Next, there is the execution of the dynamic deep learning network architecture creation and training. At this portion, the existing concepts in deep learning network architectures are utilized, which include convolutional neural networks (CNNs), LSTM networks and neural networks (NN). As the first step a user-defined base architecture stack-up is created at 103. The base architecture stack-up in example implementations is defined as the relationship between the convolutional layer, LSTM layer and NN layer. The base architecture stack-up can be defined by the user to include the if and where convolutional layers, LSTM layers and NN layers are implemented, which layers are connected to each other, and so on in accordance with the desired implementation.
Turning back to
As an example, in the case of RUL estimation, as shown in
At 106, the model is then trained. After the base architecture stack-up definition and the network architecture parameter initialization by the main optimization algorithm, the model is trained using the training dataset as shown in
At 107, the prediction and uncertainty are obtained through the use of MC dropout, whereupon the fitness function evaluation is executed at 108. In example implementations, accuracy and uncertainty are simultaneously optimized. Thus, example implementations involve a novel fitness function is developed that is evaluated using the validation dataset. As stated above, the related art shows that the MC dropout mechanism approximates a Gaussian distribution. Thus, example implementations involve a fitness function as follows:
The Gaussian distribution is represented as:
where,
x=input data
w=network parameters (e.g., deep learning architecture parameters)
d=dropout rate
μ=mean of the prediction sampled using MC dropout mechanism
σ=deviation of the prediction sampled using MC dropout mechanism
P(y|x, w, d)=likelihood of predicting y given input data x, network parameters (e.g., trained model provided from the deep learning architecture parameters) w and dropout d.
The fitness function to be maximized is defined as log-likelihood on validation set=Σi=1nP(yi|x, w, d), where n is the number of data points in the validation dataset.
Evaluating the fitness function involves a two-step procedure. In the first step the trained model is used to estimate the predictions and uncertainty in the prediction using the validation dataset and the MC dropout method. In the current practice during the inference phase or prediction phase, the dropout is de-activated; however in this case of MC dropout it is kept activated to evaluate the predictions and related uncertainty that is ultimately used to evaluate the fitness function. This is explained using the RUL estimation example below.
For example in the case of RUL estimation on validation dataset, RUL is evaluated multiple number of times for the same instance (Monte Carlo samples), using the trained model and validation dataset by keeping dropout activated.
Turning back to
In an example implementation, the flow as illustrated in
In such an example implementation, the historical data that is provided in the database 803 can serve as a basis for training the model and generating an optimized model that provides prediction and an uncertainty level. For example, data stored in the database 803 from the desired apparatuses or types of apparatuses to be modeled are prepared through the execution of the flow at 101, whereupon a training and validation set is defined from the data at 102. The user defines a base architecture stack-up at 103 by defining the architecture as illustrated in
As illustrated in
Although example implementations described herein are directed to RUL, the present disclosure is not limited thereto, and any parameter that requires a model for generation of a prediction and uncertainty level can be applied. Examples of other parameters that can be determined from the generated model can include, but are not limited to, estimated time of arrival for a vehicle, expected power consumption for a set of equipment, expected network traffic for data feedback from the apparatuses to the server, estimated cost of repairs for a month, and so on according to the desired implementation. As long as a historical dataset is provided in database 803 with the associated data of a desired parameter, the flow diagram as illustrated in
Computer device 905 in computing environment 900 can include one or more processing units, cores, or processors 910, memory 915 (e.g., RAM, ROM, and/or the like), internal storage 920 (e.g., magnetic, optical, solid state storage, and/or organic), and/or I/O interface 925, any of which can be coupled on a communication mechanism or bus 930 for communicating information or embedded in the computer device 905. I/O interface 925 is also configured to receive images from cameras or provide images to projectors or displays, depending on the desired implementation.
Computer device 905 can be communicatively coupled to input/user interface 935 and output device/interface 940. Either one or both of input/user interface 935 and output device/interface 940 can be a wired or wireless interface and can be detachable. Input/user interface 935 may include any device, component, sensor, or interface, physical or virtual, that can be used to provide input (e.g., buttons, touch-screen interface, keyboard, a pointing/cursor control, microphone, camera, braille, motion sensor, optical reader, and/or the like). Output device/interface 940 may include a display, television, monitor, printer, speaker, braille, or the like. In some example implementations, input/user interface 935 and output device/interface 940 can be embedded with or physically coupled to the computer device 905. In other example implementations, other computer devices may function as or provide the functions of input/user interface 935 and output device/interface 940 for a computer device 905.
Examples of computer device 905 may include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by humans and animals, and the like), mobile devices (e.g., tablets, notebooks, laptops, personal computers, portable televisions, radios, and the like), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded therein and/or coupled thereto, radios, and the like).
Computer device 905 can be communicatively coupled (e.g., via I/O interface 925) to external storage 945 and network 950 for communicating with any number of networked components, devices, and systems, including one or more computer devices of the same or different configuration. Computer device 905 or any connected computer device can be functioning as, providing services of, or referred to as a server, client, thin server, general machine, special-purpose machine, or another label.
I/O interface 925 can include, but is not limited to, wired and/or wireless interfaces using any communication or I/O protocols or standards (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, a cellular network protocol, and the like) for communicating information to and/or from at least all the connected components, devices, and network in computing environment 900. Network 950 can be any network or combination of networks (e.g., the Internet, local area network, wide area network, a telephonic network, a cellular network, satellite network, and the like).
Computer device 905 can use and/or communicate using computer-usable or computer-readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metal cables, fiber optics), signals, carrier waves, and the like. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.
Computer device 905 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. Computer-executable instructions can be retrieved from transitory media, and stored on and retrieved from non-transitory media. The executable instructions can originate from one or more of any programming, scripting, and machine languages (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).
Processor(s) 910 can execute under any operating system (OS) (not shown), in a native or virtual environment. One or more applications can be deployed that include logic unit 960, application programming interface (API) unit 965, input unit 970, output unit 975, and inter-unit communication mechanism 995 for the different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements can be varied in design, function, configuration, or implementation and are not limited to the descriptions provided.
In some example implementations, when information or an execution instruction is received by API unit 965, it may be communicated to one or more other units (e.g., logic unit 960, input unit 970, output unit 975). In some instances, logic unit 960 may be configured to control the information flow among the units and direct the services provided by API unit 965, input unit 970, output unit 975, in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 960 alone or in conjunction with API unit 965. The input unit 970 may be configured to obtain input for the calculations described in the example implementations, and the output unit 975 may be configured to provide output based on the calculations described in example implementations.
In an example implementation, processor(s) 910 can be configured to execute the flow as illustrated in
Through execution of the flow steps a) to e) in the order described and as similarly illustrated in
In an example implementation, the fitness function is configured to evaluate accuracy and uncertainty of the predictions of the training model through providing a probability of predicting a given output from an input, the deep learning architecture parameters, and a dropout rate, as described with respect to 107 of
such that P(y|x, w, d) is the likelihood of predicting y given input data x, network parameters w and dropout rate d, wherein x=input data; w=network parameters (e.g., deep learning architecture parameters); d=dropout rate; μ=mean of the prediction sampled using MC dropout mechanism; and σ=deviation of the prediction sampled using the MC dropout mechanism. The fitness function is indicative of the model being optimized for when the deep learning architecture parameters converge within a threshold; wherein the fitness function is evaluated based on a mean of the predictions and the wherein the uncertainties calculated from comparing the mean of the predictions to the validation set of data.
As illustrated in
Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a series of defined steps leading to a desired end state or result. In example implementations, the steps carried out require physical manipulations of tangible quantities for achieving a tangible result.
Unless specifically stated otherwise, as apparent from the discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing,” “computing,” “calculating,” “determining,” “displaying,” or the like, can include the actions and processes of a computer system or other information processing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other information storage, transmission or display devices.
Example implementations may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored in a computer readable medium, such as a computer-readable storage medium or a computer-readable signal medium. A computer-readable storage medium may involve tangible mediums such as, but not limited to optical disks, magnetic disks, read-only memories, random access memories, solid state devices and drives, or any other types of tangible or non-transitory media suitable for storing electronic information. A computer readable signal medium may include mediums such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Computer programs can involve pure software implementations that involve instructions that perform the operations of the desired implementation.
Various general-purpose systems may be used with programs and modules in accordance with the examples herein, or it may prove convenient to construct a more specialized apparatus to perform desired method steps. In addition, the example implementations are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the example implementations as described herein. The instructions of the programming language(s) may be executed by one or more processing devices, e.g., central processing units (CPUs), graphics processing units (GPUs), processors, or controllers.
As is known in the art, the operations described above can be performed by hardware, software, or some combination of software and hardware. Various aspects of the example implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software), which if executed by a processor, would cause the processor to perform a method to carry out implementations of the present application. Further, some example implementations of the present application may be performed solely in hardware, whereas other example implementations may be performed solely in software. Moreover, the various functions described can be performed in a single unit, or can be spread across a number of components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions can be stored on the medium in a compressed and/or encrypted format.
Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and/or components of the described example implementations may be used singly or in any combination. It is intended that the specification and example implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.