Patent Application
20230153699
Energetic particles (electrons and/or protons) magnetically trapped inside the Earth’s Van Allen radiation belts present a serious concern for space operations. These particles are usually present in a two-belt distribution, an inner belt in the region with equatorial distances (i.e., L-shells or simply L) within ~2-3 Earth radii and an outer belt with ~3 < L < 8 separated by the slot region in between. Deep injections of Megaelectron-volt (MeV) electrons into the outer belt, often occurred during geomagnetic storms, are called MeV electron events in which electron intensities may vary up to several orders of magnitude. These energetic electrons pose a major space radiation risk, not only in term of the ionizing dose, but also due to the deep dielectric charging and discharging phenomena, for satellites operating in orbits with high apogees including geosynchronous orbit (GEO), geosynchronous-transfer-orbit (GTO), medium- and high-earth-orbits (MEO and HEOs). When space systems are irradiated, some of the electrons are energetic enough to penetrate through satellite surfaces (e.g., ranges of 2.0 and 3.0 MeV electrons inside Aluminum are 4.53 and 6.92 mm, respectively), stop, and bury themselves inside the dielectric materials of electronic parts on board. During major MeV electron events when electron intensities across the outer belt are greatly enhanced with sustained high levels, these buried electrons accumulate faster than they can dissipate, and thus build up high electric fields (a process called “charging”, with the potential differences reaching as high as multiple kilovolts), until eventually sudden intense breakdowns occur which result discharge arcs that may cause catastrophic failure to individual electronics or to the satellite as a whole.
Accordingly, forecasting MeV electron events have become more important in recent years given the reliance on various satellite technologies and space infrastructures. One conventional method in forecasting MeV electron events includes utilizing empirical models based on past measurements. Unfortunately the empirical model results in very crude results. Another conventional method is to use first-principles models that are driven by the dynamic global distributions of various input parameters, e.g., electrons with energies less than 1 MeV and related wave intensities, to forecast MeV electron events; however, due to the lack of adequate measurement coverage, these theoretical models still fall short of making reliable prediction with high time resolutions. Some conventional systems have utilized machine learning (ML) to forecast MeV electron events in GEO because continuous in situ measurement of MeV electrons are available along with a large volume of past data. However, intensification events of ultra-relativistic electrons (with energies greater than or equal to 2 MeV) not only occur in GEO but across the whole outer belt region, in particular, in the heart region of four Earth radii where navigation satellites reside. In this vast space, unfortunately, in-situ measurements of ultra-relativistic electrons are not routinely available, and thus the conventional ML model cannot be simply applied to make forecasts.
Accordingly, a need has arisen to generate a model that reliably forecasts the intensifications of ultra-relativistic electrons (hereinafter called ultra-relativistic electron events or MeV electron events). Our solution is an ML model higher electron events based on lower-energy electrons and/or solar wind data. In some embodiments, a plurality of ML models is generated based on various inputs, e.g., solar wind data, measurements of low-energy electrons (less than 2 MeV) in low-Earth-orbits (LEO), etc., window size for the input selection, the models are trained and validated by observed (e.g., measured) ultra-relativistic electron (greater than or equal to 2 MeV) distributions in the past, and the known outputs are nowcasts and forecasts of ultra-relativistic electron distributions in the future. A subset of ML models from the plurality of ML models can be selected to form an ensemble group. An ensemble ML model can be generated from the group to make ensemble forecasts with error bars, e.g., statistical mean of each ML model of the ensemble group at each L-shell.
A system may include one or more transmitters configured to transmit measured low-energy electrons that are less than or equal to 1.5 MeV, wherein the measured low-energy electrons include at least a first and a second set of data. The system may further include a receiver configured to receive the measured low-energy electrons from the one or more transmitters, and wherein the receiver is further configured to receive solar wind data as a third set of data, and wherein the receiver is further configured to receive data associated with observation of higher-electron events with greater than or equal to 2 MeV The system also includes a processor configured to generate a plurality of machine learning (ML) models based on (1) selection of one or more ML algorithms (e.g., linear regression, feedforward neural networks (FNN), long-short-term memory (LSTM), and convolutional neural network (CNN)), (2) selection of input data from at least two or more of the first set of data, the second set of data, and the third set of data, (3) the data associated with the observation of higher-electron events with greater than or equal to 2 MeV as its output, and (4) selection of a number of window size. It is appreciated that each generated ML model of the plurality of ML models is different from one another generated ML model of the plurality of ML models based on the selection of ML algorithm, the selection of input data, and the selection of window size. Moreover, it is appreciated that the processor is further configured to select a subset of ML models from the plurality of ML models to generate an ensemble ML model.
In some embodiments, the processor is further configured to apply new data input to the generated ensemble ML model to predict higher electron events of greater than or equal to 2 MeV. According to some embodiments, the measured low-energy electrons includes a third set of data, and wherein the first set of data includes electron count rates of greater than 100 keV, the second set of data includes electron count rates of greater than 300 keV, and the third set of data includes electron count rates of greater than 1000 keV. In some embodiments, the receiver is further configured to receive a derivative of the measured low-energy electrons, and wherein the processor is further configured to make a selection of input data including the derivative of the measured low-energy electrons. It is appreciated that the solar wind data includes data associated with one or more of solar wind velocity and solar wind density.
It is appreciated that in some embodiments the plurality of ML models is generated based on a combination of at least two or more different ML algorithms. In some embodiments, the plurality of ML models is generated based on a combination of at least two or more input data. It is further appreciated that the plurality of ML models may be generated based on a combination of at least two or more window sizes.
In some embodiments the system includes a receiver configured to receive a measured low-energy electrons from one or more transmitters that are less than or equal to 1.5 MeV, wherein the measured low-energy electrons include at least a first and a second set of data, and wherein the receiver is further configured to receive solar wind data as a third set of data, and wherein the receiver is further configured to receive data associated with observation of higher-electron events with greater than or equal to 2 MeV. The system may further include a processor configured to generate a plurality of machine learning (ML) models based on: (1) selection of one or more ML algorithms (e.g., linear regression, feedforward neural networks (FNN), long-short-term memory (LSTM), and convolutional neural network (CNN)), (2) selection of input data from at least two or more of the first set of data, the second set of data, and the third set of data, (3) the data associated with the observation of higher-electron events with greater than or equal to 2 MeV as its output, and (4) selection of a number of window size. It is appreciated that each generated ML model of the plurality of ML models is different from one another generated ML model of the plurality of ML models based on the selection of ML algorithm, the selection of input data, and the selection of window size. Moreover, it is appreciated that the processor is further configured to select a subset of ML models from the plurality of ML models to generate an ensemble ML model.
In some embodiments, the processor is further configured to apply new data input to the generated ensemble ML model to predict higher electron events of greater than or equal to 2 MeV. It is appreciated that in some embodiments the measured low-energy electrons include a third set of data, and wherein the first set of data includes electron count rates of greater than 100 keV, the second set of data includes electron count rates of greater than 300 keV, and the third set of data includes electron count rates of greater than 1000 keV. According to some embodiments, the receiver is further configured to receive a derivative of the measured low-energy electrons, and wherein the processor is further configured to make a selection of input data including the derivative of the measured low-energy electrons. It is appreciated that in some embodiments the solar wind data includes data associated with one or more of solar wind velocity and solar wind density.
In some embodiments, a method includes receiving a plurality of data sets, wherein the plurality of data sets includes measured low-energy electrons that are less than or equal to 1.5 MeV, and wherein the plurality of data sets further includes data associated with solar wind, e.g., solar wind velocity and solar wind density. The method further includes receiving measured data associated with higher electron events of greater than or equal to 2 MeV. In some embodiments, the method also includes in response to a selection of at least two data sets from the plurality of data sets, and further in response to a selection of one or more machine learning (ML) algorithms from a plurality of ML algorithms (e.g., linear regression, feedforward neural networks (FNN), long-short-term memory (LSTM), and convolutional neural network (CNN)), and further in response to a selection of a number of window size, generating a plurality of ML models based on the selections as an input and the measured data associated with higher electron events of greater than or equal to 2 MeV as its output.
The method in some embodiments further includes forming an ensemble of ML models in response to a selection of a subset of ML models from the plurality of ML models. The method further includes generating an ensemble ML model based on the ensemble of ML models, wherein the ensemble ML model is a statistical means of the subset of ML models at each measured L-shell, wherein L-shell is the equatorial distance to earth.
In some embodiments, the method further includes generating an ensemble ML model based on a selection of a subset of ML models of the plurality of ML models. According to some embodiments, the method includes receiving new data sets that include a selection from the measured low-energy electrons that are less than or equal to 1.5 MeV and the data associated with solar wind; and applying the new data sets as input to the ensemble ML model to predict higher electron events of greater than or equal to 2 MeV.
In some embodiments, the measured low-energy electrons include data sets for electron count rates of greater than 100 keV, electron count rates of greater than 300 keV, and electron count rates of greater than 1000 keV.
These and other features and aspects of the concepts described herein may be better understood with reference to the following drawings, description, and appended claims.
Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures. It is noted that, in accordance with the standard practice in the industry, various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.
The following disclosure provides many different embodiments, or examples, for implementing different features of the subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. In addition, the present disclosure may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed.
Before various embodiments are described in greater detail, it should be understood that the embodiments are not limiting, as elements in such embodiments may vary. It should likewise be understood that a particular embodiment described and/or illustrated herein has elements which may be readily separated from the particular embodiment and optionally combined with any of several other embodiments or substituted for elements in any of several other embodiments described herein. It should also be understood that the terminology used herein is for the purpose of describing the certain concepts, and the terminology is not intended to be limiting. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood in the art to which the embodiments pertain.
It is appreciated that the intensifications of electrons traveling at greater than 0.98 of the speed of light--also known as ultra-relativistic electrons with energies greater than or equal to 2 MeV--cause higher electron events (i.e., energies greater than or equal to 2 MeV). The embodiments utilize one or more measurements of precipitating low-energy electrons (i.e., energies such as 100 keV, 300 keV, 1 MeV, etc.) from satellites (for example operating in low earth orbit (LEO)), ultra-relativistic electron measurements in geosynchronous orbit (GEO), etc., and/or other measurements, e.g., solar wind velocity, solar wind densities, etc., (as measured by solar wind monitors at Lagrangian 1 point of Sun-Earth, in one nonlimiting example) as inputs to one or more ML algorithms, e.g., linear regression, feedforward neural networks (FNN), long-short-term memory (LSTM), convolutional neural network (CNN), etc. It is appreciated that in some embodiments, a transmitter may transmit the measurements associated with precipitating low-energy electrons, solar wind associated information, etc., to a processor for further processing. The one or more ML algorithms use the input data as training data to generate a model in order to forecast higher MeV electron events (i.e., ultra-relativistic electrons with greater than or equal to 2 MeV). In other words, a new approach explores the use of precipitating low-energy electrons observed in LEO, MeV electrons in GEO, etc., and/or solar wind velocity and density at Lagrangian 1, to forecast higher electron events, i.e. greater than or equal to 2 MeV. The low-energy electrons may be defines as substantially less than 2 MeV, e.g., approximately 1 MeV, 100 keV, 300 keV, etc. Throughout this application low-energy electrons are considered to be at or below ~1.5 MeV.
In some embodiments, a number of different (or the same) ML algorithms with different inputs (or combination thereof) and a number of different window sizes (i.e., a number of periods such as four 5 hour periods as an example) are used to generate different ML models. Ensemble methodology (i.e., ensemble of various models) may be used to generate a broad sample of possible future states of a dynamic system based on slightly different initial conditions (e.g., input, window size, etc.) and/or different forecast models (e.g., different ML algorithms) to forecast higher electron events. The output from the ML algorithms with different data inputs, size, etc., can be combined and analyzed to identify trends and uncertainty ranges.
Accordingly, an ensemble of ML models generated can be used in order to generate a model that reliably predicts (e.g., 1-day forecast, 2-day forecast, etc.) ultra-relativistic electrons (i.e., higher electron events) based on lower electron events measured by a combination of satellites operating in LEO, GEO, and/or solar wind velocity at Lagrangian 1 point, etc. It is appreciated that while the ML model generated based on the lower electron events and/or solar wind velocity successfully forecast higher electron events, the ensemble methodology further improves the predictability of the higher electron events across the outer radiation belt.
It is appreciated that generation of the forecasting model, as described above, can be updated over time using additional data input. For example, the models may be updated based on receiving data, e.g., electron fluxes at given electron energies, etc., received from global positioning (GPS) satellites. In other words, the ML models can be changed and updated over time and on-the-fly.
During a training/on-the-fly-learning phase 100A, a processor 110, e.g., a central processing unit (CPU), an application specific integrated circuit (ASIC), a graphics pipeline unit (GPU), an ML specific processor, etc., may receive training data as its input. For example, the processor 110 may receive low-energy electron data 102, e.g., 100 keV, 300 keV, 1 MeV, etc. It is appreciated that low-energy electron is generally referred to as an electron with less than 2 MeV. Moreover, the processor 110 may receive solar wind data 104, e.g., solar wind velocity, solar wind density, etc. In some embodiments, the processor 110 also receives the ML algorithm 106, e.g., linear regression, FNN, LSTM, CNN, etc. The input data (i.e., solar wind data 104 and/or low-energy electron data 102) is used during the training stage of the specific ML algorithm 106, optimized by comparing to the target data, to generate an ML model 112. It is appreciated that the embodiments are described with respect to receiving both the low-energy electron data 102 and the solar wind data 104 for illustrative purposes, but should not be construed as limiting the scope of the embodiments. For example, in some embodiments only the solar wind data 104 may be used, or the low-energy electron data 102 may be used, or alternatively other types of data may be used, e.g., data from a GPS satellite.
It is appreciated that in some embodiments, the low-energy electron data 102 may be measured by one NOAA Polar Operational Environmental Satellite (POES) NOAA-15. In some embodiments, the solar wind data 104 may be an upstream solar wind conditions, e.g., solar wind velocity (SW), solar wind densities (SWD), temperature, etc., over the same time interval as the low-energy electron data 102. It is appreciated that the solar wind data 104 may be standardized by subtracting their mean values and dividing the results with the standard deviations.
In the described embodiments, three different channels, e.g., E2, E3, and P6, for low-energy electron measurements by POES are used. It is appreciated that the E2 channel measures electron counts having greater than 100 keV, E3 channel measures electron counts having greater than 300 keV, and P6 channel measures electron counts having greater than 1 MeV. It is appreciated that the channels and the electron energy levels described are for illustrative purposes and should not be construed as limiting the scope of the embodiments. For example, any number of channels, e.g., 2 channels or greater than 3 channels, with different electron energy levels, e.g., 600 keV, 500 keV, etc., may be used. It is appreciated that electron intensities refers to electron flux value (i.e., number of particles going through a given area within a given time interval).
It is appreciated that the low-energy electron data 102 and/or the solar wind data 104 are binned in certain time increments, e.g., 5 hour increments (5 hr time bin), at various equatorial distances (i.e., L-shells).
The ML algorithm 106 may be based on linear regression models in order to optimize linear relationship between input parameters and targets. In contrast, ML algorithm 106 may be based on FNN that uses layers of neurons to process inputs with linear transformations followed by nonlinear activation functions to optimize outputs. ML algorithm 106 may be based on LSTM networks that includes connected memory cells that learn the sequential and temporal dynamics from the previous time steps, e.g., previous 10 hours as an example, to make predictions. The ML algorithm 106 may be based on CNN that relies on a convolution kernel to filter the data and explore the local patterns inside.
For model development, the input data may be split into two stages, e.g., training and post training. At the end of the training stage, an ML model is generated that can be used to forecast higher electron events, e.g., during testing and validation stage. In the described embodiments, the training stage is approximately 65% (~835 days) while validation is approximately 14% (175 days) testing is approximately 21% (267 days) for the collected data.
It is appreciated that the ML model(s) is trained for each individual L-shell between 2.8 and 6 as well as at GEO (6.6) in the outer belt region, with the optimization goal of reducing the root-mean-square error between the target valuesy (electron fluxes in logarithm) and predicted valuesƒ.
It is appreciated that once the ML model(s) 112 is generated it may be fed into a processor 120 as one of its input, during the validation/testing/operation 100B stage. The processor 120 may also receive the low-energy electron data 102 and/or solar wind data 104, as described above. It is appreciated that the processor 120 may be similar to processor 110 and may in fact be the same. Different parameter combinations and temporal window sizes are tested for model inputs. The processor 120 may process the input data based on the ML model 112 and outputs a result to forecast high electron events forecast 124.
It is appreciated that the model performance may be gauged by Performance Efficiency (PE), which quantifies the accuracy of predictions by comparing to the variance of the target. Naming y and ƒ both with size M, PE is defined as
where
Referring now to
Panel A illustrates ultra-relativistic electron flux distributions from the in-situ observations made by the Relativistic Electron-proton Telescope aboard one Van Allen Probe spacecraft (RBSP-a) at L ≤ 6, and by the Energy Spectrometer for Particles (ESP) instrument carried by one Los Alamos National Laboratory (LANL) GEO satellite LANL-01A at L = 6.6. The ESP instrument measures the fluxes of relativistic electrons between 0.7 and 10 MeV. As illustrated integral fluxes of greater than or equal to 2 MeV electrons are the target data set that is a function of L-shell over a 1289-day interval. These greater than or equal to 2 MeV electron data are used for model training, validation, and test.
Panels B, C, and D illustrate the count rates (logarithmically) of precipitating electrons (low-energy electrons) measured by NOAA-15 in LEO, for E2, E3, and P6 channels, respectively. Panel E illustrates the solar wind speeds measured upstream of the magnetosphere from the OMNI data set while panel F illustrates the solar wind densities.
As discussed above, the ML algorithm 106, the data inputs (i.e. low-energy electron data 102 and/or solar wind data 104), and the size of the input may vary, e.g., number of certain time of periods (as discussed above). For example, linear regression as an ML algorithm 106 may be selected. For the selected ML algorithm 106 the data input and the size may be changed to generate a different model for each selection. For example, the data input may be selected and be a combination of E2, E3, E6, SW, SWD, dE2 (temporal derivative of E2), E246 (E2 measured at L-shell 4.6 which may be desirable due to high cross L-sell correlation and positive effects on forecast), etc., and the window size may be selected from 4 (i.e., 20 hours in this example) or 16 (i.e., 80 hours in this example). A number of selected combinations may generate different models as illustrated by index 1-8 of Table 1 below. It is appreciated that a similar process may be performed for other ML algorithms 106, e.g., FNN (in this example composed of two hidden layers as the first one has 64 neurons and the second has 32 neurons and the neurons use Exponential Linear Unit (ELU) as the activation function), LSTM (in this examples includes 128 memory cells), and CNN (in this example contains two convolutional layers, the first contains 64 kernels and the second contains 32 kernels and the kernels use Rectified Linear Unit (relu) as an activation function), as illustrated by index 9-32 of Table 1 below. It is appreciated that the particular data inputs, window size, and ML algorithm described are for illustrative purposes and should not be construed as limiting the scope of the embodiments.
Each generated model, after the training stage, may be tested and validated. Each model may be evaluated based on the performance efficiency, as described above, for illustrative purposes. It is appreciated that other methodology used to evaluate the performance may also be implemented, and the particular performance efficiency that has been described herein for illustrative purposes should not be construed as limiting the scope of the embodiments. The performance efficiency during the training stage, the validation stage, and the testing stage may be determined, as shown in columns 5-7 of Table 1. In some embodiments, the PE of the valuation and testing stage may be determine as well as the overall PE that includes the training stage and post training stage may be illustrated as shown in columns 8-9 of Table 1. In some embodiments, the PE for the validation and testing stage at GEO is also determined, as shown in column 10 of Table 1. It is appreciated that Table 1 illustrates the input data, model and PE for forecasting higher electron events (e.g., greater than 2 MeV) for a 1-day (25 hr) period, for illustration purposes.
It is appreciated that an ensemble methodology may be used to generate a model that almost outperforms each model individually and at various L-shells. In this illustrative example, the top generated ML model performer, after the training stage, from each category of ML algorithms are selected (i.e., top performance as determined by PE of validation and testing in column 8) and underlined and in bold font for convenience. All the selected models may be used together in an ensemble model, as illustrated by index 33 that shows an ensemble group comprising models 8, 13, 22, and 29 for illustrative purposes. It is noteworthy that the top performers for each category may be different from the top performer of GEO, e.g., the top performer based on PE during validation and testing of FNN is different from its top performer based on PE of GEO during validation and testing. The ensemble model in some embodiments may be a mean of the selected models, e.g., at L=3.2 the ensemble model may be the mean of the predicted value for models 8, 13, 22, and 29. The ensemble model is determined at every L-shell, i.e., between 2.8 to GEO in this example. It is appreciated that the use of median is for illustrative purposes and should not be construed as limiting the scope of the embodiments. For example, an average may be used or alternatively other statistical operations may be used. It is further appreciated that the ensemble model can not only be used to forecast higher electron events but it can also be used to predict the uncertainty in the model, i.e., spread.
It is appreciated that a similar approach may be used to generate a plurality of ML models and to apply an ensemble methodology to generate an ML model that reliably forecasts higher electron events (i.e., greater than 2 MeV) for a 2-day (50 hour), as illustrated in Table 2 below. As illustrated, the ensemble may include different models based on the performance. For example, in Table 1 the ensemble includes models 8, 13, 22, and 29 while in Table 2 the ensemble includes the models 6, 13, 22, and 29 because one is forecasting for 1-day and the other is forecasting for 2-days.
It is appreciated that a similar approach may be used to generate a plurality of ML models and to apply an ensemble methodology to generate an ML model that reliably forecasts higher electron events (i.e., equal to 2 MeV) for a 1-day (25 hour) and 2-day (50 hour), as illustrated in Tables 3 and 4 below. As illustrated, the ensemble may include different models based on the performance. For example, in Table 3 the ensemble includes models 8, 13, 24, and 29 while in Table 4 the ensemble includes the models 8, 9, 22, and 29 because one is forecasting for 1-day and the other is forecasting for 2-days.
Referring now to
The comparison of the performance of the linear regression models and the LSTM models illustrates the impact of feature selection, e.g., input data such as low-energy electron (E2, E3, P6, dE2, etc.), solar wind data (solar wind velocity, solar wind density, etc.), window size, etc., as a function of L-shells. In Panel A, the general trend can be observed for linear models that PE increases with the increasing number of input parameters and window sizes. All curves have similar shapes with the highest PE at L~4.0 (and decreasing in both directions, while PE values at GEO go above 0.6). Note that each PE curve has data points located at L-shells from 3.0 to 6.0 with an increment of 0.1 as well as at GEO. The high PE values at GEO can be explained by the inclusion of >2 MeV electron fluxes in-situ measured by LANL-01A satellite.
For illustration purposes three of the models are highlighted. In panel A, Models 6 and 8 have different input parameters but the same window size, while models 7 and 8 have the same input parameters but different window sizes (see Table 1). As illustrated model 8 has the highest PE with SWD included as one of its data inputs. In Panel B, LSTM models 23, 24, and 25 are highlighted for illustration purposes. Models 23-25 have different PE curves with large variations. Several LSTM models show a local minimum in PE with L-shell at ~4 and a plateau at L between 3.1-3.8. In addition, the inclusion of SWD to models 23 and 24 has decreased their respective PE at L ≤ 6 in comparison to those of model 22 (also see Table 1). PE values can drop below zero at small L < 3.0, particularly for the linear models, mainly due to the lack of training events. Therefore, hereinafter we confine our discussions on PE only for L ≥ 3.0.
In order to determine performance, mean PE values of Table 1 may be determined. The mean values are averaged over all L-shells except for GEO for individual models. Based on the mean out-of-sample (post training stage) PE values, the models may be ranked from high to low. For instance, in the linear category, model 8 is the top performer with the highest mean PE of 0.523, followed by model 6 with a PE value of 0.509. For the top performer model 8, its out-of-sample PE at GEO is 0.629, also the highest in the category and thus in bold and underscored. Similarly, the top and second performers in other categories are picked out with their mean PE in bold font and underscored. In Table 1, mean PE values of the four top (second) performers are 0.523 (0.509) for linear, 0.553 (0.488) for FNN, 0.537 (0.521) for LSTM, and 0.479 (0.477) for CNN, while their PE values at GEO are 0.629 (0.625), 0.630 (0.603), 0.600 (0.581), and 0.598 (0.566), which are not necessarily the highest of each category. As illustrated, among the four top performers, only the linear model 8 has SWD in model inputs, while at GEO three out of the four models with the highest PE, i.e., models 8, 15 and 23, have SWD included.
Similarly, in Table 2 for 2-day forecasts, mean out-of-sample PE values for the four top (second) performers are 0.438 (0.431), 0.460 (0.416), 0.456 (0.451), and 0.423 (0.408), while their PE at GEO are 0.428 (0.431), 0.423 (0.419), 0.390 (0.384), and 0.402 (0.345) which are often not the highest in the category. For 2-day forecasts, SWD are not needed for the top four performers, while at GEO the only exception is the linear model 8. Therefore, SWD is not necessary for model input except for 1-day linear forecasts at GEO.
Also, in both Tables 1 and 2, top FNN and LSTM models marginally outperform top linear models, suggesting the significance of nonlinear component for greater than 2 MeV electrons. Additionally, PE values at GEO are ~0.1 higher than the mean PE at L≤6 for 1-day forecasts, while for 2-day forecasts PE values are slightly lower at GEO.
Referring now to
PE curves for the top two performers in each category for one- and two-day forecasts are further compared as a function of L-shell. It is appreciated that comparing panels A and B illustrates that no one individual model outperforms others over all L-shells. For example, linear models (i.e., the solid gray curves) have higher PE at L-shells above ~3.8, while the top FNN (red) and LSTM (brown) models perform better at small L-shells than the quickly degrading linear ones. Moreover, the PE curves for the top linear model from a similar method for 1 MeV electrons are plotted in long-dashed gray for comparison. It can be seen that for this new model PE curves for the linear ones in solid gray and magenta have higher PE at L-shells >4.5 for 1-day (>4.0 for 2-day) but lower PE at smaller L-shells than the dashed gray curve.
Referring now to
Panels B-E illustrate that the models, as selected, that perform similar to those observed in target data. Over the entire interval, most MeV electron events are captured well in terms of both intensities and L-shell ranges. Exceptions include the significant electron dropouts, e.g., on days ~ 1080 at L > 5, and the deep electron injections into small L-shells below 3.0.
Referring now to
It is appreciated that in the validation and test periods, the lack of dark vertical strips suggests the onsets of higher electron events (i.e., greater than 2 MeV) are well predicted, while the clear vertical strips reflect the predicted high fluxes during dropouts, which is acceptable since this model aims to predict the enhancements of energetic electrons. The dark regions at small L-shells ~ 2.8 and 2.9 during the validation and test periods, particularly in Panel A, indicate models’ lack of high performance in those area. It is appreciated that lack of performance may be due to presence of low L-shells training data that is dominated by background and the ML algorithms that can learn only from the single major event starting on day ~758, while there are up to three events during the validation and test periods.
Referring now to
Referring now to
Panels B-E illustrate that the models, as selected, perform similar to those observed in target data. Over the entire interval, most MeV electron events are captured well in terms of both intensities and L-shell ranges. It is appreciated that the features, as illustrated by
Referring now to
As shown in the last row of Table 1, ensemble members include linear regression model 8, FNN model 13, LSTM model 22 and CNN model 29, which are the top four performing models in each of the four categories. At each time step the ensemble prediction of electron fluxes at one L-shell is the median of all four member model outputs, and standard deviation of the outputs is the measure of uncertainty. It is appreciated that the ensemble model may be used not only to reliably predict occurrences of higher electron events, but also to predict the uncertainty or spread in the model.
Referring now to
Referring now to
As illustrated, there are noticeable improvements, including the better predictions of low fluxes at L-shells ~3.5, e.g., centered on day 552 during the training in Panel B, and the deep injections to low L-shells during the validation and test periods when compared to the linear model in
Referring now to
As illustrated, in one nonlimiting example, the PE may be used to quantify model performance, comparing the ensemble PE curves to those of group members as a function of L-shell. It is appreciated that the ensemble PE curve (in thick black) that illustrates the ensemble model is almost always to the rightmost for all L-shells, including at GEO, when compared to PE curves of its four member models individually. In other words, the ensemble model outperforms each individual members individually. Moreover, in comparison to the PE curves in long-dashed gray from the linear model of similar methods for 1 MeV electrons, the ensemble forecasts presented herein have at least comparable performance in Panel A for 1-day or have even better performance as shown in Panel B for 2-day forecast, in particular at medium or high L-shells.
As illustrated in Tables 1 and 2, the ensemble models have a mean PE value of 0.612 for 1-day and 0.521 for 2-day at L ≤ 6, and 0.677 and 0.572 at GEO. It is appreciated that these PE values are significantly higher than those from individual top performer models, thereby displaying the performance gain and advantage of using an ensemble model over any individual model. It is appreciated that the actual measured higher electron events, e.g., using the operational Relativistic Electron Forecast Model (REFM) at NOAA, has PE values of 0.72 and 0.49 at GEO for 1-day and 2-day predictions for daily averaged fluence of great than 2 MeV electrons, whereas the discussed ensemble model has PE values of ~0.68 and ~0.57 at GEO for 1-day and 2-day forecasts with 5 hour time resolution. Additionally, besides GEO, the ensemble model also has similar predictive performance across L-shells between 3 and 6 in the heart of the outer belt.
It is appreciated that the number of members within the ensemble group, as described, is for illustrative purposes and should not be construed as limiting the scope of the embodiments. For example, in some examples more than 32 ML models may be generated and the ensemble group may include eight model members, as an example. It is further appreciated that a hybrid model may be employed where the best performing models are used at each L-shell instead of selecting the same model across all L-shells. For example, this hybrid model may combine nonlinear models at small L-shells (e.g., L<3.5) and include more linear models at large L-shells depending on their ranks in PE.
Referring now to
It is appreciated that the effects of model input parameters and window sizes are examined. The mean PE values for individual models are presented in Tables 3 and 4 for 1-day and 2-day forecasts, respectively. In Table 3, mean PE values of the four top (second) performers are 0.600 (0.590) for linear regression, 0.549 (0.548) for FNN, 0.549 (0.533) for LSTM, and 0.525 (0.518) for CNN, while their PE values at GEO are 0.566 (0.568), 0.461 (0.535), 0.509 (0.539), and 0.459 (0.437), which are lower than the highest for each category. In Table 4, the mean PE values of the four top (second) performers are 0.512 (0.506) for linear regression, 0.474 (0.461) for FNN, 0.438 (0.435) for LSTM, and 0.439 (0.425) for CNN, while their PE values at GEO are 0.234 (0.244), 0.186 (0.105), 0.138 (0.106), and 0.102 (0.125), which are often far away from the highest value for each category. It is noteworthy that in this nonlimiting example, the top (and second) linear regression models have higher mean PE than all the remaining top performers for both 1-day and 2-day forecasts. Based on the rank of mean PE values, in the last row of Tables 3-4, ensemble forecasting models are constructed including the top performers from each of the four categories.
Referring now to
It is appreciated that the model performance may be quantified by comparing the ensemble PE curves to those of group members as a function of L-shell. Similar to
Referring now to
Referring now to
Accordingly, using electron data, e.g., from NASA’s Van Allen Probes mission, various ML models have been trained (i.e., supervised), evaluated and tested for higher electron events of greater than or equal to 2 MeV electron fluxes. After evaluating the performance of these models, ensemble forecasting has performed demonstrably better than any model individually. The comparison of the performance (after the training stage) to that of the observed data demonstrates that the approach to generate the models and to leverage an ensemble model can reliably predict higher electron events, thereby eliminating the need for in-situ electron measurements.
It is appreciated that the model may be updated and evolved over time as more data becomes available. Data that becomes available may be from the same sources, as described above, or they may be from one or more different sources, e.g., GPS satellites, that can provide additional information, e.g., electron fluxes, given energy of particles, etc.
Training of the neural network 300 using one or more training input matrices, a weight matrix, and one or more known outputs is initiated by one or more computers. In an embodiment, a server may run known input data through a deep neural network in an attempt to compute a particular known output. For example, a server uses a first training input matrix and a default weight matrix to compute an output. If the output of the deep neural network does not match the corresponding known output of the first training input matrix, the server adjusts the weight matrix, such as by using stochastic gradient descent, to slowly adjust the weight matrix over time. The server computer then re-computes another output from the deep neural network with the input training matrix and the adjusted weight matrix. This process continues until the computer output matches the corresponding known output. The server computer then repeats this process for each training input dataset until a fully trained model is generated.
In the example of
In an embodiment, low-energy electron data 302 is used as one type of input data to train the model, which is described above. In some embodiments, solar wind data 304 are also used as another type of input data to train the model, as described above. Moreover, in some embodiments, input data 306 are also used as another type of input data to train the model, as described above.
In the embodiment of
Once the neural network 300 of
Referring now to
It is appreciated that a subset of the plurality of ML models is formed (selected) to form ensemble members. The ensemble group may be used to form an ensemble ML model, as described above. It is appreciated that once the ensemble ML model is generated, new input data, e.g., low-energy electrons, solar wind velocity, solar wind density, etc., can be used as an input in order for the ensemble ML model to predict a higher electron events.
According to some examples, computer system 1100 performs specific operations in which processor 1104 executes one or more sequences of one or more instructions stored in system memory 1106. Such instructions can be read into system memory 1106 from another computer readable medium, such as static storage device 1108 or disk drive 1110. In some examples, hardwired circuitry can be used in place of or in combination with software instructions for implementation. In the example shown, system memory 1106 includes modules of executable instructions for implementing an operating system (“OS”) 1132, an application 1136 (e.g., a host, server, web services-based, distributed (i.e., enterprise) application programming interface (“API”), program, procedure or others). Further, application 1136 includes a module of executable instructions associated with machine learning algorithm module 1141 to generate one or more ML models based on the observed data 1142 (as described above), based on the solar wind data 1140 (as described above), and further based on low-energy electron data 1138 (described above). The generated ML models may be used by the ML module 1141 and/or the processor 1104 to create an ensemble ML model that outperforms each member of the ensemble individually. It is appreciated that the generation of the ML model(s) and the ensemble transforms the system as shown to a special purpose computer that reliably forecasts higher electron events.
The term “computer readable medium” refers, at least in one embodiment, to any medium that participates in providing instructions to processor 1104 for execution. Such a medium can take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as disk drive 1110. Volatile media includes dynamic memory, such as system memory 1106. Transmission media includes coaxial cables, copper wire, and fiber optics, including wires that comprise bus 1102. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
Common forms of computer readable media include, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, electromagnetic waveforms, or any other medium from which a computer can read.
In some examples, execution of the sequences of instructions can be performed by a single computer system 1100. According to some examples, two or more computer systems 1100 coupled by communication link 1120 (e.g., LAN, PSTN, or wireless network) can perform the sequence of instructions in coordination with one another. Computer system 1100 can transmit and receive messages, data, and instructions, including program code (i.e., application code) through communication link 1120 and communication interface 1112. Received program code can be executed by processor 1104 as it is received, and/or stored in disk drive 1110, or other non-volatile storage for later execution. In one embodiment, system 1100 is implemented as a handheld device. But in other embodiments, system 1100 can be implemented as a personal computer (i.e., a desktop computer) or any other computing device. In at least one embodiment, any of the above-described delivery systems can be implemented as a single system 1100 or can implemented in a distributed architecture including multiple systems 1100.
In other examples, the systems, as described above can be implemented from a personal computer, a computing device, a mobile device, a mobile telephone, a facsimile device, a personal digital assistant (“PDA”) or other electronic device.
In at least some of the embodiments, the structures and/or functions of any of the above-described interfaces and panels can be implemented in software, hardware, firmware, circuitry, or a combination thereof. Note that the structures and constituent elements shown throughout, as well as their functionality, can be aggregated with one or more other structures or elements.
Alternatively, the elements and their functionality can be subdivided into constituent sub-elements, if any. As software, the above-described techniques can be implemented using various types of programming or formatting languages, frameworks, syntax, applications, protocols, objects, or techniques, including C, Objective C, C++, C#, Flex.TM., Fireworks. RTM., Java.TM., Javascript.TM., AJAX, COBOL, Fortran, ADA, XML, HTML, DHTML, XHTML, HTTP, XMPP, Python, and others. These can be varied and are not limited to the examples or descriptions provided.
The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to the practitioner skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical application, thereby enabling others skilled in the relevant art to understand the claimed subject matter, the various embodiments and the various modifications that are suited to the particular use contemplated.
The United States government has rights in this invention pursuant to Contract Number 89233218CNA000001 Between the U.S. Department of Energy and Triad National Security, LLC for operation of Los Alamos National Laboratory. The government has certain rights in the invention.
