SYSTEMS, METHODS, AND MEDIA FOR TRAINING NEURAL NETWORKS TO SOLVE OPTIMATIZATION PROBLEMS

Abstract

Training neural network(s) (NN(s)) to solve optimization problem (OP) includes: configuring a first NN of the NN(s) with a first set of (FSO) values for NN parameters (NNPs); selecting a FSO training samples (TSs) for training the NN(s); providing the FSO TSs and a FSO first variables to the first NN to produce a FSO first output variables (OVs); evaluating an OP-performance-measuring objective function (OF) based on the FSO TSs and the FSO first OVs to provide a first value of (VO) a performance metric (FVOPM) of the NN(s); updating the NNPs based on the FVOPM; selecting a second set of (SSO) TSs for training the NN(s); providing the SSO TSs and the FSO first OVs to the first NN to produce a SSO first OVs; evaluating the OF based on the SSO TSs and the SSO first OVs to provide a second VO the PM of the NN(s); and updating the NNPs based on the second VO the PM.

Description

BACKGROUND

Many applications require rapid and accurate optimization solutions in order to most-effectively operate. For example, in modern wireless communications systems, it is necessary to solving optimization problems in real-time in order to most effectively provide communication services.

Existing mechanisms for solving such optimization problems are either too slow or inaccurate.

Accordingly, new mechanisms for solving optimization problems are desirable

SUMMARY

In accordance with some embodiments, mechanisms, including systems, methods, and media, for training one or more neural networks to solve optimization problems are provided.

In some embodiments, systems for training one or more neural networks to solve an optimization problem are provided, the systems comprising: memory; and at least one hardware processor collectively configured to at least: configure a first neural network of the one or more neural networks with a first set of values for first neural network parameters; select a first set of training samples for training the one or more neural networks; provide the first set of training samples and a first set of first variables to the first neural network to produce a first set of first output variables; evaluate an objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide a first value of a performance metric of the one or more neural networks; update the first neural network parameters based on the first value of the performance metric; select a second set of training samples for training the one or more neural networks; provide the second set of training samples and the first set of first output variables to the first neural network to produce a second set of first output variables; evaluate the objective function that measures performance on the optimization problem based on the second set of training samples and the second set of first output variables to provide a second value of the performance metric of the one or more neural networks; and update the first neural network parameters based on the second value of the performance metric. In some of these embodiments, the at least one hardware processor if further collectively configured to: configure a second neural network of the one or more neural networks with a first set of values for second neural network parameters; provide the first set of training samples, a first set of second variables, and the first set of first output variables to the second neural network to produce a first set of second output variables; and update the second neural network parameters based on the first value of the performance metric, wherein providing the first set of training samples and the first set of first variables to the first neural network to produce the first set of first output variables includes providing the first set of second variables to the first neural network, wherein evaluating the objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide the first value of the performance metric of the one or more neural networks is also based on the first set of second output variables. In some of these embodiments, the first neural network and the second neural network have the same architecture. In some of these embodiments: selecting the first set of training samples comprises determining that the first set of training samples has a first level of complexity; and the at least one hardware processor if further collectively configured to: select a third set of training samples determined as having a second level of complexity that is greater than the first level of complexity; and train the one or more neural networks using the third set of training samples. In some of these embodiments, the objective function has a first level of complexity; and the at least one hardware processor if further collectively configured to train the one or more neural networks using an objective function that is determined to be more complex than the first level of complexity. In some of these embodiments, the performance metric is based on a sum of the objective function. In some of these embodiments, the optimization problem is to select optimal beamformers and wherein the method further comprises setting beamformers of a communication system based on the the second set of first output variables.

In some of embodiments, methods for training one or more neural networks to solve an optimization problem are provided, the methods comprising: configuring a first neural network of the one or more neural networks with a first set of values for first neural network parameters; selecting a first set of training samples for training the one or more neural networks; providing the first set of training samples and a first set of first variables to the first neural network to produce a first set of first output variables; evaluating an objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide a first value of a performance metric of the one or more neural networks; updating the first neural network parameters based on the first value of the performance metric using a hardware processor; selecting a second set of training samples for training the one or more neural networks; providing the second set of training samples and the first set of first output variables to the first neural network to produce a second set of first output variables; evaluating the objective function that measures performance on the optimization problem based on the second set of training samples and the second set of first output variables to provide a second value of the performance metric of the one or more neural networks; and updating the first neural network parameters based on the second value of the performance metric. In some of these embodiments, the method further comprises: configuring a second neural network of the one or more neural networks with a first set of values for second neural network parameters; providing the first set of training samples, a first set of second variables, and the first set of first output variables to the second neural network to produce a first set of second output variables; and updating the second neural network parameters based on the first value of the performance metric, wherein providing the first set of training samples and the first set of first variables to the first neural network to produce the first set of first output variables includes providing the first set of second variables to the first neural network, wherein evaluating the objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide the first value of the performance metric of the one or more neural networks is also based on the first set of second output variables. In some of these embodiments, the first neural network and the second neural network have the same architecture. In some of these embodiments: selecting the first set of training samples comprises determining that the first set of training samples has a first level of complexity; and the method further comprises: selecting a third set of training samples determined as having a second level of complexity that is greater than the first level of complexity; and training the one or more neural networks using the third set of training samples. In some of these embodiments: the objective function has a first level of complexity, and the method further comprises training the one or more neural networks using an objective function that is determined to be more complex than the first level of complexity. In some of these embodiments, the performance metric is based on a sum of the objective function. In some of these embodiments, the optimization problem is to select optimal beamformers and wherein the method further comprises setting beamformers of a communication system based on the the second set of first output variables.

In some of embodiments, non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for training one or more neural networks to solve an optimization problem are provided, the method comprising: configuring a first neural network of the one or more neural networks with a first set of values for first neural network parameters; selecting a first set of training samples for training the one or more neural networks; providing the first set of training samples and a first set of first variables to the first neural network to produce a first set of first output variables; evaluating an objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide a first value of a performance metric of the one or more neural networks; updating the first neural network parameters based on the first value of the performance metric; selecting a second set of training samples for training the one or more neural networks; providing the second set of training samples and the first set of first output variables to the first neural network to produce a second set of first output variables; evaluating the objective function that measures performance on the optimization problem based on the second set of training samples and the second set of first output variables to provide a second value of the performance metric of the one or more neural networks; and updating the first neural network parameters based on the second value of the performance metric. In some of these embodiments, the method further comprises: configuring a second neural network of the one or more neural networks with a first set of values for second neural network parameters; providing the first set of training samples, a first set of second variables, and the first set of first output variables to the second neural network to produce a first set of second output variables; and updating the second neural network parameters based on the first value of the performance metric, wherein providing the first set of training samples and the first set of first variables to the first neural network to produce the first set of first output variables includes providing the first set of second variables to the first neural network, wherein evaluating the objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide the first value of the performance metric of the one or more neural networks is also based on the first set of second output variables. In some of these embodiments, the first neural network and the second neural network have the same architecture. In some of these embodiments: selecting the first set of training samples comprises determining that the first set of training samples has a first level of complexity; and the method further comprises: selecting a third set of training samples determined as having a second level of complexity that is greater than the first level of complexity; and training the one or more neural networks using the third set of training samples. In some of these embodiments: the objective function has a first level of complexity, and the method further comprises training the one or more neural networks using an objective function that is determined to be more complex than the first level of complexity. In some of these embodiments, the performance metric is based on a sum of the objective function. In some of these embodiments, the optimization problem is to select optimal beamformers and wherein the method further comprises setting beamformers of a communication system based on the the second set of first output variables.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a first example architecture for learning to optimize that can be used in accordance with some embodiments.

FIG. 2 is an illustration of a first example algorithm for learning to optimize that can be used in accordance with some embodiments.

FIG. 3 is an illustration of a second example architecture for learning to optimize that can be used in accordance with some embodiments.

FIG. 4 is an illustration of a second example algorithm for learning to optimize that can be used in accordance with some embodiments.

FIG. 5 is an illustration of a third example algorithm for learning to optimize that can be used in accordance with some embodiments.

FIG. 6 is an illustration of a fourth example algorithm for learning to optimize that can be used in accordance with some embodiments.

FIG. 7 is an illustration of an example architecture for a neural network that can be used in accordance with some embodiments.

FIG. 8 is an illustration of example hardware that can be used in accordance with some embodiments.

DETAILED DESCRIPTION

Consider a family of optimization problems:

$P=\left\{\underset{x\in {ℝ}^n}{\mathrm{maximum}}R\left(s,x\right)❘s\in {ℝ}^m\right\}$

where R: ×→ is the objective, s∈ is a parameter that specifies the problem instance, and x∈ is the decision variable. In accordance with some embodiments, mechanisms, including systems, methods, and media for solving such problems are provided. More particularly, in accordance with some embodiments, mechanisms, including systems, methods, and media, for rapidly solving such problems in order to enable some real-time application, such as communication or control, are provided.

In some embodiments, iterative algorithms can be used to solve such problems. In some embodiments, iterative optimization entails application of a sequence of functions h_t:×→, t=1, 2, . . . , where each h_t maps s and a candidate point x_t to a new point x_t+1:=h_t(s, x_t) and is designed such that the sequence {x_t} converges to a maximizer of R(s, x) for all s. The recurrence is a composite mapping designed to approach or approximate a solution mapping:

$\begin{array}{cc}{x}^{*}\left(s\right)=\underset{x}{\arg \max }R\left(s,x\right).& \left(1\right)\end{array}$

In accordance with some embodiments, a goal is to learn an operator F_θ:×>, that can be referred to as an optimizer, that approximates the optimal mapping (1) for all s, thereby embedding in F_θ the set of all solutions of the problem family . This can be referred to herein as “learning to optimize”.

In some embodiments, F_θ is a neural network with learnable parameters θ. In some embodiments, the mapping (1) can be approximated through a recursive application of F_θ. Given s and a starting point x₀, F_θ can be applied for T steps, yielding x_t:=F_θ(s, x_t−1), t=1, 2, . . . , T, in some embodiments. An example block diagram of this scheme in accordance with some embodiments is shown in FIG. 1 for T=3 iterations. To measure the performance of a given trajectory {x_t}, the sum of the objective value over the trajectory, Σ_t=1^TR(s, x_t), can be used, in some embodiments. If s has distribution p_s, the objective function can be

$\begin{array}{cc}J\left(\theta \right):={𝔼}_{s\sim p_s}\left[{\sum }_{t=1}^{T}R\left(s,x_t\right)\right]& \left(2\right)\end{array}$

and the learning problem can be

$\underset{\theta }{\mathrm{maximum}}J\left(\theta \right)$

(to which gradient ascent can be applied, in some embodiments), in some embodiments. Training samples (i.e., problem instances specified by s that belong to ) can be obtained through simulation or measurement, in some embodiments. If p_s is known, then an arbitrary number of samples can be generated, in some embodiments. The starting point x₀ can be chosen either heuristically or randomly (or pseudo-randomly) for each s, in some embodiments. Although the quantity of interest is the final objective value, R(s, x_T), such a metric would ignore the intermediate values; summation over the entire trajectory, on the other hand, encourages each step to improve the objective.

Learning θ in effect “amortizes” the computational cost of optimization across all problem instances, shifting the computational burden from online optimization to offline learning, in some embodiments. In some embodiments, in deployment of the learned model, an optimization problem can be instantiated by s, then s can be fed to the model which outputs a near-optimal solution for that problem instance. Since the model execution entails just feedforward computation of the model, optimization can be done repeatedly and rapidly, in some embodiments.

An overall example of a procedure for learning to optimize is summarized in Algorithm 1 of FIG. 2, in accordance with some embodiments. In this example, the target family is specified by an objective function R with parameter s∈. In each epoch shown in lines 10-17 of the Algorithm, the iteration x_t:=F_θ(s, x_t−1) is carried out at line 15 for t=1, 2, . . . , T and the cumulative objective value J is computed at line 16. Finally, a gradient step updates θ at line 17. At test time, as shown in lines 18-23, for a given problem instantiated by s, x_t:=F_θ(s, x_t−1) is applied for t=1, 2, . . . , T and a final iterate x_T is returned at line 23.

Analogous to the common practice of varying the hyperparameters of an iterative optimization algorithm as the iterations progress, the parameter θ may in general be allowed to vary with t, in some embodiments. When θ is free to vary with respect to t, the optimizer can be referred to as being “untied”; Algorithm 1 as shown in FIG. 2 addresses a “tied” optimizer. The optimal parameters at t=1 need not be the same as some later step, say t=5. Allowing the parameters to vary may grant more expressive capacity. For an untied optimizer in Algorithm 1, F_θ can be replaced with a sequence of optimizers F_θ, t=1, . . . , T; hence line 15 of Algorithm 1 becomes:

$x_t^{\left(i\right)}=F_{{\theta }_t}\left(s^{\left(i\right)},x_{t-1}^{\left(i\right)}\right).$

In some embodiments, learning to optimize can be extended to problems where block coordinate optimization is appropriate. In some embodiments, a block coordinate approach splits the optimization variable into subsets and iteratively optimizes over each subset while holding the others fixed. If x=(y, z) is a partition of the optimization variable, the subproblems at iteration t have the form:

$\begin{array}{cc}{y}_t=\underset{y}{\arg \max }R\left(s,y,z_{t-1}\right)& \left(3\right)\end{array}$ $\begin{array}{cc}{z}_t=\underset{z}{\arg \max }R\left(s,y_t,z\right)& \left(4\right)\end{array}$

In some embodiments, this approach can be used to efficiently solve convex problems. In some embodiments, even if R is nonconvex, the subproblems may be tractable or admit closed-form solutions, thus providing an efficient means of finding a local optimum.

In some embodiments, two optimizers can be learned to learn the optimal mappings in (3) and (4). The rationale is twofold: a nonconvex objective may become simplified when certain coordinates are held constant; and splitting the optimization variable reduces the dimension of the output space of each policy which, by mitigating the curse of dimensionality, reduces the desired mapping's complexity.

An example procedure for doing so is summarized in Algorithm 2 of FIG. 4. Let F_θ and Go denote optimizers which output candidate points y and z, respectively. In some embodiments, the following alternating scheme can be used to iteratively compute y_t and z_t for t=1, . . . , T:

$\begin{array}{c}{y}_t=F_{\theta }\left(s,y_{t-1},z_{t-1}\right)\\ z_t=G_{\varphi }\left(s,y_t,z_{t-1}\right)\end{array}$

An example block diagram of this scheme in accordance with some embodiments is shown in FIG. 3 for T=3 iterations. As in the single-variable case (shown in FIG. 1), the average sum of the objective values obtained over the iterations can be expressed as:

$\begin{array}{cc}J\left(\theta ,\varphi \right):={𝔼}_{s\sim p_s}\left[{\sum }_{t=1}^{T}R\left(s,y_t,z_t\right)\right].& \left(5\right)\end{array}$

In some embodiments, equation (5) can be employed as a training objective function in order to learn the optimizer parameters θ and ϕ, which may be optimized via block coordinate optimization (as described above) as well. That is, at epoch 1, a gradient step for ϕ can be performed, at epoch 2, θ can be updated, at epoch 3, ϕ can be updated, and so on. To obtain a warm start for each optimizer, F_θ (or G_ϕ) may be separately pretrained via (3) (or (4)) by fixing z (or y).

In accordance with some embodiments, curriculum learning techniques can be used to learn the solution to the above-described optimization problems. In some embodiments, curriculum learning trains a model on a sequence of tasks of increasing difficulty. Each task can be defined by a particular training objective function and a particular training data distribution, in some embodiments. In some embodiments, the sequence of tasks can increase in difficulty, in that the loss functions become successively more complex and/or that the entropy of the data distributions increases as the curriculum progresses. In some embodiments, rather than solely learning a primary task, these techniques can first perform a first round of learning in which an easier subordinate task is learned, and then perform one or more subsequent rounds of learning based on a neural network produced in the previous round until training converges to a point in parameter space that might be unreachable had training solely been on the primary task.

In accordance with some embodiments, two examples of curriculum learning techniques that can be used with Algorithms 1 and 2 are described below. The first curriculum learning technique, which is referred to herein as a subspace curriculum learning technique, uses a fixed training objective and prescribes a sequence of training data distributions of increasing complexity. The second curriculum learning technique, which is referred to herein as a reward curriculum learning technique, uses a fixed training distribution and prescribes a sequence of training objectives of increasing complexity.

In accordance with some embodiments, the subspace curriculum learning technique curates the training data (problem instances) seen by the optimizer over the course of training. At each stage, in some embodiments, it samples from distributions of increasing entropy and therefore learns tasks of increasing difficulty. Observe that the distribution p_s affects the difficulty of maximizing (2). In some embodiments, for a zero-entropy distribution p_s(s)=δ(s−s₀) where s∈ is known, only a single output has to be learned, namely

$\underset{x}{\arg \max }{R}_1\left(s_0,x\right).$

In some embodiments, the subspace curriculum learning technique prescribes that, during each stage of the curriculum, the training data is restricted to a linear subspace of the state space . That is, in some embodiments, the optimizer is trained on a sequence of tasks corresponding to the problem families

$P_d=\left\{\underset{x\in {ℝ}^m}{\mathrm{maximum}}R\left(s,x\right)❘s\in S_d\right\}$

for d=1, 2, . . . , m where S_d⊂ is a d-dimensional subspace. Since S₁⊂S₂⊂ . . . ⊂, then ₁⊂₂⊂ . . . C⊂_m; so the complexity of the problem family increases with d. The entropy of the distribution increases with d, since the dimension of the sample space increases with d; for example, with p_α,d=(0, I), the entropy can be given by d(1+log 2π)/2+(log d)/2, which is increasing in d.

In some embodiments, the subspaces are generated via a particular orthonormal basis {b₁, . . . , b_m}⊂ chosen prior to training, such that S_d:=span ({b₁, . . . , b_d}). If the subspace dimension is d, training samples can be generated via s=Σ_i=1^dα_i,db_i, where the coefficients α_i,d∈ are sampled from a chosen distribution p_α,d, in some embodiments.

An example application of the subspace curriculum in a general learning environment is illustrated in Algorithm 3 of FIG. 5. Assume a training objective J(θ) where θ denotes the learnable model parameters. Before training begins, a random (or pseudo-random) orthonormal basis {b₁, . . . , b_m} (which induces the subspaces of in which training samples will reside) can be generated, in some embodiments. In some embodiments, the subspace dimension d=1 is initialized at line 8 and the subspace dimension d is incremented every N epochs at lines 12-13. Thus, in some embodiments, for the first N epochs at lines 9-13, all training samples lie on the line induced by basis vector b₁, i.e., the samples are generated according to s=α_1,1b₁, where α_1,1˜p_α,1. In some embodiments, after N epochs, the subspace dimension is increased to d=2 at lines 12-13, so that, in the following N epochs, training samples are generated at line 10 via s=α_1,2b₁+α_2,2b₂, where α_1,2, α_2,2˜p_α,2. In some embodiments, after N*(m−1) epochs, the subspace dimension is d=m, at which point the training samples span .

Since the subspace curriculum learning technique affects only the training data, it can be incorporated into various learning algorithms, in some embodiments. For example, in some embodiments, in Algorithm 1, one would only need to modify the training sample generation at line 11 to incorporate the subspace curriculum learning technique.

Suppose there are two objective functions: R₁: ×> and R₂: ×> corresponding to two distinct tasks such that the point

$\underset{x}{\arg \max }{R}_1\left(s,x\right)$

obtains a reasonably good objective value R₂ for all s. In some embodiments, θ can first be learned for the family of problems

$P_1=\left\{\underset{x\in {ℝ}^m}{\mathrm{maximum}}{R}_1\left(s,x\right)❘s\in {ℝ}^m\right\}$

using the training objective J₁:=_s˜ps[Σ_t=1^TR₁(s, x_t)]. A second family of problems can be expressed as

$P_2=\left\{\underset{x\in {ℝ}^m}{\mathrm{maximum}}{R}_2\left(s,x\right)❘s\in {ℝ}^m\right\}$

and this family of problems can use the training objective J₂:=_s˜ps[Σ_t=1^TR₂(s, x_t)].

If the optimizer converges to a point θ₁, then starting from θ₁ training can continue with primary task objective J₂:=_s˜ps[Σ_t=1^TR₂(s, x_t)] until convergence, in some embodiments. In some embodiments, θ₁ may be suboptimal with respect to ₂, but nonetheless may provide a warm start that ultimately leads to a point superior to that which would be obtained via training solely with J₂. Moreover, if R₁ is a relatively simple function (e.g., quadratic in x), then it stands to reason that the mapping

$\underset{x}{\arg \max }{R}_1\left(s,x\right)$

will be simpler and θ will converge quickly, in some embodiments.

Algorithm 4 of FIG. 6 demonstrates an example application of a reward curriculum learning technique in a general learning environment. It is assumed that there are K_c task objective functions

${\left\{J_k\right\}}_{k=1}^{K_c}$

such that the task difficulty increases with k and the desired task is represented by the final objective function J_Kc.

As illustrated in Algorithm 4, in some embodiments, training begins by initializing k=1 at line 5 and thus J₁ serves as the initial training objective. In each epoch in lines 6-10, a batch of samples are obtained at line 7 and then a gradient update is performed using J₁ at line 8. After N epochs, the task index is incremented to k=2 at lines 9-10 and J₂ is used as the training objective function. After N*(K_c−1) epochs, k=K_c and the final task objective J_Kc is used for the remainder of training.

In some embodiments, the reward curriculum learning technique can be applied to Algorithm 1 by modifying the training objective in line 16 according to the curriculum over the course of training.

Downlink beamforming is a fundamental technology in multiuser wireless communication that allows simultaneous transmission of multiple data streams from a base station (BS) to multiple users using the same time-frequency resource. The BS is equipped with an antenna array whose complex amplitudes are to be configured so that the transmitted signals add constructively in certain spatial directions and destructively in others, thereby enabling spatial multiplexing of user data streams. A particular configuration of amplitudes is called a beamformer. Each beamformer has sidelobes that interfere with other users, therefore the beamformers should be optimized jointly so as to maximize a performance function that quantifies desired system behavior. For example, the minimum user rate is an objective that promotes fairness among users and thus can be used as a performance metric in some embodiments. As another example, to achieve the best overall system performance, the weighted sum of the user rates can be used as a performance metric in some embodiments.

In accordance with some embodiments, the learning to optimize techniques described above (including Algorithms 1-4) can be applied to, for example, three beamforming scenarios: MISO, MIMO and relay. In some embodiments, this requires four main steps: (1) design the optimizer neural network architecture; (2) define a problem family with objective function R corresponding to a system performance criterion; (3) formulate an iterative scheme that outputs beamformers for given channels where the iterative operator is a learnable optimizer; and (4) devise a training procedure to learn the optimizer parameters such that the iterative scheme approaches the optimal channel-beamformer mapping.

In some embodiments, the learning to optimize techniques described above (including Algorithms 1-4) can be used to configure a multiple-input, single-output radio (MISO) communication system. When these techniques are used, in some embodiments, s described above can be equal to H below, x described above can be equal to W described below, and R described above can be equal to R(H, W).

Consider an N-antenna base station (BS) that communicates with K single-antenna users. The BS applies transmit beamformer w_i∈ to the ith user data stream, so that, if x_i∈ is the symbol intended for user i, the transmitted signal is Σ_i=1^Kw_iw_i. Let h_k denote the channel between the BS and user k (a user herein is a device with a receiver and/or transmitter), so that user k receives the signal

$y_k=h_k^H{\sum }_{i=1}^K{x}_i{w}_i+n_k,k=1,\dots ,K$

where n_k˜(0, σ²) is additive noise. User k's signal-to-interference-plus-noise ratio can be expressed as:

${\mathrm{SINR}}_k=\frac{{❘h_k^H{w}_k❘}^2}{{\sigma }^2+{\sum }_{i\ne k}{❘h_K^H{W}_i❘}^2}.$

The MISO beamforming problem can be formulated as

$\begin{array}{cc}\begin{array}{c}\underset{w_1,\dots ,w_K}{\mathrm{maximum}}f\left(r_1,\dots ,r_k\right)\\ \mathrm{subject}\mathrm{to}{\sum }_{k=1}^K{w_k}_2^2\le P\end{array}& \left(6\right)\end{array}$

where ƒ:→ is the system performance function and r_k:=log₂(1+SINR_k) is the achievable information rate of user k.

In accordance with some embodiments, (6) can be solved by applying Algorithm 1 as described below.

Let s=[h₁ . . . h_K]^H:=H∈ and x=[w₁ . . . w_K]:=W∈. The optimization problem class is

$P=\left\{\underset{\begin{array}{c}w\in {ℂ}^{N\times K}\\ {W}_F^2=P\end{array}}{\mathrm{maximum}}{R}_2\left(H,W\right)\right\},$

where R:^N×K×^N×KΘ is the performance criterion (ƒ in (6)). The learnable optimizer is denoted F_θ:^N×K×^N×K→^N×K with learnable parameters θ. The beamformers are iteratively computed via

$\begin{array}{cc}{W}_{t+1}:=F_{\theta }\left(H,W_t\right)]& \left(8\right)\end{array}$

and the training objective is

$\begin{array}{cc}J\left(\theta \right)={𝔼}_{H\sim p_H}\left[{\sum }_{t=1}^{T}R\left(H,W_t\right)\right]& \left(8\right)\end{array}$

where p_H is the channel distribution. In each epoch, a batch of channels is generated with SNR ρ_H² (the SNR can be made to vary from sample to sample so that the optimizer is trained on a range of SNRs as described above). For each sample in the batch, Algorithm 1 can carry out the iteration (7) for T steps, compute the corresponding cumulative loss (8) and perform a gradient step for θ.

In some embodiments, at least one of the above-described reward curriculum learning technique and the subspace curriculum learning technique can be used to obtain beamformers.

For example, a reward curriculum learning technique can be implemented as follows.

In some embodiments, minimum mean square error (MMSE) beamformers achieve a reasonably good sum rate and min rate, and the mean square error (MSE) objective is simply quadratic in W.

In some embodiments, linear MMSE beamformers can be given by

$W_k=\sqrt{p_k}\frac{{\left({\sigma }^2{I}_N+{\sum }_{i=1}^K\frac{P}{K}{h}_i{h}_i^H\right)}^{-1}*h_k}{{{\left({\sigma }^2{I}_N+{\sum }_{i=1}^K\frac{P}{K}{h}_i{h}_i^H\right)}^{-1}*h_k}_2}$

where {p_k} are transmit powers. In some embodiments, assume equal power allocation, p_k=P=K.

In some embodiments, the MSE can be defined as MSE(H, W):=[∥HWx+n−x∥₂²], where expectation is with respect to the noise n˜(0, σ²I) and data vector x˜(0,I).

In some embodiments, the MMSE optimization problem can be

$\underset{\begin{array}{c}W\in {ℂ}^{N\times K}\\ {W}_F^2=P\end{array}}{\mathrm{maximize}}\mathrm{MSE}\left(H,W\right).$

In some embodiments, to compute the MSE, the objective can be empirically evaluated via sampling independent and identically distribute (i.i.d.) vectors x and n, or the analytical expression MSE(H, W)=∥HW∥_F²−2Re{trace(HW)} can be used.

In terms of Algorithm 4, the reward curriculum technique can define the task loss R₁:=MSE and R₂ can be the desired performance criterion (e.g., sum rate, min rate).

For example, a subspace curriculum learning technique can be implemented as follows.

Assuming p_H˜(0,1), an arbitrary number of training samples can be generated and the training data can be curated according to the subspace curriculum. To implement subspace curriculum learning, the orthonormal basis vectors {b₁, . . . , b_n} of can be randomly (or pseudo-randomly) generated and the distributions {p_α,d|d=1, . . . , m} which are used to generate samples in a given subspace can be set to p_α,d=(0,1) for all d. For the MISO problem, the full channel space is , hence n=2NK.

In some embodiments, the learning to optimize techniques described above (including Algorithms 1-4) can be used to configure beamforming in a multiple-input, multiple-output radio (MIMO) communication systems. When these techniques are used, in some embodiments, s described above can be equal to {H_k} below, x described above can be equal to {W_k} described below, and R described above can be equal to R ({H_k}, {W_k}).

In the MIMO case, in some embodiments, each user is equipped with a receive antenna array capable of receive beamforming, or spatially filtering the impinging waveform, and hence may perform additional interference cancellation, thereby relaxing the transmit beamforming requirements.

Suppose the BS has N transmit antennas and user k has m_k receive antennas. The symbol x_k∈ is intended for user k and the BS applies the beamformers W_k∈, so that the tranmitted signal is Σ_k=1^K W_kx_k. If H_k∈ is user k's channel matrix, then user k's array measurement Y_k=[Y_k,1 . . . Y_k,mk]^T∈ has the form Y_k=H_k^HΣ_i=1^KW_ix_i+n_k, or

$Y_k=H_k^H{W}_k{x}_k+H_k^H\underset{i\ne k}{\sum ^K}{W}_i{x}_i+n_k$

Define Σ_k∈, the covariance matrix of the interference-plus-noise term,

${\sum }_k={\sigma }^{2}I+H_k^H\left({\sum }_{i\ne k}^K{W}_i{W}_i^H\right)H_k.$

Then the rate of user k is

$r_k={\log }_2❘I+{\sum }_k^{-1}{H}_k^H{W}_k{W}_k^H{H}_k❘$

and the beamformer design problem can then be (in some embodiments):

$\begin{array}{c}\underset{W_1,\dots W_K}{\mathrm{maximize}}f\left(r_1,\dots ,r_k\right)\\ \mathrm{subject}\mathrm{to}{\sum }_{i=1}^K{W_i}_F^2\le P.\end{array}$

In this example, the class of resource allocation problems under consideration can then be (in some embodiments):

$P=\left\{\underset{\begin{array}{c}{W}_k\in {ℂ}^{N\times m_k}\\ {\sum }_k{W_k}_F^2=P\end{array}}{\mathrm{maximize}}R\left(\left\{H_k\right\},\left\{W_k\right\}\right)\right\},$

where R:×→ is the performance criterion and H_k∈. In this case, in some embodiments, Algorithm 1 can be applied by defining s:=(H₁, . . . , H_K), x:=(W₁, . . . , W_K) and the objective function as follows: R({H_k}, {W_k})=Σ_k=1^K log₂|I+Σ_k⁻¹H_k^HW_kW_k^HH_k|.

In some embodiments, the learning to optimize techniques described above (including Algorithms 1-4) can be used to configure relay beamforming in a multiple-input, multiple-output radio (MIMO) communication system. When these techniques are used, in some embodiments, s described above can be equal to {H,G} below, x described above can be equal to {F, W} described below, and R described above can be equal to R(H, G, F, W).

Consider a scenario in which there is an M-antenna relay station (RS) between the N-antenna BS and the single-antenna users. Let G∈ be the MIMO channel matrix between the BS and the RS and let h_k∈ be the channel vector between the RS and user k. The transmitted signal from the BS is Σ_k=1^Kx_Kw_k. Denote the BS beamformers W=[W₁ . . . . W_k]∈.

The received signal at the RS can be given by z=G Σ_k=1^Kx_kw_k+v∈ where v˜(0,σ_r²). The RS can employ a transmit beamforming matrix F∈ and forward the signal Fz∈ to the users. The received signal at user k can be written as

$\begin{array}{c}{y}_k=h_k^H\mathrm{Fz}+n_k,\mathrm{or}\\ y_k=h_k^H{\mathrm{FGw}}_k{x}_k+\sum _{l\ne k}{h}_k^H{\mathrm{FGw}}_l{x}_l+h_k^H\mathrm{Fv}+n_k\end{array}$

where n_k˜(0,σ_r²), in some embodiments, the SINR of user k is

$\begin{array}{cc}{\mathrm{SINR}}_k=\frac{{❘h_k^H{\mathrm{FGw}}_k❘}^2}{{\sum }_{l\ne k}{❘h_k^H{\mathrm{FGw}}_l❘}^2+{h_k^{H}F}_2^2{\sigma }_r^2+{\sigma }^2}& \left(9\right)\end{array}$

and the user rates are given by r_k:=log₂(1+SINR_k), in some embodiments. The goal is to choose beamformers W and F to maximize a performance function ƒ subject to transmit power constraints at the BS and RS:

$\begin{array}{cc}\begin{array}{c}\underset{W,F}{\mathrm{maximize}}f\left(r_1,\dots ,r_k\right)\\ \begin{array}{cc}\mathrm{subject}\mathrm{to}& {W}_F^2\le P_b\\ & {F}_F^2\le P_r.\end{array}\end{array}& \left(10\right)\end{array}$

In accordance with some embodiments, (10) can be solved using the learning to optimize techniques described above as follows.

In some embodiments, the class of resource allocation problems under consideration can be

$P=\left\{\underset{\begin{array}{c}F\in {ℂ}^{M\times M},W\in {ℂ}^{N\times K}\\ {W}_F^2=P_b,{F}_F^2=P_r\end{array}}{\mathrm{maximize}}R\left(H,G,F,W\right)\right\},$

where H∈, G∈, and the performance criterion R is the sum rate as

$R\left(H,G,F,W\right)={\sum }_{k=1}^K{\log }_2\left(1+{\mathrm{SINR}}_k\right)$

where SINR_k is given by (9) below.

In some embodiments, when applying Algorithm 2 to this optimization problem, F_θ and G_ϕ are defined as optimizers and variables y:=W and z:=F given s:=(H,G) are alternatively selected.

When F is fixed, in some embodiments, the task of selecting W is equivalent to that of the MISO downlink beamforming problem described above with the channel matrix set equal to the effective channel {tilde over (H)}:=HFG.

When W is fixed, in some embodiments, the task is to choose relay beamformers F given the BS relay channel H and the relay-user effective channel GW.

In some embodiments, the beamformers can be computed by alternating between the two optimizers for t=1, . . . , T:

$\begin{array}{c}{W}_{t,0}=W_{t-1,T_W}\\ F_{t,0}=F_{t-1,T_F}\\ {\tilde{H}}_t={\mathrm{HF}}_{t,0}G\\ W_{t,k}=F_{\theta }\left({\tilde{H}}_t,W_{t,k-1}\right),k=1,\dots ,T_W\\ F_{t,k}=G_{\varphi }\left({\mathrm{GW}}_{t,T_W},H,F_{t,k-1}\right),k=1,\dots ,T_F,\end{array}$

where unrolled F_θ and G_ϕ have been unrolled for T_F and T_G steps, respectively. F_0;TF is set equal to I/√{square root over (M)} and W_0,TF is set equal to HG/∥HG∥_F. The input to F_θ is analogous to that of the MISO optimizer describe above, where {tilde over (H)}_t serves as the BS-user channel, but a key difference is that here the channel varies with t since it depends on the choice of F_t. The inputs of G_ϕ include GW_t (the effective channel between the relay and the users) along with H and F_t. The training objective is defined as

$J\left(\theta ,\varphi \right)=\left[𝔼\left[{\sum }_{t=1}^T{\sum }_{k=1}^{T_W}R\left(H,G,F_{t,0},W_{t,k}\right)\right]+{\sum }_{l=1}^{T_F}R\left(H,G,F_{t,l},W_{t,T_W}\right)\right].$

Since for fixed F the problem has the same structure has the MISO scenario, F_θ may be pretrained as a MISO beamforming optimizer (as described above) so that it outputs the optimal W for any given {tilde over (H)}. Having initialized F_θ, Algorithm 2 can then be applied.

In some embodiments, Algorithms 1-4 can be implemented using an optimizer with any suitable neural network architecture. For example, in some embodiments, the neural network architecture can include any function that: has parameters to be learned; and is a composition of linear and nonlinear functions (which can be referred to as “layers” or “activations”). In some embodiments, examples of layers (can be linear or nonlinear) can include: convolution layers-used in convolutional neural network (CNN); fully connected layers-used in multilayer perceptron (MLP); attention layers-used in the “Transformer” architecture; and message passing layers-used in graph neural networks (GNN). In some embodiments, examples of activations (nonlinear only) can include: rectified linear unit (ReLU); sigmoid function (logistic function); and hyperbolic tangent (tanh).

In some embodiments, for the beamforming applications, a neural network architecture including biconvolutional neural network (BiCNN), a DenseNet, and a normalization layer can be used.

FIG. 5 shows an example of such a neural network for the MISO application described above, in accordance with embodiment some embodiments. As illustrated in this figure, the inputs to F_θ are H (or s in Algorithms 1-4) and W_t−1 (or x_t−1 in Algorithms 1-4) and the output is W_t (or x_t in Algorithms 1-4). In some embodiments, any suitable number (e.g., three as shown in FIG. 1) of the F_θ from FIG. 5 can arranged in series in a configuration as shown in FIG. 1. Likewise, in some embodiments, the Fe shown in FIG. 5 can be used to implement F_θ and G_ϕ shown in FIG. 3.

In some embodiments, the BiCNN performs feature extraction on the network input. Convolution exploits the translation invariance of the input along both user and antenna dimensions, since for any ordering of input channel vectors the optimal beamformers are the same (up to a permutation of the user indices) and vice versa, in some embodiments.

In some embodiments, the DenseNet module mirrors the original DenseNet architecture, except that the convolutional layers are replaced by fully connected layers.

In some embodiments, each beamforming scenario includes a constraint on the transmit power. This constraint can be enforced by appending a normalization layer to the network. In particular, suppose {tilde over (W)}∈ is the beamformer matrix produced by the DenseNet module. Then, for a given transmit power P, the normalization layer can output W=√{square root over (P)}{tilde over (W)}/∥{tilde over (W)}∥_F so that the output satisfies ∥W∥_F²=P.

The above-described mechanisms for solving optimization problems can be perform on any suitable general-purpose computer or special-purpose computer. Any such general-purpose computer or special-purpose computer can include any suitable hardware. For example, as illustrated in example hardware 800 of FIG. 8, such hardware can include hardware processor 802, memory and/or storage 804, an input device controller 806, an input device 808, display/audio drivers 810, display and audio output circuitry 812, communication interface(s) 814, an antenna 816, and a bus 818.

Hardware processor 802 can include any suitable hardware processor, such as a microprocessor, a micro-controller, digital signal processor(s), dedicated logic, and/or any other suitable circuitry for controlling the functioning of a general-purpose computer or a special purpose computer in some embodiments.

Memory and/or storage 804 can be any suitable memory and/or storage for storing programs, data, and/or any other suitable information in some embodiments. For example, memory and/or storage 804 can include random access memory, read-only memory, flash memory, hard disk storage, optical media, and/or any other suitable memory.

Input device controller 806 can be any suitable circuitry for controlling and receiving input from input device(s) 808 in some embodiments. For example, input device controller 806 can be circuitry for receiving input from an input device 808, such as a touch screen, from one or more buttons, from a voice recognition circuit, from a microphone, from a camera, from an optical sensor, from an accelerometer, from a temperature sensor, from a near field sensor, and/or any other type of input device.

Display/audio drivers 810 can be any suitable circuitry for controlling and driving output to one or more display/audio output circuitries 812 in some embodiments. For example, display/audio drivers 810 can be circuitry for driving one or more display/audio output circuitries 812, such as an LCD display, a speaker, an LED, or any other type of output device.

Communication interface(s) 814 can be any suitable circuitry for interfacing with one or more communication networks. For example, interface(s) 814 can include network interface card circuitry, wireless communication circuitry, and/or any other suitable type of communication network circuitry.

Antenna 816 can be any suitable one or more antennas for wirelessly communicating with a communication network in some embodiments. In some embodiments, antenna 816 can be omitted when not needed.

Bus 818 can be any suitable mechanism for communicating between two or more components 802, 804, 806, 810, and 814 in some embodiments.

Any other suitable components can additionally or alternatively be included in hardware 800 in accordance with some embodiments.

It should be understood that at least some of the above-described functions of Algorithms 1-4 can be executed or performed in any order or sequence not limited to the order and sequence shown in and described in the figures. Also, some of the above functions of Algorithms 1-4 can be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. Additionally or alternatively, some of the above described functions of Algorithms 1-4 can be omitted.

In some embodiments, any suitable computer readable media can be used for storing instructions for performing the functions and/or processes described herein. For example, in some embodiments, computer readable media can be transitory or non-transitory. For example, non-transitory computer readable media can include media such as non-transitory magnetic media (such as hard disks, floppy disks, and/or any other suitable magnetic media), non-transitory optical media (such as compact discs, digital video discs, Blu-ray discs, and/or any other suitable optical media), non-transitory semiconductor media (such as flash memory, electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and/or any other suitable semiconductor media), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and/or any suitable tangible media. As another example, transitory computer readable media can include signals on networks, in wires, conductors, optical fibers, circuits, any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and/or any suitable intangible media.

Although the invention has been described and illustrated in the foregoing illustrative embodiments, it is understood that the present disclosure has been made only by way of example, and that numerous changes in the details of implementation of the invention can be made without departing from the spirit and scope of the invention, which is limited only by the claims that follow. Features of the disclosed embodiments can be combined and rearranged in various ways.

Claims

1. A system for training one or more neural networks to solve an optimization problem, comprising: memory; andat least one hardware processor collectively configured to at least: configure a first neural network of the one or more neural networks with a first set of values for first neural network parameters;select a first set of training samples for training the one or more neural networks;provide the first set of training samples and a first set of first variables to the first neural network to produce a first set of first output variables;evaluate an objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide a first value of a performance metric of the one or more neural networks;update the first neural network parameters based on the first value of the performance metric;select a second set of training samples for training the one or more neural networks;provide the second set of training samples and the first set of first output variables to the first neural network to produce a second set of first output variables;evaluate the objective function that measures performance on the optimization problem based on the second set of training samples and the second set of first output variables to provide a second value of the performance metric of the one or more neural networks; andupdate the first neural network parameters based on the second value of the performance metric.

2. The system of claim 1, wherein the at least one hardware processor if further collectively configured to: configure a second neural network of the one or more neural networks with a first set of values for second neural network parameters;provide the first set of training samples, a first set of second variables, and the first set of first output variables to the second neural network to produce a first set of second output variables; andupdate the second neural network parameters based on the first value of the performance metric,wherein providing the first set of training samples and the first set of first variables to the first neural network to produce the first set of first output variables includes providing the first set of second variables to the first neural network,wherein evaluating the objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide the first value of the performance metric of the one or more neural networks is also based on the first set of second output variables.

3. The system of claim 2, wherein the first neural network and the second neural network have the same architecture.

4. The system of claim 1, wherein selecting the first set of training samples comprises determining that the first set of training samples has a first level of complexity, andwherein the at least one hardware processor if further collectively configured to: select a third set of training samples determined as having a second level of complexity that is greater than the first level of complexity; andtrain the one or more neural networks using the third set of training samples.

5. The system of claim 1, wherein the objective function has a first level of complexity, andwherein the at least one hardware processor if further collectively configured to train the one or more neural networks using an objective function that is determined to be more complex than the first level of complexity.

6. The system of claim 1, wherein the performance metric is based on a sum of the objective function.

7. The system of claim 1, wherein the optimization problem is to select optimal beamformers and wherein the method further comprises setting beamformers of a communication system based on the the second set of first output variables.

8. A method for training one or more neural networks to solve an optimization problem, comprising: configuring a first neural network of the one or more neural networks with a first set of values for first neural network parameters;selecting a first set of training samples for training the one or more neural networks;providing the first set of training samples and a first set of first variables to the first neural network to produce a first set of first output variables;evaluating an objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide a first value of a performance metric of the one or more neural networks;updating the first neural network parameters based on the first value of the performance metric using a hardware processor;selecting a second set of training samples for training the one or more neural networks;providing the second set of training samples and the first set of first output variables to the first neural network to produce a second set of first output variables;evaluating the objective function that measures performance on the optimization problem based on the second set of training samples and the second set of first output variables to provide a second value of the performance metric of the one or more neural networks; andupdating the first neural network parameters based on the second value of the performance metric.

9. The method of claim 8, further comprising: configuring a second neural network of the one or more neural networks with a first set of values for second neural network parameters;providing the first set of training samples, a first set of second variables, and the first set of first output variables to the second neural network to produce a first set of second output variables; andupdating the second neural network parameters based on the first value of the performance metric,wherein providing the first set of training samples and the first set of first variables to the first neural network to produce the first set of first output variables includes providing the first set of second variables to the first neural network,wherein evaluating the objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide the first value of the performance metric of the one or more neural networks is also based on the first set of second output variables.

10. The method of claim 9, wherein the first neural network and the second neural network have the same architecture.

11. The method of claim 8, wherein selecting the first set of training samples comprises determining that the first set of training samples has a first level of complexity, andwherein the method further comprises: selecting a third set of training samples determined as having a second level of complexity that is greater than the first level of complexity; andtraining the one or more neural networks using the third set of training samples.

12. The method of claim 8, wherein the objective function has a first level of complexity, andwherein the method further comprises training the one or more neural networks using an objective function that is determined to be more complex than the first level of complexity.

13. The method of claim 8, wherein the performance metric is based on a sum of the objective function.

14. The method of claim 8, wherein the optimization problem is to select optimal beamformers and wherein the method further comprises setting beamformers of a communication system based on the the second set of first output variables.

15. A non-transitory computer-readable medium containing computer executable instructions that, when executed by a processor, cause the processor to perform a method for training one or more neural networks to solve an optimization problem, the method comprising: configuring a first neural network of the one or more neural networks with a first set of values for first neural network parameters;selecting a first set of training samples for training the one or more neural networks;providing the first set of training samples and a first set of first variables to the first neural network to produce a first set of first output variables;evaluating an objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide a first value of a performance metric of the one or more neural networks;updating the first neural network parameters based on the first value of the performance metric;selecting a second set of training samples for training the one or more neural networks;providing the second set of training samples and the first set of first output variables to the first neural network to produce a second set of first output variables;evaluating the objective function that measures performance on the optimization problem based on the second set of training samples and the second set of first output variables to provide a second value of the performance metric of the one or more neural networks; andupdating the first neural network parameters based on the second value of the performance metric.

16. The non-transitory computer-readable medium of claim 15, wherein the method further comprises: configuring a second neural network of the one or more neural networks with a first set of values for second neural network parameters;providing the first set of training samples, a first set of second variables, and the first set of first output variables to the second neural network to produce a first set of second output variables; andupdating the second neural network parameters based on the first value of the performance metric,wherein providing the first set of training samples and the first set of first variables to the first neural network to produce the first set of first output variables includes providing the first set of second variables to the first neural network,wherein evaluating the objective function that measures performance on the optimization problem based on the first set of training samples and the first set of first output variables to provide the first value of the performance metric of the one or more neural networks is also based on the first set of second output variables.

17. The non-transitory computer-readable medium of claim 16, wherein the first neural network and the second neural network have the same architecture.

18. The non-transitory computer-readable medium of claim 15, wherein selecting the first set of training samples comprises determining that the first set of training samples has a first level of complexity, andwherein the method further comprises: selecting a third set of training samples determined as having a second level of complexity that is greater than the first level of complexity; andtraining the one or more neural networks using the third set of training samples.

19. The non-transitory computer-readable medium of claim 15, wherein the objective function has a first level of complexity, andwherein the method further comprises training the one or more neural networks using an objective function that is determined to be more complex than the first level of complexity.

20. The non-transitory computer-readable medium of claim 15, wherein the performance metric is based on a sum of the objective function.

21. The non-transitory computer-readable medium of claim 15, wherein the optimization problem is to select optimal beamformers and wherein the method further comprises setting beamformers of a communication system based on the the second set of first output variables.