Nonlinear optimization system

Abstract

A computer solves a nonlinear optimization problem. An optimality check is performed for a current solution to an objective function that is a nonlinear equation with constraint functions on decision variables. When the performed optimality check indicates that the current solution is not an optimal solution, a barrier parameter value is updated, and a Lagrange multiplier value is updated for each constraint function based on a result of a complementarity slackness test. The current solution to the objective function is updated using a search direction vector determined by solving a primal-dual linear system that includes a dual variable for each constraint function and a step length value determined for each decision variable and for each dual variable. The operations are repeated until the optimality check indicates that the current solution is the optimal solution or a predefined number of iterations has been performed.

Description

BACKGROUND

The most straightforward way to handle inequality constraints in an optimization problem consists of transforming the inequality constraints into equality constraints by subtracting a vector of slack variables. The main benefit of this transformation is the ability to find an easy feasible starting point for interior-point based approaches. The second approach for handling inequality constraints consists of placing them into a barrier function. The most common two examples of barrier functions are the log-barrier function and the inverse barrier function. The log-barrier is usually preferred because the logarithm increases so slowly as the boundary is approached. The classical log-barrier (CLB) method transforms the optimization problem with inequality constraints to an unconstrained subproblem. The main difficulty associated with the CLB method is finding a strictly feasible starting point because finding the strictly feasible starting point has the same complexity as solving the original optimization problem with inequality constraints. Furthermore, it has been shown that a Hessian matrix of the log-barrier function becomes increasingly ill-conditioned as the barrier parameter converges to zero. As a result, CLB methods were quickly abandoned by practitioners.

As an alternative to CLB methods, a new method based on a modified log-barrier (MLB) function was introduced that eliminates most of the deficiencies associated with the CLB method while retaining its best features. However, though MLB methods may exhibit strong convergence properties, they require shifting a starting point via an auxiliary problem after updating the barrier parameter. The MLB method further has singularities at its boundary leading to numerical difficulties that prevent the algorithm from performing well when approaching the solution.

SUMMARY

In an example embodiment, a computer-readable medium is provided having stored thereon computer-readable instructions that when executed by a computing device, cause the computing device to determine a solution to a nonlinear optimization problem. (A) An optimality check is performed for a current solution to a predefined objective function. The predefined objective function is a nonlinear equation with a plurality of decision variables. (B) When the performed optimality check indicates that the current solution is an optimal solution, the optimal solution to the predefined objective function is output. The optimal solution includes a value for each decision variable of the plurality of decision variables that optimizes a result of the predefined objective function.

When the performed optimality check indicates that the current solution is not the optimal solution, (C) a complementarity slackness test is performed with the current solution; (D) based on the result of the performed complementarity slackness test, a barrier parameter value is updated and a Lagrange multiplier value is updated for each constraint function of a plurality of constraint functions defined for objective function optimization; (E) a search direction vector is determined by solving a primal-dual linear system that includes a dual variable defined for each constraint function of the plurality of constraint functions; (F) a step length value is determined for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated barrier parameter value; (G) the current solution to the predefined objective function is updated using the determined search direction vector and the determined step length value for each decision variable of the plurality of decision variables and for each dual variable of a plurality of dual variables; and (H) (A) to (G) are repeated until the performed optimality check indicates that the current solution is the optimal solution or a predefined number of iterations of (D) has been performed.

Each constraint function applies one or more of the plurality of decision variables. The search direction vector includes a direction value for each decision variable of the plurality of decision variables and for each dual variable of a plurality of dual variables. The search direction vector is determined based on the updated barrier parameter value and the updated Lagrange multiplier value for each constraint function of a plurality of constraint functions.

In another example embodiment, a computing device is provided. The computing device includes, but is not limited to, a processor and a computer-readable medium operably coupled to the processor. The computer-readable medium has instructions stored thereon that, when executed by the processor, cause the computing device to determine a solution to a nonlinear optimization problem.

In yet another example embodiment, a method of determining a solution to a nonlinear optimization problem is provided.

Other principal features of the disclosed subject matter will become apparent to those skilled in the art upon review of the following drawings, the detailed description, and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative embodiments of the disclosed subject matter will hereafter be described referring to the accompanying drawings, wherein like numerals denote like elements.

FIG. 1 depicts a block diagram of a nonlinear optimization solution device in accordance with an illustrative embodiment.

FIGS. 2A, 2B, and 2C depict a flow diagram illustrating examples of operations performed by a nonlinear optimization application of the nonlinear optimization device of FIG. 1 in accordance with an illustrative embodiment.

FIG. 3 depicts a flow diagram illustrating examples of operations performed by the nonlinear optimization application of the nonlinear optimization device of FIG. 1 in applying a quadratic penalty in accordance with an illustrative embodiment.

DETAILED DESCRIPTION

Referring to FIG. 1, a block diagram of a nonlinear optimization device 100 is shown in accordance with an illustrative embodiment. Nonlinear optimization device 100 may include an input interface 102, an output interface 104, a communication interface 106, a non-transitory computer-readable medium 108, a processor 110, a nonlinear optimization application 122, and a solution description 124. Fewer, different, and/or additional components may be incorporated into nonlinear optimization device 100.

Input interface 102 provides an interface for receiving information from the user or another device for entry into nonlinear optimization device 100 as understood by those skilled in the art. Input interface 102 may interface with various input technologies including, but not limited to, a keyboard 112, a microphone 113, a mouse 114, a display 116, a track ball, a keypad, one or more buttons, etc. to allow the user to enter information into nonlinear optimization device 100 or to make selections presented in a user interface displayed on display 116.

The same interface may support both input interface 102 and output interface 104. For example, display 116 comprising a touch screen provides a mechanism for user input and for presentation of output to the user. Nonlinear optimization device 100 may have one or more input interfaces that use the same or a different input interface technology. The input interface technology further may be accessible by nonlinear optimization device 100 through communication interface 106.

Output interface 104 provides an interface for outputting information for review by a user of nonlinear optimization device 100 and/or for use by another application or device. For example, output interface 104 may interface with various output technologies including, but not limited to, display 116, a speaker 118, a printer 120, etc. Nonlinear optimization device 100 may have one or more output interfaces that use the same or a different output interface technology. The output interface technology further may be accessible by nonlinear optimization device 100 through communication interface 106.

Communication interface 106 provides an interface for receiving and transmitting data between devices using various protocols, transmission technologies, and media as understood by those skilled in the art. Communication interface 106 may support communication using various transmission media that may be wired and/or wireless. Nonlinear optimization device 100 may have one or more communication interfaces that use the same or a different communication interface technology. For example, nonlinear optimization device 100 may support communication using an Ethernet port, a Bluetooth antenna, a telephone jack, a USB port, etc. Data and/or messages may be transferred between nonlinear optimization device 100 and another computing device of a distributed computing system 130 using communication interface 106.

Computer-readable medium 108 is an electronic holding place or storage for information so the information can be accessed by processor 110 as understood by those skilled in the art. Computer-readable medium 108 can include, but is not limited to, any type of random access memory (RAM), any type of read only memory (ROM), any type of flash memory, etc. such as magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips, . . . ), optical disks (e.g., compact disc (CD), digital versatile disc (DVD), . . . ), smart cards, flash memory devices, etc. Nonlinear optimization device 100 may have one or more computer-readable media that use the same or a different memory media technology. For example, computer-readable medium 108 may include different types of computer-readable media that may be organized hierarchically to provide efficient access to the data stored therein as understood by a person of skill in the art. As an example, a cache may be implemented in a smaller, faster memory that stores copies of data from the most frequently/recently accessed main memory locations to reduce an access latency. Nonlinear optimization device 100 also may have one or more drives that support the loading of a memory media such as a CD, DVD, an external hard drive, etc. One or more external hard drives further may be connected to nonlinear optimization device 100 using communication interface 106.

Processor 110 executes instructions as understood by those skilled in the art. The instructions may be carried out by a special purpose computer, logic circuits, or hardware circuits. Processor 110 may be implemented in hardware and/or firmware. Processor 110 executes an instruction, meaning it performs/controls the operations called for by that instruction. The term “execution” is the process of running an application or the carrying out of the operation called for by an instruction. The instructions may be written using one or more programming language, scripting language, assembly language, etc. Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information. Processor 110 may retrieve a set of instructions from a permanent memory device and copy the instructions in an executable form to a temporary memory device that is generally some form of RAM. Nonlinear optimization device 100 may include a plurality of processors that use the same or a different processing technology.

Some machine-learning approaches may be more efficiently and speedily executed and processed with machine-learning specific processors (e.g., not a generic central processing unit (CPU)). Such processors may also provide additional energy savings when compared to generic CPUs. For example, some of these processors can include a graphical processing unit (GPU), an application-specific integrated circuit, a field-programmable gate array, an artificial intelligence accelerator, a purpose-built chip architecture for machine learning, and/or some other machine-learning specific processor that implements a machine learning approach using semiconductor (e.g., silicon, gallium arsenide) devices. These processors may also be employed in heterogeneous computing architectures with a number of and a variety of different types of cores, engines, nodes, and/or layers to achieve additional various energy efficiencies, processing speed improvements, data communication speed improvements, and/or data efficiency targets and improvements throughout various parts of the system.

Nonlinear optimization application 122 performs operations associated with defining solution description 124 that describes a solution to a nonlinear optimization problem with inequality constraints. Some or all of the operations described herein may be embodied in nonlinear optimization application 122. The operations may be implemented using hardware, firmware, software, or any combination of these methods.

Referring to the example embodiment of FIG. 1, nonlinear optimization application 122 is implemented in software (comprised of computer-readable and/or computer-executable instructions) stored in computer-readable medium 108 and accessible by processor 110 for execution of the instructions that embody the operations of nonlinear optimization application 122. Nonlinear optimization application 122 may be written using one or more programming languages, assembly languages, scripting languages, etc. Nonlinear optimization application 122 may be integrated with other analytic tools. As an example, nonlinear optimization application 122 may be part of an integrated data analytics software application and/or software architecture such as that offered by SAS Institute Inc. of Cary, N.C., USA that solve nonlinear optimization problems individually and as part of training models. Merely for illustration, nonlinear optimization application 122 may be implemented as part of one or more SAS software tools such as JMP®, Base SAS, SAS® Enterprise Miner™, SAS® Event Stream Processing, SAS/STAT®, SAS® High Performance Analytics Server, SAS® Visual Data Mining and Machine Learning, SAS® LASR™, SAS® In-Database Products, SAS® Scalable Performance Data Engine, SAS® Cloud Analytic Services (CAS), SAS/OR®, SAS/ETS®, SAS® Visual Analytics, SAS® Viya™, SAS In-Memory Statistics for Hadoop®, etc. all of which are developed and provided by SAS Institute Inc. of Cary, N.C., USA. Data mining, statistical analytics, and response prediction are practically applied in a wide variety of industries to solve technical problems.

Nonlinear optimization application 122 may be implemented as a Web application. For example, nonlinear optimization application 122 may be configured to receive hypertext transport protocol (HTTP) responses and to send HTTP requests. The HTTP responses may include web pages such as hypertext markup language (HTML) documents and linked objects generated in response to the HTTP requests. Each web page may be identified by a uniform resource locator (URL) that includes the location or address of the computing device that contains the resource to be accessed in addition to the location of the resource on that computing device. The type of file or resource depends on the Internet application protocol such as the file transfer protocol, HTTP, H.323, etc. The file accessed may be a simple text file, an image file, an audio file, a video file, an executable, a common gateway interface application, a Java applet, an extensible markup language (XML) file, or any other type of file supported by HTTP.

Nonlinear optimization application 122 provides an optimization that is a search for a maximum or a minimum of an objective function (also called a cost function), where search variables are restricted by particular constraints, where the constraints define a feasible region for the solution. A Lagrange function for the problem minimize ƒ(x) subject to c(x)≥0 is defined by (x, z)=ƒ(x)−c(x)^Tz, where (x,z) indicates the Lagrange function, x indicates one or more decision variables, ƒ(x) indicates the objective function, c(x) indicates a set of one or more constraint functions applied to the optimization of the objective function, T indicates a transpose, and z indicates one or more dual variables defined for each constraint function of the one or more constraint functions. The maximum of ƒ(x) is simply a negation of the minimum of −ƒ(x). The objective function is a mathematical formula defined using a set of one or more variables x called optimization variables or decision variables. The one or more constraint functions are mathematical formulas that may include equalities (=) and inequalities (<, >, ≤, ≥) required to be simultaneously satisfied based on values for the one or more variables x. A problem is said to be linearly constrained if the functions in the constraints are linear. A linear programming problem is a linearly constrained problem with a linear objective function. A nonlinear programming problem occurs where some function in the objective function or constraint functions is nonlinear. The objective function and/or constraint functions may include one or more polynomial functions. For illustration, the optimization problem may be defined as minimize ƒ(x₁,x₂)=100(x₂−x²)²+(1−x_t)² subject to x_t≥5 and x_t+Sx₂≥76, where x_t and x₂ are decision variables, and x_t≥5 and x_t+Sx₂≥76 are two constraint functions.

A feasible solution for the one or more decision variables x minimizes an objective function value and is called an optimal solution to the optimization problem, and the corresponding objective value is called the optimal value. Nonlinear optimization application 122 varies values for the optimization variables when searching for the optimal value for the objective function that is output to solution description 124. The solution may further be sent to a calling function, for example, of a model training process such as for training a neural network model, a support vector machine model, etc.

For two real vectors x and y of the same length, a Euclidean scalar product can be denoted by x^Ty. A Euclidean norm can be defined as ∥x∥=(x^Tx)^1/2. A positive part of a real number t is a function defined by t⁺=max{t, 0}. ∀ƒ(x)ϵⁿ indicates a gradient of ƒ at x. ∀c(x)^T indicates a Jacobian matrix of c at x, and is an m×n matrix, whose i^th row is a vector ∀c_i(x)^T, where m is a number of the one or more constraint functions, and n is a number of the one or more decision variables. C(x) and Z are diagonal matrices whose diagonal entries are given by the components of c(x) and z, respectively, e is a vector of all ones. A gradient and a Hessian of (x,z) with respect to the primal-dual variables (x,z) can be defined as ∀_x(x, z)=∀ƒ(x)−∀c(x)z and

${\nabla }_x^2ℒ\left(x,z\right)={\nabla }^{2}f\left(x\right)-\sum _{i=1}^m{z}_i{\nabla }^2{c}_i\left(x\right),$ respectively, where ∀_x² indicates a Hessian computation with respect to x, and ∀_x indicates a derivative computation with respect to x.

The first order optimality conditions, also called Karush-Khun-Tucker (KKT) conditions, for the problem minimize ƒ(x) subject to c(x)≥0 can be defined by

$\left(\begin{array}{c}\nabla f\left(x\right)-\nabla c\left(x\right)z\\ C\left(x\right)\mathrm{Ze}\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right),c\left(x\right)\ge 0\mathrm{and}z\ge 0.$

A paper by R. Polyak, Modified barrier functions (theory and methods), Mathematical Programming, Volume 54, Issue 2, Series A, pp. 177-222 (Feb. 1992) proposed the following MLB function associated with solving the problem minimize ƒ(x) subject to c(x)≥0

$\begin{array}{cc}{ℳ}_{\lambda ,\mu }\left(x\right):=f\left(x\right)-\mu \sum _{i\in ℐ}{\lambda }_i\log \left(\frac{c_i\left(x\right)}{\mu }+1\right)& \left(1\right)\end{array}$ where ={1, . . . , m}, λ is a vector of positive estimates of m Lagrange multipliers, μ>0 is a barrier parameter value, λ_i is an i^th Lagrange multiplier, c_i(x) is an i^th constraint function, and m is a number of constraint functions included in c(x). The MLB method solves the problem minimize ƒ(x) subject to c(x)≥0 by solving a sequence of unconstrained minimization problems

$\underset{x\in {ℝ}^n}{\min }{ℳ}_{\lambda ,\mu }\left(x\right),$ where n is a number of decision variables included in ƒ(x). The KKT conditions for solving the sequence of unconstrained minimization problems is

$\nabla f\left(x\right)-\sum _{i\in ℐ}\frac{\mu {\lambda }_i}{c_i\left(x\right)+\mu }\nabla c_i\left(x\right)=0.$

In the present application,

$z_i=\frac{\mu {\lambda }_i}{c_i\left(x\right)+\mu },$ i=1, . . . m is introduced where z_i is an i^th dual variable value paired with an i^th constraint function c_i(x),

$\begin{array}{cc}r\left(w,\lambda ,\mu \right):=\left(\begin{array}{c}\nabla f\left(x\right)-\nabla c\left(x\right)z\\ C\left(x\right)\mathrm{Ze}+\mu \left(z-\lambda \right)\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right).& \left(2\right)\end{array}$

These conditions appear as a perturbation of the KKT conditions of the initial problem. Nonlinear optimization application 122 applies a Newton-like method to equation (2) while updating X and p to guarantee a global convergence. Nonlinear optimization application 122 is more flexible than the CLB method and offers two possibilities to ensure the convergence to a point for which the KKT conditions are satisfied: 1) reduce the barrier parameter or 2) update the Lagrange multiplier estimate. Using the CLB method, only the barrier parameter is driven to zero.

Nonlinear optimization application 122 executes an outer iteration and an inner iteration. A solution vector w_k=(x_k,z_k) is associated with a k^th outer iteration, where x_k is a k^th one or more decision variables, z_k is a k^th one or more dual variables paired with a k^th one or more constraint functions. A minimization process starts by updating λ_k and p_k depending on a progress toward a complementarity slackness, where λ_k is a k^th estimate of each Lagrange multiplier of the plurality of Lagrange multiplier, μ_k is a k^th barrier parameter value, and ∥C(x_k)z_k∥≤α∥C(x_k−1)z_k−1∥ is a complementarity slackness test used to determine how λ_k and μ_k are updated, where αϵ(0,1). When ∥(x_k)z_k∥≤α∥C(x_k−1)z_k−1∥, the Lagrange multiplier estimate is updated by setting λ_k+1=z_k, and the barrier parameter is updated by setting μ_k+1≤μ_k. When ∥C(x_k)z_k≥α∥C(x_k−1)z_k−1∥, the barrier parameter is updated by setting μ_k+1≤ρμ_k and the Lagrange multiplier is updated by setting λ_k+1=λ_k to ensure that it does not behave badly far from a solution. The complementarity slackness test used to measure the progress achieved by the k^th outer iteration may be too demanding especially for nonlinear optimization. As a result, a condition based on a non-monotonic decrease of the complementarity slackness may be used instead as defined by∥C(x_k)z_k∥≤α max{∥C(x_ij)z_ij∥:(k−)⁺≤j≤k}+ζ_k  (3)where ≥1, {i_k} is a sequence whose elements are the iteration indices at which λ_k+1=z_k (or ∥C(x_k)z_k∥≤α∥C(x_k−1)z_k−1∥) and {(ζ_k} is a first sequence that converges to zero.

After updating μ_k and λ_k, a search direction d={d_x, d_z} is computed by solving the following primal-dual linear system

$\begin{array}{cc}\left(\begin{array}{cc}{\nabla }_x^2ℒ\left(w_k\right)& -\nabla c\left(x_k\right)\\ Z_k\nabla {c\left(x_k\right)}^{\top }& C\left(x_k\right)+{\mu }_{k+1}I\end{array}\right)\left(\begin{array}{c}{d}_x\\ d_Z\end{array}\right)=-\left(\begin{array}{c}{\nabla }_xℒ\left(w_k\right)\\ Z_{k}C\left(x_k\right)e+{\mu }_{k+1}\left(z_k-{\lambda }_{k+1}\right)\end{array}\right)& \left(4\right)\end{array}$ where T indicates a transpose. This corresponds to the result of applying a Newton-like method to equation (2). Setting λ_k+1=z_k, the linear system appears as a regularized Newton method applied to the KKT conditions of the initial problem. Hence a high rate of convergence is expected provided that the parameters are correctly updated and the Jacobian constraint is of full rank. The complementarity slackness test is used to ensure that the update λ_k+1=z_k occurs as often as possible, where the Jacobian constraint is defined using

$J\left(w_k,{\mu }_{k+1}\right)\colon\colon =\left(\begin{array}{cc}{\nabla }_x^2ℒ\left(w_k\right)& -\nabla c\left(x_k\right)\\ Z_k\nabla {c\left(x_k\right)}^{\top }& C\left(x_k\right)+{\mu }_{k+1}I\end{array}\right).$

The linear system defined by equation (4) can be written as J(w_k,μ_k+1)d=−r(w_k,λ_k+1,μ_k+1). After the step computation, a new iteration is defined by {circumflex over (x)}_k=x_k+α_k^xd_k^x and {circumflex over (z)}_k=z_k+α_k^zd_k^z, where the step lengths α_k^x and α_k^x are computed such that

$\begin{array}{cc}\frac{c\left(x_k+{\alpha }_k^x{d}_k^x\right)}{{\mu }_{k+1}}+1>0& \left(5\right)\end{array}$ andz_k+α_k^zd_k^z>0  (6)

To decide whether the candidate iterate ŵ_k is selected, the following criterion is tested ∥r(ŵ_k,λ_k+1,μ_k+1)∥≤ε_k, where {ε_k} is a second sequence that converges to zero. If ∥r(ŵ_k,λ_k+1,μ_k+1)∥>ε_k, a sequence of inner iterations to find w_k+1 satisfying ∥r(w_k+1,λ_k+1,μ_k+1)∥≤ε_k is performed. The sequence of inner iterations minimize a new primal-dual merit function Φ_λ,μ(w)=_λ,μ(x)+ηΨ_λ,μ(w) with fixed values of the Lagrange multiplier estimate and the barrier parameter where η≥0 is a first scaling parameter and

${\Psi }_{\lambda ,\mu }\left(w\right)=\mu \sum _{i\in ℐ}\left(\frac{\left(c_i\left(x\right)+\mu \right)z_i}{\mu {\lambda }_i}-\log \left(\frac{\left(c_i\left(x\right)+\mu \right)z_i}{\mu {\lambda }_i}\right)\right).$ The control of iterates is performed in both primal and dual spaces during the minimization process including the inner iterations.

For the logarithmic term to be well defined, the following inequality should be satisfied

$\frac{c_i\left(x\right)}{\mu }+1>0,\forall i\in ℐ.$ Due to non-convexity, the inequality

$\frac{c_i\left(x\right)}{\mu }+1>0$ can restrict how quickly the barrier parameter p can be reduced. This situation mainly occurs when the complementarity slackness test is not satisfied and the current values of the constraints do not allow the barrier parameter μ to be reduced by a large factor. To respond to this occurrence, a quadratic extrapolation is applied by modifying a definition of the function by distinguishing two cases based on the infeasibility of the current iterate:

$ℳ\left(x\right)=f\left(x\right)-\mu \sum _{i\in ℐ}{\lambda }_i\{\begin{array}{ccc}\log \left(\frac{c_i\left(x\right)}{\mu }+1\right)& \mathrm{if}{c}_i\left(x\right)\ge -\beta \mu ,& \left(7a\right)\\ \frac{1}{2}{q^a\left(c_i\left(x\right)\right)}^2+q^b{c}_i\left(x\right)+q^c& \mathrm{if}{c}_i\left(x\right)<-\beta \mu ,& \left(7b\right)\end{array}$ where 0≤β≤1 and the coefficients q^a, q^b, q^c are given by

$q^a=\frac{-1}{{\left(\mu \left(1-\beta \right)\right)}^2},q^b=\frac{1-\beta }{{\mu \left(1-\beta \right)}^2},\mathrm{and}{q}^c=\frac{\beta \left(2-3\beta \right)}{2{\left(1-\beta \right)}^2}+\log \left(1-\beta \right).$

In an illustrative embodiment, β=0.9. Whenever a “large” infeasibility is detected using c_i(x)<−βμ, the quadratic extrapolation term for the corresponding constraint may be used. In this manner, quadratic approximations may be seen as a penalty rather than a barrier. In other words, if the i^th constraint function becomes increasingly infeasible, the constraint is penalized using a quadratic term. Let and Q be two non-intersecting index sets denoting the set of constraints that satisfy c_i(x)≥−βμ and c_i(x)<−βμ, respectively. The linear system J(w_k,h,μ_k)d=−r(w_k,h,λ_k,μ_k) has the following form

$\left(\begin{array}{ccc}{\nabla }_x^2ℒ\left(w\right)& -\nabla c_{ℬ}\left(x\right)& -\nabla c_Q\left(x\right)\\ Z_{ℬ}\nabla {c_{ℬ}\left(x\right)}^{\top }& c_{ℬ}\left(x\right)+\mu I& 0\\ \mu q^a{⩓}_Q{c_Q\left(x\right)}^{\top }& 0& -I\end{array}\right)\left(\begin{array}{c}{d}_x\\ d_{z_{ℬ}}\\ d_{z_Q}\end{array}\right)=-\left(\begin{array}{c}{\nabla }_xℒ\left(w\right)\\ C_{ℬ}\left(x\right)Z_{ℬ}e+\mu \left(z_{ℬ}-{\lambda }_{ℬ}\right)\\ \mu {⩓}_Q\left(q^a{c}_Q\left(x\right)+q^{b}e\right)-z_Q\end{array}\right),$ where Λ_B=diag(λ_B). During the inner iterations, the following merit function is applied

$\Phi \left(w\right)=ℳ\left(x\right)+v\mu \sum _{i\in ℬ}\left(\frac{\left(c_i\left(x\right)+\mu \right)z_i}{\mu {\lambda }_i}-\log \left(\frac{\left(c_i\left(x\right)+\mu \right)z_i}{\mu {\lambda }_i}\right)\right),$ where ν is a second scaling parameter.

Methods for solving nonlinear optimization problems may be classified into two classes: primal and primal-dual. Using the substitution

$z_i=\frac{\mu {\lambda }_i}{c_i\left(x\right)+\mu }$ is a starting point for introducing a primal-dual framework in the context of the MLB method. Nonlinear optimization application 122 implements the MLB method in the primal-dual space. The control of the iterates is done in both primal and dual spaces during the entire minimization process, including the globalization phase corresponding to an inner algorithm. A new primal-dual merit function is used, and an estimate of the Lagrange multipliers is updated depending on a progress made by the iterates towards a complementarity slackness.

Referring to FIGS. 2A to 2C, example operations associated with nonlinear optimization application 122 are described when training dataset 124 is stored on nonlinear optimization device 100. Additional, fewer, or different operations may be performed depending on the embodiment of nonlinear optimization application 122. The order of presentation of the operations of FIGS. 2A to 2C is not intended to be limiting. Some of the operations may not be performed in some embodiments. Although some of the operational flows are presented in sequence, the various operations may be performed in various repetitions and/or in other orders than those that are illustrated. For example, a user may execute nonlinear optimization application 122, which causes presentation of a first user interface window, which may include a plurality of menus and selectors such as drop-down menus, buttons, text boxes, hyperlinks, etc. associated with nonlinear optimization application 122 as understood by a person of skill in the art. The plurality of menus and selectors may be accessed in various orders. An indicator may indicate one or more user selections from a user interface, one or more data entries into a data field of the user interface, one or more data items read from computer-readable medium 108 or otherwise defined with one or more default values, etc. that are received as an input by nonlinear optimization application 122. The operations of nonlinear optimization application 122 further may be performed in parallel using a plurality of threads and/or a plurality of worker computing devices.

In operation 200, a first indicator of an objective function ƒ(x_i), i=1, . . . , n may be received, where n is a number of decision variables the values of which are to be computed by optimization based on minimization or maximization. In an alternative embodiment, the first indicator may not be received. For example, the objective function may be stored, for example, in computer-readable medium 108 and used automatically. The objective function is defined by a nonlinear equation that is not defined by a line when graphed.

In an operation 202, a second indicator may be received that indicates a set of constraint functions c_i(x), i=1, . . . , m, where m is a number of constraint functions. Each constraint function may be an equation that defines an equality (=) or an inequality (<, >, ≤, ≥) with constant values and one or more of the decision variables. At least one of the constraint functions includes an inequality. In an alternative embodiment, the second indicator may not be received. For example, the set of constraint functions may be stored, for example, in computer-readable medium 108 and used automatically.

In an operation 204, a third indicator may be received that indicates a solution starting point w₀=(x₀,z₀) for searching for the optimum solution using the primal x and dual z, where x₀ is a primal vector x_0,i, i=1, . . . , n, and z₀ is a dual vector z_0,i, i=1, . . . , m, where n≥1 and m≥1. In an alternative embodiment, the third indicator may not be received or selectable. For example, the starting point w₀=(x₀,z₀) may be stored, for example, in computer-readable medium 108 and used automatically. For illustration, the primal vector is initialized to all zeroes and the dual vector is initialized to all ones.

In an operation 206, a fourth indicator may be received that indicates an initial barrier parameter μ₀>0. In an alternative embodiment, the fourth indicator may not be received. For example, the initial barrier parameter μ₀ may be stored, for example, in computer-readable medium 108 and used automatically. For illustration, μ₀=max(−c_i(x₀)+0.1, 0.1), which guarantees that

$\frac{c\left(x_0\right)}{{\mu }_0}+e\ge 0$ to ensure that the barrier term is well defined, where e is a vector of all ones.

In an operation 208, a fifth indicator may be received that indicates an initial Lagrange multiplier value for each constraint function λ_0,i>0, i=1, . . . , m, where λ₀ is a Lagrange multiplier vector. In an alternative embodiment, the fifth indicator may not be received. For example, the initial Lagrange multiplier value for each constraint λ_0,i, i=1, . . . , m may be stored, for example, in computer-readable medium 108 and used automatically. For illustration, λ_0,i=1.

In an operation 210, a sixth indicator may be received that indicates a value for a convergence parameter a. In an alternative embodiment, the sixth indicator may not be received. For example, the value for the convergence parameter a may be stored, for example, in computer-readable medium 108 and used automatically. The convergence parameter a should be chosen such that a e (0,1), which guarantees that a right hand-side of equation (3) converges to zero. For illustration, α=0.9.

In an operation 212, a seventh indicator may be received that indicates a value for a barrier reduction parameter ρ, where ρϵ(0,1). In an alternative embodiment, the seventh indicator may not be received. For example, the value for the barrier reduction parameter ρ may be stored, for example, in computer-readable medium 108 and used automatically. For illustration, ρ=0.2.

In an operation 214, an eighth indicator may be received that indicates a predefined integer value , where ϵ. In an alternative embodiment, the eighth indicator may not be received. For example, the predefined integer value may be stored, for example, in computer-readable medium 108 and used automatically. The predefined integer value is ≥1 and is used to define a number of last iterations used to satisfy a condition for a non-monotonic decrease of the complementarity slackness to update the barrier parameter p and the estimate of the Lagrange multipliers λ. For illustration, =2.

In an operation 216, a ninth indicator of a maximum number of iterations I_max may be received. In an alternative embodiment, the ninth indicator may not be received. For example, a default value may be stored, for example, in computer-readable medium 108 and used automatically. In another alternative embodiment, the value of the maximum number of iterations I_max may not be selectable. Instead, a fixed, predefined value may be used. For illustration, a default value of the maximum number of iterations I_max may be 250 though other values may be used. The maximum number of iterations I_max indicates how many iterations are performed as part of solving the optimization problem. In an alternative embodiment, a maximum computing time may be specified in addition to or instead of the maximum number of iterations I_max and used in a similar manner to stop the optimization process when the maximum computing time is reached.

In an operation 220, a tenth indicator may be received that indicates a value for a first scaling parameter η, where ηϵ(0,1). In an alternative embodiment, the tenth indicator may not be received. For example, the value for the first scaling parameter η may be stored, for example, in computer-readable medium 108 and used automatically. For illustration, η=10⁻⁴.

In an operation 222, an eleventh indicator may be received that indicates a value for a second scaling parameter ν, where ν≥0. In an alternative embodiment, the eleventh indicator may not be received. For example, the value for the second scaling parameter ν may be stored, for example, in computer-readable medium 108 and used automatically. For illustration, ν=10⁻⁴.

In an operation 224, a twelfth indicator may be received that indicates a value for whether the quadratic penalty is to be applied. For example, a quadratic penalty application may be set to zero or one, or alternatively, true or false, to indicate whether the quadratic penalty is to be applied. In an alternative embodiment, the twelfth indicator may not be received. For example, the quadratic penalty application may be stored, for example, in computer-readable medium 108 and used automatically. For illustration, the quadratic penalty application may be defined as one or true by default.

In an operation 226, a thirteenth indicator may be received that indicates a value for a feasibility parameter β, where 0≤β≤1, when the quadratic penalty application is selected, for example, indicates true. In an alternative embodiment, the thirteenth indicator may not be received. For example, the value for feasibility parameter β may be stored, for example, in computer-readable medium 108 and used automatically. For illustration, β=0.9. The feasibility parameter β defines how close to the singularities the MLB function is extrapolated.

In an operation 228, an index i₀ and an index j are initialized, for example, as i₀=0 and j=0.

In an operation 230, an iteration index k is initialized, for example, as k=0.

In an operation 232, an optimality check is performed using a current solution w_k=(x_k,z_k), where the current decision variable vector x and dual variable vector z are indicated by the iteration index k. For example, the optimality check is performed using

$\left(\begin{array}{c}\nabla f\left(x_k\right)-\nabla c\left(x_k\right)z_k\\ C\left(x_k\right)Z_{k}e\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$ as defined above.

In operation 234, a determination is made concerning whether the current solution w_k is an optimum solution based on the optimality check or whether k≥I_max. When w_k is an optimum solution or k≥I_max, processing continues in an operation 236. When w_k is not an optimum solution and k<I_max, processing continues in an operation 240 to perform another iteration.

In operation 236, the current solution vector x_k is output as the optimized solution that includes a value for each decision variable x_k,i, i=1, . . . n. For example, the current solution vector x_k may be output to solution description 124 and/or to display 116. Additional information regarding the optimization process further may be output such as the current objective function value ƒ(x_k).

Referring to FIG. 2B, in operation 240, a value ζ_k is determined from a sequence that converges to zero. For illustration, ζ_k=10μ_k to guarantee that a right hand-side of equation (3) converges to zero.

In an operation 242, a complementarity slackness test is performed using the current solution w_k=(x_k,z_k). For example, the complementarity slackness test is performed using ∥C(x_k)z_k∥≤α∥C(x_k−1)z_k−1∥ or ∥C(x_k)z_k∥≤α max {∥C(x_ij)z_ij∥:(k−)⁺≤j≤k}+ζ_k as described relative to equation (3) above.

In operation 244, a determination is made concerning whether ∥C(x_k)z_k∥≤α max{∥C(x_ij)z_ij∥:(k−)⁺≤j≤k}+ζ_k, which indicates that the Lagrange multipliers are updated. When ∥C(x_k)z_k∥≤α max{∥C(x_ij)z_ij∥:(k−⁺≤j≤k}+ζ_k, processing continues in an operation 246. When ∥C(x_k)z_k∥>α max{∥C(x_ij)z_ij∥:(k−⁺≤j≤k}+ζ_k, processing continues in an operation 252 to instead update the barrier parameter.

In operation 246, the next barrier parameter value is defined or updated, for example, using μ_k+1=0.5μ_k such that μ_k+1≤μ_k. As another example, μ_k+1=qμ_k, where 0<q≤1 and q may be defined by a user or using a default value. An illustrative value for q may be q=0.5 or q=0.2 though other values may be used.

In an operation 248, the next Lagrange multiplier values are defined or updated, for example, using λ_k+1,h=z_k,h, h=1, . . . , m.

In an operation 250, the index i_k and the index j are updated, for example, using i_k+1=k and j=k, and processing continues in an operation 258.

In operation 252, the next barrier parameter value is defined or updated, for example, using μ_k+1=ρμ_k.

In an operation 254, the next Lagrange multiplier values are defined or updated, for example, using λ_k+1,h=λ_k,h, h=1, . . . , m.

In an operation 256, the index i_k and the index j are updated, for example, using i_k+1=i_k and j=k, and processing continues in an operation 258.

In operation 258, a search direction vector d is computed as

$d_{k,h}=\left(d_{k,g}^{x_{k,g}},d_{k,h}^{z_{k,h}}\right),g=1,\dots ,n,h=1,\dots ,m$ for each next solution value by solving the primal-dual linear system J(w_k,μ_k+1)d=−r(w_k,λ_k+1,μ_k+1), where

$r\left(w_k,{\lambda }_{k+1},{\mu }_{k+1}\right)=\left(\begin{array}{c}{\nabla }_xℒ\left(w_k\right)\\ Z_{k}C\left(x_k\right)e+{\mu }_{k+1}\left(z_k-{\lambda }_{k+1}\right)\end{array}\right),J\left(w_k,{\mu }_{k+1}\right)=\left(\begin{array}{cc}{\nabla }_x^2ℒ\left(w_k\right)& -\nabla c\left(x_k\right)\\ Z_k\nabla {c\left(x_k\right)}^{\top }& C\left(x_k\right)+{\mu }_{k+1}I\end{array}\right)$ and search direction vector d is computed using a regularized Newton method applied to the KKT conditions of the initial problem. For example, the linear system is solved by applying an LDL factorization as described in Chapter 4 of a book by Golub, G. H., and C. F. Van Loan, Matrix Computations, 3rd ed. (Johns Hopkins University Press 1996).

In an operation 260, step lengths are computed as α_k,g^xk,g, g=1, . . . , n and α_k,h^zk,h, h=1, . . . , m for each next solution value such that

$\frac{c\left(x_k+{\alpha }_k^x{d}_k^x\right)}{{\mu }_{k+1}}+1>0\mathrm{and}{z}_k+{\alpha }_k^z{d}_k^z>0$ as discussed above relative to equations (5) and (6). A fraction is applied to the boundary rule to compute α_k^x and α_k^z that satisfy equations (5) and (6). For illustration,

${\alpha }_k^x:=\max \left\{{\alpha }_k^x\in \left(0,1\right]:\frac{c\left(x_k+{\alpha }_k^x{d}_k^x\right)}{{\mu }_{k+1}}+1\ge 0.005\left(\frac{c\left(x_k\right)}{{\mu }_{k+1}}+1\right)\right\}\mathrm{and}$ ${\alpha }_k^z=\max \left\{{\alpha }_k^z\in \left(0,1\right]:z_k+{\alpha }_k^z{d}_k^z\ge 0.005z_k\right\}.$ The constraints are first linearized to compute an initial estimate of a primal step length α_k^x. For example, an initial estimate is computed using

${\alpha }_k^x:=\min \left\{1,-0.995\underset{\left(i:{\left(\nabla {c\left(x_k\right)}^{\top }{d}_k^x\right)}_i<0\right)}{\min }\left\{\frac{{c\left(\left(x_k\right)\right)}_i+{\mu }_{k+1})}{{\left(\nabla {c\left(x_k\right)}^{\top }{d}_k^x\right)}_i}\right\}\right\}.$

If the initial estimate is not acceptable because

$\frac{c\left(x_k+{\alpha }_k^x{d}_k^x\right)}{{\mu }_{k+1}}+1<0.005\left(\frac{c\left(x_k\right)}{{\mu }_{k+1}}+1\right),$ a backtracking method is applied to iterate to a solution for the primal step length α_k^x such that

$\frac{c\left(x_k+{\alpha }_k^x{d}_k^x\right)}{{\mu }_{k+1}}+1\ge 0.005\left(\frac{c\left(x_k\right)}{{\mu }_{k+1}}+1\right).$ For the dual step length α_k^z, a unit step length is tried as an initial estimate of the dual step length α_k^z=1. If the initial estimate of the dual step length α_k^z is not acceptable because z_k+α_k^zd_k^z<0.005z_k, a backtracking method is applied to iterate to a solution for the dual step length α_k^z such that z_k+α_k^zd_k^z≥0.005z_k.

In an operation 262, a next solution ŵ_k=({circumflex over (x)}_k,{circumflex over (z)}_k) is determined using {circumflex over (x)}_k=x_k,g+α_k,g^xk,gd_k,g^xk,g for g=1, . . . , n and {circumflex over (z)}_k=z_k,h+α_k,h^xk,hd_k,h^xk,h for h=1, . . . , m.

In an operation 264, a value ε_k is determined, for example, using ε_k=0.9 max{∥r(w_j,λ_j,μ_j)∥:(k−4)⁺≤j≤k}+10μ_k+1, where

$r\left(w_j,{\lambda }_j,{\mu }_j\right)=\left(\begin{array}{c}{\nabla }_xℒ\left(w_j\right)\\ {\hat{Z}}_{j}C\left(x_j\right)e+{\mu }_j\left(z_j-{\lambda }_j\right)\end{array}\right).$ A maximum of the norm of the vector ∥(w_j,z_j,μ_j)∥ over a last five iterations (iterations k, k−1, k−2, k−3, k−4) is chosen where r(w_j,λ_j,μ_j) was already computed and stored above.

In an operation 266, an iteration check is performed using the next solution ŵ_k=({circumflex over (x)}_k,{circumflex over (z)}_k), and processing continues in operation 268. For example, the iteration check is performed using ∥r(ŵ_k, λ_k+1, μ_k+1)∥≤ε_k as described relative to equation (4) above, where

$r\left({\hat{w}}_k,{\lambda }_{k+1},{\mu }_{k+1}\right)=\left(\begin{array}{c}{\nabla }_xℒ\left({\hat{w}}_k\right)\\ {\hat{Z}}_{k}C\left({\hat{x}}_k\right)e+{\mu }_{k+1}\left({\hat{z}}_k-{\lambda }_{k+1}\right)\end{array}\right).$

Referring to FIG. 2C, in operation 268, a determination is made concerning whether ∥r(ŵ_k, λ_k+1, μ_k+1)∥≤ε_k, which indicates whether inner iterations are needed. When ∥r(ŵ_k, λ_k+1, μ_k+1)∥≤ε_k, processing continues in an operation 270 because the inner iterations are not needed. When ∥r(ŵ_k, λ_k+1, μ_k+1)∥>ε_k, processing continues in an operation 274 to perform the inner iterations.

In operation 270, the next solution is updated using index w_k+1=ŵ_k.

In an operation 272, the iteration index k is incremented, for example, using k=k+1, and processing continues in operation 232 to determine if w_K+1 is the optimal solution.

In operation 274, an inner iteration index h is initialized using h=0, w_k,h=ŵ_k, λ=λ_k+1, and μ=μ_k+1.

Similar to operation 258, in an operation 276, a search direction vector d is computed as d=(d^xk,d^zk) for each next solution value by solving J(w_k,h,μ)d=−r(w_k,h, λ, μ) as discussed above.

In an operation 278, α is defined using α=1.

In an operation 280, step lengths are computed based on the value of the quadratic penalty application. For example, referring to FIG. 3, the step lengths are computed based on the value of the quadratic penalty application.

In an operation 300, a determination is made concerning the value of the quadratic penalty application. When the quadratic penalty application is zero or false, processing continues in an operation 302. When the quadratic penalty application is one or true, processing continues in an operation 304.

Similar to operation 260, in operation 302, step lengths are computed as {tilde over (α)}_k,h^wk,hϵ(0, αdo it9 for each next solution value such that

$\frac{c\left(x_{k,h}+{\tilde{\alpha }}_{k,h}^{x_{k,h}}{d}^{x_{k,h}}\right)}{\mu }+1>0\mathrm{and}{z}_{k,h}+{\tilde{\alpha }}_{k,h}^{z_{k,h}}{d}^{z_{k,h}}>0$ as discussed above relative to equations (5) and (6) and processing is done for the quadratic penalty application and continues in an operation 282. A fraction is again applied to the boundary rule to compute {tilde over (α)}_k^x and {tilde over (α)}_k^z that satisfy equations (5) and (6). For illustration,

${\tilde{\alpha }}_k^x:=\max \left\{{\tilde{\alpha }}_k^x\in \left(0,1\right]:\frac{c\left(x_k+{\tilde{\alpha }}_k^x{d}_k^x\right)}{{\mu }_k}+1\ge 0.005\left(\frac{c\left(x_k\right)}{{\mu }_k}+1\right)\right\}\mathrm{and}$ ${\tilde{\alpha }}_k^z=\max \left\{{\tilde{\alpha }}_k^z\in \left(0,1\right]:z_k+{\tilde{\alpha }}_k^z{d}_k^z\ge 0.005z_k\right\}.$

In operation 304, a determination is made concerning whether c_i(x_k,h)<−βμ. When c_i(x_k,h)<−βμ, processing continues in an operation 306. When c_i(x_k,h)≥−βμ, processing continues in operation 302.

In operation 306, quadratic penalty coefficient values are computed, for example, using coefficients q^a, q^b, q^c defined by

$q^a=\frac{-1}{{\left(\mu \left(1-\beta \right)\right)}^2},q^b=\frac{1-\beta }{{\mu \left(1-\beta \right)}^2},\mathrm{and}{q}^c=\frac{\beta \left(2-3\beta \right)}{2{\left(1-\beta \right)}^2}+\log \left(1-\beta \right).$

In an operation 308, step lengths are computed as {tilde over (α)}_k,h^wk,hϵ(0,α] for each next solution value such that

$\frac{1}{2}{q^a\left(c_i\left(x_{k,h}+{\tilde{\alpha }}_{k,h}^{x_{k,h}}{d}^{x_{k,h}}\right)\right)}^2+q^b{c}_i\left(x_{k,h}+{\tilde{\alpha }}_{k,h}^{x_{k,h}}{d}^{x_{k,h}}\right)+q^c>0\hspace{1em}\hspace{1em}\mathrm{and}{z}_{k,h}+{\tilde{\alpha }}_{k,h}^{z_{k,h}}{d}^{z_{k,h}}>0,$ and processing is done for the quadratic penalty application and continues in operation 282.

${\tilde{\alpha }}_{k,h}^{x_{k,h}}=1,\mathrm{and}$ $\frac{1}{2}{q^a\left(c_i\left(x_{k,h}+{\tilde{\alpha }}_{k,h}^{x_{k,h}}{d}^{x_{k,h}}\right)\right)}^2+q^b{c}_i\left(x_{k,h}+{\tilde{\alpha }}_{k,h}^{x_{k,h}}{d}^{x_{k,h}}\right)+q^c>0$ is applied to make sure that the dual variable z remains positive. Once the dual step length {tilde over (α)}_k,h^zk,h is computed, {tilde over (α)}_k,h^wk,h=min ({tilde over (α)}_k,h^xk,h,{tilde over (α)}_k,h^zk,h).

Referring again to FIG. 2C, in operation 282, step lengths are computed as α_k,h^wk,hϵ(0, {tilde over (α)}_k,h^wk,h] for each next solution value such that the Armijo condition is satisfied based on

${\Phi }_{\lambda ,\mu }\left(w_{k,h}+{\alpha }_{k,h}^{w_{k,h}}{d}^{w_{k,h}}\right)\le {\Phi }_{\lambda ,\mu }\left(w_{k,h}\right)+{\alpha }_{k,h}^{w_{k,h}}\eta \nabla {{\Phi }_{\lambda ,\mu }\left(w_{k,h}\right)}^{\top }{d}^{w_{k,h}},\mathrm{where}$ $\Phi \left(w\right)=ℳ\left(x\right)+v\mu \sum _{i\in ℬ}\left(\frac{\left(c_i\left(x\right)+\mu \right)z_i}{\mu {\lambda }_i}-\log \left(\frac{\left(c_i\left(x\right)+\mu \right)z_i}{\mu {\lambda }_i}\right)\right).$

In an operation 284, a determination is made concerning whether another backtracking step is needed. When another backtracking step is needed,

$\frac{c\left(x_{k,h}+{\alpha }_{k,h}^{x_{k,h}}{d}^{x_{k,h}}\right)}{\mu }+1\le 0\mathrm{or}{z}_{k,h}+{\alpha }_{k,h}^{z_{k,h}}{d}^{z_{k,h}}\le 0.$ When another backtracking step is not needed,

$\frac{c\left(x_{k,h}+{\alpha }_{k,h}^{x_{k,h}}{d}^{x_{k,h}}\right)}{\mu }+1>0\mathrm{and}{z}_{k,h}+V_{k,h}^{z_{k,h}}{d}^{z_{k,h}}>0.$ When another backtracking step is needed, processing continues in an operation 286. When another backtracking step is not needed, processing continues in an operation 288.

In operation 286, α=α, and processing continues in operation 278.

In operation 288, α_k,h=α for each next solution value.

In an operation 290, a next solution w_k,h+1=(x_k,h,z_k,h) is determined using w_k,h+1=w_k,h+α_k,hd^wk,h.

In an operation 292, a determination is made concerning whether ∥r(w_k,h+1,λ, μ)∥≥ε_k, When ∥r(w_k,h+1,λ, μ)∥≤ε_k, processing continues in an operation 296 because additional inner iterations are not needed. When ∥r(w_k,h+1,λ, μ)∥>ε_k, processing continues in an operation 294 to perform additional inner iterations.

In operation 294, the inner iteration index h is incremented, for example, using h=h+1, and processing continues in operation 276 to perform another inner iteration.

In operation 296, the next solution is updated using index w_k+1=w_k,h+1.

In an operation 298, the iteration index k is incremented, for example, using k=k+1, and processing continues in operation 232 to determine if w_k+1 is the optimal solution.

Nonlinear optimization application 122 was developed and executed in the C programming language without quadratic penalty application (indicated as 122) and with quadratic penalty application (indicated as 122 with Q). The models were written using the OPTMODEL Procedure developed and provided by SAS Institute Inc. of Cary, N.C., USA as part of SAS OR®. The numerical solutions were obtained using a 64-bit 2.50 gigahertz Intel® Xeon® CPU E7-4880 machine with 8 processors and 128 GB RAM.

A first set of problems includes two problems that naturally arise in astrophysics, engineering, and biology. The Lane-Emden problem is defined as Δu+u³=0. The variational problem associated with this problem is

$g_{LE}\left(u\right)={\int }_D\left(\frac{1}{2}{\nabla u\left(s\right)}^2-\frac{1}{4}{u\left(s\right)}^4\right)\mathrm{ds},$ where D is a unit square. The singularly perturbed Dirichlet problem defined as ε²Δu+u³−u=0. The variational problem associated with this problem is

$g_D\left(u\right)={\int }_D\left(\frac{{\varepsilon }^2}{2}{\nabla u\left(s\right)}^2+\frac{1}{2}{u\left(s\right)}^2-\frac{1}{4}{u\left(s\right)}^4\right)ds,$ where D is a unit circle. A solution for the Lane-Emden and Dirichlet problems can be obtained by discretizing the corresponding variational problems using difference approximations based on a triangularization of the corresponding domain D. Using this, an approximate solution of the two problems is found by solving the following inequality optimization problem:

Minimize w, subject to g(u_k)≤h, 1≤k≤m and |u_k−u_k+1|≤h, 0≤k≤m, where h>0 is a fixed tolerance, and m is a number of breakpoints used by an elastic string algorithm. Note that for the first set of inequality constraints, the function g is replaced with g_LE or g_D. For each problem, the results were computed using m=10 and m=40.

Table 1 below summarizes the performance comparison between nonlinear optimization application 122 without quadratic penalty application and with quadratic penalty application in terms of computing time on the first set of problems with m=10 (_10) and m=40 (_40).

	TABLE 1
	Problem	122	122 with Q
	Lane_Emden_10	2.75	1.06
	Lane_Emden_40	77.86	13.17
	Dirichlet_10	1.01	0.51
	Dirichlet_40	4.2	10.61

The results show that nonlinear optimization application 122 with quadratic penalty application provides an improved efficiency relative to nonlinear optimization application 122 without quadratic penalty application. This improvement results based on the number of function evaluations required. For the first set of problems, the number of function evaluations required by nonlinear optimization application 122 with quadratic penalty application was much less than that required by nonlinear optimization application 122 without quadratic penalty application because nonlinear optimization application 122 without quadratic penalty application has to perform backtracking to satisfy the inequality

$\frac{c\left(x_k+{\alpha }_k^x{d}_k^x\right)}{{\mu }_{k+1}}+1>0;$ whereas, when the inequality

$\frac{c\left(x_k+{\alpha }_k^x{d}_k^x\right)}{{\mu }_{k+1}}+1>0$ is not satisfied using nonlinear optimization application 122 with quadratic penalty application, the quadratic term is used instead. As a result, there is no backtracking required, and a unit step-length could be chosen.

A primal-dual CLB method was also developed and executed in the C programming language, for example, based on methods described in a paper by Silva, R. et al., Local analysis of the feasible primal-dual interior-point method, Computational Optimization and Applications, Volume 40, Issue 1, pp. 41-57 (May 2008). Again, the model was written using the OPTMODEL Procedure. Using the primal-dual CLB method, a strictly feasible initial point was assumed with respect to the inequality constraint. The computation time results are summarized in Table 2 below.

	TABLE 2
	Problem	CLB method	122 with Q
	Lane_Emden_10	89.33	1.06
	Lane_Emden_40	591.25	13.17
	Dirichlet_10	5.22	0.51
	Dirichlet_40	33.87	10.61

Table 2 confirms the superiority of nonlinear optimization application 122 with quadratic penalty application over the classical primal-dual CLB method. Again, the improvement results based on the number of function evaluations required. For example, for the problem Lane_Emden_10, the primal-dual CLB method required 32,761 function evaluations while nonlinear optimization application 122 with quadratic penalty application only required 24, which had a significant impact on the computing time that made nonlinear optimization application 122 with quadratic penalty application 84 times faster when solving problem Lane_Emden_10.

A first general-purpose nonlinear optimization solver described in a paper by A. Wachter and L. T. Biegler, On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming, Mathematical Programming, Volume 106, Issue 1, Series A, pp. 25-57 (Mar. 2006) (GPNO1) was developed and executed in the C programming language. A second general-purpose nonlinear optimization solver described in a paper by P. Armand and R. Omheni, A globally and quadraticaiiy convergent primal-dual augmented iagrangian algorithm for equality constrained optimization, Optimization Methods and Software, Volume 32, Issue 1, pp. 1-21 (2017) (GPNO2) was developed and executed in the C programming language. Again, the models were written using the OPTMODEL Procedure. Both GPNO1 and GPNO2 are well-established nonlinear optimization problem solvers. Both GPNO1 and GPNO2 convert the inequality constraints to equality constraints using slack variables. GPNO1 is understood to be the best open-source general-purpose nonlinear optimization solver. GPNO2 applies a primal-dual augmented Lagrangian interior-point algorithm while GPNO1 implements a line-search filter method. Both GPNO1 and GPNO2 were run using their default options except for the scaling procedure. All executions were done without scaling to have a similar execution stop condition.

The performance was compared using a class of degenerate problems created by manipulating the OPTMODEL models and adding the constraint −c₁(x)²≤0. As a result, the problems are appended with “_d” to indicate this addition. This transformation does not have any impact on the feasible region but makes every problem degenerate since the Mangasarian-Fromovitz Constraint Qualification is not satisfied at all solution points. This transformation makes the problem harder to solve. The computation time results are summarized in Table 3 below.

TABLE 3
Problem	GPNO1	GPNO2	122 with Q
Lane_Emden_10_d	9.02	190.76	2.71
Lane_Emden_40_d	48.27	76.28	26.93
Dirichlet_10_d	10.92	24.66	8.32
Dirichlet_40_d	1949.69	266.99	13.74

For the four different problems, nonlinear optimization application 122 with quadratic penalty application clearly outperformed GPNO1 and GPNO1 in terms of computation time. Using nonlinear optimization application 122 with quadratic penalty application provides the improvement by extrapolating the modified logarithmic term with quadratic approximations.

A second set of computational experiments explores the behavior when solving a product portfolio problem that arises in industry. The goal of this problem is to define a portfolio strategy for a laundry powder business. More precisely, the portfolio variables (additive levels, intermediate compositions, intermediate levels, intermediate assignments) that minimize a total cost are determined. Each product uses one intermediate composition and a plurality of additive compositions.

The problem can be reformulated as a nonlinear optimization with equality and inequality constraints. The objective function consists of minimizing the portfolio cost, including raw material and some manufacturing and shipment costs across all products. Equality constraints are added to make sure that a sum of a product ingredient fractions is equal to one. The inequality constraints are added to ensure that product quality predictions and product composition variables remain within specified ranges. Two sets of data were used as summarized in Table 4 below.

TABLE 4
	Number of	Number of	Number of	Linear	Nonlinear
	ingredients	premixes	variables	Constraints	Constraints
Dataset	47	53	266	4	1474
1
Dataset	49	52	872	12	7911
2

The computation time results are summarized in Table 5 below.

	TABLE 5
	Problem	GPNO1	122	122 with Q
	Dataset 1	81	187	79
	Dataset 2	210	377	194

From the results presented in Table 5, nonlinear optimization application 122 with quadratic penalty application is faster than nonlinear optimization application 122 without quadratic penalty application. For both datasets, nonlinear optimization application 122 with quadratic extrapolation is over twice as fast as nonlinear optimization application 122 without quadratic penalty application. Table 4 also shows that nonlinear optimization application 122 with quadratic extrapolation is faster in comparison to the nonlinear optimization problem solver GPNO1.

The word “illustrative” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “illustrative” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Further, for the purposes of this disclosure and unless otherwise specified, “a” or “an” means “one or more”. Still further, using “and” or “or” in the detailed description is intended to include “and/or” unless specifically indicated otherwise. The illustrative embodiments may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computer to implement the disclosed embodiments.

The foregoing description of illustrative embodiments of the disclosed subject matter has been presented for purposes of illustration and of description. It is not intended to be exhaustive or to limit the disclosed subject matter to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the disclosed subject matter. The embodiments were chosen and described in order to explain the principles of the disclosed subject matter and as practical applications of the disclosed subject matter to enable one skilled in the art to utilize the disclosed subject matter in various embodiments and with various modifications as suited to the particular use contemplated.

Claims

1. A non-transitory computer-readable medium having stored thereon computer-readable instructions that when executed by a computing device cause the computing device to: (A) perform an optimality check for a current solution to a predefined objective function, wherein the predefined objective function is a nonlinear equation with a plurality of decision variables;(B) when the performed optimality check indicates that the current solution is an optimal solution, output the optimal solution to the predefined objective function, wherein the optimal solution includes a value for each decision variable of the plurality of decision variables that optimizes a result of the predefined objective function;when the performed optimality check indicates that the current solution is not the optimal solution, (C) perform a complementarity slackness test with the current solution;(D) based on the result of the performed complementarity slackness test, update a barrier parameter value and update a Lagrange multiplier value for each constraint function of a plurality of constraint functions defined for objective function optimization, wherein each constraint function applies one or more of the plurality of decision variables;(E) determine a search direction vector by solving a primal-dual linear system that includes a dual variable defined for each constraint function of the plurality of constraint functions, wherein the search direction vector includes a direction value for each decision variable of the plurality of decision variables and for each dual variable of a plurality of dual variables, wherein the search direction vector is determined based on the updated barrier parameter value and the updated Lagrange multiplier value for each constraint function of the plurality of constraint functions;(F) determine a step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated barrier parameter value;(G) update the current solution to the predefined objective function using the determined search direction vector and the determined step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables; and(H) perform an iteration check for the updated current solution;when the performed iteration check indicates that a second set of iterations is needed, (I) determine a second search direction vector by solving the primal-dual linear system using the updated current solution, the updated barrier parameter value, and the updated Lagrange multiplier value for each constraint function of the plurality of constraint functions, wherein the second search direction vector includes the direction value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables;(J) determine a second step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated barrier parameter value, wherein the second step length value is determined for each decision variable of the plurality of decision variables and each dual variable of the plurality of dual variables using quadratic coefficient values;(K) determine a third step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated current solution, the updated barrier parameter value, and the determined second step length value by minimizing a primal-dual merit function;(L) update the current solution to the predefined objective function using the determined second search direction vector and the determined third step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables; and(M) repeat (H) to (L) until the performed iteration check indicates that the second set of iterations is complete; and(N) repeat (A) to (M) until the performed optimality check indicates that the current solution is the optimal solution or a predefined number of iterations of (D) has been performed.

2. The non-transitory computer-readable medium of claim 1, wherein, before a first iteration of (A), the current solution is initialized to a predefined solution starting point that includes values for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables.

3. The non-transitory computer-readable medium of claim 1, wherein, before a first iteration of (A), the barrier parameter value is initialized to a predefined barrier parameter value.

4. The non-transitory computer-readable medium of claim 1, wherein, before a first iteration of (A), the Lagrange multiplier value for each constraint function of the plurality of constraint functions is initialized to a predefined Lagrange multiplier value for each constraint function of the plurality of constraint functions.

5. The non-transitory computer-readable medium of claim 1, wherein the optimality check is performed using

6. The non-transitory computer-readable medium of claim 5, wherein

7. The non-transitory computer-readable medium of claim 1, wherein the complementarity slackness test is performed using ∥C(xk)zk∥≤α∥C(xk−1)zk−1∥, where k is a value of a number of iterations of (C), xk indicates a kth plurality of decision variables, zk indicates a kth plurality of dual variables, C(xk) indicates a first diagonal matrix whose diagonal entries are defined by components of c(xk), c(xk) indicates the plurality of constraint functions solved with xk, a indicates a predefined convergence parameter value, xk−1 indicates a (k−1)th plurality of decision variables, C(xk−1) indicates a first diagonal matrix whose diagonal entries are defined by components of c(xk−i), c(xk−1) indicates the plurality of constraint functions solved with xk−x, and ∥ ∥ indicates a Euclidean norm.

8. The non-transitory computer-readable medium of claim 1, wherein the complementarity slackness test is performed using ∥C(xk)zk∥≤α max {∥C(xij)zij∥:(k−+≤j≤k}+ζk, where k is a value of a number of iterations of (C), xk indicates a kth plurality of decision variables, zk indicates a kth plurality of dual variables, C(xk) indicates a first diagonal matrix whose diagonal entries are defined by components of c(xk), c(xk) indicates the plurality of constraint functions solved with xk, a indicates a predefined convergence parameter value, ∥ ∥ indicates a Euclidean norm, j indicates a first index value defined in (D) based on the result of the performed complementarity slackness test, ij indicates a second index value defined in (D) based on the result of the performed complementarity slackness test, xij indicates an ijth plurality of decision variables, C(xij) indicates the first diagonal matrix whose diagonal entries are defined by components of c(xij), c(xij) indicates the plurality of constraint functions solved with xij, zij indicates an ijth plurality of dual variables, is a predefined integer, t+=max{t,0}, and {ζk} is a first sequence of values that converge to zero.

9. The non-transitory computer-readable medium of claim 8, wherein when ∥C(xk)zk≤α max{∥C(xij)zij∥:(k−)+≤j≤k}+ζk, the updated barrier parameter value is updated using μk+1=qμk, where μk indicates a current barrier parameter value, μk+1 indicates the updated barrier parameter value, q is a predefined multiplier, and the Lagrange multiplier value for each constraint function is updated using λk+1,h=zk,h, h=1, . . . , m, where λk,h indicates a current hth Lagrange multiplier value, λk+1,h indicates the updated hth Lagrange multiplier value, and m is a number of the plurality of constraint functions.

10. The non-transitory computer-readable medium of claim 9, wherein when ∥C(xk)zk∥≤α max{∥C(xij)zij∥:(k−)+≤j≤k}+ζk, ik+1=k, and j=k.

11. The non-transitory computer-readable medium of claim 8, wherein when ∥C(xk)zk∥>α max{∥C(xij)zij∥:(k−)+≤j≤k}+ζk, the updated barrier parameter value is updated μk+1=ρμk, where μk indicates a current barrier parameter value, μk+1 indicates the updated barrier parameter value, and ρ indicates a predefined value, and the Lagrange multiplier value for each constraint function is updated using λk+1,h=λk,h, h=1, . . . , m, where λk,h indicates a current hth Lagrange multiplier value, λk+1,h indicates the updated hth Lagrange multiplier value, and m is a number of the plurality of constraint functions.

12. The non-transitory computer-readable medium of claim 11, wherein 0<ρ<1.

13. The non-transitory computer-readable medium of claim 11, wherein when ∥C(xk)zk∥>α max{∥C(xij)zij∥:(k−)+≤j≤k}+ζk, ik+1=ik, and j=k.

14. The non-transitory computer-readable medium of claim 8, wherein when ∥C(xk)zk∥≤α max{∥C(xij)zij∥:(k−)+≤j≤k}+ζk, the barrier parameter value is reduced, and the updated Lagrange multiplier value for each constraint function of the plurality of constraint functions is unchanged.

15. The non-transitory computer-readable medium of claim 8, wherein, when the performed optimality check indicates that the current solution is not the optimal solution, before the kth iteration of (C), the computer-readable instructions further cause the computing device to update ζk+1 using ζk+1=10μk, where μk indicates the barrier parameter value before the kth iteration of (C).

16. The non-transitory computer-readable medium of claim 1, wherein the current solution to the predefined objective function is updated using {circumflex over (x)}k=xk,h+αk,hxk,hdk,hxk,h and {circumflex over (z)}k=zk,h+αk,hzk,hdk,hzk,h for h=1, . . . , n, where k indicates a value of a number of iterations of (C), {circumflex over (x)}k indicates an updated solution for each decision variable of the plurality of decision variables, {circumflex over (z)}k indicates an updated solution for each dual variable of the plurality of dual variables, xk,h indicates an hth decision variable value of a kth plurality of decision variables, αk,hxk,h indicates the determined step length value for xk,h, dk,hxk,h indicates the determined search direction vector for xk,h, zk,h indicates an hth dual variable value of a kth plurality of dual variables, αk,hzk,h indicates the determined step length value for zk,h, dk,hzk,h indicates the determined search direction vector for zk,h, and n is a number of the one or more decision variables.

17. The non-transitory computer-readable medium of claim 1, wherein the primal-dual linear system is defined by

18. The non-transitory computer-readable medium of claim 1, wherein the step length value is determined for each decision variable of the plurality of decision variables using a backtracking method applied to iterate to a solution such that

19. The non-transitory computer-readable medium of claim 1, wherein the step length value is determined for each dual variable of the plurality of dual variables using a backtracking method applied to iterate to a solution such that zk+αkzdkz≥0.005zk, where k is a counter of iterations of (C), zk indicates a kth plurality of dual variables, αkz indicates a kth step length value for the kth plurality of dual variables, and dkz indicates a kth direction vector for the kth plurality of dual variables.

20. The non-transitory computer-readable medium of claim 1, wherein the primal-dual merit function is

21. The non-transitory computer-readable medium of claim 1, wherein the third step length value is determined for each dual variable of the plurality of dual variables using a backtracking method applied to iterate to a solution that minimizes the primal-dual merit function.

22. The non-transitory computer-readable medium of claim 1, wherein the quadratic coefficient values are computed using

23. The non-transitory computer-readable medium of claim 22, wherein the second step length value is determined for each decision variable of the plurality of decision variables using a backtracking method applied to iterate to a solution such that

24. The non-transitory computer-readable medium of claim 22, if wherein the second step length value is determined for each dual variable of the plurality of dual variables using a backtracking method applied to iterate to a solution such that zk+{tilde over (α)}kzkdzk>0, where k is a counter of iterations of (C), zk indicates a kth plurality of dual variables, {tilde over (α)}kzk indicates a kth step length value for the kth plurality of dual variables, and dkz indicates a kth direction vector for the kth plurality of dual variables.

25. A computing device comprising: a processor; anda computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to (A) perform an optimality check for a current solution to a predefined objective function, wherein the predefined objective function is a nonlinear equation with a plurality of decision variables;(B) when the performed optimality check indicates that the current solution is an optimal solution, output the optimal solution to the predefined objective function, wherein the optimal solution includes a value for each decision variable of the plurality of decision variables that optimizes a result of the predefined objective function;when the performed optimality check indicates that the current solution is not the optimal solution, (C) perform a complementarity slackness test with the current solution;(D) based on the result of the performed complementarity slackness test, update a barrier parameter value and update a Lagrange multiplier value for each constraint function of a plurality of constraint functions defined for objective function optimization, wherein each constraint function applies one or more of the plurality of decision variables;(E) determine a search direction vector by solving a primal-dual linear system that includes a dual variable defined for each constraint function of the plurality of constraint functions, wherein the search direction vector includes a direction value for each decision variable of the plurality of decision variables and for each dual variable of a plurality of dual variables, wherein the search direction vector is determined based on the updated barrier parameter value and the updated Lagrange multiplier value for each constraint function of the plurality of constraint functions;(F) determine a step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated barrier parameter value;(G) update the current solution to the predefined objective function using the determined search direction vector and the determined step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables; and(H) perform an iteration check for the updated current solution;when the performed iteration check indicates that a second set of iterations is needed, (I) determine a second search direction vector by solving the primal-dual linear system using the updated current solution, the updated barrier parameter value, and the updated Lagrange multiplier value for each constraint function of the plurality of constraint functions, wherein the second search direction vector includes the direction value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables;(J) determine a second step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated barrier parameter value, wherein the second step length value is determined for each decision variable of the plurality of decision variables and each dual variable of the plurality of dual variables using quadratic coefficient values;(K) determine a third step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated current solution, the updated barrier parameter value, and the determined second step length value by minimizing a primal-dual merit function;(L) update the current solution to the predefined objective function using the determined second search direction vector and the determined third step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables; and(M) repeat (H) to (L) until the performed iteration check indicates that the second set of iterations is complete; and(N) repeat (A) to (M) until the performed optimality check indicates that the current solution is the optimal solution or a predefined number of iterations of (D) has been performed.

26. A method of determining a solution to a nonlinear optimization problem, the method comprising: (A) performing, by a computing device, an optimality check for a current solution to a predefined objective function, wherein the predefined objective function is a nonlinear equation with a plurality of decision variables;(B) when the performed optimality check indicates that the current solution is an optimal solution, outputting, by the computing device, the optimal solution to the predefined objective function, wherein the optimal solution includes a value for each decision variable of the plurality of decision variables that optimizes a result of the predefined objective function;when the performed optimality check indicates that the current solution is not the optimal solution, (C) performing, by the computing device, a complementarity slackness test with the current solution;(D) based on the result of the performed complementarity slackness test, updating, by the computing device, a barrier parameter value and updating, by the computing device, a Lagrange multiplier value for each constraint function of a plurality of constraint functions defined for objective function optimization, wherein each constraint function applies one or more of the plurality of decision variables;(E) determining, by the computing device, a search direction vector by solving a primal-dual linear system that includes a dual variable defined for each constraint function of the plurality of constraint functions, wherein the search direction vector includes a direction value for each decision variable of the plurality of decision variables and for each dual variable of a plurality of dual variables, wherein the search direction vector is determined based on the updated barrier parameter value and the updated Lagrange multiplier value for each constraint function of the plurality of constraint functions;(F) determining, by the computing device, a step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated barrier parameter value;(G) updating, by the computing device, the current solution to the predefined objective function using the determined search direction vector and the determined step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables; and(H) performing, by the computing device, an iteration check for the updated current solution;when the performed iteration check indicates that a second set of iterations is needed, (I) determining, by the computing device, a second search direction vector by solving the primal-dual linear system using the updated current solution, the updated barrier parameter value, and the updated Lagrange multiplier value for each constraint function of the plurality of constraint functions, wherein the second search direction vector includes the direction value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables;(J) determining, by the computing device, a second step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated barrier parameter value, wherein the second step length value is determined for each decision variable of the plurality of decision variables and each dual variable of the plurality of dual variables using quadratic coefficient values;(K) determining, by the computing device, a third step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables based on the updated current solution, the updated barrier parameter value, and the determined second step length value by minimizing a primal-dual merit function;(L) updating, by the computing device, the current solution to the predefined objective function using the determined second search direction vector and the determined third step length value for each decision variable of the plurality of decision variables and for each dual variable of the plurality of dual variables; and(M) repeating, by the computing device, (H) to (L) until the performed iteration check indicates that the second set of iterations is complete; and(N) repeating, by the computing device, (A) to (M) until the performed optimality check indicates that the current solution is the optimal solution or a predefined number of iterations of (D) has been performed.

27. The method of claim 26, wherein the primal-dual merit function is

28. The method of claim 26, wherein the third step length value is determined for each dual variable of the plurality of dual variables using a backtracking method applied to iterate to a solution that minimizes the primal-dual merit function.

29. The method of claim 26, wherein the quadratic coefficient values are computed using

30. The method of claim 29, wherein the second step length value is determined for each decision variable of the plurality of decision variables using a backtracking method applied to iterate to a solution such that