Patent Application
20250131285
The present invention relates generally to the field of evolutionary neural networks, and more particularly, to a system and a method for carrying out optimized evolutionary neural architecture search.
Evolutionary Neural Architecture Search (NAS) technique is generally carried out using evolutionary algorithms (EAs), which are based on natural evolution, for carrying out a population-based search. The NAS technique maintains a population of solutions, which are evolved using mutation and crossover operations. The mutation operation provides local search (i.e., refinement) and the crossover operation provides a directed global search, which leads to evolution of solutions.
Typically, it has been observed that the evolutionary NAS techniques are limited to carrying out only mutation operations without crossover operations due to ‘permutation problem’, also known as ‘competing conventions problem’. ‘Permutation problem’ arises due to isomorphisms in graph space, i.e., when functionally identical architectures are mapped to different encodings/representations, thereby making crossover operations disruptive. In other words, ‘permutation problem’ arises when the same architecture (i.e., a phenotype) can have multiple distinct genotypes. As a result, crossover on these genotypes has a disruptive effect on the information encoded in parents, leading to damaged child (i.e., offspring). Further, it has been observed that existing techniques either work only on fixed or constrained neural network topologies or are limited to one particular algorithm or search space, and none of the techniques operate on arbitrary graphs or NAS based architectures.
In light of the aforementioned drawbacks, there is a need for a system and a method which provides for an optimized evolutionary neural architecture search. There is a need for a system and a method which provides for application of improved crossover operations by the evolutionary NAS technique.
In various embodiments of the present invention, a system for optimized evolutionary neural architecture search is provided. The system comprises a memory storing program instructions, a processor executing instructions stored in the memory and an evolutionary neural architecture search optimization engine. The evolutionary neural architecture search optimization engine is configured to generate at least two neural network architectures as two parents based on one or more received inputs. The neural network architectures are in the form of organized structures represented as computation graphs. The evolutionary neural architecture search optimization engine computes a similarity between the generated two neural network architectures by computing a Graph Edit Distance (GED) between the neural network architectures corresponding to the computation graphs. One or more graph edit operations are executed for computing the GED. Further, the evolutionary neural architecture search optimization engine carries out a Shortest Edit Path (SEP) crossover operation by analyzing computed edit paths between the graphs. A shuffling operation is carried out randomly for shuffling the edit paths in the SEP between the parents by selecting half of the shuffled edit paths randomly and applying selected edit paths to one of the parents to generate an offspring graph for optimizing Neural Architecture Search (NAS) to solve real-world problems.
In various embodiments of the present invention, a method for optimized evolutionary neural architecture search is provided. The method is implemented by a processor executing instructions stored in the memory. The method comprises generating at least two neural network architectures as two parents based on one or more received inputs. The neural network architectures are in the form of organized structures represented as computation graphs. The method comprises computing a similarity between the generated two neural network architectures by computing a Graph Edit Distance (GED) between the neural network architectures corresponding to the computation graphs. One or more graph edit operations are executed for computing the GED. Further, the method comprises carrying out a Shortest Edit Path (SEP) crossover operation by analyzing computed edit paths between the computation graphs. A shuffling operation is carried out randomly for shuffling the edit paths in the SEP between the two parents by selecting half of the shuffled edit paths randomly and applying selected edit paths to one of the parents to generate an offspring graph for optimizing Neural Architecture Search (NAS) to solve real-world problems.
In various embodiments of the present invention, a computer program product is provided. The computer program product comprises a non-transitory computer-readable medium having computer program code stored thereon, the computer-readable program code comprising instructions that, when executed by a processor, causes the processor to generate at least two neural network architectures as two parents based on one or more received inputs. The neural network architectures are in the form of organized structures represented as computation graphs. A similarity between the generated two neural network architectures is computed by computing a Graph Edit Distance (GED) between the neural network architectures corresponding to the computation graphs. One or more graph edit operations are executed for computing the GED. Further, a Shortest Edit Path (SEP) crossover operation is carried out by analyzing computed edit paths between the graphs. A shuffling operation is carried out randomly for shuffling the edit paths in the SEP between the parents by selecting half of the shuffled edit paths randomly and applying selected edit paths to one of the parents to generate an offspring graph for optimizing Neural Architecture Search (NAS) to solve real-world problems.
The present invention is described by way of embodiments illustrated in the accompanying drawings wherein:
The present invention discloses a system and a method which provides an optimized evolutionary Neural Architecture Search (NAS) by overcoming ‘permutation problem’. The present invention provides for application of a new crossover operator by the evolutionary NAS technique based on a Shortest Edit Path (SEP) in an original graph space. Further, the present invention provides for a system and a method for carrying out NAS without any constraints on encoding or other algorithmic components and can be directly applied to any arbitrary neural architecture.
The disclosure is provided in order to enable a person having ordinary skill in the art to practice the invention. Exemplary embodiments herein are provided only for illustrative purposes and various modifications will be readily apparent to persons skilled in the art. The general principles defined herein may be applied to other embodiments and applications without departing from the scope of the invention. The terminology and phraseology used herein is for the purpose of describing exemplary embodiments and should not be considered limiting. Thus, the present invention is to be accorded the widest scope encompassing numerous alternatives, modifications, and equivalents consistent with the principles and features disclosed herein. For purposes of clarity, details relating to technical material that is known in the technical fields related to the invention have been briefly described or omitted so as not to unnecessarily obscure the present invention.
The present invention would now be discussed in context of embodiments as illustrated in the accompanying drawings.
In an embodiment of the present invention, the system 100 is configured with a built-in-intelligent mechanism for carrying out optimized evolutionary Neural Architecture Search (NAS). In an exemplary embodiment of the present invention, the subsystem 102 is configured to compute a new crossover operator based on a Shortest Edit Path (SEP) crossover technique between at least two neural network architectures (i.e., parents) in an original graph space for carrying out population-based search in NAS. The subsystem 100 is configured to overcome the ‘permutation problem’ for carrying out optimized evolutionary NAS based on the SEP crossover technique. In an embodiment of the present invention, a crossover operator associated with the SEP crossover technique generates an offspring architecture by recombining the two parents. The SEP crossover technique consists of an encoding (i.e., genotype) and a recombination strategy for adequately integrating the data associated with both the parents for generating the offspring, elaborated later in the specification.
In an embodiment of the present invention, the subsystem 102 comprises an evolutionary neural architecture search optimization engine 104 (engine 104), a processor 106 and a memory 108. In various embodiments of the present invention, the engine 104 is configured to carry out optimized evolutionary NAS. The various units of the engine 104 are operated via the processor 106 specifically programmed to execute instructions stored in the memory 108 for executing respective functionalities of the units of the engine 104 in accordance with various embodiments of the present invention.
In an embodiment of the present invention, the engine 104 comprises a neural network architecture generation unit 112, a Graph Edit Distance (GED) computation unit 114, a Short Edit Path (SEP) crossover computation unit 116, an analysis and error computation unit 118, and a verification unit 120.
In an embodiment of the present invention, the neural network architecture generation unit 112 is configured to receive inputs from the input unit 110 for generating at least two neural network architectures as parents. The neural network architectures are in the form of organized structures represented as computation graphs. The computation graph is represented by a directed graph (G). The inputs, therefore, relate to the directed graph, which includes a set of vertices (vi) (i.e., nodes) and a set of directed edges (ei) associated with the directed graph. The order of the directed graph (G) equals the number of its vertices, represented by |G|. For an attributed directed graph, a function γv assigns an attribute (e.g., an integer) to each vertex, and a function γe assigns an attribute to each edge. For the NAS, each vertex with an attribute denotes an operation in a neural architecture, and the directed edges denote data flows.
In an embodiment of the present invention, the GED computation unit 114 is configured to compute similarity between the generated two neural network architectures based on computing a Graph Edit Distance (GED) between neural network architectures corresponding graphs (G1 and G2). In an embodiment of the present invention, the GED computation unit 114 is configured to execute one or more graph edit operations for computing the GED. The graph edit operation comprises application of an elementary graph edit technique which transforms G to G′. which represents an edited graph. In an exemplary embodiment of the present invention, a set of elementary graph edit technique associated with the NAS carries out at least a vertex deletion and insertion operation, an edge deletion and insertion operation, or a vertex attribute substitution operation. In an exemplary embodiment of the present invention, the graph edit operation is associated with computing an edit path, and is a sequence of graph edit operations. Further, GED between the two graphs (G1 and G2) associated with the neural network architectures relates to the set of all edit paths that transform the computation graph G1 to G2 or isomorphisms of G2. In an embodiment of the present invention, all the edit operations have a value of 1. Therefore, the edit path that minimizes the total edit value comprises the SEP between G1 and G2. In an embodiment of the present invention, multiple SEP may be computed between G1 and G2 having same length.
In an embodiment of the present invention, the SEP crossover computation unit 116 is configured to analyze the computed edit paths between the two graphs (G1 and G2) associated with neural network architectures for carrying out SEP crossover operation. The SEP crossover computation unit 116 is configured to carry out a shuffling operation for randomly shuffling the edit paths in the SEP between parents, as illustrated in
In an embodiment of the present invention, the analysis and error computation unit 118 is configured to generate an Attributed Adjacency matrix (AA-matrix) (AG), as represented below, for analysis of efficiency of SEP crossover technique by communicating with the SEP crossover computation unit 116. Further, the analysis and error computation unit 118 is configured to generate an Attributed Adjacency matrix (AA-matrix) (AG) for a mutation technique and subsequently the AA-matrix for both the SEP technique and mutation technique are compared for determining improvement of the SEP technique with respect to the mutation technique, as elaborated later in the specification. The AA-matrix represents the attributed directed graph (G). The AA-matrix is a ‘n×n’ matrix, where ‘n’ represents number of vertices (v) in ‘G’. The diagonal entries in the AA-matrix represents node attributes and non-diagonal entries in the AA-matrix represents edges. The entry in ith row and jth column is represented by AGi,j. As such, AGi,j=0, if there is no edge from vi to vj and, AGi,j=γe(ei,j), if there exists an edge from vi to vj for i,j∈1, 2 . . . , n and i≠j. AGi,j=γv(vi), for i∈1, 2 . . . , n.
In an embodiment of the present invention, the analysis and error computation unit 118 is configured to analyze SEP crossover technique by utilizing the generated AA-matrix. The analysis and error computation unit 118 is configured to firstly process the two computation graphs G1 and G2. G1 and G2 have a same order which is determined by adding null vertices resulting in G{circumflex over ( )}1 and G{circumflex over ( )}2. A crossover operation is carried out between G1 and G2 for generating a new offspring graph (Gnew) by recombining AG{circumflex over ( )}1 and AG{circumflex over ( )}2 as AG{circumflex over ( )}new=r (AG{circumflex over ( )}1, Pπ AG{circumflex over ( )}2 PπT), where AGA and AG{circumflex over ( )}2 is AA-matrix of G{circumflex over ( )}1 and G{circumflex over ( )}2, Pπ is a permutation matrix based on permutation π which is decided by the specific crossover operator utilized and function r(A,B) returns a matrix that inherits each entry from A or B with probability 0.5. For the SEP crossover technique, permutation is defined as π=π*G{circumflex over ( )}1,G{circumflex over ( )}2, which minimizes the GED between G1 and G2. In an example, for the standard crossover technique, since the vertices may be in any order in the original AA-matrix representation and there is no particular vertex/edge matching mechanisms during crossover, a random permutation πrand is used to represent the randomness. In an embodiment of the present invention, the analysis and error computation unit 118 therefore generates an AA-matrix (AG{circumflex over ( )}new) of the generated new graph with null vertices associated with the SEP crossover technique. By removing all null vertices from G{circumflex over ( )}new (new graph with null vertices), the crossover offspring graph is generated by the analysis and error computation unit 118.
In another embodiment of the present invention, the analysis and error computation unit 118 is configured to analyze mutation technique by utilizing the generated AA-matrix. The analysis and error computation unit 118 is configured to determine mutation operation by generating the new offspring graph (Gnew) by mutating the graph G1. In the AA-matrix representation, a mutation operation is then defined as AG{circumflex over ( )}new=m(AG{circumflex over ( )}1), where function m(A) alters each element of A with an equal probability pm, and pm is usually selected so that on average one element is altered during each mutation operation. The G{circumflex over ( )}1 is the extended graph of G1 with null vertices so that node additions can be performed in AG{circumflex over ( )}1. An element AG{circumflex over ( )}1i,j can be altered in order to randomly resample an allowed value that is different from the original AG{circumflex over ( )}1i,j. The result AG new, is the AA-matrix of the generated new graph with null vertices. By removing all null vertices from G{circumflex over ( )}new, the mutated new offspring graph Gnew is generated by the analysis and error computation unit 118.
In an embodiment of the present invention, the analysis and error computation unit 118 is configured to compute a main performance metric for SEP crossover and mutation operations for determining efficiency of SEP crossover and mutation operations based on a topological similarity to a global optimal graph. The expected improvement is referred to as E(max(de(AG{circumflex over ( )}opt, AG{circumflex over ( )}1 to G{circumflex over ( )}opt)−de(AG{circumflex over ( )}opt, AG{circumflex over ( )}1 to G{circumflex over ( )}opt),0)), which compares the new offspring graph (Gnew) with one parent graph G1 in terms of the expected edge differences to Gopt. Further, the max(., 0) part takes into account the selection pressure in standard EAs, that is, only the offspring that is better than its parent can survive and become the next parent. Further, the analysis and error computation unit 118 is configured to determine improvement of the SEP crossover by computing the below mentioned formula and the entries that are present in different AA-matrix and the permuted AA-matrix are assumed to be uniformly distributed on the positions other than those ns (number of common entries among multiple matrices) common entries:
let nse=max (n·(n−1)−d*e, G{circumflex over ( )}opt,G{circumflex over ( )}1−d*e, G{circumflex over ( )}1,G{circumflex over ( )}2,0), where nse is number of common non-diagonal entries among multiple matrices and suppose AG{circumflex over ( )}new=r(AG{circumflex over ( )}1, Pπ*G{circumflex over ( )}1,G{circumflex over ( )}2 AG{circumflex over ( )}1, PπT*G{circumflex over ( )}1,G{circumflex over ( )}2). Then E(max(de(AG{circumflex over ( )}opt, AG{circumflex over ( )}1 to G{circumflex over ( )}opt)−de(AG{circumflex over ( )}opt, AG{circumflex over ( )}new to G{circumflex over ( )}opt), 0))≥E(max(d*e, G{circumflex over ( )}opt, G{circumflex over ( )}1−d*e, G{circumflex over ( )}1,G{circumflex over ( )}2−B(d*e, G{circumflex over ( )}1,G{circumflex over ( )}2, 0.5),0))=LBEISEPX
n·(n−1)−nse
where B (d*e, G{circumflex over ( )}1,G{circumflex over ( )}2, 0.5),0) denotes the number of successful trials after sampling from a binomial distribution with d*e, G{circumflex over ( )}1,G{circumflex over ( )}2 trials and success probability of 0.5, and LBEISEPX denotes the lower bound of expected improvement of the SEP crossover.
Further, the analysis and error computation unit 118 is configured to determine improvement of the mutation operation by computing the below mentioned formula:
Suppose AG{circumflex over ( )}new=m(AG{circumflex over ( )}1), then E(max(de(AG{circumflex over ( )}opt, AG{circumflex over ( )}1 to G{circumflex over ( )}opt)−de(AG{circumflex over ( )}opt, AG{circumflex over ( )}new to G{circumflex over ( )}opt), 0))≥E(max(d*e, G{circumflex over ( )}opt,G{circumflex over ( )}1−B(n·(n−1)−d*e, G{circumflex over ( )}opt,G{circumflex over ( )}1,pm)−B(d*e, G{circumflex over ( )}opt,G{circumflex over ( )}1, 1−pm),0))=LBEIMUTA, where pm is the mutation rate usually chosen to be pm=1/n. (n−1) and LBEIMUTA denotes the lower bound of expected improvement of mutation.
In an embodiment of the present invention, the analysis and error computation unit 118 is configured to compute error during the GED determination between two neural network architectures extended graphs G{circumflex over ( )}1, G{circumflex over ( )}2. The error is computed based on the below mentioned formula:
where ∈ (error ratio)>0 is the error ratio and d∈e, G{circumflex over ( )}1, G{circumflex over ( )}2 is the expectation of GED computation result. The analysis and error computation unit 118 is configured to determine effect of GED errors on LBEISPEX and it is determined that the SEP crossover has an advantage in expected improvement compared to mutation and a standard crossover even with a very high error ratio (∈) of 30% in the GED calculations.
In an embodiment of the present invention, the verification unit 120 is configured to verify that parameter values, as elaborated above, used in analysis of the SEP crossover technique apply to real-world problems. The verification unit 120 is configured to firstly, verify that parameter values used in a Monte Carlo simulation lie within favorable regions in real NAS problems, prior to verifying that the parameter values used in analysis of the SEP crossover technique apply to real-world problems. The parameter values used for d*e,G{circumflex over ( )}opt, G{circumflex over ( )}1, d*e,G{circumflex over ( )}1, G{circumflex over ( )}2, n11, n12 are critical for the expected improvement and are verified in standard NAS benchmarks and with a standard NAS algorithm. The verification unit 120 is configured to apply the standard NAS benchmarks including a NAS-bench-101 and a NAS-bench-NLP to evaluate the parameter values used in the SEP crossover technique. In an embodiment of the present invention, the verification unit 120, firstly, evaluates the SEP crossover technique with the standard NAS technique by incorporating the SEP crossover technique into a Regularized Evolution (RE) method. As such, the RE method employs only a mutation operator and SEP crossover is integrated into the RE technique by alternating crossover with mutation. The verification unit 120, secondly, determines the parameter values by executing the RE on both the benchmarks (i.e., the NAS-bench-101 and the NAS-bench-NLP) and recording the relative frequency distributions of the above parameters. In an exemplary embodiment of the present invention,
The newly developed SEP crossover technique, in accordance with various embodiments of the present invention, is employed in real-world NAS problems for determining search efficiency of the SEP crossover technique in a noise-free environment and a noisy environment. In one experiment, the search efficiency of the SEP crossover technique was determined in a noise-free environment. The evaluation step in NAS, i.e., the training and testing of an architecture, can be very noisy. In order to determine the search efficiency of the SEP crossover technique without the effects of such noise, a noise-free evaluation function was first employed as the GED between a candidate architecture and a target architecture. The global optimum was selected as the target in NAS-bench-101, while Gated Recurrent Unit (GRU) and Long-Short Term Memory (LSTM) models were used as targets in NAS-bench-NLP. As the NAS-bench-NLP is not queryable, the global optimum is unknown. However, GRU and LSTM are two known top-performing models in this search space and can therefore be used as a proxy for the global optimum.
In another experiment, the search efficiency of the SEP crossover technique was determined in a noisy environment. The robustness of the SEP crossover was evaluated by applying the SEP crossover to NAS problems with noisy evaluations. Noise was caused from two sources: (1) the direct fitness/reward, e.g., the validation accuracy, used for search strategy was noisy; (2) mapping between the final objective e.g., the test accuracy, and direct fitness/reward was noisy. Further, the validation accuracy in NAS-bench-101, which consists of random sampling of three real-world training trials was used as the direct fitness/reward. The average test accuracy in NAS-bench-101 was used as the final objectives.
At step 702, inputs are received for generating at least two neural network architectures as parents. In an embodiment of the present invention, the neural network architectures are in the form of organized structures represented as computation graphs. The computation graph is represented by a directed graph (G). The inputs, therefore, relate to the directed graph, which includes a set of vertices (vi) (i.e., nodes) and a set of directed edges (ei) associated with the directed graph. The order of the directed graph (G) equals the number of its vertices, represented by |G|. For an attributed directed graph, a function γv assigns an attribute (e.g., an integer) to each vertex, and a function γe assigns an attribute to each edge. For the NAS, each vertex with an attribute denotes an operation in a neural architecture, and the directed edges denote data flows.
At step 704, similarity between the generated two neural network architectures is computed by computing a Graph Edit Distance (GED). In an embodiment of the present invention, the similarity is computed by computing a Graph Edit Distance (GED) between neural network architectures corresponding graphs (G1 and G2). In an embodiment of the present invention, one or more graph edit operations are executed for computing the GED. The graph edit operation comprises application of an elementary graph edit technique which transforms G to G′ which represents an edited graph. In an exemplary embodiment of the present invention, a set of elementary graph edits technique associated with the NAS carries out at least a vertex deletion and insertion operation, an edge deletion and insertion operation, or a vertex attribute substitution operation. In an exemplary embodiment of the present invention, the graph edit operation is associated with computing an edit path, and is a sequence of graph edit operations. GED between the two graphs (G1 and G2) associated with the neural network architectures relates to the set of all edit paths that transform computation graph G1 to G2 or isomorphisms of G2. In an embodiment of the present invention, all the edit operations have a value of 1. Therefore, the edit path that minimizes the total edit value comprises the SEP between G1 and G2. In an embodiment of the present invention, multiple SEP may be computed between G1 and G2 having same length.
At step 706, Shortest Edit Path (SEP) crossover operation is carried out. In an embodiment of the present invention, the computed edit paths between the two graphs (G1 and G2) associated with neural network architectures are analyzed for carrying out SEP crossover operation. A shuffling operation is carried out for randomly shuffling the edit paths in the SEP between parents. SEP crossover operation is carried out by selecting half of the shuffled edit paths randomly and thereafter applying the selected edit paths to one of the parents to generate a child graph (i.e., an offspring graph). The two parent architectures share vertices ‘A’ and ‘B’. Although the two vertices ‘A’ and ‘B’ appear in a different order, together they implement the same function, and this function should not be disrupted during crossover. As such, the standard crossover technique cannot identify the subgraph isomorphism, and it loses this substructure (disruptive structure). However, the SEP computation recognizes the isomorphism, and as a result, the SEP crossover preserves this substructure (preserving structure). Thus, the SEP crossover technique analyses the parts that are functionally inconsistent between the two parents and is therefore employed to generate an offspring graph.
At step 708, an Attributed Adjacency matrix is generated for analysis of SEP crossover and mutation technique. In an embodiment of the present invention, the Attributed Adjacency matrix (AA-matrix) (AG), as represented below, is generated for analysis of efficiency of SEP crossover technique. Further, an Attributed Adjacency matrix (AA-matrix) (AG) is generated for a mutation technique and subsequently the AA-matrix for both the SEP technique and mutation technique are compared for determining improvement of SEP technique with respect to the mutation technique. The AA-matrix represents the attributed directed graph (G). The AA-matrix is a ‘n×n’ matrix, where ‘n’ represents number of vertices (v) in ‘G’. The diagonal entries in the AA-matrix represents node attributes and non-diagonal entries in the AA-matrix represents edges. The entry in ith row and jth column is represented by AGi,j. As such, AGi,j=0, if there is no edge from vi to vj and, AGi,j=γe(ei,j), if there exists an edge from vi to vj for i,j∈1, 2 . . . , n and i≠j. AGi,j=γv (vi), for i∈1, 2 . . . , n.
In an embodiment of the present invention, SEP crossover technique employs the generated AA-matrix for analysis. Firstly, the two computation graphs G1 and G2 are processed. G1 and G2 have a same order which is determined by adding null vertices resulting in G{circumflex over ( )}1 and G{circumflex over ( )}2. A crossover operation between G1 and G2 for generating a new offspring graph (Gnew) is carried out by recombining AG{circumflex over ( )}1 and AG{circumflex over ( )}2 as AG{circumflex over ( )}new=r (AG{circumflex over ( )}1, Pπ AG{circumflex over ( )}2 PπT), where AG{circumflex over ( )}1 and AG{circumflex over ( )}2 is AA-matrix of G{circumflex over ( )}1 and G{circumflex over ( )}2, Pπ is a permutation matrix based on permutation π which is decided by the specific crossover operator employed and function r(A,B) returns a matrix that inherits each entry from ‘A’ or ‘B’ with probability 0.5. For the SEP crossover, permutation π is defined as π=π*G{circumflex over ( )}1,G{circumflex over ( )}2, which minimizes the GED between G1 and G2. In an example, for the standard crossover technique, since the vertices may be in any order in the original AA-matrix representation and there are no particular vertex/edge matching mechanisms during crossover, a random permutation πrand is used to represent the randomness. In an embodiment of the present invention, an AA-matrix (AG{circumflex over ( )}new) of the generated new graph is generated with null vertices associated with the SEP crossover technique. By removing all null vertices from G{circumflex over ( )}new, the crossover offspring graph is generated.
In another embodiment of the present invention, mutation technique is analyzed by employing the generated AA-matrix. Mutation operation is determined based on generating a new offspring graph (Gnew) by mutating the graph G1. In the AA-matrix representation, a mutation operation is defined as AG{circumflex over ( )}new=m (AG{circumflex over ( )}1), where function m(A) alters each element of A with an equal probability pm, and pm is usually selected so that on average one element is altered during each mutation operation. The G{circumflex over ( )}1 is the extended graph of G1 with null vertices so that node additions can be performed in AG{circumflex over ( )}1. An element AG{circumflex over ( )}1i,j can be altered in order to randomly resample an allowed value that is different from the original AG{circumflex over ( )}1i,j. The result AG{circumflex over ( )}new, is the AA-matrix of the generated new graph with null vertices. By removing all null vertices from G{circumflex over ( )}new, the mutated new offspring graph Gnew is generated.
At step 710, a main performance metric is computed for SEP crossover and mutation operations for determining improvement of SEP crossover and mutation operations. In an embodiment of the present invention, the main performance metric is computed based on a topological similarity to the global optimal graph. The expected improvement is referred to as E(max(de(AG{circumflex over ( )}opt, AG{circumflex over ( )}1 to G{circumflex over ( )}opt)−de(AG{circumflex over ( )}opt, AG{circumflex over ( )}1 to G{circumflex over ( )}opt),0)), which compares the new offspring graph (Gnew) with one parent graph G1 in terms of the expected edge differences to Gopt. Further, the max(., 0) part takes into account the selection pressure in standard EAs, that is, only the offspring that is better than its parent can survive and become the next parent. Improvement of the SEP crossover is determined by computing the below mentioned formula and the entries that are present in different AA-matrix. The permuted AA-matrix are assumed to be uniformly distributed on the positions other than those ns (number of common entries among multiple matrices) common entries:
let nse=max(n·(n−1)−d*e, G{circumflex over ( )}opt,G{circumflex over ( )}1−d*e, G{circumflex over ( )}1,G{circumflex over ( )}2,0), where nse is number of common non-diagonal entries among multiple matrices and suppose AG{circumflex over ( )}new=r(AG{circumflex over ( )}1, Pπ*G{circumflex over ( )}1,G{circumflex over ( )}2 AG{circumflex over ( )}1, PπT*G{circumflex over ( )}1,G{circumflex over ( )}2). Then E(max(de(AG{circumflex over ( )}opt, AG{circumflex over ( )}1 to G{circumflex over ( )}opt)−de(AG{circumflex over ( )}opt, AG{circumflex over ( )}new to G{circumflex over ( )}opt), 0))≥E(max(d*e, G{circumflex over ( )}opt, G{circumflex over ( )}1−d*e, G{circumflex over ( )}1,G{circumflex over ( )}2−B(d*e, G{circumflex over ( )}1,G{circumflex over ( )}2, 0.5),0))=LBEISE
n·(n−1)−nse
where B(d*e, G{circumflex over ( )}1,G{circumflex over ( )}2, 0.5),0) denotes the number of successful trials after sampling from a binomial distribution with d*e, G{circumflex over ( )}1,G{circumflex over ( )}2 trials and success probability of 0.5, and LBEISEPX denotes the lower bound of expected improvement of the SEP crossover.
Further, the improvement of the mutation operation is determined based on computing the below mentioned formula:
Suppose AG{circumflex over ( )}new=m(AG{circumflex over ( )}1), then E(max(de(AG{circumflex over ( )}opt, AG{circumflex over ( )}1 to G{circumflex over ( )}opt)−de(AG{circumflex over ( )}opt, AG{circumflex over ( )}new to G{circumflex over ( )}opt), 0))≥E(max(d*e, G{circumflex over ( )}opt,G{circumflex over ( )}1−B(n·(n−1)−d*e, G{circumflex over ( )}opt,G{circumflex over ( )}1,pm)−B(d*e, G{circumflex over ( )}opt,G{circumflex over ( )}1, 1−pm),0))=LBEIMUTA, where pm is the mutation rate usually chosen to be pm=1/n. (n−1) and LBEIMUTA denotes the lower bound of expected improvement of mutation.
At step 712, effect of GED errors is determined on SEP crossover operation. In an embodiment of the present invention, error is computed during the GED determination between two neural network architecture extended graphs G{circumflex over ( )}1, G{circumflex over ( )}2. The error is computed based in the below mentioned formula:
where ∈ (error ratio)>0 is the error ratio and d∈e,G{circumflex over ( )}1,G{circumflex over ( )}2 is the expectation of GED computation result. Effect of GED errors are determined on LBEISPEX and it is ascertained that the SEP crossover has an advantage in expected improvement compared to mutation and a standard crossover even with a very high error ratio (∈) of 30% in the GED calculations.
At step, 714, parameter values which are used in analysis of the SEP crossover technique are verified that they apply to real-world problems. In an embodiment of the present invention, firstly, it is verified that parameter values used in a Monte Carlo simulation lie within favorable regions in real NAS problems, prior to verifying that the parameter values used in analysis of the SEP crossover technique apply to real-world problems. The parameter values used for d*e,G{circumflex over ( )}opt, G{circumflex over ( )}1, d*e,G{circumflex over ( )}1, G{circumflex over ( )}2, n11, n12 are critical for the expected improvement and are verified in standard NAS benchmarks with a standard NAS algorithm. The standard NAS benchmarks apply a NAS-bench-101 and a NAS-bench-NLP to evaluate the parameter values used in the SEP crossover technique. In an embodiment of the present invention, firstly, the SEP crossover technique is evaluated with the standard NAS algorithm by incorporating the SEP crossover technique into a Regularized Evolution (RE) method. The RE method employs only a mutation operator and SEP crossover is integrated into the RE technique by alternating crossover with mutation. Secondly, the parameter values are determined by executing the RE on both the benchmarks (i.e., the NAS-bench-101 and the NAS-bench-NLP) and recording the relative frequency distributions of the above parameters.
Advantageously, in accordance with various embodiments of the present invention, the present invention provides for a system and method for carrying out optimized evolutionary neural architecture search by employing the new SEP crossover technique. The present invention provides for effectively resolving the ‘permutation problem’ associated with existing NAS techniques, thereby providing an improved NAS system and method. The present invention provides for carrying out NAS without any constraints on encoding or other algorithmic components and can be directly applied to any arbitrary neural architectures. Further, the present invention provides for robust and effective application of SEP crossover in realistic noisy environments as well. Furthermore, the present invention provides for reducing the computational costs during the NAS operations.
The communication channel(s) 808 allow communication over a communication medium to various other computing entities. The communication medium provides information such as program instructions, or other data in a communication media. The communication media includes, but not limited to, wired or wireless methodologies implemented with an electrical, optical, RF, infrared, acoustic, microwave, Bluetooth or other transmission media.
The input device(s) 810 may include, but not limited to, a keyboard, mouse, pen, joystick, trackball, a voice device, a scanning device, touch screen or any another device that is capable of providing input to the computer system 802. In an embodiment of the present invention, the input device(s) 810 may be a sound card or similar device that accepts audio input in analog or digital form. The output device(s) 812 may include, but not limited to, a user interface on CRT or LCD, printer, speaker, CD/DVD writer, or any other device that provides output from the computer system 802.
The storage 814 may include, but not limited to, magnetic disks, magnetic tapes, CD-ROMs, CD-RWs, DVDs, flash drives or any other medium which can be used to store information and can be accessed by the computer system 802. In various embodiments of the present invention, the storage 814 contains program instructions for implementing the described embodiments.
The present invention may suitably be embodied as a computer program product for use with the computer system 802. The method described herein is typically implemented as a computer program product, comprising a set of program instructions which is executed by the computer system 802 or any other similar device. The set of program instructions may be a series of computer readable codes stored on a tangible medium, such as a computer readable storage medium (storage 814), for example, diskette, CD-ROM, ROM, flash drives or hard disk, or transmittable to the computer system 802, via a modem or other interface device, over either a tangible medium, including but not limited to optical or analogue communications channel(s) 808. The implementation of the invention as a computer program product may be in an intangible form using wireless techniques, including but not limited to microwave, infrared, Bluetooth or other transmission techniques. These instructions can be preloaded into a system or recorded on a storage medium such as a CD-ROM, or made available for downloading over a network such as the internet or a mobile telephone network. The series of computer readable instructions may embody all or part of the functionality previously described herein.
The present invention may be implemented in numerous ways including as a system, a method, or a computer program product such as a computer readable storage medium or a computer network wherein programming instructions are communicated from a remote location.
While the exemplary embodiments of the present invention are described and illustrated herein, it will be appreciated that they are merely illustrative. It will be understood by those skilled in the art that various modifications in form and detail may be made therein without departing from or offending the scope of the invention.
