Hybrid and Hierarchical Multi-Trial and OneShot Neural Architecture Search on Datacenter Machine Learning Accelerators

Abstract

According to various implementations, generally disclosed herein is a hybrid and hierarchical neural architecture search (NAS) approach. The approach includes performing a search space partitioning scheme to divide the search space into sub-search spaces. The approach further includes performing a first type of NAS, such as a Multi-trial NAS, to cover a search across the sub-search spaces. The approach also includes performing a second type of NAS, such as a One-Shot NAS, to cover each sub-search space. The approach further includes automatically stopping the second type of NAS based on one or more early stopping criteria.

Description

Claims

1. A method for performing a neural architecture search (NAS), the method comprising: partitioning, with one or more processors, a search space into a plurality of sub-search spaces;performing, with the one or more processors, a first type of NAS across the plurality of sub-search spaces;performing, with the one or more processors, a second type of NAS within each sub-search space based on the performance of the first type of NAS; andstopping, with the one or more processors, the second type of NAS based on an early stopping criterion.

2. The method of claim 1, wherein the first type of NAS is Multi-trial and the second type of NAS is One-Shot.

3. The method of claim 1, wherein partitioning the search space further comprises partitioning the search space based on one or more hyperparameters that influence the search space.

4. The method of claim 3, wherein: partitioning the search space further comprises selecting a principal model architecture parameter as a dimension for the first type of NAS; andperforming a second type of NAS further comprises searching for a remainder of model architecture dimensions.

5. The method of claim 1, wherein partitioning the search space further comprises: computing a size of the search space based on a capacity influenced by the search space; andautomatically partitioning the search space based on the computed size.

6. The method of claim 5, wherein the capacity influenced by the search space further comprises one of a machine learning hardware memory capacity, compute throughput, memory or network bandwidth, or power.

7. The method of claim 1, wherein partitioning the search space further comprises partitioning the search space based on one or more hyperparameters that influence at least one of a quality or efficiency of machine learning model results.

8. The method of claim 7, wherein performing the first type of NAS further comprises searching for one of a compiler flag or model hyperparameter.

9. The method of claim 1, further comprising monitoring, with the one or more processors, an early stopping criterion of the second type of NAS.

10. The method of claim 1, wherein the early stopping criterion comprises one or more of an architecture searchable parameter approaching a convergence, a quality threshold, a threshold amount of data consumed, or a convergence rate threshold.

11. A system comprising: one or more processors; andone or more storage devices coupled to the one or more processors and storing instructions, when performed by the one or more processors, causes the one or more processors to perform operations for performing a neural architecture search (NAS), the operations comprising: partitioning a search space into a plurality of sub-search spaces;performing a first type of NAS across the plurality of sub-search spaces;performing a second type of NAS within each sub-search space based on the performance of the first type of NAS; andstopping the second type of NAS based on an early stopping criterion.

12. The system of claim 11, wherein the first type of NAS is Multi-trial and the second type of NAS is One-Shot.

13. The system of claim 11, wherein partitioning the search space further comprises partitioning the search space based on one or more hyperparameters that influence the search space.

14. The system of claim 13, wherein: partitioning the search space further comprises selecting a principal model architecture parameter as a dimension for the first type of NAS; andperforming a second type of NAS further comprises search for a remainder of model architecture dimensions.

15. The system of claim 11, wherein partitioning the search space further comprises: computing a size of the search space based on a capacity influenced by the search space; andautomatically partitioning the search space based on the computed size.

16. The system of claim 15, wherein the capacity influenced by the search space further comprises one of a machine learning hardware memory capacity, compute throughput, memory or network bandwidth, or power.

17. The system of claim 11, wherein partitioning the search space further comprises partitioning the search space based on one or more hyperparameters that influence at least one of quality or efficiency of machine learning model results.

18. The system of claim 17, wherein performing the first type of NAS further comprises search for one of a compiler flag or a model hyperparameter.

19. The system of claim 11, wherein the early stopping criterion comprises one or more of an architecture searchable parameter approaching a convergence, a quality threshold, a threshold amount of data consumed, or a convergence rate threshold.

20. A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, causes the one or more processors to perform operations for performing a neural architecture search (NAS), the operations comprising: partitioning a search space into a plurality of sub-search spaces;performing a first type of NAS across the plurality of sub-search spaces;performing a second type of NAS within each sub-search space based on the performance of the first type of NAS; andstopping the second type of NAS based on an early stopping criterion.