Patent Application
20210350234
Machine learning includes the study and construction of algorithms that can learn from and make predictions on data. Deep neural networks may implement algorithms to perform a type of machine learning referred to as deep learning. Typically, deep learning may utilize a cascade of many layers of artificial neurons, or operators, such as nonlinear processing units. Frequently, each successive layer, or operator, uses the output of the previous layer as input. Collectively, the artificial neurons may perform feature extraction and transformation with deep learning algorithms. Deep learning may include supervised and unsupervised algorithms. Generally, unsupervised algorithms are used for pattern analysis and supervised algorithms are used for pattern classification.
Various embodiments are generally directed to techniques to detect fusible operators with machine learning, such as by evaluating a set of operators in a graph of a machine learning model to identify fusion candidates comprising subgraphs of the graph with two or more operators to combine, for instance. Some embodiments are particularly directed to utilizing a machine learning classifier to evaluate fusion candidates using a set of features of the fusion candidate. In one embodiment, for example, an apparatus may comprise a processor and a memory comprising instructions that when executed by the processor cause the processor to perform one or more of the following. In some embodiments, the processor may identify input comprising one or more machine learning models that each include a graph of operators. In various embodiments, the processor may mine the one or more machine learning models based on one or more operational parameters to determine one or more fusion candidates. In various such embodiments, each of the one or more fusion candidates may include a subgraph of at least one graph of operators corresponding to a machine learning model, and each subgraph may include two or more operators as candidates to combine. In many embodiments, the processor may extract a feature set from each of the one or more fusion candidates. In several embodiments, the processor may utilize a machine learning classifier to evaluate the one or more fusion candidates based on the feature sets extracted from each of the one or more fusion candidates. In some embodiments, the processor may provide, as output, a proposed candidate of the one or more fusion candidates to fuse based on evaluation of the one or more fusion candidates. These and other embodiments are described and claimed.
Some challenges facing machine learning include limited operational efficiencies, such as in deep neural network models. Such challenges may arise from an inability to accurately identify and combine operators, or layers, in machine learning models without extensive manual intervention. The need for manual intervention results in issues such as limited coverage, inconsistencies, and excessive lag. For example, humans can only find a small fraction of valuable fusible operators, resulting in a large number of fusible operators with higher frequency and heavier computation cost being missed. In another example, hand-crafted fusible operator detection relies heavily on the skillfulness of the developer and their understanding of the usage domain, leading to inconsistent performance. In yet another example, the deep learning industry is quickly evolving with new operators and operator combinations being continually developed, making it difficult or impossible for manual interventions to stay relevant. These and other factors may result is machine learning models with excessive overhead, limited applicability, and poor adaptability. Such limitations can drastically reduce the usability and performance of machine learning models, contributing to inefficient machine learning models.
Various embodiments described herein include a fusible operator detector (FOD), such as in an inference framework, to increase the efficiency of machine learning models. In various such embodiments, the FOD may analyze machine learning models that include machine learning topologies and/or graphs to identify fusible operators and/or fusible operator patterns (e.g., fusion candidates). Sometimes, the fusion candidates may be provided as output for use in further inference jobs and/or improved machine learning models. In many embodiments, the FOD may utilize data-driven and/or machine learning techniques to identify fusion candidates with better coverage and improved consistency than techniques that require extensive manual intervention. For instance, the FOD may efficiently and automatically identify two or more operators, or layers, in a deep neural network model to combine using a machine learning classifier.
Further, the FOD can find new fusible operators quickly and accurately to evolve with the evolution of topologies used in machine learning a while coping with the fast evolution of the machine learning industry. In these and other ways, components described here may identify methods to increase efficiency, decrease performance costs, decrease computational cost, and/or reduce resource requirements to implement machine learning models, in an accurate, robust, efficient, dynamic, and scalable manner, resulting in several technical effects and advantages over conventional computer technology, including increased capabilities and improved adaptability. In various embodiments, one or more of the components may be implemented in a practical application via one or more computing devices, and thereby provide additional and useful functionality to the one or more computing devices, resulting in more capable, better functioning, and improved computing devices. In many embodiments, one or more aspects of fusible operator detection described herein may be implemented via familiar, user-friendly interface objects.
In several embodiments, components described herein may provide specific and particular manners of automatically detecting and/or evaluating the fusibility of two or more operators in a machine learning model. In many embodiments, one or more of the components described herein may be implemented as a set of rules that improve computer-related technology by allowing a function not previously performable by a computer that enables an improved technological result to be achieved. For example, the function allowed may include automatic fusible operator detection and/or evaluation in machine learning models. In some examples, the function allowed may include fusible operator detection and/or evaluation in machine learning models using machine learning classifiers. In numerous examples, the function allowed may include fusible operator detection and/or evaluation in machine learning models using a set of features extracted from fusion candidates. In many examples, the function allowed may include providing a set of features extracted from a fusion candidate as input to a machine learning classifier to evaluate the fusion candidate.
With general reference to notations and nomenclature used herein, one or more portions of the detailed description which follows may be presented in terms of program procedures executed on a computer or network of computers. These procedural descriptions and representations are used by those skilled in the art to most effectively convey the substances of their work to others skilled in the art. A procedure is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It proves convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be noted, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to those quantities.
Further, these manipulations are often referred to in terms, such as adding or comparing, which are commonly associated with mental operations performed by a human operator. However, no such capability of a human operator is necessary, or desirable in most cases, in any of the operations described herein that form part of one or more embodiments. Rather, these operations are machine operations. Useful machines for performing operations of various embodiments include general purpose digital computers as selectively activated or configured by a computer program stored within that is written in accordance with the teachings herein, and/or include apparatus specially constructed for the required purpose. Various embodiments also relate to apparatus or systems for performing these operations. These apparatuses may be specially constructed for the required purpose or may include a general-purpose computer. The required structure for a variety of these machines will be apparent from the description given.
Reference is now made to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding thereof. It may be evident, however, that the novel embodiments can be practiced without these specific details. In other instances, well known structures and devices are shown in block diagram form to facilitate a description thereof. The intention is to cover all modification, equivalents, and alternatives within the scope of the claims.
In some embodiments, FOD 105 may include an automatic fusible operator detection module integrated into an inference framework. Accordingly, in various embodiments described herein, an inference frame work may include one or more of fusible operator detection, construction of a fused topology, and make inferences with the fused topology. In many embodiments, FOD 105 may analyze the machine learning models 104 provided in input 102. In various embodiments, machine learning models 104 may include one or more graphs of topologies of machine learning models. In many embodiments, each graph includes a set of operators. For example, machine learning models 104 may include a deep learning model and the operators may correspond to layers in the deep learning model.
In various embodiments, FOD 105 may identify and/or evaluate operators in a machine learning model to fuse, or combine, them to improve the efficiency with which future machine learning inference workloads can be performed. For example, FOD 105 may identify opportunities to combine multiple adjacent operators into a single operator to save memory traffic and/or leverage potential mathematical compounding opportunities. With better efficiency, an improved user experience may be provided to more users at reduced performance and/or computational cost. In some embodiments, computational, or computation, costs may refer to costs while doing a job, such as how much time and power the computation costs. In several embodiments, deep learning inference may include a process where new data is fed into one or more pre-trained deep neural network models for classification. For example, a photo may be fed into a deep learning model that classifies people in the photo.
As will be described in more detail below, in one or more embodiments, FOD 105 may include, or implement, a fusion candidate selection stage and a fusion candidate filtering stage. In several embodiments, the fusion candidate selection stage may include mining the fusion candidates, or fusible operators candidates, with a metric designed to factor both frequency and computation cost. In many embodiments, the fusion candidate filtering stage may include extracting feature sets 107 from the fusion candidates. In many such embodiments, the fusion candidate filtering stage may include evaluating fusibility of one or more of the fusion candidates 106 based on the feature sets 107 with a machine learning classifier, such as a recurrent neural network (RNN). In various embodiments, operator fusion may be utilized to improve deep learning inference computational efficiency across multiple platforms.
In one or more embodiments described herein, fusion candidate 210 may be detected and/or evaluated. For instance, the machine learning model 204 may be mined based one or more operational parameters, such as frequency of use and/or computational cost. In various embodiments, FOD 105 may identify and/or evaluate operators in machine learning model 204 to fuse, or combine, them to improve the efficiency with which future machine learning inference workloads can be performed. For example, operators 218-2, 218-3, 218-4 of fusion candidate 210 may be integrated to produce combined operator 220. In one or more embodiments, machine learning model 204, fusion candidate model 214, and fused model 216 may include deep neural network (DNN) models. In several embodiments, each of the operators 218, 220 may comprise a layer in the DNN.
In several embodiments, fusion candidate 210 may provide an opportunity for operator fusion. In many embodiments, operator fusion may include combining multiple adjacent operators (e.g., operators 218-2, 218-3, 218-4) into a single operator (e.g., combined operator 220). In various embodiments, one or more aspects of operator fusion described herein may save memory traffic and/or leverage potential mathematical compounding opportunities via fused model 216. With better efficiency, an improved user experience may be provided to more users at a reduced cost. In several embodiments, deep learning inference may include a process where new data is fed into one or more pre-trained deep neural network models for classification. For example, a photo may be fed into a deep learning model that classifies people in the photo.
Various embodiments described herein may implement one or more of the following operations, procedures, settings, and/or configurations. For example, such embodiments may include an apparatus comprising a processor and a memory comprising instructions that when executed by the processor cause the processor to implement one or more of the following operations, procedures, settings, and/or configurations. Some embodiments may identify input comprising one or more machine learning models that each include a graph of operators. Many embodiments may mine the one or more machine learning models based on one or more operational parameters to determine one or more fusion candidates, each of the one or more fusion candidates comprising a subgraph of at least one graph of operators, wherein each subgraph includes two or more operators. One or more embodiments may extract a feature set from each of the one or more fusion candidates.
Some embodiments may utilize a machine learning classifier to evaluate the one or more fusion candidates based on the feature sets extracted from each of the one or more fusion candidates. Several embodiments may provide, as output, a proposed candidate of the one or more fusion candidates to fuse based on evaluation of the one or more fusion candidates. In many embodiments, the machine learning classifier may implement a machine learning algorithm to identify patterns in fusible operators. In many such embodiments, the patterns in fusible operators may be used to increase the efficiency of future machine learning models, such as by fusing operators based on the pattern.
One or more embodiments may combine each operator in the subgraph of the proposed candidate to fuse the proposed candidate into a fused candidate. One or more such embodiments may evaluate computational efficiency of a first machine learning model with the proposed candidate and a second machine learning model with the fused candidate to validate the proposed candidate. Many embodiments may utilize compiler stacks to evaluate computational efficiency of the first and second machine learning models. Various embodiments may utilize a tensor virtual machine (TVM) to evaluate computational efficiency of the first and second machine learning models. In some embodiments the machine learning model may comprise a deep neural network (DNN) model and each operator includes a layer in the DNN model. In one or more embodiments, a layer may be the basic unit of a deep learning network. In one or more such embodiments, a layer may take data from a predecessor operator, transform the data according to specified parameters, and output the transformed data to the next operator.
Many embodiments may rank each of the one or more fusion candidates based on the feature sets to identify the proposed candidate. In some embodiments, a confidence score of a machine learning classifier may be used to rank each of the one or more fusion candidates. In several embodiments the feature set may include the one or more operational parameters. In various embodiments, the one or more operational parameters may include one or more of a frequency of utilization, a computational cost, and a memory cost.
One or more embodiments may utilize weighted frequent subgraph mining to mine the one or more machine learning models based on the one or more operational parameters to determine the one or more fusion candidates. One or more such embodiments may generate an edge weight metric based on the one or more operational parameters to mine the one or more machine learning models.
In various embodiments, each feature set may include one or more core features and one or more uncore features. In several such embodiments, the core features may comprise one or more of instructions retired, elapsed core clock ticks, core frequency, L2 cache hits and misses, and L3 cache hits and misses. In many such embodiments, the uncore features may comprise one or more of read bytes from memory controllers, bytes written to memory controllers, and data traffic transferred via interconnect links.
Some embodiments may utilize a performance counter monitor (PCM) to extract the feature sets. In several embodiments, each feature set may include indications of one or more of data movement patterns, computation patterns, system resource utilization, frequency, computation cost, and memory cost. In many embodiments, the machine learning classifier may comprise a recurrent neural network (RNN). Various such embodiments may map the feature sets to vectors corresponding to fusibility. some such embodiments may compute a probability that each fusion candidate is fusible with the vectors corresponding to fusibility.
In many embodiments, FOD 305 may include, or implement, the fusion candidate selection stage with candidate selector 320 to identify one or more fusion candidates 306 based on the one or more machine learning models 304 in input 302. In several embodiments, the fusion candidate selection stage may include subgraph mining 322 of input 304 to identify the fusion candidates 306. In some embodiments, subgraph mining 322 may be performed based on one or more operational parameters such as frequency and computation cost. In some such embodiments, the one or more operational parameters may include a metric designed to factor both frequency and computation cost. In various embodiments, the one or more fusion candidates 306 may be provided to candidate filter 326 for feature extraction 328.
In some embodiments, candidate selector 320 may perform one or more of the following, such as during the fusion candidate selection stage. Candidate selector 320 may continually collect online machine learning models that serve deep learning inference workloads. In various embodiments, the topologies of each workload may be modeled as directed graphs. In several embodiments, the fusible operator candidate selection procedure may be modeled as a weighted frequent subgraph mining problem. In several such embodiments, the GraMi algorithm may be utilized to solve the weighted frequent subgraph mining problem. Psuedo code for the fusible operator candidate selection procedure may include one or more portions below:
In various embodiments, an edge weight metric may be used to avoid difficult cases, such as a frequent subgraph with trivial computation costs or a rare subgraph with excessive computation cost. In many embodiments, the weight metric may take one or more of frequency, computation cost, and memory cost into account. In some embodiments, for every subgraph, g, the total computation cost may be computed. In some such embodiments, the total computation cost of a subgraph may be determined by summing the cost of every operator, as shown in Equation (1):
cost(g)=Σop∈gcost(op) (1)
In one or more embodiments, in addition to frequency and cost, the product of frequency and computation cost of every subgraph as another operational parameter, as shown in Equation (2):
freqCost(g)=freq(g)·cost(g) (2)
In some embodiments, a min max normalization technique may be utilized to ensure scaling of the operational parameters are consistent. Accordingly, a given operational parameter, f of a subgraph, g, may be normalized with respect to the original graph, G, as shown in Equation (3):
Further, in many embodiments, the normalized features may be combined as the weight of a subgraph, g, with respect to the original graph, G, as shown in Equation (4):
Weight(g|G)=normFreqG(g)+normCostG(g)+normFreqCostG(g) (4)
In several embodiments, FOD 305 may include, or implement, the fusion candidate filtering stage based on one or more feature sets 307 extracted from fusion candidates 306 at feature extraction 328. In some embodiments, the feature sets 307 may include one or more of the operational parameters utilized during subgraph mining 322 to detect fusion candidates 306. In the illustrated embodiments, the feature sets 307 include frequency 332, computation cost 334, and memory cost 336. In some embodiments, feature sets 307 may include a set of features for each of fusion candidates 306.
In many embodiments, the fusion candidate filtering stage may include extracting feature sets 107 from the fusion candidates 306. In many such embodiments, the fusion candidate filtering stage may include evaluating fusibility of one or more of the fusion candidates 106 based on the feature sets 107 with an operator fusion classifier 338. In several embodiments, the operator fusion classifier 338 may include a machine learning classifier, such as a recurrent neural network (RNN).
In some embodiments, candidate filter 326 may perform one or more of the following, such as during the fusion candidate filtering stage. Candidate filter 326 may evaluate each of the fusion candidates 306 to determine fusibility. In various embodiments, determining fusibility may be implemented as a binary classification problem. For instance, feature sets 307 may be extracted from each fusion candidate at feature extraction 328. In such instances, a respective feature set may be provided to operator fusion classifier 338 to determine fusibility of a respective fusion candidate. As will be discussed in more detail below, in many embodiments, operator fusion classifier 338 may be trained as part of the fusion candidate filtering stage.
In many embodiments, for each fusion candidate (e.g., subgraph candidates 424), one or more types of features may be extracted. In some embodiments, data movement patterns and computation patterns may be extracted from machine code, as features 430. In one or more embodiments, system resources utilization may be extracted, as features 430. In various embodiments, machine code may include a collection of machine instructions utilized to realize specified functionalities. In some embodiments, machine code may be generated from compilers and/or hand-written by programmers.
In several embodiments, how a CPU and/or memory are utilized when executing the operators can indicate their fusibility. In various embodiments described herein, performance counter monitor (PCM) may be utilized to extract features 430. In some embodiments, each feature set may include one or more core features and one or more uncore features. In several such embodiments, the core features may comprise one or more of instructions retired, elapsed core clock ticks, core frequency, L2 cache hits and misses, and L3 cache hits and misses. In many such embodiments, the uncore features may comprise one or more of read bytes from memory controllers, bytes written to memory controllers, and data traffic transferred via interconnect links.
In one or more embodiments, the features 430 may be input as time series data to an RNN model, such as at classifier trainer 440. In one or more such embodiments, inputting the features 430 as times series data may enable the RNN model to learn the underlying patterns of fusible operators. Generally, the RNN may map the extracted features to vectors that preserve information corresponding to the operator's fusibility. The resulting vectors may then be used to compute the probability that the operators are fusible. In one or more embodiments, techniques similar to those used in sentiment classification of natural language processing may be used. to learn the underlying patterns of fusible operators.
In various embodiments, a loose threshold may be initially chosen to bootstrap the training process due to lack of positive samples. Among the samples predicted as fusible (e.g., fusion candidates), fusibility may be validated, such as by comparing computational efficiency between the original operators and the optimized operators using compiler stacks, such as a deep learning stack, end-to-end deep learning stack, and/or a tensor virtual machine (TVM) that supports low-level optimization. In many embodiments, the true positive may be included in a training data set as positive samples. In several embodiments, this process may be iterated along the data growth. In some embodiments, the threshold may be gradually raised as the classification becomes more and more accurate.
In the illustrated embodiment, proposed fusion candidate 510-1 may include operators 518-1, 518-2, 518-3, proposed fusion candidate 510-2 may include operators 518-4, 518-5, 518-6, 518-7, 518-8, proposed fusion candidate 510-4 may include operators 518-9, 518-10, 518-11, 518-12, 518-13, and proposed fusion candidate 510-4 may include operators 518-14, 518-15, 518-16, 518-17, 518-18. In various embodiments, each proposed fusion candidate 510-1, 510-2, 510-3, 510-4 may correspond to a proposed candidate features 512-1, 512-2, 512-3, 512-4. In various such embodiments, proposed candidate features 512-1, 512-2, 512-3, 512-4 may include, respectively, frequency 550-1, 550-2, 550-3, 550-4, computation cost 552-1, 552-2, 552-3, 552-4, score metric 554-1, 554-2, 554-3, 554-4, and rank 556-1, 556-2, 556-3, 556-4. In some embodiments, proposed candidate features 512 may be utilized to rank each of the proposed fusion candidates 512. In various embodiments, score metric 554 may be generated based on one or more other proposed candidate features 556, such as frequency 550 and computation cost 552. In one or more embodiments, the proposed fusion candidates 510 may be ranked based on the score metric 554.
In some embodiments, output 508 may be based on one or more online cloud computing workloads. In some such embodiments, the one or more online cloud computing workloads may be collected based on one or more machine learning models, such as convolutional neural network (CNN) models. In various embodiments, the number of occurrences of each machine learning model may be set to a range. For example, the number of occurrences of every machine learning model may be limited to a number between 10 and 50. In one or more embodiments, output 508 may identify deep operator composition that have high frequency and/or heavy computation cost.
In the illustrated embodiments, logic flow 600 may begin at block 602. At block 602 “identify input comprising one or more machine learning models that each include a graph of operators” input including one or more machine learning models that each include a graph of operators may be identified. For example, fusible operator detector (FOD) 105 may identify input 102 comprising one or more machine learning models 104. Continuing to block 604 “mine the one or more machine learning models based on one or more operational parameters to determine one or more fusion candidates, each of the one or more fusion candidates comprising a subgraph of at least one graph of operators, wherein each subgraph includes two or more operators” the one or more machine learning models may be mined based on one or more operational parameters to determine one or more fusion candidates that each include a subgraph of at least one graph of operators with two or more operators. In some embodiments, FOD 105 may identify one or more fusion candidates 106 in machine learning models 104 based on one or more operational parameters. In various embodiments, the machine learning model 204 may be mined based one or more operational parameters, such as frequency of use and/or computational cost, to identify fusion candidate 210.
At block 606 “extract a feature set from each of the one or more fusion candidates” a feature set from each of the one or more fusion candidates may be extracted. For example, candidate filter 326 of FOD 305 may extract feature sets 307 from each of the fusion candidates 306. Proceeding to block 608 “utilize a machine learning classifier to evaluate the one or more fusion candidates based on the feature sets extracted from each of the one or more fusion candidates” a machine learning classifier may be utilized to evaluate the one or more fusion candidates based on the extracted feature sets. For instance, candidate filter 326 may utilize operator fusion classifier 338 to evaluate each of fusion candidates 306 based on the feature sets 307. At block 610 “provide, as output, a proposed candidate of the one or more fusion candidates to fuse based on evaluation of the one or more fusion candidates” output may be provided that includes a proposed candidate of the one or more fusion candidates to fuse. For example, proposed fusion candidate 510-1 may be provided as output 508.
As used in this application, the terms “system” and “component” and “module” are intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution, examples of which are provided by the exemplary computing architecture 800. For example, a component can be, but is not limited to being, a process running on a processor, a processor, a hard disk drive, multiple storage drives (of optical and/or magnetic storage medium), an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a component. One or more components can reside within a process and/or thread of execution, and a component can be localized on one computer and/or distributed between two or more computers. Further, components may be communicatively coupled to each other by various types of communications media to coordinate operations. The coordination may involve the uni-directional or bi-directional exchange of information. For instance, the components may communicate information in the form of signals communicated over the communications media. The information can be implemented as signals allocated to various signal lines. In such allocations, each message is a signal. Further embodiments, however, may alternatively employ data messages. Such data messages may be sent across various connections. Exemplary connections include parallel interfaces, serial interfaces, and bus interfaces.
The computing architecture 800 includes various common computing elements, such as one or more processors, multi-core processors, co-processors, memory units, chipsets, controllers, peripherals, interfaces, oscillators, timing devices, video cards, audio cards, multimedia input/output (I/O) components, power supplies, and so forth. The embodiments, however, are not limited to implementation by the computing architecture 800.
As shown in
The system bus 808 provides an interface for system components including, but not limited to, the system memory 806 to the processing unit 804. The system bus 808 can be any of several types of bus structure that may further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. Interface adapters may connect to the system bus 808 via a slot architecture. Example slot architectures may include without limitation Accelerated Graphics Port (AGP), Card Bus, (Extended) Industry Standard Architecture ((E)ISA), Micro Channel Architecture (MCA), NuBus, Peripheral Component Interconnect (Extended) (PCI(X)), PCI Express, Personal Computer Memory Card International Association (PCMCIA), and the like.
The system memory 806 may include various types of computer-readable storage media in the form of one or more higher speed memory units, such as read-only memory (ROM), random-access memory (RAM), dynamic RAM (DRAM), Double-Data-Rate DRAM (DDRAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory (e.g., one or more flash arrays), polymer memory such as ferroelectric polymer memory, ovonic memory, phase change or ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, magnetic or optical cards, an array of devices such as Redundant Array of Independent Disks (RAID) drives, solid state memory devices (e.g., USB memory, solid state drives (SSD) and any other type of storage media suitable for storing information. In the illustrated embodiment shown in
The computer 802 may include various types of computer-readable storage media in the form of one or more lower speed memory units, including an internal (or external) hard disk drive (HDD) 814, a magnetic floppy disk drive (FDD) 816 to read from or write to a removable magnetic disk 818, and an optical disk drive 820 to read from or write to a removable optical disk 822 (e.g., a CD-ROM or DVD). The HDD 814, FDD 816 and optical disk drive 820 can be connected to the system bus 808 by a HDD interface 824, an FDD interface 826 and an optical drive interface 828, respectively. The HDD interface 824 for external drive implementations can include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 994 interface technologies. In various embodiments, these types of memory may not be included in main memory or system memory.
The drives and associated computer-readable media provide volatile and/or nonvolatile storage of data, data structures, computer-executable instructions, and so forth. For example, a number of program modules can be stored in the drives and memory units 810, 812, including an operating system 830, one or more application programs 832, other program modules 834, and program data 836. In one embodiment, the one or more application programs 832, other program modules 834, and program data 836 can include or implement, for example, the various techniques, applications, and/or components described herein.
A user can enter commands and information into the computer 802 through one or more wire/wireless input devices, for example, a keyboard 838 and a pointing device, such as a mouse 840. Other input devices may include microphones, infra-red (IR) remote controls, radio-frequency (RF) remote controls, game pads, stylus pens, card readers, dongles, finger print readers, gloves, graphics tablets, joysticks, keyboards, retina readers, touch screens (e.g., capacitive, resistive, etc.), trackballs, trackpads, sensors, styluses, and the like. These and other input devices are often connected to the processing unit 804 through an input device interface 842 that is coupled to the system bus 808, but can be connected by other interfaces such as a parallel port, IEEE 994 serial port, a game port, a USB port, an IR interface, and so forth.
A monitor 844 or other type of display device is also connected to the system bus 808 via an interface, such as a video adaptor 846. The monitor 844 may be internal or external to the computer 802. In addition to the monitor 844, a computer typically includes other peripheral output devices, such as speakers, printers, and so forth.
The computer 802 may operate in a networked environment using logical connections via wire and/or wireless communications to one or more remote computers, such as a remote computer 848. In various embodiments, one or more migrations may occur via the networked environment. The remote computer 848 can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer 802, although, for purposes of brevity, only a memory/storage device 850 is illustrated. The logical connections depicted include wire/wireless connectivity to a local area network (LAN) 852 and/or larger networks, for example, a wide area network (WAN) 854. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which may connect to a global communications network, for example, the Internet.
When used in a LAN networking environment, the computer 802 is connected to the LAN 852 through a wire and/or wireless communication network interface or adaptor 856. The adaptor 856 can facilitate wire and/or wireless communications to the LAN 852, which may also include a wireless access point disposed thereon for communicating with the wireless functionality of the adaptor 856.
When used in a WAN networking environment, the computer 802 can include a modem 858, or is connected to a communications server on the WAN 854, or has other means for establishing communications over the WAN 854, such as by way of the Internet. The modem 858, which can be internal or external and a wire and/or wireless device, connects to the system bus 808 via the input device interface 842. In a networked environment, program modules depicted relative to the computer 802, or portions thereof, can be stored in the remote memory/storage device 850. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used.
The computer 802 is operable to communicate with wire and wireless devices or entities using the IEEE 802 family of standards, such as wireless devices operatively disposed in wireless communication (e.g., IEEE 802.16 over-the-air modulation techniques). This includes at least Wi-Fi (or Wireless Fidelity), WiMax, and Bluetooth™ wireless technologies, among others. Thus, the communication can be a predefined structure as with a conventional network or simply an ad hoc communication between at least two devices. Wi-Fi networks use radio technologies called IEEE 802.11x (a, b, g, n, etc.) to provide secure, reliable, fast wireless connectivity. A Wi-Fi network can be used to connect computers to each other, to the Internet, and to wire networks (which use IEEE 802.3-related media and functions).
As shown in
The clients 902 and the servers 904 may communicate information between each other using a communication framework 906. The communications framework 906 may implement any well-known communications techniques and protocols. The communications framework 906 may be implemented as a packet-switched network (e.g., public networks such as the Internet, private networks such as an enterprise intranet, and so forth), a circuit-switched network (e.g., the public switched telephone network), or a combination of a packet-switched network and a circuit-switched network (with suitable gateways and translators).
The communications framework 906 may implement various network interfaces arranged to accept, communicate, and connect to a communications network. A network interface may be regarded as a specialized form of an input output interface. Network interfaces may employ connection protocols including without limitation direct connect, Ethernet (e.g., thick, thin, twisted pair 10/100/1900 Base T, and the like), token ring, wireless network interfaces, cellular network interfaces, IEEE 802.11a-x network interfaces, IEEE 802.16 network interfaces, IEEE 802.20 network interfaces, and the like. Further, multiple network interfaces may be used to engage with various communications network types. For example, multiple network interfaces may be employed to allow for the communication over broadcast, multicast, and unicast networks. Should processing requirements dictate a greater amount speed and capacity, distributed network controller architectures may similarly be employed to pool, load balance, and otherwise increase the communicative bandwidth required by clients 902 and the servers 904. A communications network may be any one and the combination of wired and/or wireless networks including without limitation a direct interconnection, a secured custom connection, a private network (e.g., an enterprise intranet), a public network (e.g., the Internet), a Personal Area Network (PAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN), an Operating Missions as Nodes on the Internet (OMNI), a Wide Area Network (WAN), a wireless network, a cellular network, and other communications networks.
Various embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints.
One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium which represents various logic within the processor, which when read by a machine causes the machine to fabricate logic to perform the techniques described herein. Such representations, known as “IP cores” may be stored on a tangible, machine readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that actually make the logic or processor. Some embodiments may be implemented, for example, using a machine-readable medium or article which may store an instruction or a set of instructions that, if executed by a machine, may cause the machine to perform a method and/or operations in accordance with the embodiments. Such a machine may include, for example, any suitable processing platform, computing platform, computing device, processing device, computing system, processing system, computer, processor, or the like, and may be implemented using any suitable combination of hardware and/or software. The machine-readable medium or article may include, for example, any suitable type of memory unit, memory device, memory article, memory medium, storage device, storage article, storage medium and/or storage unit, for example, memory, removable or non-removable media, erasable or non-erasable media, writeable or re-writeable media, digital or analog media, hard disk, floppy disk, Compact Disk Read Only Memory (CD-ROM), Compact Disk Recordable (CD-R), Compact Disk Rewriteable (CD-RW), optical disk, magnetic media, magneto-optical media, removable memory cards or disks, various types of Digital Versatile Disk (DVD), a tape, a cassette, or the like. The instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, and the like, implemented using any suitable high-level, low-level, object-oriented, visual, compiled and/or interpreted programming language.
The following examples pertain to further embodiments, from which numerous permutations and configurations will be apparent.
Example 1 is an apparatus, comprising: a processor; and a memory comprising instructions that when executed by the processor cause the processor to: identify input comprising one or more machine learning models that each include a graph of operators; mine the one or more machine learning models based on one or more operational parameters to determine one or more fusion candidates, each of the one or more fusion candidates comprising a subgraph of at least one graph of operators, wherein each subgraph includes two or more operators; extract a feature set from each of the one or more fusion candidates; utilize a machine learning classifier to evaluate the one or more fusion candidates based on the feature sets extracted from each of the one or more fusion candidates; and provide, as output, a proposed candidate of the one or more fusion candidates to fuse based on evaluation of the one or more fusion candidates.
Example 2 includes the subject matter of Example 1, the memory comprising instructions that when executed by the processor cause the processor to combine each operator in the subgraph of the proposed candidate to fuse the proposed candidate into a fused candidate.
Example 3 includes the subject matter of Example 2, the memory comprising instructions that when executed by the processor cause the processor to evaluate computational efficiency of a first machine learning model with the proposed candidate and a second machine learning model with the fused candidate to validate the proposed candidate.
Example 4 includes the subject matter of Example 3, the memory comprising instructions that when executed by the processor cause the processor to utilize compiler stacks to evaluate computational efficiency of the first and second machine learning models.
Example 5 includes the subject matter of Example 3, the memory comprising instructions that when executed by the processor cause the processor to utilize a tensor virtual machine (TVM) to evaluate computational efficiency of the first and second machine learning models.
Example 6 includes the subject matter of Example 1, the machine learning model comprising a deep neural network (DNN) model and each operator includes a layer in the DNN model.
Example 7 includes the subject matter of Example 1, the memory comprising instructions that when executed by the processor cause the processor to rank each of the one or more fusion candidates based on the feature sets to identify the proposed candidate.
Example 8 includes the subject matter of Example 1, wherein the feature set includes the one or more operational parameters.
Example 9 includes the subject matter of Example 1, wherein the one or more operational parameters include one or more of a frequency of utilization, a computational cost, and a memory cost.
Example 10 includes the subject matter of Example 1, the memory comprising instructions that when executed by the processor cause the processor to utilize weighted frequent subgraph mining to mine the one or more machine learning models based on the one or more operational parameters to determine the one or more fusion candidates.
Example 11 includes the subject matter of Example 10, the memory comprising instructions that when executed by the processor cause the processor to generate an edge weight metric based on the one or more operational parameters to mine the one or more machine learning models.
Example 12 includes the subject matter of Example 1, each feature set comprising one or more core features and one or more uncore features.
Example 13 includes the subject matter of Example 12, the core features comprising one or more of instructions retired, elapsed core clock ticks, core frequency, L2 cache hits and misses, and L3 cache hits and misses.
Example 14 includes the subject matter of Example 12, the uncore features comprising one or more of read bytes from memory controllers, bytes written to memory controllers, and data traffic transferred via interconnect links.
Example 15 includes the subject matter of Example 1, the memory comprising instructions that when executed by the processor cause the processor to utilize a performance counter monitor (PCM) to extract the feature sets.
Example 16 includes the subject matter of Example 1, wherein each feature set includes indications of one or more of data movement patterns, computation patterns, system resource utilization, frequency, computation cost, and memory cost.
Example 17 includes the subject matter of Example 1, the machine learning classifier comprising a recurrent neural network (RNN).
Example 18 includes the subject matter of Example 17, the memory comprising instructions that when executed by the processor cause the processor to map the feature sets to vectors corresponding to fusibility.
Example 19 includes the subject matter of Example 18, the memory comprising instructions that when executed by the processor cause the processor to calculate a probability that each fusion candidate is fusible with the vectors corresponding to fusibility.
Example 20 is at least one non-transitory computer-readable medium comprising a set of instructions that, in response to being executed by a processor circuit, cause the processor circuit to: identify input comprising one or more machine learning models that each include a graph of operators; mine the one or more machine learning models based on one or more operational parameters to determine one or more fusion candidates, each of the one or more fusion candidates comprising a subgraph of at least one graph of operators, wherein each subgraph includes two or more operators; extract a feature set from each of the one or more fusion candidates; utilize a machine learning classifier to evaluate the one or more fusion candidates based on the feature sets extracted from each of the one or more fusion candidates; and identify a proposed candidate of the one or more fusion candidates to fuse based on evaluation of the one or more fusion candidates.
Example 21 includes the subject matter of Example 20, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to combine each operator in the subgraph of the proposed candidate to fuse the proposed candidate into a fused candidate.
Example 22 includes the subject matter of Example 21, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to evaluate computational efficiency of a first machine learning model with the proposed candidate and a second machine learning model with the fused candidate to validate the proposed candidate.
Example 23 includes the subject matter of Example 22, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to utilize compiler stacks to evaluate computational efficiency of the first and second machine learning models.
Example 24 includes the subject matter of Example 22, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to utilize a tensor virtual machine (TVM) to evaluate computational efficiency of the first and second machine learning models.
Example 25 includes the subject matter of Example 20, the machine learning model comprising a deep neural network (DNN) model and each operator includes a layer in the DNN model.
Example 26 includes the subject matter of Example 20, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to rank each of the one or more fusion candidates based on the feature sets to identify the proposed candidate.
Example 27 includes the subject matter of Example 20, wherein the feature set includes the one or more operational parameters.
Example 28 includes the subject matter of Example 20, wherein the one or more operational parameters include one or more of a frequency of utilization, a computational cost, and a memory cost.
Example 29 includes the subject matter of Example 20, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to utilize weighted frequent subgraph mining to mine the one or more machine learning models based on the one or more operational parameters to determine the one or more fusion candidates.
Example 30 includes the subject matter of Example 29, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to generate an edge weight metric based on the one or more operational parameters to mine the one or more machine learning models.
Example 31 includes the subject matter of Example 20, each feature set comprising one or more core features and one or more uncore features.
Example 32 includes the subject matter of Example 31, the core features comprising one or more of instructions retired, elapsed core clock ticks, core frequency, L2 cache hits and misses, and L3 cache hits and misses.
Example 33 includes the subject matter of Example 31, the uncore features comprising one or more of read bytes from memory controllers, bytes written to memory controllers, and data traffic transferred via interconnect links.
Example 34 includes the subject matter of Example 20, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to utilize a performance counter monitor (PCM) to extract the feature sets.
Example 35 includes the subject matter of Example 20, wherein each feature set includes indications of one or more of data movement patterns, computation patterns, system resource utilization, frequency, computation cost, and memory cost.
Example 36 includes the subject matter of Example 20, the machine learning classifier comprising a recurrent neural network (RNN).
Example 37 includes the subject matter of Example 36, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to map the feature sets to vectors corresponding to fusibility.
Example 38 includes the subject matter of Example 37, comprising instructions that, in response to being executed by the processor circuit cause the processor circuit to calculate a probability that each fusion candidate is fusible with the vectors corresponding to fusibility.
Example 39 is a computer-implemented method, comprising: identifying input comprising one or more machine learning models that each include a graph of operators; mining the one or more machine learning models based on one or more operational parameters to determine one or more fusion candidates, each of the one or more fusion candidates comprising a subgraph of at least one graph of operators, wherein each subgraph includes two or more operators; extracting a feature set from each of the one or more fusion candidates; utilizing a machine learning classifier to evaluate the one or more fusion candidates based on the feature sets extracted from each of the one or more fusion candidates; and identifying a proposed candidate of the one or more fusion candidates to fuse based on evaluation of the one or more fusion candidates.
Example 40 includes the subject matter of Example 39, comprising combining each operator in the subgraph of the proposed candidate to fuse the proposed candidate into a fused candidate.
Example 41 includes the subject matter of Example 40, comprising evaluating computational efficiency of a first machine learning model with the proposed candidate and a second machine learning model with the fused candidate to validate the proposed candidate.
Example 42 includes the subject matter of Example 41, comprising utilizing compiler stacks to evaluate computational efficiency of the first and second machine learning models.
Example 43 includes the subject matter of Example 41, comprising utilizing a tensor virtual machine (TVM) to evaluate computational efficiency of the first and second machine learning models.
Example 44 includes the subject matter of Example 39, the machine learning model comprising a deep neural network (DNN) model and each operator includes a layer in the DNN model.
Example 45 includes the subject matter of Example 39, comprising ranking each of the one or more fusion candidates based on the feature sets to identify the proposed candidate.
Example 46 includes the subject matter of Example 39, wherein the feature set includes the one or more operational parameters.
Example 47 includes the subject matter of Example 39, wherein the one or more operational parameters include one or more of a frequency of utilization, a computational cost, and a memory cost.
Example 48 includes the subject matter of Example 39, comprising utilizing weighted frequent subgraph mining to mine the one or more machine learning models based on the one or more operational parameters to determine the one or more fusion candidates.
Example 49 includes the subject matter of Example 48, comprising generating an edge weight metric based on the one or more operational parameters to mine the one or more machine learning models.
Example 50 includes the subject matter of Example 39, each feature set comprising one or more core features and one or more uncore features.
Example 51 includes the subject matter of Example 50, the core features comprising one or more of instructions retired, elapsed core clock ticks, core frequency, L2 cache hits and misses, and L3 cache hits and misses.
Example 52 includes the subject matter of Example 50, the uncore features comprising one or more of read bytes from memory controllers, bytes written to memory controllers, and data traffic transferred via interconnect links.
Example 53 includes the subject matter of Example 39, comprising utilizing a performance counter monitor (PCM) to extract the feature sets.
Example 54 includes the subject matter of Example 39, wherein each feature set includes indications of one or more of data movement patterns, computation patterns, system resource utilization, frequency, computation cost, and memory cost.
Example 55 includes the subject matter of Example 39, the machine learning classifier comprising a recurrent neural network (RNN).
Example 56 includes the subject matter of Example 55, comprising mapping the feature sets to vectors corresponding to fusibility.
Example 57 includes the subject matter of Example 56, comprising calculating a probability that each fusion candidate is fusible with the vectors corresponding to fusibility.
Example 58 is an apparatus, comprising: means for identifying input comprising one or more machine learning models that each include a graph of operators; means for mining the one or more machine learning models based on one or more operational parameters to determine one or more fusion candidates, each of the one or more fusion candidates comprising a subgraph of at least one graph of operators, wherein each subgraph includes two or more operators; means for extracting a feature set from each of the one or more fusion candidates; means for utilizing a machine learning classifier to evaluate the one or more fusion candidates based on the feature sets extracted from each of the one or more fusion candidates; and means for identifying a proposed candidate of the one or more fusion candidates to fuse based on evaluation of the one or more fusion candidates.
Example 59 includes the subject matter of Example 58, comprising means for combining each operator in the subgraph of the proposed candidate to fuse the proposed candidate into a fused candidate.
Example 60 includes the subject matter of Example 59, comprising means for evaluating computational efficiency of a first machine learning model with the proposed candidate and a second machine learning model with the fused candidate to validate the proposed candidate.
Example 61 includes the subject matter of Example 60, comprising means for utilizing compiler stacks to evaluate computational efficiency of the first and second machine learning models.
Example 62 includes the subject matter of Example 60, comprising means for utilizing a tensor virtual machine (TVM) to evaluate computational efficiency of the first and second machine learning models.
Example 63 includes the subject matter of Example 58, the machine learning model comprising a deep neural network (DNN) model and each operator includes a layer in the DNN model.
Example 64 includes the subject matter of Example 58, comprising means for ranking each of the one or more fusion candidates based on the feature sets to identify the proposed candidate.
Example 65 includes the subject matter of Example 58, wherein the feature set includes the one or more operational parameters.
Example 66 includes the subject matter of Example 58, wherein the one or more operational parameters include one or more of a frequency of utilization, a computational cost, and a memory cost.
Example 67 includes the subject matter of Example 58, comprising means for utilizing weighted frequent subgraph mining to mine the one or more machine learning models based on the one or more operational parameters to determine the one or more fusion candidates.
Example 68 includes the subject matter of Example 67, comprising means for generating an edge weight metric based on the one or more operational parameters to mine the one or more machine learning models.
Example 69 includes the subject matter of Example 58, each feature set comprising one or more core features and one or more uncore features.
Example 70 includes the subject matter of Example 69, the core features comprising one or more of instructions retired, elapsed core clock ticks, core frequency, L2 cache hits and misses, and L3 cache hits and misses.
Example 71 includes the subject matter of Example 69, the uncore features comprising one or more of read bytes from memory controllers, bytes written to memory controllers, and data traffic transferred via interconnect links.
Example 72 includes the subject matter of Example 58, comprising means for utilizing a performance counter monitor (PCM) to extract the feature sets.
Example 73 includes the subject matter of Example 58, wherein each feature set includes indications of one or more of data movement patterns, computation patterns, system resource utilization, frequency, computation cost, and memory cost.
Example 74 includes the subject matter of Example 58, the machine learning classifier comprising a recurrent neural network (RNN).
Example 75 includes the subject matter of Example 74, comprising means for mapping the feature sets to vectors corresponding to fusibility.
Example 76 includes the subject matter of Example 75, comprising means for calculating a probability that each fusion candidate is fusible with the vectors corresponding to fusibility.
The foregoing description of example embodiments has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Many modifications and variations are possible in light of this disclosure. It is intended that the scope of the present disclosure be limited not by this detailed description, but rather by the claims appended hereto. Future filed applications claiming priority to this application may claim the disclosed subject matter in a different manner, and may generally include any set of one or more limitations as variously disclosed or otherwise demonstrated herein.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CN2019/073331 | 1/28/2019 | WO | 00 |
