Patent Application
20230139106
This application claims priority to Chinese Patent Application No. 202010015495.0 filed with the China National Intellectual Property Administration (CNIPA) on Jan. 7, 2020, the disclosure of which is incorporated herein by reference in its entirety.
Embodiments of the present application relate to deep learning technology, for example, a conversion method and apparatus for a deep learning model, a server and a storage medium.
A deep learning network is generally trained by algorithms. In most cases, an algorithm developer tends to use an open deep learning framework for model training. One deep learning framework may develop multiple deep learning models, and most open deep learning frameworks are designed for computing devices such as a central processing unit/graphics processing unit (CPU/GPU). The CPU/GPU uses a conventional instruction set architecture with relatively low architecture efficiency, relatively small operator granularity and relatively high flexibility. With the development of deep learning-related technology, a requirement for computing power is increasing. A defect of the architecture efficiency of the conventional instruction set cannot meet a requirement of an application scenario. In contrast, a data flow architecture has higher efficiency and is more suitable for a development trend of deep learning technology from the perspective of technical route. However, large differences between the data flow architecture and the instruction set architecture exist in data representation. The data flow architecture has much larger operator granularity than the instruction set architecture, and before the operator granularity of the data flow architecture is calculated, an order of calculation modules is predetermined according to data dependence. This difference determines that a model trained under the instruction set architecture cannot be directly deployed in the data flow architecture, significantly hindering the application development of the data flow architecture.
Embodiments of the present application provide a conversion method and apparatus for a deep learning model, a server and a storage medium so that a deep learning model developed based on an instruction set architecture is converted to operate under a data flow architecture.
In an embodiment, the embodiments of the present application provide a conversion method for a deep learning model. The method includes the steps below.
A target deep learning model is parsed into an intermediate representation of an instruction set computation graph.
The intermediate representation of the instruction set computation graph is converted into an intermediate representation of a data flow computation graph.
The intermediate representation of the data flow computation graph is adjusted to an intermediate representation of a customized architecture.
A converted target data flow network model corresponding to the target deep learning model is obtained according to the intermediate representation of the customized architecture.
In an embodiment, the embodiments of the present application provide a conversion apparatus for a deep learning model. The apparatus includes a target deep learning model parsing module, an instruction set computation graph intermediate representation conversion module, a data flow computation graph intermediate representation adjustment module and a target data flow network model generation module.
The target deep learning model parsing module is configured to parse a target deep learning model into an intermediate representation of an instruction set computation graph.
The instruction set computation graph intermediate representation conversion module is configured to convert the intermediate representation of the instruction set computation graph into an intermediate representation of a data flow computation graph.
The data flow computation graph intermediate representation adjustment module is configured to adjust the intermediate representation of the data flow computation graph to an intermediate representation of a customized architecture.
The target data flow network model generation module is configured to obtain a target data flow network model corresponding to the target deep learning model according to the intermediate representation of the customized architecture.
Optionally, the target deep learning model includes first operator granularity, the intermediate representation of the instruction set computation graph includes second operator granularity, and the intermediate representation of the data flow computation graph includes third operator granularity.
Optionally, the first operator granularity is the same as the second operator granularity.
Optionally, the second operator granularity is less than the third operator granularity.
Optionally, the intermediate representation of the instruction set computation graph further includes a second operator, and the intermediate representation of the data flow computation graph further includes a third operator.
Optionally, second operators form the third operator through fusion.
In an embodiment, the embodiments of the present application provide a server. The server includes one or more processors and a storage apparatus configured to store one or more programs.
The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method according to any embodiment of the present application.
In an embodiment, the embodiments of the present application provide a computer-readable storage medium storing a computer program, where the computer program, when executed by a processor, implements the method according to any embodiment of the present application.
In the embodiments of the present application, the target deep learning model is parsed into the intermediate representation of the instruction set computation graph; the intermediate representation of the instruction set computation graph is converted into the intermediate representation of the data flow computation graph; the intermediate representation of the data flow computation graph is adjusted to the intermediate representation of the customized architecture; and the converted target data flow network model corresponding to the target deep learning model is obtained according to the intermediate representation of the customized architecture. Therefore, the deep learning model developed based on the instruction set architecture is converted to operate under the data flow architecture. The intermediate representation of the instruction set computation graph, the intermediate representation of the data flow computation graph and the intermediate representation of the customized architecture are used for describing the deep learning model, which may make a design more flexible through a tradeoff in readability and execution efficiency according to an actual requirement.
The present application is further described in detail hereinafter in conjunction with the drawings and embodiments. It is to be understood that the embodiments described herein are intended to illustrate and not to limit the present application. Additionally, it is to be noted that to facilitate description, only part, not all, of structures related to the present application are illustrated in the drawings.
Some example embodiments are described as processing or methods depicted in flowcharts. Although multiple steps are described as sequential processing in the flowcharts, many of the steps may be implemented concurrently, coincidentally or simultaneously. Additionally, the sequence of the multiple steps may be rearranged. The processing may be terminated when the operations of the multiple steps are completed, but the processing may further have additional steps that are not included in the drawings. The processing may correspond to a method, a function, a procedure, a subroutine, a subprogram or the like.
Additionally, the terms “first”, “second” and the like may be used herein to describe multiple directions, actions, steps, elements or the like, but these directions, actions, steps or elements are not limited by these terms. These terms are only used for distinguishing one direction, action, step or element from another direction, action, step or element. For example, without departing from the scope of the present application, first operator granularity may be referred to as second operator granularity, and similarly, the second operator granularity may be referred to as the first operator granularity. Both the first operator granularity and the second operator granularity are operator granularity but not the same operator granularity. Terms like “first”, “second” and the like are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features as indicated. Thus, a feature defined as a “first” feature or a “second” feature may explicitly or implicitly include one or more of such features. As described herein, “multiple” is defined as at least two, for example, two, three or the like, unless otherwise expressly limited.
As shown in
In S110, a target deep learning model is parsed into an intermediate representation of an instruction set computation graph.
In an embodiment, a deep learning framework includes a large amount of base codes for an algorithm developer to perform model training such as TensorFlow, Caffe, Mxnet, Torch and the like. A deep learning model is a neural network model developed under the deep learning framework to implement a specific algorithm, and one deep learning framework may develop multiple deep learning models. A set of all instructions that CPU/GPU can execute is referred to as an instruction set, and an instruction set architecture is an interface between CPU/GPU physical hardware and higher-level software. Most open deep learning models are designed for computing devices such as the CPU/GPU, that is, most open deep learning models use the instruction set architecture.
The intermediate representation of the instruction set computation graph defines a network structure of the deep learning model, that is, types of operator and a connection relationship between operators. An operator is composed of one or more minimum operation units that can be executed by a target operation device. The connection relationship between operators represents an operation rule between the operators. Operator granularity represents a complexity of the operator and is generally represented by the number of minimum operation units included in the operator granularity, where an operator with large operator granularity is referred to as a large-granularity operator, and an operator with small granularity is referred to as a small-granularity operator. For example, in a CPU/GPU device, minimum operation units are A1, A2, A3 and A4, and operators are also A1, A2, A3 and A4. Then, corresponding operator granularity equals to 1, and there are four types of operator: A1, A2, A3 and A4. A connection relationship between operators may be that (A1+A2) is operated first and then (A1+A2+A3+A4) is operated. The deep learning model using the instruction set architecture generally includes the small-granularity operator with relatively small granularity, relatively high flexibility and low efficiency. When data to be computed is too large in volume, a relatively long computation time is needed.
The target deep learning model is parsed into the intermediate representation of the instruction set computation graph, that is, types of operator and an operation rule between operators of the target deep learning model are parsed out so that the operators in the target deep learning model developed based on the instruction set architecture can be fused and converted and the target deep learning model can be operated under a data flow framework.
Operator granularity of the target deep learning model is first operator granularity, and operator granularity of the intermediate representation of the instruction set computation graph is second operator granularity. Since the operator granularity is not varied when the target deep learning model is parsed into the intermediate representation of the instruction set computation graph, the first operator granularity is the same as the second operator granularity, and an operator in the target deep learning model is also the same as an operator in the intermediate representation of the instruction set computation graph, each of which is a first operator. That is, in the intermediate representation of the instruction set computation graph, the second operator granularity is obtained for the first operator. That is, the operator/operator granularity of the target deep learning model is consistent with the operator/operator granularity of the intermediate representation of the instruction set computation graph. Moreover, the intermediate representation of the instruction set computation graph is closest to a representation of an original computation graph of the target deep learning model.
In an embodiment, the first operator/first operator granularity is closer to a design level of a neural network algorithm and has relatively high readability, facilitating a developer interpreting a network structure.
In S120, the intermediate representation of the instruction set computation graph is converted into an intermediate representation of a data flow computation graph.
In an embodiment, the intermediate representation of the data flow computation graph represents types of operator and a connection relationship between operators under the data flow architecture. The operator in the intermediate representation of the instruction set computation graph is the first operator, and an operator in the intermediate representation of the data flow computation graph is a second operator. The intermediate representation of the instruction set computation graph is converted into the intermediate representation of the data flow computation graph, that is, the intermediate representation of the instruction set computation graph is reconstructed according to operator granularity of a data flow, and the first operator in the intermediate representation of the instruction set computation graph is fused into the second operator in the intermediate representation of the data flow computation graph according to the operator granularity of the data flow, that is, small-granularity operators in the intermediate representation of the instruction set computation graph are fused into a large-granularity operator. For example, in the intermediate representation of the instruction set computation graph, there are four types of operator: A1, A2, A3 and A4. A connection relationship between operators may be that (A1+A2) is operated first and then (A1+A2+A3+A4) is operated. When the intermediate representation of the instruction set computation graph is converted into the intermediate representation of the data flow computation graph, (A1+A2) (A1 and A2 are small-particle operators) is fused into B (B is a large-particle operator), and (A3+A4) is fused into C. In this case, B has operator granularity of 2. In the intermediate representation of the data flow computation graph, there are two types of operator: B and C, and a connection relationship between the operators is (B+C).
In an embodiment, the fusion here does not mean simple superimposition, but includes meanings of fusion and conversion.
The intermediate representation of the data flow computation graph includes third operator granularity, and the third operator granularity included in the intermediate representation of the data flow computation graph is larger than the second operator granularity included in the intermediate representation of the instruction set computation graph.
In S130, the intermediate representation of the data flow computation graph is adjusted to an intermediate representation of a customized architecture.
In an embodiment, the intermediate representation of the customized architecture represents operators in the data flow architecture operating the target deep learning model and a connection relationship between the operators. The intermediate representation of the data flow computation graph is adjusted to the intermediate representation of the customized architecture, that is, the operator in the intermediate representation of the data flow computation graph is reconstructed and rewritten according to a design principle of the data flow architecture operating the target deep learning model. The intermediate representation of the customized architecture is close to an underlying operation and has relatively high operation efficiency.
The operator in the intermediate representation of the data flow computation graph represents a minimum operation unit that can be executed under the data flow architecture. In the intermediate representation of the customized architecture, the minimum operation unit may be divided according to a module. For example, in the intermediate representation of the data flow computation graph, there are four types of operator: B, C, D and E. An operation relationship between operators is that (B+C) is operated first and then (B+C+D+E) is operated. The intermediate representation of the customized architecture may be that a first module operates (B+C) and a second module operates (D+E). In a design, the first module and the second module may perform the computation simultaneously, reducing the computation time and improving the efficiency.
In S140, a converted target data flow network model corresponding to the target deep learning model is obtained according to the intermediate representation of the customized architecture.
In an embodiment, the target data flow network model is the deep learning model operating under the data flow architecture, and the intermediate representation of the customized architecture may be regarded as a computation graph of the target data flow network model which includes both types of operator and a corresponding data parameter of the target data flow network model and a connection relationship between operators of the target data flow network model. The target deep learning model can be operated according to the intermediate representation of the customized architecture so that the deep learning model developed based on the instruction set architecture is converted to operate under the data flow architecture.
In the embodiments of the present application, the target deep learning model is parsed into the intermediate representation of the instruction set computation graph; the intermediate representation of the instruction set computation graph is converted into the intermediate representation of the data flow computation graph; the intermediate representation of the data flow computation graph is adjusted to the intermediate representation of the customized architecture; and the converted target data flow network model corresponding to the target deep learning model is obtained according to the intermediate representation of the customized architecture. Therefore, the deep learning model developed based on the instruction set architecture is converted to operate under the data flow architecture. The intermediate representation of the instruction set computation graph, the intermediate representation of the data flow computation graph and the intermediate representation of the customized architecture are used for describing the deep learning model, which may make a design more flexible through a tradeoff in readability and execution efficiency according to an actual requirement.
In an embodiment, the target deep learning model includes the first operator granularity, the intermediate representation of the instruction set computation graph includes the second operator granularity, and the intermediate representation of the data flow computation graph includes the third operator granularity.
In an embodiment, the first operator granularity is the same as the second operator granularity.
In an embodiment, the second operator granularity is less than the third operator granularity.
In an embodiment, the intermediate representation of the instruction set computation graph further includes the first operator, and the intermediate representation of the data flow computation graph further includes the second operator. In an embodiment, the third operator granularity is obtained for the second operator.
In an embodiment, multiple first operators form the second operator through the fusion and conversion.
In embodiment one of the present application, the target deep learning model is parsed into the intermediate representation of the instruction set computation graph; the intermediate representation of the instruction set computation graph is converted into the intermediate representation of the data flow computation graph; the intermediate representation of the data flow computation graph is adjusted to the intermediate representation of the customized architecture; and the converted target data flow network model corresponding to the target deep learning model is obtained according to the intermediate representation of the customized architecture. Therefore, the deep learning model developed based on the instruction set architecture is converted to operate under the data flow architecture. The intermediate representation of the instruction set computation graph, the intermediate representation of the data flow computation graph and the intermediate representation of the customized architecture are used for describing the deep learning model, which may make the design more flexible through the tradeoff in the readability and execution efficiency according to the actual requirement.
As shown in
The target deep learning model parsing module 210 is configured to parse a target deep learning model into an intermediate representation of an instruction set computation graph.
The instruction set computation graph intermediate representation conversion module 220 is configured to convert the intermediate representation of the instruction set computation graph into an intermediate representation of a data flow computation graph.
The data flow computation graph intermediate representation adjustment module 230 is configured to adjust the intermediate representation of the data flow computation graph to an intermediate representation of a customized architecture.
The target data flow network model generation module 240 is configured to obtain a converted target data flow network model corresponding to the target deep learning model according to the intermediate representation of the customized architecture.
In an embodiment, each of the target deep learning model parsing module 210, the instruction set computation graph intermediate representation conversion module 220 and the data flow computation graph intermediate representation adjustment module 230 is an independent module.
In an embodiment, each of the target deep learning model parsing module 210, the instruction set computation graph intermediate representation conversion module 220 and the data flow computation graph intermediate representation adjustment module 230 is an independent module.
In a case where one of the target deep learning model parsing module 210, the instruction set computation graph intermediate representation conversion module 220 and the data flow computation graph intermediate representation adjustment module 230 is modified, operating logics of other modules are not affected. For example, if the target deep learning model needs to be replaced and a target deep learning model after the replacement and the target deep learning model before the replacement are developed based on different deep learning frameworks, a related logic of the target deep learning model parsing module 210 is modified to correspond to a deep learning framework corresponding to the target deep learning model after the replacement, and the instruction set computation graph intermediate representation conversion module 220 and the data flow computation graph intermediate representation adjustment module 230 may remain constant and continue to be used. If the target data flow network model needs to be varied, a related variation is made to the data flow computation graph intermediate representation adjustment module 230, and the target deep learning model parsing module 210 and the instruction set computation graph intermediate representation conversion module 220 may remain constant and continue to be used.
In an embodiment, the target deep learning model includes first operator granularity, the intermediate representation of the instruction set computation graph includes second operator granularity, and the intermediate representation of the data flow computation graph includes third operator granularity.
In an embodiment, the first operator granularity is the same as the second operator granularity.
In an embodiment, the second operator granularity is less than the third operator granularity.
In an embodiment, the intermediate representation of the instruction set computation graph further includes a first operator, and the intermediate representation of the data flow computation graph further includes a second operator.
In an embodiment, multiple first operators form the second operator through fusion and conversion.
In the embodiments of the present application, the target deep learning model parsing module, the instruction set computation graph intermediate representation conversion module, the data flow computation graph intermediate representation adjustment module and the target data flow network model generation module are included so that the deep learning model developed based on the instruction set architecture is converted to operate under the data flow architecture. The intermediate representation of the instruction set computation graph, the intermediate representation of the data flow computation graph and the intermediate representation of the customized architecture are used for describing the deep learning model, which may make a design more flexible through a tradeoff in readability and execution efficiency according to an actual requirement. Each of the target deep learning model parsing module, the instruction set computation graph intermediate representation conversion module and the data flow computation graph intermediate representation adjustment module is the independent module, increasing expansibility of the conversion apparatus for a deep learning model and improving a development speed.
As shown in
The bus 318 represents one or more of several types of bus structures including a storage apparatus bus or a storage apparatus controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any one of multiple bus structures. For example, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus and a Peripheral Component Interconnect (PCI) bus.
The server 312 includes multiple computer system readable media. Such media may be available media that can be accessed by the server 312, including volatile and non-volatile media, and removable and non-removable media.
The storage apparatus 328 may include a computer system readable medium in the form of volatile memory, such as a random-access memory (RAM) 330 and/or a cache 332. The server 312 may include other removable/non-removable and volatile/non-volatile computer system storage media. Just for example, a storage system 334 may be configured to perform reading and writing on a non-removable and non-volatile magnetic medium (not shown in
A program/utility 340 having a group of program modules 342 (at least one program module 342) may be stored in, for example, the storage apparatus 328. Such program modules 342 include an operating system, one or more application programs, other program modules and program data. Each or some combination of such examples may include an implementation of a network environment. The program modules 342 generally perform functions and/or methods in the embodiments of the present application.
The server 312 may communicate with one or more external devices 314 (such as a keyboard, a pointing server and a display 324), and may also communicate with one or more servers that enable a user to interact with the server 312, and/or communicate with a server (such as a network card or a modem) that enables the server 312 to communicate with one or more other computing servers. Such communication may be performed through an input/output (I/O) interface 322. Moreover, the server 312 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN) and/or a public network such as the Internet) through a network adapter 320. As shown in
The one or more processors 316 executes a program stored in the storage apparatus 328 to perform multiple functional applications and data processing, for example, to perform the method according to any embodiment of the present application. The method may include the steps below. A target deep learning model is parsed into an intermediate representation of an instruction set computation graph.
The intermediate representation of the instruction set computation graph is converted into an intermediate representation of a data flow computation graph.
The intermediate representation of the data flow computation graph is adjusted to an intermediate representation of a customized architecture.
A converted target data flow network model corresponding to the target deep learning model is obtained according to the intermediate representation of the customized architecture.
Embodiment four of the present application further provides a computer-readable storage medium storing a computer program, where the computer program, when executed by a processor, implements the method according to any embodiment of the present application. The method may include the steps below.
A target deep learning model is parsed into an intermediate representation of an instruction set computation graph.
The intermediate representation of the instruction set computation graph is converted into an intermediate representation of a data flow computation graph.
The intermediate representation of the data flow computation graph is adjusted to an intermediate representation of a customized architecture.
A converted target data flow network model corresponding to the target deep learning model is obtained according to the intermediate representation of the customized architecture.
A computer storage medium in the embodiments of the present application may use a combination of one or more computer-readable media. The computer-readable media may be computer-readable signal media or computer-readable storage media. The computer-readable storage medium may be, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination thereof. Examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer magnetic disk, a hard disk, an RAM, a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or a suitable combination thereof. In this document, the computer-readable storage medium may be multiple tangible media containing or storing a program. The program may be used by or used in conjunction with an instruction execution system, apparatus or device.
A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier. Computer-readable program codes are carried in the data signal. The data signal propagated in this manner may be in multiple forms and includes an electromagnetic signal, an optical signal or a suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium that is not a computer-readable storage medium and that can send, propagate or transmit a program used by or used in conjunction with an instruction execution system, apparatus or device.
The program codes contained in the computer-readable medium may be transmitted in a suitable medium, including a wireless medium, a wire, an optical cable, radio frequency (RF) or a suitable combination thereof.
Computer program codes for performing the operations of the present application may be written in one or more programming languages or a combination thereof, the programming languages including object-oriented programming languages such as Java, Smalltalk and C++ and further including conventional procedural programming languages such as C programming language or similar programming languages. The program codes may be executed entirely or partially on a user computer, as a separate software package, partially on the user computer and partially on a remote computer, or entirely on the remote computer or terminal. In the case where the remote computer is involved, the remote computer may be connected to the user computer via multiple types networks including an LAN or a WAN, or may be connected to an external computer (for example, via the Internet through an Internet service provider).
| Number | Date | Country | Kind |
|---|---|---|---|
| 202010015495.0 | Jan 2020 | CN | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CN2021/070223 | 1/5/2021 | WO |
