Patent Grant
12099869
This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2020-0132759 filed on Oct. 14, 2020, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.
The following description relates to an electronic device and method with scheduling.
As artificial intelligence (AI) technology develops, use of hardware for AI is increasing. AI may perform inference and learning through operations. Thus, various devices are being developed as hardware for the implementation of AI.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In one general aspect, a processor-implemented method includes receiving respective requests for execution of a plurality of models to be independently executed in an accelerator, and performing queuing of the respective requests and layer-wise scheduling of the plurality of models, for execution by the accelerator, based on estimated idle times for respective candidate layers of each of the plurality of models corresponding to the queued respective requests.
The performing of the layer-wise scheduling may include selecting one candidate layer, of the respective candidate layers, that has a corresponding idle time that is minimum among the estimated idle times with respect to a state of the accelerator.
The state of the accelerator may be determined based on consideration of at least one of usage information of a memory of the accelerator, a difference between a point in time at which an operation resource of the accelerator will be completed and a point in time at which a memory access resource of the accelerator will be available to start being used, or a state of a progression of each of the plurality of models.
The usage information may be information of an entire capacity, a used capacity, and/or a remaining capacity of an on-chip memory of the accelerator.
The performing of the queuing of the respective requests and the layer-wise scheduling of the plurality of models may be performed in real-time.
Each of the estimated idle times, for the respective candidate layers, may be based on an idle time of a corresponding operation resource of the accelerator and an idle time of a corresponding memory access resource of the accelerator.
The idle time of the corresponding operation resource, for each of the respective candidate layers, may be determined based on a difference between a point in time at which the corresponding operation resource completes execution of a previously scheduled layer and a point in time at which a memory access resource, for the previously scheduled layer, completed execution, an execution time of the corresponding memory access resource.
The idle time of the corresponding operation resource, with respect to each of the respective candidate layers, may occur when an execution time of the corresponding memory access resource is respectively greater than an execution time of an operation resource for a previous layer that is most recently scheduled.
The corresponding idle time of the memory access resource, with respect to each of the respective candidate layers, may be determined based on a point in time at which execution of the memory access resource, with respect to each of the respective candidate layers, is suspended due to a constraint on a size of an on-chip memory of the accelerator, and a point in time at which execution of an operation resource for a previous layer that is most recently scheduled is completed.
The performing of the layer-wise scheduling of the plurality of models based on the estimated idle times may include selecting for execution a candidate layer, from among multiple candidate layers that have a same minimum estimated idle time, that has a lowest idle time of a corresponding memory access resource.
The performing of the layer-wise scheduling of the plurality of models based on the estimated idle times may include determining whether a candidate layer, among the respective candidate layers, has had a delayed execution a preset number of other layer execution times or more, and may be based on a result of the determining selecting the candidate layer to next be executed before remaining candidate layers of the respective candidate layers.
The estimated idle times may be estimated based on consideration of multiple layers currently being executed in the accelerator.
The performing of the layer-wise scheduling of the plurality of models may be performed independently of an order of the requests being received.
Two or more of the plurality of models may have no data dependency with one another when executed in the accelerator.
An operation resource of the accelerator may be based on one or more processing elements of the accelerator, and a memory access resource of the accelerator may be based on an on-chip memory and/or an off-chip memory of the accelerator.
In one general aspect, a non-transitory computer-readable storage medium may store instructions that, when executed by a processor, cause the processor to perform one or more or all operations and methods described herein.
In one general aspect, a non-transitory computer-readable storage medium may store instructions that, when executed by a processor distinct from the accelerator, may cause the processor to perform any one, combination, or all scheduling operations and methods described herein, and cause respective executions of the plurality of candidate layers based on the scheduling using the accelerator.
In one general aspect, an electronic device includes one or more processors configured to perform, in real-time, a queuing of respective requests for execution of a plurality of models to be independently executed in an accelerator, and a layer-wise scheduling of the plurality of models, for execution by the accelerator, based on estimated idle times for respective candidate layers of each of the plurality of models corresponding to the queued respective requests.
The device may further include an off-chip memory, and the accelerator that may include an on-chip memory.
For the performing in real-time of the layer-wise scheduling, the one or more processors may be configured to select one candidate layer, of the respective candidate layers, that has a corresponding idle time that is minimum among the estimated idle times with respect to a state of the accelerator.
Each of the estimated idle times, for the respective candidate layers, may be based on an idle time of a corresponding operation resource of the accelerator and an idle time of a corresponding memory access resource of the accelerator.
The idle time of the corresponding operation resource, for each of the respective candidate layers, may be determined based on a difference between a point in time at which the corresponding operation resource completes execution of a previously scheduled layer and a point in time at which a memory access resource, for the previously scheduled layer, completed execution, an execution time of the corresponding memory access resource.
The idle time of the corresponding memory access resource, with respect to each of the respective candidate layers, may be determined based on a point in time at which an execution of the corresponding memory access resource is suspended due to a constraint on a size of an on-chip memory of the accelerator, a point in time at which execution, of an operation resource for a previous layer that is most recently scheduled, is completed.
In one general aspect, an electronic device includes a scheduler configured to queue plural requests for execution of a plurality of models to be independently executed, and perform layer-wise scheduling on the plurality of models, for execution by the accelerator, based on estimated idle times for respective candidate layers of each of the plurality of models, and the accelerator configured to execute respective layers of the plurality of models based on the performed layer-wise scheduling by the scheduler.
In one general aspect, a processor-implemented method includes performing real-time layer-wise scheduling, of a plurality of models requested for execution in an accelerator, where the real-time layer-wise scheduling is based on estimated idle times for plural independent candidate layers of the plurality of models, and where the real-time layer-wise scheduling is performed after each time a previous candidate layer has begun scheduled execution in the accelerator and candidate layers remain to be scheduled with respect to the plurality of models, and instructing the accelerator to execute one or more of the plural independent candidate layers that have lowest estimated idle times of the estimated idle times.
The method may further include queuing respective requests for execution of the plurality of models, and performing the layer-wise scheduling based on the queued respective requests, a state of the accelerator, and respective workload characteristics of the plural independent candidate layers of the plurality of models.
The estimated idle times for the plural independent candidate layers may each be based on consideration of respective operation and memory access resources for the accelerator.
Other features and aspects will be apparent from the following detailed description, the drawings, and the claims.
Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.
The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order. Descriptions of features that are known after an understanding of the present disclosure may also be omitted for increased clarity and conciseness.
The features described herein may be embodied in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.
The terminology used herein is for describing various examples only and is not to be used to limit the disclosure. As further used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. For example articles “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should be further understood that the terms “comprises,” “includes,” and “has” specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements, and/or combinations thereof. The use of the term “may” herein with respect to an example or embodiment (for example, as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.
Although terms such as “first,” “second,” and “third” may be used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms. Rather, these terms are only used to distinguish one member, component, region, layer, or section from another member, component, region, layer, or section. Thus, a first member, component, region, layer, or section referred to in the examples described herein may also be referred to as a second member, component, region, layer, or section without departing from the teachings of the examples.
Throughout the specification, when a component is described as being “connected to,” or “coupled to” another component, it may be directly “connected to,” or “coupled to” the other component, or there may be one or more other components intervening therebetween. In contrast, when an element is described as being “directly connected to,” or “directly coupled to” another element, there can be no other elements intervening therebetween. Likewise, similar expressions, for example, “between” and “immediately between,” and “adjacent to” and “immediately adjacent to,” are also to be construed in the same way. As used herein, the term “and/or” includes any one and any combination of any two or more of the associated listed items.
Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.
Also, in the description of example embodiments, detailed description of structures or functions that are thereby known after an understanding of the disclosure of the present application will be omitted when it is deemed that such description could cause ambiguous interpretation of the example embodiments. Hereinafter, examples will be described in detail with reference to the accompanying drawings, and like reference numerals in the drawings refer to like elements throughout.
Referring to
The host processor 110 may be a device configured to control various operations of components included in the electronic device 100 and may include a central processing unit (CPU), for example. The host processor 110 may receive one or more requests for processing a neural network, for example, in the accelerator 140 and may generate respective instructions that are executable in the accelerator 140 for the received requests. A request described herein may be made for a neural network-based data inference, and for obtaining a result of the data inference by allowing the accelerator 140 to execute a neural network for object recognition, pattern recognition, computer vision, speech recognition, machine translation, machine interpretation, and the like, in various examples. The host processor 110 may transfer target inference data and particular parameters of the neural network to the accelerator 140.
The off-chip memory 120 may be a memory disposed outside the accelerator 140, and may be a dynamic random-access memory (DRAM) used as a main memory of the electronic device 100, as a non-limiting example. The off-chip memory 120 may store the target inference data and/or the parameters of the neural network to be executed in the accelerator 140, and the stored data may be transferred to the accelerator 140 for the performance of a subsequent inference. The off-chip memory 120 may also store input data for a first layer of the neural network along with the corresponding parameters of the neural network to be executed in the accelerator 140, and/or the host processor 110 may also store such input data, when the host processor 110 collects or receives information from an I/O or capturing device/component of the electronic device 100, such as a microphone or camera(s) in the example of
The off-chip memory 120 may have a larger memory capacity than the on-chip memory in the accelerator 140. However, when executing the example neural network, a cost for access by the accelerator 140 to the off-chip memory 120 may be greater than a cost for access to the on-chip memory. Such a memory access cost may indicate an amount of power and/or time that is used for accessing a memory and then reading or writing data from or in the memory.
The accelerator 140 may be an artificial intelligence (AI) accelerator configured to execute neural networks or neural network operations according to an instruction of the host processor 110 and, through the execution, infer a resultant data, and is a separate processor distinguished from the host processor 110. The accelerator 140 may be a neural processing unit (NPU), a graphics processing unit (GPU), a tensor processing unit (TPU), or the like in various examples. The accelerator 140 is also representative of one or more accelerators 140.
The accelerator 140 may process a workload that is more effectively processed by a separate dedicated processor, for example, the accelerator 140, than by the host processor 110 used for general purposes, based on characteristics of operations of the neural network. Here, one or more processing elements (PEs) are included in the accelerator 140, and the on-chip memory may be used. The on-chip memory may be a device including a global shared buffer and/or a local buffer that are included in the accelerator 140 and is distinguished from the off-chip memory 120 disposed outside the accelerator 140. The on-chip memory may include, for example, a scratchpad memory accessible through an address space, a static random-access memory (SRAM), or the like.
The neural network may include a plurality of layers. In an example, the neural network may include an input layer, a plurality of hidden layers, and an output layer. Each of the layers may include a plurality of nodes, each of which may also be referred to as an artificial neuron, though references to a neuron or neurons, or connections between the same, are not intended to infer any relation to biological formations, but rather are merely terms of art. Each of the nodes may be a computation unit having at least one input and output, and the nodes may be connected to one another in various ways in various examples. A weight may be set for a connection between respective nodes and the weights may be adjustable or changeable, e.g., for different layers of a same neural network, or for different neural networks configured for different purposes. Further, during training, the weights may represent interim weights, with the finally trained weights being trained weights. For example, during training as a non-limiting example, respective weight may be increased, decreased, or maintained a related data value, determining an influence of the data value on a final result. In a non-limiting example, each node included in the output layer may have weighted inputs from nodes included in a previous layer. A process in which weighted data is input from a layer to a subsequent layer of the layer may be referred to as forward propagation.
In an example, when a plurality of requests are received by the host processor 110 for neural network implementations, the accelerator 140 may respectively execute the plurality of neural networks according to instructions transferred from the host processor 110. In this example, the neural networks to be executed in the accelerator 140 may have different structures and/or different weights and connections, or the same neural network may be executed several times. In an additional example, plural parallel layers of a same neural network may be requested to be executed, or executed along with plural other neural networks. However, in the case in which the neural networks are executed in the accelerator 140 based simply on the order in which the requests are received by the host processor 110, idle times may exist during which a hardware resource of the accelerator 140 is not used in the course of the respective executions due to workload characteristics of each of the neural networks. In addition, a great tail latency may occur where a request received late is significantly delayed while a previous request is processed.
To prevent such a degradation of utilization of the accelerator 140, scheduling may be desirably controlled for which neural network to execute in the accelerator 140 at specific times.
In addition, in one or more examples, the scheduling on the neural networks may be performed in layer units, which may reduce or minimize the idle time occurring during the execution. A neural network is also referred to herein as a model for the convenience of description.
Referring to
The LV 0 memory 141-1 may be a memory accessible by the corresponding PE 141. That is, each LV 0 memory 141-1 in each respective PE 141 may only be accessible by that respective PE 141.
For each PE 141, the LV0 DMA 141-3 may control input data and/or output data of the LV0 memory 141-1 based on an instruction from the LV0 controller 141-7. The LV0 DMA 141-3 may read data from the LV0 memory 141-1 or write data in the LV0 memory 141-1 based on information associated with a source, a destination, and a data size that are included in the instruction from the LV0 controller 141-7.
Here, data input to the LV 0 memory 141-1 or data output from LV 0 memory 141-1 may be monitored and/or profiled. Such monitoring and/or profiling may be performed in the LV0 DMA 141-3 or a separate hardware element. Through the monitoring and/or profiling, it is possible to verify an access cost of the LV 0 memory 141-1, usage information of the LV 0 memory 141-1, and a type of data stored in the LV0 memory 141-1. For example, each LV0 DMA 141-3 may verify what percentage is indicated as the usage information of the corresponding LV0 memory 141-1, and which workload is involved with the data stored in the corresponding LV0 memory 141-1.
The MAC 141-5 may perform an operation or computation involved with a workload assigned to the PE 141. For example, the MAC 141-5 may perform a multiply-accumulate operation on given data. In addition, the MAC 141-5 may apply an activation function to the given data. The activation function may be sigmoid, hyperbolic tangent (tanh), or a rectified linear unit (ReLU), as non-limiting examples.
Each LV0 controller 141-7 may also be a device configured to control components included in the corresponding PE 141. For example, the LV0 controller 141-7 may control the LV0 memory 141-1, the LV0 DMA 141-3, and the MAC 141-5.
The foregoing description of the illustrated PE 141 of
In an example, each of n PEs, among all PEs, may cluster together. In this example, n is a natural number greater than 1 and less than the total number of the PEs included in the accelerator 140. That is, respective portions of the PEs included in the accelerator 140 may respectively cluster together to form plural clusters, for example, a PE cluster 142. PE cluster sizes may be the same or may vary in the accelerator 140.
PEs included in each cluster, e.g., the cluster 142, may share one LV1 memory 142-1. That is, the LV1 memory 142-1 may be accessible by the plural PEs included in the corresponding cluster 142. For example, even though operations respectively performed in a first PE and a second PE among the PEs in the cluster 142 may be different from each other, a same portion of data used for the operations may be commonly available to all PEs in the cluster 142. As this common data is stored in the LV1 memory 142-1, rather than being stored in an LV0 memory 141-1 included in each of a first PE and a second PE, the first PE and the second PE may share the common data, which may improve the efficiency of the accelerator 140. In the example of
In addition with respect to
In addition, an entirety 143 of the PEs may share the LV2 memory 143-1. That is, the LV2 memory 143-1 may be accessible by all the PEs included in accelerator 140. For example, there may be PEs that share a portion of data used to perform an operation, although those PEs are not clustered together to form a same cluster, among the PEs included in the accelerator 140. In this example, such PEs may not share the data through the LV1 memory 142-1, and may effectively share the common data through the LV2 memory 143-1, which may increase the efficiency of the accelerator 140. In addition with respect to
As described above, each of the PEs may access a respective LV0 memory 141-1, an LV1 memory 142-1 adjacent to each of the PEs, and the LV2 memory 143-1 of the accelerator 140, and use these memories to perform an assigned or instructed workload. The accelerator 140 may include a multilevel memory including hierarchical memories.
In addition, respective DMAs and controllers included in the accelerator 140 may be of a hierarchical multilevel type. In addition, resultant information from the monitoring and/or profiling of the respective data multilevel memories, monitoring and/or profiling data input to or output the respective memories, may be considered for the memory access cost, memory use or availability, and idle information of the accelerator 140. Accordingly, previous neural network executions by the accelerator 140 of various neural networks or network layers may be informative for estimating idle information for previously implemented or new candidate neural network layer, among a plurality of candidate neural network layers. Thus, the information of the memory accesses into memories, from memories to PEs, corresponding workloads and start and end completion times of the corresponding PEs, memory accesses from the PEs to memories, and eventual memory access to the off-chip memory 220 and/or the host processor 110 may be considered for each of the candidate layers, along with the workload characteristics, such as the number of parameters and the extent of input and output information, and type of operations to be performed for the execution of each of the candidate layers, for estimating PE and memory idle times for each of the candidate layers each time at least one resulting candidate layer is ultimately scheduled and executed, e.g., ultimately scheduled and executed after completion of a previously scheduled candidate layer.
In a non-limiting example, the PEs included in the accelerator 140 may simultaneously perform multiple workloads. One workload with a relatively greater operation amount may be assigned to a greater number of PEs (e.g., two or more clusters) and processed therein, and a second workload with a relatively less operation amount may be assigned to a smaller number of PEs (e.g., only one cluster) and processed therein. Alternatively, there may be multiple workloads being performed with an equal number of clusters for each workload.
For the convenience of description,
In the example of
In the example, the accelerator 210 includes a global shared buffer, and a plurality of PE arrays sharing the global shared buffer. Each of the PE arrays includes a local buffer, and a plurality of PEs sharing the local buffer. The global shared buffer and the local buffer may be referred to as an on-chip memory disposed inside the accelerator 210.
To execute a model in the accelerator 210, the processes of reading data used to execute the model through memory accesses, performing operations or computations in one or more respective PEs, and storing results of the operations or computations in a memory may be performed repeatedly or iteratively, e.g., the off-chip memory 220 in addition to the respective on-chip memory.
The on-chip memory may be disposed inside the accelerator 210 and have a lower access cost than the off-chip memory 220. However, the on-chip memory may have a smaller memory capacity than the off-chip memory 220, and thus the on-chip memory may not be sufficient to store all data for processing operations in PEs. Thus, the off-chip memory 220 may be used in such a case.
To execute a model in the accelerator 210, various hardware resources may be used. For example, an operation resource (or a computation resource) based on one or more PEs and a memory access resource based on the on-chip memory and/or the off-chip memory 220 may be used.
For example, the operation resource may indicate an operation quantity that is processible in a PE and may be represented by a unit of measure, such as, for example, in floating point operations per second (FLOPS) or tera operations per second (TOPS). The memory access resource may indicate an NoC bandwidth between PE arrays and a memory bandwidth between the accelerator 210 and the off-chip memory 220, and may be represented by another unit of measure, such as, for example, gigabytes per second (GB/s). In addition, the memory access resource may indicate a memory capacity of the global shared buffer and the local buffer and be represented by a still further unit of measure, such as, for example, megabyte (MB).
In an example, the memory bandwidth may be the memory bandwidth for transferring data stored in the off-chip memory 220, which may have a relatively high capacity, to the global shared buffer, which may have a relatively low capacity. The NoC bandwidth may be for transferring the data, which has been transferred to the global shared buffer, for example, to a PE array that performs an actual operation. Thus, in general, the memory bandwidth may ultimately be smaller than the NoC bandwidth in the accelerator 210.
As noted above, models and/or layers included in each of the models may have different workload characteristics, and thus the operation resource and the memory access resource that would be used for each model or layer may differ for each model or layer. Thus, by performing scheduling based on the workload characteristics, to increase or maximally overlap times for which the memory and computation/operation resources in the accelerator 210 are used and to reduce or minimize idle times, various examples may improve an overall system performance.
In an example, for model scheduling, data dependency and the availability of the on-chip memory may be further considered.
The data dependency may indicate a computation order of sets of data intended by a design or a compiler to obtain a desired result, and a plurality of layers included in one model may be sequentially processed in a preset order. However, there may be no data dependency among a plurality of models to be processed in the accelerator 210, and thus a change in a processing order of the models may not have a significant effect. For example, after one layer included in a first model is processed, a subsequent layer of the layer may be processed or a layer of a second model to be subsequently processed may be processed. As described in the foregoing example, a processing order between the first model and the second model may change by each layer execution.
The availability of the on-chip memory may restrict the processing of the accelerator 210. The on-chip memory may be an internal memory of the accelerator 210 that is fast accessible, but may not have a memory capacity sufficient for PEs to efficiently perform a particular operation. In such a case, when using the off-chip memory 220 corresponding to an external memory of the accelerator 210, a memory access time may be considered for performing scheduling because memory access time for the off-chip memory 220 is greater than that of the on-chip memory. That is, a method of reusing intermediate data of each model in the on-chip memory of the accelerator 210 may also affect the memory access cost, and thus it may also be considered for the scheduling.
Referring to
The host device 310 may include a host memory, a host processor, and an input storage. The host memory may include a request queue in which requests from a single or multiple users or functions are stored. In the request queue, execution requests for a model supported by the accelerator device 320 may be continuously accumulated. An execution request for a model described herein may refer to a request for executing the model. In the example of
The host processor may include a greedy scheduler, for example, configured to perform scheduling on a layer to be executed subsequently among models corresponding to requests stored in the request queue. The greedy scheduler will be simply referred to as a scheduler for the convenience of description.
The scheduler may be called each time execution of a scheduled layer is completed in the accelerator device 320 and the scheduler may perform scheduling for a layer of that model or another model that minimizes an idle time of the accelerator device 320 at a corresponding time. That is, the scheduler may calculate or estimate an idle time occurring when a candidate layer, which is a target for the scheduling, in each of a plurality of models corresponding to available user requests is executed at a point in time at which the scheduler is called, perform scheduling on a selected layer with a minimal or minimum idle time, and allow the layer to be executed in the accelerator device 320. In an example set of models, there may be no data dependency between the models, and thus the scheduler may perform layer-wise scheduling on the different models independently of an order of requests. As described above, as the scheduler calculates or estimates an idle time of the accelerator device 320 that occurs when each candidate layer is selected each time the execution of each layer is completed, and performs scheduling on a layer with a minimal or minimum idle time, it is possible to increase or maximize throughput and performance of the accelerator device 320 even through runtime scheduling that is based on a portion of layers without considering the execution of all the layers included in each model. In addition, even though the scheduler is called each time a layer to be executed in the accelerator device 320 is switched (that is, content switching), real-time scheduling and support the scalability to a plurality of models may be provided in various examples.
In an example, a subsequent accelerator state may be tracked and recorded, and the scheduler may perform scheduling based on the accelerator state. Such as discussed above with respect to the memory and operation resources, an accelerator state described herein may include at least one of usage information of a memory included in an accelerator (e.g., an entire capacity, a used capacity, and/or a remaining capacity of an on-chip memory in MB units of measure), a difference between a point in time at which an operation resource of the accelerator is most recently used and a point in time at which a memory access resource of the accelerator starts being used (e.g., in cycles unit of measure), or a state of a progression of each of models (e.g., represented by an n-th layer, considering the presence of data dependency among layers included in a same model). Hereinafter, scheduling based on such an accelerator state will be described in greater detail with reference to
In addition, the scheduler may calculate/estimate a potential probability that an idle time will occur in the future based on a current state of an on-chip memory, and perform scheduling based on a determined influence of the selection of a layer made at a current time on future layer scheduling.
The scheduler may perform the scheduling as described above until the execution of all models stored in the request queue is completed.
The input storage may include the various model parameters for the multiple models to be executed, as well as the respective input data that are targets for respective inferences. An input data may refer to data to be initially input to a model, or output data from one or more layers of a model that have been previously executed, as non-limiting examples.
The host device 310 may transfer, to the accelerator device 320, an accelerator instruction as to which layer is to be performed at which point in time determined by the scheduler. The accelerator device 320 may then execute a layer according to the accelerator instruction and return an inference result of a model (or layer of the model) for which the layer execution is completed to the host device 310.
As described above, various example embodiments may effectively implement a runtime scheduler without the addition of separate dedicated or auxiliary hardware for performing layer-wise runtime scheduling.
In one non-limiting example, the host devices of
For an operation to be performed in an operation resource, a process of reading data that is a target for the operation may need to be performed first through a memory access resource. The memory access resource and the operation resource may operate in parallel. Thus, it is possible to reduce an unnecessary idle time by reading in advance data for a subsequent operation through the memory access resource while an operation is being performed in the operation resource. As such an idle time of the memory access resource and the operation resource decreases, the utilization of an accelerator may be improved, and thus performance may be improved.
Referring to
In the example of
Referring to
The scheduler may determine an idle time of the memory access resource based on a point in time t1 at which the execution of the memory access resource for each candidate layer, which is a target for scheduling, is suspended due to a limited size of the on-chip memory of the accelerator and a point in time t2 at which the execution of the operation resource for a previous layer that is most recently scheduled is completed. For example, the scheduler may calculate, as the idle time of the memory access resource, a difference between the time t1 and the time t2. In addition, when calculating the idle time of the memory access resource, the scheduler may also use an accelerator state described above.
In an example, the scheduler may perform scheduling on a selected layer having a minimum sum of the idle time of the memory access resource and the idle time of the operation resource, among the candidate layers that are the targets for the scheduling for a plurality of models. In this example, when there are a plurality of candidate layers having the same minimum sum of the idle time of the memory access resource and the idle time of the operation resource, the scheduler may perform the scheduling on a selected layer having a minimum idle time of the memory access resource, for example. That is, the scheduling may be performed preferentially on a layer at which a difference between a point in time at which the operation resource of the accelerator is most recently used and a point in time at which the memory access resource starts being used is maintained at a similar level to an idle time of the memory access resource that occurs by the on-chip memory. Through this, it is possible to reduce or minimize an idle time that may occur in a next scheduling.
Referring to
In some case, due to a great idle time of the accelerator for a certain candidate layer, the candidate layer may not be selected by the scheduler. In such a case, a latency of a model in which the candidate layer is included may increase greatly. To prevent this, when there is a layer for which execution is delayed a preset number of times or more among candidate layers of a plurality of models, the scheduler may perform scheduling on the layer and allow the layer to be forced to be executed. Through this, it is possible to effectively manage a latency of the accelerator.
Although it is illustrated in
Referring to
As described above, a degree of usage of each resource may differ for each layer, and thus the scheduler may allocate two layers with different workloads among layers included in each of the first model and the second model to the operation resource and the memory access resource of an accelerator, respectively. For example, while one layer included in the first model is being allocated to the operation resource of the accelerator, the scheduler may allocate, to the memory access resource of the accelerator, a subsequent layer of the first model or a layer of the second model needed to be subsequently processed. In this example, the layer of the first model to be allocated to the operation resource may have a workload characteristic different from that of the subsequent layer of the first model or the layer of the second model to be allocated to the memory access resource.
As described, the scheduler may perform layer-wise scheduling on the first model and the second model based on a workload characteristic of each layer of the first model and the second model and a hardware resource of the accelerator. Thus, idle times may be minimized or reduced from occurring in each resource and the utilization of an accelerator. The scheduler may perform scheduling to change an execution order to a layer level between models independent of each other, or another equivalent operation unit (e.g., residual block, inception module, etc.)
Referring to
In an example, the scheduler may perform such layer-unit scheduling that selects a layer with a minimum idle time from among candidate layers of the models, and thus perform optimized scheduling that minimizes or reduces an idle time independently of an order of user or function requests. In this example, the scheduler may be called for each time of the execution of each layer.
The above discussions with respect to
Referring to
The server 900 may be a separate device distinguished from an example user terminal, or other electronic device, that is controlled by a user, and may communicate with one or more such user terminals through a wired and/or wireless network. The server 900 may receive requests that are simultaneously (or soon in time, e.g., with overlapping in time execution and/or execution requests) transmitted from multiple users through their user terminals. The server 900 may also receive requests that are transmitted from a same user from multiple user terminals of the user. Through a scheduler 910, the server 900 may perform layer-wise scheduling on a plurality of models to be executed in an accelerator 920. The accelerator 920 is representative of one or two or more accelerators 920. The accelerator 920 may execute the models based on the scheduling and determine inference results. The server 900 may then return the inference results to corresponding user terminals. A user terminal (or electronic device) described herein may include any or any combination of any two or more of, for example, a computing device such as a smartphone, a personal computer (PC), a tablet PC, and a laptop, a wearable device such as a smart watch and smart eyeglasses, a home appliance such as a smart speaker, a smart TV, and/or a smart refrigerator, and other devices such as a smart vehicle, a smart kiosk, and an Internet of things (IoT) device. In addition, an example exists where the server operation is performed by one of such electronic devices that are in communication with each other and model execution requests may be received from each of the other electronic devices as well as for the one such electronic device.
Referring to
The server 900 of
The host processors, host devices, schedulers, memory controllers, off-chip memory, accelerators, electronic devices, user terminals, and other devices, apparatuses, units, modules, and components described herein with respect to
The methods illustrated in
Instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler. In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computer using an interpreter. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions used herein, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.
The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computers.
While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2020-0132759 | Oct 2020 | KR | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5293620 | Barabash | Mar 1994 | A |
| 7634776 | Parameswaran | Dec 2009 | B2 |
| 8856777 | Sundarrajan et al. | Oct 2014 | B2 |
| 10175980 | Temam et al. | Jan 2019 | B2 |
| 10181000 | Vardhan | Jan 2019 | B1 |
| 10255118 | Yu et al. | Apr 2019 | B2 |
| 10540202 | Smaldone | Jan 2020 | B1 |
| 10740674 | Ambrose | Aug 2020 | B2 |
| 20020040381 | Steiger | Apr 2002 | A1 |
| 20120266176 | Vojnovic | Oct 2012 | A1 |
| 20130232495 | Rossbach | Sep 2013 | A1 |
| 20170344882 | Ambrose | Nov 2017 | A1 |
| 20180046903 | Yao et al. | Feb 2018 | A1 |
| 20190114533 | Ng et al. | Apr 2019 | A1 |
| 20190294959 | Vantrease | Sep 2019 | A1 |
| 20190294968 | Vantrease | Sep 2019 | A1 |
| 20190370085 | Gross | Dec 2019 | A1 |
| 20200057581 | Choi | Feb 2020 | A1 |
| 20200133898 | Therene et al. | Apr 2020 | A1 |
| 20200326766 | Tsirkin | Oct 2020 | A1 |
| 20210064433 | Nakfour | Mar 2021 | A1 |
| 20210373944 | Lee | Dec 2021 | A1 |
| 20220107837 | Youn | Apr 2022 | A1 |
| 20220114015 | Lee | Apr 2022 | A1 |
| 20230143270 | Lee | May 2023 | A1 |
| Number | Date | Country |
|---|---|---|
| 10-2020-0020366 | Feb 2020 | KR |
| 10-2020-0062323 | Jun 2020 | KR |
| WO 2016195840 | Dec 2016 | WO |
| WO 2017214728 | Dec 2017 | WO |
| Entry |
|---|
| Baek, et al. “A multi-neural network acceleration architecture.” 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA). IEEE, May 30-Jun. 3, 2020. pp. 940-953. |
| Extended European Search Report issued Feb. 4, 2022 in counterpart European Patent Application No. 21192131.7. |
| Number | Date | Country | |
|---|---|---|---|
| 20220114015 A1 | Apr 2022 | US |
