Patent Application
20230195509
To improve processing efficiency, some processing systems employ one or more processing units to execute specified operations with specialized hardware. For example, some processing systems employ one or more graphics processing units (GPUs) to execute operations associated with the generation of images. To perform the specified operations, in at least some cases the processing unit hardware is configured to perform repeated operations (e.g., vector or matrix operations) on relatively large sets of data. For example, some GPUs employ shader hardware to perform repeated processing passes on image data in order to apply a specified texture or other image processing to a set of image data (e.g., pixels).
The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items.
To illustrate, in many cases, the data sets processed by the processing unit are larger than are able to be accommodated by at least some of the memory hierarchy of the processing unit. In such cases, the data sets and associated operations are divided into units referred to as threads, and the threads themselves are divided into sets referred to as thread groups (or sometimes, threadgroups). To perform a processing operation, the hardware of the processing unit accesses subsets of the thread groups in a specified sequence, referred to as dispatch order, and processes the accessed subsets in the dispatch order. To process some data subsets, the processing unit loads corresponding data into one or more levels of the processing unit's memory hierarchy, while transferring data associated with a previous subset to main memory. This loading and transfer process is referred to herein as a “data swap.”
The overhead associated with a data swap (e.g., the loading and transferring of data) consumes power and other system resources. This overhead is reduced when the number of data swaps, or the amount of data moved for a particular data swap, is reduced. Furthermore, in some cases different thread groups share at least some data, such that processing those thread groups in sequence reduces one or both of the number of data swaps or the amount of data moved during a given data swap. However, the particular thread groups that share data differs based on the particular set of thread groups, the type of processing operations to be done on the thread groups, or a combination thereof. Conventionally, the thread groups are dispatched in a fixed specified order, without regard to the thread group to be processed or the processing operations to be done. Using the techniques described herein, thread groups are accessed and dispatched for processing based on a programmable access pattern. In some cases, the access pattern is programmed to reduce the number of data swaps, or the amount of data moved during one or more data swaps, thereby reducing the overall overhead associated with the data swaps. This in turn reduces power consumption at the processing unit and improves overall processing efficiency.
To execute the sets of commands received from the CPU, the processing unit 100 includes a command processor 102, a dispatch unit 104, a shader 110, and a cache 115. The command processor 102 is a set of hardware configured to receive the commands from the CPU and to prepare the received commands for processing. For example, in some embodiments the command processor 102 buffers the received commands, organizes the received commands into one or more queues for processing, performs operations to decode or otherwise interpret the received commands, and the like.
The dispatch unit 104 is a set of hardware configured to perform dispatch operations in response to dispatch commands (e.g., dispatch command 120) received from the command processor 102. Responsive to a dispatch command, the dispatch unit 104 accesses a set of thread groups 123 and dispatches subsets (e.g., thread group 124) of the thread groups to the shader 110 for processing. In some embodiments, the dispatch unit 104 dispatches the subsets of the thread groups 123 to the shader 110 in a particular order, designated dispatch order 122, wherein the dispatch order 122 is indicated by an access pattern 125 stored at a programmable table 106. That is, the access pattern 125 represents the pattern of how the thread groups 123 are accessed, and the dispatch order 122 is the order in which the thread groups 123 are dispatched to the shader 110. The dispatch order 122 is thus based on the access pattern 125.
In some embodiments, the programable table 106 is a set of registers, a content addressable memory (CAM) or other storage structure that is programmed based on pattern commands (e.g., pattern command 121). Each pattern command indicates an access pattern to be stored at the programmable table 106. In some embodiments, the pattern commands, such as pattern command 121, are issued by a device driver or other program, and are based on the particular thread groups 123 to be processed. As described further below, the device driver thereby tailors the access pattern 125, and the corresponding dispatch order 122, to reduce the number of data swaps, or the amount of data moved during one or more data swaps, thereby improving processing efficiency.
The shader 110 includes a plurality of processing elements (e.g., processing element 111) that together perform shading operations on thread groups (e.g., thread group 124) dispatched by the dispatch unit 104. In different embodiments, the processing elements of the shader 110 perform geometry operations, texture operations, tessellation operations, vertex operations, mesh operations, primitive operations, ray tracing operations, compute operations, and the like or any combination thereof, based on commands received from the command processor 102. In some embodiments, to perform these operations the processing elements each include one or more SIMD elements configured to execute the specified operations using the threads of the received thread groups.
The cache 115 stores data for the shader 110. Thus, in the course of executing shader operations, the shader 110 stores and retrieves data from the cache 115, wherein the stored and retrieved data is based on the particular thread group (or set of thread groups) being processed. For example, in some embodiments each thread of the thread group 124 corresponds to an individual pixel of an image, and the cache 115 stores data (e.g., pixel values) for each individual pixel, or a subset of the individual pixels, included in the thread group 124. In some embodiments, the processing unit 100 is associated with a memory hierarchy having multiple cache levels as well as a system memory (not shown), and the cache 115 represents one of the multiple cache levels (e.g., a Level 1 (L1) cache.
In some cases, when a new thread group is dispatched to the shader 110 by the dispatch unit 104, the shader 110 performs a data swap by loading data corresponding to the new thread group to the cache 115 and transferring data corresponding to the previous thread group to system memory. However, for some thread groups, the corresponding shader operations share at least some data. For example, in some embodiments each thread group of the thread groups 123 corresponds to a portion (e.g., a tile) of an image. For some shader operations, and for a given thread group 124, the shader 110 uses at least some pixel data from portions that adjoin the portion of the image associated with the thread group 124. Accordingly, by dispatching the thread groups 123 to the shader 110 in a particular order, the dispatch unit 104 is able to increase the likelihood that processing of a given thread group is able to use data already stored at the cache 115. This in turn reduces the amount of overhead associated with data swaps, reducing power consumption and increasing processing efficiency.
However, the way in which the shader operations share data between thread groups varies according to one or more of the data represented by the thread groups, the operations performed by the shader 110, or a combination thereof. For example, in some cases the shader operations share data between thread groups in one way for a particular type of texture operation but share data between thread groups differently for a different type of texture operation. As noted above, the dispatch order 122 is governed by the access pattern 125, because the access pattern 125 is the pattern according to which the thread groups 123 are accessed. Accordingly, by programming the access pattern 125 to a particular access pattern for the particular thread groups 123, a device driver or other program tailors the dispatch order 122 for the thread groups 123 to reduce the impact of data swaps, such as by reducing the amount of data loaded to the cache 115 when a new thread group 124 is dispatched for processing. For example, in some embodiments a programmer tests different access patterns for a given set of thread groups 123 and measures the impact of the data swaps for each of the different access patterns. For example, the programmer measures the number of cache lines of the cache 115 loaded or evicted when the thread groups 123 are processed at the shader 110. The programmer then selects the access pattern corresponding to the lowest impact (e.g., the fewest number of cache lines loaded or evicted). The programmer then programs the device driver or other program to issue the pattern command 121 to store the selected access pattern when the thread groups 123 are to be processed at the shader 110. The programmer thus ensures that the dispatch order 122 is based on an access pattern than reduces the impact of the data swaps, thereby reducing the overhead associated with the data swaps.
In other embodiments, the thread groups 123 represent non-image data. For example, in some embodiments each thread of the thread groups 123 represents a corresponding node of a neural network or other machine learning network. In still other embodiments, each thread of the thread groups 123 represents an entry of a matrix. Further, it will be appreciated that in some embodiments the thread groups 123 are organized in an array having more or fewer than two dimensions. For example, in some embodiments, the thread groups 123 are organized as a three-dimensional array with each thread of the thread groups 123 corresponding to a point in a three-dimensional space or volume, and the access pattern 125 indicates a three-dimensional access pattern.
In operation, the dispatch unit 104 dispatches the thread groups 123 to the shader 110 one at a time in a particular order, referred to as the dispatch order 122. In response to an indication that the shader 110 has completed processing a dispatched thread group, the dispatch unit 104 dispatches the next thread group in the dispatch order 122, until all of the thread groups 123 have been processed at the shader 110. This process of dispatching the thread groups 123 according to the dispatch order 122 is referred to as the dispatch unit 104 “walking” the thread groups 123, or sometimes as the dispatch unit 104 performing a “dispatch walk” with respect to the thread groups 123. It will be appreciated that the dispatch order 122 is based on the pattern with which the dispatch unit 104 accesses the thread groups 123, and that this pattern is indicated by the access pattern 125 programmed at the programmable table 106.
An example access pattern and corresponding dispatch walk is illustrated at
Thus, in the depicted example, based on the access pattern 125 indicating a typewriter pattern, the dispatch unit 104 first dispatches the thread group at entry (0,0) of the thread groups 123 to the shader 110. As indicated by arrow 333, in response to the shader 110 indicating processing of the previous thread group is complete, the dispatch unit 104 next dispatches the thread group at entry (0,1). Next, as illustrated by arrow 334, the dispatch unit 104 dispatches the thread group at entry (0, 2). The dispatch unit 104 then completes dispatching of row 0 of the thread groups 123 by dispatching the thread groups at the entry (0,3), as represented by the arrow 335.
After completing dispatch of row 0, the dispatch unit 104 moves to the first column of the next row (i.e., row 1), as indicated by arrow 336. The dispatch unit 104 dispatches the thread group at entry (1,0) to the shader 110. In response to the shader 110 indicating processing of the previous thread group is complete, the dispatch unit 104 next dispatches the thread group at entry (1,1) as shown by arrow 337. The dispatch unit 104 continues to dispatch the thread groups 123 according to this typewriter fashion, until finally dispatching the thread group at entry (7,3).
Thus, in the depicted example, based on the access pattern 125 indicating a first order Hilbert pattern, the dispatch unit 104 first dispatches the thread group at entry (0,0) of the thread groups 123 to the shader 110. As indicated by arrow 440, in response to the shader 110 indicating processing of the previous thread group is complete, the dispatch unit 104 next dispatches the thread group at entry (1,0). Next, as illustrated by arrow 441, the dispatch unit 104 dispatches the thread group at entry (1, 1). The dispatch unit 104 then dispatches the thread group at entry (0, 1), as indicated by the arrow 443. Next, the dispatch unit 104 dispatches the thread group at entry (0, 2). The dispatch unit 104 continues to follow this pattern until it dispatches the thread group at entry (0, 3). The dispatch unit 104 then moves to entry (2,0) and repeats the pattern. The dispatch unit 104 continues to repeat the first order Hilbert curve pattern until all of the thread groups 123 have been dispatched.
It will be appreciated that the access patterns and corresponding dispatch orders illustrated at
As is apparent from the examples of
Thus, in this example, dispatching thread groups according to the Hilbert curve access pattern illustrated at
In different embodiments, the pattern command 121 indicates the access pattern 528 in different ways. For example, in some embodiments, the pattern command 121 includes a field that reflects the access pattern 528 itself, such as by indicating a set of rules that designate the implementation of the access pattern 528. In other embodiments, the processing unit 100 stores a set of pre-defined access patterns, and the pattern command 121 indicates a selection of one of the set of pre-defined access patterns. In response to the pattern command 121, the command processor 102 or other module transfers the selected access pattern to the programmable table 106. In still other embodiments, the programmable table 106 itself stores different access patterns, and the pattern command 121 indicates which of the stored access patterns is to be used by the dispatch unit 104 to determine the access pattern 528.
It will be appreciated that in some embodiments the pattern command 121 is not a separate command, but instead is a field of the dispatch command 120. For example, in at least one embodiment a field of the dispatch command 120 indicates a selected access pattern, and in response to the dispatch command 120 the programmable table 106 stores the selected access pattern. Thus, in some embodiments a single command is used both to program the programmable table 106 with an access pattern, and to trigger dispatch of the thread groups 123 to the shader 110.
At block 602, the programmable table 106 receives the pattern command 121. In response, at block 604, the programmable table 106 selects and stores the access pattern 121. For example, in some embodiments the processing unit 100 stores a set of pre-defined access patterns, and the pattern command 121 indicates a selection of one of the set of pre-defined access patterns. In response to the pattern command 121, programmable table stores the selected access pattern indicated by the pattern command 121. In other embodiments, the pattern command 121 itself indicates access pattern 125 is a field of the command.
At block 606, the dispatch unit 104 receives the dispatch command 120. In response, at block 608, the dispatch unit 104 performs a dispatch walk of the thread groups 123 according to the access pattern 125 that was programmed at block 604. That is, the dispatch unit 104 uses the access pattern 125 to determine the dispatch order 122 of the thread groups 123. The dispatch unit 104 dispatches individual thread groups to the shader 110 according to the determined dispatch order.
A computer readable storage medium may include any non-transitory storage medium, or combination of non-transitory storage media, accessible by a computer system during use to provide instructions and/or data to the computer system. Such storage media can include, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic hard drive), removably attached to the computing system (e.g., an optical disc or Universal Serial Bus (USB)-based Flash memory) or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).
In some embodiments, certain aspects of the techniques described above may implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer readable storage medium. The software can include the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer readable storage medium can include, for example, a magnetic or optical disk storage device, solid state storage devices such as Flash memory, a cache, random access memory (RAM) or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.
Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device may not be required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.
Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.
