Patent Application
20230169621
This disclosure relates generally to computing technologies, and more specifically to accessibility of on-chip memory of a processor by a compute shader.
Central processing unit (CPU) and graphics processing unit (GPU) play significant roles in nowadays computing technologies. Architecturally, a CPU is composed of several cores with lots of cache memory, and is optimized for serial processing. In contrast, a GPU is composed of hundreds of cores that can handle thousands of threads simultaneously, and is optimized for parallel processing.
From an operational perspective, a CPU processes various functions therein in a same way. For example, functions can access resources at any memory location without limitations defined in the CPU, and each function is assigned with a general-purpose register. By contrast, a GPU defines a plurality of different functions (also called shaders). The various functions are associated with their dedicated accessible resources.
With the development of semiconductor technology, the computing power of mobile devices, such as smartphones, is getting stronger and stronger due to more and more powerful CPUs integrated therein. There is an increasing trend to implement mobile GPUs into mobile devices to boost parallel processing capabilities. Therefore, it is needed to provide techniques for implementing mobile GPU into the mobile devices.
A method, device and non-transitory computer-readable medium are disclosed for processing a data workload including related segments of data. A compute shader is enabled with accessibilities to memory space (e.g., a buffer) of an on-chip memory of a processor (e.g., a GPU), such that performance of the processor is improved by utilizing the bandwidth of an on-chip memory (e.g., a buffer) of the processor and saving the bandwidth of an off-chip memory (e.g., a device memory) that is not integrated on a monolithic chip of the processor.
In some instances, a method is provided. The method includes loading a first segment of data to a buffer in an on-chip memory of the processor, receiving a first trigger signal for the first segment of data, instantiating a first compute shader in response to the first trigger signal and loading the first segment of data from the buffer to the first compute shader for execution by the first compute shader. The buffer is used for temporarily storing one or more segments of data of the data workload. The first trigger signal is generated in response to the first segment of data being loaded to the buffer.
In some variations, the trigger signal is generated by circuitry associated with the buffer when the first segment of data is written into the buffer.
In some examples, the method further includes sending information of the first segment of data to the first compute shader. The information of the first segment of data comprises a start address of the first segment of data in the buffer and a size of the first segment of data. The first segment of data in the buffer is located based on the information of the first segment of data.
In some instances, the method further includes closing the first compute shader after the first compute shader completes processing of the first segment of data.
In some variations, the method further includes loading a second segment of data to the buffer, receiving a second trigger signal, instantiating a second compute shader in response to the second trigger signal and loading the second the first trigger signal is generated in response to the first segment of data being loaded to the buffer from the buffer to the second compute shader for execution by the second compute shader. The second trigger signal is generated in response to the second segment of data being loaded to the buffer.
In some examples, the method further includes sending information of the second segment of data to the second compute shader. The information of the second segment of data comprises a start address of the second segment of data in the buffer and a size of the second segment of data. The second segment of data in the buffer is located based on the information of the second segment of data.
In some instances, wherein the number of instantiated compute shaders that run concurrently is less than or equal to a predefined value. The predefined value is determined based on a number of threads included in each segment of data associated with an instantiated compute shader and a preset maximum number of threads that can run concurrently.
In some variations, the method further includes monitoring the buffer, stopping loading additional segments of data to the buffer in response to the buffer reaching a preset capacity. The additional segments of data are segments of the data workload. Memory space allocated in the buffer for storing the first segment of data is released after the processor loads the first segment of data from the buffer to the first compute shader.
In some examples, a device is provided. The device includes one or more processors and a non-transitory computer-readable media storing computer instructions thereon. When the instructions are executed by the one or more processors, causing the one or more processors to further perform the steps of loading a first segment of data to a buffer in an on-chip memory of the processor, receiving a first trigger signal for the first segment of data, instantiating a first compute shader in response to the first trigger signal and loading the first segment of data from the buffer to the first compute shader for execution by the first compute shader. The buffer is used for temporarily storing one or more segments of data of the data workload. The first trigger signal is generated in response to the first segment of data being loaded to the buffer.
In some instances, the trigger signal is generated by circuitry associated with the buffer when the first segment of data is written into the buffer.
In some variations, the one or more processers of the device executes the instructions to perform an additional step of sending information of the first segment of data to the first compute shader. The information of the first segment of data comprises a start address of the first segment of data in the buffer and a size of the first segment of data. The first segment of data in the buffer is located based on the information of the first segment of data.
In some examples, the one or more processers of the device executes the instructions to perform an additional step of closing the first compute shader after the first compute shader completes processing of the first segment of data.
In some instances, the one or more processers of the device executes the instructions to perform additional steps of loading a second segment of data to the buffer, receiving a second trigger signal, instantiating a second compute shader in response to the second trigger signal and loading the second the first trigger signal is generated in response to the first segment of data being loaded to the buffer from the buffer to the second compute shader for execution by the second compute shader. The second trigger signal is generated in response to the second segment of data being loaded to the buffer.
In some variations, the one or more processers of the device executes the instructions to perform an additional step of sending information of the second segment of data to the second compute shader. The information of the second segment of data comprises a start address of the second segment of data in the buffer and a size of the second segment of data. The second segment of data in the buffer is located based on the information of the second segment of data.
In some examples, the number of instantiated compute shaders that run concurrently is less than or equal to a predefined value. The predefined value is determined based on a number of threads included in each segment of data associated with an instantiated compute shader and a preset maximum number of threads that can run concurrently.
In some instances, the one or more processers of the device executes the instructions to perform additional steps of monitoring the buffer, stopping loading additional segments of data to the buffer in response to the buffer reaching a preset capacity. The additional segments of data are segments of the data workload. Memory space allocated in the buffer for storing the first segment of data is released after the processor loads the first segment of data from the buffer to the first compute shader.
In some variations, a non-transitory computer-readable medim is provided. Instructions stored on the non-transitory computer-readable medim, when executed by one or more processors, cause the one or more processors to perform the steps of loading a first segment of data to a buffer in an on-chip memory of the processor, receiving a first trigger signal for the first segment of data, instantiating a first compute shader in response to the first trigger signal and loading the first segment of data from the buffer to the first compute shader for execution by the first compute shader. The buffer is used for temporarily storing one or more segments of data of the data workload. The first trigger signal is generated in response to the first segment of data being loaded to the buffer.
In some examples, the trigger signal is generated by circuitry associated with the buffer when the first segment of data is written into the buffer.
In some instances, the instructions stored on the non-transitory computer-readable medim, when executed by one or more processors, cause the one or more processors to perform an additional step of sending information of the first segment of data to the first compute shader. The information of the first segment of data comprises a start address of the first segment of data in the buffer and a size of the first segment of data. The first segment of data in the buffer is located based on the information of the first segment of data.
In some variations, the instructions stored on the non-transitory computer-readable medim, when executed by one or more processors, cause the one or more processors to perform an additional step of closing the first compute shader after the first compute shader completes processing of the first segment of data.
In some examples, the instructions stored on the non-transitory computer-readable medim, when executed by one or more processors, cause the one or more processors to perform additional steps of loading a second segment of data to the buffer, receiving a second trigger signal, instantiating a second compute shader in response to the second trigger signal and loading the second the first trigger signal is generated in response to the first segment of data being loaded to the buffer from the buffer to the second compute shader for execution by the second compute shader. The second trigger signal is generated in response to the second segment of data being loaded to the buffer.
In some instances, the instructions stored on the non-transitory computer-readable medim, when executed by one or more processors, cause the one or more processors to perform an additional step of sending information of the second segment of data to the second compute shader. The information of the second segment of data comprises a start address of the second segment of data in the buffer and a size of the second segment of data. The second segment of data in the buffer is located based on the information of the second segment of data.
In some variations, the number of instantiated compute shaders that run concurrently is less than or equal to a predefined value. The predefined value is determined based on a number of threads included in each segment of data associated with an instantiated compute shader and a preset maximum number of threads that can run concurrently.
In some examples, the instructions stored on the non-transitory computer-readable medim, when executed by one or more processors, cause the one or more processors to perform additional steps of monitoring the buffer, stopping loading additional segments of data to the buffer in response to the buffer reaching a preset capacity. The additional segments of data are segments of the data workload. Memory space allocated in the buffer for storing the first segment of data is released after the processor loads the first segment of data from the buffer to the first compute shader.
The following detailed description is exemplary in nature and is not intended to limit the disclosure or the application and uses of the disclosure. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding background, summary and brief description of the drawings, or the following detailed description.
In the following detailed description, numerous specific details are set forth in order to provide a more thorough understanding of the disclosed technology. However, it will be apparent to one of ordinary skill in the art that the disclosed technology may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.
When the GPU 160 renders an object (e.g., a visual image), the GPU 160 performs a number of functions following a sequence of steps, which is called a rendering pipeline. At each step, the GPU 160 performs a specialized function called a shader. The GPU 160 renders the object by performing the various functions (e.g., the shaders) following the steps defined in the rendering pipeline, so as to generate a desired final product. For instance, the GPU 160 may render a visual image following a rendering pipeline to generate a desired photorealistic image for displaying.
A GPU defines a plurality of different functions (e.g., the shaders) originally used for shading in graphic scenes. A shader is a type of computer program used for a specialized function. The plurality of shaders defined in the GPU include 2D shaders, such as a pixel shader, and 3D shaders, such as a vertex shader. For example, a pixel shader, also known as fragment shader, computes attributes (e.g., color, depth, etc.) of each fragment and outputs values for each pixel displayed on a screen. A fragment is a collection of values produced by a rasterizer that produces a plurality of primitives from an original image frame. Each fragment represents a sample-sized segment of a rasterized primitive. In some variations, a fragment has a size of one pixel. In another example, a vertex shader computes transformation of each vertex's 3D position in virtual space to a set of 2D coordinates for displaying on a screen, where a primitive uses vertices to reference points. The various shaders are associated with their dedicated accessible resources.
Among these shaders, compute shader (CS) is a relatively flexible one that is capable of performing any calculations (e.g., executing any type of shader) on the GPU thus supporting general-purpose computing on GPU (GPGPU). CS provides memory sharing and thread synchronization features allowing for implementation of more effective parallel programming methods. However, accessibility of CS to resources (e.g., memory storages) is limited by the existing graphic standard like Open Graphic Library (openGL) or Vulkan (which is an application programing interface (API) with focus on 2D and 3D graphics). According to the existing graphic standard, when accessing data output from the other types of shaders, CS can only access data stored in a main memory (e.g., the memory 190) of the device that is an off-chip memory and is not integrated on a monolithic chip of a GPU.
Various examples described in the present disclosure provides techniques to enable accessibility of a CS to memory space (e.g., a buffer) of an on-chip memory of a GPU, such that the GPU performance is improved by utilizing the bandwidth of an on-chip memory (e.g., a tile buffer) of the GPU and saving the bandwidth of an off-chip memory (e.g., a device memory) that is not integrated on a monolithic chip of the GPU. In some examples, a buffer (e.g., a tile buffer) is defined in the on-chip memory of the GPU for temporarily storing one or more data segments (e.g., tiles of data). A dependency relationship is established between the buffer and a CS launcher that instantiates one or more CSs. When a data segment is written into the buffer, circuitry associated with the buffer generates a trigger signal for the data segment. The circuitry associated with the buffer may be a logic IC that is integrated on the on-chip memory and electrically connected to the buffer. The trigger signal is sent to the CS launcher indicating that the data segment (e.g., a tile of data) is loaded into the buffer. After receiving the trigger signal, the CS launcher instantiates a CS (e.g., by calling a dispatch method). Once the data segment is ready in the buffer, the CS retrieves the data segment from the buffer and processes the data segment. The allocated memory space for the data segment in the buffer is released after the data segment is retrieved by the CS. After the CS completes processing of the data segment, the CS is closed. In some variations, capacity of the buffer may be continuously monitored. If the buffer does not exceed a preset capacity, additional data segments may be continuously loaded into the buffer. The circuitry associated with the buffer generates a trigger signal for each data segment written into the buffer. A plurality of trigger signals corresponding to a plurality of data segments are sent to the CS launcher to instantiate a plurality of CSs. The CS launcher instantiates one CS in response to each trigger signal. A maximum number of CSs that can run concurrently may be predefined in the GPU, and/or defined depending on actual implementation. When the GPU determines that the capacity of the buffer exceeds a preset capacity, the GPU may determine to stop loading data segments into the buffer and/or stop executing a preceding step that outputs the data segments.
The present disclosure provides techniques to establish dependency relationship between a buffer (e.g., the tile buffer 180 in the GPU 160) included in an on-chip memory of a GPU and a CS launcher that launchers a CS, such that the CS can directly retrieve data from the buffer, so as to improve the performance of the GPU by utilizing the bandwidth of the on-chip memory of the GPU.
In some examples, the process 500 performs a part of the pipeline 400 that includes tile-based rendering processes, as shown in
At block 510, the GPU 160 loads data from a preceding step (e.g., the Job 1 410 shown in
At block 520, the GPU 160 instantiates a CS through the CS launcher. The CS launcher when executed by the GPU 160 instantiates a CS in response to a received trigger signal. The CS launcher may instantiate a plurality of CSs in response to a plurality of trigger signals, where each CS is instantiated in response to one trigger signal. The maximum number of CSs that can be instantiated may be predefined in the GPU 160, and/or defined depending on actual implementation. For instance, the GPU 160 may define a workgroup including a number of threads (e.g., 256 threads) that can run concurrently. Each thread may be associated with a pixel included in a piece of workload. A tile may include a number of pixels, for example, 4×4 pixels. As such, 16 CSs may be instantiated and run concurrently when the size of the workgroup is set to be 256 threads. A workgroup may be defined as one-dimensional (1D), two-dimensional (2D) or three-dimensional (3D) space. Accordingly, the size of a CS may be defined in 1D, 2D or 3D. The size of the CS may be associated with the size of a tile that is processed by the CS.
At block 530, the GPU 160 loads the data from the tile buffer 180 to the CS. Once a tile is ready in the tile buffer 180, the instantiated CS may retrieve the tile from the tile buffer 180. In some instances, the CS obtains the tile from the tile buffer 180 based on the information of the tile, which may include the start address of the tile and/or the size of the tile.
After the CS retrieves the tile from the tile buffer 180, the memory space allocated in the tile buffer 180 for storing the tile may be released. Once the CS completes processing of the tile, the CS is closed.
In some variations, the GPU 160 continuously loads tiles from a preceding step into the tile buffer 180, as long as the tile buffer 180 does not reach a preset capacity. The preset capacity may be a maximum capacity of the tile buffer 180. Whenever a tile is ready in the tile buffer 180, the GPU 160 may instantiate a CS through the CS launcher and the CS may read the tile from the tile buffer 180. When a plurality of the tiles are ready in the tile buffer 180, the GPU 160 may instantiate a plurality of CSs through the CS launcher one by one, and each CS reads a tile from the tile buffer 180. In some variations, the GPU 160 may execute instructions to query execution time of one or more CSs and determine whether to stop loading data into the tile buffer 180 based on the results of the query. If the execution time of one or more CSs is beyond a preset time limit, the GPU may determine to stop loading additional tiles from the preceding step and/or to stop the preceding step. In some instances, the GPU 160 continuously monitors the capacity of the tile buffer 180. If the GPU 160 determines the tile buffer 180 reaches the preset capacity, the GPU 160 may determine to stop loading additional tiles from the preceding step and/or to stop the preceding step.
In some examples, a succeeding Job 2 635 is performed by a second pixel shader (pixel shader 2). Once the memory space 645 is loaded with the tile 605, the GPU 160 instantiates the pixel shader 2 through a pixel shader launcher 630. A pixel shader launcher 630 is a program executed by the GPU 160 to instantiate one or more pixel shaders. The pixel shader 2 reads the tile 605 stored in the memory space 645 of the tile buffer 180 and perform computations defined in the Job 2 635. Once the pixel shader 2 completes the processing of the tile 605, the pixel shader 2 is closed. The GPU 160 may continuously load tiles 605 from the preceding Job 1 620 to the tile buffer 180. Whenever a tile 605 is ready in the tile buffer 180, the GPU 160 may instantiate a pixel shader through the pixel shader launcher 630 to perform computations defined in the Job 2 635.
In some instances, a succeeding Job 2 655 is performed by a CS. The GPU 160 sends a trigger signal to a CS launcher 650 to instruct the CS launcher 650 to instantiate a CS for the Job 2 655. In some variations, the GPU 160 sends the information of the tile 605 to the CS launcher 650. The information of the tile 605 may be sent before or after the trigger signal is sent to the CS launcher. Once the memory space 645 is loaded with the tile 605, the GPU 160 instantiates a CS through the CS launcher 650. The CS reads the tile 605 stored in the memory space 645 of the tile buffer 180 and performs the computations defined in the Job 2 655. Once the CS completes the computing of the tile 605, the CS is closed.
In some examples, the GPU 160 may continuously load tiles 605 from the preceding Job 1 620 to the tile buffer 180. Whenever a tile 605 is ready in the tile buffer 180, the GPU 160 may instantiate a CS through the CS launcher 650 to perform computations defined in the Job 2 655. The CS launcher 650 may instantiate a plurality of CSs in response to a plurality of trigger signals, where each CS is instantiated in response to one trigger signal. A maximum number of CSs that can run concurrently may be predefined in the GPU, and/or defined depending on actual implementation. For instance, the GPU 160 may define a workgroup including a number of threads (e.g., 256 threads) that can run concurrently. Each thread may be associated with a pixel included in the frame 610. A tile 605 may include a number of pixels, for example, 4×4 pixels. As such, 16 CSs may be instantiated and run concurrently when the size of the workgroup is set to be 256 threads. A workgroup may be defined as one-dimensional (1D), two-dimensional (2D) or three-dimensional (3D) space. Accordingly, the size of a CS may be defined in 1D, 2D or 3D. The size of the CS may be associated with the size of a tile that is processed by the CS.
At block 710, the GPU 160 loads a tile 605 into the tile buffer 180 of the GPU 160. The tile 605 may be an output from a preceding step in the rendering pipeline. The size of the tile 605 may be defined by a user while defining the rendering pipeline. The tile 605 stored in the tile buffer 180 may be located based on information of the tile 605. The information of the tile 605 may include a start address of the tile 605 in the tile buffer 180 and/or a size of the tile 605.
At block 720, the GPU 160 sends a trigger signal to a CS launcher. The trigger signal is generated by the circuitry associated with the tile buffer 180 when the tile 605 is written into the tile buffer 180. The trigger signal may be generated at the beginning, in the middle or at the end of the process of writing the tile 605 into the tile buffer 180. The trigger signal is sent to the CS launcher after being generated. The GPU 160 monitors the tile buffer 180 and determines whether the tile buffer 180 reaches a preset capacity (e.g., a maximum capacity of the tile buffer 180). If the tile buffer 180 does not reach the preset capacity, the GPU 160 continuously loads tiles 605 from the preceding step. A trigger signal is generated for each tile 605 loaded into the tile buffer 180. As such, the GPU 160 sends a plurality of trigger signals to the CS launcher whenever a trigger signal is generated. The CS launcher is instructed to instantiate a plurality of CSs in response to the plurality of trigger signals, and each CS is instantiated for a respective trigger signal to process a respective tile 605 in the tile buffer 180. If the tile buffer 180 reaches the preset capacity, the GPU 160 may determine to stop loading tiles from the preceding step and/or stop the execution of the preceding step. In some instances, the GPU 160 sends the tile information of the tiles 605 to the CS launcher. The tile information may be sent before or after the trigger signals are sent to the CS launcher.
At block 730, the GPU 160 instantiates a CS through the CS launcher. Once the tile 605 is ready in the tile buffer 180, the CS launcher instantiates a CS to perform computations defined in a succeeding step in the rendering pipeline. When a plurality of tiles 605 are loaded into the tile buffer 180, the CS launcher instantiates a plurality of CSs one by one, where each CS processes a respective tile 605.
A maximum number of CSs that can run concurrently may be predefined in the GPU, and/or defined depending on actual implementation. For instance, the GPU 160 may define a workgroup including a number of threads (e.g., 256 threads) that can run concurrently. Each thread may be associated with a pixel included in the frame 610. A tile 605 may include a number of pixels, for example, 4×4 pixels. As such, 16 CSs may be instantiated and run concurrently when the size of the workgroup is set to be 256 threads. A workgroup may be defined as one-dimensional (1D), two-dimensional (2D) or three-dimensional (3D) space. Accordingly, the size of a CS may be defined in 1D, 2D or 3D. The size of the CS may be associated with the size of a tile 605 that is processed by the CS.
At block 740, the GPU 160 loads the tile 605 from the tile buffer 180 to the CS. Once the tile 605 is ready in the tile buffer 180, the CS retrieves the tile 605 from the tile buffer 180 and processes the tile 605. The CS may locate the tile 605 stored in the tile buffer 180 based on the tile information, which may include the start address of the tile 605 in the tile buffer 180 and/or the size of the tile 605. Memory space allocated for the tile 605 in the tile buffer 180 may be released after the CS retrieves the tile 605 from the tile buffer.
At block 750, the GPU 160 processes the tile 605 by executing the CS. After the CS completes processing of the tile 605, the CS is closed by the GPU 160 through the CS launcher. In some examples, the GPU 160 may execute instructions to query an execution time of one or more CSs that are instantiated to process the tiles 605. The GPU 160 may determine whether to stop loading tiles 605 from the preceding step and/or stop the execution of the preceding step based on the results of the query.
In some variations, the GPU 160 further causes display of an image based on one or more tiles 605 that are processed and output from a step performed by the CSs in the rendering pipeline. In some examples, the GPU 160 may cause display of the one or more tiles 605 one by one whenever a tile 605 is output from a CS. In some instances, the GPU 160 may cause display of the tiles 605 that are synchronized in a step performed by the CSs.
It is noted that the techniques described herein may be embodied in executable instructions stored in a computer readable medium for use by or in connection with a processor-based instruction execution machine, system, apparatus, or device. It will be appreciated by those skilled in the art that, for some embodiments, various types of computer-readable media can be included for storing data. As used herein, a “computer-readable medium” includes one or more of any suitable media for storing the executable instructions of a computer program such that the instruction execution machine, system, apparatus, or device may read (or fetch) the instructions from the computer-readable medium and execute the instructions for carrying out the described embodiments. Suitable storage formats include one or more of an electronic, magnetic, optical, and electromagnetic format. A non-exhaustive list of conventional exemplary computer-readable medium includes: a portable computer diskette; a random-access memory (RAM); a read-only memory (ROM); an erasable programmable read only memory (EPROM); a flash memory device; and optical storage devices, including a portable compact disc (CD), a portable digital video disc (DVD), and the like.
It should be understood that the arrangement of components illustrated in the attached Figures are for illustrative purposes and that other arrangements are possible. For example, one or more of the elements described herein may be realized, in whole or in part, as an electronic hardware component. Other elements may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other elements may be combined, some may be omitted altogether, and additional components may be added while still achieving the functionality described herein. Thus, the subject matter described herein may be embodied in many different variations, and all such variations are contemplated to be within the scope of the claims.
To facilitate an understanding of the subject matter described herein, many aspects are described in terms of sequences of actions. It will be recognized by those skilled in the art that the various actions may be performed by specialized circuits or circuitry, by program instructions being executed by one or more processors, or by a combination of both. The description herein of any sequence of actions is not intended to imply that the specific order described for performing that sequence must be followed. All methods described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context.
The use of the terms “a” and “an” and “the” and “at least one” and similar referents in the context of describing the invention (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The use of the term “at least one” followed by a list of one or more items (for example, “at least one of A and B”) is to be construed to mean one item selected from the listed items (A or B) or any combination of two or more of the listed items (A and B), unless otherwise indicated herein or clearly contradicted by context. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including, but not limited to,”) unless otherwise noted. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate the invention and does not pose a limitation on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.
All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.
