1. Technical Field
The present invention relates generally to a design structure, and in particular to a design structure for implementing dynamically scalable queues for transferring input/output (IO) data from an IO device of a computer system.
2. Description of the Related Art
Connections of Input/Output (IO) devices to computer systems and the associated transfer of data to and from the connected computer system are typically supported by one of several available hardware devices and associated protocols. In most conventional computer systems, the transfer protocol utilized for local interconnection of these IO devices is Peripheral Component Interconnect (PCI) Express. PCI Express (supported by specific hardware) is an implementation of the PCI computer bus that enables faster physical layer communications via use of a network of serial interconnects (in lieu of a single bus). PCI Express utilizes a single hub with many pins on the mainboard to enable switching and parallel data transfers.
The higher speeds accomplished by PCI Express has enabled PCI express to become the new backplane standard in a majority of personal computers. This is also due in part to PCI Express' design, which enables PCI Express to be completely transparent to software developers. Thus, an operating system designed for PCI is able to boot in a PCI Express system without any code modification.
Conventional methods for enabling IO data transfer include the utilization of IO queues. However, developing queues for IO devices is currently application specific, particularly when transferring data via PCI Express. PCI Express utilizes a variable size packet-driven serial protocol to transfer data. A queuing structure is required to execute these transfers in a coherent manner. These IO queues are statically configured and support only a single type of data transfer well. For example, if the IO devices that drive the traffic are varied in (1) the sizes of transfers and/or (2) the number of outstanding transactions on the link, developing the queues becomes a choice of exclusively supporting (a) many large transfers, (b) a few large transfers, (c) many small transfers, or (d) a few small transfers. Each category of data transfer operates best at a particular (single) type of queue configuration and losses operational quality for all other types of transfers and corresponding queue configurations.
The determination of which queue configuration works best for the particular IO transfer depends on what the computer system (or executing application) requests/requires. Thus, when the system/application is concurrently or sequentially providing different combinations of sizes and numbers of transactions, the statically-configured IO queues are unable to deliver high performance on all of the various configurations. The present invention recognizes and corrects this limitation in the existing IO data transfer methods, particularly those that utilize PCI Express.
Disclosed are a method, computer system, and PCI Express device/protocol for enabling a design structure that implements high performance IO data transfers for multiple, different IO configurations, which include variable packet sizes and/or variable/different numbers of transactions on the IO link. PCI Express protocol is enhanced to support utilization of counters and dynamically variable queue sizes. In addition to the standard queue entries, several (or a selected number of) dynamically changeable queue entries are provided/reserved and a dynamic queue modification (DQM) logic is provided within the enhanced PCI Express protocol. The DQM logic monitors ongoing, current data transfer and manages when the size(s) of the queue entries are modified (increased or decreased) based on current data traffic transmitting on the PCI Express IO link.
When data traffic tends towards a single stream of large data packets, queue entries are automatically combined and utilized to transfer the large data as quickly as possible. However, if the data traffic tends towards smaller data packets, the queue entries are broken up into many independent entries to handle the individual, smaller data packets. The enhanced PCI Express protocol provides an equilibrium point at which many large data packets are transferred efficiently, while imposing a limit on the number of each size of packets outstanding.
The above as well as additional objectives, features, and advantages of the present invention will become apparent in the following detailed written description.
The invention itself, as well as a preferred mode of use, further objects, and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:
The present invention provides a method, computer system, and PCI Express device/protocol for enabling high performance IO data transfers for multiple, different IO configurations, which include variable packet sizes and/or variable/different numbers of transactions on the IO link. PCI Express protocol is enhanced to support utilization of counters and dynamically variable queue sizes. In addition to the standard queue entries, several (or a selected number of) dynamically changeable queue entries are provided/reserved and a dynamic queue modification (DQM) logic is provided within the enhanced PCI Express protocol. The DQM logic monitors ongoing, current data transfer and manages when the size(s) of the queue entries are modified (increased or decreased) based on current data traffic transmitting on the PCI Express IO link.
When data traffic tends towards a single stream of large data packets, queue entries are automatically combined and utilized to transfer the large data as quickly as possible. However, if the data traffic tends towards smaller data packets, the queue entries are broken up into many independent entries to handle the individual, smaller data packets. The enhanced PCI Express protocol provides an equilibrium point at which many large data packets are transferred efficiently, while imposing a limit on the number of each size of packets outstanding.
With reference now to the figures, and in particular to
As provided within the invention, the PC1 Express specification/protocol defines data sizes to be between 1 to 4096 Bytes in length. The transfer of this range of data sizes in a coherent structure requires large packets be broken into manageable chunks. With conventional implementation, if a queue has many entries for small data packets, a large data packet would consume all the queue entries or take a long time to execute when/if one queue entry was utilized to transfer the entire data packet. However, with the invention, the sizes of these queues are variable and depend on the type of traffic actually received (rather then statically configured for the traffic that is expected). Also, the queues are broken into entries that are able to handle the data expected. These entries are either many small entries or a few large entries or a combination of both.
In addition to the above hardware components, various features of the invention may be implemented via software executing on CPU 102. For illustrative purposes, those software components are represented within system memory 109 as operating system (OS) 120 and applications 122. Program code of OS 120 and applications 122 execute on CPU 102 and may generate IO data that is transferred to a PCI connected device (not specifically shown) via PCI bus controller 114. Actual processing/manipulation of data during data transfer from the processor to the PCI device via/through the PCI fabric, as well as processing/manipulation of data received from the connected PCI device utilizing dynamically configurable IO queues is described in detail below.
PCI bridge 212 is also coupled to PCI bus 116, to which several PCI compliant devices 232, 234, 236 are coupled. Each of these PCI compliant devices 232, 234, 236 has a request/grant pair of buses 230, 228, 226, respectively. The request/grant buses 230, 228, 226 are coupled to PCI bus arbiter 220. PCI bus 116 is coupled to PCI bus interface 218. PCI bus interface 218 is coupled by a two way bus to PCI transaction queues 216. PCI transaction queues 216 are shown in greater details in
In operation, when the slave circuit in PCI bus interface 218 receives the data, the circuit processes the data into the PCI transaction queues. Concurrently, DQM logic 118 performs the necessary operations to modify queue sizes, if required, based on the data characteristics. The process by which the size(s) of the queue entries are modified is described in details below with reference to
Those of ordinary skill in the art will appreciate that the hardware depicted in
The number of transactions supported and queue entries provided are finite based on a maximum amount of available space allocated to the queues. In the exemplary embodiment, the maximum size single queue supported contains a single queue entry of 2096 bytes, and the queue is capable of supporting up to 32 entries at 64 bytes each. Also, in the described embodiment, the queue entries are sized at 512 bytes each, resulting in only four (4) queue entries being available during system initialization (device power on or setup).
Processing of the inventive steps is completed via DQM logic 118 once the initial parameters are established. According to the invention, DQM logic 118 comprises one or more performance counters, which take on one of several predefined characteristics based on the number of outstanding transactions (NTR). To understand the processing completed by the utility, the following parameters are defined and utilized within the example DQM logic 118:
Parameter NTR is reset after reaching the reset limit (RLT). In the specific illustrative embodiment, when NTR reaches RLT, an evaluation of queue entry sizes is completed by DQM logic 118. This evaluation may occur only once if RLEn is set to “on” (e.g., the value of RLEn=1). However, when RLEn is not set to “on,” (e.g., the value of RLEn=0), the evaluation of queue entry sizes is triggered and completed every time NTR reaches RLT. DQM logic 118 performs the consolidation or expansion of queue entries within queues 216 based on a series of conditions, which are presented below. Each condition includes a determination of the number of outstanding transactions, where P(x)% is the maximum threshold percentage for that particular number of transactions to trigger the resize operation. Thus:
(a) If N64 is greater than P1% of NTR, set the queues to 64 B entries;
(b) If N256 is greater than P2% of NTR, set the queues to 256 B entries;
(c) If N512 is greater than P3% of NTR, set the queues to 512 B entries;
(d) If N64+N512 is greater than P4% of NTR set the queues to 256 B;
(e) If N256+N512 is greater than P5% or NTR set queues to 256 B;
(f) If N256+N64 is greater than P6% of NTR set queues to 128 B; and
(g) If none of the above proves to be true, the utility leaves the queues in the current configuration.
Turning now to
At block 308, the counter values are compared to a set of pre-established threshold values (one for each counter), and a determination is made at block 310 whether the counter values have reach the associated threshold. If any one of the counter values have reached the associated threshold, DQM logic 118 initiates a resizing of the queue entries to accommodate for that particular sized data packet, as shown by block 312.
Returning to decision block 314, if DQM logic 118 determines that the number of smaller data packets outstanding is not greater than the associated threshold for that size packet, DQM logic 118 checks, as depicted at block 318, whether there are a greater number of larger data packets outstanding than the associated threshold for that size packet. When there is a greater number of outstanding larger data packets than the associated threshold, DQM logic 118 combines or consolidates two or more or the queue entries, as shown at block 320, to generate larger queue entries to accommodate a greater number of larger data packets. For example, assuming there are four queue entries having an initial 512 byte capacity and there are only two large data transfers outstanding at a time, DQM logic 118 consolidates the queue to two (2) entries at 1024 bytes each. The performance counters are thus utilized to show trends in the data traffic, and DQM logic 118 is eventually able to change the queue sizes based on the performance counters in order to gain better performance in different IO data transfer situations. The monitoring and resizing of queue entries continues for additional data traffic received as indicated by the return to block 304 from each end point of the sub-process.
Design process 510 preferably employs and incorporates hardware and/or software modules for synthesizing, translating, or otherwise processing a design/simulation functional equivalent of the components, circuits, devices, or logic structures shown in
Design process 510 may include hardware and software modules for processing a variety of input data structure types including netlist 580. Such data structure types may reside, for example, within library elements 530 and include a set of commonly used elements, circuits, and devices, including models, layouts, and symbolic representations, for a given manufacturing technology (e.g., different technology nodes, 32 nm, 45 nm, 90 nm, etc.). The data structure types may further include design specifications 540, characterization data 550, verification data 560, design rules 570, and test data files 585 which may include input test patterns, output test results, and other testing information. Design process 510 may further include modules for performing standard circuit design processes such as timing analysis, verification, design rule checking, place and route operations, etc.
Design process 510 employs and incorporates well-known logic and physical design tools such as HDL compilers and simulation model build tools to process design structure 520 together with some or all of the depicted supporting data structures to generate a second design structure 590. Similar to design structure 520, design structure 590 preferably comprises one or more files, data structures, or other computer-encoded data or instructions that reside on transmission or data storage media and that when processed by an ECAD system generate a logically or otherwise functionally equivalent form of one or more of the embodiments of the invention shown in
Design structure 590 may also employ a data format used for the exchange of layout data of integrated circuits and/or symbolic data format (e.g. information stored in a GDSII (GDS2), GL1, OASIS, map files, or any other suitable format for storing such design data structures). Design structure 590 may comprise information such as, for example, symbolic data, map files, test data files, design content files, manufacturing data, layout parameters, wires, levels of metal, vias, shapes, data for routing through the manufacturing line, and any other data processed by semiconductor manufacturing tools to fabricate embodiments of the invention as shown in
As a final matter, it is important that while an illustrative embodiment of the present invention has been, and will continue to be, described in the context of a fully functional computer system with installed management software, those skilled in the art will appreciate that the software aspects of an illustrative embodiment of the present invention are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the present invention applies equally regardless of the particular type of signal bearing media used to actually carry out the distribution. Examples of signal bearing media include recordable type media such as floppy disks, thumb drives, hard disk drives, CD ROMs, DVDs, and transmission type media such as digital and analogue communication links.
While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention.
The present application is a continuation-in part of U.S. patent application Ser. No. 11/466,142, filed Aug. 22, 2006. Benefit of priority is hereby claimed under 35 U.S.C. §120 to U.S. patent application Ser. No. 11/466,142, which is incorporated by reference herein in its entirety and for all purposes.
Number | Date | Country | |
---|---|---|---|
Parent | 11466142 | Aug 2006 | US |
Child | 12132445 | US |