The technology of the disclosure relates generally to Peripheral Component Interconnect (PCI) express (PCIE) links and, more particularly, to long distance PCIE links.
Computing devices have evolved from their early forms that were large and had limited use into compact, multifunction, multimedia devices. The increase in functionality has come, in part, as a function of using integrated circuits (ICs) in place of the original vacuum tubes. Many computing devices include multiple ICs having different dedicated functions.
Various internal buses may be used to exchange data between the ICs, such as Inter-integrated circuit (I2C), serial AT attachment (SATA), serial peripheral interface (SPI), or other serial interfaces. One popular bus is based on the Peripheral Component Interconnect (PCI) express (PCIE) standard published by the PCI Special Interest Group (PCI-SIG), PCIE is a high-speed point-to-point serial bus. PCIE version 4 was officially announced on Jun. 8, 2017 and version 5 has been preliminary proposed at least as early as June 2017 with expected release in 2019.
PCIE is an ordered and reliable link. To help effectuate this order and reliability, PCIE uses, amongst other tools, a credit system, that tells a transmitter how much data a receiver can manage. The transmitter uses a credit with each packet of data sent to the receiver, and then, if the transmitter exhausts the available credits, the transmitter waits for the receiver to return a credit for a managed packet. PCIE initially started as a short distance chip-to-chip or chip-to-card communication link, with typical distances under ten centimeters (10 cm) and usually under 1 cm. These short distances meant that credits from the receiver were rapidly returned. However, the simplicity of PCIE has led to its adoption in environments that have substantially longer distances. For example, in an automotive setting, distances on the order of ten meters (10 m) may not be unusual. In such instances, the transmitter may use all of the credits before the first packet even arrives at the receiver. The transmitter then waits for the packet to arrive and the receiver to return the credit. One way to decrease this latency is to advertise more credits at the receiver. However, because PCIE is reliable, for the receiver to advertise more credits, the receiver must have sufficient buffer space to handle packets corresponding to each of those credits. Similarly, the transmitter must have sufficient replay buffers to store each packet until a credit or acknowledgment is returned. These buffers use relatively large amounts of space in the silicon of the devices and thus increase the cost of the devices. As link distances increase, the amount of buffers required to utilize full link bandwidth increases, adding to the size and cost of the device. Thus, there needs to be a way to reduce the size and cost of the devices coupled to long PCIE links while keeping latency to a minimum.
Aspects disclosed in the detailed description include systems and methods for reducing latency on long distance point-to-point links. In an exemplary aspect, the point-to-point link is a Peripheral Component Interconnect (PCI) express (PCIE) link. A receiver on the PCIE link advertises infinite or unlimited credits. A transmitter sends packets to the receiver. If the receiver's buffers fill, the receiver, contrary to PCIE doctrine, drops the packet and returns a negative acknowledgement (NAK) packet to the transmitter. The transmitter, on receipt of the NAK packet, resends packets beginning with the one for which the NAK packet was sent. By the time these resent packets arrive, the receiver will have had time to manage the packets in the buffers and be ready to receive the resent packets. This process results in an overall reduction of latency relative to the normal Kw approach without requiting additional butters.
In this regard in one aspect, a method of communicating over a point-to-point communication link is disclosed. The method includes, at a receiver, receiving packets from a transmitter until a buffer is full. The method also includes, responsive to the buffer being full, sending a NAK packet to the transmitter. The method also includes receiving retransmitted packets after sending the NAK packet to the transmitter.
In another aspect, an apparatus is disclosed. The apparatus includes a receiver. The receiver includes a communication link interface configured to be coupled to a communication link. The receiver also includes a butler configured to store packets received through the communication link interface. The receiver also includes a control system. The control system, responsive to the buffer being filled with packets, is configured to send a NAK packet to a transmitter through the communication link interface.
With reference now to the drawing figures, several exemplary aspects of the present disclosure are described. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.
Aspects disclosed in the detailed description include systems and methods for reducing latency on long distance point-to-point links. In an exemplary aspect, the point to point link is a Peripheral Component interconnect (PCI) express (PCIE) link. A receiver on the PCIE link advertises infinite or unlimited credits. A transmitter sends packets to the receiver. If the receiver's buffers fill, the receiver, contrary to PCIE doctrine, drops the packet and returns a negative acknowledgement (NAK) packet to the transmitter. The transmitter, on receipt of the NAK packet, resends packets beginning with the one for which the NAK packet was sent. By the time these resent packets arrive, the receiver will have had time to manage the packets in the buffers and be ready to receive the resent packets. This process results in an overall reduction of latency relative to the normal PCIE approach without requiring additional buffers.
A brief overview of a computing system with PCIE links is provided with reference to
In this regard,
In this regard, the process 300 begins much as the process outlined in
If, however, there is an issue with the TLP packet, the process 300 has various ways of handling, depending on the nature of the issue. Thus, if the answer to block 306 is yes, the physical layer indicates the TLP packet was nullified, then the control system determines if the CRC is equal to logical NOT of the received value (block 318). If the answer to block 318 is yes, then the TLP packet is discarded and any storage allocated is freed (block 320) before the process ends (block 322). Likewise, if the answer to block 318 is no, or the answer to block 308 is no, then the control system indicates an error: bad TLP packet (block 324).
If the answer to block 310 is no, the sequence number is not correct, then the control system checks whether the received sequence number is in a window (2k) of sequence numbers before the expected sequence number. This check is made using a modulo 4096 operand on the difference of the expected sequence number from the received sequence number compared to 2048 (2k) (block 326). If the answer to block 326 is no, then the control system concludes that the TLP packet is a bad TLP packet (block 324). If the received sequence number is in the window, the PCIE protocol assumes that this is a packet for which an ACK was previously sent but not received for some reason and for which the transmitter has sent a duplicate. This duplication causes the receiver to resend the ACK through an ACK transmission (block 334).
Once there is a determination of a bad TLP packet at block 324, or after block 312 is answered affirmatively (i.e., the buffers are full), the control system determines if the NAK_SCHEDULED flag is clear (block 328) to see if a NAK packet has already been sent. If the flag is set, meaning there is already a NAK packet pending, then the control system discards the TLP packet and frees any allocated storage (block 330), and the process ends (block 316). If, however, the flag is clear at block 328, then the control system sends a NAK data link layer packet (DLLP) and sets the NAK_SCHEDULED flag (block 332).
Additionally, if block 326 is answered affirmatively, there is a duplicate, the control system schedules an ACK DLLP for transmission (block 334) and then moves to block 330 previously described.
In the absence of the present disclosure, a transmitter may run out of credits even though the buffers of the receiver are not full. This situation is exacerbated on long PCIE links where the length of the link uses all of the credits before the first packet arrives at the receiver. This situation is illustrated in simplified form in
Exemplary aspects of the present disclosure reduce this latency by allowing the receiver to publish infinite credits and drop packets when the buffers are full. When the receiver drops a packet, a NAK packet is sent indicating what sequence number was lost, and the transmitter resends the packet and all packets with higher sequence numbers that had been sent before arrival of the NAK packet. If the buffer size on the receiver matches the transfer rate, then no packets should be dropped. This situation is illustrated by signal flow 400B of
If for some reason, the buffers cannot handle the transfer rate, then the buffers will fill and begin to drop packets. At the point when the buffer is full, a NAK packet is sent to the transmitter to alert the transmitter to resend packets. While the use of a NAK packet to resend packets is known, it has never been used for intentionally dropped packets resulting from full buffers. However, because it is known to use NAK packets to resend packets, no change in the transmitter is required and backwards compatibility is maintained.
The systems and methods for reducing latency on long distance point-to-point links according to aspects disclosed herein may be provided in or integrated into any processor-based device. Examples, without limitation, include a set top box, an entertainment unit, a navigation device, a communications device, a fixed location data unit, a mobile location data unit, a global positioning system (GPS) device, a mobile phone, a cellular phone, a smart phone, a session initiation protocol (SIP) phone, a tablet, a phablet, a server, a computer, a portable computer, a mobile computing device, a wearable computing device (e.g., a smart watch, a health or fitness tracker, eyewear, etc.), a desktop computer, a personal digital assistant (PDA), a monitor, a missile, a computer monitor, a television, a tuner, a radio, a satellite radio, a music player, a digital music player, a portable music player, a digital video player, a video player, a digital video disc (DVD) player, a portable digital video player, an automobile, a vehicle component, avionics systems, a drone, and a multicopter.
In this regard,
With continued reference to
With continued reference to
With continued reference to
Those of skill in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithms described in connection with the aspects disclosed herein may be implemented as electronic hardware, instructions stored in memory or in another computer readable medium and executed by a processor or other processing device, or combinations of both. The devices described herein may be employed in any circuit, hardware component, IC, or IC chip, as examples. Memory disclosed herein may be any type and size of memory and may be configured to store any type of information desired. To clearly illustrate this interchangeability, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. How such functionality is implemented depends upon the particular application, design choices, and/or design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The various illustrative logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration),
The aspects disclosed herein may be embodied in hardware and in instructions that are stored in hardware, and may reside, for example, in Random Access Memory (RAM), flash memory, Read Only Memory (ROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), registers, a hard disk, a removable disk, a CD-ROM, or any other form of computer readable medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a remote station. In the alternative, the processor and the storage medium may reside as discrete components in a remote station, base station, or server.
It is also noted that the operational steps described in any of the exemplary aspects herein are described to provide examples and discussion. The operations described may be performed in numerous different sequences other than the illustrated sequences. Furthermore, operations described in a single operational step may actually be performed in a number of different steps. Additionally, one or more operational steps discussed in the exemplary aspects may be combine& It is to be understood that the operational steps illustrated in the flowchart diagrams may be subject to numerous different modifications as will be readily apparent to one of skill in the art. Those of skill in the art will also understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.