This patent application relates generally to data generation and transfer, and more specifically, to a selective acknowledgement framework utilizing bit vector arrays to track the transmission and reception of data packets.
In some examples, to enable bi-directional transit of data packets between network components, communication networks may use packet sequences or acknowledgement numbers to keep track of transferred data. However, only ideally and rarely are all of the data packets delivered to the network entities successfully and in order.
When one or more data packets are missed or lost during a transmission, a receiving entity may send a request for re-transmission to the transmitting entity. In some instances, the transmitting entity may respond with all of the data packets that were previously transmitted due to the network delay in receiving the acknowledgements from the receiving entity for successfully delivered data packets.
Features of the present disclosure are illustrated by way of example and not limited in the following figures, in which like numerals indicate like elements. One skilled in the art will readily recognize from the following that alternative examples of the structures and methods illustrated in the figures can be employed without departing from the principles described herein.
For simplicity and illustrative purposes, the present application is described by referring mainly to examples thereof. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. It will be readily apparent, however, that the present application may be practiced without limitation to these specific details. In other instances, some methods and structures readily understood by one of ordinary skill in the art have not been described in detail so as not to unnecessarily obscure the present application. As used herein, the terms “a” and “an” are intended to denote at least one of a particular element, the term “includes” means includes but not limited to, the term “including” means including but not limited to, and the term “based on” means based at least in part on.
The use of selective acknowledgements to trigger retransmissions has been shown to improve over tail latency without needing to enable priority flow controls (PFC) for high-performance networks. However, in some instances, tracking packets that are lost on the network and conveying that information from the receiver back to the transmitter may be quite complex, especially when the network latencies may be quite large.
Generally, exact sequence numbers of lost data packets may be conveyed to a transmitter in the form of a negative acknowledgement (NACKs) or a selective acknowledgement (SACKs) where such sequence numbers are stacked. In some instances, this may add significant complexity to the implementation on both the transmitter and receiver sides.
In some examples, a selective acknowledgement framework implemented according to the examples described may eliminate complexities via the use of bit-vectors across the transmitter, receiver, and acknowledgement packets. In some examples, the examples described may implement a compact form of representing different combinations of acknowledgements (ACKs) and negative acknowledgements (NACKs) when multiple packet drops occur.
According to some examples, a transmitter may transmit a segment or a group of one or more data packets to a receiver, wherein each data packet of the one or more data packets may be identified via a corresponding packet sequence number (PSN). The transmitter may include a transmission tracker having one or more transmit bit vector arrays that may store bit vector representations of the packet sequence numbers (PSNs) of the data packets.
In some examples, the transmitter may maintain at least two pointers. In some examples, a first pointer of the two pointers may include at least a highest scheduled packet sequence number (PSN) pointer that may store the packet sequence number (PSN) of the data packet from the segment of data packets scheduled for the next transmission. In some examples, a second pointer of the two pointers may include a contiguously acknowledged packet sequence number (PSN) pointer that stores a packet sequence number (PSN) of a last data packet of a sequence of data packets that may be acknowledged by the receiver as contiguously received from the segment of data packets so that no retransmission may be needed. In some examples, the last data packet may be identified in terms of the sequence of packet sequence numbers (PSNs) rather than a temporal sequence. The difference between the packet sequence numbers (PSNs) of the pointers may correspond to the packet sequence numbers (PSNs) of the in-flight data packets that may have been ejected from the transmitter but are yet to arrive at the receiver.
In some examples, a receiver that may receive the segment of data packets from the transmitter may also maintain a receiver tracker including one or more receive bit vector arrays. In some examples, the receiver tracker may be utilized to store bit vector representations of the packet sequence numbers (PSNs) of the data packets received from a segment of data packets. The size of each of the transmission tracker and the receiver tracker may be a variable of the Bandwidth Delay Product (BDP) of the network to accommodate all the in-flight data packets which may require tracking. The receiver may also maintain at least two pointers that may be based on the receiver tracker as well. In some examples, the at least two pointers may include a contiguously received packet sequence (PSN) pointer that may store the packet sequence number (PSN) of a last data packet of a sequence of data packets contiguously received from the segment of data packets. It may be appreciated that, in some examples, the last data packet may be identified based on the packet sequence number (PSN) sequence rather than a temporal order in which a data packet may be received. In addition, in some examples, a second pointer may include a highest received packet sequence number (PSN) pointer storing a packet sequence number (PSN) of a last data packet successfully received from the segment of data packets. Again, in some examples, a difference in the packet sequence numbers (PSNs) of the two pointers maintained by the receiver may correspond to the packet sequence numbers (PSNs) of the in-flight data packets.
In some examples, a receiver may generate an acknowledgement packet from values stored in a receiver tracker. In some examples, the acknowledgement packet may include at least a select acknowledgement bit vector, wherein each bit of the select acknowledgement bit vector may represent a corresponding data packet of a segment of data packets. Additionally, a value of each bit of the select acknowledgement bit vector may indicate an acknowledgement (ACK) or negative acknowledgement (NACK) of the corresponding data packet.
In order to enable a transmitter to identify corresponding data packets, an acknowledgement packet may also include a last contiguously acknowledged sequence number as an anchor value from where a select acknowledgement bit vector may be applicable. In some examples, the acknowledgement packet may also include a maximum sequence number that may represent a number of the packet sequence numbers (PSNs) being acknowledged as well. Furthermore, optionally, the acknowledgement packet may include an optional parameter of chained acknowledgements if there may be a series of acknowledgement packets for one segment of data packets.
Upon receipt of an acknowledgement packet by a transmitter, various values may be extracted and an update operation may be executed to update a transmission tracker for an identification of acknowledgements (ACKs) and/or the negative acknowledgements (NACKs) corresponding to a segment of data packets. For example, the update may include an exclusive OR operation between the select acknowledgement bit vector and one or more of the transmit bit vector arrays to eliminate data packets with duplicate acknowledgements. In some examples, the data packets associated with the negative acknowledgements (NACKs) may be re-transmitted upon updating the transmission tracker. In some examples, the transmitter may be configured with one or more timers that may sweep the transmit bit vector arrays so that upon timeout, the data packets associated with negative acknowledgements (NACKs) may be automatically re-transmitted.
In some examples, a selective acknowledgement scheme described above may mitigate the need for the transmitter and the receiver to exchange the complete packet sequence numbers (PSNs) of the data packets as each data packet may be represented by a bit in the select acknowledgement bit vector. In some examples, a combination of a last contiguously acknowledged sequence number as an anchor value and a maximum sequence number may enable a network entity to accurately identify the data packets that may be referred to within the acknowledgement packet. In some examples, such a compact communication scheme may improve the efficiency of the network while reducing latency and conserving network resources as it may mitigate a need for exchanging all packet sequence numbers (PSNs) of data packets that may be acknowledged or negative acknowledged.
In some examples, a number of data packets in the segment of data packets 150 may be a configurable quantity within the network 190. In particular, the configurable quantity may be based on various factors, including but not limited to a capacity of the transmitter 102, a capacity of the receiver 104, a network bandwidth, etc. Moreover, each data packet 152, 154, ... of the segment of data packets 150 may be uniquely identified by a corresponding packet sequence number (PSN).
In some examples, corresponding packet sequence numbers (PSNs) of the plurality of data packets e.g., 152, 154, ... may be in sequential order. In an example, the packet sequence numbers (PSNs) of the segment of data packets 150 may be maintained within the transmitter 102 in one or more transmit bit vector arrays that may form a transmission tracker 124. As the data packets are received by the receiver 104, the receiver 104 may transmit acknowledgements (ACKs) for each data packet received correctly and negative acknowledgements (NACKs) for data packets that may include errors or were never received.
The receiver 104 may transmit one acknowledgement packet for a given segment of data packets received in a data stream. For example, one acknowledgement packet 160 may be transmitted for the segment of data packets 150. The acknowledgement packet 160 may include at least three fields, viz., a select acknowledgement bit vector (SelAckBitVector) 162, a last contiguously acknowledged sequence number (LastContiguouslyAckedSeqNumber) 164 and a maximum sequence number (MaxSequenceNumber) 166. In an example, the acknowledgement packet 160 may also include an additional field of ChainedAcks 168, wherein a flag which when set may represent that a total acknowledgement for the segment of data packets 150 may be sent over multiple acknowledgement packets.
In some examples, a data packet (e.g., data packet 152 of the segment of data packets 150) may be represented by a bit in the SelAckBitVector 162. Furthermore, in some examples, a value of each bit of the SelAckBitVector 162 may represent one of an acknowledgement (ACK) or a negative acknowledgement (NACK) of the corresponding data packet of the segment of data packets 150. In an example, the transmitter 102 may be configured with a predetermined timeout so that if neither an acknowledgement (ACK) nor a negative acknowledgement (NACK) is received from the receiver 104 for a data packet, such a data packet may be automatically retransmitted after the timeout. Furthermore, information regarding the data packets may be exchanged between the transmitter 102 and the receiver 104 in a more compact form that may include bit vector representations of a plurality of packet sequence numbers (PSNs). Accordingly, in some examples, the acknowledgement packet 160 may include the LastContiguouslyAckedSeqNumber 164 that may represent an anchor value in the sequence number space from where the SelAckBitVector 162 may be applicable. In addition, in some examples, the MaxSequenceNumber 166 may represent a number of packet sequence numbers (PSNs) being acknowledged in the acknowledgement packet 160 based on the data packets that may be received by the receiver 104. In particular, since the receiver 104 may lag the transmitter 102 due to the latency in the network 190, the MaxSequenceNumber 166 field may prevent retransmission of data packets that may still be in flight and may yet reach the receiver 104.
In order to decode the compact acknowledgement format implemented by the acknowledgement packet 160, the transmitter 102 and the receiver 104 maintain certain pointers for storing certain values that may be changed dynamically as needed during the data communication. In an example, the transmitter 102 may maintain within the transmission tracker 124, an increasing number representing new data packets that are transmitted. A highest scheduled packet sequence number (PSN) pointer 122 may store a monotonically increasing number representing a new entry that may be created in the transmission tracker 124 if space is available. In case there is no space available in the transmission tracker 124 due to backpressure, the data packets may be held back from transmission until the space is available in the transmission tracker 124. In some examples, the transmission tracker 124 may include one or more transmit bit vector arrays that may store bit vector representations of packet sequence numbers (PSNs) of data packets that were sent to the receiver 104. Accordingly, in some examples, the size of the transmission tracker 124 may equal the number of data packets of maximum transmission unit (MTU) size possible within the Bandwidth Delay Product (BDP) of the network. In some examples, the transmitter 102 may also maintain a contiguously acknowledged packet sequence number (PSN) pointer 126 that may store the packet sequence number (PSN) of the latest data packet of a sequence of data packets that may be contiguously transmitted from the segment of data packets 150 and that may require no retransmission. In some examples, the application software within the transmitter 102 may use the contiguously acknowledged packet sequence number (PSN) pointer 126 to free up and to reuse the packet data memory. In an example, the communication system 100 may also generate completion updates to the software based on the highest scheduled packet sequence number (PSN) pointer 122 and the contiguously acknowledged packet sequence number (PSN) pointer 126. In some instances, a difference between the highest scheduled PSN pointer 122 and the contiguously acknowledged packet sequence number (PSN) pointer 126 may correspond to the amount of inflight data.
In some examples, the receiver 104 may also maintain a plurality of pointers. In some examples, the receiver 104 may include a highest received packet sequence number (PSN) pointer 142 and a contiguously received packet sequence number (PSN) pointer 146. In some examples, the highest received packet sequence number (PSN) pointer 142 may store a number representing the highest packet sequence number (PSN) number of the data packet received successfully by the receiver 104. In some examples, a new entry may be created in a receiver tracker 144 if space may be available. In some examples, where no space may be available in the receiver tracker 144 to store the packet sequence number (PSN) of a received data packet, then such a data packet may be dropped by the receiver 104. In these examples, this may be considered an error scenario as it may represent an out-of-range packet reception wherein the transmission rate of the data packets may exceed the tracking capacity of the receiver 104.
The receiver 104, may periodically acknowledge the received data packets. The contiguously received packet sequence number (PSN) pointer 146 may store the packet sequence number (PSN) of the last data packet of a sequence of data packets contiguously received with no drops (including post-retransmission) from the segment of data packets 150. Again, the applications software on the receiver side may use the contiguously received packet sequence number (PSN) pointer 146 to pass on the received data to the upper-level software stack. Transport engines may also generate completion updates based on these two pointers. The difference between the highest received packet sequence number (PSN) pointer 142 and the contiguously received packet sequence number (PSN) pointer 146 corresponds to the amount of inflight data. Any data successfully received in this range by the receiver 104 may be written into the memory of the receiver 104 directly and tracked as part of the receiver tracker 144. The size of the receiver tracker 144 may equal the number of Maximum Transmission Unit (MTU) packets possible within the Bandwidth Delay Product (BDP) of the communication network 190.
The data packets 220 may represent one or more data packets of the segment of data packets 150 that are transmitted to the receiver 104. The receiver 104 may generate an acknowledgement (ACK) or a negative acknowledgement (NACK) message for each of the data packets 220. As the data packet from the transmitter 102 is received at the receiver 104, the packet sequence number (PSN) of the received data packet may be stored in the receiver tracker 144 and the value of the highest received packet sequence number (PSN) 206 may be incremented. In an example, the currently updated packet sequence number (PSN) of the receiver tracker 144 can be compared to the packet sequence number (PSN) of the previously stored data packet and if it is determined that the packet sequence numbers (PSNs) of the current data packet and the prior data packet are consecutive, then the contiguously received packet sequence number (PSN) 208 may be incremented and the bit corresponding to the data packet in the SelAckBitVector 162 may be set to indicate an acknowledgement. Additionally, it may be determined if the acknowledgement packet 160 is to be generated. In an example, the acknowledgement packet 160 may be generated when a predetermined number of data packets are received by the receiver 104 or the acknowledgement packet 160 may be generated upon the expiration of a predetermined time (e.g., timeout). The SelAckBitVector 162 may be generated by including bits representing an acknowledgement (ACK) (e.g., ‘1’) or a negative acknowledgement (NACK)(e.g., ‘0’) for each received data packet. The value of the contiguously received packet sequence number (PSN) pointer 146 may be included as LastContiguouslyAckedSeqNumber 164 in the acknowledgement packet 160. The number of data packets or packet sequence numbers (PSNs) being acknowledged in the acknowledgement packet 160 may be included as MaxSequenceNumber 166 and optionally, if more than one acknowledgement packet is being transmitted, the ChainedAcks 168 flag may be set. The acknowledgement packet 160 thus generated may be provided to the transmitter 102.
Upon receiving the acknowledgement packet 160, the transmitter 102 may extract the LastContiguouslyAckedSeqNumber 164 to identify the data packet that was last received contiguously by the receiver 104 and may update the value of the contiguously acknowledged packet sequence number (PSN) 204 stored in the contiguously acknowledged packet sequence number (PSN) pointer 126. The packet sequence numbers (PSNs) s of the data packets associated with the bits in the SelAckBitVector 162 may be counted from the LastContiguouslyAckedSeqNumber 164 until the MaxSequenceNumber 166. Therefore, the last contiguously acknowledged sequence number (i.e., LastContiguouslyAckedSeqNumber 164) forms an anchor value in a sequence number space from where the select acknowledgement bit vector (i.e., SelAckBitVector 162) may be applied. The difference between the highest scheduled packet sequence number (PSN) pointer 122 and the contiguously acknowledged packet sequence number (PSN) pointer 126 may be identified as data packets in flight between the transmitter 102 and the receiver 104. The packet sequence numbers (PSNs) of the in-flight packets may be tracked using the transmission tracker 124. Similarly, the difference between the highest received packet sequence number (PSN) 206 and the contiguously received packet sequence number (PSN) 208 correspond to the inflightPacketPSNs 212 of either dropped packets that were never received or data packets that are in-flight from the transmitter 102 that are yet to be received by the receiver 104.
Offset into sequence number space may be calculated based on the LastContiguouslyAckedSeqNumber 164. Each time an acknowledgement (ACK) is received, 64 bytes may be read from the transmission tracker 124 and the above-described operation may be performed. The transmission tracker 124 may be further associated with a timer 408. In case any of the data packets remain unacknowledged so that an acknowledgement (ACK) or a negative acknowledgement (NACK) message is not received by the transmitter 102 at timeout, such data packets may also be added to the retransmission queue 414. In an example, when a contiguous stream of data packets is being transmitted, the transmitter 102 may not wait for the timeout before scheduling retransmissions as the receiver 104 may keep track of data packets received in an out of order manner and may be issuing selective acknowledgements aggressively. In an example, each entry in the transmission queue 402 may include 2 bits and together with the transmission tracker 124, the transmission queue 402 may keep track of scheduled packets, outstanding packets on the wire and acknowledged packets.
The methods detailed in the flowcharts below are provided by way of an example. There may be a variety of ways to carry out the method described herein. Although the method detailed below are primarily described as being performed by the transmitter 102 or the receiver 104, as shown in
An initial transmit operation is illustrated in state 802 in
Assuming that only 32 data packets were received by the receiver 104 with half of the data packets being dropped, the state 804 shows the status of the indices and tables maintained by receiver 104. Further assuming that the first and the third entries (odd-numbered data packets) in the received packets table 842 are set to 1 indicating accurate receipt of these packets, the LastFullyAckedPSN may therefore be 0 and the MaxPSNReceived (or highest received packet sequence number (PSN) 206) maybe 63 while the MaxPSNReceivedIndex (or highest received packet sequence number (PSN) pointer 142) may be set to 3 as the even-numbered data packets are not received. These values may be used to generate the Ack Packet 850 which may be provided at 860 to the transmitter 102. The updated table entries 870 of the transmit table 812 and the acknowledgement table 814 are obtained using the update operation described herein wherein a value of 1 represents an acknowledged packet while 0 may represent an outstanding packet. Thus, the transmitter 102 may identify the outstanding data packets that need to be re-transmitted based on updated table entries 870.
It should be appreciated that the processor 912 may be a semiconductor-based microprocessor, a central processing unit (CPU), or may include, one or more programmable general-purpose or special-purpose microprocessors, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), digital signal processors (DSPs), programmable controllers, programmable logic device (PLDs), trust platform modules (TPMs), other processing circuits, or a combination of these and/or other suitable hardware devices. The processor 912 may control the overall operation of the computing system 900. In some examples, the processor 912 may accomplish this by executing software or firmware stored in system memory 918 or other data via the storage adapter 920.
In some examples, the system memory 918 may have stored thereon machine-readable instructions (which may also be termed computer-readable instructions) that the processor 912 may execute. The system memory 918 may be an electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. The system memory 918 may be, for example, Random Access Memory (RAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a storage device, an optical disc, and the like. The system memory 918, which may also be referred to as a computer-readable storage medium, maybe a non-transitory machine-readable storage medium, where the term “non-transitory” does not encompass transitory propagating signals.
The interconnect 910 may interconnect various subsystems, elements, and/or components of the computer system 900. As shown, the interconnect 910 may be an abstraction that may represent any one or more separate physical buses, point-to-point connections, or both, connected by appropriate bridges, adapters, or controllers. In some examples, the interconnect 610 may include a system bus, a peripheral component interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA)) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1364 bus, or “firewire,” or other similar interconnection element.
In some examples, the interconnect 910 may allow data communication between the processor 912 and system memory 918. The system memory 918 may include read-only memory (ROM) or flash memory (neither shown), and random access memory (RAM) (not shown). It should be appreciated that the RAM may be the main memory into which an operating system and various application programs may be loaded. The ROM or flash memory may contain, among other code, the Basic Input-Output system (BIOS) which controls basic hardware operations such as the interaction with one or more peripheral components.
The multimedia adapter 914 may connect to various multimedia elements or peripherals. These may include devices associated with visual (e.g., video card or display), audio (e.g., sound card or speakers), and/or various input/output interfaces (e.g., mouse, keyboard, touchscreen).
The network interface 916 may provide the computing device with an ability to communicate with a variety of remote devices over a network and may include, for example, an Ethernet adapter, a Fiber Channel adapter, and/or other wired- or wireless-enabled adapter. The network interface 916 may provide a direct or indirect connection from one network element to another, and facilitate communication between various network elements.
The storage adapter 920 may connect to a standard computer-readable medium for storage and/or retrieval of information, such as a fixed disk drive (internal or external).
Many other devices, components, elements, or subsystems (not shown) may be connected in a similar manner to the interconnect 910 or via a network. Conversely, all of the devices shown in
The figures and description are not intended to be restrictive. The terms and expressions that have been employed in this disclosure are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof. The word “example” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “example” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.
Although the methods and systems as described herein may be directed mainly to digital content, such as videos or interactive media, it should be appreciated that the methods and systems as described herein may be used for other types of content or scenarios as well. Other applications or uses of the methods and systems as described herein may also include social networking, marketing, content-based recommendation engines, and/or other types of knowledge or data-driven systems.