Patent Grant
12132800
The Internet protocol suite is a set of communication protocols used for servicing data transmissions between two devices communicating information over the Internet or other computer networks. Transmission Control Protocol (“TCP”) is a part of the Internet protocol suite that provides for connection-oriented, reliable, and ordered delivery of a stream of data packets between, for example, a web-browser application running on a client device and a web-server application running on a server device over a local or wide area network. Currently, datacenters using communication protocols such as TCP encounter certain issues. For instance, incast is a many-to-one communication pattern commonly found in datacenters, which may result in incast congestion when multiple synchronized computing devices send data to a same receiver computing device in parallel. Further, because the current communication protocols require ordered delivery of packets over a connection, a long tail latency, which is the amount of time for the last few packets among a series of packets to be transmitted, may prevent transmission of the next series of packets
The present disclosure provides for determining, by an initiator entity, that outgoing data is to be transmitted to a target entity; transmitting, by the initiator entity to the target entity, a solicited push request requesting the outgoing data to be placed at the target entity; receiving, by the initiator entity from the target entity, a push grant in response to the solicited push request; and transmitting, by the initiator entity to the target entity, the outgoing data to be placed at the target entity in response to the push grant.
The method may further comprise determining, by the initiator entity, that a size of the outgoing data meets a predetermined threshold, wherein transmitting the solicited push request is based on the determination that the size of the outgoing data meeting the predetermined threshold. The push request may originate from an upper layer protocol of the initiator entity, based on which a reliable transport protocol layer of the initiator entity transmits the solicited push request as a packet over a connection between the initiator entity and the target entity.
The method may further comprise determining, by the initiator entity, that a size of the outgoing data does not meet a predetermined threshold; and transmitting, by the initiator entity, the outgoing data to be placed at the target entity without sending the solicited push request or receiving the push grant. The push request may originate from an upper layer protocol of the initiator entity, based on which a reliable transport protocol layer of the initiator entity sends the outgoing data as a packet over a connection between the initiator entity and the target entity.
The method may further comprise receiving, by the initiator entity from the target entity, an acknowledgment indicating that the outgoing data is received and placed at the target entity.
The method may further comprise determining, by the initiator entity, that incoming data is needed from the target entity; transmitting, by the initiator entity to the target entity, a pull request requesting the incoming data to be transmitted to the initiator entity; receiving, by the initiator entity from the target entity, the incoming data in response to the pull request. The method may further comprise scheduling, by the initiator entity based on one or more congestion parameters, the pull request for incoming data.
The present disclosure further provides for transmitting, by a sender entity over a connection to a receiver entity, a plurality of packets in a first order; maintaining, by the sender entity, at least one sliding window including a plurality of bits, wherein each bit of the sliding window represents a respective packet of the plurality of packets; receiving, by the sender entity, one or more acknowledgments indicating that one or more of the plurality of packets have been received by the receiver entity, each of the acknowledgments referencing a respective packet of the plurality of packets, wherein the acknowledgments are received in a second order different from the first order; and modifying, by the sender entity, values of one or more of the plurality of bits in the sliding window corresponding to the one or more acknowledgments received.
The method may further comprise adjusting, by the sender entity, a size of the sliding window based on one or more congestion parameters.
The plurality of packets may include one or more of: requests for data packets, data packets, acknowledgments. The at least one sliding windows may include a request sliding window. The at least one sliding windows may include a data sliding window. The plurality of packets may include at least one data packet in response to a pull request. The plurality of packets may include at least one push grant packet in response to a solicited push request.
The present disclosure still further provides for transmitting, by an initiator entity to a target entity over a connection, a plurality of packets; determining, by the initiator entity, that neither an acknowledgment nor a negative acknowledgment has been received in response to a particular packet of the plurality of packets within a predetermined period of time; retransmitting, by the initiator entity to the target entity based on the determination, the particular packet; receiving, by the initiator entity from the target entity in response to the retransmission, a negative acknowledgement; and determining, by the initiator entity based on the negative acknowledgment, whether to wait for an acknowledgment for the particular packet or to resynchronize.
The method may further comprise determining, by the initiator entity, that the negative acknowledgment indicates that the target entity is not ready for the particular packet; and waiting, by the initiator entity, for an acknowledgment from the target entity in response to the negative acknowledgment without another retransmission of the particular packet to the target entity.
The method may further comprise determining, by the initiator entity, that the negative acknowledgment indicates that operation for the particular packet is completed in error by the target entity; and transmitting, by the initiator entity to the target entity, a resynchronization packet without tearing down the connection. The method may further comprise receiving, by the initiator entity from the target entity, an acknowledgment in response to the resynchronization packet; and transmitting, by the initiator entity, a next plurality of packets in response to the acknowledgment to the resynchronization packet.
The plurality of packets may be transmitted according to requests from an upper layer protocol of the initiator entity, and the retransmission of the particular packet is performed by a reliable transport protocol layer of the initiator entity.
The technology generally relates to communication protocols for reliable transport of packets over a connection. The technology provides solicitation based push transactions, which provides a receiver entity control over incoming data and thus reduce incast congestion and tail latency. The technology further supports unordered transactions over a connection using sliding windows and bitmaps, which may increase overall efficiency in handling of packets over the connection. The technology further provides handling of failed transmissions that reduces retransmission attempts and uses resynchronization to prevent tearing down of connections, thus resulting in more resilient connections.
A connection may be identified by a pair of Connection IDs (“CIDs”), one in each direction of communication. CIDs may be allocated by a receiver entity during connection setup process and have no global significance outside of the parties involved. Thus, the connection 110 between entities A and B may have a CID with value 5 for the direction from A to B, and a CID with value 10 for the direction from B to A. The connection 120 between entities A and C may have a CID value 5 for the direction from A to C and a CID with value 11 for the direction from C to A. Further, CIDs assigned by an entity or “Source CIDs” of an entity must have different values. Thus in the example shown, the CIDs assigned by entity A or Source CIDs of entity A have different values 10 and 11. In contrast, “Destination CIDs” of an entity are assigned by other entities and may have the same value. Thus in the example shown, the Destination CIDs of entity A are assigned by entities B and C respectively, which may have the same value 5.
Packets may be transmitted over the connections between the entities. In this regard, a packet is a basic unit of communication across a connection. A packet may have a predetermined size, for example up to a maximum transfer unit (“MTU”) in length. A packet may have a header including information about the packet and its transmission, and a payload of data. To ensure reliable transport, a reliable transport packet may include the Destination CID, such as in a header. For example, when entity B receives a packet over the connection 110 with the Destination CID of 5, entity B may identify the packet as coming from entity A, and may then notify A that the packet has been received by sending an acknowledgment over the connection 110 referencing this packet and its CID of 5. The acknowledgment itself may be sent as a packet including the Destination CID of 10.
Entities A, B, and C may be any type of device capable of communicating over a network, such as personal computing devices, sever computing devices, mobile devices, wearable devices, virtual machines, etc.
The one or more processors 220 can be any conventional processors, such as a commercially available CPU. Alternatively, the processors can be dedicated components such as an application specific integrated circuit (“ASIC”) or other hardware-based processor. Although not necessary, the one or more of the computing devices 210 may include specialized hardware components to perform specific computing processes.
The memory 230 can be of any non-transitory type capable of storing information accessible by the processor, such as a hard-drive, memory card, ROM, RAM, DVD, CD-ROM, write-capable, and read-only memories. Memory 230 of the computing devices 210 can store information accessible by the one or more processors 220, including data 232 and instructions 234.
Memory 230 can include data 232 that can be retrieved, manipulated or stored by the processors 220. For example, data such as communication protocols, connection information such as CIDs, definitions of headers, etc., as described with respect to
Memory 230 of the computing devices 210 can also store instructions 234 that can be executed by the one or more processors 220. For instance, instructions such as communication protocols as described with reference to
Data 232 may be retrieved, stored, or modified by the one or more processors 220 in accordance with the instructions 234. For instance, although the subject matter described herein is not limited by any particular data structure, the data can be stored in computer registers, in a relational database as a table having many different fields and records, or XML documents. The data can also be formatted in any computing device-readable format such as, but not limited to, binary values, ASCII or Unicode. Moreover, the data can comprise any information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories such as at other network locations, or information that is used by a function to calculate the relevant data.
The instructions 234 can be any set of instructions to be executed directly, such as machine code, or indirectly, such as scripts, by the one or more processors. In that regard, the terms “instructions,” “application,” “steps,” and “programs” can be used interchangeably herein. The instructions can be stored in object code format for direct processing by a processor, or in any other computing device language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance.
Although not shown, computing devices 210 may further include other components typically present in general purpose computing devices. For example, computing devices 210 may include output devices, such as displays (e.g., a monitor having a screen, a touch-screen, a projector, a television, or other device that is operable to display information), speakers, haptics, etc. The computing devices 210 may also include user input devices, such as a mouse, keyboard, touch-screen, microphones, sensors, etc.
Although
The computing devices 210 may be capable of directly and indirectly communicating with other entities of a network, such as computing devices 260. Computing devices 210 and 260 may be interconnected using various protocols and systems, such that computing devices in the network can be part of the Internet, World Wide Web, specific intranets, wide area networks, or local networks. Computing devices in the network can utilize standard communication protocols, such as Ethernet, WiFi and HTTP, protocols that are proprietary to one or more companies, and various combinations of the foregoing. Although certain advantages are obtained when information is transmitted or received as noted above, other aspects of the subject matter described herein are not limited to any particular manner of transmission of information.
Returning to
As described with reference to
Also described with reference to
Referring to the TX sliding window 410, to keep track of the packets, each packet is assigned a Packet Sequence Number (“PSN”) by the sender entity A. As shown, the bit number increases from left to right. The receiver entity B may acknowledge the packets it has received within the sliding window by communicating to the sender entity A the PSN it has received within the window in an acknowledgement packet. In this regard, a Sequence Number Bitmap may be provided on both the sender entity A and the receiver entity B. Each bit of the Sequence Number Bitmap represents one packet within a sliding window at the entity. For example, for the TX sliding window 410, a bit is set to 1 if a sent packet has been acknowledged. Otherwise the bit is 0. Once all packets within the TX sliding window 410 are received and acknowledged, the sender entity A may move the sliding window 410 forward to the next set of packets to be transmitted. The sliding window moves forward once the base sequence number packet is acknowledged, thus referring to the example in
PSN for the sender entity may include Base Sequence Number (“BSN”) and Next Sequence Number (“NSN”). As shown, BSN is the PSN value of the oldest packet that is yet to be acknowledged by the receiver entity B. Further as shown, NSN is the PSN value that should be assigned to the next packet transmitted over the connection to receiver entity B. For instance, when a packet is received from ULP 310 for transmission, the current PSN may be updated to NSN. Then when the packet is transmitted over the connection, NSN may be incremented, for example with NSN=(NSN+1) mod 232. As such, within the sliding window 410, Bit 0 represents a PSN value of BSN and Bit n represents a PSN value of (BSN+n).
Although not shown, the receiver entity may also keep one or more sliding windows. For example, a RX sliding window may be kept by receiver entity B for the packets received, where each bit represents a packet to be received with the sliding window. The bit is set to 1 if the packet has been received by the receiver entity B. Otherwise the bit is 0. The receiver entity B may also use PSN to keep track of received packets. For instance, BSN may be the PSN value of the oldest packet that is yet to be received by the receiver entity. When a packet is received with a PSN value of BSN, the BSN may be updated to the next lowest PSN of the packet that has not yet been received, for example with BSN=(BSN+1) mod 232. The update of the BSN may clear the bits in the Sequence Number Bitmap corresponding to packets from the previous BSN to the PSN. As such, within the RX sliding window for the receiver entity B, Bit 0 represents a PSN value of BSN and Bit n represents a PSN value of (BSN+n). Because sender entity A does not acknowledge the acknowledgements sent by receiver entity B, that is, PSN is not used for the acknowledgment packets, the receiver entity B need not keep a TX sliding window for the acknowledgements it sends.
The sender entity and receiver entity may handle the packets and the respective acknowledgements according to a set of rules. For instance, if the receiver BSN in a received packet is smaller than the sender entity's BSN, the sender entity discards the ACK information; otherwise, the sender entity updates its BSN to match the receiver entity's BSN. After adjusting its BSN, the sender entity applies an OR operation on the receiver entity's Sequence Number Bitmap in the ACK packet with its own Sequence Number Bitmap. After a packet is transmitted, it is buffered by the sender entity until it is acknowledged by the receiver entity. With respect to retransmission of failed packets, the sender entity may be configured to free up resources allocated to all ACK packets in a retransmit buffer. Further, upon per packet retransmit timer expiry, the sender entity retransmits the packet with the same PSN as the original packet, and increment a retransmission counter for that packet.
The receiver entity may also implement a number of rules. For instance, if the PSN value of the received packet is less than the BSN of the received packet, the receiver entity discards the packet and sends an ACK packet with the current BSN. If the PSN value falls within the receiver entity's sliding window, the receiver entity updates the Sequence Number Bitmap by setting the bit at location (PSN−BSN) to 1. If the bit at location (PSN−BSN) was already 1, the packet is discarded; otherwise the packet is delivered to the ULP of the receiver entity and a cumulative ACK counter is incremented. If the PSN of the received packet is equal to BSN of the received packet, the receiver entity updates the BSN to be equal to the next highest PSN that has not been received.
Note that, because the packets are tracked according to bitmaps, the sliding windows are configured to allow the entities to keep track of packets received and/or acknowledged out-of-order within the respective sliding window. Thus as shown, although packets represented by bits 3 and 4 may be sent by entity A before the packets represented by bits 0, 1, and 2, the packets represented by bits 3 and 4 may be received and/or acknowledged before the packets represented by bits 0, 1, 2 in the TX sliding window 410.
Network congestion may be detected by monitoring packet retransmission and/or packet round-trip latencies. To perform congestion control, a size of the one or more sliding windows may be adjusted. For example, if congestion is high, it may take longer for all packets within the TX sliding window 410 to be received and/or acknowledged by entity B. As such, to reduce congestion, the number of outstanding packets in the network may be reduced by decreasing the size of the sliding window 410. In addition to or as alternative to changing the size of the sliding window, retransmission timer expiry value in response to network congestion status may be adjusted. For example, retransmitting less frequently might reduce network congestion.
The communication protocol system 300 of
Referring to
As shown, a pull request (“pullReq”) originates from the initiator entity A, for instance from the initiator ULP 510, which is sent to the initiator RT 530. The initiator entity A may send the pullReq to the target entity B from which incoming data is requested, for instance over the connection 110. This may be performed by the respective RTs, thus the initiator RT 530 is shown sending the pullReq to the target RT 540. Once the pullReq is received by the target entity B, the target RT 540 subsequently sends the pullReq to the target ULP 520 to request permission. The target ULP 520 may then send an acknowledgment message (“ULP-ACK”) to the target RT 540 acknowledging the pullReq, as well as a pull response (“pullResp”) instructing the target RT 540 to pull the requested data. In response to the pullResp, the target RT 540 may pull the requested data (“pullData”), and send the pulled data to the initiator RT 530, for instance over the connection 110. Once the requested data is received by the initiator RT 530, the initiator RT 530 may send a pullResp to the initiator ULP 510 so that the initiator ULP 510 may place or store the received data packet.
As described with reference to
As illustrated by
Referring to
As shown, a push request (“pushReq”) may originate from the initiator entity A, for instance from the initiator ULP 610, which is sent to the initiator RT 630. The initiator entity A may then push unsolicited data onto the target entity B, for instance over the connection 110. This may be performed by the respective RTs, thus the initiator RT 630 is shown pushing unsolicited data (“pushUnslctdData”) to the target RT 640. The data is unsolicited because the target entity B did not request this data. Once the data is received by the target entity B, the target RT 640 may request for the received data to be placed or stored at the target entity B, and does so by sending a pushReq to the target ULP 620. In response, the target ULP 620 may place or store the received data, and then sends an acknowledgment message ULP-ACK to the target RT 640 acknowledging that the received data has been placed or stored according to the pushReq. For reliable transport, the target entity B sends an acknowledgment message (“ACK”) to notify initiator entity A of the receipt and placement of the pushed data, for instance over the connection 110. This is performed by the respective RTs, thus as shown the target RT 640 sends the ACK message to the initiator RT 630. Once the ACK message is received by the initiator RT 630, the initiator RT 630 may send a push complete message (“pushCmpl”) to initiator ULP 610 to notify that the data packet has been received and placed by the target entity.
As described with reference to
As illustrated by
In contrast to
Similarly to
Once the pushGrnt is received by the initiator entity A, the initiator entity A may push the solicited data to the target entity B, for instance over the connection 110. This may be performed by the respective RTs, thus the initiator RT 730 is shown pushing solicited data to the target RT 740. In contrast to the unsolicited data pushed in
As described with reference to
As illustrated by
In some instances, the communication protocol system may be configured to perform one of the push transactions shown in
In another aspect, the communication protocol system 300 of
Referring to
Referring to the timing diagram 800, a number of requests may originate from the initiator entity A, including pull requests such as pullReq_1, and push requests such as pushReq_0, pushReq_2, and pushReq_3. As described above, these requests may be sent by the initiator ULP 810 to the initiator RT 830. Once the initiator RT 830 receives these requests, initiator RT 830 may optionally determine whether the push requests should be sent as solicited or unsolicited as described above with reference to
The requests may be sent by the initiator ULP 810 in a particular order as indicated by the Request Sequence Numbers (“RSN”), which may be assigned by the initiator ULP 810. In some instances, the initiator RT 830 may also assign Solicited Sequence Numbers (“SSN”) specifically to solicited push requests, which may be an incremental number as shown. When the requests are sent as packets between two entities, the requests may be assigned with a sequence of numbers in ascending order according to the order of the RSN. Thus as shown, the requests may be assigned PSNs within one or more TX sliding windows maintained by initiator entity A according to the RSNs. For example, pushSlctdReq_0 is assigned PSN=0, pullReq_1 is assigned PSN=1, pushSlctdReq_2 is assigned PSN=2 within a request TX sliding window of entity A (indicated by dash lines pointing towards B). Note that since pushReq_3 from the initiator ULP 810 does not require solicitation as shown in
In response to the solicited push requests, push grants may be sent by the target RT 840 to the initiator RT 830 in the order of the received requests, such as pushGnt_0 and pushGnt_2. The push grants may be assigned with PSNs in ascending order within one or more TX sliding windows maintained by the target entity B according to the same order as the RSNs of the push requests. For example, pushGrnt_0 is assigned PSN=1000 and pushGrnt_2 is assigned PSN=1001 within a data TX sliding window of entity B (indicated by dash dot lines pointing towards A). However, the push grants may not be received in the same order by the initiator RT 830 as the order of transmission for the push requests. Thus as shown, pushGrnt_2 is received by the initiator RT 830 before the pushGrnt_0.
Nonetheless, the initiator RT 830 may determine the correct order of the push grants based on their respective PSNs, and push the data packets based on that order. As such, although pushGrnt_2 was received by the initiator RT 830 before pushGrnt_0, the initiator RT 830 may first push the data solicited by pushGrnt_0 with pushSlctdData_0 and then push the data solicited by pushGrnt_2 with pushSlctdData_2 to target RT 840. The pushed data packets are also assigned PSNs in ascending order within one or more TX sliding windows maintained by initiator entity A according to the order of transmission. For example, pushSlctdData_0 is assigned PSN=200 and pushSlctdData_2 is assigned PSN=201 within a data TX sliding window of entity A (indicated by dash dot lines pointing towards B). Note that the pushReq_3 does not require a grant as described with reference to
Target RT 840 receives the requests, and then sends corresponding requests to the target ULP 820 in the order of ULP-Req-0-1-2-3, which is in the same order as the transmission order of the requests from the initiator ULP 810 shown at the top of the timing diagram 800. As described above with reference to
Following the ULP-ACKs, with respect to the push transactions, ACKs acknowledging the data packets (or data acknowledgments) are then sent by target RT 840 to initiator RT 830 to notify the safe receipt and placement of the reliable transport data packets. As an example, ACK-eBSN=3, 203 is sent by entity B to notify entity A that all request packets up to PSN=3 and all data packets up to PSN=203 have been received and placed. Once the ACK is received, initiator RT 830 may send a completion message pushCompl_0 to initiator ULP 810. Further, in some instances acknowledgment packets may be opportunistically piggybacked on other reliable transport packets. For example, the requests pushSlctdReq_0, pullReq_1, and pushSlctdReq_2, are reliable transport packets requiring an ACK, but these acknowledgments to requests (or request ACKs) are not explicitly shown in timing diagram 800 because they may be piggybacked on reliable transport packets such as pushGrnt_0 and pushGrnt_2.
Also following the ULP-ACKs, pull requests may also be responded to. Thus as shown, the target ULP 820 may send a pullResp_1 instructing target RT 840 to pull the requested data. Target RT 840 then sends the pulled data to the initiator RT 830 with pullData_1. In this example, pullData_1 is assigned PSN=1002 within the same data TX sliding window of entity B as the pushGrnts (indicated by dash dot line pointing towards A). The initiator RT 830 then sends a pullResp_1 to the initiator ULP 810 so that the initiator ULP 810 may place or store the received data packet at entity A. After the data packet is placed or stored at entity A, an acknowledgment may be sent to notify entity B of safe receipt. Thus as shown, ACK-eBSN=3, 203 is sent by entity A to notify entity B that the pullData_1 packet has been safely received.
In this ordered system, the completion messages received by the initiator ULP 810 near the bottom of timing diagram 800 are in the same order as the requests that were sent by initiator ULP 810 near the top of the timing diagram 800. This order is maintained on ULPs of both initiator and target entities, where the target RT presents requests to the target ULP in the same order as the initiator ULP sends requests to the initiator RT. This ordered system ensures that the requests are delivered once and only once over the connection. In contrast, there may not be ordering requirement between transactions going in different directions over the connection.
Referring to timing diagram 900, many of the same transactions as timing diagram 800 are shown, and are labeled as such. For instance, once the various requests are sent from initiator ULP 910 to initiator RT 930, the requests are then transmitted to the target RT 940 similar to timing diagram 800. For the solicited push requests such as pushSlctdReq_0 and pushSlctdReq_2, push grants such as pushGrnt_0 and pushGrnt_2 are sent by target RT 940 to initiator RT 930. In response to the push grants, solicited data are pushed by initiator RT 930 to target RT 940 as shown with pushSlctdData_0 and pushSlctdData_2. For the unsolicited requests such as pushUnslctdReq_3, unsolicited data may be pushed by initiator RT 930 to target RT 940 as shown with pushUnslctdData_3. For the pull requests such as pullReq_1, pull requests may be sent by initiator RT 930 to target RT 940, then to target ULP 920. Target ULP 920 may then respond with pullResp_1, which target RT 940 responds with pullData_1 to initiator RT 930. Initiator RT 930 then sends pullResp_1 to initiator ULP 910, which handles the placing and/or storing of the pulled data at entity A. These packets may also be kept tracked of by sliding windows as described above with reference to
Timing diagram 900 also illustrates several aspects where the unordered system is different from the ordered system of timing diagram 800. One difference is that the PSNs for the packets are assigned according to the order of transmission, rather than in accordance with the respective RSNs. For instance, the pushSlctdReq_0, pullReq_1, and pushSlctdReq_2 packets are assigned PSNs according to the order of transmission by the initiator RT 930, which happens to be the same order as the respective RSNs. The pushGrnt_0 and pushGrnt_2 packets also have PSNs in the order of the respective RSNs due to the transmission order. However, the pushUnslctdData_3, pushSlctdData_0, and pushSlctdData_2 packets are transmitted in a different order than the order of the respective RSNs, and thus resulting in PSNs not in the same order as the respective RSNs. As such, pushUnslctdData_3 has PSN=200 that is smaller than pushSlctdData_0 with PSN=201 and pushSlctdData_2 with PSN=202.
Another difference is that the target entity may handle the transactions out of order. Thus as shown, the ULP-Reqs and the ULP-ACKs in timing diagram 900 are not sent all after the various requests and grants have been passed around the respective RTs. Rather, the ULP-Req and ULP-ACK corresponding to a particular transaction are sent as soon as the request and/or grant between the RTs are completed with respect to that particular transaction. For example, with respect to the transaction pullReq_1, ULP-Req_1 and ULP-ACK-1 are transmitted as soon as the pullReq_1 is transmitted from the initiator RT 930 to the target RT 940. As another example with respect to the transaction pushUnslctdReq_3, ULP-Req_3 and ULP-ACK-3 are transmitted as soon as the pushUnslctdData_3 is transmitted from the initiator RT 930 to the target RT 940. With respect to transactions pushSlctdReq_0 and pushSlctdReq_2, the ULP-Reqs and ULP-ACKs are sent later, after the push grants and the solicited data packets are pushed from initiator RT 930 to target RT 940. Further, because pushSlctdData_0 is received at target RT 940 after the pushSlctdData_2, the ULP-Reqs and ULP-ACKs for these two are sent in reversed order. Still further, although ULP-Req-1 and ULP-ACK-1 may be transmitted before ULP-Req_3, ULP-ACK_3, ULP-Req-O, ULP-Req-2, ULP-ACK-0, and ULP-ACK-2, the target ULP 920 may nonetheless handle the pull request after the push requests.
As a consequence of the unordered handling of transactions by the target entity, the acknowledgements and completion messages may be sent out of order to the initiator entity. As such, the initiator may also handle the transactions out of order. Thus as shown, target RT 940 sends ACK for pushUnslctdData_3, which prompts initiator RT 930 to send pushCompl_3 to initiator ULP 910 before the ACK for pushSlctdData_0 and pushCompl_0, before pushCompl_2, and before pullResp_1. Further as shown, because the solicited pushes are completed in this example before the pull request, ACK for pushSlctdData_0, pushComp_0, and pushComp_2 may be transmitted before the pullResp_1 from initiator RT 930.
The unordered handling of transactions by the entities as shown in
In still another aspect, the communication protocol system 300 of
Referring to
Referring to timing diagram 1000, many of the same transactions as timing diagram 800 are shown, and are labeled as such. For instance, timing diagram 1000 shows pushReq_0, pullReq_1, and pushReq_2 originating from the initiator ULP 1010. Once the various requests are sent from initiator ULP 1010 to initiator RT 1030, the requests are then transmitted to the target RT 1040 similar to timing diagram 800. For the solicited push requests pushSlctdReq_0 and pushSlctdReq_2, push grants pushGrnt_0 and pushGrnt_2 are sent by target RT 1040 to initiator RT 1030. In response to the push grants, solicited data are pushed by initiator RT 1030 to target RT 1040 as shown with pushSlctdData_0 and pushSlctdData_2. Also, the pull request pullReq_1 is sent by initiator RT 1030 to target RT 1040. The target RT 1040 then sends pushSlctdReq_0, pullReq_1, and pushSlctdReq_2 to target ULP 1020. The target RT 1040 also sends acknowledgment to the initiator RT 1030 for the requests with ACK-0-1-2.
However, in timing diagram 1000, the target entity is not ready for the push and pull requests. Accordingly, target ULP 1020 sends negative acknowledgements (“NACK”) notifying that the target ULP 1020 is not ready, and that the target RT 1040 should try again later. Thus as shown, target ULP 1020 sends pushNACK_0-retry in response to the pushSlctdReq_1, pullNACK_1-retry in response to the pullReq_1, and pushNACK_2-retry in response to the pushSlctdReq_2. These NACKs may include the reason for the NACK, which in this example being that the target entity is not ready, and/or may include a new timer expiry value for retransmissions. The target RT 1040 then sends corresponding NACKs to the initiator RT 1030, shown as NACK-200 and NACK-201 referencing the PSNs of the pushSlctdData_0 and pushSlctdData_2 packets. Note that there is no NACK to the initiator for pullReq_1 because it had already been acknowledged earlier by ACK-0-1-2. Instead, target RT 1040 keeps track of the pullNACK-1-retry, and will re-deliver to target ULP 1020 upon timer expiry. Note that the NACKs have the same eBSN because the sliding window is stuck due to the inability to acknowledge to sender. However, while NACK-201 reaches the initiator RT 1030, for any of a number of reasons, the NACK-200 packet may be dropped in the network and does not reach the initiator RT 1030, and as mentioned earlier, ACK and NACK messages are not transmitted by reliable transport.
Because the initiator RT 1030 receives the NACK for pullSlctdReq_2, initiator RT does not attempt retransmission of pushSlctdData_2. In contrast, because the initiator RT 1030 does not receive the NACK for pushSlctdReq_0, initiator RT 1030 attempts retransmission with pushData_0-retry. For example, the initiator RT 1030 may be configured to attempt retransmission with the same PSN if it does not receive an ACK for the pushed data within a predetermined period of time. Once the target RT 1040 receives the pushData_0-retry, the target RT 1040 recognizes that pushData_0-retry is a retransmission of a data packet, for example based on the PSN of the pushData_0-retry being the same as the earlier received pushSlctdData_0. Target RT 1040 then sends ULP-pushReq_0 to target ULP 1020. This time, the target ULP 1020 is ready, and responds with ULP-ACK-0.
Then, after some time, target RT 1040 attempts again with ULP-pullReq_1 and ULP-pushSlctdReq_2 to target ULP 1020. As shown, this time the target ULP 1020 is ready and completes the requests. The rest of the timing diagram 1000 is similar to timing diagram 800, where the target ULP 1020 sends to target RT 1040 acknowledgments ULP-ACK-0, ULP-ACK-1, and ULP-ACK-2, then the target RT 1040 sends acknowledgments ACKs-200-201 to initiator RT 1030, and the initiator RT 1030 in turn sends completion messages pushComp_0 and pushCompl_2. For the pull request, the target ULP 1020 sends pullResp_1 to target RT 1040, which then sends pullResp_1 to initiator RT 1030, which then sends completion message pullCompl_1 to initiator ULP 1010.
Thus,
Referring to
Referring to timing diagram 1100, many of the same transactions as timing diagram 800 are shown, and are labeled as such. For instance, timing diagram 1100 shows pushReq_0, pullReq_1, and pushReq_2 originating from initiator ULP 1110. Once the various requests are sent from initiator ULP 1110 to initiator RT 1130, the requests are then transmitted to the target RT 1140 similar to timing diagram 800. For the solicited push requests pushSlctdReq_0 and pushSlctdReq_2, push grants pushGrnt_0 and pushGrnt_2 are sent by target RT 1140 to initiator RT 1130. In response to the push grants, solicited data are pushed by initiator RT 1130 to target RT 1140 as shown with pushSlctdData_0 and pushSlctdData_2. The pull request pullReq_1 is sent by initiator RT 1130 to target RT 1140. The target RT 1140 then sends pushSlctdReq_0, pullReq_1, and pushSlctdReq_2 to target ULP 1120.
However, in timing diagram 1100, for any of a number of reasons, the target ULP 1120 completes the pushSlctdData_0 in error. Accordingly, target ULP 1120 sends a pushNACK_0-compl-in-err notifying the target RT 1140 of the error. The NACK may include reason for the NACK, which in this example is that the placement or storing of the pushed data was completed in error. The NACK may optionally include information on the error, such as the reason. The target RT 1140 then sends a corresponding NACK to the initiator RT 1130, shown as NACK-200 referencing the PSN of the pushSlctdData_0. However, for any of a number of reasons, NACK-200 does not reach the initiator RT 1130. Further as shown, the target ULP 1120 completes the other requests successfully, and sends ULP-ACK_1, ULP-ACK_2, and pullResp_1 to target RT 1140, which prompts target RT 1140 to send ACK-201 and pullResp_1 to initiator RT 1130.
Because the initiator RT 1130 does not receive the NACK for pushSlctdReq_0, initiator RT 1130 attempts retransmission with pushData_0-retry. For example, the initiator RT 1130 may be configured to attempt retransmission with the same PSN if it does not receive an ACK for the pushed data within a predetermined period of time. Once the target RT 1140 receives the pushData_0-retry, the target RT 1140 recognizes that pushData_0-retry is a retransmission of a data packet, for example based on the PSN of the pushData_0-retry being the same as the earlier received pushSlctdData_0. As such, the target RT 1140 resends the NACK-200 without having to send another push request to the target ULP 1120.
Further, in response to the complete-in-error negative acknowledgement, a resynchronization may be performed. For instance as shown, the resynchronization may be initiated by the initiator RT 1130 with “resync-pkt.” The resynchronization prompts the target RT 1140 to send an acknowledgement that allows the one or more current sliding windows to move to a next set of packets to be transmitted and/or received. As shown, this is done by the target RT 1140 sending an acknowledgement ACK-200, rather than a negative acknowledgment, to the initiator RT 1130 in response to the resync-pkt. The initiator RT 1130 then sends completion messages pushCompl_0-in-err, pullCompl_1, and pushCompl_2 to the initiator ULP 1110, which notifies of both successful completions and the completion in error.
Although the examples of
Returning to
Rate update engine may report the results back to the communication protocol system 300, based on which congestion control may be implemented. For example, the report may include congestion window (“Cwnd”), which is the total number of outstanding TX packets. When this value is between 0 and 1, the communication protocol system 300, for example the sender RT, may apply additional inter-packet gap to limit the number of packet transmission to be less than 1 per RTT. As another example, the report may include retransmission timeout (“RTO”), which is the time the sender entity waits before retransmitting a pending TX packet if no ACK is received.
Referring to
Turning to
Referring to
Referring to
The technology generally relates to communication protocols for reliable transport of packets over a connection. The technology provides solicitation based push transactions, which provides a receiver entity control over incoming data and thus reduce incast congestion and tail latency. The technology further supports unordered transactions over a connection using sliding windows and bitmaps, which may increase overall efficiency in handling of packets over the connection. The technology further provides handling of failed transmissions that reduces retransmission attempts and uses resynchronization to prevent tearing down of connections, thus resulting in more resilient connections.
Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as “such as,” “including” and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. Further, the same reference numbers in different drawings can identify the same or similar elements.
The present application is a divisional application of U.S. patent application Ser. No. 17/857,620 filed on Jul. 5, 2022, which is a divisional application of U.S. patent application Ser. No. 16/819,327 filed on Mar. 16, 2020, now issued as U.S. Pat. No. 11,463,547, which claims the benefit of the filing date of U.S. Provisional Patent Application No. 62/947,036 filed on Dec. 12, 2019, the disclosures of which are hereby incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6330226 | Chapman et al. | Dec 2001 | B1 |
| 6330451 | Sen et al. | Dec 2001 | B1 |
| 7305486 | Ghose et al. | Dec 2007 | B2 |
| 7756036 | Druke | Jul 2010 | B2 |
| 8306062 | Cohen | Nov 2012 | B1 |
| 20050081246 | Barrett et al. | Apr 2005 | A1 |
| 20050135394 | Sethi et al. | Jun 2005 | A1 |
| 20070008883 | Kobayashi | Jan 2007 | A1 |
| 20070211631 | Rahman et al. | Sep 2007 | A1 |
| 20070253401 | Jiang | Nov 2007 | A1 |
| 20070275656 | Chang | Nov 2007 | A1 |
| 20080008211 | Zilbershtein et al. | Jan 2008 | A1 |
| 20080165684 | Sridharan et al. | Jul 2008 | A1 |
| 20080317051 | Dantzig et al. | Dec 2008 | A1 |
| 20090007141 | Blocksome et al. | Jan 2009 | A1 |
| 20100118780 | Umesh et al. | May 2010 | A1 |
| 20100274848 | Altmaier et al. | Oct 2010 | A1 |
| 20100312941 | Aloni | Dec 2010 | A1 |
| 20110116512 | Crupnicoff | May 2011 | A1 |
| 20110134930 | McLaren et al. | Jun 2011 | A1 |
| 20120278400 | Elson et al. | Nov 2012 | A1 |
| 20120287814 | Roberts et al. | Nov 2012 | A1 |
| 20130055263 | King et al. | Feb 2013 | A1 |
| 20130163417 | Gupta | Jun 2013 | A1 |
| 20130294236 | Beheshti-Zavareh et al. | Nov 2013 | A1 |
| 20150195747 | Ho et al. | Jul 2015 | A1 |
| 20160345366 | Brännlund | Nov 2016 | A1 |
| 20170063606 | Babu | Mar 2017 | A1 |
| 20170149913 | Olomskiy | May 2017 | A1 |
| 20170187598 | Andreyev et al. | Jun 2017 | A1 |
| 20170201601 | Bright et al. | Jul 2017 | A1 |
| 20170244643 | Lawrence et al. | Aug 2017 | A1 |
| 20180004705 | Menachem | Jan 2018 | A1 |
| 20180069827 | Logue | Mar 2018 | A1 |
| 20180097853 | Wiley et al. | Apr 2018 | A1 |
| 20180102975 | Rankin | Apr 2018 | A1 |
| 20180218294 | Dantzig et al. | Aug 2018 | A1 |
| 20190190542 | Wang et al. | Jun 2019 | A1 |
| 20200065269 | Balasubramani | Feb 2020 | A1 |
| 20200084150 | Burstein et al. | Mar 2020 | A1 |
| 20200145881 | Mitra et al. | May 2020 | A1 |
| 20210037442 | Ong et al. | Feb 2021 | A1 |
| Number | Date | Country |
|---|---|---|
| 2011203511 | Jul 2011 | AU |
| 104885477 | Sep 2015 | CN |
| Entry |
|---|
| First Office Action for Chinese Application No. 202011269309.2 dated Feb. 26, 2024. 6 pages. |
| Montazeri, Behnam et al. “Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities.” SIGCOMM '18, Aug. 20-25, 2018, Budapest, Hungary. pp. 221-235. |
| Rajiullah, Mohammad. “Towards a Low Latency Internet: Understanding and Solutions.” Sep. 2015. Dissertation. Department of Computer Science, Karlstad University, Sweden. 58 pages. |
| Vernersson, Andreas. “Analysis of UDP-based Reliable Transport using Network Emulation.” 2015. Master's Thesis. Master of Science in Engineering Technology Computer Science and Engineering. Luleå University of Technology. 93 pages. |
| Wu, Haitao et al. “ICTCP: Incast Congestion Control for TCP in Data-Center Networks.” IEEE/ACM Transactions on Networking, vol. 21, No. 2, Apr. 2013. pp. 345-358. |
| Partial European Search Report for European Patent Application No. 20207349.0 dated Apr. 1, 2021. 14 pages. |
| Extended European Search Report for European Patent Application No. 20207349.0 dated Jul. 5, 2021. 13 pages. |
| Office Action for European Patent Application No. 20207349.0 dated Jan. 12, 2023. 7 pages. |
| Notice of Grant for Chinese Patent Application No. 202011269309.2 dated Aug. 8, 2024. 4 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20230421657 A1 | Dec 2023 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62947036 | Dec 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 17857620 | Jul 2022 | US |
| Child | 18367679 | US | |
| Parent | 16819327 | Mar 2020 | US |
| Child | 17857620 | US |
