Low impact network debugging

Abstract

A method for debugging network connections that involves receiving a packet for a first destination by a network interface card (NIC), classifying the packet, sending the packet to a first receive ring associated with the NIC, and sending the packet from the first receive ring to a first virtual NIC, where the first virtual NIC is associated with a second destination, and where the second destination is used to debug the first destination.

Description

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows a schematic diagram of a system in accordance with one or more embodiments of the invention.

FIG. 2 shows a flow diagram in accordance with one or more embodiments of the invention.

FIG. 3 shows a computer system in accordance with one or more embodiments of the invention.

DETAILED DESCRIPTION

Specific embodiments of the invention will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

In the following detailed description of embodiments of the invention, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

In general, embodiments of the invention provide a method and apparatus to debug network connections. A network connection involves the transport of packets between a source and a destination. When packets between the source and destination are dropped, missing, misdirected, or arrive out of order, the connection between the source and destination is examined at various levels (e.g., transport layer, network layer, etc.) in order to determine where errors are occurring.

Embodiments of the invention provide a method and apparatus to debug network connections at high bandwidth sites while incurring minimal computational cost. In one embodiment of the invention, packets from a problematic connection are isolated by a network interface card (NIC) and redirected to a virtual NIC, where the packets may be further redirected to a test machine, offline storage, or a processor for real-time processing. Each packet may also be copied and the copy sent to the original packet destination so that the original connection may be observed for anomalies. In addition, embodiments of the invention allow the connection state to be captured and stored once a packet is transmitted or received by the NIC to further aid in determining the condition of the network connection.

FIG. 1 shows a schematic diagram of a system in accordance with one or more embodiments of the invention. As shown in FIG. 1, the system includes a host (100), a NIC (105), a virtual NIC (130), a debug virtual NIC (140), a virtual network stack (145), a debug virtual network stack (155), a service (175), a debug service (185), a network (110), a test client (160), an offline storage (165), and a test machine (170). Each of these components is described below.

The NIC (105) provides an interface between the host (100) and the network (110) (e.g., a local area network, a wide area network, a wireless network, etc.). More specifically, the NIC (105) includes a network interface (NI) (i.e., the hardware on the NIC used to interface with the network (110)). For example, the NI may correspond to an RJ-45 connector, a wireless antenna, etc. The packets received by the NI are then forwarded to other components on the NIC (105) for processing. In one embodiment of the invention, the NIC (105) includes one or more receive rings (e.g., receive ring 1 (115), receive ring 2 (125)). In one embodiment of the invention, the receive rings (e.g., receive ring 1 (115), receive ring 2 (125)) correspond to portions of memory within the NIC (105) used to temporarily store packets received from the network (110). In one or more embodiments of the invention, the receive rings (e.g., receive ring 1 (115), receive ring 2 (125)) are implemented as ring buffers in the NIC (105). In one embodiment of the invention, the classifier (108) is configured to analyze the incoming network traffic, typically in the form of packets, received from the network (110).

In one embodiment of the invention, analyzing individual packets includes determining to which of the receive rings (e.g., receive ring 1 (115), receive ring 2 (125)) each packet is sent. In one embodiment of the invention, analyzing the packets by the classifier (108) includes analyzing one or more fields in each of the packets to determine to which of the receive rings (e.g., receive ring 1 (115), receive ring 2 (125)) the packets are sent. As an alternative, the classifier (108) may use the contents of one or more fields in each packet as an index into a data structure that includes information necessary to determine to which receive ring (e.g., receive ring 1 (115), receive ring 2 (125)) that packet is sent. The classifier (108) may be implemented entirely in hardware (i.e., the classifier (108) may be a separate microprocessor embedded on the NIC (105)), or the classifier (108) may be implemented in software stored in memory (e.g., firmware, etc.) on the NIC and executed by a microprocessor on the NIC (105).

In one or more embodiments of the invention, the host (100) may include a device driver (not shown) and one or more virtual NICs (e.g., virtual NIC (130), debug virtual NIC (140)). In one embodiment of the invention, the device driver provides an interface between the receive rings (e.g., receive ring 1 (115), receive ring 2 (125)) and the host (100). More specifically, the device driver (not shown) exposes the receive rings (e.g., receive ring 1 (115), receive ring 2 (125)) to the host (100). In one embodiment of the invention, each of the virtual NICs (e.g., virtual NIC (130), debug virtual NIC (140)) is associated with one or more receive rings (e.g., receive ring 1 (115), receive ring 2 (125)). In other words, a virtual NIC (e.g., virtual NIC (130), debug virtual NIC (140)) receives incoming packets from the corresponding receive ring (e.g., receive ring 1 (115), receive ring 2 (125)).

In one or more embodiments of the invention, the virtual NICs (e.g., virtual NIC (130), debug virtual NIC (140)) are operatively connected to services (e.g., service (175), debug service (185)) via virtual network stacks (e.g., virtual network stack (145), debug virtual network stack (155)). The virtual NICs (e.g., virtual NIC (130), debug virtual NIC (140)) provide an abstraction layer between the NIC (105) and the services (e.g., service (175), debug service (185)) on the host (100). More specifically, each virtual NIC (e.g., virtual NIC (130), debug virtual NIC (140)) operates like a NIC (105). For example, in one embodiment of the invention, each virtual NIC (e.g., virtual NIC (130), debug virtual NIC (140)) is associated with one or more Internet Protocol (IP) addresses, one or more ports, and is configured to handle one or more protocol types. Thus, while the host (100) may be operatively connected to a single NIC (105), services (e.g., service (175), debug service (185)) executing on the host (100) operate as if the host (100) is bound to multiple NICs. To each service (e.g., service (175), debug service (185)), the corresponding virtual NIC (e.g., virtual NIC (130), debug virtual NIC (140)) is indistinguishable from a physical NIC (105). Further, each virtual NIC (e.g., virtual NIC (130), debug virtual NIC (140)) includes properties of a physical NIC, such as link speed, Media Access Control (MAC) address, etc. Those skilled in the art will appreciate that an arbitrary number of virtual NICs (e.g., virtual NIC (130), debug virtual NIC (140)) may be created for a given physical NIC (105), subject to resource limitations.

In one embodiment of the invention, each virtual network stack (e.g., virtual network stack (145), debug virtual network stack (155)) includes functionality to process packets in accordance with various protocols used to send and receive packets (e.g., Transmission Communication Protocol (TCP), Internet Protocol (IP), User Datagram Protocol (UDP), etc.). Further, each virtual network stack (e.g., virtual network stack (145), debug virtual network stack (155)) may also include functionality, as needed, to perform additional processing on the incoming and outgoing packets. This additional processing may include, but is not limited to, cryptographic processing, firewall routing, debugging, rerouting, etc.

In one or more embodiments of the invention, the virtual network stacks (e.g., virtual network stack (145), debug virtual network stack (155)) correspond to network stacks with network layer and transport layer functionality. In one embodiment of the invention, network layer functionality corresponds to functionality to manage packet addressing and delivery on a network (e.g., functionality to support Internet Protocol, Address Resolution Protocol (ARP), Internet Control Message Protocol (ICMP), etc.). In one embodiment of the invention, transport layer functionality corresponds to functionality to manage the transfer of packets on the network (e.g., functionality to support TCP, UDP, Stream Control Transmission Protocol (SCTP), etc.).

In one or more embodiments of the invention, the system of FIG. 1 is used to debug a network connection between a client and a service (175). A test client (160) is configured to send packets to the service (175) over the network (110). In one or more embodiments of the invention, the test client (160) corresponds to a system with an IP address that is recognized by the classifier (108). Those skilled in the art will appreciate that the test client (160) does not have to be a physical system; the test client (160) may be run on a virtual machine or container, which may or may not reside on the host (100). In one or more embodiments of the invention, packets sent from the test client (160) are identified by the classifier (108) using fields for the source address and destination address in the packet header. For example, if the test client (160) has an IP address of 10.1.10.5 and the virtual NIC (130) has an IP address of 192.14.28.15, packets with a source address of 10.1.10.5 and a destination address of 192.14.28.15 are identified by the classifier as part of the debugging system.

In one or more embodiments of the invention, a debug virtual NIC (140), debug virtual network stack (155), and debug service (185) are set up to isolate packets originating from the test client (160) from packets originating from other sources. For example, the debug virtual NIC (140), debug virtual network stack (155), and debug service (185) are used to examine a network connection that sends packets to the service (175) by receiving and redirecting packets originating from the test client (160) without interfering with or accessing packets sent from other sources, such as real client machines communicating with the host (100). While debugging takes place, the service (175) is allowed to receive and process packets from other sources, as well as transmit packets to other clients (not shown). Those skilled in the art will appreciate that the system of FIG. 1 may be used to transmit packets to the test client (160) as well as receive packets from the test client (160). In addition, those skilled in the art will appreciate that multiple test clients (160) and multiple debug virtual NICs (140), debug virtual network stacks (155), and debug services (185) may be set up on the same host (100) to examine multiple connections.

In one or more embodiments of the invention, the debug virtual network stack (155) and debug service (185) are duplicates of the virtual network stack (145) and service (175), respectively. Packets sent from the test client (160) are received by the NIC (105) and classified by the classifier (108), which places the packets in the receive ring (e.g., receive ring 2 (125)) corresponding to the debug virtual NIC (140). The debug virtual NIC (140) may then send the packets to the debug virtual network stack (155). Errors in the virtual network stack (145) may be observed when the packets proceed through the debug virtual network stack (155). In addition, errors in the service (175) may be observed once the packets arrive at the debug service (185) after going through the debug virtual network stack (155).

Alternatively, packets that arrive at the debug virtual NIC (140) from the test client (160) may be sent to offline storage (165) or a test machine (170) for further processing. The host (100) may be directly connected to the offline storage (165) or test machine (170), or the debug virtual NIC (140) may send the packets to the offline storage (165) or test machine (170) through a network (110). In addition, packets that arrive at the debug virtual NIC (140) from the test client (160) may be processed in real-time by a processor (not shown) operatively connected to the debug virtual NIC (140).

In one or more embodiments of the invention, packets from the test client (160) arrive at the NIC (105), or are sent from the NIC (105) to the test client (160). Once the packets arrive or leave, the connection state between the test client (160) and the host (100) is stored. In one or more embodiments of the invention, the connection state corresponds to a TCP connection state. For example, the connection state between the test client (160) and the host (100) may be LISTEN, SYN-SENT, SYN-RECEIVED, ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT, or CLOSED. Storing the connection state at the transmission or receipt of a packet allows more information about the connection to be known, and as a result, easier identification of problems occurring within the connection.

In addition, a packet sent from the test client (160) may be copied once the packet arrives at the NIC (105). The original packet is placed in receive ring 2 (125) and sent to the debug virtual NIC (140) as described above, and the copy of the packet is placed in receive ring 1 (115) and sent to the virtual NIC (130). Those skilled in the art will appreciate that the original packet and the copy of the packet are identical; either the original packet or the copy may be sent to either receive ring (e.g., receive ring 1 (115), receive ring 2 (125)) with the same result in both scenarios. As the packet, or the copy of the packet, travels between the virtual NIC (130) and the service (175), problems occurring at the virtual network stack (145) and service (175) may be identified; the debug virtual network stack (155) and debug service (185) may not be implemented to save overhead. In such cases, the debug virtual NIC (140) may be used to send the packet, or a copy of the packet, as well as a snapshot of the connection state upon receipt of the packet, to a test machine (170), offline storage (165), or a processor (not shown) for storage and/or further processing.

FIG. 2 shows a flow diagram in accordance with one or more embodiments of the invention. First, a suspect connection is identified (Step 201). A suspect connection corresponds to a network connection with possible issues from, for example, the transmission of packets over the network infrastructure (e.g., routers, CAT-5 cable, etc.), the virtual network stack, the service, or socket. Next, the classifier is programmed to direct packets from the suspect connection to a particular virtual NIC (Step 203). In one or more embodiments of the invention, the virtual NIC corresponds to the debug virtual NIC of FIG. 1. As stated above, a test client may be set up to transmit packets along the suspect connection. As a result, only packets from that test client may be directed to the debug virtual NIC, while packets from other clients may be directed to the original packet destination.

Once the classifier has been programmed to direct packets from the suspect connection to the virtual NIC, a packet from the suspect connection is sent by the test client and received by the classifier (Step 205). The classifier then forwards the packet to the debug virtual NIC; the classifier may also optionally create a copy of the packet (Step 207) and forward the copy to the virtual NIC corresponding to the original packet destination. In addition, a snapshot of the connection state associated with the suspect connection (Step 209) is optionally taken once the packet is received by the classifier.

Once the packet arrives at the debug virtual NIC, the packet is further processed based on the debug mode of the debug virtual NIC (Step 211). If the debug mode is offline, the packet is written to offline storage (Step 213) and processed offline (Step 215) so that processing does not interfere with the flow of traffic through the NIC. Alternatively, if the debug mode is test machine, the packet is transmitted to a test machine (Step 217), where the packet is processed (Step 219) so as to not interfere with activity on the host. As stated above, the packet may be sent to offline storage or the test machine through a direct connection with the storage or test machine, or through the network. If the debug mode is real-time, a processor is used to process the packet in real-time (Step 221). The processor may be a Central Processing Unit (CPU) operatively connected to the debug virtual NIC, or the processor may be located elsewhere, such as on the NIC. In one or more embodiments of the invention, multiple debug modes take place once the packet is received by the debug virtual NIC. For example, the packet may be sent to a test machine as well as processed in real-time, or the packet may be processed using all three debug modes.

The following is an example covering one or more embodiments of the invention. The example is not intended to limit the scope of the invention. Assume that an network error is detected by the service (175 in FIG. 1). However, it is not clear what the source of the network error. In accordance with one or more embodiments of the invention, a test machine is configured to send packets to the suspect connection (i.e., VNIC (130 in FIG. 1), virtual network stack (145 in FIG. 1) and the service (175 in FIG. 1)).

Prior to sending the packets from the test machine, the classifier (108 in FIG. 1) in the NIC (105 in FIG. 1) is configured to identify packets from the test machine (based on, for example, source IP address) and send the packets to the first receive ring (115 in FIG. 1) and a copy of the packet to the second receive ring (125 in FIG. 1). In addition, prior to sending the packets from the test machine, a debug virtual NIC (140 in FIG. 1), a debug virtual network stack (155 in FIG. 1) and a debug service (185 in FIG. 1) are also created.

Packets from the test machine are subsequently sent to the NIC (105 in FIG. 1). The packets sent from the test machine are for the service (175 in FIG. 1) (i.e., the destination IP address in the packet corresponds to the IP address of the service). The NIC (105 in FIG. 1), upon receiving the packets, sends the packet to the first receive ring (115 in FIG. 1) and a copy of the packet to the second receive ring (125 in FIG. 1).

At this stage, the progress of the packet through the suspect connection may be monitored. In addition, the progress of the copy of the packet may also be monitored as it progresses through the debug virtual NIC (140 in FIG. 1), the debug virtual network stack (155 in FIG. 1) and the debug service (185 in FIG. 1). In one embodiment of the invention, a user may have root access to the debug virtual NIC (140 in FIG. 1), the debug virtual network stack (155 in FIG. 1) and the debug service (185 in FIG. 1) and, accordingly, may perform more in-depth analysis of progress of the copy of the packet as it progresses through the debug virtual NIC (140 in FIG. 1), the debug virtual network stack (155 in FIG. 1) and the debug service (185 in FIG. 1).

In one embodiment of the invention, the debug service (185 in FIG. 1) may be useful in cases where root access to the service (175 in FIG. 1) has not been granted. Further, by issuing packets from a test machine and analyzing the progress of the packets through the suspect connection, the user may be able to debug the suspect connection without viewing packets associated with other clients of the service.

The invention may be implemented on virtually any type of computer regardless of the platform being used. For example, as shown in FIG. 3, a computer system (300) includes a processor (302), associated memory (304), a storage device (306), and numerous other elements and functionalities typical of today's computers (not shown). The computer (300) may also include input means, such as a keyboard (308) and a mouse (310), and output means, such as a monitor (312). The computer system (300) is connected to a local area network (LAN) or a wide area network (e.g., the Internet) (not shown) via a network interface connection (not shown). Those skilled in the art will appreciate that these input and output means may take other forms.

Further, those skilled in the art will appreciate that one or more elements of the aforementioned computer system (300) may be located at a remote location and connected to the other elements over a network. Further, the invention may be implemented on a distributed system having a plurality of nodes, where each portion of the invention (e.g., test machine, test client, network interface card, etc.) may be located on a different node within the distributed system. In one embodiment of the invention, the node corresponds to a computer system. Alternatively, the node may correspond to a processor with associated physical memory. The node may alternatively correspond to a processor with shared memory and/or resources. Further, software instructions to perform embodiments of the invention may be stored on a computer readable medium such as a compact disc (CD), a diskette, a tape, a file, or any other computer readable storage device.

While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.

Claims

1. A method for debugging network connections, comprising: receiving a packet for a first destination by a network interface card (NIC);classifying the packet;sending the packet to a first receive ring associated with the NIC; andsending the packet from the first receive ring to a first virtual NIC,wherein the first virtual NIC is associated with a second destination, andwherein the second destination is used to debug the first destination.

2. The method of claim 1, further comprising: creating a copy of the packet after classification;sending the copy of the packet to a second receive ring associated with the NIC; andsending the copy of the packet from the second receive ring to a second virtual NIC, wherein the second virtual NIC is associated with the first destination.

3. The method of claim 1, further comprising: recording a Transmission Control Protocol (TCP) connection state associated with the packet when the packet is received.

4. The method of claim 1, further comprising: writing the packet from the first virtual NIC to an offline storage.

5. The method of claim 1, further comprising: sending the packet from the first virtual NIC to a test machine.

6. The method of claim 1, further comprising: processing the packet from the first virtual NIC using a processor in real-time.

7. The method of claim 1, further comprising: sending the packet from the first virtual NIC to a debug virtual network stack, wherein the debug virtual network stack is associated with the second destination.

8. A system, comprising: a host, comprising: a first destination;a second destination, wherein the second destination is used to debug the first destination;a first virtual NIC associated with the second destination; anda network interface card (NIC), operatively connected to the host, comprising a first receive ring,wherein the NIC is configured to: receive a packet for the first destination;classify the packet;send the packet to the first receive ring; andsend the packet from the first receive ring to the first virtual NIC.

9. The system of claim 8, wherein the NIC further comprises a second receive ring, wherein the host further comprises a second virtual NIC, and wherein the NIC is further configured to: create a copy of the packet after classification;send the copy of the packet to the second receive ring; andsend the copy of the packet from the second receive ring to the second virtual NIC, wherein the second virtual NIC is associated with the first destination.

10. The system of claim 8, further comprising: an offline storage, wherein the host is further configured to write the packet from the first virtual NIC to the offline storage.

11. The system of claim 8, further comprising: a test machine, wherein the host is further configured to pass the packet from the first virtual NIC to the test machine.

12. The system of claim 8, wherein the host further comprises: a processor, wherein the host is further configured to process the packet from the virtual NIC using the processor in real-time.

13. The system of claim 8, wherein the host is further configured to: record a Transmission Control Protocol (TCP) connection state associated with the packet when the packet is received.

14. The system of claim 8, wherein the host is further configured to: pass the packet from the virtual NIC to a debug virtual network stack, wherein the debug virtual network stack is associated with the second destination.

15. A computer usable medium having computer readable program code embodied therein for causing a computer system to execute a method for debugging network connections, the method comprising: receiving a packet for a first destination by a network interface card (NIC);classifying the packet;sending the packet to a first receive ring associated with the NIC; andsending the packet from the first receive ring to a first virtual NIC,wherein the first virtual NIC is associated with a second destination, andwherein the second destination is used to debug the first destination.

16. The computer usable medium of claim 15, the method further comprising: creating a copy of the packet after classification;sending the copy of the packet to a second receive ring associated with the NIC; andsending the copy of the packet from the second receive ring to a second virtual NIC, wherein the second virtual NIC is associated with the first destination.

17. The computer usable medium of claim 15, the method further comprising: recording a Transmission Control Protocol (TCP) connection state associated with the packet when the packet is received.

18. The computer usable medium of claim 15, the method further comprising: writing the packet from the first virtual NIC to an offline storage.

19. The computer usable medium of claim 15, the method further comprising: sending the packet from the first virtual NIC to a test machine.

20. The computer usable medium of claim 15, the method further comprising: processing the packet from the first virtual NIC using a processor in real-time.