Patent Application
20230291673
The subject matter of this application relates to improved systems and methods that deliver CATV, digital, and Internet services to customers.
Cable Television (CATV) services have historically provided content to large groups of subscribers from a central delivery unit, called a “head end,” which distributes channels of content to its subscribers from this central unit through a branch network comprising a multitude of intermediate nodes. Modern Cable Television (CATV) service networks, however, not only provide media content such as television channels and music channels to a customer, but also provide a host of digital communication services such as Internet Service, Video-on-Demand, telephone service such as VoIP, and so forth. These digital communication services, in turn, require not only communication in a downstream direction from the head end, through the intermediate nodes and to a subscriber, but also require communication in an upstream direction from a subscriber and to the content provider through the branch network.
To this end, such CATV head ends included a separate Cable Modem Termination System (CMTS), used to provide high speed data services, such as video, cable Internet, Voice over Internet Protocol, etc. to cable subscribers. Typically, a CMTS will include both Ethernet interfaces (or other more traditional high-speed data interfaces) as well as RF interfaces so that traffic coming from the Internet can be routed (or bridged) through the Ethernet interface, through the CMTS, and then onto the optical RF interfaces that are connected to the cable company's hybrid fiber coax (HFC) system. Downstream traffic is delivered from the CMTS to a cable modem in a subscriber's home, while upstream traffic is delivered from a cable modem in a subscriber's home back to the CMTS. Many modern CATV systems have combined the functionality of the CMTS with the video delivery system (EdgeQAM) in a single platform called the Converged Cable Access Platform (CCAP). Still other modern CATV architectures (referred to as Distributed Access Architectures or DAA) relocate the physical layer (e.g., a Remote PHY or R-PHY architecture) and sometimes the MAC layer as well (e.g., a Remote MACPHY or R-MACPHY architecture) of a traditional CCAP by pushing it/them to the network's fiber nodes. Thus, while the core in the CCAP performs the higher layer processing, the remote device in the node converts the downstream data sent by the core from digital-to-analog to be transmitted on radio frequency, and converts the upstream RF data sent by cable modems from analog-to-digital format to be transmitted optically to the core.
Regardless of which architectures were employed, historical implementations of CATV systems bifurcated available bandwidth into upstream and downstream transmissions i.e., data was only transmitted in one direction across any part of the spectrum. For example, early iterations of the Data Over Cable Service Interface Specification (DOC SIS) specified assigned upstream transmissions to a frequency spectrum between 5 MHz and 42 MHz and assigned downstream transmissions to a frequency spectrum between 50 MHz and 750 MHz. Later iterations of the DOCSIS standard expanded the width of the spectrum reserved for each of the upstream and downstream transmission paths, the spectrum assigned to each respective direction did not overlap.
Packet Loss is a natural part of the Internet, occurring in cables, network elements (like routers), etc. The cause can be from noise on a channel (causing the packet's bits to be corrupted), can be caused by packet congestion in a network element that leads to a buffer overflow (causing the packet to be dropped at the tail of the buffer), or can be caused by the Transmission Control Protocol (TCP) probing for new maximum bandwidth capacities.
TCP and other higher-layer apps (like QUIC—which runs on top of UDP) can ameliorate packet loss by re-transmissions, but this solution increases latencies and also degrades throughputs of the connections in TCP and higher-layers, since it couples into the TCP or higher-layer app congestion control algorithms that limit throughputs as a result of detected packet loss.
When packet losses are causing undesirable side-effects (like higher latencies and lower throughputs), it may be desirable to find a technique that permits network operators to quickly identify the location of the packet loss so that corrective actions can be taken, such as increasing the link capacity on a particular network link or adding more links between network endpoints.
Even when packets are not lost, packet delay and jitter also degrade quality of service in communications networks. Packet delay is the time taken to send data packets over a network connection, and this delay varies based on factors such as network congestion, changes in the path taken by a packet when traversing the network between a source and destination, and variations in buffer depths in routers. The variation in that delay is called jitter, and adversely affects the services provided over the network, particularly in real-time applications, such as video conferencing, VoIP calls, live streaming, online gaming, etc. Jitter is noticed in the form of video or audio artifacts, static, distortion, and dropped calls.
What is desired, therefore, are systems and methods that locate the source of packet loss, packet latency, and/or packet jitter in the network.
For a better understanding of the invention, and to show how the same may be carried into effect, reference will now be made, by way of example, to the accompanying drawings, in which:
As noted previously, packet loss, packet latency, and packet jitter are each phenomenon that adversely impact quality of service provided over a communications network, and therefore any systems or methods that would assist in determining the location of conditions that are causing these phenomena e.g., packets being dropped, would be immensely helpful in managing the network in that it would help operators more quickly locate and correct the issue, leading to greatly improved customer satisfaction. Such solutions would be beneficial in a wide variety of communications architectures and services, including DOC SIS services, PON architectures, any communications system employing routers, including wireless networks such as WiFi and 5G, as well as Citizen's Broadcast Radio Service (CBRS). The present specification discloses systems and methods that provide such solutions across this broad array of architectures, and in a low-cost manner that that does not require complex additions to the network.
For example, the systems and methods disclosed in the present specification leverage the Transmission Control Protocol (TCP) that is already ubiquitously used in modern communications technologies.
For every packet transmitted from a Server process Ps on processor X (with IP Address Ix) to a client process Pc on processor Y (with IP Address Iy), there is a unique TCP port number (S_Port) assigned to the TCP port on the Server process and another unique TCP port number (C_Port) assigned to the TCP port on the Client process. The S_Port is unique within the scope of the Server processor X with IP Address Ix, and the C_Port is unique within the scope of the Client processor Y with IP Address Iy).
The TCP protocol used by the disclosed systems and methods utilizes a TCP “sequence value” (SEQ) associated with packet flows in each direction on the TCP connection between the server 12 and the client 14. A TCP Sequence Number is a 4-byte field in the TCP header (shown and described later in this specification with respect to
For the Left-to-Right (L2R) Flowing Packet Stream (shown in
Referring specifically to
The SEQ number of the next packet sent by the server will be N0+B0, i.e. each packet sent by the server 12 includes a SEQ number that is a running track of all the bytes sent in the process. Thus, the SEQ numbers of the packets sent by the server 12 are determined solely by the data stored on the server, and do not account for acknowledgments received from the client. Assuming that the number of bytes in the next data packet's payload is B1, then the ACK number sent back from the client after receiving that packet would be ACK=N0+B0+B1, again keeping a running count of the bytes of all data received. Those of ordinary skill in the art will appreciate that ACKs can be piggybacked in a normal data packet or sent in their own packet.
Referring to
Disclosed in the present specification is a novel network monitoring unit 22 positioned at a location in a network that both monitors traffic exchanged between tow endpoints, to extract relevant data by which a lost packet may be detected, as well as divides the network into quadrants such that the quadrant in which the lost packet may be identified. Referring specifically to
The network monitoring unit 22 may be positioned in a network in any appropriate manner. For example,
Referring to
As can be seen in this procedure, both the server 12 and the client device 14 can easily determine whether any packets have not yet been acknowledged, perhaps have being dropped, simply by comparing adjacent SEQ/ACK numbers; every ACK packet received by a server should have a value that matches the SEQ number of a packet already, or to be sent and every packet with an SEQ number received from the client should match the ACK number of a response already sent.
The right side of
The disclosed systems and methods provide for enhanced information about packet loss not previously attainable in the techniques previously described. The disclosed systems and methods not only identify when packet loss has occurred, but also are preferably capable of identifying the packet loss rate i.e., the number of packet losses occurring in the forward-going packet stream per second, and in some embodiments are also capable of estimating changes in average throughput of the forward-going packet stream resulting from the loss of a packet, which impacts the TCP Congestion Control Algorithm. The packet loss rate may be identified by dividing the packet loss count by the time of observation. The estimate of the change in average throughput may be determined by calculating the bps rate for a window of time before the packet loss occurred to the bps rate for a window of time after the packet loss occurred; the bps rates may for example calculated by dividing the total bytes passing by the time of observation.
The disclosed systems and methods are also preferably capable of identifying locational information as to where the packet loss occurred, and in particular, identifying which one of the four quadrants, shown in
If (at the network monitoring unit 22) the SEQ Number S(i+1) for a packet P(i+1) ever shows up and is greater than the predicted value that was predicted by the formula above, then that is likely to identify a packet loss that occurred in the Forward-Ingress Quadrant . . . where packet P(i+1) was actually dropped and the packet that came in at the apparent spot for P(i+1) is actually packet P(i+2) with the SEQ Number S(i+2). Typically, S(i+2)>S(i+1), so seeing that SEQ Number arrive as a value that is higher than expected is the trigger indicating that a packet may have been dropped in the Forward-Ingress Quadrant. As previously noted, there are circumstances when packets are delayed, but not dropped, when traversing a network, thus, the network monitoring unit 22 may not initially flag a packet as being dropped until three consecutive subsequent packets (i.e., packets P(3), P(4), and P(5)) have all been received without receipt of packet P(2). This example is analogous to employing the “triple duplicate acknowledgment” rule, but of course any other threshold may be used consistently with the disclosed system and methods.
Referring specifically to
It should be noted that, although
Similarly, both the server 12 and the client 14 may also be connected to a wide area network through respective content delivery networks (CDNs), and therefore some embodiments will have a first network monitoring unit 22 proximate the edge of the CDN serving the server, and a second network monitoring unit serving the client device.
As noted earlier, in addition to dropped packets, network latency and jitter also degrade quality of service provided by communications networks. The disclosed network monitoring unit 22 is therefore also preferably capable of measuring the latency and jitter as packets traverse specific portions of a communications network. Referring for example to
Thus, the north round trip latency 42 adds together the latency in the Reverse-Egress Quadrant, the packet processing delay in the server 12, and the latency in the Forward-Ingress Quadrant, Similarly, the south round trip latency 44 adds together the latency in the Forward-Egress Quadrant, the packet processing delay in the client device 14, and the latency in the Reverse-Ingress Quadrant. Those of ordinary skill in the art will recognize that the packets leaving the network monitoring unit are not the same packets returning in either of these “round trips.”
Determining the north round-trip latency 42 and south round-trip latency 44 at the network monitoring unit 22 can help operators determine where excessive latency is occurring in a network with latency issues. This can help to steer maintenance personnel directly to problems. For example, in a DOC SIS network with the network monitoring unit 22 near the CMTS, north latency issues point to the Internet as the source of the problem, while South latency issues point to the DOC SIS network as the source of the problem.
As just noted, embodiments of the disclosed network monitoring unit may preferably be capable of measuring the north round trip latency 42. Specifically, for every packet entering from the client device 14—i.e., packets going from south-to-north, the network monitoring unit may record the SEQ Number S(i), the TCP Payload Length L(i), and the start timestamp Ts(i) when the packet passed through the network monitoring unit 22. Also, the network monitoring unit 22 may preferably store the packet's Source IP, Dest IP, Source Port, Dest Port, and Protocol (TCP or UDP), collectively referred to as a “5-tuple.” Similarly, for every acknowledgment entering the network monitoring unit from the server 12, i.e., packets going from north-to-south, the network monitoring unit 22 may record the ACK Number A(i) and the final timestamp Tf(i) when the packet passed by the network monitoring unit 22. Also, the network monitoring unit 22 may preferably store the Source IP, Dest IP, Source Port, Dest Port, and Protocol (TCP or UDP) of the packets (the “5-tuple” containing these acknowledgments.
With this information, the network monitoring unit may, for each ACK number monitored within a particular 5-tuple, calculate the associated “north round trip” Latency Delay time D(i) as being D(i)=Tf(i)−Ts(i). All of the calculated Latency Delay times D(i) may be stored, along with various statistics (avg, min, max, pdf) that can be calculated from the collection of latency delay times.
Embodiments of the disclosed network monitoring unit may preferably also be capable of measuring the south round trip latency 44. Specifically, for every packet entering from the server 12—i.e., packets going from north-to-south, the network monitoring unit may record the SEQ Number S(i), the TCP Payload Length L(i), and the start timestamp Ts(i) of when the packet passed through the network monitoring unit 22. Also, the network monitoring unit may preferably store the packet's Source IP, Dest IP, Source Port, Dest Port, and Protocol (TCP or UDP). Similarly, for every acknowledgment entering the network monitoring unit from the client 14, i.e., packets going from south-to-north, the network monitoring unit 22 may record the ACK Number A(i) and the final timestamp Tf(i) when the packet passed through the network monitoring unit 22. Also, the network monitoring unit 22 may preferably store the Source IP, Dest IP, Source Port, Dest Port, and Protocol (TCP or UDP) of the packets containing these acknowledgments.
With this information, the network monitoring unit may, for each ACK number monitored within a particular 5-tuple, calculate the associated “south round trip” Latency Delay time D(i) as being D(i)=Tf(i)−Ts(i). All the calculated Latency Delay times D(i) may be stored, along with various statistics (avg, min, max, pdf) that can be calculated from the collection of latency delay times.
With respect to measuring performance characteristics related to jitter, along with the location of a source of such jitter, one technique may simply be approximated based on the foregoing latency measurements by calculating the maximum latency minus the minimum latency over sequential temporal windows Twi. Disclosed, however, are other embodiments that determine jitter statistics in more detail. Such disclosed embodiments collect data in a manner similar to that with respect to latency as described above, meaning that data-collection/calculations are performed on a 5-tuple basis and that measurements are made with respect to a northbound round-trip jitter and a southbound round-trip jitter, thereby permitting location of the source of the jitter.
Specifically, for purposes of illustration and in reference to
With this information, the network monitoring unit may, for each ACK number monitored within a particular 5-tuple, calculate the associated “north round trip” Latency Delay time D(i) as being D(i)=Tf(i)−Ts(i). All of the calculated Latency Delay times D(i) may be stored.
From this stored data, the network monitoring unit may preferably collect a variety of statistics related to delay and jitter that occurs over the north-round-trip segment of the quadrants shown in
Referring to
The variable delay for all packets in the scatter plot may be plotted as a probability mass function (pmf) 60, which charts the number of occurrences (y-axis) in the data set of packets of a particular variable delay (x-axis). From pmf 60, statistics may be collected (mean, mode, min, max, std deviation, etc) for the variable delay for that particular flow. This process can be repeated for other 5-tuple flows, and the results can be blended and compared. Jitter for a particular packet flow is measured as the x-axis width 62 of the pmf 60. A pmf 60 of vertical distances to the line 58 for all points in all of the delay vs packet length scatter plots for all 5-tuple flows creates average jitter statistics for all subscribers, in the north-round trip portion of the network.
Those of ordinary skill in the art wis appreciate that the procedure that was just described with respect to the north-round-trip of the network may be repeated with respect to the south round-trip portion of the network.
An IP network 108 may include a web server 110 and a data source 112. The web server 110 is a streaming server that uses the IP protocol to deliver video-on-demand, audio-on-demand, and pay-per view streams to the IP network 108. The IP data source 112 may be connected to a regional area or backbone network (not shown) that transmits IP content. For example, the regional area network can be or include the Internet or an IP-based network, a computer network, a web-based network or other suitable wired or wireless network or network system.
At the head end 102, the various services are encoded, modulated and up-converted onto RF carriers, combined onto a single electrical signal and inserted into a broadband optical transmitter. A fiber optic network extends from the cable operator's master/regional head end 102 to a plurality of fiber optic nodes 104. The head end 102 may contain an optical transmitter or transceiver to provide optical communications through optical fibers 103. Regional head ends and/or neighborhood hub sites may also exist between the head end and one or more nodes. The fiber optic portion of the example HFC network 100 extends from the head end 102 to the regional head end/hub and/or to a plurality of nodes 104. The optical transmitter converts the electrical signal to a downstream optically modulated signal that is sent to the nodes. In turn, the optical nodes convert inbound signals to RF energy and return RF signals to optical signals along a return path.
Each node 104 serves a service group comprising one or more customer locations. By way of example, a single node 104 may be connected to thousands of cable modems or other subscriber devices 106. In an example, a fiber node may serve between one and two thousand or more customer locations. In an HFC network, the fiber optic node 104 may be connected to a plurality of subscriber devices 106 via coaxial cable cascade 111, though those of ordinary skill in the art will appreciate that the coaxial cascade may comprise a combination of fiber optic cable and coaxial cable. In some implementations, each node 104 may include a broadband optical receiver to convert the downstream optically modulated signal received from the head end or a hub to an electrical signal provided to the subscribers' devices 106 through the coaxial cascade 111. Signals may pass from the node 104 to the subscriber devices 106 via the RF cascade of amplifiers, which may be comprised of multiple amplifiers and active or passive devices including cabling, taps, splitters, and in-line equalizers. It should be understood that the amplifiers in the RF cascade may be bidirectional, and may be cascaded such that an amplifier may not only feed an amplifier further along in the cascade but may also feed a large number of subscribers. The tap is the customer's drop interface to the coaxial system. Taps are designed in various values to allow amplitude consistency along the distribution system.
The subscriber devices 106 may reside at a customer location, such as a home of a cable subscriber, and are connected to the cable modem termination system (CMTS) 120 or comparable component located in a head end. A client device 106 may be a modem, e.g., cable modem, MTA (media terminal adaptor), set top box, terminal device, television equipped with set top box, Data Over Cable Service Interface Specification (DOCSIS) terminal device, customer premises equipment (CPE), router, or similar electronic client, end, or terminal devices of subscribers. For example, cable modems and IP set top boxes may support data connection to the Internet and other computer networks via the cable network, and the cable network provides bi-directional communication systems in which data can be sent downstream from the head end to a subscriber and upstream from a subscriber to the head end.
References are made in the present disclosure to a Cable Modem Termination System (CMTS) in the head end 102. In general, the CMTS is a component located at the head end or hub site of the network that exchanges signals between the head end and client devices within the cable network infrastructure. In an example DOCSIS arrangement, for example, the CMTS and the cable modem may be the endpoints of the DOCSIS protocol, with the hybrid fiber coax (HFC) cable plant transmitting information between these endpoints. It will be appreciated that architecture 100 includes one CMTS for illustrative purposes only, as it is in fact customary that multiple CMTSs and their Cable Modems are managed through the management network.
The CMTS 120 hosts downstream and upstream ports and contains numerous receivers, each receiver handling communications between hundreds of end user network elements connected to the broadband network. For example, each CMTS 120 may be connected to several modems of many subscribers, e.g., a single CMTS may be connected to hundreds of modems that vary widely in communication characteristics. In many instances several nodes, such as fiber optic nodes 104, may serve a particular area of a town or city. DOCSIS enables IP packets to pass between devices on either side of the link between the CMTS and the cable modem.
It should be understood that the CMTS is a non-limiting example of a component in the cable network that may be used to exchange signals between the head end and subscriber devices 106 within the cable network infrastructure. For example, other non-limiting examples include a Modular CMTS (M-CMTSTM) architecture or a Converged Cable Access Platform (CCAP).
An EdgeQAM (EQAM) 122 or EQAM modulator may be in the head end or hub device for receiving packets of digital content, such as video or data, re-packetizing the digital content into an MPEG transport stream, and digitally modulating the digital transport stream onto a downstream RF carrier using Quadrature Amplitude Modulation (QAM). EdgeQAMs may be used for both digital broadcast, and DOCSIS downstream transmission. In CMTS or M-CMTS implementations, data and video QAMs may be implemented on separately managed and controlled platforms. In CCAP implementations, the CMTS and edge QAM functionality may be combined in one hardware solution, thereby combining data and video delivery.
The techniques disclosed herein may be applied to systems compliant with DOCSIS. The cable industry developed the international Data Over Cable System Interface Specification (DOCSIS®) standard or protocol to enable the delivery of IP data packets over cable systems. In general, DOCSIS defines the communications and operations support interface requirements for a data over cable system. For example, DOCIS defines the interface requirements for cable modems involved in high-speed data distribution over cable television system networks. However, it should be understood that the techniques disclosed herein may apply to any system for digital services transmission, such as digital video or Ethernet PON over Coax (EPoc). Examples herein referring to DOCSIS are illustrative and representative of the application of the techniques to a broad range of services carried over coax
Those of ordinary skill in the art will also recognize that the architecture of
Similarly, those of ordinary skill in the art will recognize that, although many embodiments were described in relation to the hairpin architecture of
It will be appreciated that the invention is not restricted to the particular embodiment that has been described, and that variations may be made therein without departing from the scope of the invention as defined in the appended claims, as interpreted in accordance with principles of prevailing law, including the doctrine of equivalents or any other principle that enlarges the enforceable scope of a claim beyond its literal scope. Unless the context indicates otherwise, a reference in a claim to the number of instances of an element, be it a reference to one instance or more than one instance, requires at least the stated number of instances of the element but is not intended to exclude from the scope of the claim a structure or method having more instances of that element than stated. The word “comprise” or a derivative thereof, when used in a claim, is used in a nonexclusive sense that is not intended to exclude the presence of other elements or steps in a claimed structure or method.
This application claims benefit of priority under 35 USC 119(e) to the filing date of U.S. Provisional Application No. 63/314,457 filed on Feb. 7, 2023, the contents of which are hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63314457 | Feb 2022 | US |
