At least some embodiments of the present invention generally relate to networking, and more particularly, to providing time markers for data streams.
Generally, to monitor and troubleshoot network operations, network traffic packets may be captured and stored for post-processing analysis later. The volume of data that might be stored can be large in high speed, high traffic volume networks. The post-processing aspect of the analysis takes the analysis out of a real-time mode of operation.
Typically, to perform a timing analysis, a network analyzing system needs to inspect a time stamp of each data packet. Typically, to perform data traffic analysis, the system needs to inspect every data packet to compare a time stamp of the data packet against a system's time. This results in an increased requirement for storage and processing capability of the network analyzing system, and reduces network analysis efficiency.
Exemplary embodiments of methods and apparatuses provide time markers for a data packet stream are described. In at least some embodiments, a time marker for a stream of data packets is generated. The packets are grouped into a block based on the time marker. The block of data packets is stored in a memory. In one embodiment, a block of the data packets is terminated based on the time marker. In one embodiment, a block of the data packets is terminated based on a size. In one embodiment, a block header is generated for the block. The block header can include, for example, data identifying a number of the packets in the block, and data identifying a duration of the block. In one embodiment, the time marker identifies a one second time boundary. In one embodiment, statistics associated with the packet stream is captured based on the time marker.
Other features of the present invention will be apparent from the accompanying drawings and from the detailed description which follows.
Embodiments of the present invention are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.
Exemplary embodiments of methods and apparatuses to provide time markers for a data packet stream are described. In at least some embodiments, a time marker for a data packet stream is generated. The packets are grouped into a block based on the time marker. The block of data packets is stored in a memory. In one embodiment, a block of the data packets is terminated based on the time marker. In one embodiment, a block of the data packets is terminated based on a size. In one embodiment, a block header is generated for the block. The block header can include, for example, data identifying a number of the packets in the block, and data identifying a duration of the block. In one embodiment, the time marker identifies a one second time boundary. In one embodiment, statistics associated with the packet stream is captured based on the time marker.
In at least some embodiments, a packet analyzing application needs to know when 1 second time boundaries passed, so the packet analyzing application can put statistics about the packets it sees into a correct time record. In at least some embodiments, by placing time markers into the packet stream it is analyzing, the application no longer has to break from what it is doing and go out to check with the system time stamp to see when time has passed. This makes overall packet analysis by a data processing system more efficient. These one second time boundaries further allow the processing system to handle time functions only once a second rather than performing timing analysis on every packet. The system can also take advantage of time based indexing because the mechanism allows the system to quickly determine if data packets matching a time filter are stored in a set of packets instead of having to inspect every packet.
In at least some embodiments, hardware that delivers data packets to software keeps its own timestamp counter. In at least some embodiments, the hardware time stamp counter is also synchronized with the system time stamp. In at least some embodiments, as packets are delivered from the hardware to the software, the hardware simultaneously keeps track of its time stamp counter. In at least some embodiments, when the timestamp passes a 1 second boundary, the hardware inserts a special (time) marker into the data packet stream indicating when the 1 second time boundary was crossed. As packets enter the software stack, they are further pre-aggregated into meaningful time bounded sets further decreasing the amount of processing required.
Various embodiments and aspects of the inventions will be described with reference to details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative of the invention and are not to be construed as limiting the invention. Numerous specific details are described to provide a thorough understanding of various embodiments of the present invention. It will be apparent, however, to one skilled in the art, that embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form, rather than in detail, in order to avoid obscuring embodiments of the present invention. Reference in the specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification do not necessarily refer to the same embodiment.
Unless specifically stated otherwise, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a data processing system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
Embodiments of the present invention can relate to an apparatus for performing one or more of the operations described herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a machine (e.g.; computer) readable storage medium, such as, but is not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable ROMs (EPROMs), electrically erasable programmable ROMs (EEPROMs), magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a bus.
The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required machine-implemented method operations. The required structure for a variety of these systems will appear from the description below.
In addition, embodiments of the present invention are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of embodiments of the invention as described herein.
Generally, a network refers to a collection of computers and other hardware components interconnected to share resources and information. Networks may be classified according to a wide variety of characteristics, such as the medium used to transport the data, communications protocol used, scale, topology, and organizational scope. Communications protocols define the rules and data formats for exchanging information in a computer network, and provide the basis for network programming. Well-known communications protocols include Ethernet, a hardware and link layer standard that is ubiquitous in local area networks, and the Internet protocol (IP) suite, which defines a set of protocols for internetworking, i.e. for data communication between multiple networks, as well as host-to-host data transfer e.g., Transmission Control Protocol (TCP), and application-specific data transmission formats, for example, Hypertext Transfer Protocol (HTTP). Methods and apparatuses to provide time markers for a packet stream described herein can be used for any of networks, protocols, and data formats.
As shown in
While
It will be apparent from this description that aspects of the present invention may be embodied, at least in part, in software. That is, the techniques may be carried out in a computer system or other data processing system in response to its processor, such as a microprocessor, executing sequences of instructions contained in a memory, such as ROM 107, volatile RAM 105, non-volatile memory 106, or a remote storage device. In various embodiments, hardwired circuitry may be used in combination with software instructions to implement embodiments of the present invention. Thus, the techniques are not limited to any specific combination of hardware circuitry and software nor to any particular source for the instructions executed by the data processing system. In addition, throughout this description, various functions and operations are described as being performed by or caused by software code to simplify description. However, those skilled in the art will recognize what is meant by such expressions is that the functions result from execution of the code by one or more processing units 103, e.g., a microprocessor, a field-programmable gate array (FPGA) and/or a microcontroller.
A machine readable medium can be used to store software and data which when executed by a data processing system causes the system to perform various methods of the present invention. This executable software and data may be stored in various places including for example ROM 107, volatile RAM 105, and non-volatile memory 106 as shown in
Thus, a machine readable medium includes any mechanism that provides (i.e., stores and/or transmits) information in a form accessible by a machine (e.g.; a computer, network device, cellular phone, personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.). For example, a machine readable medium includes recordable/non-recordable media (e.g., read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; and the like.
The methods of the present invention can be implemented using a dedicated hardware (e.g., using Field Programmable Gate Arrays (FPGAs), or Application Specific Integrated Circuit (ASIC) or shared circuitry (e.g., microprocessors or microcontrollers under control of program instructions stored in a machine readable medium). The methods of the present invention can also be implemented as computer instructions for execution on a data processing system, such as system 100 of
Generally, a FPGA is an integrated circuit designed to be configured by a customer or a designer after manufacturing. The FPGA configuration is generally specified using a hardware description language (HDL). FPGAs can be used to implement a logical function.
FPGAs typically contain programmable logic components (“logic blocks”), and a hierarchy of reconfigurable interconnects to connect the blocks. In most FPGAs, the logic blocks also include memory elements, which may be simple flip-flops or more complete blocks of memory.
Referring back to
In at least some embodiments, network processing unit 302 comprises a custom hardware that receives the packets coming over the network. In at least some embodiments, network processing unit has a function to keep track of the current time and insert a time marker into the data stream at every second boundary so that a downstream software knows that a second boundary has passed. Then all the next packets or events will be associated with that time second and then as the next second boundary passes then those packets are associated with the next time boundary. This allows the software downstream to not have to do many look ups or checks with the system operating system to figure out in which time interval the events happened.
In at least some embodiments, unit 302 has time counter 304 that counts current time and generates time markers to actually insert these time markers into the data stream that goes to the rest of the system or the software. The time markers are directly inserted into the data stream that provides an in-band messaging scheme. The time marker directly inserted into the data stream are formatted into a format complying with the software downstream.
That is, the time markers are inserted into the data stream in such a way that the format is not disturbed and this format is going to be understood by the downstream software.
Packets of a packet stream 402 arriving along a time line 401 are grouped into corresponding blocks. In at least some embodiments, the size of the block of packets does not exceed a predetermined upper threshold, e.g., 1 Megabyte or any other predetermined upper threshold size. In at least some embodiments, the size of the block of packets is not smaller than a predetermined lower threshold, e.g., 512 bytes, or any other predetermined lower threshold size.
As shown in
As shown in
In at least some embodiments, a block header is inserted at the beginning of the block. In at least some embodiments, a block header includes data identifying a number of the packets in the block, and a duration of the block. As shown in
Referring to
That is, the packets are grouped and stored within a large block of memory. So a downstream network analyzing application does not process each packet as it comes in; the network analyzing application will wait until a whole block of packets is stored into a host memory. The network application keeps track of whether or not it has a new block to read by checking a counter, such as counter 304.
In one embodiment, after storing a predetermined number of bytes of packets into a memory, a block header is generated. In one embodiment, a block header is generated and stored in a memory when a time marker is passed (e.g., every predetermined amount of seconds). For example, even if the packets are not received, block headers are generated once every predetermined time interval (e.g., every second) and are stored into a memory. In at least some embodiments, time markers are implemented as being embedded in a header block memory mechanism to store packets. In at least some embodiments, individual packets are received and grouped into blocks and stored in a host memory by a custom hardware, such as unit 302.
In at least some embodiments, memory 510 contains three levels of data: packet data, a packet report on top of the packet data, and a block header on top of the packet report. In at least some embodiments, packet data contain packet itself having data bytes. In at least some embodiments, the packet report header is generated for each packet. Generally, the packet report header contains data describing the packet, for example, an indication of what is in the packet. In at least some embodiments, the packet report header has a timestamp of the packet, an indication of number of bytes in the packet, and other data to assist processing the packet. A block header is generated on top of the packet report for each block stored in a memory.
In at least some embodiments, the block header is inserted at the beginning of a block. In at least some embodiments, the block header has information regarding how many packets are stored in that block. In at least some embodiments, the block header has a time stamp information, start of block, end of block timestamp information.
In at least some embodiments, the block header has the time stamp of a first packet in the block and the time stamp of a last packet in the block. In at least some embodiments, the block header is generated each time when a time boundary is passed (e.g., every second). In at least some embodiments, a block is terminated and a new block header is generated on the time boundary.
A block header is generated on the time boundary acts as a time marker inserted directly into the data stream. Inserting the time markers into the data stream increases the efficiency and simplifies packet stream analysis for a packet analyzing application.
In at least some embodiments, a processing unit including hardware, such as unit 302 of
Embodiments described herein make the packet analyzing more efficient. Because the time markers are added into the data stream, the packet analyzing application does not have to look somewhere else to see which packets are associated with each time bracket. The packets being analyzed have already been pre-categorized based on the time boundaries for efficient retrieval from the memory.
Inserting time markers on a time boundary for a data stream provides a mechanism for timing, for aggregating packets and for collecting statistics about the data stream even when there are no packets arriving. In at least some embodiments, inserting the time markers is a synchronized event that occurs on a time boundary so that even when no packets are received, statistics and information about the data stream on a predetermined time interval can still be collected.
For example, a packet analyzing application measures an application response time and for example, it takes 10 seconds for an event to occur. The typical application won't know that until the last packet comes in. If the packet never comes in the packet analyzing application can never have an updated statistic. By inserting time markers at a predetermined time interval into the data stream constantly as described herein, the statistics over that time interval can be dynamically obtained without a need to wait for a last packet to arrive. The processing overhead is greatly reduced by providing the time markers at a predetermined time interval, because the time interval does not need to be re-calculated each time the packet arrives.
Currently, if there's a time interval where packets don't arrive, a packet analyzing application cannot update the packet statistics. By inserting time markers into a data stream as described herein, packet statistics is updated based on the time mechanism (clock). In one embodiment, the same hardware that's providing a time stamp for each packet in a packet report also provides time markers (e.g., a predetermined interval ticks) for the statistics. That is, a real time information analysis of the traffic during a certain amount of time can be obtained based on the data that have been pre-organized based on the time markers. The network analyzing application doesn't need to look at every received packet or wait until the last packet comes to obtain the packet statistics.
In the foregoing specification, embodiments of the invention have been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope of the embodiments of the invention. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.