Programmable protocol parser for NIC classification and queue assignments

Information

  • Patent Grant
  • 11394610
  • Patent Number
    11,394,610
  • Date Filed
    Tuesday, July 23, 2019
    5 years ago
  • Date Issued
    Tuesday, July 19, 2022
    2 years ago
Abstract
Technologies for controlling operation of a compute node coupled to a computer network via a computing device that includes communications for communicating with the computer network and persistent instructions such as firmware for providing control functions to the computing device, wherein the control functions being defined at least in part by protocol data. An update control module of the computing device may receive update data from a remote node in the computer network via the communications, wherein the update data comprising new protocol data for the persistent instructions. A protocol parser module may parse the update data and generate metadata relating to the update data. A classifier module may receive rules for the control functions, wherein the rules are based at least in part on the update data and metadata. A compiler may compile the parsed update data to the persistent instructions for providing new control functions to the computing device based at least in part on the received rules.
Description
BACKGROUND

Modern computing devices have become ubiquitous tools for personal, business, and social uses. As such, many modern computing devices are capable of connecting to various data networks, including the Internet and corporate intranets, to retrieve and receive data communications over such networks. Oftentimes, a computing device connected to one network needs to communicate with a computing device connected on a different network.


A networking device, such as a network interface controller (NIC) (also known as a network interface card, network adapter, LAN adapter, and by similar terms) is a computer hardware component that connects a computer to a computer network. A NIC implements the electronic circuitry required to communicate using a specific physical layer and data link layer standard such as Ethernet, Wi-Fi or Token Ring. This provides a base for a full network protocol stack, allowing communication among small groups of computers on the same LAN and large-scale network communications through routable protocols, such as IP.


Current NICs classify packets and assign actions based on pre-defined, fixed packet types and/or offsets that are typically embodied in microcode that is flashed onto the NIC. Standard NIC architecture does not allow a user to efficiently define new protocols, packet types, and/or queue assignments to accommodate new or updated target applications. Such capabilities may be useful in a computing user experience (UX), particularly, but not exclusively, in software defined networking (SDN).


SDN is a networking architecture in which decisions regarding how network traffic is to be processed and the devices or components that actually process the network traffic are decoupled into separate planes (i.e., the control plane and the data plane). In SDN environments, a centralized SDN controller may be used to make forwarding decisions for network traffic instead of a network device such as, for example, a network switch. Typically, forwarding decisions are communicated to a network device operating in the SDN environment, which in turn forwards network packets associated with the network traffic to the next destination based on the forwarding decisions made by the SDN controller.





BRIEF DESCRIPTION OF THE DRAWINGS

The concepts described herein are illustrated by way of example and not by way of limitation in the accompanying figures. For simplicity and clarity of illustration, elements illustrated in the figures are not necessarily drawn to scale. Where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements.



FIG. 1 is a simplified block diagram of at least one embodiment of a system for managing network packet flow;



FIG. 2 is a simplified block diagram of at least one embodiment of an environment of a compute node of the system FIG. 1;



FIG. 3 is a simplified block diagram of at least one embodiment of an environment of a controller node of the system of FIG. 1;



FIG. 4 is a simplified illustration of at least one embodiment of a protocol specification and actions messaging configuration for a controller node;



FIGS. 5 and 6 are a simplified flow diagram of at least one embodiment of a method for updating protocol sequences suitable for use in the embodiments of FIGS. 1-3.





DETAILED DESCRIPTION OF THE DRAWINGS

While the concepts of the present disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the concepts of the present disclosure to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives consistent with the present disclosure and the appended claims.


References in the specification to “one embodiment,” “an embodiment,” “an illustrative embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may or may not necessarily include that particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. Additionally, it should be appreciated that items included in a list in the form of “at least one of A, B, and C” can mean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C). Similarly, items listed in the form of “at least one of A, B, or C” can mean (A); (B); (C); (A and B); (A and C); (B and C); or (A, B, and C).


The disclosed embodiments may be implemented, in some cases, in hardware, firmware, software, or any tangibly-embodied combination thereof. The disclosed embodiments may also be implemented as instructions carried by or stored on one or more non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors. A machine-readable storage medium may be embodied as any storage device, mechanism, or other physical structure for storing or transmitting information in a form readable by a machine (e.g., a volatile or non-volatile memory, a media disc, or other media device).


In the drawings, some structural or method features may be shown in specific arrangements and/or orderings. However, it should be appreciated that such specific arrangements and/or orderings may not be required. Rather, in some embodiments, such features may be arranged in a different manner and/or order than shown in the illustrative figures. Additionally, the inclusion of a structural or method feature in a particular figure is not meant to imply that such feature is required in all embodiments and, in some embodiments, may not be included or may be combined with other features.


Referring now to FIG. 1, an illustrative embodiment of system 100 for managing network packet flow includes a controller node 114 communicatively coupled to a plurality of compute nodes (104-110) via a network 112. In use, the system 100 of FIG. 1, as well as other embodiments disclosed herein, provides various functionalities relating packet forwarding, routing, filtering, etc. that may be accomplished on a local (NIC) level as opposed to a network switch level. As such, a controller and/or its north bound APIs can be configured to fully define a protocol. As discussed in more detail below, each compute node 104-110 include an associated NIC, or other networking device, which include a parser and a compiler to facilitate updating of control functions of the associated NIC (or other networking device). The parser may be programmed in substantially real-time or “on-the-fly” (e.g., while data traffic and/or packet flow is occurring) based on structured programmable parsing approaches that allow a user to define both protocol descriptors and a parse graph. Utilizing the disclosed parser in the NIC advantageously provides a better offload engine (e.g., in terms of dropping ACLs or pushing/popping tags based on on-the fly customized protocols) and a way to make packet processing more efficient in the host (e.g., by forwarding packets of a specific flow on the same PCIe bus as the consuming application or VM so there is no cross-QPI traffic). As such, the parser has the ability to understand optional tags and headers and filter accordingly. The compiler of the NIC (or other networking device) may be configured as a just-in-time compiler that uses a protocol specification language representation of a parse graph to create the required hardware updates to support the parse graph. The compiler may generate microcode, persistent instructions such as firmware updates, register writes, and so forth.


In the embodiment illustrated in FIG. 1, the compute node 104 (as well as the compute nodes 106-110) may be embodied as any type of computing device capable of performing the functions described herein. For example, the compute node 104 may be embodied as, but is not limited to, a computer, a desktop computer, a personal computer (PC), a tablet computer, a laptop computer, a notebook computer, a mobile computing device, a smart phone, a cellular telephone, a handset, a messaging device, a work station, a network appliance, a web appliance, a distributed computing system, a multiprocessor system, a processor-based system, a consumer electronic device, a digital television device, a set top box, and/or any other computing device configured to store and access data, and to execute electronic cloud software and related applications. Compute nodes 2-n (106-110) are similarly configured to the arrangement illustrated in compute node 104.


In the illustrative embodiment of FIG. 1, the compute node 104 includes a processor 120, an I/O subsystem 122, a memory 124, a data storage 126, communication circuitry comprising a NIC 130, and one or more peripheral devices 128. As can be seen in the illustrative figure, the NIC 130 may be equipped with its own memory 132 and a processor 134 for processing and updating protocol sequences and protocol parsing. Of course, the compute node 104 may include other or additional components, such as those commonly found in a digital apparatus and/or computer (e.g., various input/output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory 124, or portions thereof, may be incorporated in the processor 120 in some embodiments.


The processor 120 may be embodied as any type of processor capable of performing the functions described herein. For example, the processor 120 may be embodied as a single or multi-core processor(s), a single or multi-socket processor(s), digital signal processor, microcontroller, or other processor or processing/controlling circuit. Similarly, the memory 124 may be embodied as any type of volatile or non-volatile memory or data storage currently known or developed in the future and capable of performing the functions described herein. In operation, the memory 124 may store various data and software used during operation of the compute node 104 such as operating systems, applications, programs, libraries, and drivers.


The memory 124 may be communicatively coupled to the processor 120 via the I/O subsystem 122, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor 120, the memory 124, and other components of the compute node 104. For example, the I/O subsystem 122 may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, firmware devices, communication links (i.e., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem 122 may form a portion of a system-on-a-chip (SoC) and be incorporated, along with the processor 120, the memory 124, and other components of the compute node 104, on a single integrated circuit chip.


The NIC 130 of the compute node 104 may be embodied as any number of devices and/or circuitry for enabling communications between the compute node 104 and one or more other external electronic devices and/or systems and performing the functions described herein. Similarly, the peripheral devices 128 may be embodied as any number of additional input/output devices, interface devices, and/or other peripheral devices. The peripheral devices 128 may also include a display, along with associated graphics circuitry and, in some embodiments, may further include a keyboard, a mouse, audio processing circuitry (including, e.g., amplification circuitry and one or more speakers), and/or other input/output devices, interface devices, and/or peripheral devices.


In some embodiments, the memory 124 and/or the data storage 126 has stored therein one or more application programs, and the processor 120 is operable to execute the one or more application programs and control a display screen to display corresponding graphical information thereon. In some embodiments, the system 100 may include a controller node 114 configured to store and/or execute the one or more application programs, and in such embodiments the compute node 104 may operate as a thin-client device. Although only one such controller node 114 is shown in FIG. 1, it should be appreciated that, in other embodiments, system 100 may include any number of controller nodes.


The controller node 114 may be embodied as any type of server (e.g., a SDN server, web server, etc.) or similar computing device capable of performing the functions described herein. In the illustrative embodiment of FIG. 1, the controller node 114 includes a processor 150, an I/O subsystem 152, a memory 154, a data storage 156, a communication circuitry 162, and one or more peripheral devices 158. Components of the controller node 114 may be similar to the corresponding components of the compute node 104, the description of which is applicable to the corresponding components of the controller node 114 and is not repeated herein for clarity of the description.


In the illustrative embodiment, communication between compute nodes 104-110 and the controller node 114 takes place via the network 112, which may be operatively coupled to one or more network switches 170. In an embodiment, the network 112 may represent a wired and/or wireless network and may be or include, for example, a local area network (LAN), personal area network (PAN), storage area network (SAN), backbone network, global area network (GAN), wide area network (WAN), or collection of any such computer networks such as an intranet, extranet or the Internet (i.e., a global system of interconnected network upon which various applications or service run including, for example, the World Wide Web). Generally, the communication circuitry of compute node 104 (e.g., NIC 130) and the communication circuitry 162 of controller node 114 may be configured to use any one or more, or combination, of communication protocols to communicate with each other such as, for example, a wired network communication protocol (e.g., TCP/IP), a wireless network communication protocol (e.g., Wi-Fi.RTM., WiMAX), a cellular communication protocol (e.g., Wideband Code Division Multiple Access (W-CDMA)), and/or other communication protocols. As such, the network 112 may include any number of additional devices, such as additional computers, routers, and switches, to facilitate communications between compute node 104 and controller node 114.


Under various embodiments described in further detail below, the controller node 114 may be configured to provide updates to a NIC 130 located in any of compute nodes 104-110. In a SDN application, software, such as a remote controller, can define protocols at controller node 114 using a protocol definition language at runtime. The controller node 114 can generate a parse graph and device-specific microcode for transmission to compute node 104 to allow controller node 114 to compile the information and update the operation of the NIC 130. In another embodiment, the controller node 114 can generate a parse graph and register parameters for transmission to the compute node 104 to allow the controller node 114 to update the NIC 130 operation. In a server administration application, a system administration tool can define protocols at the controller node 114 using a protocol definition language at runtime. Similar to the SDN application, the controller node 114 can generate a parse graph and device-specific microcode for transmission and compiling in the compute node 104 to allow controller node to update the operation of the NIC 130. Also, controller node 114 can generate a parse graph and register parameters for transmission to compute node 104 to allow the controller node 114 to update operation of the NIC 130.


Such configurations can advantageously provide improved control over NIC functionality such as classification and queue assignment, which may affect server performance in a datacenter. Additionally, the configurations provide a mechanism to enable NIC actions, such as rate limitation, access control lists (ACLs) and the like, according to user-defined protocols. In various embodiments described herein, NICs are configured to register or “learn” new protocols “on the fly,” meaning that no reset is required (i.e., no reset of the NIC, no reset the networking device, and/or not reset of particular sub-components of the NIC or networking device). Once the new protocols are registered on the NIC 130, the controller node 114 can push actions and/or filters to the NIC 130 to perform new functions for the defined protocols (e.g., classification, queue assignment). This in turn provides better granularity and flexibility in terms of how packets are classified, filtered, and assigned to queues by providing users an interface for defining protocols, packet types and derived classification actions.


While embodiments described below are illustrated in the context of a SDN, it should be appreciated by those skilled in the art that the techniques described herein may be applicable to other networking protocols and/or architectures. In certain embodiments, the system 100 may be based generally on a modified OpenFlow communications protocol. The OpenFlow protocol may provide access to a forwarding plane of a network switch (e.g., 170) or router over network 112. This enables remote controllers, such as controller node 114, to determine a path of network packets through a network of switches (e.g., 170). The separation of the control from the forwarding allows for more flexible and/or sophisticated traffic management.


Furthermore, OpenFlow allows switches from different suppliers, often comprising their own proprietary interfaces and scripting languages, to be managed remotely using a single, open protocol. OpenFlow further allows remote administration of a switch's packet forwarding tables, by adding, modifying and removing packet matching rules and actions. This way, routing decisions can be made periodically or ad hoc by the controller and translated into rules and actions with a configurable lifespan, which are then deployed to a switch's flow table, leaving the actual forwarding of matched packets to the switch at wire speed for the duration of those rules. Packets that are unmatched by a switch can be forwarded to controller node 114. Controller node 114 can then decide to modify existing flow table rules on one or more switches or to deploy new rules, to prevent a structural flow of traffic between switch and controller. It could even decide to forward the traffic itself, provided that it has informed the switch to forward entire packets instead of just their header.


Referring now to FIG. 2, in use, each compute node 104 may establish an environment 200 during operation. In the illustrative embodiment, the environment 200 comprises the NIC 130, which includes persistent instructions (e.g., firmware) 206 for providing control program(s) for NIC 130. The persistent instructions 206 may be stored in, for example, the memory 132 of the NIC 130. The NIC 130 also includes a compiler module 234 and an update control module 236 as shown, wherein the update control module 236 may be operatively coupled to a classifier rules database 238 and a protocol descriptor database 240. The update control module 236 may further comprise persistent instructions (e.g., firmware) and driver interfaces to provide data transfer to/from the NIC 130 outside the confines of low-level code and code updates. In one embodiment, updates received via the NIC 130 are programmed in the appropriate specification language for the particular NIC model and loaded into the driver to avoid the need for having the user interact on an assembly language level. In other exemplary embodiments, the compiler module 234 may be configured as part of the web service module 226 as shown in FIG. 2. Under this configuration, the web service 226 receives and compiles a parse graph (using compiler 234 shown in dotted lines) and pushes it to the NIC 130.


The persistent instructions (e.g., firmware) 206 and related components may be embodied in non-volatile memory (e.g., ROM, EPROM, flash memory). The persistent instructions (e.g., firmware) 206 is operatively coupled to the administrative queue 216 (AQ) via update the control module 236, where the administrative queue 216 may be used for communications to store system-generated acknowledgement messages that message queuing or connector applications create. The administrative queue 216 may be embodied as any type of data storage capable of storing messages such as a portion of a memory. The administrative queue 216 may be coupled to web service module 226, which connects to the management port 228 configured to receive and communicate data such as representational state transfer application programming interfaces (REST API) 230 from controller node 114.


Update control module 236 is also operatively coupled to a programmable protocol parser module 202, which may also receive data from data path network 232. In this example, the protocol parser module 202 handles processing and communication of protocol descriptors and is configured with a protocol field extractor and packet type identifier. The classifier module 204 contains a set of rules that may be ordered by priority (e.g., n-tuple classification) to find matching rules for each incoming packet. Classifier module 204 is configured to process flow tables for packets using a classifier switch 203 and filters 205. In the illustrative embodiment, various rules and flow tables may be enabled in the classifier module 204 according to characteristics such as source IP, destination IP, source port, destination port, protocol (e.g., TCP, UDP), and assigned an action (e.g., permit, deny). Internal flow of data in the compute node 104 from the classifier module 204 may be directed through a plurality of queues (208-214) associated with respective virtual machines (VM1-VM4; 218-224) or other suitable modules. The update control module 236 may provide switch rules for classification to classifier module 204 and protocol descriptors to protocol parser module 202 for processing.


The various modules of the environment 200 may be embodied as hardware, firmware, software, or a combination thereof. For example the various modules, logic, and other components of the environment 200 may form a portion of, or otherwise be established by, the NIC 130, the processor 120, or other hardware components of the compute node 104. As such, in some embodiments, any one or more of the modules of the environment 200 may be embodied as a circuit or collection of electrical devices (e.g., an update control circuit and/or a classifier circuit).


Referring now to FIG. 3, in use, the controller node 114 may establish an environment 300 during operation. The environment 300 includes an SDN controller module 302, operatively coupled to a management port 328 for communicating data, such as REST APIs, to compute nodes (104-110). SDN controller module 302 may be configured as a strategic control point in an SDN network, relaying information to the switches/routers “below” (via southbound APIs) and the applications and logic “above” (via northbound APIs). In certain embodiments, SDN controller module 302 may be tasked with federating between SDN controller domains, using common application interfaces, such as OpenFlow and open virtual switch database (OVSDB). The SDN controller module 302 platform may also comprise a collection of “plug-in” modules that can perform different network tasks, including inventorying what devices are within the network and the capabilities of each, gathering network statistics, etc. Extensions can be inserted that enhance the functionality and support more advanced capabilities, such as running algorithms to perform analytics and orchestrating new rules throughout the network.


In certain embodiments, the SDN controller module 302 may communicate over the network using OpenFlow or OVSDB. In other embodiments, SDN controller module 302 may communicate over the network using other protocols such as YANG or NetConf, among others. The various modules of the environment 300 may be embodied as hardware, firmware, software, or a combination thereof. For example the various modules, logic, and other components of the environment 300 may form a portion of, or otherwise be established by the processor 150 or other hardware components of the controller node 114. As such, in some embodiments, any one or more of the modules of the environment 300 may be embodied as a circuit or collection of electrical devices (e.g., a SDN controller circuit, an update circuit).


The SDN controller module 302 further includes an update module 304 that is communicatively coupled to classifier rules database 306 and protocol descriptors database 308. As will be discussed in greater detail below, the update module may initiate configuration and/or updating of persistent instructions (e.g., firmware) for one or more NICs. In an illustrative embodiment, the controller node 114 begins NIC configuration by generating an AQ message describing a new protocol from database 308 using a protocol-specific language that can be compiled and loaded into the NIC 130 to update the programmable protocol parser module 202. AQ messages transmitted by the SDN controller module 302 are received in the compute node 104 via the management port 228 and transmitted to update the control module 236 for persistent instructions (e.g., firmware) 206. The update control module 236 processes the AQ message and, using protocol descriptors contained in the message, updates programmable protocol parser module 202 for persistent instructions (e.g., firmware) 206.


If the update is successful, the persistent instructions (e.g., firmware) 206 transmits an acknowledgement signal through the update control module 236 via the management port 228 to the controller node 114. If the update is not successful, persistent instructions (e.g., firmware) 206 transmits an error signal via management port 228 to controller node 114. After a successful update, the persistent instructions (e.g., firmware) 206 includes an update identifier, such as a protocol number, with the acknowledgement indicating the new protocol present on the NIC 130. Once the update identifier is received in the controller node 114, further AQ messages may be transmitted to the compute node 104 for configuring operation for NIC 130. In one embodiment, after receiving an update identifier, the SDN controller module 302 of the controller node 114 may transmit classification rules and queue assignments for an embedded switch of the NIC 130. In some embodiments, controller node 114 may be configured with a compiler module 310 that is coupled to update module 304. After generating a parse graph, controller node 114 may compile it locally using compiler module 310, provided there is information on the controller node 114 specifying the type of NIC (130) the parse graph will be pushed to. In such a configuration, compiler module 310 may be used instead of, or in addition to, compiler module 234 in FIG. 2.


The protocol specification language from the SDN controller module 302 may be configured to unambiguously describe a parse graph to be compiled by the compiler module 234 into device-specific microcode or low-level register instructions, and loaded into the NIC 130. This acts as a common high-level description between higher layer software and a hardware parser to describe the protocols in use. In addition to programming the flexible parser, the compiler module 234 may also generate a mapping between the user-provided protocols/fields and the internal labels used by the hardware. This advantageously simplifies software by allowing applications to use protocol/field names that are common to the environment instead of device specific protocol labels. This is particularly advantageous in large heterogeneous networks that may have many types of flexible parsers available. In certain embodiments, the compiler module 234 may be configured support multiple different hardware devices.


In an illustrative embodiment, during initial operation, protocol descriptors for one or more known protocols may be preloaded into a non-volatile memory (NVM), e.g., flash memory, of the NIC 130 (e.g., memory 132) for automatic configuration. In the presence of a preloaded NVM, the SDN controller module 302 would only need to program additional new protocols as they are introduced into the system. For example, customer-specific proprietary protocols, new standards and the like may be provided to the NIC 130 from the SDN controller module 302 after startup by extending the parse graph using the protocol specification language.


In a simplified example, the SDN controller module 302 may be configured under a HTTP protocol using the protocol specification language. An illustrative example 400 of such a protocol specification and actions message is provided in FIG. 4 showing one configuration for Ethernet 402, VLAN 404, IP 406 and TCP 408. The information illustrated 400 is compiled via the compiler module 234 into persistent instructions (e.g., firmware) 206 required by the compute node 104 (200). The compiler module 234 also generates a map table between a user-provided parse graph and labels on compute node 104 (200). In the example of FIG. 4, the illustrated configuration allows the creation of a match/action tuple such that TCP Destination Port field 80 is matched with an action to push traffic to queue 3. A compiler module 234 map function then translates the high level instructions into low level AQ commands, as illustrated in pertinent part below:

















<parse_graph : text >



packet ether {dst:48 src:48 type:16};



packet vlan {VID:12;TPID:16} extends ether with type:0x8100;



packet ip {ver:4 ihl:4 tos:8 len:16 id:16 flags:3









frag_offset:13 ttl:8 proto:8 csum:16



sip:32 dip:32} extends ether with type:0x800









packet tcp {src_port:16 dport:16









 seq:32 ack:32



 offset:4 reserved:3



 ecn:3 control:6



 window:16 csum:16 urgent:16} extends ip with proto:6









packet http {<http-specification>} extends tcp with dport: 80



</parse_graph>



<actions : text>



type qselect = Queue of int



</actions>



<rule : text>



match http action qselect 3



</rule>










Referring now to FIGS. 5 and 6, in use, each NIC 130 may execute a method 500 for identifying and updating protocol sequences and protocol parsing. The illustrative method 500 begins with block 502 in which a packet is received by the NIC 130 and buffered. The programmable protocol parser module 202 performs a protocol lookup in block 504 to identify the structure of the packet (e.g., a sequence of protocol headers) and determine if the received protocol descriptor exists in a database in block 506. In one embodiment, a hardware-based programmable parser could provide packet parsing according to a programmable protocol descriptor database and parse graph database, such as paths on the graph that represent valid protocol sequences. If the protocol descriptor is found in the database (“YES”), the parser collects protocol properties in block 508 by producing metadata (e.g., offset, instance number, etc.) describing the protocols found in the packet, including new protocols derived from software via the protocol descriptors, and updates the protocol sequence(s) in block 510. The packet and the metadata may be transmitted to the NIC switch that classifies the packet and further assigns it to a required target queue. It should be appreciated by those skilled in the art that additional classification/filtering and packet modification may be performed as necessary. After updating the protocol sequence in block 510, the process proceeds with parsing (“A”) described below in the illustrative embodiment of FIG. 6.


Referring back to block 506, if the protocol descriptor is not found in the database, (“NO”) the protocol descriptor is further processed in block 512 to determine if a protocol (graph) exception action is defined or generated. A protocol exception may be defined such that an action is performed in block 514 when a type of protocol or sequence is detected. Some possible actions comprise discarding or capturing the frame. If in block 512 it is determined that a protocol exception is not defined, the received frame may be transmitted to controller/supervisor (i.e., controller node 114) in block 516 for further processing and to determine validity of protocol descriptor in block 518. In some instances, invalid protocols and/or sequences may be part of a DoS attack that would need to be identified. As the validity of protocol descriptors is determined, valid descriptors may be used to update protocol descriptor in block 520, while invalid descriptors may be used to update a protocol exception database in block 522.


Turning now to FIG. 6, after the protocol sequence is updated, a protocol parser may perform a parse graph lookup of protocol sequences in block 524 to determine if a protocol sequence is in a graph database in block 526. If the protocol sequence is recognized in block 526 (“YES”), the parser determines whether a graph leaf or header end has been reached in block 528. If the graph leaf or header end has been reached (“YES”), the parsing is completed in block 530. If the graph leaf or header end has not been reached (“NO”), the parsing returns back to block 504 via designator “B” to continue with a next protocol lookup. As can be appreciated by those skilled in the art, the process may repeat until a graph leaf or header end of the parser has been reached.


If, in block 526, a protocol sequence is not identified from the graph database (“NO”), the parser determines if a graph exception action is defined in block 532. If an exception action is defined (“YES”), the parser performs the action in block 534. Possible actions comprise discarding or capturing the frame containing the sequence. If in block 532 it is determined that a graph exception is not defined (“NO”), the received frame may be transmitted to controller/supervisor (i.e., controller node 114) in block 536 for further processing and to determine validity of protocol sequence in block 538. As the validity of protocol sequence is determined, valid sequences may be used to update protocol sequences in block 540, while invalid sequences may be used to update a graph exception database in block 542.


It should be appreciated by those skilled in the art that principles provided in the present disclosure may be further used in other applications, such as switches with NIC-like host-interfaces (e.g., Intel® Red Rock Canyon (RRC) architecture). As one example, a hybrid switch-NIC packet processing application can program a parser in the switch using protocol descriptors to support new protocols. Classification results from the switch tables may be conveyed to the host-interface section of the switch as metadata in an F-tag container where queue assignments in the host may be derived from the metadata carried in the F-Tag.


Utilizing the techniques described herein, one skilled in the art will recognize that the configurations provide administrators with improved flexibility in directing traffic to desired queues. From these queues traffic may be further directed to CPU Cores, NUMA Nodes, VMs and/or applications. Furthermore, in an SDN environment, a control plane having knowledge of the resources of a server (NUMA nodes, number of CPU cores, PCIe topology etc.), is able to direct traffic much more accurately and efficiently and maximize the use of resources. Moreover, actions such as ACL drops (i.e., pushing flows to an inactive queue that is not attended by higher level software) can be implemented alleviating concerns about denial of service (DoS) attacks. Furthermore, tag pushing and popping will be possible in hardware switches can add (“push”), remove (“pop”) or modify single or multiple tags.


EXAMPLES

Illustrative examples of the technologies disclosed herein are provided below. An embodiment of the technologies may include any one or more, and any combination of, the examples described below.


Example 1 includes a networking device for controlling operation of a compute node coupled to a computer network, the networking device comprising a memory having stored therein persistent instructions such as firmware for providing control functions to the networking device, wherein the control functions are defined at least in part by protocol data; an update control module for receiving update data from a remote node in the computer network, the update data comprising new protocol data for the persistent instructions; a protocol parser module for parsing the update data and generating metadata relating to the update data without a reset of the networking device; a classifier module for receiving rules for the control functions, wherein the rules are based at least in part on the update data and metadata; and a compiler module for compiling the parsed update data to the persistent instructions for providing new control functions to the networking device based at least in part on the received rules.


Example 2 includes the subject matter of example 1, and wherein the new protocol data comprises a protocol specification language data relating to the persistent instructions.


Example 3 includes the subject matter of any of Examples 1 and 2, and wherein the new protocol data comprises a parse graph.


Example 4 includes subject matter of any of the Examples 1-3, and wherein the protocol data further comprises one of networking device-specific microcode and register parameters.


Example 5 includes the subject matter of any of the Examples 1-4, and wherein the new protocol data comprises protocol descriptors, and wherein the protocol parser module parses the update data using the protocol descriptors.


Example 6 includes the subject matter of any of the Examples 1-5, and wherein the protocol parser module parses the update data using protocol field extraction and packet type identification.


Example 7 includes the subject matter of any of the Examples 1-6, and wherein the remote node comprises a software defined networking (SDN) node.


Example 8 includes the subject matter of any of the Examples 1-7, and wherein the classifier module comprises at least one classifier switch and filter.


Example 9 includes the subject matter of any of the Examples 1-8, and wherein the compiler module is to generate a map for the parsed update data to at least a portion of the protocol data.


Example 10 includes the subject matter of any of the Examples 1-9, and wherein the networking device comprises a network interface controller.


Example 11 includes a network node for controlling operation of a networking device on a compute node coupled to a computer network, the network node comprising a memory having stored therein update data for updating persistent instructions of the networking device; and a controller module coupled to the memory, wherein the controller module is to retrieve update data from the memory and transmit the update data to the networking device, and wherein the update data comprises (i) new protocol data for controlling functions of the persistent instructions, the new protocol data comprising parseable and compilable protocol data and (ii) rules data relating to the new protocol data, the rules data comprising instructions for providing new control functions to the networking device based at least in part on the rules data.


Example 12 includes the subject matter of Example 11, and wherein the new protocol data comprises a protocol specification language data relating to the persistent instructions.


Example 13 includes the subject matter of any of Examples 10 and 11, and wherein the new protocol data comprises a parse graph.


Example 14 includes the subject matter of any of the Examples 11-13, and wherein the protocol data further comprises one of networking device-specific microcode and register parameters.


Example 15 includes the subject matter of any of the Examples 11-14, and wherein the new protocol data comprises parseable protocol descriptors.


Example 16 includes the subject matter of any of the Examples 11-15, and wherein the controller module is configured to receive an acknowledgement signal in response to transmitting the new protocol data, and wherein the controller module is configured to transmit the rules data after receiving the acknowledgement signal.


Example 17 includes the subject matter of any of the Examples 11-16, and wherein the network node comprises a software defined networking (SDN) node.


Example 18 includes the subject matter of any of the Examples 11-17, and wherein the memory comprises a protocol database for providing protocol data and a classifier database for providing rules data.


Example 19 includes the subject matter of any of the Examples 11-18, and wherein the networking device comprises a network interface controller.


Example 20 includes the subject matter of any of any of the Examples 11-19 and further a compiler module coupled to the controller module to compile the update data.


Example 21 includes a method for controlling operation of a compute node coupled to a computer network via a networking device, the method comprising providing, by the networking device, control functions to the networking device via persistent instructions, such as firmware, the control functions being defined at least in part by protocol data; receiving, by the networking device, update data from a remote node in the computer network, the update data comprising new protocol data for the persistent instructions; parsing, by the networking device, the update data to generate metadata relating to the update data without a reset of the networking device; receiving, by the networking device, rules for the control functions, wherein the rules are based at least in part on the update data and metadata; and compiling, by the networking device, the parsed update data to the persistent instructions for providing new control functions to the networking device based at least in part on the received rules.


Example 22 includes the subject matter of example 210, and wherein receiving update data comprises receiving update data comprising new protocol data for the persistent instructions, the new protocol data comprising a protocol specification language data relating to the persistent instructions.


Example 23 includes the subject matter of any of Examples 21 and 22, and wherein receiving update data comprises receiving update data comprising new protocol data for the persistent instructions, the new protocol data comprising a parse graph.


Example 24 includes the subject matter of any of Examples 21-23, and wherein the new protocol data further comprises one of networking device-specific microcode and register parameters.


Example 25 includes the subject matter of any of Examples 21-24, and wherein the new protocol data comprises protocol descriptors, and wherein the update data is parsed by the protocol parser module using the protocol descriptors.


Example 26 includes the subject matter of any of Examples 21-25, and wherein the update data is parsed by the protocol parser module using protocol field extraction and packet type identification.


Example 27 includes the subject matter of any of Examples 21-26, and wherein the remote node comprises a software defined networking (SDN) node.


Example 28 includes the subject matter of any of Examples 21-27, and wherein the received rules comprises data relating to at least one classifier switch and filter in the classifier module.


Example 29 includes the subject matter of any of Examples 21-28, and further comprising generating a map via the compiler for the parsed update data to at least a portion of the protocol data.


Example 30 includes the subject matter of any of Examples 21-29, and wherein the networking device comprises a network interface controller.


Example 31 includes a method for controlling operation of a networking device on a compute node coupled to a computer network via a network node, comprising storing, in a memory of the network node, update data for updating persistent instructions such as firmware of the networking device; retrieving, by a controller module of the network node, update data from the memory; and transmitting, by the network node, the update data to the networking device, wherein the update data comprises (i) new protocol data for controlling functions of the persistent instructions, the new protocol data comprising parseable and compilable protocol data and (ii) rules data relating to the new protocol data, the rules data comprising instructions for providing new control functions to the networking device based at least in part on the rules data.


Example 32 includes the subject matter of example 31, and wherein transmitting the update data comprises transmitting the new protocol data, the new protocol data comprising a protocol specification language data relating to the persistent instructions.


Example 33 includes the subject matter of any of Examples 31 and 32, and wherein transmitting the update data comprises transmitting the new protocol data, the new protocol data comprising a parse graph.


Example 34 includes the subject matter of any of Examples 31-33, and wherein the new protocol data further comprises one of networking device-specific microcode and register parameters.


Example 35 includes the subject matter of any of Examples 31-34, and wherein the new protocol data comprises parseable protocol descriptors.


Example 36 includes the subject matter of any of Examples 31-35, and further comprising receiving, by the network node, an acknowledgement signal in response to transmitting the new protocol data, wherein transmitting the rules data occurs after receiving the acknowledgement signal.


Example 37 includes the subject matter of any of Examples 31-36, and wherein the network node comprises a software defined networking (SDN) node.


Example 38 includes the subject matter of any of Examples 31-37, and wherein storing update data for updating persistent instructions of the networking device comprises storing a protocol database of the memory for providing protocol data and storing a classifier database of the memory for providing rules data.


Example 39 includes the subject matter of any of Examples 31-38, and wherein the networking device comprises a network interface controller.


Example 40 includes the subject matter of any of Examples 31-39 and further including compiling the update data by a compiler module coupled to the controller module to compile the update data


Example 41 includes one or more machine-readable media comprising a plurality of instructions stored thereon that, in response to execution by a computing device, causes the computing device to perform the method of any of Examples 21-40.


Example 40 includes a computing device for updating persistent instructions of a networking device of the computing device, the computing device comprising means for performing the method of any of claims 20-38


Example 42 includes a networking device for controlling operation of a compute node coupled to a computer network. The networking device includes means for providing control functions to the networking device via persistent instructions such as firmware, the control functions being defined at least in part by protocol data; means for receiving update data from a remote node in the computer network, the update data comprising new protocol data for the persistent instructions; means for parsing the update data to generate metadata relating to the update data without a reset of the networking device; means for receiving rules for the control functions, wherein the rules are based at least in part on the update data and metadata; and means for compiling the parsed update data to the persistent instructions for providing new control functions to the networking device based at least in part on the received rules.


Example 43 includes the subject matter of Example 42, and wherein the means for receiving update data comprises means for receiving update data comprising new protocol data for the persistent instructions, the new protocol data comprising a protocol specification language data relating to the persistent instructions.


Example 44 includes the subject matter of any of Examples 42 or 43, and wherein the means for receiving update data comprises means for receiving update data comprising new protocol data for the persistent instructions, the new protocol data comprising a parse graph.


Example 45 includes the subject matter of any of Examples 42-44, and wherein the new protocol data further comprises one of networking device-specific microcode and register parameters.


Example 46 includes the subject matter of any of Examples 42-45, and wherein the new protocol data comprises protocol descriptors, and wherein the update data is parsed by the protocol parser module using the protocol descriptors.


Example 47 includes the subject matter of any of Examples 42-46, and wherein the update data is parsed by the protocol parser module using protocol field extraction and packet type identification.


Example 48 includes the subject matter of any of Examples 42-47, and wherein the remote node comprises a software defined networking (SDN) node.


Example 49 includes the subject matter of any of Examples 42-48, and wherein the received rules comprises data relating to at least one classifier switch and filter in the classifier module.


Example 50 includes the subject matter of any of Examples 42-49, and further comprising means for generating a map via the compiler for the parsed update data to at least a portion of the protocol data.


Example 51 includes the subject matter of any of Examples 42-50, and wherein the networking device comprises a network interface controller.


Example 52 includes a network node for controlling operation of a networking device on a compute node coupled to a computer network. The network node includes means for storing, in a memory of the network node, update data for updating persistent instructions of the networking device; means for retrieving update data from the memory; and means for transmitting the update data to the networking device, wherein the update data comprises (i) new protocol data for controlling functions of the persistent instructions, the new protocol data comprising parseable and compilable protocol data and (ii) rules data relating to the new protocol data, the rules data comprising instructions for providing new control functions to the networking device based at least in part on the rules data.


Example 53 includes the subject matter of Example 52, and wherein the means for transmitting the update data comprises means for transmitting the new protocol data, the new protocol data comprising a protocol specification language data relating to the persistent instructions.


Example 54 includes the subject matter of any of Examples 52 or 53, and wherein the means for transmitting the update data comprises means for transmitting the new protocol data, the new protocol data comprising a parse graph.


Example 55 includes the subject matter of any of Examples 52-54, and wherein the new protocol data further comprises one of networking device-specific microcode and register parameters.


Example 56 includes the subject matter of any of Examples 52-55, and wherein the new protocol data comprises parseable protocol descriptors.


Example 57 includes the subject matter of any of Examples 52-56, and further includes means for receiving an acknowledgement signal in response to transmitting the new protocol data, wherein transmitting the rules data occurs after receiving the acknowledgement signal.


Example 58 includes the subject matter of any of Examples 52-57, and wherein the network node comprises a software defined networking (SDN) node.


Example 59 includes the subject matter of any of Examples 52-58, and wherein the means for storing update data for updating persistent instructions of the networking device comprises means for storing a protocol database of the memory for providing protocol data and storing a classifier database of the memory for providing rules data.


Example 60 includes the subject matter of any of Examples 52-59, and wherein the networking device comprises a network interface controller.


Example 61 includes the subject matter of any of Examples 52-60, and further includes means for compiling the update data by a compiler module coupled to the controller module to compile the update data.


Example 62 includes a networking device for controlling operation of a compute node coupled to a computer network, the networking device comprising means for storing persistent instructions for providing control functions to the networking device, wherein the control functions are defined at least in part by protocol data; means for receiving update data from a remote node in the computer network, the update data comprising new protocol data for the persistent instructions; means for parsing the update data and generating metadata relating to the update data without a reset of the networking device; means for receiving rules for the control functions, wherein the rules are based at least in part on the update data and metadata; and means for compiling the parsed update data to the persistent instructions for providing new control functions to the networking device based at least in part on the received rules.

Claims
  • 1. A network device comprising physical layer and data link layer communication circuitry and being for use in receiving network traffic, the network device comprising: packet parser circuitry for use in (1) identifying and parsing at least one header of at least one incoming packet and (2) for generating output data for use in association with at least one packet switching-related operation, the parser circuitry being programmable, at least in part, based upon compilation of user-provided instructions in a language, the user-provided instructions describing, at least in part, a parse graph, the parse graph being to define, at least in part, (1) header structure of the at least one incoming packet and (2) manner of parsing of the at least one header;wherein: the user-provided instructions also describe match/action tuples that associate, at least in part, possible incoming packet characteristics with corresponding actions to be taken by the network device;the corresponding actions comprise the at least one packet switching-related operation; andwhen the network device is in operation, the network device is to determine a particular action to take with respect to the at least one incoming packet based, at least in part, upon the output data and the match/action tuples.
  • 2. The network device of claim 1, wherein: the possible incoming packet characteristics comprise one or more of the following: source port;protocol type;destination port;source address; anddestination address.
  • 3. The network device of claim 2, wherein: the network device is useable in association with software-defined networking; andthe network device comprises embedded switch circuitry.
  • 4. The network device of claim 3, wherein: when the network device is in the operation, the network device is to receive the compilation from a computing device.
  • 5. The network device of claim 4, wherein: the compilation is to be generated, at least in part, by compiler software that is to be executed by the computing device, based upon the user-provided instructions; andthe compiler software is usable in programming multiple different hardware devices.
  • 6. The network device of claim 1, wherein: the possible incoming packet characteristics vary, at least in part, among the match/action tuples.
  • 7. A method implemented, at least in part, by a network device, the network device comprising physical layer and data link layer communication circuitry and being for use in receiving network traffic, the method comprising: identifying and parsing, by packet parser circuitry of the network device, at least one header of at least one incoming packet;generating, by the packet parser circuitry, output data for use in association with at least one packet switching-related operation, the parser circuitry being programmable, at least in part, based upon compilation of user-provided instructions in a language, the user-provided instructions describing, at least in part, a parse graph, the parse graph being to define, at least in part, (1) header structure of the at least one incoming packet and (2) manner of parsing of the at least one header;wherein: the user-provided instructions also describe match/action tuples that associate, at least in part, possible incoming packet characteristics with corresponding actions to be taken by the network device;the corresponding actions comprise the at least one packet switching-related operation; andwhen the network device is in operation, the network device is to determine a particular action to take with respect to the at least one incoming packet based, at least in part, upon the output data and the match/action tuples.
  • 8. The method of claim 7, wherein: the possible incoming packet characteristics comprise one or more of the following: source port;protocol type;destination port;source address; anddestination address.
  • 9. The method of claim 8, wherein: the network device is useable in association with software-defined networking; andthe network device comprises embedded switch circuitry.
  • 10. The method of claim 9, wherein: when the network device is in the operation, the network device is to receive the compilation from a computing device.
  • 11. The method of claim 10, wherein: the compilation is to be generated, at least in part, by compiler software that is to be executed by the computing device, based upon the user-provided instructions; andthe compiler software is usable in programming multiple different hardware devices.
  • 12. The method of claim 7, wherein: the possible incoming packet characteristics vary, at least in part, among the match/action tuples.
  • 13. One or more non-transitory machine-readable storage media storing instructions for being executed by a network device, the network device comprising physical layer and data link layer communication circuitry and being for use in receiving network traffic, the instructions when executed by the network device resulting in the network device being configured for performance of operations comprising: identifying and parsing, by packet parser circuitry of the network device, at least one header of at least one incoming packet;generating, by the packet parser circuitry, output data for use in association with at least one packet switching-related operation, the parser circuitry being programmable, at least in part, based upon compilation of user-provided instructions in a language, the user-provided instructions describing, at least in part, a parse graph, the parse graph being to define, at least in part, (1) header structure of the at least one incoming packet and (2) manner of parsing of the at least one header;wherein: the user-provided instructions also describe match/action tuples that associate, at least in part, possible incoming packet characteristics with corresponding actions to be taken by the network device;the corresponding actions comprise the at least one packet switching-related operation; andwhen the network device is in operation, the network device is to determine a particular action to take with respect to the at least one incoming packet based, at least in part, upon the output data and the match/action tuples.
  • 14. The one or more non-transitory machine-readable storage media of claim 13, wherein: the possible incoming packet characteristics comprise one or more of the following: source port;protocol type;destination port;source address; anddestination address.
  • 15. The one or more non-transitory machine-readable storage media of claim 14, wherein: the network device is useable in association with software-defined networking; andthe network device comprises embedded switch circuitry.
  • 16. The one or more non-transitory machine-readable storage media of claim 15, wherein: when the network device is in the operation, the network device is to receive the compilation from a computing device.
  • 17. The one or more non-transitory machine-readable storage media of claim 16, wherein: the compilation is to be generated, at least in part, by compiler software that is to be executed by the computing device, based upon the user-provided instructions; andthe compiler software is usable in programming multiple different hardware devices.
  • 18. The one or more non-transitory machine-readable storage media of claim 13, wherein: the possible incoming packet characteristics vary, at least in part, among the match/action tuples.
  • 19. One or more non-transitory machine-readable storage media storing instructions for being executed by at least one computing device, the at least one computing device to be used in association with a network device, the instructions, when executed by the at least one computing device resulting in the at least one computing device being configured for performance of operations comprising: generating, at least in part, by the computing device, a compilation of user-provided instructions in a language, the compilation to be provided to the network device and being usable in programming of packet parser circuitry of the network device, the user-provided instructions describing, at least in part, a parse graph, the packet parser circuitry being for identifying and parsing, when the network device is in operation, at least one header of at least one incoming packet to the network device, the packet parser circuitry also being for generating output data for use in association with at least one packet switching-related operation, the parse graph being to define, at least in part, (1) header structure of the at least one incoming packet and (2) manner of parsing of the at least one header;wherein: the user-provided instructions also describe match/action tuples that associate, at least in part, possible incoming packet characteristics with corresponding actions to be taken by the network device;the corresponding actions comprise the at least one packet switching-related operation; andwhen the network device is in the operation, the network device is to determine a particular action to take with respect to the at least one incoming packet based, at least in part, upon the output data and the match/action tuples.
  • 20. The one or more non-transitory machine-readable storage media of claim 19, wherein: the possible incoming packet characteristics comprise one or more of the following: source port;protocol type;destination port;source address; anddestination address.
  • 21. The one or more non-transitory machine-readable storage media of claim 20, wherein: the network device is useable in association with software-defined networking; andthe network device comprises embedded switch circuitry.
  • 22. The one or more non-transitory machine-readable storage media of claim 21, wherein: when the network device is in the operation, the network device is to receive the compilation from the at least one computing device.
  • 23. The one or more non-transitory machine-readable storage media of claim 22, wherein: the instructions that are executable by the at least one computing device comprise compiler instructions that are usable in programming multiple different hardware devices.
  • 24. The one or more non-transitory machine-readable storage media of claim 19, wherein: the possible incoming packet characteristics vary, at least in part, among the match/action tuples.
  • 25. A method implemented using at least one computing device, the method comprising: generating, at least in part, by the at least one computing device, a compilation of user-provided instructions in a language, the compilation being usable in programming of packet parser circuitry of a network device, the user-provided instructions describing, at least in part, a parse graph, the packet parser circuitry being for identifying and parsing, when the network device is in operation, at least one header of at least one incoming packet to the network device, the packet parser circuitry also being for generating output data for use in association with at least one packet switching-related operation, the parse graph being to define, at least in part, (1) header structure of the at least one incoming packet and (2) manner of parsing of the at least one header;wherein: the user-provided instructions also describe match/action tuples that associate, at least in part, possible incoming packet characteristics with corresponding actions to be taken by the network device;the corresponding actions comprise the at least one packet switching-related operation; andwhen the network device is in the operation, the network device is to determine a particular action to take with respect to the at least one incoming packet based, at least in part, upon the output data and the match/action tuples.
  • 26. The method of claim 25, wherein: the possible incoming packet characteristics comprise one or more of the following: source port;protocol type;destination port;source address; anddestination address.
  • 27. The method of claim 26, wherein: the network device is useable in association with software-defined networking; andthe network device comprises embedded switch circuitry.
  • 28. The method of claim 27, wherein: when the network device is in the operation, the network device is to receive the compilation from the at least one computing device.
  • 29. The method of claim 28, wherein: the generating comprises executing, by the at least one computing device, compiler instructions that are usable in programming multiple different hardware devices.
  • 30. The method of claim 25, wherein: the possible incoming packet characteristics vary, at least in part, among the match/action tuples.
CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. application Ser. No. 16/026,318, entitled “PROGRAMMABLE PROTOCOL PARSER FOR NIC CLASSIFICATION AND QUEUE ASSIGNMENTS,” which was filed on Jul. 3, 2018 and which is a application is a divisional application of U.S. application Ser. No. 14/583,664, entitled “PROGRAMMABLE PROTOCOL PARSER FOR NIC CLASSIFICATION AND QUEUE ASSIGNMENTS,” which was filed on Dec. 27, 2014.

US Referenced Citations (289)
Number Name Date Kind
5243596 Port et al. Sep 1993 A
5642483 Topper Jun 1997 A
5784003 Dahlgren Jul 1998 A
6157955 Narad et al. Dec 2000 A
6442172 Wallner et al. Aug 2002 B1
6453360 Muller Sep 2002 B1
6735679 Herbst et al. May 2004 B1
6836483 Lee Dec 2004 B1
6948099 Tallam Sep 2005 B1
6976149 Brandt et al. Dec 2005 B1
6980552 Belz et al. Dec 2005 B1
7042888 Berggreen May 2006 B2
7046685 Matsuoka et al. May 2006 B1
7177276 Epps et al. Feb 2007 B1
7203740 Putzolu et al. Apr 2007 B1
7389462 Wang et al. Jun 2008 B1
7492714 Liao et al. Feb 2009 B1
7499941 Michaeli et al. Mar 2009 B2
7539777 Aitken May 2009 B1
7643486 Belz et al. Jan 2010 B2
7826470 Aloni Nov 2010 B1
7872774 Okamoto Jan 2011 B2
7873959 Zhu Jan 2011 B2
7889750 Parker Feb 2011 B1
7961734 Panwar et al. Jun 2011 B2
8077611 Bettink et al. Dec 2011 B2
8155135 Aloni Apr 2012 B2
8514855 Robertson et al. Aug 2013 B1
8527613 Malone Sep 2013 B2
8638793 Ben-Mayor et al. Jan 2014 B1
8693374 Murphy et al. Apr 2014 B1
8738860 Griffin et al. May 2014 B1
8788512 Asaad et al. Jul 2014 B2
8971338 Mishra et al. Mar 2015 B2
9049153 Casado et al. Jun 2015 B2
9049271 Hobbs et al. Jun 2015 B1
9055004 Edsall et al. Jun 2015 B2
9055114 Talaski et al. Jun 2015 B1
9112818 Arad et al. Aug 2015 B1
9124644 Patel et al. Sep 2015 B2
9213537 Bandakka Dec 2015 B2
9258224 Bosshart et al. Feb 2016 B2
9276846 Goyal et al. Mar 2016 B2
9294386 Narad Mar 2016 B2
9298446 Chang Mar 2016 B2
9450817 Bahadur et al. Sep 2016 B1
9467363 Gao Oct 2016 B2
9686209 Arad et al. Jun 2017 B1
9712439 Bosshart et al. Jul 2017 B2
9755932 Godbole et al. Sep 2017 B1
9825862 Bosshart Nov 2017 B2
9826071 Bosshart Nov 2017 B2
9838268 Mattson Dec 2017 B1
9891898 Tonsing Feb 2018 B1
9912610 Bosshart et al. Mar 2018 B2
9912774 Daly et al. Mar 2018 B2
9923816 Kim et al. Mar 2018 B2
10009276 Bosshart et al. Jun 2018 B2
10015048 Gasparakis Jul 2018 B2
10044646 Detwiler Aug 2018 B1
10091137 Tran et al. Oct 2018 B2
10135734 Singh et al. Nov 2018 B1
10225381 Bosshart Mar 2019 B1
10341242 Srinivasan et al. Jul 2019 B2
10361914 Gasparakis Jul 2019 B2
10412018 Feng et al. Sep 2019 B1
10419242 Tonsing et al. Sep 2019 B1
10419366 Kim et al. Sep 2019 B1
10432527 Bosshart Oct 2019 B1
10686735 Watson Jun 2020 B1
20010043611 Kadambi et al. Nov 2001 A1
20020001356 Shenoi Jan 2002 A1
20020136163 Kawakami et al. Sep 2002 A1
20020172210 Wolrich et al. Nov 2002 A1
20030009466 Ta et al. Jan 2003 A1
20030043825 Magnussen Mar 2003 A1
20030046414 Pettyjohn et al. Mar 2003 A1
20030046429 Sonksen Mar 2003 A1
20030063345 Fossum et al. Apr 2003 A1
20030107996 Black et al. Jun 2003 A1
20030118022 Kulkarni et al. Jun 2003 A1
20030147401 Kyronaho Aug 2003 A1
20030154358 Seong et al. Aug 2003 A1
20030167373 Winters Sep 2003 A1
20030219026 Sun et al. Nov 2003 A1
20040024894 Osman et al. Feb 2004 A1
20040031029 Lee Feb 2004 A1
20040042477 Bitar et al. Mar 2004 A1
20040105384 Gallezot et al. Jun 2004 A1
20040123220 Johnson et al. Jun 2004 A1
20040165588 Pandya Aug 2004 A1
20040213156 Smallwood et al. Oct 2004 A1
20050013251 Wang et al. Jan 2005 A1
20050041590 Olakangil et al. Feb 2005 A1
20050060428 Corl et al. Mar 2005 A1
20050078651 Lee et al. Apr 2005 A1
20050086353 Shirakawa et al. Apr 2005 A1
20050108518 Pandya May 2005 A1
20050120173 Minowa Jun 2005 A1
20050129059 Jiang et al. Jun 2005 A1
20050149823 Lee Jul 2005 A1
20050198531 Kaniz et al. Sep 2005 A1
20050243852 Bitar et al. Nov 2005 A1
20060002386 Yik et al. Jan 2006 A1
20060039374 Belz et al. Feb 2006 A1
20060050690 Epps et al. Mar 2006 A1
20060072480 Deval et al. Apr 2006 A1
20060092857 Ansari et al. May 2006 A1
20060114895 Anand et al. Jun 2006 A1
20060114914 Anand et al. Jun 2006 A1
20060117126 Leung et al. Jun 2006 A1
20060174242 Zhu Aug 2006 A1
20060277346 Doak et al. Dec 2006 A1
20070008985 Lakshmanamurthy et al. Jan 2007 A1
20070050426 Dubai Mar 2007 A1
20070055664 Michaeli et al. Mar 2007 A1
20070104102 Opsasnick May 2007 A1
20070104211 Opsasnick May 2007 A1
20070153796 Kesavan et al. Jul 2007 A1
20070195761 Tatar et al. Aug 2007 A1
20070195773 Tatar et al. Aug 2007 A1
20070208876 Davis Sep 2007 A1
20070230493 Dravida et al. Oct 2007 A1
20070236734 Okamoto Oct 2007 A1
20070280277 Lund et al. Dec 2007 A1
20080130670 Kim et al. Jun 2008 A1
20080144662 Marcondes Jun 2008 A1
20080175449 Fang et al. Jul 2008 A1
20080285571 Arulambalam et al. Nov 2008 A1
20090006605 Chen et al. Jan 2009 A1
20090096797 Du et al. Apr 2009 A1
20090106523 Steiss Apr 2009 A1
20090180475 Hashimoto Jul 2009 A1
20100085891 Kind et al. Apr 2010 A1
20100128735 Lipschutz May 2010 A1
20100135158 Adams Jun 2010 A1
20100140364 Nordberg et al. Jun 2010 A1
20100145475 Bartels et al. Jun 2010 A1
20100150164 Ma Jun 2010 A1
20100182920 Matsuoka Jul 2010 A1
20100191951 Malone Jul 2010 A1
20100228733 Harrison Sep 2010 A1
20100238812 Boutros et al. Sep 2010 A1
20100312941 Aloni Dec 2010 A1
20110149960 Gutierrez Jun 2011 A1
20120159235 Suganthi et al. Jun 2012 A1
20120170585 Mehra et al. Jul 2012 A1
20120173661 Mahaffey et al. Jul 2012 A1
20120177047 Roitshtein Jul 2012 A1
20120284438 Zievers Nov 2012 A1
20130003556 Boden et al. Jan 2013 A1
20130100951 Ishizuka Apr 2013 A1
20130108264 Deruijter et al. May 2013 A1
20130163426 Beliveau et al. Jun 2013 A1
20130163427 Beliveau et al. Jun 2013 A1
20130163475 Beliveau et al. Jun 2013 A1
20130166703 Hammer et al. Jun 2013 A1
20130227051 Khakpour et al. Aug 2013 A1
20130227519 Maleport Aug 2013 A1
20130315054 Shamis et al. Nov 2013 A1
20130318107 Asaad et al. Nov 2013 A1
20130346814 Zadigian et al. Dec 2013 A1
20140033489 Kawashima Feb 2014 A1
20140040527 Kanigicheria et al. Feb 2014 A1
20140043974 Kwan et al. Feb 2014 A1
20140082302 Rommelmann Mar 2014 A1
20140115571 Wang Apr 2014 A1
20140115666 Garcia Morchon Apr 2014 A1
20140140342 Narad May 2014 A1
20140181232 Manula et al. Jun 2014 A1
20140204943 Palmer Jul 2014 A1
20140233568 Dong Aug 2014 A1
20140241359 Bosshart et al. Aug 2014 A1
20140241361 Bosshart et al. Aug 2014 A1
20140241362 Bosshart et al. Aug 2014 A1
20140244966 Bosshart et al. Aug 2014 A1
20140269432 Goyal et al. Sep 2014 A1
20140301192 Lee et al. Oct 2014 A1
20140321473 Chen et al. Oct 2014 A1
20140328180 Kim et al. Nov 2014 A1
20140328344 Bosshart Nov 2014 A1
20140334489 Bosshart et al. Nov 2014 A1
20150003259 Gao Jan 2015 A1
20150009796 Koponen et al. Jan 2015 A1
20150010000 Zhang et al. Jan 2015 A1
20150020060 Bandakka Jan 2015 A1
20150023147 Lee et al. Jan 2015 A1
20150081833 Pettit et al. Mar 2015 A1
20150092539 Sivabalan et al. Apr 2015 A1
20150110114 Wohlgemuth et al. Apr 2015 A1
20150121355 Chang Apr 2015 A1
20150131666 Kang et al. May 2015 A1
20150131667 Ko et al. May 2015 A1
20150142932 Hallivuori et al. May 2015 A1
20150142991 Zaloom May 2015 A1
20150146527 Kishore et al. May 2015 A1
20150156288 Lu et al. Jun 2015 A1
20150172198 Levy et al. Jun 2015 A1
20150178395 Tiwari et al. Jun 2015 A1
20150180769 Wang et al. Jun 2015 A1
20150195206 Memon et al. Jul 2015 A1
20150222560 Kakadia et al. Aug 2015 A1
20150249572 Mack-Crane et al. Sep 2015 A1
20150319086 Tripathi et al. Nov 2015 A1
20150363522 Maurya Dec 2015 A1
20150381418 Fausak et al. Dec 2015 A1
20150381495 Cherian et al. Dec 2015 A1
20160014073 Reddy et al. Jan 2016 A1
20160019161 Patel et al. Jan 2016 A1
20160094460 Shelar et al. Mar 2016 A1
20160139892 Atreya et al. May 2016 A1
20160149784 Zhang et al. May 2016 A1
20160173371 Bays Jun 2016 A1
20160173383 Liu et al. Jun 2016 A1
20160188313 Dubai Jun 2016 A1
20160188320 Chang Jun 2016 A1
20160191306 Gasparakis Jun 2016 A1
20160191370 Wood Jun 2016 A1
20160191406 Xiao et al. Jun 2016 A1
20160197852 Hutchison et al. Jul 2016 A1
20160212012 Young et al. Jul 2016 A1
20160232019 Shah et al. Aug 2016 A1
20160234097 Chang Aug 2016 A1
20160234103 Kotalwar et al. Aug 2016 A1
20160241459 Zheng et al. Aug 2016 A1
20160301601 Anand et al. Oct 2016 A1
20160323243 Levasseur et al. Nov 2016 A1
20160330127 Kim et al. Nov 2016 A1
20160330128 Wang Nov 2016 A1
20160337329 Sood et al. Nov 2016 A1
20160342510 Pani Nov 2016 A1
20160344629 Gray Nov 2016 A1
20160357534 Krishnamoorthi et al. Dec 2016 A1
20160359685 Yadav et al. Dec 2016 A1
20170005951 Labonte et al. Jan 2017 A1
20170013452 Mentze et al. Jan 2017 A1
20170034082 Pfaff Feb 2017 A1
20170041209 Joshi et al. Feb 2017 A1
20170048144 Liu Feb 2017 A1
20170053012 Levy et al. Feb 2017 A1
20170063690 Bosshart Mar 2017 A1
20170064047 Bosshart Mar 2017 A1
20170075692 Naresh et al. Mar 2017 A1
20170085414 Castaldelli et al. Mar 2017 A1
20170085477 Li et al. Mar 2017 A1
20170085479 Wang et al. Mar 2017 A1
20170091258 Rajahalme Mar 2017 A1
20170093707 Kim et al. Mar 2017 A1
20170093986 Kim et al. Mar 2017 A1
20170093987 Kaushalram et al. Mar 2017 A1
20170118041 Bhattacharya et al. Apr 2017 A1
20170118042 Bhattacharya et al. Apr 2017 A1
20170126588 Anand et al. May 2017 A1
20170134282 Agarwal et al. May 2017 A1
20170134310 Koladi et al. May 2017 A1
20170142000 Cai et al. May 2017 A1
20170142011 Zhang et al. May 2017 A1
20170180273 Daly et al. Jun 2017 A1
20170195229 Ulas et al. Jul 2017 A1
20170208015 Volkening et al. Jul 2017 A1
20170220499 Gray Aug 2017 A1
20170222881 Holbrook et al. Aug 2017 A1
20170223575 Duda et al. Aug 2017 A1
20170251077 Eerpini et al. Aug 2017 A1
20170264571 Aibester et al. Sep 2017 A1
20170289034 Bosshart et al. Oct 2017 A1
20180006945 Flajslik et al. Jan 2018 A1
20180006950 Flajslik et al. Jan 2018 A1
20180054385 Dharmapurikar et al. Feb 2018 A1
20180124183 Kozat et al. May 2018 A1
20180191640 Calderon et al. Jul 2018 A1
20180234340 Kim et al. Aug 2018 A1
20180234355 Kim et al. Aug 2018 A1
20180262424 Roeland et al. Sep 2018 A1
20180287819 Mayer-Wolf et al. Oct 2018 A1
20180316549 Gasparakis Nov 2018 A1
20180375755 Joshi et al. Dec 2018 A1
20190394086 Gasparakis Dec 2019 A1
20200007473 Kim et al. Jan 2020 A1
20200021486 Gasparakis Jan 2020 A1
20200076737 Bosshart Mar 2020 A1
20200084093 Gasparakis Mar 2020 A1
20200099617 Bosshart Mar 2020 A1
20200099618 Bosshart Mar 2020 A1
20200099619 Bosshart Mar 2020 A1
20200228433 Lee Jul 2020 A1
20200244576 Wetterwald et al. Jul 2020 A1
20200280428 Kovacs et al. Sep 2020 A1
20200280518 Lee et al. Sep 2020 A1
Foreign Referenced Citations (4)
Number Date Country
1589551 Mar 2005 CN
3229424 Oct 2017 EP
2013101024 Jul 2013 WO
2017112165 Jun 2017 WO
Non-Patent Literature Citations (111)
Entry
Advisory Action for U.S. Appl. No. 15/835,238, dated Nov. 22, 2019, 3 pages.
Advisory Action for U.S. Appl. No. 15/888,054, dated Feb. 10, 2020.
Final Office Action for U.S. Appl. No. 15/784,191, dated Feb. 26, 2021.
Final Office Action for U.S. Appl. No. 15/784,191, dated May 7, 2020, 13 pages.
Final Office Action for U.S. Appl. No. 15/784,192, dated Jun. 1, 2020, 14 pages.
Final Office Action for U.S. Appl. No. 15/888,050, dated Dec. 12, 2019.
Final Office Action for U.S. Appl. No. 15/888,054, dated Sep. 18, 2019.
Non-Final Office Action for U.S. Appl. No. 16/569,554, dated Feb. 19, 2021.
Notice of Allowance for U.S. Appl. No. 14/583,664, dated Feb. 28, 2018.
Notice of Allowance for U.S. Appl. No. 14/836,850, dated Jun. 20, 2017.
Notice of Allowance for U.S. Appl. No. 14/836,855, dated Jun. 30, 2017.
Notice of Allowance for U.S. Appl. No. 15/678,549, dated Apr. 8, 2020.
Notice of Allowance for U.S. Appl. No. 15/678,549, dated Dec. 27, 2019.
Notice of Allowance for U.S. Appl. No. 15/678,556, dated Feb. 4, 2020.
Notice of Allowance for U.S. Appl. No. 15/729,555, dated May 2, 2019.
Notice of Allowance for U.S. Appl. No. 15/729,593, dated Nov. 15, 2018.
Notice of Allowance for U.S. Appl. No. 15/784,190, dated May 10, 2019, 20 pages.
Notice of Allowance for U.S. Appl. No. 15/784,191, dated Apr. 19, 2019, 7 pages.
Notice of Allowance for U.S. Appl. No. 15/784,191, dated Aug. 21, 2019, 8 pages.
Notice of Allowance for U.S. Appl. No. 15/784,191, dated May 5, 2021.
Notice of Allowance for U.S. Appl. No. 15/784,192, dated Mar. 17, 2021.
Notice of Allowance for U.S. Appl. No. 15/784,192, dated Jun. 30, 2021.
Notice of Allowance for U.S. Appl. No. 15/835,233, dated Jul. 3, 2019, 8 pages.
Notice of Allowance for U.S. Appl. No. 15/835,233, dated Oct. 29, 2019.
Notice of Allowance for U.S. Appl. No. 15/835,235, dated Apr. 24, 2020.
Notice of Allowance for U.S. Appl. No. 15/835,235, dated Aug. 20, 2019, 16 pages.
Notice of Allowance for U.S. Appl. No. 15/835,238, dated Sep. 30, 2020.
Notice of Allowance for U.S. Appl. No. 15/835,239, dated Nov. 13, 2019, 10 pages.
Notice of Allowance for U.S. Appl. No. 15/835,242, dated Jun. 24, 2020.
Notice of Allowance for U.S. Appl. No. 15/835,242, dated Jul. 1, 2019, 7 pages.
Notice of Allowance for U.S. Appl. No. 15/835,249, dated Jul. 25, 2019.
Notice of Allowance for U.S. Appl. No. 15/835,250, dated Jul. 25, 2019, 17 pages.
Notice of Allowance for U.S. Appl. No. 16/026,318, dated Mar. 12, 2019.
Notice of Allowance for U.S. Appl. No. 16/789,339, dated Jul. 29, 2021.
Office Action for U.S. Appl. No. 14/583,664, dated Feb. 27, 2017.
Office Action for U.S. Appl. No. 14/583,664, dated Jul. 28, 2016.
Office Action for U.S. Appl. No. 14/583,664, dated Oct. 18, 2017.
Office Action for U.S. Appl. No. 14/863,961, dated Jun. 16, 2017.
Office Action for U.S. Appl. No. 14/864,032, dated Feb. 14, 2017.
Office Action for U.S. Appl. No. 15/678,549, dated Feb. 26, 2019.
Office Action for U.S. Appl. No. 15/678,549, dated Jul. 30, 2019.
Office Action for U.S. Appl. No. 15/678,556, dated Jun. 19, 2019.
Office Action for U.S. Appl. No. 15/678,565, dated Jun. 13, 2019.
Office Action for U.S. Appl. No. 15/729,593, dated Aug. 10, 2018.
Office Action for U.S. Appl. No. 15/784,191, dated Aug. 26, 2020, 14 pages.
Office Action for U.S. Appl. No. 15/784,191, dated Dec. 19, 2018, 11 pages.
Office Action for U.S. Appl. No. 15/784,191, dated Jan. 24, 2020, 12 pages.
Office Action for U.S. Appl. No. 15/784,192, dated Sep. 19, 2019, 14 pages.
Office Action for U.S. Appl. No. 15/835,233, dated Feb. 8, 2019, 17 pages.
Office Action for U.S. Appl. No. 15/835,235, dated Feb. 25, 2019, 24 pages.
Office Action for U.S. Appl. No. 15/835,238, dated Dec. 11, 2019.
Office Action for U.S. Appl. No. 15/835,238, dated Feb. 7, 2019.
Office Action for U.S. Appl. No. 15/835,238, dated Jun. 19, 2019.
Office Action for U.S. Appl. No. 15/835,238, dated Jun. 5, 2020.
Office Action for U.S. Appl. No. 15/835,239, dated Feb. 7, 2019, 20 pages.
Office Action for U.S. Appl. No. 15/835,239, dated Jun. 19, 2019, 18 pages.
Office Action for U.S. Appl. No. 15/835,242, dated Oct. 18, 2019, 8 pages.
Office Action for U.S. Appl. No. 15/835,247, dated Dec. 31, 2018, 18 pages.
Office Action for U.S. Appl. No. 15/835,247, dated Jul. 10, 2019.
Office Action for U.S. Appl. No. 15/835,249, dated Dec. 31, 2018.
Office Action for U.S. Appl. No. 15/835,250, dated Apr. 4, 2019, 16 pages.
Office Action for U.S. Appl. No. 15/888,050, dated Jun. 11, 2019.
Office Action for U.S. Appl. No. 15/888,054, dated Mar. 11, 2019.
Office Action for U.S. Appl. No. 16/026,318, dated Sep. 20, 2018.
Office Action for U.S. Appl. No. 16/288,074, dated Oct. 7, 2020.
Office Action for U.S. Appl. No. 16/288,074, dated Mar. 5, 2020.
Office Action for U.S. Appl. No. 16/569,554, dated Aug. 18, 2020.
Office Action for U.S. Appl. No. 16/569,554, dated Jul. 2, 2021.
Office Action for U.S. Appl. No. 16/573,847 dated Jan. 6, 2021.
Office Action for U.S. Appl. No. 16/573,847, dated Aug. 2, 2021.
Office Action for U.S. Appl. No. 16/582,798, dated Jun. 24, 2021.
Office Action for U.S. Appl. No. 16/687,271, dated Jun. 24, 2021.
Office Action for U.S. Appl. No. 16/695,044, dated Jul. 8, 2021.
Office Action for U.S. Appl. No. 16/695,049, dated Jul. 21, 2021.
Office Action for U.S. Appl. No. 16/695,053 dated Aug. 4, 2021.
Office Action for U.S. Appl. No. 16/804,960, dated May 12, 2021.
“Selection of Cyclic Redundancy Code and Checksum Algorithms to Ensure Critical Data Integrity”, Federal Aviation Administration William J. Hughes Technical Center Aviation Research Division Atlantic City International Airport New Jersey 08405, DOT/FAA/TC-14/49, Mar. 2015, 111 pages.
Arashloo, Mina Tahmasbi , et al. , “SNAP: Stateful Network-Wide Abstractions for Packet Processing”, SIGCOMM '16, Aug. 22-26, 2016, 27 pages, ACM, Florianopolis, Brazil.
Bosshart, P. , et al. , “Forwarding Metamorphosis: Fast Programmable Match-Action Processing in Hardware or SDN”, SIGCOMM'13, Aug. 12-16, 2013, 12 pages, ACM, Hong Kong, China.
Kaufmann, A. , et al. , “High Performance Packet Processing with FlexNIC”, ASPLOS'16, Apr. 2-6, 2016, 15 pages, ACM, Atlanta, GA, USA.
Moshref, Masoud , et al. , “Flow-level State Transition as a New Switch Primitive for SON” , HotSDN'14, Aug. 22, 2014, 6 pages, ACM, Chicago, IL, USA.
Sivaraman, A. , et al. , “Towards Programmable Packet Scheduling”, HotNets'15, Nov. 16-17, 2015, 7 pages, ACM, Philadelphia, PA, USA.
Sivaraman, Anirudh , et al. , “Packet Transactions: A Programming Model for Data-Plane Algorithms at Hardware Speed”, arXiv:1512.05023v1, Dec. 16, 2015, 22 pages.
Sivaraman, Anirudh , et al. , “Packet Transactions: High-level Programming for Line-Rate Switches”, ?rXiv:1512.05023v2, Jan. 30, 2016, 16 pages.
Sivaraman, Anirudh , et al. , “Packet Transactions: High-level Programming for Line-Rate Switches”, SIGCOMM'16, Aug. 22-26, 2016, 14 pages, ACM, Florianopolis, Brazil.
Sivaraman, Anirudh , et al. , “Programmable Packet Scheduling at Line Rate”, SIGCOMM'16, Aug. 22-26, 2016, 14 pages, ACM, Florianopolis, Brazil.
Song , “Protocol-Oblivious Forwarding: Unleashe the Power of SDN through a Future-Proof Forwarding Plane”, Huawei Technologies, USA, 6 pages.
International Preliminary Reporton Patentability for PCT Application No. PCT/US2016/062511, dated Jun. 26, 2018.
International Search Report and Written Opinion for PCT Application No. PCT/US2016/062511, dated Feb. 28, 2017.
Notice of Allowance for U.S. Appl. No. 14/977,810, dated Oct. 20, 2017.
Notice of Allowance for U.S. Appl. No. 15/784,192, dated Sep. 30, 2021.
Notice of Allowance for U.S. Appl. No. 15/878,966, dated May 15, 2019.
Notice of Allowance for U.S. Appl. No. 16/460,798, dated May 27, 2021.
Notice of Allowance for U.S. Appl. No. 16/582,798 U.S. Appl. No. 16/582,798, dated Aug. 27, 2021.
Notice of Allowance for U.S. Appl. No. 16/687,271, dated Aug. 30, 2021.
Office Action for Chinese Patent Application No. 201680075637, dated Mar. 2, 2021.
Office Action for Chinese Patent Application No. 201680075637.4, dated Sep. 23, 2021.
Office Action for U.S. Appl. No. 15/878,966, dated Jan. 11, 2019.
Office Action for U.S. Appl. No. 14/977,810, dated Jun. 29, 2017.
Office Action for U.S. Appl. No. 16/569,554, dated Sep. 27, 2021.
Office Action for U.S. Appl. No. 16/804,960, dated Aug. 19, 2021.
Final Office Action for U.S. Appl. No. 16/695,044, dated Dec. 20, 2021.
Notice of Allowance for U.S. Appl. No. 15/784,191, dated Aug. 31, 2021.
Notice of Allowance for U.S. Appl. No. 15/835,247 dated Dec. 29, 2021.
Notice of Allowance for U.S. Appl. No. 16/573,847, dated Dec. 15, 2021.
Notice of Allowance for U.S. Appl. No. 16/582,798, dated Dec. 1, 2021.
Notice of Allowance for U.S. Appl. No. 16/687,271, dated Dec. 1, 2021.
Notice of Allowance for U.S. Appl. No. 16/695,049, dated Jan. 5, 2022.
Office Action for U.S. Appl. No. 16/695,053 dated Jan. 5, 2022.
Office Action for U.S. Appl. No. 16/804,960, dated Dec. 13, 2021.
Office Action in Chinese Patent Application No. 201680075637.4, dated Jan. 5, 2022.
Related Publications (1)
Number Date Country
20190394086 A1 Dec 2019 US
Divisions (1)
Number Date Country
Parent 14583664 Dec 2014 US
Child 16026318 US
Continuations (1)
Number Date Country
Parent 16026318 Jul 2018 US
Child 16519873 US