Patent Grant
10506434
This disclosure relates to networking. This disclosure also relates to a system for accelerated network route updating.
High speed data networks form part of the backbone of what has become indispensable worldwide data connectivity. Within the data networks, network devices such as switching devices direct data packets from source ports to destination ports, helping to eventually guide the data packets from a source to a destination. Improvements in packet handling, including improvements in path resolution, will further enhance performance of data networks.
Network packet/data routing (also referred to as packet routing or data routing) can require use of routing tables that are maintained and updated as routing among networked devices changes. Prior to the implementation of network function virtualization (NFV), updates to established routes were relatively infrequent. With the implementation of NFV, allocation and de-allocation of data compute services offered by numerous processor cores possibly distributed throughout one or more networks has become more prevalent. These changes in routing should be operations that occur efficiently and simply to optimize utilization of compute resources and thus increase economic value/potential. The virtualization of network functions may employ methods such as: (a) Standardization of packet/data route update application program interfaces (APIs), which may be referred to as RUAs, and (b) Fast execution of these update APIs to shift or otherwise change the network routing possibly as much as multiple times a millisecond. NFV can make available standardized RUAs that can be used by compute and storage service management software to establish and/or tear down packet/data routes between data generation/storage and data compute elements at frequencies not previously contemplated.
The route update system allows such RUAs to manage compute services in smaller and smaller granularities such that dynamic changes in routing can complement and enable efficient use of compute resources allocated to NFV function. Thus, since the route update system enables frequent and reliable dynamic changes to network data routing, changes in allocations of NFV resources is unconstrained by associated dynamic changes in network data routing. In some examples, the routing table update rates that at present, range in few thousand to few tens of thousands per second can be transformed by the routing update system to few hundreds of thousands to few millions per second. Thus, the routing update system can enable/accelerate the adoption of Software Defined Networking (SDN) and Network Function Virtualization (NFV). The route update system enables atomic update of multiple live-routing entries associated with a routing entry, which can also be referred to as all or none routing entry updates. Thus, the system can create a new routing entry and update all memory locations related to this new routing entry in a single continuous linear action without interruption by other processes or activities. The single continuous linear action also includes updates to routing entry information shared with other live routing entries. The atomic update enables better utilization of routing table memory since no additional memory space allocation is needed for partially updated routes due to portions of a route update request being separately executed to allow for intervening other functions to be executed. The routing table update rates that in previous systems, ranged from a few thousand updates per second to a few tens of thousands of updates per second can be transformed using the route update system to hundreds of thousands of updates per second to millions of updates per second. Thus, use of the routing update system should enable/accelerate the adoption of Software Defined Networking (SDN) and Network Function Virtualization (NFV)
A routing table update may occur through a process referred to as a control plane update. Control plane updates create a new routing entry when all memory locations related to a new routing entry have been updated. Once all memory locations are updated, the root location of the routing entry is changed, which then diverts the search process to go through new entry related memory locations. If certain portions of routing entry information is shared with other live routing entries, this information must be created/copied to all memory locations related to those routing entries as well (and then change the root memory locations for each of those routing entries one by one). Thus, control plane updates face multiple performance challenges that introduce significant delay in completion of routing table updates. For example, first, control plane updates execute on control plane central processing unit (CPU), and thus are subject to an operating system's allocation of CPU resources. Secondly, the updates may invoke a driver that can take multiples of micro seconds (pS) of time to execute, such as Peripheral Component Interconnect express (PCIe) drivers, as one example. Thirdly, control plane updates may face arbitration with data plane traffic in different ways and the control plane traffic may be de-prioritized by the arbitration thereby introducing additional undesirable delay. The situation becomes even more acute when a control plane CPU is located in a different device (such as in a centralized software defined network (SDN) controller) and connects to a local control plane CPU through switches or other network devices. Given all these undesirable delays points, control plane performance (which may be measured as updates/sec) may be many orders of magnitude lesser than data plane performance. Software Defined Networks and Network Function Virtualization may implement dynamic route updates, thus increasing the control plane update performance requirement.
The route update system as described by the methods, systems, devices, circuitry and logic disclosed herein may bypass some or all of the control plane update bottlenecks noted above, and may result in increased speed and efficiency in performing routing table updates. In some implementations, the route update system picks up and/or pushes out the update instruction from a control plane close point. A control plane close point refers to a position within the source, such as on a chip, or in close vicinity of the source. The system can transfer the update instruction through an update communication path that can involve traditional bridging, switching and routing to a device close to a table storage location, such as in a memory table within a switch. A device close to table storage location refers to a location within a destination, such as a routing chip or in close vicinity of the routing chip. Accordingly, portions of the route update system can be on chip, on package, such as part of a multi-chip configuration, on circuit board, or in area, such as in a datacenter. A standards based Route Update System (RUS) connects the control plane CPU chip/box to a switching chip/box, whether both are local to a chip (e.g., on a System on a Chip (SOC)); local to a physical package (e.g., a Multi-Chip Module (MCM)); local to a board (e.g., on a common board with a control plane processor), or local to a network. Accordingly, a Route Update System (RUS) is implementable in an SoC, MCM, printed circuit board (PCB), in a network, in other alternative or additional implementations, and in any combination thereof. As one effect, the RUS may transform control plane communication into data plane communication.
Table 1 below shows example terminology and operational parameters of the RUS system, which may or may not be stored in registers and which are discussed further below.
Regarding conventions for defining a register value, in the examples provided herein, a Verilog like syntax can be used. For example, as used herein, a syntax of {WIDTH{1′b0}} may refer to a set of “WIDTH” related bits, having a value of 0. In another example, “1′b0” implies a one bit wide binary number having value of 0 and “1′b1” implies a one bit wide binary number having value of 1. As another example, “4′h7” implies a 4 bit wide number in hexadecimal notation having a value of 7.
Many of the example parameter values described herein have been included based on experience and experimental results for switches and network processors. In other examples, any number of alternative parameter values may be used.
The control plane processors can be specialized computers, such as a server computer, a router, a switch, a gateway, or any other device that includes a processor, memory, and at least one user interface. The data routing switches can be one or more devices that can connect devices together on a computer network, and can forward data packets between computer networks, such as a line card.
In some implementations of the present disclosure, the RUS 100 connects a route update source, such as the CPP and a route update target, such as the DRS (target router), with reduced or minimal intrusiveness to the operation of the CPPs 101-103 and the DRSs 111-113. For the DRSs 111-113, operation may be affected only to the extent that the DRSs 111-113 see an arbitration for (routing) table access, which may cause a DRS to push back, or delay execution of a search instruction in certain situations. The RUS 100 may include three components, in one example implementation: a) an Update Snoop Module (USM) circuitry, b) an Update Inject Module (UIM) circuitry and c) a RUS connectivity system (RCS) through which the USM can talk to the UIM. The RCS may include the RUS connectivity switches 121-124 for example. A USM is an element that may be embedded in RUS compliant CPP devices and a UIM is an element that may be embedded in RUS compliant DRS devices. Alternatively, or in addition, a USM component may be embedded in RUS compliant DRS devices and a UIM component may be embedded in RUS compliant CPP. The functions of the USM and UIM are described in greater detail below. Actual implementation of these components may take any number of forms, such as dedicated hardware embedded into the CPP or DRS.
Multiple Master/Remote Master RUS
In the example shown in
In an example of operation of two or more masters using the illustration of
Single Master RUS
In some communication systems, a DRS may use routing tables without updating the routing tables themselves. In such systems, a CPP may take care of route table updates. Further, a single CPP may own the routing tables of all DRS devices in the RUS. The RUS 100 may be implemented in such a case, which may be referred to as a simple RUS.
Connectivity System and Data Structures
The RUS may include one or more communication interfaces that carry update information (such as a routing table update instruction) between a CPP and a DRS, and implementations of the interfaces may vary. These communication interfaces may include the snoop_out and update_in busses. The communication interface(s) of this example could be implemented as parallel busses if the CPP and DRS are on same die or package. On a circuit board or other circuit interconnection device, IL/ILA or PCIe would be possible bus implementations to carry the update information from the CPP to the DRS. If the CPP and DRS are located on different circuit boards, boxes, or devices, the physical representation of the communication interface may be PCIe/Infiniband or Ethernet, or via any other wired or wireless communication link.
The CPP 502 also includes a hard engine 512 that includes update snoop module (USM) circuitry 514 operating as a source of hardAPI messages. As used herein, the term “hard engine” is defined as an entity implemented using hardware, such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a micro engine, or processing circuitry, such as circuitry included on the chip adjacent to the a processor, and which can communicate using hardAPI packets to update a routing table. Accordingly, the USM circuitry 514 can be logic hardware in the form of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or processing circuitry, such as circuitry included on the chip adjacent to the processor 510. As used herein, the term hardAPI refers to an Application Programming Interface that uses hardAPI packets to communicate between entities such as an NFV/SDN controller and/or a packet forwarding engine.
During operation, the USM circuitry 514 can cooperatively operate with the CPU 510 to generate a route update instruction. The route update instruction thus can be transmitted by the CPU 510 as a hardAPI packet over a hardAPI packet communication path 516. As used herein, a hardAPI packet is defined as a network packet that has a payload limited to Open System Interconnect layer three or below (using the OSI (Open System Interconnection) reference model). Accordingly, a hardAPI packet is directly readable and useable by hardware (a hard engine) without complex transport or application layer states that are present in Open System Interconnect layer four and above packets. The hardAPI packet is provided to the connectivity system 506 via a hardAPI packet communication path 516. The connectivity system 506 can receive and transmit the hardAPI packet to the DRS 504 via the hardAPI packet communication path 516 using a protocol, such as PCIe, Infiniband, or Ethernet, for example, as previously discussed, based on a destination address included in the hardAPI packet. The hardAPI packet need not be converted or changed by the connectivity system 506 since the hardAPI packet is readable by a hard engine included in the DRS 504 (the destination) without any conversion or translation.
The DRS 504 can include a processor 518, and a hard engine 520 that includes update inject module (UIM) circuitry 522. By being included in the hard engine 520, the UIM circuitry 522 can be logic hardware in the form of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or processing circuitry, such as circuitry included on a chip that includes the processor 518. Since the hardAPI packet is an Open System Interconnect layer three or below packet that is directly readable and useable by the DRS 504, the hardAPI packet may be provided directly to the UIM circuitry 522, bypassing processing by the CPU 518. Accordingly, the route update processing rate is substantially a function of the time required to generate the hardAPI packet.
In communication of route update instructions between the CPP and the DRS without the USM 514 and the UIM 522, a communication packet can be generated by a soft engine, such as the CPU 510, and transmitted as a softAPI packet along a softAPI packet communication path 528 using software defined networking (SDN). The softAPI packet communication path can include Control Data-Plane Interface (CDPI) drivers and CDPI agents such that the softAPI packet is processed through one or more routers 530, which each include at least one soft engine, such as a CPU 532, that processes the softAPI packet, and through the connectivity system 506 (if needed). In addition, the softAPI packet is received and processed by the CPU 518 at the DRS 504 to convert the softAPI packet to a format of individual memory update instructions. Thus, the softAPI packet is subject to processing along the softAPI packet communication path through multiple softAPIs CPU's so the rate at which the routing table is updated is dictated by the processing speeds of the multiple CPUs along the softAPI packet communication path 528. In some implementations, the routers 530 may be omitted and SoftAPI packets may travel from the CPP CPU 510 to the connectivity system 506 and from the connectivity system 506 to the DRS CPU 518 without going through the routers 530.
A network device may include one or more elements, circuitry, components, or other logic to support RUS connectivity. In some implementations, a device may implement these elements to support RUS connectivity in hardware or a combination of both hardware and software. For example, the device may implement a software driver that programs registers in the system, e.g., within a CPP, within a DRS, or both. The driver may also provide one or more APIs to manage a routing table update, for example to make manual updates, to see the performance of the update, and to perform table swapping in case of failures. As previously discussed, a device may implement the Update Snoop Module (USM) 514 or Update Inject Module (UIM) 522, as dedicated hardware or circuitry in the form of a hard engine. The USM 514 may be implemented by or added to the CPP device 502 and the UIM 522 may be implemented by or added to the DRS device 504. If a device is both a master as well as route table holder (such as a DRS 504 that can update its own routing tables), the device may implement both the USM 514 and UIM 522.
A communication protocol for the hardAPI packets is fully and directly useable by the UIM 522. Table 2 provides some exemplary data structure details of a UIM instruction buffer, data structure that illustrates the communication protocols used by the USM 514 and the UIM 522 to generate hardAPI packets.
Example implementations of a data structure for register/memory details in the USM 514 and the UIM 522 for the purpose of recovery from a loss of data events, such as failures, are provided in Table 3:
The USM circuitry 514 can be physically positioned as close to the processor 510 in the CPP 502 as possible. In an example implementation, the USM circuitry 514 is a hard engine in the form of a hardware circuit that is placed on the same die as that of processing cores of CPP 502. Also, in an example, the CPP 502 can be positioned as close to a SDN/NFV controller as possible. In some examples, the CPP 502 can be the SDN/NFV controller itself. The translation of the APIs enables update instructions at the UIM circuitry 522 that a hardware state machine and/or micro-coded engine can directly interpret and make an update to routing table memory.
Update Snoop Module (USM) Circuitry
The API translator circuitry 722 can execute high level route maintenance/update APIs, such as RUA APIs, which can be defined by a standards body such as OpenFlow, as previously discussed, to generate a set of local memory updates. These local memory updates can be packaged and converted by the USM circuitry 700 into HardAPIs that are hardware interpretable at the DRS, and more specifically, by the UIM circuitry. These local memory update instructions are transmitted over a memory interface bus 726 for use by the CPP 701 to update onchip or offchip local memory storage (local data memory) of the CPP 701. The memory storage of the CPP 701 can mirror a routing table stored in the memory storage of one or more DRS so that the CPP 701 can make changes locally in the local data memory to the currently stored routing.
The API translator circuitry 722, in addition to translation functionality, can also include functionality for enabling the UIM 522 to perform data processing, authentication and data integrity, such as by base address shift, data encryption, Endian (Big Endian, Little Endian) conversion and/or compression. For example, as part of the translation process, an authentication token, such as a certificate or a key may be included in the hard engine interpretable route update instructions so that UIM circuitry that receives one of the update instructions can authenticate that the originator of the update instruction was the CPP 701 from which the update instruction was received. In addition, the API translator circuitry 722, may provide some form of encryption or other form of data integrity techniques so that UIM circuitry that receives one of the update instructions can verify that the update instruction was not tampered with or otherwise compromised after being generated at the CPP 701 from which the update instruction was received. In an example, the authentication and data processing can be indicated in the update attribute bits of Table 2, as previously discussed. By making the API translator circuitry 722 work on local memory, a major overhead related to invocation of a driver (for example a PCI driver) is avoided.
Updates, including routing updates may be performed by the CPU core 720 using cache to read programs and data efficiently from memory. The illustrated L2/L3 cache 716 can be coupled with the CPU core 720 by a read/write address and data path 729, and can be positioned between data memory that is located on or off chip where update instructions are to be written and the CPU core 720. In an example implementation, the L2/L3 cache 716 may be the last cache prior to the memory interface bus 726 with which the local memory update instructions (also referred below as write transactions) are transmitted to the onchip or offchip memory storage of the CPP 701, as illustrated. The L2/L3 cache 716 is the last cache so that the USM circuitry 700 is not positioned between the CPU core 720 and the L2/L3 cache 716, but is instead positioned as close as possible (close coupled) with the CPU core 720, but after all the cache layers. The reset and clocks circuitry 718 may represents utilities that are provided to the USM circuitry 700 and/or the CPP 701, when the USM circuitry 700 is implemented as on-die circuitry, or as a separate chip/FPGA entity.
The USM circuitry 700 uses predetermined snoop address filters that can be stored in the configuration circuitry 704 and provided over an address filter line 732. The snoop address filters can define address ranges that need be snooped (for update purpose) and captured. The configuration circuitry 704 can also define update atomicity behavior performed with the atomic update boundary logic circuitry 706. In general, update atomicity behavior refers to the capability of the USM/UIM circuitry to perform all or none updates or changes in routing. The configuration circuitry 704 may include one or more registers in the form of, for example, the address range registers included in the example communication protocols described in Table 2. In an example configuration, the USM 700 includes one register per update address space. These registers can be updated through, for example, a separate I/O bus, which can be connected to internal peripheral bus of the CPU core 720.
During operation, the snoop and decode logic circuitry 702 may monitor write transactions occurring on the memory interface bus 726 to identify routing update transactions. When a write transaction message satisfies predetermined conditions, the snoop and decode logic circuitry 702 can identify the write message as a routing update transaction. Once write transaction messages are identified as routing update transactions, the snoop and decode logic circuitry 702 can capture a copy of such write transaction messages. In an example configuration, there can be two types of capture conditions for routing update transaction messages—a first condition to identify and capture update messages for local configuration registers of the CPP 701 based on write transactions, and a second condition to identify and capture write transactions that are route update instructions mapped to routing table memory. If the address in the write transaction being snooped belongs to a local configuration register, the local configuration register included in the configuration circuitry 704 can be updated by the snoop and decode logic circuitry 702 with write data over a register line 732. Examples of such configuration registers are the address range registers that define mapping to routing tables of Data Routing Switches (DRS) illustrated in Table 2. Where the second capture condition occurs, because the write routing update transaction is a routing table update instruction, the routing table update instruction may be captured and indicated to the hardware packet generation circuitry 707. In an example, a copy of the route update instruction may be transmitted to the packet generation circuitry 707, for temporary storage in the update instruction buffer circuitry 708. In other examples, other forms of indications of a routing updating instruction may be provided to the packet generation circuitry 707 as an indication of the captured route update instruction.
The atomic update boundary logic circuitry 706 may monitor those routing update instructions captured by the snoop and decode logic circuitry 702 to determine if atomicity is warranted because the update to the routing table has to be done in an atomic fashion to update a group of memory locations in the routing table. The atomic update boundary logic circuitry 706 processes the route update instructions and determines whether to associate/embed a parameter in the write instructions that is indicative of an atomicity requirement. The atomicity parameter can be a flag or other indicator, which is read by the UIM circuitry upon receipt of the hardAPI packet. The atomicity parameter may also include an indicator that associates multiple hardAPI packets that should be part of an atomic routing table memory update.
In examples the packet transmitter circuitry 712 need not wait for an “acknowledge” message or other acknowledgement indication from an external source before transferring the route update instructions and update message out, though such acknowledgement capability may be optionally implemented. Having such a capability can make the USM 700 compatible with various bus interfaces (connectivity systems). It is noted, however, that connectivity systems such as PCIe or Infiniband may automatically provide such a capability and therefore any form of acknowledgment may be omitted and the accompanying delay may be avoided. In some examples, the address space of interest for the USM circuitry 700 may require write-through capabilities for the L2/L3 cache circuitry 716. In such examples, the write through capabilities could be implemented through a hardware update or software programming.
The update instruction buffer circuitry 708 of the USM 700 may store route update instructions to meet atomicity (and, optionally or in addition, transport efficiency) requirements and to allow for frequency differences between clocks, such as between a connectivity system clock and the RUS clock (Table 1). The update instruction buffer 708 may support instruction buffering to meet the needs of packetization requirements and atomicity grouping requirements, and also to address differences in transport speed and local clock speed. The update instruction buffer 708 may be implemented as a first in, first out (FIFO) buffer, and may have a number of predetermined threshold parameters associated therewith. In an example, the predetermined threshold parameters can include AlmostFull and AlmostEmpty threshold values discussed with regard to Table 1. The predetermined threshold parameters may be contained in the configuration circuitry 704 and provided to the update instruction buffer 708 on a threshold parameters line 734. When the occupancy of route update instructions in the update instruction buffer 708 reaches a value of update instructions equal to the a predetermined values, such as AlmostFull level, the USM circuitry 700 may assert flow control such as usm_cpu_hold on a flow control line. Once asserted, the USM circuitry 700 may maintain flow control until buffer occupancy of the update instruction buffer 708 reaches a predetermined threshold level, such as the AlmostEmpty level.
The instruction packetizer circuitry 710 creates/generates each of the hardAPI packets as a single data layer packet using the route update instructions stored in the update instruction buffer circuitry 708. Since an update to a route can include updates to many different memory locations in a routing table, the instruction packetizer 710 can confirm that all update instructions that belong to one atomic update are packaged together in a group in a single hardAPI packet, or in a series of hardAPI packets. Where the update instructions are in multiple hardAPI packets, an indicator can be included to delay update processing at the UIM circuitry until the entire set of route update instructions are received at the UIM circuitry. Since the payload of each hardAPI packet includes a group of route update instructions in a payload limited to Open System Interconnect layer three or below that is directly compatible with the UIM circuitry, the UIM circuitry can directly parse the hardAPI packet and execute the update instructions without processing by a CPU.
In addition to extracting and creating a payload for each of the hardAPI packets, the instruction packetizer circuitry 710 can also generate and attach a packet transport header to each hardAPI packet. The packet transport header is in a hardware compatible format, and can be directly parsed by the UIM in the DRS, which is identified in the packet transport header. Information included in the packet transport header can include, for example, a source media access control (MAC) and/or internet protocol (IP) Address, a destination MAC and/or IP Address, a VLAN ID, a PCIe protocol, or some other unique identifier(s) so that the hardAPI packet can directly reach the UIM circuitry of the intended DRS without further CPU processing. Examples of communication protocols/path addressing for example hardAPI packets have been described elsewhere.
The packet transmitter circuitry 712 can transmit hardAPI packets generated by the instruction packetizer circuitry 710 over the snoop_out bus (see
Thus, the USM 700 may perform two different types of flow control. In a first type of flow control, the update instruction buffer circuitry 708 can perform flow control with a first flow control signal provide to the L2/L3 cache 716 on the flow control line 728 to limit the input of route update instructions to the memory interface bus 726, if the buffer circuitry 708 reaches the maximum continuous allowed update for a given address space (for example, based on internal FIFO threshold). The USM 700 may use the first flow control signal, such as a usm_cpu_hold signal, where the CPU core 720 provides for such a freezing capability. The advantage of such a freezing capability is to make the update speed of the USM circuitry 700 transparent to the API translator circuitry 722. In a second type of flow control, the packet transmitter circuitry 712 can match the output of the USM to the available bandwidth or throughput of the connectivity system with a second flow control signal provided on the flow control line 736 to regulate the generation of hardAPI packets by the packet generation circuitry 707 by regulation of the instruction packetizer circuitry 710. As one or the other of the first and second flow control signals are initiated, the other of the first and second flow control signals also may be implemented as needed to manage throughput of route update instructions. Since a routing table update rate may be less or more than a data transfer rate that the USM 700 included with the CPP 701 can transport to DRS devices, the first or second flow control signals may cooperatively operate to maximize the efficiency of the CPU core 720 and the connectivity system. In addition, such cooperative operation using the two flow control signals may be used to coordinate operation of the UIM 700, such as by implementing (average) update speed matching.
Update Inject Module (UIM) Circuitry
The CPU 834 is a soft engine that is capable of communicating with a network 836 and receive and process softAPI packets. The softAPI packets are received by the CPU 834 over a network data packet line 844. Similar to the CPU 518 discussed with reference to
In addition, the CPU 834 may be used to program and read back the configuration registers circuitry 818 of the UIM circuitry 800 over a registry line 848. Programming of the UIM circuitry 800 and the DRS device 801 may occur through a common programmed input/output (PIO) bus forming the registry line 848. The CPU 834 can manage the reading/writing of various resources of the DRS device 801. The CPU 834 can be implemented with a combination of hardware and software as a soft engine, resulting in undesirable delay due to CPU cycle time, and therefore can be used mainly during debugging or initialization time. During run-time, the CPU 834 is generally not used to maintain the routing table in the routing table memory 828 since individual instructions to update an individual location (or image) in the routing table memory 828 received via the CPU 834 result in undesirably slow operation.
The search logic circuitry 830 can receive over a network data packet line 850 network data packets from the network 836 that are in need of routing by the DRS 801. As used herein, the terms network data packets refers to data packets that are be transmitted from a source to a destination by passing through the DRS 801. The destination can be an address of a destination device, which is other than the DRS 801. Accordingly, such network data packets are different than the hardAPI packets since they are not addressed to local DRS 801 whereas the hardAPI packets are addressed to the DRS 801 for receipt by the UIM 800. The search logic circuitry 830 may perform routing decisions by parsing certain portions of the network data packets, such as a header included in such network data packets. The parsed portions of the header can be used by the search logic circuitry 830 to look up routing information stored in the routing table memory 828 and make one of two decisions: (a) Whether or not to forward the packet and, (b) If it is decided to forward the packet, decide where to send the packet based on one or more stored entries retrieved from the routing table.
The search logic circuitry 830 may also include network data packet editing functionality, as well as functionality to generate route search requests. Thus, the search logic circuitry 830 can learn the network 836, generate route search requests to read routing information from the routing table memory 828, and modify the routing of the network data packets based on the routing information retrieved. Route search requests may be used by the search logic circuitry 830 to search for and retrieve from the routing table memory 828 a route for a received network data packet. Thus, the search logic circuitry 830 can access the routing table memory 828 to extract stored routing table information by generation of route search requests. Routing information provided by the routing table memory 828 in response to the route search requests may be used by the search logic circuitry 830 to route the network data packets by modifying or confirming the header information included in the network data packets.
The search logic circuitry 830 can send read route search request messages through the table memory arbiter 822 to the routing table memory 828 to request stored information. The search logic circuitry 830 may also perform other functions, such as management of aging routing information stored in the routing table memory 828. For example, the search logic circuitry 830 can search for and eliminate routing table entries in the routing table memory that are not used/accessed for a predetermined period of time, such as six months.
The routing table memory 828 is where route information is stored and used by the search logic circuitry 830 to decide the forwarding (destination as well as desirability) of the packets. The routing table memory 828 can be on chip or off chip or can be a mix of the two. Accordingly, the routing table memory 828 may be located in the DRS 801 and/or external to the DRS 801. As previously discussed, the routing table memory 828 may be a mirror image of the CPP memory routing table information. The reset and clock circuitry 832 can provide utilities for the UIM circuitry 800, whether the UIM circuitry 800 is implemented as on-die circuit or as separate chip/FPGA entity.
The configuration registers circuitry 818 includes various configurations of registers that maintain parameters which configure the operation of the UIM circuitry 800. Such parameters may include data and information related to translation, decryption and other operational functionality used by the UIM circuitry 800. As previously mentioned, the CPU 834 may be used to program (write) and read back the configuration registers 818 of the UIM 800. The reading and writing of the configuration registers 818 with the CPU 834 can take place over the registry line 848, such as a PIO bus.
The update recovery and synchronization module circuitry 820 can dynamically capture summary information regarding route instruction updates, along with a current time stamp. The captured summary information and current time stamp can be stored in memory. The dynamic capture of summary information can provide for data checking and also recovery, based on the time stamps, of a most recent routing table update of the routing table memory when certain failures occur. Recovery and synchronization related data structures have been discussed elsewhere. The update synchronization and recovery module circuitry 820 can alternatively, or in addition, dynamically capture memory table updates initiated by the CPU 834 and/or the table memory update module circuitry 814.
The table memory arbiter circuitry 822 can operate to provide multiple paths to a memory bus 840, which provides access to the routing table memory 828. The table memory arbiter circuitry 822 can receive access requests from either the CPU 834, or the table memory update module circuitry 814, or from the search logic circuitry 830. Upon granting a request, the table memory arbiter circuitry 822 may open an exclusive read/write path from the CPU 834, the table memory update module circuitry 814, or the search logic circuitry 830 requesting access to the routing table memory 828 to allow updates to, or data extraction from, the routing table memory 828. Thus, in the illustrated example, the table memory arbiter circuitry 822 can operate as a switch that switches between the CPU 834, the search logic 830 and the table memory update module 814 to selectively provide exclusive access to the routing table memory 828. In the illustrated example, the table memory arbiter 822 is included in the UIM 800.
The table memory arbiter circuitry 822 may also perform flow control by generating flow control signals on a flow control line 838. Flow control may be initiated by the table memory arbiter circuitry 822 as a result of too many route search requests (read requests) being received from the search logic circuitry 830. In addition, or alternatively, flow control to reduce, limit or stop the frequency of provision of network data packets (and corresponding route update requests) may be initiated by the table memory arbiter circuitry 822 during a period of atomic update requests being performed by the table memory update module circuitry 814. In other words, the table memory arbiter circuitry 822 can initiate flow control to stop the search logic circuitry 830 from searching the routing table memory, in order for a route update instruction identified as atomic to be fully completed while the table memory arbiter circuitry 822 is allowing access by the table memory update module circuitry 822 to the routing table memory 828. Thus, flow control by the table memory arbiter circuitry 822 can be initiated in coordination with atomic update being performed based on the atomicity control module circuitry 812. This allows a completed update of the routing table memory 828 to take effect prior to receipt of access from the search logic circuitry 830.
As another implementation, the UIM 800 may not have direct access to the routing table memory 828, and the UIM 800 may access an intermediate bus to reach the target memory instead. (see
The table memory update module circuitry 814 can generate and perform read/writes via the table memory arbiter 822. The read/writes may be performed as read/writes, or as read/modify/writes. An example of a read/modify/write is in the case where the number of bits in the route modification instructions do not match the width of the memory location in the routing table memory 828, and the table memory update module circuitry 814 modifies accordingly prior to sending a write instruction that includes a route instruction update. The table memory update module circuitry 814 can perform at least two things, which are: 1) to request the table memory arbiter 822 to provide exclusive access to a memory bus 840 to write the table memory; and 2) to send the write/update instructions to table memory once exclusive access is granted. Alternatively, or in addition, the table memory update module circuitry 814 could include read-modify-write to table memory. The arbitration interface works in conjunction with Atomicity Control Module.
The atomicity control module circuitry 812 can monitor for and detect atomicity identifiers in the hardAPI packet. In addition, the atomicity control module circuitry 812 can instruct the table memory update module circuitry 814 to direct the table memory arbiter 822 to maintain the grant of access to the routing table memory 828 via the memory bus 840, until all route update instructions that are indicated as subject to an atomicity requirement are completed. Route update instructions indicated as subject to an atomicity requirement may include those that are indicated as a group of atomic route update instructions, and/or one or more route update instructions that update a group of memory locations in the routing table memory. In these situations, the table memory arbiter circuitry 822 may perform flow control via the flow control line 838 to the search logic circuitry 830, as previously discussed.
During operation, hardAPI packets can be received at the hardAPI packet processing circuitry 810. The hardAPI packets can be received from the connectivity system as previously discussed, which, in some examples could be the network 836, such as in an Ethernet protocol, but in other examples may be other communication paths, such as PCIe, or Infiniband, or other communication protocol/path. The hardAPI packets can be sent by USM circuitry 700 included in an upstream CPP device 701. (
Within the hardAPI packet processing circuitry 810, the update instruction packet receiver 802 can receive hardAPI packets on the update_in bus (see
The instruction unpacker circuitry 804, included within the hardAPI packet processing circuitry 810, can remove any route update header present in the payload (single data layer) or transport header of the packet. For example, the instruction unpacker circuitry 804 can remove and analyze an instruction packet header. In addition, the instruction unpacker circuitry 804 can perform authentication checks on the received hardAPI packet to protect the sanctity of the routing table. Authentication may include, for example, verification of the validity of a source signature (USMSignature embedded in Update Instructions or source) or other unique identifier of the source included in the packet. Such security checks may be performed by the instruction unpacker circuitry 804 based on the content of the hardAPI packet. Authentication related information may be encoded into the hardAPI frame by the instruction packetizer 710 (
The update instruction buffer 806, included within the hardAPI packet processing circuitry 810, may receive and store the unpacked route instructions. The route instructions may be stored in the queue of the update instruction buffer 806 until extracted by the instruction translator circuitry 808. If the update instruction buffer 806 gets to be full according to a predetermined threshold, a suitable flow control signal may be asserted to the update instruction packet receiver 802 over a flow control line 842. In response to receipt of the flow control signal, the update instruction packet receiver 802 can transmit a flow control message 850 back to the USM over the connectivity system, to temporarily halt the flow of hardAPI packets to the UIM. The flow control message may be transmitted over the connectivity system to the upstream module/device, such as the USM circuitry, based on the source MAC address and/or IP address received in the hardAPI packet.
The instruction translator circuitry 808, which is also included in the hardAPI packet processing circuitry 810, can perform a number of different types of memory write related preprocessing and security related processing, of contents of the hardAPI packets stored in the update instruction buffer 806. In an example, the instruction translator circuitry 808 can perform a base address translation, which can include offsetting and masking. The related offsets and masking information can come from the configuration registers circuitry 818. The route update instructions provided from USM circuitry can generally be at different absolute addresses than the address location where the actual table routing resides in the routing table memory 828 of the UIM circuitry 800. This may be due to any of various reasons such as due to better address management at the DRS 800, multiple different DRS circuitry supporting the route being updated, and/or security of the routing table's data structure.
In another example, the instruction translator circuitry 808 can perform translation related formatting after successful authentication of a signature. Formatting by the instruction translator circuitry 808 can involve conversion, such as Endian (Big Endian and/or Little Endian) conversion. In this example, the instruction translator circuitry 808 can receive, within the hardAPI data packet, parameters indicative of Endian conversion such as, for the case of Big Endian, the size of each word included in the hardAPI packet, and for Little Endian, the number of bytes included in the hardAPI packet. In an example, the instruction translator circuitry 808 can receive data and operating instructions in Little Endian, and routing update instructions in Big Endian with corresponding parameters included in, for example, the route update attributes (Table 2) included in the hardAPI data packets.
In yet another example the instruction translator circuitry 808 can provide decryption related security protection of the route table entry. In addition (or alternatively) to the address authentication, such as MAC and/or IP address authentication, provided by the update instruction packet receiver 802, and the signature authentication provided by the instruction unpacker circuitry 804, the data coming to the routing table can be encrypted. Since the instruction translator circuitry 808 knows the USM and corresponding API translator circuitry 722 (
In addition, the instruction translator circuitry 808 can perform de-compression related data processing. Decompression can include the capability of the instruction translator circuitry 808 to translate a single route update instruction into multiple memory location updates in the routing table stored in the routing table memory 828.
In the case of a single update instruction being used to update multiple memory locations in the routing table memory 828, translation by the instruction translator circuitry 808 may include state machines or micro-coded engines, without the use of a CPU or other processor to avoid and/or minimize undesirable cycle time during the route update instruction processing by maintaining the processing in hardware.
In
Existing CPP Example Enhancements
CPP enhancement may include adding a USM to an existing core-to-memory communication bus of a CPP device. A core may include an existing cache (L1 or L1/L2 or L1/L2/L3), the bus or communication link of interest for adding the USM can be between the last (e.g., highest) cache to the external memory (e.g., external to the core). The enhancement of adding the USM may vary depending on a particular implementation of the CPP.
Existing DRS Example Enhancement
The DRS enhancement may include adding a UIM (or both a USM and a UIM) to a “search logic”-to-memory communication bus of a DRS device. Since the UIM may inject route update instructions (as opposed to just snoop), the UIM may be positioned serially in the communication path of the search logic to the routing table memory. Thus, to install the UIM, the path can be “broken” and the UIM table arbiter circuitry can be inserted in order to operate as an arbiter between the search logic and the table memory update module of the UIM, as previously discussed. Optionally, the DRS may implement or add the UIM such that the UIM is inserted at a point where the UIM is able to flow control an upstream data flow coming from search (or non-RUS processor input/output) logic. The enhancement of adding the UIM may vary depending on a particular implementation of the DRS.
Multiple Master Configurations
A RUS may include multiple master devices or logic, as previously discussed. In such a scenario, one or more of the following exemplary specification may optionally be implemented in the RUS:
As a first exemplary specification, the RUS may specify that a particular address space is owned by single master. For example, L2 (MAC) entries may be owned by a DRS (with master functionality) and L3 entries may be owned by a CPU or CPP.
As a second exemplary specification, the RUS may specify that ownership of a shared address space is separated in or according to a predetermined time dimension or operational window. This may be implemented by time based sharing or operational based sharing. Thus, a CPU or CPP may “own” a shared address space in memory or cache or the configuration registers at the time of initialization of the RUS system, and the DRS may own the shared address space at run time of the RUS. For example, a shared address space of cache L2 entries may be shared between a CPP and DRS in this way. Optionally or in addition, the RUS may specify a handshake of ownership transfer between masters.
As a third exemplary specification, the RUS may specify that a shared ownership table is tolerant of mis-ordering of updates. The RUS may also specify that hardware that “owns” (such as directly, and therefore has priority) the shared ownership table can arbitrate among the other co-owners in order to ensure multiple updates do not garble or corrupt the content. An example application of a shared ownership table may be in the case of aging entries and/or hit bit entries.
Additional Example Specifications for Control Plane Processor (CPP)
The RUS may specify one or more of the following exemplary specifications for CPP devices operating in connection with the RUS.
As a first exemplary CPP specification, the CPP may read the lowest bits first and then higher bits while reading the LastUpdate register (Table 3). This specification may apply when the whole register cannot be read in one go, such as, when requiring multiple read operations to read the LastUpdate register.
As a second exemplary CPP specification, the CPP may enforce parameters to ensure read-update coherency. Since the RUS may be defined and used for table route update purposes, read-update coherency should be handled uniformly within the RUS. In some cases, there is no read involved, so maintaining read-update coherency may be inapplicable. In some cases where a read may be called for, the RUS may enforce or provide specifications for a master to follow to ensure that such coherency is maintained across different platforms, devices and circuitry. For example, the RUS may specify that a master (e.g., CPP or DRS) use only a single consistent method of update for a given address space. In an operational example, the RUS may use a predetermined RUS specific method, or a conventional method, but not both, for a given address space. If, for example, the update method is a predetermined RUS specific method, the master device may read the routing tables to get “a copy” of the routing table of interest, and use the “UpdatesSinceLastRead” parameter (Table 3) to determine if (and how many) updates took place to a given address space since the last read. The master device may also access the MaxUpdateAddrSinceLastRead” parameter and the “MinUpdateAddressSinceLastRead” parameter (Table 3) as part of the determination. Also, the master device (such as a CPP) may use the LastUpdate register (and UpdateList storage) of Table 3 to read an address as well as data and the time of a last update. Optionally, as an extra check, the master device may determine if the update occurred through the RUS or not and use the “time” parameter or other corresponding data to do precise synchronization using the update recovery and synchronization module.
Additional Example Specifications for Data Route Switch (DRS)
An RUS compatible DRS device may meet one or more of the following exemplary specification.
As a first exemplary specification, the DRS may allow push back (delay) of a search for up to MaxSrchPushBack clocks either after occurrence of “IDLE” on search or after MaxUpdateHold clocks of continuous search instructions are encountered (while update is pending). Alternatively or additionally, the DRS may allow another equivalent mechanism that ensures the UIM update instruction buffer does not overflow.
As a second exemplary specification, the DRS may implement the RUS update_in port, bus, or interface as part of the processor input/output circuitry 834. The DRS may implement the update_in port as physical interface (e.g., a dedicated IL/ILA or PCIe or Infiniband or Ethernet port). Alternatively or additionally, the DRS may implement the update_in part as a logical port (borrowed from or physically implemented via another hardware communication port) that can provide the required performance without dynamic involvement of software.
As a third exemplary specification, if the DRS is a master to a routing table, the DRS may implement a method or functionality to disable the local update of the routing table. This specification is to allow for a deterministic golden image of the routing table (master copy of the routing table) at a CPP, if required.
As a fourth exemplary specification, if the DRS also works as a master, the DRS may implement a snoop_out port, bus, or interface. If the DRS does not implement the snoop_out port, or some equivalent communication path, the responsibility of coherency of a CPP's image of a particular table and the DRS's image of the particular table may be performed by the CPP.
As a fifth exemplary specification, the DRS may maintain one or more “LastUpdate” registers. For example, the DRS may implement one such register on a per Address Space basis. Each register may have Update Attribute Bits (see LastUpdate register—Table 3). If the read data width does not fully cover these registers, the DRS may maintain a shadow copy of the upper bits of the LastUpdate registers. The DRS may fill the shadow register with an upper portion of the main (e.g., non-shadow) register when the lowest bits of the register are read. All the higher bits may be returned from shadow copy.
As a sixth exemplary specification, the DRS may maintain one or more “UpdatesSinceLastRead” registers (Table 3). For example, the DRS may implement one such register on a per Address Space basis.
As a seventh exemplary specification, the DRS may maintain a number of updated addresses according to the UpdateListDepth parameter value (Table 1), and the DRS may implement one such set of registers for each update address space. The DRS may implement two copies, current and shadow. In some implementations, the updates that happen after the lowest bits of LastUpdate register (Table 3) is read go to the current copy and the update that happened before (or at the clock) go to shadow copy (from where the actual data is read).
RUS Connectivity System (RCS) Example Specifications
A RUS may be implemented such that some or all of the RUS-related functionality is handled by the USM(s) and the UIM(s) in the RUS, which may allow for flexibility in intermediate connectivity systems linking the USMs and the UIMs. Various intermediate connectivity system may be used. Some optional guidelines that the RCS may meet are presented next:
As a first exemplary guideline, the performance of the RCS may match with the peak and average update traffic. For example, the performance of the RCS may meet or exceed the equivalent of the RUS Clock period (RCP) parameter (Table 1).
As a second exemplary guideline, the hardAPI packetization (manifested as packet size or burst length) supported by the RCS may be consistent with Update Group Size (UGS) requirements.
Update Performance
The disclosure herein, including the RUS, may increase routing table update performance. This section analyses the gain in the update performance. One possible maximum (guaranteed) performance is determined by the following parameters:
At the CPP end, the performance may be determined from parameters included in the RUS at least some of which are described in Table 1, such as: (CPPFrequency*CPPUpdateWordSize*CPPMaxUpdateCountLimit)/(CPPMaxUpdateCountLimit+CPPMinUpdateBreak)
For the RUS connectivity system, the update rate (without conflict) may be tied to the RUS frequency. For example, at a RUS Clock Period of 4 nanoseconds (ns), the RUS frequency can be about 250M updates/sec (each update in unit of RUS Word size that may be 64 bits).
At the DRS end, the update rate may be determined from parameters of the RUS system such as those discussed with reference to Table 1, such as: (DRSFrequency*DRSUpdateWordSize*(MaxSrchPushBack/(MaxSrchPushBack+MaxUpdateWait)).
Of the above equations and performance metrics, the RUS may measure performance as a metric that provides a minimum value, which may be the metric for the DRS. As one illustration, assume the following example values:
DRSFrequency=500 MHz
DRSUpdateWordSize=1 (same as RUS Word Size)
MaxSrchPushBack=MaxUpdateWait
For these assumptions, the value of update rate can be, for example, about 250M updates/sec. For comparison, the typical control plane update rate for a non-RUS system may range from 1K to 100K/sec. In other words, the RUS may increase update performance by many orders of magnitude. When it comes to such update rates, the extent to which a main path (search for example) can be pushed may more greatly affect performance than the capability of an update path (that includes CPP software) itself. Also, various designs may not allow access to the memory bus of the memory routing table, and the RUS may have to use intermediate bus controllers, which may control the effective throughput.
Update Speed Matching
The CPP may perform a string of routing table updates at very high frequency. The RUS may make these updates in the Table Holder device (such as a DRS), which is transparent to the CPP. The USM may implement a buffer to make such transparency possible. The USM may limit the buffering so as not to break (e.g., overflow) if the updates continue indefinitely. The RUS may implement one or both of the follow two methods to handle this situation:
Software method: RUS may include hardware executing software to limit the number of updates to a CPPMaxUpdateCountLimit parameter value. (Table 1) Once the limit is reached, the update software can wait for CPPMinUpdateBreak number of clocks (e.g., the clock that drives the USM) before resuming updates. (Table 1)
Hardware Method: The RUS may assume that the actual update rate will always be lower than what the RUS can handle based on, for example, an upper performance limit. Hardware in the RUS may use a flow control signal from the USM, the “usm_cpu_hold” signal. This flow control signal can be used to freeze the CPU. Accordingly, this method may be software-transparent as software never comes to know about such a situation. For debugging, the RUS may implement two counters that count the number of clocks while such a flow control freeze is applied. One counter can hold a predetermined minimum freeze duration threshold, and other counter can hold a predetermined maximum freeze duration. The RUS may also count or track the number of such freeze incidents.
Data Synchronization in RUS
RUS functionality may also extend to systems requiring or supporting portability of routing tables. The portability of routing tables may be used or implemented for redundancy or failure recovery, for example. The redundancy or failure could apply to one or both the CPP and the DRS ends.
When Master is CPP
In this scenario, the table within the CPP master is already the assumed-correct reference table. The only check needed is to access the LastUpdate register value from a Table Holder DRS to ensure that a last update expected has been performed. This step (of validating through LastUpdate register) may be a safety method and helps when there may be a buffer resident at the RCS that may take some time to flush.
Scenario Where the Master is Table Holder DRS
This scenario is dynamic and may not allow a deterministic point when a CPP's routing table image can be said to be fully correct or mirroring the routing table in the DRS (which can referred to as “golden”), unless the CPP disables the route update mechanism. In many cases, an image can be assumed correct (“golden”) until a precise point in time. The RUS may define the precision in terms of a clock that is used in a table update process by the table holder DRS. There may or may not be a better precision than the clock that is used to update the routing table itself. One possible methodology to update and maintain the assumed-correct (e.g., “golden”) image for a CPP when the DRS is the master is presented next.
Before reading the table, the CPP or DRS may read the LastUpdate register. (Table 1) This can include the time (in units of the clock of the Table Holder) when the last update took place to the table. For illustrative purposes, this time (in units of a clock of the table holder device) may be referred to as time T1. Also the CPP or DRS may read the UpdatesSinceLastRead register and the UpdateList storage. (Table 1) This will give the count of updates (such as C1), an address zone where update(s) took place (such as Z1), and specific addresses that were updated (such as L1 cache for example).
If C1 is the same or less than the UpdateListDepth parameter (Table 1), then the CPP or DRS may update the locations that are in the L1 cache, for example. Or else, the CPP or DRS may get the full dump of the table in the address zone Z1 (and the count of update may not help in this scenario).
The CPP or DRS may read the LastUpdate register again, at a subsequent time, which may be referred to as time is T2. Also, the CPP or DRS may read the UpdatesSinceLastRead register and UpdateList storage. UpdatesSinceLastRead can include all updates that happened between time T2 and time T1. In an example scenario, the UpdateCount as extracted from UpdatesSinceLastRead is C2, the address zone is Z2 and the update address list is L2.
Now the image that exists at CPP is what the DRS had at time T1 plus some of the C2 updates that happened in between. In other words, C2 entries spanning the address zone Z2, are under uncertainty. As one illustration, if a line card (holding the DRS) was to be replaced (due to potential failure or just for maintenance), an image of the routing table stored in the CPP could be used to create the routing table in the DRS. If for some reason, an exact image of the DRS routing table was desired, the CPP can disable a local update mechanism(s) of the routing table holder device and then perform the above described process. If the C2 is non-zero, the CPP or DRS may read the image in address zone Z2 again, and then repeat the steps until a subsequent count Cn is 0. If the CPP itself (or card component) is to be replaced, the image along with Tn, Cn, Zn, Ln info can be transferred to next the CPP, and that next CPP can continue from there.
The methods, devices, processing, modules, and logic described above may be implemented in many different ways and in many different combinations of hardware and software. For example, all or parts of the implementations may be circuitry that includes an instruction processor, such as a Central Processing Unit (CPU), microcontroller, or a microprocessor; an Application Specific Integrated Circuit (ASIC), Programmable Logic Device (PLD), or Field Programmable Gate Array (FPGA); or circuitry that includes discrete logic or other circuit components, including analog circuit components, digital circuit components or both; or any combination thereof. The circuitry may include discrete interconnected hardware components and/or may be combined on a single integrated circuit die, distributed among multiple integrated circuit dies, or implemented in a Multiple Chip Module (MCM) of multiple integrated circuit dies in a common package, as examples.
The circuitry may further include or access instructions for execution by the circuitry. The instructions may be stored in a tangible storage medium that is other than a transitory signal, such as a flash memory, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM); or on a magnetic or optical disc, such as a Compact Disc Read Only Memory (CDROM), Hard Disk Drive (HDD), or other magnetic or optical disk; or in or on another machine-readable medium. A product, such as a computer program product, may include a storage medium and instructions stored in or on the medium, and the instructions when executed by the circuitry in a device may cause the device to implement any of the processing described above or illustrated in the drawings.
The implementations may be distributed as circuitry among multiple system components, such as among multiple processors and memories, optionally including multiple distributed processing systems. Parameters, databases, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, and may be implemented in many different ways, including as data structures such as linked lists, hash tables, arrays, records, objects, or implicit storage mechanisms. Programs may be parts (e.g., subroutines) of a single program, separate programs, distributed across several memories and processors, or implemented in many different ways, such as in a library, such as a shared library (e.g., a Dynamic Link Library (DLL)). The DLL, for example, may store instructions that perform any of the processing described above or illustrated in the drawings, when executed by the circuitry.
Various implementations have been specifically described. However, many other implementations are also possible.
This application is a continuation of U.S. patent application Ser. No. 14/700,709, filed Apr. 30, 2015, which claims the benefit of priority to U.S. provisional patent application Ser. No. 61/986,804, filed Apr. 30, 2014. The entire contents of both of which are incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9986434 | Tripathi | May 2018 | B2 |
| 20120300780 | Hasan | Nov 2012 | A1 |
| 20140040321 | Cohen | Feb 2014 | A1 |
| 20140044135 | Sankaralingam | Feb 2014 | A1 |
| 20140086240 | Assarpour | Mar 2014 | A1 |
| 20140245262 | Hill | Aug 2014 | A1 |
| 20140281508 | Akhter | Sep 2014 | A1 |
| 20140282968 | Aboughanaima | Sep 2014 | A1 |
| 20150121476 | Zheng | Apr 2015 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20180249335 A1 | Aug 2018 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61986804 | Apr 2014 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14700709 | Apr 2015 | US |
| Child | 15962521 | US |
