The present invention relates generally to computer networks, and particularly to routing data packets between subnets.
Routers are commonly used to forward data packets among subnets of a Layer-3 network. (A subnet, short for subnetwork, is a logical subdivision of a Layer-3 network.) Network ports of nodes within a given subnet share the same Layer-3 network address prefix. For example, in Internet Protocol (IP) networks, the ports in each subnet share the same most-significant bit-group in their IP address, so that the IP address is logically divided into two fields: a network or routing prefix, and the rest field or host identifier. Similarly, in InfiniBand™ (IB) networks, each subnet is uniquely identified with a subnet identifier known as the Subnet Prefix. For each port in the subnet, this prefix is combined with a respective Globally-Unique Identifier (GUID) to give the IB Layer-3 address of the port, known as the Global Identifier (GID).
Typically, the logical subdivision of a Layer-3 network into subnets reflects the underlying physical division of the network into Layer-2 local area networks. Routers forward packets among subnets on the basis of their Layer-3 (IP or GID) destination addresses, while within a given subnet packets are forwarded among ports by Layer-2 switches or bridges. These Layer-2 devices operate in accordance with the applicable Layer-2 protocol and forward packets within the subnet according to the Layer-2 destination address, such as the Ethernet™ medium access control (MAC) address or the IB link-layer Local Identifier (LID). In general, Layer-2 addresses in a given subnet are recognized only within that subnet, and routers will swap the Layer-2 address information of packets that they forward from one subnet to another.
Routers in most existing networks—particularly IP networks—use the Address Resolution Protocol (ARP) to resolve Layer-3 addresses of packets that they receive into Layer-2 addresses that can be used for local forwarding within the destination subnet. ARP was first defined by Plummer in “An Ethernet Address Resolution Protocol,” published as Request for Comments (RFC) 826 of the Internet Engineer Task Force (IETF), in 1982. ARP is a request/reply protocol that specifies a packet structure that can be used by nodes in a network to query and inform one another as to the Ethernet address that corresponds to any given IP address. Routers use ARP to build lookup tables of the correspondence between IP and Layer-2 addresses, which they can then use in packet forwarding.
In some multicast applications, MAC addresses may be defined on the basis of corresponding IP multicast addresses. A scheme of this sort is defined, for example, by Hinden et al., in “IP Version 6 Addressing Architecture,” published as IETF RFC 2373 (1998). In this scheme, IPv6 multicast addresses are mapped into Ethernet MAC addresses by taking the low-order 32 bits of the IPv6 multicast address and using them to create a MAC multicast address. Thus, new IPv6 multicast addresses are assigned so that the group identifier is always in the low-order 32 bits.
Embodiments of the present invention that are described hereinbelow provide improved methods and systems for packet routing and forwarding in and among subnets.
There is therefore provided, in accordance with an embodiment of the invention, a method for communication, which includes routing unicast data packets among nodes in a network using respective Layer-3 addresses that are uniquely assigned to each of the nodes. Respective Layer-2 unicast addresses are assigned to the nodes in accordance with an algorithmic mapping of the respective Layer-3 addresses, and the unicast data packets are forwarded within subnets of the network using the assigned Layer-2 addresses.
Typically, the Layer-3 addresses each include a subnet prefix and a respective globally-unique identifier (GUID), and assigning the respective Layer-2 unicast addresses includes mapping the GUID to a local identifier (LID). Additionally or alternatively, assigning the respective Layer-2 unicast addresses includes mapping the Layer-3 addresses to unicast medium access control (MAC) addresses.
In a disclosed embodiment, forwarding the unicast data packets includes selecting respective ports of a switch through which the unicast data packets are to be transmitted to the nodes by applying the algorithmic mapping to the Layer-3 addresses.
In one embodiment, assigning the respective Layer-2 unicast addresses includes mapping a first field within the respective Layer-3 addresses to generate the Layer-2 unicast addresses, and forwarding the unicast data packets includes assigning the unicast data packets to respective traffic classes within the subnets by mapping respective values of a second field within the Layer-3 addresses to the traffic classes.
The method may also include routing multicast data packets among the nodes in the network using a Layer-3 multicast address, and forwarding the multicast data packets within the subnets using a common Layer-2 multicast address that is derived by algorithmic mapping from the Layer-3 multicast address.
There is also provided, in accordance with an embodiment of the invention, a method for communication, which includes receiving in a router a multicast data packet, having a Layer-3 multicast address, for transmission to nodes in multiple subnets of a Layer-3 network using a common Layer-2 multicast address corresponding to the Layer-3 address in all of the subnets. A single lookup operation is performed in the router, based on the Layer-2 or Layer-3 multicast address, in order to identify output ports in two or more of the subnets through which the multicast data packet is to be forwarded by the router. The multicast data packet is forwarded from the router to the two or more of the subnets via the identified output ports using the common Layer-2 address.
Typically, the common Layer-2 multicast address is derived by an algorithmic mapping from the Layer-3 multicast address.
The output ports identified in the single lookup operation may include both a first port in a source subnet of the multicast data packet and a second port in another of the subnets, other than the source subnet, to which the router is connected for forwarding of the multicast data packet.
In a disclosed embodiment, forwarding the multicast data packet includes deriving respective subnet mask vectors for the subnets, and selecting the output ports of the router through which the multicast data packet is to be transmitted using the subnet mask vectors.
The method may also include, upon receiving the forwarded multicast data packet at a host in the network, filtering the data packet in a network interface controller of the host according to the Layer-3 multicast address.
There is additionally provided, in accordance with an embodiment of the invention, a system for communication, including a plurality of routers, which are configured to route unicast data packets among nodes in a network using respective Layer-3 addresses that are uniquely assigned to each of the nodes. A management processor is configured to assign respective Layer-2 unicast addresses to the nodes in accordance with an algorithmic mapping of the respective Layer-3 addresses, whereby the unicast data packets are forwarded within subnets of the network using the assigned Layer-2 addresses.
There is further provided, in accordance with an embodiment of the invention, apparatus for communication, including a plurality of ports, which are configured to receive from a Layer-3 network a multicast data packet, having a Layer-3 multicast address, for transmission to nodes in multiple subnets of the Layer-3 network using a common Layer-2 multicast address corresponding to the Layer-3 address in all of the subnets. Routing logic is configured to perform a single lookup operation based on the Layer-2 or Layer-3 multicast address in order to identify the ports in two or more of the subnets through which the multicast data packet is to be forwarded and to forward the multicast data packet via the identified ports to the two or more of the subnets using the common Layer-2 address.
The present invention will be more fully understood from the following detailed description of the embodiments thereof, taken together with the drawings in which:
Routing and forwarding of data packets among subnets by routers that are known in the art typically require multiple lookup operations, both to ascertain the egress port through which each packet should be transmitted and to translate the Layer-3 address of the packet into the appropriate Layer-2 address for forwarding in the destination subnet. These lookup operations consume processing resources and chip area and increase the latency of packet transfer through the router.
Embodiments of the present invention that are described hereinbelow use algorithmic mapping in order to reduce the computational load and latency that are associated with packet routing. In some of these embodiments, unicast data packets are routed among nodes in a network using respective Layer-3 addresses that are uniquely assigned to each of the nodes, while respective Layer-2 unicast addresses are assigned to the nodes by algorithmic mapping of the respective Layer-3 addresses. In other words, the Layer-2 addresses are not arbitrarily allocated or fixed in advance (as is usually the case in Ethernet networks, for example), but are rather assigned values that can be derived by applying a predefined mathematical algorithm to the Layer-3 addresses. A router can thus determine the Layer-2 destination address to use for each incoming Layer-3 unicast packet by simply applying the appropriate algorithm, without ARP queries or lookup, and can then transmit these packets via the appropriate egress ports for forwarding within subnets of the network using these Layer-2 addresses.
Additionally or alternatively, algorithmic mapping may be applied to Layer-3 multicast addresses and may be configured so that the same Layer-2 multicast address is commonly used in forwarding a given multicast data packet to multiple different subnets (or all subnets). In such implementations, the router looks up the common Layer-2 multicast address only once, performing a single lookup to identify all of the multiple subnets to which the multicast data packet is to be transmitted. No Layer-3 lookup is required at all in this case. The Layer-2 multicast address is used to look up a respective subnet mask vector for each subnet, and the router selects the egress ports through which the multicast data packet is to be transmitted using these subnet mask vectors.
Alternatively, the principles of the present invention may similarly be applied in other networks in which Layer-2 addresses can be assigned at will. Conventional Ethernet networks do not generally satisfy this criterion, since MAC addresses are typically fixed in hardware. On the other hand, in virtualized and other tunneled IP and Ethernet networks (such as “MAC-in-MAC” provider backbone bridges, as defined by the IEEE 802.1ah standard), MAC addresses may be assigned and applied as provided by embodiments of the present invention.
Network 20 comprises multiple subnets 22 (labeled subnets A, B and C), which are interconnected by Layer-3 routers 24 (labeled R0, R1 and R2). Each subnet 22 comprises multiple Layer-2 switches 26, which connect to hosts 28 via suitable network interface controllers (NICs) 34, which are referred to in IB parlance as host channel adapters. Switches 26 within each subnet may be interconnected in any suitable topology, such as a “fat tree” topology. Certain of the switches (for example, spine switches in the case of a fat tree) connect to routers 24 and thus enable packet transfer between subnets. In the pictured implementation, any given pair of subnets 22 is separated by no more than a single routing hop, but the principles of the present invention may also be extended to networks in which traffic between certain subnets must traverse two or more routers in sequence. Typically (although not necessarily), each pair of subnets 22 is connected by two or more routers 24, for purposes of load balancing and failure protection.
A subnet manager (SM) 30 in each subnet 22 performs management and administration functions defined by the above-mentioned IB specification. (Optionally, more than one subnet manager may exist in a given subnet to provide backup in case of failure, but typically only a single subnet manager is active in performing these functions at any given time.) Specifically, SM 30 in each subnet assigns a Layer-2 address, in the form of a LID, to each physical port of each host 28 within the given subnet. A subnet administration (SA) function provides nodes with information gathered by the SM, including communication of the LID information to a Subnet Management Agent (SMA) in each node of the subnet. For simplicity and clarity in the description that follows, however, all of these subnet management and administration functions will be assumed to be carried out by the SM. Layer-2 switches 26 within each subnet 22 are configured by SM 30 to forward packets among the ports on the basis of the destination LID (D-LID) in the packet header.
SM 30 is typically a combined hardware/software element, comprising a computing platform, such as an embedded or stand-alone central processing unit (CPU) with a memory and suitable interfaces, which runs management software that performs the functions described herein. This sort of hardware/software entity is also referred to, in the context of the present description and in the claims, as a management processor. The computing platform may be dedicated to subnet management functions, or it may alternatively be integrated with one of the nodes in the subnet, such as a host computer, switch or appliance, and possibly shared with other computing and communication functions. The software components of the SM may be downloaded to the computing platform in electronic form, for example over network 20 or via a separate control network (not shown). Alternatively or additionally, these software components may be stored on tangible, non-transitory computer-readable media, such as in optical, magnetic, or electronic memory.
SM 30 in each subnet 22 assigns a Layer-2 address, in the form of a LID, to each port of each switch 26 and host 28 within the subnet. Each port also receives a GID Layer-3 address (which may also be assigned by SM 30), wherein all ports in a given subnet have the same GID prefix, as explained above, but each port in the subnet has a different GUID. Each SM 30 assigns the LIDs in its subnet so that they correspond algorithmically to the respective GIDs, as explained in greater detail hereinbelow. In addition, a distributed name server (DNS) 32, at a predefined address in network 20, may collect and store names and/or numbers besides the LID and GID, such as a domain name, IP address or MAC address, that identify nodes and/or processes in the network. DNS 32 may provide information on the correspondence between these names and/or numbers and GID values in response to queries from nodes in the network.
In the present embodiment, router 24 comprises a number of Layer-2 switches 40, each serving a respective subnet via local ports 46 and capable of forwarding data packets among these ports on the basis of the DLID carried in the packet header. For inter-subnet traffic, switches 40 are interconnected by routing logic 42 via respective routing ports 44. Inter-subnet data packets are transmitted by nodes in each subnet to the LID that is assigned (by the subnet manager) to routing port 44 of the respective switch 40.
Routing logic 42 receives these packets and examines the DGID of each packet in order to determine the subnet (and hence the switch 40) or subnets (and switches) to which it is to be routed. For unicast packets, the routing logic identifies the destination subnet and switch according to the GID prefix, while computing the DLID by algorithmic mapping of the DGID and thus identifying the proper egress port 46. For multicast packets, the multicast DLID and DGID do not change, and routing logic 42 simply identifies the one or more egress switches and ports corresponding to the multicast DLID. These functions are described in greater detail hereinbelow.
Packet 50 comprises a Layer-2 header 52, referred to as a Local Routing Header (LRH), followed by a Layer-3 header 54, referred to as a Global Routing Header (GRH), followed by transport headers (not shown), a payload 56, and an error detection field 58, which typically contains a cyclic redundancy code (CRC). The Layer-2 destination address, or DLID, is contained in a DLID field 60 in header 52, while the Layer-3 destination address, or DGID, is contained in a DGID field 62 in header 54. For unicast packets, such as the packet shown in
As illustrated by the dashed arrow in
Alternatively, other algorithmic mapping functions may be applied in mapping the GUID of each port in a subnet to the LID. The term “algorithmic mapping function” means that the function can be computed mathematically by an appropriate computing entity (such as routing logic 42) without resort to a lookup table. Any suitable function can be used for this purpose, including arithmetic, algebraic, and Boolean functions, as long as it is chosen so as to create a one-to-one mapping between GUIDs and LIDs in the subnet.
In some networks, Layer-2 packets are assigned to different traffic classes, which are referred to in IB networks as service levels (SLs). The SL is identified by four bits in Layer-2 header 52 (not marked in
Although the example shown in
In the table above, each row corresponds to a particular multicast LID, and each column corresponds to one of ports 46. The entries “1” and “0” indicate whether or not packets with the corresponding multicast LID are to be forwarded to the corresponding port. This single table covers all of the subnets to which router 24 is connected, including the source subnet of any given multicast packet.
The method of
If the result of step 74 is positive, on the other hand, switch 40 forwards the packet to routing logic 42. The routing logic checks Layer-3 header 54 and identifies the destination subnet of the packet based on GID prefix 64. The routing logic thus selects the routing port 44 through which the packet will be passed to the corresponding egress switch 40, at a port lookup step 77. Routing logic 42 also applies algorithmic mapping, as described above, to convert the DGID value (specifically GUID 66) of the packet to the appropriate DLID, at an address mapping step 78. This mapping is carried out once for each packet, by routing logic 42, regardless of whether the packet is multicast or unicast.
Routing logic 42 passes each unicast packet via the selected port 44 to the appropriate switch 40, which then chooses the appropriate local port 46 for egress of the packet, at a Layer 2 lookup step 80. The choice of egress port is based on the DLID assigned at step 78 and will typically have different values in each subnet to which a given packet is transmitted. Switch 40 inserts the DLID in the appropriate field 60 of the outgoing packet, and also updates the source LID (SLID), at a packet transmission step 82. The traffic class (service level) in the Layer-2 header of the outgoing packet may also be updated by mapping of GUID 66, as explained above, and the TTL value in the packet header is decremented. The switch then transmits the packet into the destination subnet 22.
If the packet is found at step 72 to contain a multicast DGID, on the other hand, the switch 40 receiving the packet looks up the Layer 2 ports 46 through which the packet is to be forwarded, at a multicast lookup step 90. The lookup is typically referred to a multicast forwarding table of the general form that is illustrated above in Table 1. As each multicast GID is mapped to the same multicast LID over the entire network 20, a single lookup of this sort is sufficient to cover all subnets 22 to which any given multicast packet is to be forwarded.
Each switch 40 applies a subnet mask, in the form of a vector that indicates the egress ports 46 to use for each multicast DLID, at a masking step 92. When the ingress switch forwards a multicast subnet within its own subnet, the packet is transmitted through the applicable ports 46 without modification, at step 76, as described above. On the other hand, when a switch forwards a multicast packet that originated from another subnet, the Layer 2 header and the TTL value are appropriately updated, at a packet transmission step 94.
As Layer-3 multicast addresses are global, the mapping applied at step 90 may result in two or more multicast DGIDs being mapped to the same multicast DLID. Consequently, a given host 28 may receive multicast packets that are destined for a multicast group to which the host does not belong. To avoid confusion in this sort of situation, NIC 34 may be configured to filter incoming multicast packets according to the DGID, and discard multicast packets whose GID is not in a list of multicast groups for which the host has registered.
It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.
This application is a continuation of U.S. patent application Ser. No. 14/046,976, filed Oct. 6, 2013.
Number | Name | Date | Kind |
---|---|---|---|
6115385 | Vig | Sep 2000 | A |
6614758 | Wong | Sep 2003 | B2 |
6665297 | Harigochi et al. | Dec 2003 | B1 |
6795886 | Nguyen | Sep 2004 | B1 |
6807175 | Jennings | Oct 2004 | B1 |
6912589 | Jain | Jun 2005 | B1 |
7274869 | Pan et al. | Sep 2007 | B1 |
7899930 | Turner | Mar 2011 | B1 |
7924837 | Shabtay | Apr 2011 | B1 |
8489718 | Brar et al. | Jul 2013 | B1 |
8495194 | Brar et al. | Jul 2013 | B1 |
8621111 | Marr et al. | Dec 2013 | B2 |
8625427 | Terry et al. | Jan 2014 | B1 |
8681641 | Sajassi | Mar 2014 | B1 |
8755389 | Poutievski et al. | Jun 2014 | B1 |
8873567 | Mandal et al. | Oct 2014 | B1 |
9042234 | Liljenstolpe | May 2015 | B1 |
9571400 | Mandal et al. | Feb 2017 | B1 |
20010043614 | Viswanadham | Nov 2001 | A1 |
20020009073 | Furukawa | Jan 2002 | A1 |
20020026525 | Armitage | Feb 2002 | A1 |
20020071439 | Reeves et al. | Jun 2002 | A1 |
20020085586 | Tzeng | Jul 2002 | A1 |
20020141412 | Wong | Oct 2002 | A1 |
20020176363 | Durinovic-Johri et al. | Nov 2002 | A1 |
20030016624 | Bare | Jan 2003 | A1 |
20030097438 | Bearden | May 2003 | A1 |
20040024903 | Costatino | Feb 2004 | A1 |
20040062242 | Wadia | Apr 2004 | A1 |
20050157641 | Roy | Jul 2005 | A1 |
20060143300 | See et al. | Jun 2006 | A1 |
20060215645 | Kangyu | Sep 2006 | A1 |
20070030817 | Arunachalam | Feb 2007 | A1 |
20070091911 | Watanabe et al. | Apr 2007 | A1 |
20070104192 | Yoon | May 2007 | A1 |
20070183418 | Riddoch | Aug 2007 | A1 |
20080101378 | Krueger | May 2008 | A1 |
20080112413 | Pong | May 2008 | A1 |
20080186981 | Seto et al. | Aug 2008 | A1 |
20080267078 | Farinacci et al. | Oct 2008 | A1 |
20080298248 | Roeck et al. | Dec 2008 | A1 |
20090010159 | Brownell et al. | Jan 2009 | A1 |
20090022154 | Kiribe | Jan 2009 | A1 |
20090097496 | Nakamura | Apr 2009 | A1 |
20090262741 | Jungck | Oct 2009 | A1 |
20100020796 | Park | Jan 2010 | A1 |
20100049942 | Kim et al. | Feb 2010 | A1 |
20100111529 | Zeng | May 2010 | A1 |
20100141428 | Mildenberger et al. | Jun 2010 | A1 |
20100216444 | Mariniello et al. | Aug 2010 | A1 |
20100290385 | Ankaiah et al. | Nov 2010 | A1 |
20100290458 | Assarpour et al. | Nov 2010 | A1 |
20110080913 | Liu | Apr 2011 | A1 |
20110085440 | Owens et al. | Apr 2011 | A1 |
20110090784 | Gan | Apr 2011 | A1 |
20110164518 | Daraiseh et al. | Jul 2011 | A1 |
20110225391 | Burroughs et al. | Sep 2011 | A1 |
20110255410 | Yamen et al. | Oct 2011 | A1 |
20120075999 | Ko et al. | Mar 2012 | A1 |
20120147752 | Ashwood-Smith et al. | Jun 2012 | A1 |
20120163797 | Wang | Jun 2012 | A1 |
20120170582 | Abts et al. | Jul 2012 | A1 |
20120207175 | Raman et al. | Aug 2012 | A1 |
20120287791 | Xi et al. | Nov 2012 | A1 |
20130044636 | Koponen et al. | Feb 2013 | A1 |
20130083701 | Tomic et al. | Apr 2013 | A1 |
20130114619 | Wakumoto | May 2013 | A1 |
20130159548 | Vasseur et al. | Jun 2013 | A1 |
20130204933 | Cardona | Aug 2013 | A1 |
20130208720 | Ellis et al. | Aug 2013 | A1 |
20130297757 | Han | Nov 2013 | A1 |
20130329727 | Rajagopalan | Dec 2013 | A1 |
20130336164 | Yang et al. | Dec 2013 | A1 |
20140016457 | Enyedi et al. | Jan 2014 | A1 |
20140022942 | Han | Jan 2014 | A1 |
20140043959 | Owens et al. | Feb 2014 | A1 |
20140105034 | Sun | Apr 2014 | A1 |
20140140341 | Bataineh et al. | May 2014 | A1 |
20140169173 | Naouri et al. | Jun 2014 | A1 |
20140198636 | Thayalan et al. | Jul 2014 | A1 |
20140269305 | Nguyen | Sep 2014 | A1 |
20140313880 | Lu et al. | Oct 2014 | A1 |
20140328180 | Kim et al. | Nov 2014 | A1 |
20150092539 | Sivabalan et al. | Apr 2015 | A1 |
20150124815 | Beliveau et al. | May 2015 | A1 |
20150127797 | Attar et al. | May 2015 | A1 |
20150131663 | Brar et al. | May 2015 | A1 |
20150172070 | Csaszar | Jun 2015 | A1 |
20150194215 | Douglas et al. | Jul 2015 | A1 |
20150249590 | Gusat et al. | Sep 2015 | A1 |
20160043933 | Gopalarathnam | Feb 2016 | A1 |
20160080120 | Unger et al. | Mar 2016 | A1 |
20160080321 | Pan et al. | Mar 2016 | A1 |
20160182378 | Basavaraja et al. | Jun 2016 | A1 |
20170054591 | Hyoudou et al. | Feb 2017 | A1 |
20180139132 | Edsall et al. | May 2018 | A1 |
Number | Date | Country |
---|---|---|
2016105446 | Jun 2016 | WO |
Entry |
---|
U.S. Appl. No. 14/745,488 Office Action dated Dec. 6, 2016. |
U.S. Appl. No. 14/337,334 Office Action dated Oct. 20, 2016. |
Dally et al., “Deadlock-Free Message Routing in Multiprocessor Interconnection Networks”, IEEE Transactions on Computers, vol. C-36, No. 5, May 1987, pp. 547-553. |
Prisacari et al., “Performance implications of remote-only load balancing under adversarial traffic in Dragonflies”, Proceedings of the 8th International Workshop on Interconnection Network Architecture: On-Chip, Multi-Chip, 4 pages, Jan. 22, 2014. |
Garcia et al., “On-the-Fly 10 Adaptive Routing in High-Radix Hierarchical Networks,” Proceedings of the 2012 International Conference on Parallel Processing (ICPP), pp. 279-288, Sep. 10-13, 2012. |
U.S. Appl. No. 14/970,608 Office Action dated May 30, 2017. |
U.S. Appl. No. 14/673,892 Office Action dated Jun. 1, 2017. |
U.S. Appl. No. 14/970,608 office action dated Nov. 1, 2017. |
U.S. Appl. No. 15/152,077 office action dated Dec. 1, 2017. |
U.S. Appl. No. 15/050,480 office action dated Jan. 22, 2018. |
U.S. Appl. No. 15/387,718 office action dated Mar. 9, 2018. |
Zahavi et al., “Distributed Adaptive Routing for Big-Data Applications Running on Data Center Networks,” Proceedings of the Eighth ACM/IEEE Symposium on Architectures for Networking and Communication Systems, New York, USA, pp. 99-110, Oct. 29-30, 2012. |
U.S. Appl. No. 14/732,853 Office Action dated Jan. 26, 2017. |
U.S. Appl. No. 15/050,480 office action dated Nov. 2, 2018. |
U.S. Appl. No. 15/152,077 office action dated Jul. 16, 2018. |
U.S. Appl. No. 15/050,480 office action dated Jun. 28, 2019. |
U.S. Appl. No. 15/896,088 office action dated Jun. 12, 2019. |
U.S. Appl. No. 15/218,028 office action dated Jun. 26, 2019. |
U.S. Appl. No. 15/050,480 office action dated Apr. 9, 2020. |
Number | Date | Country | |
---|---|---|---|
20170070474 A1 | Mar 2017 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 14046976 | Oct 2013 | US |
Child | 15356588 | US |