1. Field of the Invention
The present invention relates data transfer between servers and remote entities including remote peripherals. In one example, the present invention relates to methods and apparatus for efficiently transferring data between servers and remote peripherals such as network interface cards (NICs), host bus adapters (HBAs), and cryptographic accelerators.
2. Description of Related Art
A server or computing system generally includes one or more processors, memory, and peripheral components and peripheral interfaces. Examples of peripheral components include cryptographic accelerators, graphics accelerators, and eXtensible Markup Language (XML) accelerators. Examples of peripheral interfaces include network interface cards (NIC), serial ATA (SATA) adapters, and host bus adapters (HBA). Processors, memory, and peripherals are often connected using one or more buses and bus bridges. To provide fault-tolerance, individual servers are often configured with redundant resources.
Because resources such as peripheral components and peripheral interfaces are assigned on a per server or a per processor basis, other servers do not typically have access to these resources. In order to provide adequate resources for each server, resources are typically over-provisioned. For example, more hardware acceleration is provided than is typically needed. More network interface bandwidth is allocated than is typically used simply to handle worst-case or expected worst-case scenarios. Resources are over-provisioned resulting in overall waste and low utilization. Resource assignment on a per server or a per processor basis also limits the ability to reconstruct or reconfigure a resource environment.
A variety of peripherals are sometimes provided remotely over a network. However, protocols for communications between servers and remote peripherals are often inefficient, inadequate, or even unavailable. Consequently, the techniques and mechanisms of the present invention allow efficient data transfer between servers and remote peripherals over a networking technology.
Methods and apparatus are provided for transferring data between servers and a remote entity having multiple peripherals. Multiple servers are connected to a remote entity over an Remote Direct Memory Access capable network. The remote entity includes peripherals such as network interface cards (NICs) and host bus adapters (HBAs). Server descriptor rings and descriptors are provided to allow efficient and effective communication between the servers and the remote entity.
In one embodiment a technique for transferring data from a server to a remote entity is provided. A write index is sent to virtualization logic associated with the remote entity. The remote entity includes multiple peripheral interfaces. The write index is provided from the server to virtualization logic over an RDMA capable networking technology. One or more server descriptor entries are provided to virtualization logic. An RDMA read data buffer request is received from virtualization logic. Data is transferred in response to the RDMA read data buffer request.
In another example, a technique for transferring data from a remote peripheral to a server is provided. A write index is sent to virtualization logic associated with a remote entity including multiple peripherals. The write index is provided from the server to virtualization logic over an RDMA capable networking technology. A server descriptor entry referenced by the write index is provided to virtualization logic. An RDMA write response buffer is received from virtualization logic. A completion message is received from virtualization logic.
A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings.
The invention may best be understood by reference to the following description taken in conjunction with the accompanying drawings, which are illustrative of specific embodiments of the present invention.
Reference will now be made in detail to some specific examples of the invention including the best modes contemplated by the inventors for carrying out the invention. Examples of these specific embodiments are illustrated in the accompanying drawings. While the invention is described in conjunction with these specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. On the contrary, it is intended to cover alternatives, modifications, and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims.
For example, the techniques of the present invention will be described in the context of RDMA and Infiniband as one example of an RDMA capable network. However, it should be noted that the techniques of the present invention can be applied to a variety of different standards and variations to RDMA and Infiniband. Similarly, a server is described throughout. However, a server can be a single processing system, a multiple processor system, a guest operating system, a system image, or a virtual machine. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. The present invention may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail in order not to unnecessarily obscure the present invention.
Furthermore, techniques and mechanisms of the present invention will sometimes be described in singular form for clarity. However, it should be noted that some embodiments include multiple iterations of a technique or multiple instantiations of a mechanism unless noted otherwise. For example, a processor is used in a variety of contexts. However, it will be appreciated that multiple processors can also be used while remaining within the scope of the present invention unless otherwise noted.
Servers having processors and memory typically have access to physical peripherals, including peripheral components and peripheral interfaces. However, physically tying resources to particular servers can lead to a number of drawbacks, including management and usage inefficiencies. In some examples, peripherals are moved from individual servers onto a remote device. In some embodiments, resources such as hardware accelerators, NICs, and HBAs are removed from individual servers and aggregated at a remote entity. The remote entity can be configured to include multiple peripherals and virtualization logic allowing server access to the peripherals.
In some examples, the remote entity is an appliance or switch that is connected to multiple servers over a network. However, offloading resources onto a remote entity requires that the servers be able to access the resources in an efficient manner. Typical networks do not allow efficient access to the resources in a manner providing low-latency, high-throughput, processor efficiency, support for quality of service (QoS). Consequently, the techniques and mechanisms of the present invention contemplate providing efficient access to remote peripherals including remote storage adapters, remote network adapters, and remote hardware accelerators. In one example, the remote entity receives requests from individual servers over an remote direct memory access (RDMA) capable network such as Infiniband and provides resources to handle individual requests.
Examples of RDMA capable networks include Infiniband and RDMA over Ethernet. The RDMA capable network provides a low latency, high bandwidth connection between the multiple servers and the resources in the remote entity. The technique takes into account the fact that the RDMA-capable network adapter within each server may have different performance characteristics than the specialized device that resides on the I/O entity. According to various embodiments, a higher processing burden is placed on the target device within the remote entity rather than on the RDMA network adapter within each server. This improves performance and reduces the overhead imposed on the computer and its RDMA network adapter.
According to various embodiments, an RDMA network adapter on a server has a large set of capabilities, but is limited because of the need to support a variety of protocols. The RDMA adapters on the remote entity, however, can be configured to be much more specialized. Each remote entity RDMA adapter does not need to support all host functionality. In some examples, each remote entity RDMA adapter only needs to support a particular data transfer protocol. However, the remote entity RDMA adapter needs to support high bandwidth and low latency, as a remote entity RDMA adapter may be configured to support many server adapters at the same time. According to various embodiments, the techniques of the present invention place processing burden on the remote entity adapter and not on the server. RDMAs are issued by the remote entity adapter and not by the server adapter.
Various embodiments of the present invention provide a unified infrastructure for supporting storage, network, and other types of remote input/output (I/O) traffic. Message coalescing is used to improve performance by reducing the number of messages. Support for QoS is maintained and the techniques are suitable for either hardware or software implementation. According to various embodiments, the remote entity allows dynamic reconfiguration and remapping of virtual resources to servers to support true stateless computing and utility computing and allows charge back by application, user, departments, etc. Storage resources including local disks can be shared and virtualized to allow stateless computing.
Requests from multiple servers may also be forwarded to the same NIC. An administrator can provision and partition resources including server resources, peripheral component resources, and peripheral interface resources at the remote entity based on particular needs and requirements. According to various embodiments, a remote entity can also perform packet inspection or deep packet inspection to classify server traffic. Traffic policies can be applied to allow application level quality of service. Quality of service (QOS) and traffic engineering schemes can be implemented. Traffic associated with particular devices or servers can be given priority or guaranteed bandwidth. The total amount of resources can be decreased while increasing resource utilization. The remote entity can be introduced into existing server racks and farms with little disruption to system operation.
The various NICs 107, 119, and 129 are also associated with IP addresses and media access control (MAC) addresses. Each server and associated NIC encapsulates data into IP packets for transmission to a network router 141. Encapsulation may involve adding appropriate Telnet Control Protocol (TCP) and IP headers and addresses. Each NIC is also configured to remove TCP/IP headers and addresses and provide data to an associated processor over a system bus when IP packets are received from an IP network. An HBA 127 may be configured to perform fabric login (FLOGI) and port login (PLOGI) processes associated with operating in a fibre channel storage area network. Each server 101, 111, and 121 may be individual computers, UNIX based low-end to high-end servers, mainframes, or blade servers. Any system including a processor and memory and an address space is referred to herein as a server. A server may include components noted above as well as a variety of other components.
In some examples, a series of servers each include a processor and a number of resources such as HBAs, NICs, iSCSI TOE, SATA as well as other peripherals interfaces and peripheral components. Cables and lines are then connected to each server to allow access to Ethernet switches, Fibre Channel switches, appliances, and mainframe or KVM switches. In some examples, redundant cables are used to connect each server to each switch. In an example with 40 servers, 80 cables would be used to connect the servers to an Ethernet switch. An additional 80 cables would be used to connect the servers to Fibre Channel switch, etc. The resource allocation and system management inefficiencies are magnified by the physical complexities of routing the individual cables.
Virtualization logic 241 may be a standalone entity, integrated within a particular server, or provided with a remote entity 243. According to various embodiments, components such as HBA 253, NIC 255, a cryptographic accelerator 257, an XML accelerator 259, Internet SCSI (iSCSI)/TCP Offload Engine (TOE)/Serial ATA (SATA) 251, can be offloaded from servers 201, 211, and 221 onto a remote entity 243. The resources including HBA 253, NIC 255, cryptographic accelerator 257, XML accelerator 259, and iSCSI/TOE/SATA 251 are maintained in a shared and virtualized manner on a remote entity 243. Links are provided between the remote entity and external switches/routers such as an IP network switch/router.
According to various embodiments, a series of servers is connected to the remote entity using an Infiniband network. Infiniband allows interconnection of processing nodes and I/O nodes. The Infiniband stack includes support of multiple queues including send and receive queues. In one example, the queues used are descriptor rings. Descriptors are used to hold instructions, commands, and data addresses. For example, descriptors hold instructions to transmit data and receive data.
By using a remote entity, the number of resources and links can be significantly reduced while increasing allocation efficiency. Resources can be accessed at the driver level. Virtual resources including service engines can be dynamically bound to servers including system images, guest operating systems, and virtual machines. In one instance, a cryptographic accelerator is bound to a particular system image.
In this example, to perform cryptographic operations associated with sending or receiving data, a server 201 passes data to the remote entity cryptographic accelerator 257 as though the remote entity cryptographic accelerator 257 were included in the server 201. To perform XML operations, a server 201 passes data to the XML accelerator 259 as though the XML accelerator 259 is included in the server 201. Alternatively, XML and cryptographic operations can be performed as data is received at a remote entity from a network. Consequently, additional server 201 processor involvement is not needed.
Similarly, descriptor ring 313 has write index 341 and read index 343 and descriptor ring 323 has read index 353 and write index 351. In one embodiment, the server descriptor rings 303, 313, and 323 are associated with different servers. Each server descriptor ring may also have an associated ring size value and a base address value.
The virtualization logic 403 sends an optional RDMA write to provide a read index 417 to the server 401. The virtualization logic 403 processes the descriptor information and sends an RDMA read data buffer request 421 to the server 401. One or more address ranges may be read. The server provides the data 423 to the virtualization logic 403. When the data has been received the virtualization logic 403 responds with a completion message 425.
The virtualization logic 503 sends an optional RDMA write to provide a read index 517 to the server 501. The virtualization logic 503 processes the descriptor information and sends an RDMA write to the server with write response data 521. According to various embodiments, data is written directly into server memory at addresses specified by descriptor entries. One or more address ranges may be written. The virtualization logic 503 sends a completion message 523 when the data transfer is complete.
The platform 911 is associated with memory 919 and a processor subsystem 913, a power subsystem 915, and a storage subsystem 917. In some embodiments, the platform 911 includes tables with information mapping various servers connected through the fabric interface 921 and various port adapter resources and network interfaces. The processor subsystem 913 is configured to manage port adapter resources as though the port adapters and network interfaces 981-887 were included in individual servers. In one example, the processor subsystem 913 is configured to initialize an IP network connection regardless of whether servers have been connected to the server platform 911.
According to various embodiments, a VSSL card 961 is coupled to translation logic 923. In some embodiments, a VSSL card 961 is separate from a VNIC or VN chip or card 951. The VSSL card includes a VSSL chip 967. In some embodiments, the VSSL chip 967 polls transmit and receive queues associated with various servers. When it sees a packet in a server transmit queue, it pulls the packet and forwards associated commands to an SSL chip 969. SSL chip 969 can be one of a number of available SSL accelerators, such as the Nitrox Chip available from Cavium Networks of Mountain View, Calif. It should be noted that although a VSSL card 961 and a VN chip 951 are shown, virtualization logic and accelerator functionality can also be implemented in a single chip. Alternatively, a VN chip 951 may be separated out into multiple components and implemented as a card. In still other implementations, all virtualization logic for the VN chip 951, the VSSL card 961, and any other components is implemented in a single device. In one embodiment, an SSL chip 969 includes general purpose processor cores and specialized cryptography cores included on a single ASIC. The cryptography cores may include SSL cores, IP Security (IPSec) cores, SSL-VPN cores, XML/web services cryptography cores, storage and fibre channel data cryptography cores, and email data cryptography cores.
IOP 963 is configured to perform context management for SSL chip 969. According to various embodiments, an SSL agent runs on IOP 963. The SSL agent manages and allocates key memory and context memory. A context is used for each SSL connection. A context is allocated before a new SSL connection is established. According to various embodiments, context can be established in several manners.
A server can send a context allocation request to the SSL transmit queue. The VSSL chip 967 pulls this request and notifies the SSL agent running on IOP 963. The SSL agent allocates a new context and passes an index back to the VSSL chip 967. The VSSL chip 967 sends the response back to the server's receive queue. An application will now have a context allocated for it and the context can be passed in for all SSL requests.
Alternatively, an SSL agent could allocate a pool of contexts per server. Whenever an application needs a new context, it could get the context locally from the VSSL driver (which keeps a cache of the contexts). Once the VSSL driver's cache runs out of contexts, it sends a request to the SSL Agent to obtain additional contexts.
According to various embodiments, a VNIC chip or VN chip 951 is also coupled to the servers. In one example, the VN chip 951 is connected to the fabric interface 921 through optional translation logic 923. The VN chip 951 also has a connection with the processor subsystem 913 and a series of network interfaces 981-987 connecting the VN chip 951 to external network entities. In other examples, the VN chip may not include NIC interfaces and instead may be connected to conventional NICs.
The VNIC chip includes classifier logic 947, a queue manager 945, and a buffer manager 943. According to various embodiments, the classifier logic 947 includes parse and lookup logic configured to identify information such as a packet destination server and priority. Classifier logic can also be used to filter incoming data or apply traffic policing policies. In some instances, classifier logic can be used to block packets in order to implement a firewall. In one embodiment, classifier logic 947 parses a packet and uses the information in the packet to identify entries in lookup tables. The data is then buffered. Buffer manager 943 manages data in memory associated with the VN chip 951. Queue manager 945 manages descriptors for data posted. A descriptor can include a reference to a memory location, a length, a source port, and a multicast count, as well as other parameters.
In one example, classifier logic 947 determines that the packet received is a high priority packet and should be placed in a high priority queue by the buffer manager 943. Parameters provided may include a pointer, a length, a source port, a multicast count, and a queue identifier. The data is then placed into memory and information referencing the data such as a pointer and a length is posted into a buffer ring or a descriptor ring. When a connected server successfully arbitrates for bus access, the server reads the buffer ring or descriptor ring and obtains the data from memory associated with the VN chip. According to various embodiments, the server reads the data directly into its own memory.
Each individual server may also include descriptor queues. As will be appreciated, the servers connected to the network switch and the remote entity arbitrate for access to the I/O Bus. When access is obtained, data can be read from memory associated with one of the server based on the information provided in the descriptor queues.
Redundancy mechanisms are also provided to allow continued operation in the event that a NIC or other resource fails or a remote entity itself fails. Redundancy mechanisms can be managed by a VNIC device or VN chip, a remote entity, or by the individual servers themselves.
A fabric interface 1021 is connected to multiple computer systems using a network such as Infiniband and translation logic 1023. Virtualization logic 1061 obtains data such as descriptor data from individual servers and provides the data to hardware resources 1071. In some examples, data transfer is performed using virtualized Direct Memory Access (DMA) mechanisms to allow minimal processor involvement during data transfers. In some embodiments, a descriptor is provided on a descriptor queue. The descriptor includes addresses, lengths, and other data parameters. A virtualized DMA engine 1063 in virtualization logic 1061 reads the descriptor and directly transfers the block of data in memory to appropriate hardware resources 1071.
In addition, although exemplary techniques and devices are described, the above-described embodiments may be implemented in a variety of manners, media, and mechanisms. For instance, instructions and data for implementing the above-described invention may be stored on a disk drive, a hard drive, a floppy disk, a server computer, or a remotely networked computer. Hardware used to implement various techniques may be embodied as racks, cards, integrated circuited devices, or portions of semiconductor chips. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
Number | Name | Date | Kind |
---|---|---|---|
5621913 | Tuttle et al. | Apr 1997 | A |
5754948 | Metze | May 1998 | A |
5815675 | Steele et al. | Sep 1998 | A |
5898815 | Bluhm et al. | Apr 1999 | A |
6003112 | Tetrick | Dec 1999 | A |
6069895 | Ayandeh | May 2000 | A |
6145028 | Shank et al. | Nov 2000 | A |
6157955 | Narad et al. | Dec 2000 | A |
6247086 | Allingham | Jun 2001 | B1 |
6253334 | Amdahl et al. | Jun 2001 | B1 |
6282647 | Leung et al. | Aug 2001 | B1 |
6308282 | Huang et al. | Oct 2001 | B1 |
6314525 | Mahalingham et al. | Nov 2001 | B1 |
6331983 | Haggerty et al. | Dec 2001 | B1 |
6343324 | Hubis et al. | Jan 2002 | B1 |
6377992 | Plaza et al. | Apr 2002 | B1 |
6393483 | Latif et al. | May 2002 | B1 |
6401117 | Narad et al. | Jun 2002 | B1 |
6418494 | Shatas et al. | Jul 2002 | B1 |
6430191 | Klausmeier et al. | Aug 2002 | B1 |
6466993 | Bonola | Oct 2002 | B1 |
6470397 | Shah et al. | Oct 2002 | B1 |
6578128 | Arsenault et al. | Jun 2003 | B1 |
6594329 | Susnow | Jul 2003 | B1 |
6628608 | Lau et al. | Sep 2003 | B1 |
6708297 | Bassel | Mar 2004 | B1 |
6725388 | Susnow | Apr 2004 | B1 |
6757725 | Frantz et al. | Jun 2004 | B1 |
6779064 | McGowen et al. | Aug 2004 | B2 |
6804257 | Benayoun et al. | Oct 2004 | B1 |
6807581 | Starr et al. | Oct 2004 | B1 |
6823458 | Lee et al. | Nov 2004 | B1 |
6898670 | Nahum | May 2005 | B2 |
6931511 | Weybrew et al. | Aug 2005 | B1 |
6937574 | Delaney et al. | Aug 2005 | B1 |
6963946 | Dwork et al. | Nov 2005 | B1 |
6970921 | Wang et al. | Nov 2005 | B1 |
7011845 | Kozbor et al. | Mar 2006 | B2 |
7046668 | Pettey et al. | May 2006 | B2 |
7093265 | Jantz et al. | Aug 2006 | B1 |
7096308 | Main et al. | Aug 2006 | B2 |
7103064 | Pettey et al. | Sep 2006 | B2 |
7103888 | Cayton et al. | Sep 2006 | B1 |
7111084 | Tan et al. | Sep 2006 | B2 |
7120728 | Krakirian et al. | Oct 2006 | B2 |
7127445 | Mogi et al. | Oct 2006 | B2 |
7143227 | Maine | Nov 2006 | B2 |
7159046 | Mulla et al. | Jan 2007 | B2 |
7171434 | Ibrahim et al. | Jan 2007 | B2 |
7171495 | Matters et al. | Jan 2007 | B2 |
7181211 | Phan-Anh | Feb 2007 | B1 |
7188209 | Pettey et al. | Mar 2007 | B2 |
7203842 | Kean | Apr 2007 | B2 |
7209439 | Rawlins et al. | Apr 2007 | B2 |
7213246 | van Rietschote et al. | May 2007 | B1 |
7219183 | Pettey et al. | May 2007 | B2 |
7240098 | Mansee | Jul 2007 | B1 |
7260661 | Bury et al. | Aug 2007 | B2 |
7269168 | Roy et al. | Sep 2007 | B2 |
7281030 | Davis | Oct 2007 | B1 |
7281077 | Woodral | Oct 2007 | B2 |
7281169 | Golasky et al. | Oct 2007 | B2 |
7307948 | Infante et al. | Dec 2007 | B2 |
7308551 | Arndt et al. | Dec 2007 | B2 |
7334178 | Aulagnier | Feb 2008 | B1 |
7345689 | Janus et al. | Mar 2008 | B2 |
7346716 | Bogin et al. | Mar 2008 | B2 |
7360017 | Higaki et al. | Apr 2008 | B2 |
7366842 | Acocella et al. | Apr 2008 | B1 |
7386637 | Arndt et al. | Jun 2008 | B2 |
7395352 | Lam et al. | Jul 2008 | B1 |
7412536 | Oliver et al. | Aug 2008 | B2 |
7421710 | Qi et al. | Sep 2008 | B2 |
7424529 | Hubis | Sep 2008 | B2 |
7433300 | Bennett et al. | Oct 2008 | B1 |
7457897 | Lee et al. | Nov 2008 | B1 |
7457906 | Pettey et al. | Nov 2008 | B2 |
7493416 | Pettey | Feb 2009 | B2 |
7502884 | Shah et al. | Mar 2009 | B1 |
7509436 | Rissmeyer | Mar 2009 | B1 |
7516252 | Krithivas | Apr 2009 | B2 |
7602774 | Sundaresan et al. | Oct 2009 | B1 |
7606260 | Oguchi et al. | Oct 2009 | B2 |
7609723 | Munguia | Oct 2009 | B2 |
7634650 | Shah et al. | Dec 2009 | B1 |
7669000 | Sharma et al. | Feb 2010 | B2 |
7711789 | Jnagal et al. | May 2010 | B1 |
7733890 | Droux et al. | Jun 2010 | B1 |
7782869 | Chitlur Srinivasa | Aug 2010 | B1 |
7783788 | Quinn et al. | Aug 2010 | B1 |
7792923 | Kim | Sep 2010 | B2 |
7793298 | Billau et al. | Sep 2010 | B2 |
7821973 | McGee et al. | Oct 2010 | B2 |
7836332 | Hara et al. | Nov 2010 | B2 |
7843907 | Abou-Emara et al. | Nov 2010 | B1 |
7849153 | Kim | Dec 2010 | B2 |
7865626 | Hubis | Jan 2011 | B2 |
7870225 | Kim | Jan 2011 | B2 |
7899928 | Naik et al. | Mar 2011 | B1 |
7933993 | Skinner | Apr 2011 | B1 |
7937447 | Cohen et al. | May 2011 | B1 |
7941814 | Okcu et al. | May 2011 | B1 |
8041875 | Shah et al. | Oct 2011 | B1 |
8180872 | Marinelli et al. | May 2012 | B1 |
8180949 | Shah et al. | May 2012 | B1 |
8185664 | Lok et al. | May 2012 | B1 |
8195854 | Sihare | Jun 2012 | B1 |
8200871 | Rangan et al. | Jun 2012 | B2 |
8218538 | Chidambaram et al. | Jul 2012 | B1 |
8228820 | Gopal Gowda et al. | Jul 2012 | B2 |
8261068 | Raizen et al. | Sep 2012 | B1 |
8285907 | Chappell et al. | Oct 2012 | B2 |
8291148 | Shah et al. | Oct 2012 | B1 |
8387044 | Yamada et al. | Feb 2013 | B2 |
8392645 | Miyoshi | Mar 2013 | B2 |
8397092 | Karnowski | Mar 2013 | B2 |
8443119 | Limaye et al. | May 2013 | B1 |
8458306 | Sripathi | Jun 2013 | B1 |
8677023 | Vengataraghavan et al. | Mar 2014 | B2 |
8892706 | Dalal | Nov 2014 | B1 |
9064058 | Daniel | Jun 2015 | B2 |
9083550 | Cohen et al. | Jul 2015 | B2 |
9264384 | Sundaresan et al. | Feb 2016 | B1 |
9331963 | Krishnamurthi et al. | May 2016 | B2 |
20010032280 | Osakada et al. | Oct 2001 | A1 |
20010037406 | Philbrick et al. | Nov 2001 | A1 |
20020023151 | Iwatani | Feb 2002 | A1 |
20020065984 | Thompson et al. | May 2002 | A1 |
20020069245 | Kim | Jun 2002 | A1 |
20020146448 | Kozbor et al. | Oct 2002 | A1 |
20020152327 | Kagan et al. | Oct 2002 | A1 |
20030007505 | Noda et al. | Jan 2003 | A1 |
20030028716 | Sved | Feb 2003 | A1 |
20030037177 | Sutton et al. | Feb 2003 | A1 |
20030051076 | Webber | Mar 2003 | A1 |
20030081612 | Goetzinger et al. | May 2003 | A1 |
20030093501 | Carlson et al. | May 2003 | A1 |
20030099254 | Richter | May 2003 | A1 |
20030110364 | Tang et al. | Jun 2003 | A1 |
20030118053 | Edsall et al. | Jun 2003 | A1 |
20030126315 | Tan et al. | Jul 2003 | A1 |
20030126320 | Liu et al. | Jul 2003 | A1 |
20030126344 | Hodapp, Jr. | Jul 2003 | A1 |
20030131182 | Kumar et al. | Jul 2003 | A1 |
20030165140 | Tang et al. | Sep 2003 | A1 |
20030172149 | Edsall et al. | Sep 2003 | A1 |
20030200315 | Goldenberg et al. | Oct 2003 | A1 |
20030208614 | Wilkes | Nov 2003 | A1 |
20030212755 | Shatas et al. | Nov 2003 | A1 |
20030226018 | Tardo et al. | Dec 2003 | A1 |
20030229645 | Mogi et al. | Dec 2003 | A1 |
20040003140 | Rimmer | Jan 2004 | A1 |
20040003141 | Matters et al. | Jan 2004 | A1 |
20040003154 | Harris et al. | Jan 2004 | A1 |
20040008713 | Knight et al. | Jan 2004 | A1 |
20040025166 | Adlung et al. | Feb 2004 | A1 |
20040028063 | Roy et al. | Feb 2004 | A1 |
20040030857 | Krakirian et al. | Feb 2004 | A1 |
20040034718 | Goldenberg et al. | Feb 2004 | A1 |
20040054776 | Klotz et al. | Mar 2004 | A1 |
20040057441 | Li et al. | Mar 2004 | A1 |
20040064590 | Starr et al. | Apr 2004 | A1 |
20040078632 | Infante et al. | Apr 2004 | A1 |
20040081145 | Harrekilde-Petersen et al. | Apr 2004 | A1 |
20040107300 | Padmanabhan et al. | Jun 2004 | A1 |
20040123013 | Clayton et al. | Jun 2004 | A1 |
20040139237 | Rangan et al. | Jul 2004 | A1 |
20040151188 | Maveli et al. | Aug 2004 | A1 |
20040160970 | Dally et al. | Aug 2004 | A1 |
20040172494 | Pettey et al. | Sep 2004 | A1 |
20040179529 | Pettey et al. | Sep 2004 | A1 |
20040210623 | Hydrie et al. | Oct 2004 | A1 |
20040218579 | An | Nov 2004 | A1 |
20040225719 | Kisley et al. | Nov 2004 | A1 |
20040225764 | Pooni et al. | Nov 2004 | A1 |
20040233933 | Munguia | Nov 2004 | A1 |
20040236877 | Burton | Nov 2004 | A1 |
20050010688 | Murakami et al. | Jan 2005 | A1 |
20050033878 | Pangal et al. | Feb 2005 | A1 |
20050039063 | Hsu et al. | Feb 2005 | A1 |
20050044301 | Vasilevsky et al. | Feb 2005 | A1 |
20050050191 | Hubis | Mar 2005 | A1 |
20050058085 | Shapiro et al. | Mar 2005 | A1 |
20050066045 | Johnson et al. | Mar 2005 | A1 |
20050080923 | Elzur | Apr 2005 | A1 |
20050080982 | Vasilevsky et al. | Apr 2005 | A1 |
20050091441 | Qi et al. | Apr 2005 | A1 |
20050108407 | Johnson et al. | May 2005 | A1 |
20050111483 | Cripe et al. | May 2005 | A1 |
20050114569 | Bogin et al. | May 2005 | A1 |
20050114595 | Karr et al. | May 2005 | A1 |
20050120160 | Plouffe et al. | Jun 2005 | A1 |
20050141425 | Foulds | Jun 2005 | A1 |
20050160251 | Zur et al. | Jul 2005 | A1 |
20050182853 | Lewites et al. | Aug 2005 | A1 |
20050188239 | Golasky et al. | Aug 2005 | A1 |
20050198410 | Kagan et al. | Sep 2005 | A1 |
20050198523 | Shanbhag et al. | Sep 2005 | A1 |
20050203908 | Lam et al. | Sep 2005 | A1 |
20050232285 | Terrell et al. | Oct 2005 | A1 |
20050238035 | Riley | Oct 2005 | A1 |
20050240621 | Robertson et al. | Oct 2005 | A1 |
20050240932 | Billau et al. | Oct 2005 | A1 |
20050262269 | Pike | Nov 2005 | A1 |
20060004983 | Tsao et al. | Jan 2006 | A1 |
20060007937 | Sharma | Jan 2006 | A1 |
20060010287 | Kim | Jan 2006 | A1 |
20060013240 | Ma et al. | Jan 2006 | A1 |
20060045089 | Bacher et al. | Mar 2006 | A1 |
20060045098 | Krause | Mar 2006 | A1 |
20060050693 | Bury et al. | Mar 2006 | A1 |
20060059400 | Clark et al. | Mar 2006 | A1 |
20060092928 | Pike et al. | May 2006 | A1 |
20060129699 | Kagan et al. | Jun 2006 | A1 |
20060136570 | Pandya | Jun 2006 | A1 |
20060168286 | Makhervaks et al. | Jul 2006 | A1 |
20060168306 | Makhervaks et al. | Jul 2006 | A1 |
20060179178 | King | Aug 2006 | A1 |
20060182034 | Klinker et al. | Aug 2006 | A1 |
20060184711 | Pettey et al. | Aug 2006 | A1 |
20060193327 | Arndt et al. | Aug 2006 | A1 |
20060200584 | Bhat | Sep 2006 | A1 |
20060212608 | Arndt et al. | Sep 2006 | A1 |
20060224843 | Rao et al. | Oct 2006 | A1 |
20060233168 | Lewites et al. | Oct 2006 | A1 |
20060242332 | Johnsen et al. | Oct 2006 | A1 |
20060253619 | Torudbakken et al. | Nov 2006 | A1 |
20060282591 | Krithivas | Dec 2006 | A1 |
20060292292 | Brightman et al. | Dec 2006 | A1 |
20070011358 | Wiegert et al. | Jan 2007 | A1 |
20070050520 | Riley | Mar 2007 | A1 |
20070067435 | Landis et al. | Mar 2007 | A1 |
20070101173 | Fung | May 2007 | A1 |
20070112574 | Greene | May 2007 | A1 |
20070112963 | Dykes et al. | May 2007 | A1 |
20070130295 | Rastogi et al. | Jun 2007 | A1 |
20070220170 | Abjanic et al. | Sep 2007 | A1 |
20070286233 | Latif et al. | Dec 2007 | A1 |
20080025217 | Gusat et al. | Jan 2008 | A1 |
20080082696 | Bestler | Apr 2008 | A1 |
20080159260 | Vobbilisetty et al. | Jul 2008 | A1 |
20080192648 | Galles | Aug 2008 | A1 |
20080205409 | McGee et al. | Aug 2008 | A1 |
20080225877 | Yoshida | Sep 2008 | A1 |
20080270726 | Elnozahy et al. | Oct 2008 | A1 |
20080288627 | Hubis | Nov 2008 | A1 |
20080301692 | Billau et al. | Dec 2008 | A1 |
20080307150 | Stewart et al. | Dec 2008 | A1 |
20090070422 | Kashyap et al. | Mar 2009 | A1 |
20090106470 | Sharma et al. | Apr 2009 | A1 |
20090141728 | Brown et al. | Jun 2009 | A1 |
20090307388 | Tchapda | Dec 2009 | A1 |
20100088432 | Itoh | Apr 2010 | A1 |
20100138602 | Kim | Jun 2010 | A1 |
20100195549 | Aragon et al. | Aug 2010 | A1 |
20100232450 | Maveli et al. | Sep 2010 | A1 |
20100293552 | Allen et al. | Nov 2010 | A1 |
20110153715 | Oshins et al. | Jun 2011 | A1 |
20110154318 | Oshins et al. | Jun 2011 | A1 |
20120072564 | Johnsen | Mar 2012 | A1 |
20120079143 | Krishnamurthi et al. | Mar 2012 | A1 |
20120110385 | Fleming et al. | May 2012 | A1 |
20120144006 | Wakamatsu et al. | Jun 2012 | A1 |
20120158647 | Yadappanavar et al. | Jun 2012 | A1 |
20120163376 | Shukla et al. | Jun 2012 | A1 |
20120163391 | Shukla et al. | Jun 2012 | A1 |
20120166575 | Ogawa et al. | Jun 2012 | A1 |
20120167080 | Vilayannur et al. | Jun 2012 | A1 |
20120209905 | Haugh et al. | Aug 2012 | A1 |
20120239789 | Ando et al. | Sep 2012 | A1 |
20120304168 | Raj Seeniraj et al. | Nov 2012 | A1 |
20130031200 | Gulati et al. | Jan 2013 | A1 |
20130080610 | Ando | Mar 2013 | A1 |
20130117421 | Wimmer | May 2013 | A1 |
20130117485 | Varchavtchik et al. | May 2013 | A1 |
20130138836 | Cohen et al. | May 2013 | A1 |
20130145072 | Venkataraghavan et al. | Jun 2013 | A1 |
20130159637 | Forgette et al. | Jun 2013 | A1 |
20130179532 | Tameshige et al. | Jul 2013 | A1 |
20130201988 | Zhou et al. | Aug 2013 | A1 |
20140122675 | Cohen et al. | May 2014 | A1 |
20150134854 | Tchapda | May 2015 | A1 |
Number | Date | Country |
---|---|---|
104823409 | Aug 2015 | CN |
2912805 | Sep 2015 | EP |
2912805 | Jun 2016 | EP |
2014070445 | May 2014 | WO |
Entry |
---|
Liu et al., “High Performance RDMA-Based MPI Implementation over InfiniBand”, ICS'03, Jun. 23-26, 2003, San Francisco, California, USA, Copyright 2003, ACM 1-58113-733-8/03/0006. |
Office Action dated May 31, 2007, from U.S. Appl. No. 11/179,085. |
Final Office Action dated Oct. 30, 2007, from U.S. Appl. No. 11/179,085. |
Notice of Allowance dated Aug. 11, 2008, from U.S. Appl. No. 11/179,085. |
Shah, et al., U.S. Appl. No. 12/250,842, titled “Resource Virtualization Switch,” filed Oct. 14, 2008. |
Office Action dated Aug. 10, 2010, from U.S. Appl. No. 12/250,842. |
Notice of Allowance for Application No. dated Feb. 18, 2011. |
Allowed Claims dated Feb. 18, 2011. |
Figueiredo et al., “Resource Virtualization Renaissance,” May 2005, IEEE Computer Society, pp. 28-31. |
Ajay V. Bhatt, “Creating a Third generation I/O Interconnect,” Intel ® Developer Network for PCI Express Architecture, www.express-lane.or, originally printed on Aug. 22, 2005, pp. 1-11. |
U.S. Appl. No. 12/544,744, Non-Final Office Action dated Apr. 4, 2014, 30 pages. |
U.S. Appl. No. 11/083,258, Final Office Action dated Apr. 18, 2014, 37 pages. |
International Search Report and written Opinion of PCT/US2013/065008 dated Apr. 16, 2014, 17 pages. |
Marshall, Xsigo Systems Launches Company and 1/0 Virtualization Product, vmblog.com, http:/lvmblog.com/archive/2007/09/15/xsigo-systems-launches-company-and-i-o-virtualization-product.aspx, accessed on Mar. 24, 2014, Sep. 15, 2007. |
U.S. Appl. No. 11/083,258, Non-Final Office Action dated Sep. 10, 2014, 34 pages. |
U.S. Appl. No. 12/544,744, Final Office Action dated Nov. 7, 2014, 32 pages. |
U.S. Appl. No. 12/890,498, Advisory Action dated Apr. 16, 2012, 4 pages. |
Wikipedia's article on ‘Infiniband’, Aug. 2010. |
U.S. Appl. No. 11/083,258, Final Office Action dated Feb. 2, 2009, 13 pages. |
U.S. Appl. No. 11/083,258, Final Office Action dated Jun. 10, 2010, 15 pages. |
U.S. Appl. No. 11/083,258, Final Office Action dated Oct. 26, 2012, 30 pages. |
U.S. Appl. No. 11/083,258, Non-Final Office Action dated Jul. 11, 2008, 12 pages. |
U.S. Appl. No. 11/083,258, Non-Final Office Action dated Nov. 12, 2009, 13 pages. |
U.S. Appl. No. 11/083,258, Non-Final Office Action dated Mar. 28, 2011, 14 pages. |
U.S. Appl. No. 11/083,258, Non-Final Office Action dated Apr. 25, 2012, 30 pages. |
U.S. Appl. No. 11/086,117, Final Office Action dated Dec. 23, 2008, 11 pages. |
U.S. Appl. No. 11/086,117, Final Office Action dated Dec. 10, 2009, 18 pages. |
U.S. Appl. No. 11/086,117, Non-Final Office Action dated May 6, 2009, 12 pages. |
U.S. Appl. No. 11/086,117, Non-Final Office Action dated Jul. 22, 2008, 13 pages. |
U.S. Appl. No. 11/086,117, Non-Final Office Action dated Jul. 22, 2010, 24 pages. |
U.S. Appl. No. 11/086,117, Notice of Allowance dated Dec. 27, 2010, 15 pages. |
U.S. Appl. No. 11/145,698, Final Office Action dated Aug. 18, 2009, 22 pages. |
U.S. Appl. No. 11/145,698, Final Office Action dated Jul. 6, 2011, 26 pages. |
U.S. Appl. No. 11/145,698, Non-Final Office Action dated May 9, 2013, 13 pages. |
U.S. Appl. No. 11/145,698, Non-Final Office Action dated Mar. 31, 2009, 22 pages. |
U.S. Appl. No. 11/145,698, Non-Final Office Action dated Mar. 16, 2011, 24 pages. |
U.S. Appl. No. 11/179,085, Pre Appeal Brief Request dated Jan. 24, 2008, 6 pages. |
U.S. Appl. No. 11/179,085, Preliminary Amendment dated May 27, 2008, 9 pages. |
U.S. Appl. No. 11/179,085, filed Jul. 11, 2005. |
U.S. Appl. No. 11/179,085, Response to Non-final Office Action filed on Aug. 10, 2007, 8 pages. |
U.S. Appl. No. 11/179,437, Final Office Action dated Jan. 8, 2009, 13 pages. |
U.S. Appl. No. 11/179,437, Non-Final Office Action dated May 8, 2008, 11 pages. |
U.S. Appl. No. 11/179,437, Notice of Allowance dated Jun. 1, 2009, 8 pages. |
U.S. Appl. No. 11/179,437, dated Jul. 11, 2005. |
U.S. Appl. No. 11/184,306, Non-Final Office Action dated Apr. 10, 2009, 5 pages. |
U.S. Appl. No. 11/184,306, Notice of Allowance dated Aug. 10, 2009, 4 pages. |
U.S. Appl. No. 11/200,761, U.S. Patent Application dated Aug. 9, 2005, 32 pages. |
U.S. Appl. No. 11/222,590, Non-Final Office Action dated Mar. 21, 2007, 6 pages. |
U.S. Appl. No. 11/222,590, Notice of Allowance dated Sep. 18, 2007, 5 pages. |
U.S. Appl. No. 12/250,842, Allowed Claims dated Jun. 10, 2011. |
U.S. Appl. No. 12/250,842, Notice of Allowance dated Feb. 18, 2011, 5 pages. |
U.S. Appl. No. 12/250,842, Notice of Allowance dated Jun. 10, 2011, 5 pages. |
U.S. Appl. No. 12/250,842, Response to Non-Final Office Action filed on Nov. 19, 2010, 8 pages. |
U.S. Appl. No. 12/544,744, Final Office Action dated Feb. 27, 2013, 27 pages. |
U.S. Appl. No. 12/544,744, Non-Final Office Action dated Jun. 6, 2012, 26 pages. |
U.S. Appl. No. 12/862,977, Non-Final Office Action dated Mar. 1, 2012, 8 pages. |
U.S. Appl. No. 12/862,977, Non-Final Office Action dated Aug. 29, 2012, 9 pages. |
U.S. Appl. No. 12/862,977, Notice of Allowance dated Feb. 7, 2013, 11 pages. |
U.S. Appl. No. 12/890,498, Non-Final Office Action dated Nov. 13, 2011, 10 pages. |
U.S. Appl. No. 12/890,498, Non-Final Office Action dated May 21, 2013, 22 pages. |
U.S. Appl. No. 13/229,587, Non-Final Office Action dated Oct. 6, 2011, 4 pages. |
U.S. Appl. No. 13/229,587, Notice of Allowance dated Jan. 19, 2012, 5 pages. |
U.S. Appl. No. 13/229,587, Response to Non-Final Office Action filed on Jan. 4, 2012, 4 pages. |
U.S. Appl. No. 13/445,570, Notice of Allowance dated Jun. 20, 2012, 5 pages. |
Kesavan et al., Active CoordinaTion (ACT)—Toward Effectively Managing Virtualized Multicore Clouds, IEEE, 2008. |
Poulton, Xsigo—Try it out, I dare you, Nov. 16, 2009. |
Ranadive et al., IBMon: Monitoring VMM-Bypass Capable InfiniBand Devices using Memory Introspection, ACM, 2009. |
Wong et al., Effective Generation of Test Sequences for Structural Testing of Concurrent Programs, IEEE International Conference of Complex Computer Systems (ICECCS'05), 2005. |
Xu et al., Performance Virtualization for Large-Scale Storage Systems, IEEE, 2003, 10 pages. |
U.S. Appl. No. 11/179,085, Non-Final Office Action dated May 31, 2007, 14 pages. |
U.S. Appl. No. 11/179,085, Final Office Action dated Oct. 30, 2007, 13 pages. |
U.S. Appl. No. 11/179,085, Notice of Allowance dated Aug. 11, 2008, 4 pages. |
U.S. Appl. No. 11/083,258, Advisory Action dated Jan. 24, 2013, 3 pages. |
U.S. Appl. No. 12/250,842, Non-Final Office Action dated Aug. 10, 2010, 9 pages. |
U.S. Appl. No. 12/250,842, filed Oct. 14, 2008. |
U.S. Appl. No. 11/083,258, Non-Final Office Action dated Sep. 18, 2013, 35 pages. |
U.S. Appl. No. 11/145,698, Notice of Allowance dated Oct. 24, 2013, 15 pages. |
U.S. Appl. No. 12/890,498, Final Office Action dated Feb. 7, 2012, 9 pages. |
HTTP Persistent Connection Establishment, Management and Termination, section of ‘The TCP/IP Guide’ version 3.0, Sep. 20, 2005, 2 pages. |
TCP Segment Retransmission Timers and the Retransmission Queue, section of ‘The TCP/IP Guide’ version 3.0, Sep. 20, 2005, 3 pages. |
TCP Window Size Adjustment and Flow Control, section of ‘The TCP/IP Guide’ version 3.0, Sep. 20, 2005, 2 pages. |
Balakrishnan et al., Improving TCP/IP Performance over Wireless Networks, Proc. 1st ACM Int'l Conf. on Mobile Computing and Networking (Mobicom), Nov. 1995, 10 pages. |
U.S. Appl. No. 12/890,498, Final Office Action dated Jun. 17, 2015, 24 pages. |
U.S. Appl. No. 11/083,258, Final Office Action dated Mar. 19, 2015, 37 pages. |
U.S. Appl. No. 13/663,405, Notice of Allowance dated Mar. 12, 2015, 13 pages. |
U.S. Appl. No. 12/890,498, Advisory Action dated Jan. 27, 2015, 3 pages. |
U.S. Appl. No. 12/890,498, Final Office Action dated Nov. 19, 2014, 21 pages. |
U.S. Appl. No. 12/890,498, Non-Final Office Action dated Mar. 5, 2015, 24 pages. |
U.S. Appl. No. 13/663,405, Non-Final Office Action dated Nov. 21, 2014, 19 pages. |
Spanbauer, Wired or Wireless, Choose Your Network, PCWorld, Sep. 30, 2003, 9 pages. |
U.S. Appl. No. 12/544,744, Non-Final Office Action dated Sep. 24, 2015, 29 pages. |
U.S. Appl. No. 12/890,498, Advisory Action dated Aug. 25, 2015, 3 pages. |
European Application No. 13850840.3, Extended European Search Report dated May 3, 2016, 6 pages. |
International Application No. PCT/US2013/065008, International Preliminary Report on Patentability dated May 14, 2015, 6 pages. |
U.S. Appl. No. 12/544,744, Final Office Action dated Jun. 1, 2016, 34 pages. |
U.S. Appl. No. 12/890,498, Supplemental Notice of Allowability dated Jan. 14, 2016, 2 pages. |
U.S. Appl. No. 11/083,258, Corrected Notice of Allowability dated Oct. 15, 2015, 5 pages. |
U.S. Appl. No. 11/083,258, Notice of Allowance dated Oct. 5, 2015, 8 pages. |
U.S. Appl. No. 12/890,498, Notice of Allowance dated Dec. 30, 2015, all pages. |
U.S. Appl. No. 11/086,117, dated Jul. 22, 2008, Non-Final Office Action. |
U.S. Appl. No. 11/086,117, dated Dec. 23, 2008, Final Office Action. |
U.S. Appl. No. 11/086,117, dated May 6, 2009, Non-Final Office Action. |
U.S. Appl. No. 11/086,117, dated Dec. 10, 2009, Final Office Action. |
U.S. Appl. No. 11/086,117, dated Jul. 22, 2010, Non-Final Office Action. |
U.S. Appl. No. 11/086,117, dated Dec. 27, 2010, Notice of Allowance. |
U.S. Appl. No. 11/145,698, dated Mar. 31, 2009, Non-Final Office Action. |
U.S. Appl. No. 11/145,698, dated Aug. 18, 2009, Final Office Action. |
U.S. Appl. No. 11/145,698, dated Mar. 16, 2011, Non-Final Office Action. |
U.S. Appl. No. 11/145,698, dated Jul. 6, 2011, Final Office Action. |
U.S. Appl. No. 11/145,698, dated May 9, 2013, Non-Final Office Action. |
U.S. Appl. No. 11/145,698, dated Oct. 24, 2013, Notice of Allowance. |
U.S. Appl. No. 11/179,085, dated May 31, 2007, Non-Final Office Action. |
U.S. Appl. No. 11/179,085, dated Oct. 30, 2007, Final Office Action. |
U.S. Appl. No. 11/179,085, dated Jan. 24, 2008, Pre Appeal Brief Request. |
U.S. Appl. No. 11/179,085, dated May 27, 2008, Preliminary Amendment. |
U.S. Appl. No. 11/179,085, dated Aug. 11, 2008, Notice of Allowance. |
U.S. Appl. No. 11/179,437, dated May 8, 2008, Non-Final Office Action. |
U.S. Appl. No. 11/179,437, dated Jan. 8, 2009, Final Office Action. |
U.S. Appl. No. 11/179,437, dated Jun. 1, 2009, Notice of Allowance. |
U.S. Appl. No. 11/184,306, dated Apr. 10, 2009, Non-Final Office Action. |
U.S. Appl. No. 11/184,306, dated Aug. 10, 2009, Notice of Allowance. |
U.S. Appl. No. 11/083,258, dated Jul. 11, 2008, Non-Final Office Action. |
U.S. Appl. No. 11/083,258, dated Feb. 2, 2009, Final Office Action. |
U.S. Appl. No. 11/083,258, dated Nov. 12, 2009, Non-Final Office Action. |
U.S. Appl. No. 11/083,258, dated Jun. 10, 2010, Final Office Action. |
U.S. Appl. No. 11/083,258, dated Mar. 28, 2011, Non-Final Office Action. |
U.S. Appl. No. 11/083,258, dated Apr. 25, 2012, Non-Final Office Action. |
U.S. Appl. No. 11/083,258, dated Oct. 26, 2012, Final Office Action. |
U.S. Appl. No. 11/083,258, dated Jan. 24, 2013, Advisory Action. |
U.S. Appl. No. 11/083,258, dated Sep. 18, 2013, Non-Final Office Action. |
U.S. Appl. No. 11/083,258, dated Apr. 18, 2014, Final Office Action. |
U.S. Appl. No. 11/083,258, dated Sep. 10, 2014, Non-Final Office Action. |
U.S. Appl. No. 11/083,258, dated Mar. 19, 2015, Final Office Action. |
U.S. Appl. No. 11/083,258, dated Oct. 5, 2015, Notice of Allowance. |
U.S. Appl. No. 11/083,258, dated Oct. 15, 2015, Notice of Allowability. |
U.S. Appl. No. 11/222,590, dated Mar. 21, 2007, Non-Final Office Action. |
U.S. Appl. No. 11/222,590, dated Sep. 18, 2007, Notice of Allowance. |
U.S. Appl. No. 12/250,842, dated Aug. 10, 2010, Non-Final Office Action. |
U.S. Appl. No. 12/250,842, dated Aug. 18, 2011, Notice of Allowance. |
U.S. Appl. No. 12/250,842, dated Jun. 10, 2011, Notice of Allowance. |
U.S. Appl. No. 12/250,842, dated Jun. 10, 2011, Allowed Claims. |
U.S. Appl. No. 12/544,744, dated Jun. 6, 2012, Non-Final Office Action. |
U.S. Appl. No. 12/544,744, dated Feb. 27, 2013, Final Office Action. |
U.S. Appl. No. 12/544,744, dated Apr. 4, 2014, Non-Final Office Action. |
U.S. Appl. No. 12/544,744, dated Nov. 7, 2014, Final Office Action. |
U.S. Appl. No. 12/544,744, dated Sep. 24, 2015, Non-Final Office Action. |
U.S. Appl. No. 12/862,977, dated Mar. 1, 2012, Non-Final Office Action. |
U.S. Appl. No. 12/862,977, dated Aug. 29, 2012, Non-Final Office Action. |
U.S. Appl. No. 12/862,977, dated Feb. 7, 2013, Notice of Allowance. |
U.S. Appl. No. 12/890,498, dated Nov. 13, 2011, Non-Final Office Action. |
U.S. Appl. No. 12/890,498, dated Feb. 7, 2012, Final Office Action. |
U.S. Appl. No. 12/890,498, dated May 21, 2013, Non-Final Office Action. |
U.S. Appl. No. 12/890,498, dated Apr. 16, 2014, Advisory Action. |
U.S. Appl. No. 12/890,498, dated Jan. 27, 2015, Advisory Action. |
U.S. Appl. No. 12/890,498, dated Nov. 19, 2014, Final Office Action. |
U.S. Appl. No. 12/890,498, dated Mar. 5, 2015, Non-Final Office Action. |
U.S. Appl. No. 12/890,498, dated Jun. 17, 2015, Final Office Action. |
U.S. Appl. No. 12/890,498, dated Aug. 25, 2015, Advisory Action. |
U.S. Appl. No. 12/890,498, dated Dec. 30, 2015, Notice of Allowance. |
U.S. Appl. No. 13/229,587, dated Oct. 6, 2011, Non-Final Office Action. |
U.S. Appl. No. 13/229,587, dated Jan. 19, 2012, Notice of Allowance. |
U.S. Appl. No. 13/445,570, dated Jun. 20, 2012, Notice of Allowance. |
U.S. Appl. No. 13/663,405, dated Nov. 21, 2014, Non-Final Office Action. |
U.S. Appl. No. 13/663,405, dated Mar. 12, 2015, Notice of Allowance. |
Number | Date | Country | |
---|---|---|---|
20130138758 A1 | May 2013 | US |