This application relates to data processing systems and networking. More particularly, the present disclosure relates to a method and system for port pooling.
The demand on certain physical servers may be greater than others because of actions being performed thereon. Additional physical servers may be added or existing physical servers may be re-assigned to meet demand.
Further, servers may experience excessive demand because of bandwidth limitations for physical server access through gateway ports on a server chassis. When adding new physical servers or re-assigning existing physical servers to meet demand, an administrator may modify a mapping of the gateway ports to the physical servers to provide sufficient bandwidth, and/or may add additional gateway ports to provide additional bandwidth. The administrator may then manually reconfigure or re-initialize the virtual servers to use the modified gateway port allocation. The administrator may seek to implement the changes at off-periods as the virtual servers may be unavailable for a period of time during reconfiguration or re-initialization.
The particular embodiments of the invention are illustrated by way of example, and not limitation, in the figures of the accompanying drawings, in which like reference numerals indicate the same or similar features unless otherwise indicated.
In the drawings,
Methods and systems for port pooling are described. An interface may communicate with at least one physical server. The at least one physical server may host a plurality of virtual servers and be connectable via a plurality of gateway ports to a storage area network (SAN). A virtual server manager configured to arrange the plurality of gateway ports in a plurality of port pools, define a virtual server group including a plurality of virtual servers, associate each virtual server with one or more port pools, the one or more port pools defining available gateway ports for access by the particular virtual server; and provide configuration instructions to allow the particular virtual server to communicate with the SAN through the available gateway ports.
In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration, specific embodiments in which the disclosed subject matter can be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the disclosed subject matter.
As described further below, according to various example embodiments of the disclosed subject matter described herein, there is provided methods and systems for port pooling. The system may include a computer program embedded within the memory and executable by the processor, the computer program comprising instructions to implement a port pooling system.
In a virtual server system that includes physical servers connected through a gateway to Storage Area Networks (SANs), port pools may be defined on the gateway by selecting a plurality of gateway ports. A number of the gateway ports in the port pools may be made available to a virtual server group. A virtual server may be then be defined within the virtual server group and World Wide Port Names (WWPNs) for the virtual server may be bound to the ports pools to provide the virtual server with an appropriate amount of bandwidth and to enable the virtual server to dynamically use a modified gateway port mapping.
In an example embodiment, as the server personality is disembodied or separated from the physical structure, it may be possible to provision virtual servers on-demand out of industry standard components to enable differing configurations of virtual servers without reconfiguring the server. Each virtual server deployed on a physical server defines a state of a physical server. This may include the logical definitions and configuration information stored in, and used by, a VSS director (described by way of example in more detail below) to program a server fabric, an Operating System (OS), and applications of the virtual server. The state may be stored on a logical unit on a SAN 30, as described in more detail below. Thus, in
The system 10 is shown, by way of example, to include a switch group 12 including one or more switches 14, 16. The switch group 12 is connected, for example, via an InfiniBand link 18 (e.g., a switched fabric communications link) to one or more server pools 20. Each server pool 20.1-20.3 is shown to include a plurality of physical servers 22.1-22.n linked via one or more InfiniBand links 18 to the switch group 12. By way of example, three physical server pools 20.1-20.3 (on which the virtual servers are deployed) are shown in
When the link 18 is an InfiniBand link, each switch 14 may include an InfiniBand interface 24 to interface the server pools 20.1-20.3 to the switch group 12. The InfiniBand architecture or link may define a high speed network for interconnecting processing nodes and I/O nodes. In an InfiniBand network, processing nodes and I/O nodes are connected to the fabric by Host Channel Adapters (HCAs) and Target Channel Adapters (TCAs). Other links may be provided in addition to, or instead of, the InfiniBand link 18.
In this example embodiment, the physical server infrastructure may be stateless computational resources with CPUs and memory. For example, as shown in
The switch 14 communicates with a plurality of different networks (Local Area Networks, Wide Area Networks, or the like) via communication links 38, 40, 42. For example, the communication links 38, 40, 42 may be Ethernet connections and, each switch 14, 16 may include one or more Ethernet gateways 44. In the example system 10, the communication link 38 is shown to connect to a network 46 interconnecting a plurality of hosts 48.1-48.5. The hosts 48.1-48.5 may form part of another data network, or be any other network host.
The switch 14 also communicates via the communication link 40 to a network 50 which may, for example, be an enterprise network. The network 50 communicates with desktop computers 52.1-52.2 and a subnet 54 which, in turn, is connected to desktop computers 56.1-56.3. Further, the switch 14 connects via the communication link 42 to a network such as the Internet 58. The aforementioned networks are merely example networks and different configurations, and different numbers of networks and subnets may be provided that connect a wide range of network devices.
The system 10 may allow virtualization of servers deployed on physical servers to be managed by a management module 60. The management module 60 may be provided at the switch 14 or in other components. The management module 60 communicates with a VSS director 62 that controls the provisioning of the server pools 20.1-20.3.
In an example embodiment, the VSS director 62 communicates via a network 64 with the management module 60. The system 10 also includes a third party management module 65 that communicates with the VSS director 62 and/or with the management module 60 to manage the provisioning of virtual servers. In an example embodiment, the network 64 is an Ethernet network and, accordingly, the switch 14 may thus include one or more Ethernet ports 66. However, the various communication links linking the various components/devices in the system 10 are not restricted to InfiniBand connections, Ethernet connections, or the like. Any communication means may be provided to interconnect the various components.
As shown by way of example in
A virtual server group 610, which includes one or more virtual servers 610.1-610.n, may be implemented by the VSS director 605. A server bank 640 is provided which may include one or more physical servers 640.1-640.n. The system 600 may also include a SAN 650 including target memory devices 650.1-650.n. Each of the target memory devices 650.1-650.n, such as target memory device 650.1, includes one or more logical units (LUs) 655.1-655.n. In an example embodiment, each LU 655.1-655.n serves as a memory location for a virtual server. A local disk 660 may be a memory device for the VSS director 605, and may have an image repository 670 that includes one or more images 670.1-670.n. The image repository 670 in an example embodiment may be located at another location, such as at the SAN 650. The VSS director 605, the server bank 640, and the SAN 650 can be interconnected through a switch group 620 (which is shown by way of example to include switches 620.1-620.n) and a fibre channel gateway (FC GW) group 630 (which includes FC GWs 630.1-630.n).
The gateway device 702.1 may include a number (e.g., one or a plurality) of gateway ports 704.1-704.n while the gateway device 702.2 may include a number of gateway ports 706.1-706.n. In an example embodiment, each fibre channel gateway 630.1-630.n (see
The SANs 710 may include a number of target memory devices 714.1-714.n, 716.1-716.m. In an example embodiment, the use of more than one SAN 710 may enable higher system availability and/or provide a separate storage mechanism. Each of the target memory devices 714.1-714.n, 716.1-716.m, such as the target memory device 714.1, may include a number of logical units (LUs) 718.1-718.n.
The gateway ports 704.1-704.n, 706.1-706.n may be bound to the SANs 710. By way of example, the gateway ports 704.1, 704.3, 706.1, 706.4 may be bound to the SAN 710.1 while the gateway ports 704.6, 706.5, 706.7 may be bound to the SAN 710.2. However, other gateway port-to-SAN configurations may also be used with the system 700.
Each of the SANs 710 may include access controls to enable access to a number of the logical units 718.1-718.n, 720.1-720.n of the target memory devices 714.1-714.n, 716.1-716.m. In an example embodiment, the SAN 650 (see
The use of a number of port pools 708.1-708.3 for the gateway ports 704.1-704.n, 706.1-706.n is described in greater detail by way of an example below.
A plurality of gateway ports 704.1-704.n, 706.1-706.n may be arranged in a plurality of port pools 708.1-708.3 at block 802. For example, the port pools 708.1-708.3 may be defined on the gateway device 702.1, 702.2 (see
As illustrated in
The port pools 708.1-708.3 may optionally be defined by selecting from among the gateway ports 704.1-704.n, 706.1-706.n to include in each of the port pools 708.1-708.3. For example, a first port pool 708.1 is shown to include eight gateway ports 704.1-704.4, 706.1-706.4, while a second port pool 708.2 is shown to include a gateway port 704.5. A third port pool 708.3 is shown to include gateway ports 704.6, 706.5-706.n.
A gateway port 704, 706 may belong to only one port pool 708 or the gateway port 704, 706 may belong to more than one port pool 708. Each of the gateway ports 704.1-704.n, 706.1-706.n in the one or more port pools 708.1-708.3 may be connected to the same server fabric. Each of the port pools 708.1-708.3 may optionally span more than one fibre channel gateway 900 and/or more than one gateway device 702.
In an example embodiment, the VSS director 62 (see
The port pools 708.1-708.3 may optionally be limited to the gateway ports 704.1-704.n, 706.1-706.n on the same SAN 710. For example, a dynamic assignment of the gateway ports 704.1-704.n, 706.1-706.n may be enabled by having the gateway ports 704.1-704.n, 706.1-706.n for each of the port pools 708.1-708.3 connected to the same SAN 710.
In an example embodiment, the user may not have to manually rebalance an entire physical server mapping to the gateway ports 704.1-704.n, 706.1-706.n to shift some load to a newly added gateway port 704, 706. For example, the VSS director 62 (see
A virtual server group may be defined to include a plurality of virtual servers to be hosted on at least one physical server at block 804. The virtual server group includes a pool listing of a number of gateway ports 704.1-704.n, 706.1-706.n available to the members of the virtual server group from each of the port pools 704.1-704.n.
As illustrated in
The user may optionally define the port pools 708.1-708.3 and select a number of the gateway ports 704.1-704.n, 706.1-706.n from among each of the port pools 708.1-708.3 that a virtual server group 1000 can access. Providing the virtual server group 1000 access to the port pools 708.1-708.3 as opposed to selected gateway ports 704.1-704.n, 706.1-706.n may reduce configuration work to be performed by a user of the system 10 (see
A number of virtual servers may be defined (e.g., within the virtual server group 1000) during the operations at block 804. For example, defining a virtual server may include generating a World-Wide Node Name (WWNN) with associated World-Wide Port Names (WWPNs) for the virtual server to use when communicating through the gateway ports 704.1-704.n, 706.1-706.n.
The virtual servers may be associated with one or more ports pools at block 806. The association of the virtual servers with the one or more port pools 708.1-708.3 provides the virtual servers with access to a number of available gateway ports 704.1-704.n, 706.1-706.n in one or more of the port pools 708.1-708.3. For example, the VSS director 62 may control an actual mapping of physical and/or virtual servers to the gateway ports 704.1-704.n, 706.1-706.n to achieve a desired and/or selected resource allocation of resources of the virtual server system 10 (see
As illustrated in
Each of the WWPNs 1104.1-1104.n may optionally be bound to the gateway ports 704.1-704.n, 706.1-706.n in a selected port pool 708. In an example embodiment, the WWPNs 1104.1-1104.n may be generated for a virtual server-to-gateway port binding. The VSS director 62 may optionally perform a WWPN generation and maintain a WWPN-to-port pool association. A list of the WWPNs 1104 for the virtual servers 1100 may optionally be provided (e.g., by an administrator of the virtual server system 10) to a storage administrator of the SANs 710.1-710.2 after the WWPNs 1104 are generated.
By way of example, if the port listing 1002 of the virtual server group 1000 provides that that each virtual server 1102 in the virtual server group 1000 has access to two gateway ports 704, 706 from a first port pool PP1 and three gateway ports from a second port pool PP2, then, when a virtual server VS1 is defined in the virtual server group 1000, the VSS director 62 (see
A user may optionally keep track of the port pools 708.1-708.3 to which the WWPNs 1104 are bound to instead of keeping track of an exact gateway port 704, 706 to which the WWPNs 1104 are bound. For example, a user may keep track of only the port pools 708.1-708.3 for a particular SAN 710 to which the WWPNs 1104 are bound to instead of keeping track of the exact gateway port 704, 706 to which the WWPNs 1104 are bound.
The WWPNs 1104 when defining the virtual server 1102 may differ depending on the gateway ports 704.1-704.n, 706.1-706.n available in a particular port pool 708 associated with the virtual server group 1000 of the virtual server 1102. For example, if the particular port pool 708 has four gateway ports 704, 706 and the port listing 1002 with the virtual server group 1000 indicates that two of the four gateway ports 704, 706 may be used, any two of the four gateway ports 704, 706 may be used to provide the WWPNs 1104 to the virtual server 1102.
A number of the WWPNs 1104 assigned to the virtual server 1102 may optionally be limited. Limiting the number of the WWPNs 1104 may reduce a configuration time to implement security on the SANs 710.1-710.2 (see
A mapping of the gateway ports 704.1-704.n, 706.1-706.n may optionally dynamically change as the VSS director 62 (see
In an example embodiment, each of the WWPNs 1104 may be associated with a particular port pool 708 and may be bound only to the gateway ports 704.1-704.n, 706.1-706.n in that particular port pool 708. The WWPNs 1104 may not appear in more than one port pool 708.
In an example embodiment, the WWPNs 1104 associated with each of the port pools 708.1-708.3 may be shown to a user through use of the user interface module 70 (see
The virtual server 1102 may have access to the gateway ports 704.1-704.n, 706.1-706.n when a WWPN 1104 is bound to a gateway port 704, 706. In an example embodiment, the VSS director 62 may bind the WWPNs 1104 to the gateway ports 704.1-704.n, 706.1-706.n for the virtual server 1102.
In an example embodiment, the use of the port pools 708.1-708.3 may enable the VSS director 62 to automatically handle errors as well as load balance connections for the virtual servers 1102 that are live. For example, if a particular gateway port 704, 706 fails, the VSS director 62 may migrate all virtual servers 1102 using that particular gateway port 704, 706 to other gateway ports 704, 706 in a same port pool 708. In this way, the virtual servers 1102 may continue to be provided with an appropriate amount of available port bandwidth.
When new gateway ports 704, 706 are added to a particular port pool 708, the VSS director 62 may, in an example embodiment, dynamically migrate some of the virtual servers 1102 to the new gateway ports 704, 706 to balance a load without waiting for the virtual server 1102 to be re-activated. Dynamically balancing a load through the use of port pools may save processing time and/or operator time that might otherwise be spent re-activating the virtual servers, and may also enable the virtual servers to more quickly use newly available resources without interference with the operation of the virtual servers from shut downs and reactivations.
The VSS director 62 may optionally balance a load across the gateway ports 704.1-704.n, 706.1-706.n to provide high availability for the virtual server 1102. For example, an algorithm used by the VSS director 62 for balancing the load may be enhanced to use input from one or more user defined policies.
Returning to
Upon completion of the operations at block 808, the method 800 may terminate.
In an example embodiment, a physical server 1202 may be assigned to the virtual server 1102 upon completion of the operations at block 808. The VSS director 62 (see
As illustrated in
In an example embodiment, the one or more port pools 708.1-708.3 may enable the VSS director 62 to assist a user in detecting storage configuration errors. For example, if it was determined that the virtual server 1102 could access a target memory device 716 (see
In an example embodiment, an alternative virtualization engine may be used instead of or in additional to the VSS director 62 (see
A port pool configuration may be accessed at block 1302. The port pool configuration may be a desired port pool behavior by an administrator based on a number of gateway ports 704.1-704.n, 706.1-706.n made available to a particular virtual server 1102 (see
A determination may be made (e.g., based on the port pool configuration accessed) at decision block 1304 whether to allow the virtual server group 1000 use of a particular port pool 708. If a determination is made not to allow the virtual server group 1000 to use the port pool 708, zero gateway ports 704.1-704.n, 706.1-706.n may be designated as being accessible to the virtual server 1102 from the port pool 708. If the determination is made to allow the virtual server group 1000 use of the port pool 708, the method 1300 may proceed to decision block 1308.
At decision block 1308, a determination may be made (e.g., based on the port pool configuration accessed) as to whether a load on the gateway ports 704.1-704.n, 706.1-706.n may be distributed (e.g., by the VSS director 62 of
If a determination is made not to allow the load to be dynamically distributed, manual control may be enabled to use available bandwidth for the virtual servers 1102 of the virtual server group at block 1312. Upon completion of the operations at block 1312, a determination may be made (e.g., based on the port pool configuration accessed) at decision block 1314 as to whether additional bandwidth (e.g., the addition of a gateway port 704, 706) should be used when available. If a determination is made that the additional bandwidth should not be used when available, the number of gateway ports 704.1-704.n, 706.1-706.n in the port pool 708 may be designated as the number of ports accessible to the port pool 708. The use of every one of the gateway ports 704.1-704.n, 706.1-706.n may provide the administrator with a maximum amount of control on which of the one of the gateway ports 704.1-704.n, 706.1-706.n the virtual server 1102 of the virtual server group 1000 may use.
If a determination is made to use the additional bandwidth when available at decision block 1314, a greater number than the number of gateway ports 704.1-704.n, 706.1-706.n in the port pool 708 may be designated as the number of ports accessible to the port pool 708 for use by the virtual server 1102 of the virtual server group 1000 at block 1318. For example, if an additional gateway port 704, 706 is added to the port pool, the virtual server 1102 may then use the additional gateway port 704, 706 for greater bandwidth.
Upon completion of the operations at block 1306, 1310, 1316, or block 1318, the method 1300 may terminate.
By way of an example, if a port pool 708 has only two gateway ports 704, 706 but the user wants to provide for future growth, the user may provide a virtual server 1102 access to four gateway ports 704, 706 in the port pool 708 even though there are currently only two gateway ports 704, 706 in the port pool 708. When two more gateway ports 704, 706 are added later, the new gateway ports 704, 706 may automatically be used (e.g., by VSS director 62).
In an example embodiment, the VSS director 62 may make an allocation decision based on a particular selection of gateway ports 704, 706 from the gateway ports 704.1-704.n 706.1-706.8 that are in a particular port pool 708, then seek to first spread the particular selection of gateway ports 704, 706 across the one or more gateway device 702.1-702.2 and then spread across the one or more gateways 812.1-812.n, 814.1-814.4. In an example embodiment, the gateway ports 704, 706 may be selected for high availability and/or load balancing.
When the VSS director 62 has a port pool 708 with multiple available gateway ports 704.1-704.n, 706.1-706.n, the VSS director 62 may balance an assignment of the multiple available gateway ports 704, 706 for the virtual server 1102 by selecting a first gateway port 704 from a first gateway device 702.1 and a second gateway port 706 from a second gateway device 702.2. By selecting multiple gateway devices 702 for the plurality of gateway ports 704.1-704.n, 706.1-706.n, if one of the gateway devices 702 fails, the virtual server 1102 may still have access to another gateway device 702, thereby providing high availability of the virtual server system 10. For example, a user may define one or more allocation policies and control how the gateway ports 704.1-704.n, 706.1-706.n are allocated using the one or more allocation policies.
During unavailability of a gateway port 704, 706, the VSS director 62 may assign a new gateway port 704, 706 from the same port pool 708 of the port pools 708.1-708.3 when available. For example, a selection of the gateway ports 704.1-704.n, 706.1-706.n may differ each time a virtual server 1102 is provisioned based on available of the gateway ports 704.1-704.n, 706.1-706.n to a port pool 708.
The physical server 1202 (see
The physical server 1202 may be reassigned to the virtual server 1102 at block 1404. For example, block 1404 may include the operations at block 808 (see
Upon completion of the operations at block 1404, the method 1400 may terminate.
The example computer system 1500 includes a processor 1502 (e.g., a central processing unit (CPU) a graphics processing unit (GPU) or both), a main memory 1504 and a static memory 1506, which communicate with each other via a bus 1508. The computer system 1500 may further include a video display unit 1510 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 1500 also includes an alphanumeric input device 1512 (e.g., a keyboard), a cursor control device 1514 (e.g., a mouse), a disk drive unit 1516, a signal generation device 1518 (e.g., a speaker) and a network interface device 1520.
The disk drive unit 1516 includes a machine-readable medium 1522 on which is stored one or more sets of instructions (e.g., software 1524) embodying any one or more of the methodologies or functions described herein. The software 1524 may also reside, completely or at least partially, within the main memory 1504 and/or within the processor 1502 during execution thereof by the computer system 1500, the main memory 1504 and the processor 1502 also constituting machine-readable media.
The software 1524 may further be transmitted or received over a network 1526 via the network interface device 1520.
While the machine-readable medium 1522 is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, optical and magnetic media, and carrier wave signals.
Although example embodiments of the present invention have been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the invention. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.
This application is a continuation of U.S. application Ser. No. 11/734,610, filed on Apr. 12, 2007 now U.S. Pat. No. 7,706,303, which claims the priority benefit of U.S. Provisional Application No. 60/805,853, filed 26 Jun. 2006, each of which is incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
6308315 | Dice et al. | Oct 2001 | B1 |
6421711 | Blumenau et al. | Jul 2002 | B1 |
6499063 | Chessell et al. | Dec 2002 | B1 |
6631395 | Chessell | Oct 2003 | B1 |
6801949 | Bruck et al. | Oct 2004 | B1 |
7069296 | Moller et al. | Jun 2006 | B2 |
7076801 | Gong et al. | Jul 2006 | B2 |
7111300 | Salas et al. | Sep 2006 | B1 |
7213065 | Watt | May 2007 | B2 |
7379990 | Tsao | May 2008 | B2 |
7546354 | Fan et al. | Jun 2009 | B1 |
7706303 | Bose et al. | Apr 2010 | B2 |
7747816 | Nourmohamadian et al. | Jun 2010 | B1 |
8176153 | Bose | May 2012 | B2 |
20020001307 | Nguyen et al. | Jan 2002 | A1 |
20020165961 | Everdell et al. | Nov 2002 | A1 |
20020188870 | Gong et al. | Dec 2002 | A1 |
20030005028 | Dritschler et al. | Jan 2003 | A1 |
20030018927 | Gadir et al. | Jan 2003 | A1 |
20030217131 | Hodge et al. | Nov 2003 | A1 |
20040054725 | Moller et al. | Mar 2004 | A1 |
20040236633 | Knauerhase et al. | Nov 2004 | A1 |
20050120160 | Plouffe et al. | Jun 2005 | A1 |
20050262505 | Esfahany et al. | Nov 2005 | A1 |
20060041660 | Bishop et al. | Feb 2006 | A1 |
20060107087 | Sieroka et al. | May 2006 | A1 |
20060114917 | Raisch | Jun 2006 | A1 |
20060143617 | Knauerhase et al. | Jun 2006 | A1 |
20060294516 | Winner et al. | Dec 2006 | A1 |
20070027973 | Stein et al. | Feb 2007 | A1 |
20070067435 | Landis et al. | Mar 2007 | A1 |
20070094367 | Esfahany et al. | Apr 2007 | A1 |
20070234334 | Araujo et al. | Oct 2007 | A1 |
20070234337 | Suzuki et al. | Oct 2007 | A1 |
20070250608 | Watt | Oct 2007 | A1 |
20070258388 | Bose | Nov 2007 | A1 |
20070283015 | Jackson et al. | Dec 2007 | A1 |
20070297428 | Bose et al. | Dec 2007 | A1 |
20070299906 | Bose et al. | Dec 2007 | A1 |
20090282404 | Khandekar et al. | Nov 2009 | A1 |
Entry |
---|
“U.S. Appl. No. 11/734,610, Non-Final Office Action mailed Jul. 23, 2009”, 17 pgs. |
“U.S. Appl. No. 11/734,610, Notice of Allowance mailed Dec. 17, 2009”, 4 pgs. |
“U.S. Appl. No. 11/734,610, Response filed Nov. 16, 2009 to Non Final Office Action mailed Jul. 23, 2009”, 13 pgs. |
“U.S. Appl. No. 11/743,537, Preliminary Amendment filed Jun. 17, 2008”, 7 pgs. |
“U.S. Appl. No. 11/692,736, Final Office Action mailed Mar. 28, 2011”, 22 pgs. |
“U.S. Appl. No. 11/692,736, Response filed Jan. 4, 2011 to Non Final Office Action mailed Aug. 4, 2010”, 10 pgs. |
“U.S. Appl. No. 11/743,537, Final Office Action mailed Feb. 9, 2011”, 16 pgs. |
“U.S. Appl. No. 11/743,537, Response filed May 9, 2011 to Final Office Action mailed Feb. 9, 2011”, 12 pgs. |
“U.S. Appl. No. 11/743,537, Response filed Jun. 23, 2010 to Non Final Office Action mailed Mar. 23, 2010”, 13 pgs. |
“U.S. Appl. No. 11/743,537, Response filed Dec. 3, 2010 to Non Final Office Action mailed Sep. 3, 2010”, 12 pgs. |
“U.S. Appl. No. 11/692,736, Examiner Interview Summary mailed Jan. 30, 2012”, 3 pgs. |
“U.S. Appl. No. 11/692,736, Final Office Action mailed Jun. 21, 2012”, 22 pgs. |
“U.S. Appl. No. 11/692,736, Non Final Office Action mailed Oct. 12, 2011”, 22 pgs. |
“U.S. Appl. No. 11/692,736, Response filed Aug. 10, 2011 to Final Office Action mailed Mar. 28, 2011”, 10 pgs. |
“U.S. Appl. No. 11/743,537, Non Final Office Action mailed Aug. 12, 2011”, 13 pgs. |
“U.S. Appl. No. 11/743,537, Notice of Allowance mailed Jan. 3, 2012”, 13 pgs. |
“U.S. Appl. No. 11/743,537, Response filed Nov. 14, 2011 to Non Final Office Action mailed Aug. 12, 2011”, 8 pgs. |
Number | Date | Country | |
---|---|---|---|
20100228840 A1 | Sep 2010 | US |
Number | Date | Country | |
---|---|---|---|
60805853 | Jun 2006 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 11734610 | Apr 2007 | US |
Child | 12754489 | US |