This invention relates to recovery from device and communication link failures in a virtual private network (VPN).
Virtual private networks are used to provide virtual communication links, called “tunnels,” between different sites that are coupled by a communication network. Typically, the communication network separating the sites is a public network, such as the Internet, which may not provide secure communication. The tunnels are then established to provide secure communication paths between the sites. One approach to establishing secure tunnels is according to the IPSec standard (see IETF RFCs 2401-2411 and 2451).
Network based services are often configured to have more than one gateway to the public network to provide high availability and communication capacity. For example, a service may have two separate gateways that make use of different communication links to different points-of-presence (POPs) for network service providers on the public network. In this way, if one of the communication links fails, then traffic may be able to pass over the other link. By connecting the gateways to different POPs, this approach also provides redundancy in case a POP fails, or a particular network service provider's network infrastructure (e.g., backbone network) fails. Multiple gateways also provide a basis for balancing communication load, and to reacting to performance degradation, such as congestion, that does not necessarily result in complete failure of a communication path.
Internet Protocol (IP) based networks can make use of dynamic routing protocols to maintain routing information at routing hosts and devices (routers) in the network in order to route IP packets through the network to their destination based on destination address information in packet headers. Two standard protocols for dynamically maintaining routing information are RIP (see RFC 1058) and OSPF (see RFC 2328). One characteristic of these protocols is that if a router determines that a neighboring router is inaccessible, for example because is does not receive responses to a number of requests for routing information from that neighbor, that router updates its routing tables and propagates an indication that the neighbor is inaccessible to its other neighboring routers.
In some virtual private networks, routing information passes between devices over the tunnels such that the tunnels act as virtual links and the devices at each end of the tunnels act as routing devices that are coupled by the virtual links. In this way devices at different sites can dynamically update routing information for paths that use those virtual links.
One approach to providing fault tolerance in a VPN network is to link computers using redundant virtual links. When communication through one of the virtual links is interrupted, active communication sessions are re-routed to one of the redundant virtual links. In an IP-based virtual network, this re-routing can be performed by passing routing information through the virtual links of the VPN according to the OSPF, RIP, or other dynamic routing protocols. As the routing information is updated, IP packets follow the new route through the redundant virtual link.
Transport layer communication sessions are typically configured to have timeout periods such that if communication is interrupted for more than that timeout period, the transport session is terminated. For example, a TCP session may have a timeout period in the order of 10 seconds. Therefore, if routing information is not dynamically updated in the VPN quickly enough, the transport session may not be rerouted to a redundant virtual link before its timeout period and therefore may be terminated. It is desirable to avoid such interruption of the transport layer sessions.
In general, the invention features an approach to rapid failover of a communication path between computers that are linked by redundant virtual links in a virtual private network (VPN). The approach features detection of communication link and device failures through an active monitoring approach and rerouting of communication through a redundant link when a failure is detected.
In a one aspect, in general, the invention features a method for providing remote access to a server system over a data network. Multiple communication paths through the data network are maintained between each of one or more client systems and the server system. Each path is associated with one of a number of gateway devices that are coupled between the data network and the server system. For each of the client systems, the paths between that client system and the server system are associated with different of the gateway devices. Communication is routed between a first of the client systems and the server system over a first of the communication paths passing through a first of the gateway devices. In the event of a communication failure associated with the first gateway device, the communication is re-routed to a second of the communication paths though a second of the gateway devices.
The method can include one or more of the following features:
The communication paths are through geographically distributed of the gateway devices, and these geographically distributed gateway devices are coupled through a trusted data network.
A separate virtual communication link is maintained between each of the client systems and multiple of the gateway devices.
At least some of the client systems include a client computer coupled to an access device. The virtual links are maintained between the access devices and the gateway devices.
A communication tunnel is maintained between the client system and the gateway device. The tunnel can use the PPTP, L2TP, IPSec, or other tunneling approach.
Network layer communication passes through the tunnel in a Point-to-Point Protocol (PPP) session.
Routing communication between the client system and the server system includes transmitting data from the client system over the data network to a first network addressed of the first gateway device.
Re-routing the communication includes transmitting data from the client system over the data network to a network address of the second gateway device, which is different than the address of the first gateway device.
The method further includes monitoring communication associated with the first gateway device.
Communication characteristics between the first gateway device and one or more devices of the server system are monitored.
The monitoring includes polling the devices from the first gateway device and detecting responses at the first gateway device from those devices.
Polling the devices includes transmitting ICMP echo requests.
Communication characteristics between the first gateway device and the data network are monitored.
A device coupled between the first gateway device and the data network is monitored.
Communication characteristics between the first gateway device and a network Point of Presence (POP) of the data network are monitored.
Communication characteristics between the first gateway device and one or more devices accessible over the data network are monitored.
Communication characteristics between the gateway device and the client system are monitored.
Heartbeat messages from the first gateway device are transmitted to devices and responses from the devices are detected at the first gateway device.
Heartbeat messages include ICMP echo requests, or LCP echo requests.
Re-routing the communication includes terminating the communication paths passing through the first gateway device.
Re-routing the communication includes updating routing data to indicate the second gateway device provides a path between the first client system and the server system.
Updating the routing data includes passing routing data from the second gateway device to the first client system.
Updating the routing data includes passing routing data from the second gateway device to one or more devices, such as routers and host computers, of the server system.
Aspects of the invention can include one or more of the following advantages:
Failover to a redundant link can be rapid enough to avoid timeout of transport layer communication sessions between the computers. For example, a TCP/IP communication session may have a timeout of less than 10 seconds, and the invention provides a mechanism by which communication is reestablished within this timeout period. In this way, ongoing communication, such as a lengthy TCP-based data transfer session does not have to be restarted.
Active monitoring of particularly important communication paths allows IP routing information to be updated quickly in the event of a failure and this updating may be achieved more quickly than may be possible using standard dynamic routing protocols.
Active monitoring using small heartbeat messages may load the communication links less than using standard “hello” messages for dynamic routing update protocols.
Redundant gateway devices do not have to be co-located or tightly coordinated to protect against failures of the gateway devices. For example, for increased reliability, the gateway devices can be geographically separated while still allowing a rapid failover in case of failure of one of the gateway devices or a failure of another device or communication link on the path through the gateway device.
The approach can be combined with redundant device and link backup approaches that allow a backup device or link to take the role of a primary device or link on a failure. If both a primary and backup of such a device fail, the invention provides a mechanism for further failure protection without interrupting communication sessions.
The approach can be combined with control of a variety of types of devices, such as load balancing devices, to route communication through backup gateways.
Other features and advantages of the invention are apparent from the following description, and from the claims.
Referring to
Each client 110 is configured to communicate through two or more gateways 140, each of which links public network 130 and private network 150. Two representative gateways 140 are shown in
Communication between clients 110 and servers 160 passes through public network 130 over virtual links of a virtual private network (VPN). At the site of each client 110, an access device 120 is coupled on the communication path between client 110 and public network 130. Each access device 120 establishes VPN tunnels 135 to two or more gateways 140. In this embodiment, each tunnel makes use of the IPSec protocol to transport and encrypt packets passing between access devices 120 and gateways 140. Each IPSec tunnel carries a PPP (Point-to-Point Protocol) data stream that is transported using a GRE (Generic Routing and Encapsulation) protocol. In this way, each tunnel provides a virtual link between the access device 120 and the gateway 140 at its endpoints. In alternative embodiments, different tunneling protocols, such as native IPSec tunneling, or L2TP or PPTP based tunnels are used to link the access devices and the gateways, and combinations of different tunneling protocols are used to link different clients to the servers.
Each access device 120 is preconfigured with the addresses on public network 130 of two or more gateways 140. For example, the IP host addresses of the gateways are stored in non-volatile memory of the access device. Access device 120 establishes tunnels 135 to those preconfigured gateways when it initially begins to establish communication with servers 160. Access device 120 treats one gateway is treated as a primary gateway while it treats the others as backups. After each tunnel 135 is established, gateway 140 sends updated routing information over private network 150 that identifies that gateway as a providing a route to the client 110 that is accessible over the tunnel. In this embodiment, the routing protocol used by gateways 140 on private network 150 is configurable from a set of protocols that include the OSPF (Open Shortest Path First) protocol and RIP (Router Interchange Protocol).
Client 110 uses its access device 120 as its default IP gateway. IP packets therefore first pass from client 110 to its access device 120. Based on that access device's current primary gateway 140, the access device routes packets addressed to server systems 160 over the tunnel 135 corresponding to its primary gateway 140. The receiving gateway 140 then routes the packet to the destination server 160. IP communication from client 110 to server 160 may use a transport protocol that acknowledges packets, such as TCP, and therefore server 160 sends acknowledgement packets addressed to client 110. These acknowledgments are routed back to client 110 according to routing information at the originating server 160 and at routing devices in private network 150.
Referring to
The general approach to handling each of these types of failures is to rapidly reroute communication from the tunnel 135 linking access device 120A and primary gateway 140A to the tunnel 135 linking access device 120A and backup gateway 140B. For example, if client 110A had active transport layer sessions with both server 160A and 160B (illustrated as paths 210), then after the failover, these transport sessions follow paths through backup gateway 140B (illustrated as paths 220).
Referring to
Gateway 140 also includes a heartbeat module 330 that communicates with devices on private network 150 and with access devices 120 over tunnels 135. Heartbeat module 330 is configurable. Configuration data 335 includes an identification of the particular devices on private network 150 with which heartbeat module should attempt to maintain communication. In addition, configuration data 335 identifies with which of these devices an ability to communicate is critical. In general, heartbeat module 330 sends periodic heartbeat messages to the identified devices. The period of the heartbeat messages is also configurable, typically in the range of 2.0 to 4.0 seconds. The period is chosen to be long enough to not unduly load the network and is chosen short enough such that a failure can be detected and acted upon quickly. During normal operation, the devices that receive the heartbeat messages immediately respond to each heartbeat message. Configuration data 335 also identifies how many sequential heartbeat messages to a particular device must be unanswered to declare a failure of communication with that device. Typically, this number is in the range of 2-4 messages. Therefore, a period of 3.5 seconds and declaration of a failure after 2 unanswered heartbeats yields detection within approximately 7.0 seconds of a failure, at which point failover is initiated.
Configuration data 335 also identifies whether devices on private network 150 are “critical.” If any critical device fails to respond to a heartbeat message from gateway 140, the entire gateway is disabled and all tunnels 135 from that gateway are shut down and no new tunnels are accepted until the critical device again starts responding to heartbeat messages. When gateway 140 shuts down its tunnels, routing module 320 sends updated routing information to devices on private network 150, for example using the OSPF protocol, to identify the fact that it is no longer a valid gateway to the access devices 120 and clients 110 that it previously served.
Other devices on private network 150 are not marked as critical, and gateway 140 is shut down only after all the non-critical devices concurrently fail to respond to heartbeat messages. Configuration data 335 also identifies the characteristics of heartbeat messages sent to access devices 120 over tunnels 135. If heartbeats through a particular tunnel 135 fail to be answered the configured number of sequential times, that tunnel is terminated, but the gateway otherwise continues to perform its functions. Routing module 320 sends updated routing information to devices on private network 150 to identify the fact that it is no longer a valid gateway to the access device 120 and client 110 that it previously served through the terminated tunnel.
Each access device 120 has a similar logical arrangement as that shown for gateway 140 in
Referring to
Each gateway 140 also sends heartbeat messages 420 to each access device 120 to which it has a tunnel connected. Heartbeat messages 420 do not necessarily use the same protocol as do heartbeat messages 410. The protocol is configurable. For example, LCP (Link Control Protocol) Echo Requests are used to implement heartbeats 420.
Access device 120 can also be configured to send heartbeat messages (440) to each of gateways 140. Access devices 120 can also be configured to send heartbeat messages (430) to particular devices (e.g., servers 160) through each gateway 140, thereby directly monitoring critical communication through each of the gateways.
Referring to
Referring to
If access device 120 is configured to initiate heartbeat messages 430 or 440, the access device itself initiates shutdown of a tunnel when a heartbeat message is not answered. Referring to
In various examples of this approach, private network 150 can include a variety of different types of routing “fabrics.” For example, private network 150 can be configured to be statically routed or to use a dynamic routing protocol such as OSPF.
Referring to
Referring to
Referring to
In alternative embodiments, gateways 140 pass information regarding accessibility to various clients 110 using approaches other than using standard dynamic routing protocols. For example, each gateway 140 may communicate directly with a load management device to indicate with clients are accessible using a special purpose communication protocol.
In other embodiments, or in combination with the approaches described above, devices on private network 150 can poll gateways 140 to determine their status. For example, a load balancing device can poll the gateways to determine which access devices 120 are accessible through the gateways.
In alternative embodiments, the primary gateway 140 advertises to devices on private network 150 a lower routing cost for routing traffic to client 110 than does the backup gateway 140. In this way, traffic flowing from private network 150 to client 110 is preferentially directed through the primary gateway 110. Since the primary gateway 140 has advertised a lower cost to route packets to client 110, acknowledgement packets will typically pass back to client 110 through the primary gateway 140 for that client.
It is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention. Other embodiments are within the scope of the following claims.
This application claims the benefit of U.S. Provisional Application No. 60/332,839 filed Nov. 6, 2001, which is incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
4627052 | Hoare et al. | Dec 1986 | A |
4734907 | Turner | Mar 1988 | A |
4823338 | Chan et al. | Apr 1989 | A |
4939726 | Flammer et al. | Jul 1990 | A |
5076688 | Bowen et al. | Dec 1991 | A |
5090025 | Marshall et al. | Feb 1992 | A |
5095480 | Fenner | Mar 1992 | A |
5132926 | MacEachern et al. | Jul 1992 | A |
5136580 | Videlock et al. | Aug 1992 | A |
5173933 | Garner et al. | Dec 1992 | A |
5243652 | Teare et al. | Sep 1993 | A |
5289460 | Drake, Jr. et al. | Feb 1994 | A |
5331637 | Francis et al. | Jul 1994 | A |
5355371 | Auerbach et al. | Oct 1994 | A |
5355375 | Christensen | Oct 1994 | A |
5361256 | Doeringer et al. | Nov 1994 | A |
5367667 | Wahlquist | Nov 1994 | A |
5394402 | Ross | Feb 1995 | A |
5396493 | Sugiyama | Mar 1995 | A |
5400326 | Smith | Mar 1995 | A |
5428615 | Backes et al. | Jun 1995 | A |
5434855 | Perlman et al. | Jul 1995 | A |
5442633 | Perkins et al. | Aug 1995 | A |
5444702 | Burnett et al. | Aug 1995 | A |
5448565 | Chang et al. | Sep 1995 | A |
5475781 | Chang et al. | Dec 1995 | A |
5481540 | Huang | Jan 1996 | A |
5485455 | Dobbins et al. | Jan 1996 | A |
5491694 | Oliver et al. | Feb 1996 | A |
5500860 | Perlman et al. | Mar 1996 | A |
5506838 | Flanagan | Apr 1996 | A |
5511168 | Perlman et al. | Apr 1996 | A |
5517494 | Green | May 1996 | A |
5517620 | Hashimoto et al. | May 1996 | A |
5519760 | Borkowski et al. | May 1996 | A |
5521910 | Matthews | May 1996 | A |
5530703 | Liu et al. | Jun 1996 | A |
5550816 | Hardwick et al. | Aug 1996 | A |
5553083 | Miller | Sep 1996 | A |
5583861 | Holden | Dec 1996 | A |
5606602 | Coyle et al. | Feb 1997 | A |
5608726 | Virgile | Mar 1997 | A |
5613069 | Walker | Mar 1997 | A |
5621793 | Bednarek et al. | Apr 1997 | A |
5634011 | Auerbach et al. | May 1997 | A |
5640452 | Murphy | Jun 1997 | A |
5659617 | Fischer | Aug 1997 | A |
5675582 | Hummel et al. | Oct 1997 | A |
5684800 | Dobbins et al. | Nov 1997 | A |
5727057 | Emery et al. | Mar 1998 | A |
5734865 | Yu | Mar 1998 | A |
5740171 | Mazzola et al. | Apr 1998 | A |
5742604 | Edsall et al. | Apr 1998 | A |
5745685 | Kirchner et al. | Apr 1998 | A |
5752003 | Hart | May 1998 | A |
5754657 | Schipper et al. | May 1998 | A |
5757916 | MacDoran et al. | May 1998 | A |
5781726 | Pereira | Jul 1998 | A |
5781737 | Schmidt | Jul 1998 | A |
5790074 | Rangedahl et al. | Aug 1998 | A |
5812819 | Rodwin | Sep 1998 | A |
5825772 | Dobbins et al. | Oct 1998 | A |
5862338 | Walker et al. | Jan 1999 | A |
5874964 | Gille | Feb 1999 | A |
5881236 | Dickey | Mar 1999 | A |
5892451 | May et al. | Apr 1999 | A |
5892910 | Safadi | Apr 1999 | A |
5892912 | Suzuki et al. | Apr 1999 | A |
5898686 | Virgile | Apr 1999 | A |
5905779 | Steinmetz | May 1999 | A |
5920699 | Bare | Jul 1999 | A |
5922073 | Shimada | Jul 1999 | A |
5963556 | Varghese et al. | Oct 1999 | A |
5983364 | Botcosh | Nov 1999 | A |
5999126 | Ito | Dec 1999 | A |
6005864 | Krause | Dec 1999 | A |
6006259 | Adelman et al. | Dec 1999 | A |
6012088 | Li et al. | Jan 2000 | A |
6018771 | Hayden | Jan 2000 | A |
6035105 | McCloghrie et al. | Mar 2000 | A |
6041166 | Hart et al. | Mar 2000 | A |
6044400 | Golan et al. | Mar 2000 | A |
6061797 | Jade et al. | May 2000 | A |
6070079 | Kuwahara | May 2000 | A |
6076114 | Wesley | Jun 2000 | A |
6078957 | Adelman et al. | Jun 2000 | A |
6085243 | Fletcher et al. | Jul 2000 | A |
6094434 | Kotzur et al. | Jul 2000 | A |
6105027 | Schneider et al. | Aug 2000 | A |
6105064 | Davis et al. | Aug 2000 | A |
6108365 | Rubin et al. | Aug 2000 | A |
6115754 | Landgren | Sep 2000 | A |
6122664 | Boukobza et al. | Sep 2000 | A |
6130890 | Leinwand et al. | Oct 2000 | A |
6131120 | Reid | Oct 2000 | A |
6151324 | Belser et al. | Nov 2000 | A |
6151631 | Ansell et al. | Nov 2000 | A |
6157647 | Husak | Dec 2000 | A |
6167275 | Oros et al. | Dec 2000 | A |
6167513 | Inoue et al. | Dec 2000 | A |
6192045 | Williams | Feb 2001 | B1 |
6192403 | Jong et al. | Feb 2001 | B1 |
6201789 | Witkowski et al. | Mar 2001 | B1 |
6205126 | Moon | Mar 2001 | B1 |
6212391 | Saleh et al. | Apr 2001 | B1 |
6216159 | Chintakrindi et al. | Apr 2001 | B1 |
6222840 | Walker et al. | Apr 2001 | B1 |
6230018 | Watters et al. | May 2001 | B1 |
6233242 | Mayer et al. | May 2001 | B1 |
6236365 | LeBlanc et al. | May 2001 | B1 |
6256338 | Jalloul et al. | Jul 2001 | B1 |
6259404 | Parl et al. | Jul 2001 | B1 |
6273622 | Ben-David | Aug 2001 | B1 |
6286044 | Aoyama et al. | Sep 2001 | B1 |
6304218 | Sugiura et al. | Oct 2001 | B1 |
6308273 | Goertzel et al. | Oct 2001 | B1 |
6317500 | Murphy | Nov 2001 | B1 |
6327474 | Ruutu et al. | Dec 2001 | B1 |
6327535 | Evans et al. | Dec 2001 | B1 |
6343317 | Glorikian | Jan 2002 | B1 |
6363422 | Hunter et al. | Mar 2002 | B1 |
6370629 | Hastings et al. | Apr 2002 | B1 |
6388618 | Stilp et al. | May 2002 | B1 |
6421009 | Suprunov | Jul 2002 | B2 |
6442394 | Valentine et al. | Aug 2002 | B1 |
6442616 | Inoue et al. | Aug 2002 | B1 |
6453237 | Fuchs et al. | Sep 2002 | B1 |
6456853 | Arnold | Sep 2002 | B1 |
6460084 | Van Horne et al. | Oct 2002 | B1 |
6466786 | Wallenius | Oct 2002 | B1 |
6480495 | Mauger et al. | Nov 2002 | B1 |
6523064 | Akatsu et al. | Feb 2003 | B1 |
6539229 | Ali | Mar 2003 | B1 |
6542813 | Kovacs | Apr 2003 | B1 |
6556831 | Buppelmann | Apr 2003 | B1 |
6580914 | Smith | Jun 2003 | B1 |
6583713 | Bates | Jun 2003 | B1 |
6640184 | Rabe | Oct 2003 | B1 |
6640248 | Jorgensen | Oct 2003 | B1 |
6665715 | Houri | Dec 2003 | B1 |
6701864 | Watson et al. | Mar 2004 | B2 |
6716101 | Meadows et al. | Apr 2004 | B1 |
6741863 | Chiang et al. | May 2004 | B1 |
6757545 | Nowak et al. | Jun 2004 | B2 |
6757740 | Parekh et al. | Jun 2004 | B1 |
6771639 | Holden | Aug 2004 | B1 |
6778818 | O'Neil | Aug 2004 | B1 |
6795688 | Plasson et al. | Sep 2004 | B1 |
6799049 | Zellner et al. | Sep 2004 | B1 |
6807427 | Sakamoto et al. | Oct 2004 | B1 |
6813501 | Kinnunen et al. | Nov 2004 | B2 |
6826385 | Kujala | Nov 2004 | B2 |
6826617 | Ansell et al. | Nov 2004 | B1 |
6834195 | Brandenberg et al. | Dec 2004 | B2 |
6859791 | Spagna et al. | Feb 2005 | B1 |
6889051 | Ogino et al. | May 2005 | B2 |
6889053 | Chang et al. | May 2005 | B1 |
6920329 | Kennedy et al. | Jul 2005 | B2 |
6934548 | Gould et al. | Aug 2005 | B1 |
6937988 | Hemkumar et al. | Aug 2005 | B1 |
6938096 | Greschler et al. | Aug 2005 | B1 |
6983313 | Korkea-Aho | Jan 2006 | B1 |
6985731 | Johnson et al. | Jan 2006 | B1 |
7010583 | Aizono et al. | Mar 2006 | B1 |
7089264 | Guido et al. | Aug 2006 | B1 |
7120449 | Muhonen et al. | Oct 2006 | B1 |
7136915 | Rieger, III | Nov 2006 | B2 |
7139829 | Wenzel et al. | Nov 2006 | B2 |
7197556 | Short et al. | Mar 2007 | B1 |
7266839 | Bowers et al. | Sep 2007 | B2 |
20010022558 | Karr, Jr. et al. | Sep 2001 | A1 |
20010039623 | Ishikawa | Nov 2001 | A1 |
20020010866 | McCullough et al. | Jan 2002 | A1 |
20020016831 | Peled et al. | Feb 2002 | A1 |
20020023010 | Rittmaster et al. | Feb 2002 | A1 |
20020034953 | Tricarico | Mar 2002 | A1 |
20020046073 | Indseth et al. | Apr 2002 | A1 |
20020051540 | Glick et al. | May 2002 | A1 |
20020052180 | Ravishankar et al. | May 2002 | A1 |
20020062379 | Widegren et al. | May 2002 | A1 |
20020063656 | Gutowski | May 2002 | A1 |
20020107029 | Caughran et al. | Aug 2002 | A1 |
20020122055 | Parupudi et al. | Sep 2002 | A1 |
20020138632 | Bade et al. | Sep 2002 | A1 |
20020164996 | Dorenbosch | Nov 2002 | A1 |
20020188842 | Willeby | Dec 2002 | A1 |
20030035544 | Herle et al. | Feb 2003 | A1 |
20030041167 | French et al. | Feb 2003 | A1 |
20030065571 | Dutta | Apr 2003 | A1 |
20030095509 | Ramanan et al. | May 2003 | A1 |
20030185233 | Ji et al. | Oct 2003 | A1 |
20030208523 | Gopalan et al. | Nov 2003 | A1 |
20040064334 | Nye | Apr 2004 | A1 |
20050199792 | Argast et al. | Sep 2005 | A1 |
Number | Date | Country |
---|---|---|
1154624 | Nov 2001 | EP |
9428683 | Dec 1994 | WO |
9705749 | Feb 1997 | WO |
9741654 | Nov 1997 | WO |
9819482 | May 1998 | WO |
0022862 | Apr 2000 | WO |
0044149 | Jul 2000 | WO |
0067450 | Nov 2000 | WO |
0069198 | Nov 2000 | WO |
0108425 | Feb 2001 | WO |
0122656 | Mar 2001 | WO |
0169956 | Sep 2001 | WO |
0176093 | Oct 2001 | WO |
0182259 | Nov 2001 | WO |
0194967 | Dec 2001 | WO |
0195505 | Dec 2001 | WO |
0209456 | Jan 2002 | WO |
0212914 | Feb 2002 | WO |
Number | Date | Country | |
---|---|---|---|
20030088698 A1 | May 2003 | US |
Number | Date | Country | |
---|---|---|---|
60332839 | Nov 2001 | US |