The present invention relates generally to computer systems. In particular, the present invention relates to computer systems that share resources such as storage.
Servers are typically used for big applications and work loads such as those used in conjunction with large web services and manufacturing. Often, a single server does not have enough power to perform the required application. Several servers may be used in conjunction with several storage devices in a storage area network (SAN) to accommodate heavy traffic. As systems get larger, applications often become important enough to avoid shutting off access to perform maintenance.
A typical server management system uses a single management control station that manages the servers and the shared storage. A potential problem of such a system is that it may have a single point of failure which can cause a shut-down of the entire storage area network to perform maintenance. Another potential problem is that there is typically no dynamic cooperation between the servers in case a change to the system occurs.
Such systems typically use large mainframes. A problem with the mainframes is that they are very expensive. Another possible system may use smaller computers but this solution typically requires customized hardware as well as a customized operating system that coordinates the computers to work as though it was one large machine with one operating system between them. Obtaining and maintaining customized hardware can be very costly.
What is needed is a system and method for a multi-node environment with shared storage provided at a lower price. The present invention addresses such a need.
The present invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements, and in which:
It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication links. It should be noted that the order of the steps of disclosed processes may be altered within the scope of the invention.
A detailed description of one or more preferred embodiments of the invention are provided below along with accompanying figures that illustrate by way of example the principles of the invention. While the invention is described in connection with such embodiments, it should be understood that the invention is not limited to any embodiment. On the contrary, the scope of the invention is limited only by the appended claims and the invention encompasses numerous alternatives, modifications and equivalents. For the purpose of example, numerous specific details are set forth in the following description in order to provide a thorough understanding of the present invention. The present invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the present invention is not unnecessarily obscured.
What is needed is a system and method for a multi-node environment that does not require customized hardware components. The present invention addresses such a need.
Servers 300A-300D can be any computer, preferable an off-the-shelf computer or server or any equivalent thereof. Servers 300A-300D can each run operating systems that are independent of each other. Accordingly, each server 300A-300D can, but does not need to, run a different operating system. For example, server 300A may run Microsoft windows, while server 300B runs Linux, and server 300C can simultaneously run a Unix operating system. An advantage of running independent operating systems for the servers 300A-300D is that the entire multi-node system can be dynamic. For example, one of the servers 300A-300D can fail while the other servers 300A-300D continue to operate.
The shared storage 306A-306D can be any storage device, such as hard drive disks, compact disks, tape, and random access memory. A filesystem is a logical entity built on the shared storage. Although the shared storage 306A-306D is typically considered a physical device while the filesystem is typically considered a logical structure overlaid on part of the storage, the filesystem is sometimes referred to herein as shared storage for simplicity. For example, when it is stated that shared storage fails, it can be a failure of a part of a filesystem, one or more filesystems, or the physical storage device on which the filesystem is overlaid. Accordingly, shared storage, as used herein, can mean the physical storage device, a portion of a filesystem, a filesystem, filesystems, or any combination thereof.
In this manner, there is redundancy in the system such that if any of the components or connections fail, the entire system can continue to operate.
In the example shown in
In this embodiment, the following components are shown:
The Distributed Lock Manager (DLM) 500 manages matrix-wide locks for the filesystem image 306a-306d, including the management of lock state during crash recovery. The Matrix Filesystem 504 uses DLM 500—managed locks to implement matrix-wide mutual exclusion and matrix-wide filesystem 306a-306d metadata and data cache consistency. The DLM 500 is a distributed symmetric lock manager. Preferably, there is an instance of the DLM 500 resident on every server in the matrix. Every instance is a peer to every other instance; there is no master/slave relationship among the instances.
The lock-caching layer (“LCL”) 502 is a component internal to the operating system kernel that interfaces between the Matrix Filesystem 504 and the application-level DLM 500. The purposes of the LCL 502 include the following:
1. It hides the details of the DLM 500 from kernel-resident clients that need to obtain distributed locks.
2. It caches DLM 500 locks (that is, it may hold on to DLM 500 locks after clients have released all references to them), sometimes obviating the need for kernel components to communicate with an application-level process (the DLM 500) to obtain matrix-wide locks.
3. It provides the ability to obtain locks in both process and server scopes (where a process lock ensures that the corresponding DLM (500) lock is held, and also excludes local processes attempting to obtain the lock in conflicting modes, whereas a server lock only ensures that the DLM (500) lock is held, without excluding other local processes).
4. It allows clients to define callouts for different types of locks when certain events related to locks occur, particularly the acquisition and surrender of DLM 500—level locks. This ability is a requirement for cache-coherency, which depends on callouts to flush modified cached data to permanent storage when corresponding DLM 500 write locks are downgraded or released, and to purge cached data when DLM 500 read locks are released.
The LCL 502 is the only kernel component that makes lock requests from the user-level DLM 500. It partitions DLM 500 locks among kernel clients, so that a single DLM 500 lock has at most one kernel client on each node, namely, the LCL 502 itself. Each DLM 500 lock is the product of an LCL 502 request, which was induced by a client's request of an LCL 502 lock, and each LCL 502 lock is backed by a DLM 500 lock.
The Matrix Filesystem 504 is the shared filesystem component of The Matrix Server. The Matrix Filesystem 504 allows multiple servers to simultaneously mount, in read/write mode, filesystems living on physically shared storage devices 306a-306d. The Matrix Filesystem 504 is a distributed symmetric matrixed filesystem; there is no single server that filesystem activity must pass through to perform filesystem activities. The Matrix Filesystem 504 provides normal local filesystem semantics and interfaces for clients of the filesystem.
SAN (Storage Area Network) Membership Service 506 provides the group membership services infrastructure for the Matrix Filesystem 504, including managing filesystem membership, health monitoring, coordinating mounts and unmounts of shared filesystems 306a-306d, and coordinating crash recovery.
Matrix Membership Service 508 provides the Local, matrix-style matrix membership support, including virtual host management, service monitoring, notification services, data replication, etc. The Matrix Filesystem 504 does not interface directly with the MMS 508, but the Matrix Filesystem 504 does interface with the SAN Membership Service 506, which interfaces with the MMS 508 in order to provide the filesystem 504 with the matrix group services infrastructure.
The Shared Disk Monitor Probe 510 maintains and monitors the membership of the various shared storage devices in the matrix. It acquires and maintains leases on the various shared storage devices in the matrix as a protection against rogue server “split-brain” conditions. It communicates with the SMS 506 to coordinate recovery activities on occurrence of a device membership transition.
Filesystem monitors 512 are used by the SAN Membership Service 508 to initiate Matrix Filesystem 504 mounts and unmounts, according to the matrix configuration put in place by the Matrix Server user interface.
The Service Monitor 514 tracks the state (health & availability) of various services on each server in the matrix so that the matrix server may take automatic remedial action when the state of any monitored service transitions. Services monitored include HTTP, FTP, Telnet, SMTP, etc. The remedial actions include service restart on the same server or service fail-over and restart on another server.
The Device Monitor 516 tracks the state (health & availability) of various storage-related devices in the matrix so that the matrix server may take automatic remedial action when the state of any monitored device transitions. Devices monitored may include data storage devices 306a-306d (such as storage device drives, solid state storage devices, ram storage devices, JOBDs, RAID arrays, etc.)and storage network devices 304′ (such as FibreChannel Switches, Infiniband Switches, iSCSI switches, etc.). The remedial actions include initiation of Matrix Filesystem 504 recovery, storage network path failover, and device reset.
The Application Monitor 518 tracks the state (health & availability) of various applications on each server in the matrix so that the matrix server may take automatic remedial action when the state of any monitored application transitions. Applications monitored may include databases, mail routers, CRM apps, etc. The remedial actions include application restart on the same server or application fail-over and restart on another server.
The Notifier Agent 520 tracks events associated with specified objects in the matrix and executes supplied scripts of commands on occurrence of any tracked event.
The Replicator Agent 522 monitors the content of any filesystem subtree and periodically replicates any data which has not yet been replicated from a source tree to a destination tree. Replication is preferably used for subtrees not placed in shared storage
The Matrix Communication Service 524 provides the network communication infrastructure for the DLM 500, Matrix Membership Service 508, and SAN Membership Service 506. The Matrix Filesystem 504 does not use the MCS 524 directly, but it does use it indirectly through these other components.
The Storage Control Layber (SCL) 526 provides matrix-wide device identification, used to identify the Matrix Filesystems 504 at mount time. The SCL 526 also manages storage fabric configuration and low level I/O device fencing of rogue servers from the shared storage devices 306a-306d containing the Matrix Filesystems 504. It also provides the ability for a server in the matrix to voluntarily intercede during normal device operations to fence itself when communication with rest of the matrix has been lost.
The Storage Control Layer 526 is the Matrix Server module responsible for managing shared storage devices 306a-306d. Management in this context consists of two primary functions. The first is to enforce I/O fencing at the hardware SAN level by enabling/disabling host access to the set of shared storage devices 306a-306d. And the second is to generate global (matrix-wide) unique device names (or “labels”) for all matrix storage devices 306a-306d and ensure that all hosts in the matrix have access to those global device names. The SCL module also includes utilities and library routines needed to provide device information to the UI.
The Pseudo Storage Driver 528 is a layered driver that “hides” a target storage device 306a-306d so that all references to the underlying target device must pass through the PSD layered driver. Thus, the PSD provides the ability to “fence” a device, blocking all I/O from the host server to the underlying target device until it is unfenced again. The PSD also provides an application-level interface to lock a storage partition across the matrix. It also has the ability to provide common matrix-wide ‘handles’, or paths, to devices such that all servers accessing shared storage in the Matrix Server can use the same path to access a given shared device.
Group membership is used herein to refer to a set of cooperating processes (programs) that form a communication group. For example, a group membership can be formed between servers. SAN membership is used herein to refer to a storage area membership. SAN membership refers to the group of servers that are communicating which are allowed to read/write the Storage area network resources such as disks and switches. Shared storage group membership is used herein to refer to the set of servers actively using a single filesystem located on the SAN. An example is the set of servers that have “mounted” the filesystem to make it accessible to user programs. Cluster membership refers to the set of servers forming a single cluster which actively share filesystems and network resources as a logical whole.
In this example, a membership of a cluster of servers is determined (700). As previously mentioned, servers are used as an example, however, any node, computer or processor can be used. A cluster, as used herein, can be any two or more servers, computers, processors, or any combination thereof, that is associated with a membership.
It is then determined whether the membership of the cluster has changed (702). At time zero, there will be no change to the membership since there is no history of the membership. At some later time, the membership may be changed, for example, if a server is taken off line, or a new server is added. In addition to the change in the numbers of servers that are utilized, determining whether the membership has changed can also accommodate a situation where a server has failed, a server has been added, or has been taken out of the cluster.
If the cluster membership has not changed (702), then each server sends messages to the other servers and monitors the messages to see if the other servers are active (704). It is then determined whether all servers respond (706). If all the other servers in the cluster respond to the messages, then the storage area network (SAN) membership service (SMS) is informed of the active members of the cluster (708). If, however, all servers in the cluster do not respond (706), then message activity is blocked (716). Blocking message activity avoids stale messages between membership changes. Once message activity is blocked, the membership of the cluster is again determined (700).
If the cluster membership has changed (702), then new locks are no longer granted (710). It is then determined whether there is an administrator (ADM) in this cluster (712). If there is no administrator in this cluster, then one of the members of the cluster is elected as administrator (714). In the example shown in
The administrator verifies that the other servers in this cluster are part of this storage area membership (720) of
There may be separate clusters operating in the storage area network either by design or by accident. In either case servers misconfigured software may result in a server attaching to the shared storage and attempting to access it, without knowledge of the valid cluster. If nodes are outside the cluster then the administrator excludes (fences) those servers to prevent corruption of data on shared storage. Servers that successfully gain membership to the network cluster are then allowed access to the shared storage and are then part of the SAN membership.
All cluster non-members are then excluded and all cluster members are allowed into the shared storage group (722). Accordingly, servers outside the cluster membership are excluded from accessing the discs 306A-306D of
Shared storage is then set to equal one (724). It is determined by the ADM whether recovery is necessary for a member of this shared storage group (726).
If no recovery is required for a member of this shared storage group, then it is determined whether this is the last shared storage (732). If it is not the last shared storage, then the shared storage is then set to shared storage plus one (730), and the next shared storage is evaluated by having the administrator determine whether recovery is necessary for a member of that shared storage (726). If, however, this shared storage was the last shared storage, then the granting of new locks are resumed for shared storages that are accessible (734).
If the administrator determines that recovery is necessary for a member of this shared storage (726), then it is determined which member(s) of this shared storage needs recovery (728). For example, the SAN membership service (SMS) of each server can tell their distributed lock manager (DLM) whether that server needs recovery.
The server that needs recovery is analyzed. In this example, the DLM for the member that needs recovery is set equal to one (750). Locks which can trigger journal recovery in the shared storage by the matrix file system (MFS) and the lock caching layer (LCL) are then recovered (752).
It is then determined whether recovery was successful (754). If the recovery was successful then it is determined whether this server was the last server that required recovery (762). If this server was not the last server that required recovery, then the next server that needed recovery is analyzed by setting DLM equals DLM plus one (764). The locks are then recovered for this next server (752). If, however, this server was the last server that needed recovery (762), then it is determined whether this shared storage is the last shared storage that needed to be analyzed (732 of
If recovery of the locks was not successful (754 of
The failed shared storage is then isolated and access to that shared storage is prevented (760). It is then determined whether this shared storage is the last shared storage that needs to be analyzed (732 of
A shared lock request is provided (802). For example, the matrix file system (MFS) 504 of
If there is a share storage failure, then the shared storage is isolated if necessary (808). If, however, there is no shared storage failure then the MFS asks the LCL for a shared lock (802). It is then determined whether the shared lock request is granted (804).
If the shared lock request is granted (804), then it is determined whether the server is able to read a piece of the requested file from the shared storage into the operating system buffer cache (810). If the server is able to read the file into the OS buffer cache then the process is complete. If, however, the server is unable to read into the OS buffer cache then the shared lock is dropped and an error is returned to the operating system or application (812). An example of when such an error might occur is if there is a disk failure or a connection failure.
In this example, the operating system of a server requests to write to a part of a file (900). The matrix file system (MFS) 504 of
If the request for an exclusive lock is granted (904), then it is determined whether there is a stable copy of the requested part of the file in the local cache (906). If there is a stable copy in the cache then the piece of file is modified in the local cache (934 of
It is then determined whether a request for a lock for this particular file has been received (936). If a request for a lock for this file has been received, and the file has been modified, then the modified data is written to the shared storage (940). The server then gives up the exclusive lock on this file (938).
If no request has been received for a lock for this file (936), then it is determined whether the operating system is cleaning out the local cache (942). If the cache is being cleaned, then the modified data is written to the shared storage (940) and any locks are maintained unless there is an outstanding request from another server. Otherwise, the modified file is kept in the local cache (944). By keeping the modified file in the local cache until it is needed by another server, access to the file is made faster for the server that maintains the file in its local cash.
In this example, it is determined whether there is an administrator (ADM) in the cluster (1400). The cluster includes the set of servers that cooperate to share a shared resource such as the shared storage. One of the servers in the cluster act as an administrator to manage the cluster. If there is no administrator in the cluster, then it is determined whether this server can try to become the administrator (1408). If this server can try to become the administrator then the server reads the device information from the database and it is now the new administrator (1410).
If there is an administrator in the cluster (1400), or if this server can not become the new administrator (1408), then it asks the existing administrator to be imported into the cluster (1402). An example of how this server can not become the administrator (1408) is if another server became the administrator during the time this server established that there was no administrator and then tried to become the administrator.
It is then determined whether it is permissible for this server to be imported into the cluster (1404). If it is not permissible then the process of adding this server to the cluster has failed (1412). Examples of reasons why adding the server would fail include this server not being healthy or having a storage area network generation number mismatch with the generation number used by the administrator.
If this server can be imported (1404), then it receives device names from the administrator (1406). Examples of device names include names of shared storage.
The administrator grants physical storage area network access to this server (1410 of
The storage area network generation number is then updated and stored to the database (1710). Thereafter, normal operation continues (1712).
If there is no longer an administrator (1702), then a server is picked to try to become the new administrator (1704). There are several ways to select a server to try to become the new administrator. One example is a random selection of one of the servers. The elected server is then told to try to become the new administrator (1706). One example of how the server is selected and told to become the new administrator is through the use of a group coordinator.
In one embodiment, the group coordinator is elected during the formation of a process communication group using an algorithm that can uniquely identify the coordinator of the group with no communication with any server or node except that required to agree on the membership of the group. For example, the server with the lowest numbered Internet Protocol (IP) address of the members can be selected. The coorindator can then make global decisions for the group of servers, such as the selection of a possible administrator. The server selected as administrator is preferably one which has a high probability of success of actually becoming the administrator. The group coordinator attempts to place the administrator on a node which might be able to connect the SAN hardware and has not recently failed in an attempt to become the SAN administrator.
The selected server then attempts to acquire the storage area network locks (1720). If it cannot acquire the SAN locks, then it has failed to become the administrator (1724). If it succeeds in acquiring the SAN locks (1720), then it attempts to read the SAN generation number from the membership database (1722). The database can be maintained in one of the membership partitions on a shared storage and can be co-resident with the SAN locks
If the server fails to read the SAN generation number from the database (1722), then it drops the SAN locks (1726), and it has failed to become the administrator (1724). Once the server has failed to become the administrator (1724), the group coordinator tells a server to try to become the new administrator (1706
If the server can read the SAN generation number from the database, then it increments the SAN generation number and stores it back into the database (1728). It also informs the group coordinator that this server is now the administrator (1730). The group coordinator receives the administrator update (1732). It is then determined if it is permissible for this server to be the new administrator (1750). If it is not okay, then a message to undo the administrator status is sent to the current server trying to become the administrator (1752). Thereafter, the group coordinator tells another server to try to become the new administrator (1706 of
If it is okay for this server to be the new administrator, the administrator is told to commit (1754), and the administrator is committed (1756). The coordinator then informs the other servers in the cluster about the new administrator (1758).
The SAN generation number is then incremented (1604). Each server in the cluster is then informed of the SAN generation number and the addition or deletion of the new disk (1606). When all the servers in the cluster acknowledge, the new SAN generation number is written to the database (1608). The requesting server is then notified that the addition/removal of the disk is complete (1610).
Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. It should be noted that there are many alternative ways of implementing both the process and apparatus of the present invention. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may modified within the scope and equivalents of the appended claims.
This application claims priority to U.S. Provisional Patent Application No. 60/324,196 entitled SHARED STORAGE LOCK: A NEW SOFTWARE SYNCHRONIZATION MECHANISM FOR ENFORCING MUTUAL EXCLUSION AMONG MULTIPLE NEGOTIATORS filed Sep. 21, 2001, which is incorporated herein by reference for all purposes. This application claims priority to U.S. Provisional Patent Application No. 60/324,226 entitled JOUNALING MECHANISM WITH EFFICIENT, SELECTIVE RECOVERY FOR MULTI-NODE ENVIRONMENTS filed Sep. 21, 2001, which is incorporated herein by reference for all purposes. This application claims priority to U.S. Provisional Patent Application No. 60/324,224 entitled COLLABORATIVE CACHING IN A MULTI-NODE FILESYSTEM filed Sep. 21, 2001, which is incorporated herein by reference for all purposes. This application claims priority to U.S. Provisional Patent Application No 60/324,242 entitled DISTRIBUTED MANAGEMENT OF A STORAGE AREA NETWORK filed Sep. 21, 2001, which is incorporated herein by reference for all purposes. This application claims priority to U.S. Provisional Patent Application No. 60/324,195 entitled METHOD FOR IMPLEMENTING JOURNALING AND DISTRIBUTED LOCK MANAGEMENT filed Sep. 21, 2001, which is incorporated herein by reference for all purposes. This application claims priority to U.S. Provisional Patent Application No. 60/324,243 entitled MATRIX SERVER: A HIGHLY AVAILABLE MATRIX PROCESSING SYSTEM WITH COHERENT SHARED FILE STORAGE filed Sep. 21, 2001, which is incorporated herein by reference for all purposes. This application claims priority to U.S. Provisional Patent Application No. 60/324,787 entitled A METHOD FOR EFFICIENT ON-LINE LOCK RECOVERY IN A HIGHLY AVAILABLE MATRIX PROCESSING SYSTEM filed Sep. 24, 2001, which is incorporated herein by reference for all purposes. This application claims priority to U.S. Provisional Patent Application No. 60/327,191 entitled FAST LOCK RECOVERY: A METHOD FOR EFFICIENT ON-LINE LOCK RECOVERY IN A HIGHLY AVAILABLE MATRIX PROCESSING SYSTEM filed Oct. 1, 2001, which is incorporated herein by reference for all purposes. This application is related to co-pending U.S. patent application Ser. No. 10/251,689 entitled A SYSTEM AND METHOD FOR SYNCHRONIZATION FOR ENFORCING MUTUAL EXCLUSION AMONG MULTIPLE NEGOTIATORS filed concurrently herewith, which is incorporated herein by reference for all purposes; and U.S. patent application Ser. No. 10/251,626 entitled SYSTEM AND METHOD FOR JOURNAL RECOVERY FOR MULTINODE ENVIRONMENTS filed concurrently herewith, which is incorporated herein by reference for all purposes; and co-pending U.S. patent application Ser. No. 10/251,690 entitled A SYSTEM AND METHOD FOR MANAGEMENT OF A STORAGE AREA NETWORK filed concurrently herewith, which is incorporated herein by reference for all purposes; and co-pending U.S. patent application Ser. No. 10/251,894 entitled SYSTEM AND METHOD FOR IMPLEMENTING JOURNALING IN A MULTI-NODE ENVIRONMENT filed concurrently herewith, which is incorporated herein by reference for all purposes; and co-pending U.S. patent application Ser. No. 10/251,895 entitled A SYSTEM AND METHOD FOR A MULTI-NODE ENVIRONMENT WITH SHARED STORAGE filed concurrently herewith, which is incorporated herein by reference for all purposes; and co-pending U.S. patent application Ser. No. 10/251,893 entitled A SYSTEM AND METHOD FOR EFFICIENT LOCK RECOVERY filed concurrently herewith, which is incorporated herein by reference for all purposes.
Number | Name | Date | Kind |
---|---|---|---|
5276872 | Lomet et al. | Jan 1994 | A |
5438464 | Lewis et al. | Aug 1995 | A |
5678026 | Vartti et al. | Oct 1997 | A |
5751992 | Bhargava et al. | May 1998 | A |
5813016 | Sumimoto | Sep 1998 | A |
5850507 | Ngai et al. | Dec 1998 | A |
5909540 | Carter et al. | Jun 1999 | A |
5913227 | Raz et al. | Jun 1999 | A |
5920872 | Grewell et al. | Jul 1999 | A |
5953719 | Kleewein et al. | Sep 1999 | A |
5960446 | Schmuck et al. | Sep 1999 | A |
5974250 | Angelo et al. | Oct 1999 | A |
5987506 | Carter et al. | Nov 1999 | A |
6009466 | Axberg et al. | Dec 1999 | A |
6016505 | Badovinatz et al. | Jan 2000 | A |
6021508 | Schmuck et al. | Feb 2000 | A |
6026426 | Badovinatz et al. | Feb 2000 | A |
6026474 | Carter et al. | Feb 2000 | A |
6044367 | Wolff | Mar 2000 | A |
6108654 | Chan et al. | Aug 2000 | A |
6154512 | Homan | Nov 2000 | A |
6163855 | Shrivastava et al. | Dec 2000 | A |
6226717 | Reuter et al. | May 2001 | B1 |
6256740 | Muller et al. | Jul 2001 | B1 |
6269410 | Spasojevic | Jul 2001 | B1 |
6272491 | Chan et al. | Aug 2001 | B1 |
6370625 | Carmean et al. | Apr 2002 | B1 |
6421723 | Tawil | Jul 2002 | B1 |
6530004 | King et al. | Mar 2003 | B1 |
6633904 | Shinkai | Oct 2003 | B1 |
20010042221 | Moulton et al. | Nov 2001 | A1 |
20020069340 | Tindal et al. | Jun 2002 | A1 |
20020091854 | Smith | Jul 2002 | A1 |
20020101946 | Hartman et al. | Aug 2002 | A1 |
20020150126 | Kovacevik | Oct 2002 | A1 |
20020161855 | Manczak et al. | Oct 2002 | A1 |
20020188590 | Curran et al. | Dec 2002 | A1 |
Number | Date | Country |
---|---|---|
H9-114721 | Oct 1995 | JP |
2001-084168 | Sep 1998 | JP |
H11-143843 | May 1999 | JP |
WO 0062502 | Oct 2000 | WO |
Number | Date | Country | |
---|---|---|---|
20030065686 A1 | Apr 2003 | US |
Number | Date | Country | |
---|---|---|---|
60327191 | Oct 2001 | US | |
60324787 | Sep 2001 | US | |
60324243 | Sep 2001 | US | |
60324195 | Sep 2001 | US | |
60324242 | Sep 2001 | US | |
60324224 | Sep 2001 | US | |
60324226 | Sep 2001 | US | |
60324196 | Sep 2001 | US |