Patent Grant
7702791
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
The invention relates generally to data replication and specifically to providing redundancy for a client network session.
When a client connects to a server on a network and begins a session, there can be information stored on the server that is particular to that client session. For example, a user of the client might place items in a virtual shopping cart. The selection of those items can be stored, at least temporarily, on the server. In this example, there is no need for other users or servers to have access to this information. It is desirable, however, that this data be highly available across a network or server cluster such that if the server storing the session data fails, it is possible to recover the data on another server.
One way to accomplish data recovery in such a situation is to store the information in a database during the session, although the information could also be stored by other means such as in a data file. Every time a change is made to the session data, an update is written to the database such that the data is accessible to every server having access to the database. The data is stored in a persistent place, and can easily be retrieved by another server.
A problem exists with this approach, however, in that it is fairly expensive to fetch session information from the database for each request. The multiple hits to the database can create a bottleneck and bog the system down to the point where it is basically inoperable, as the throughput of the system can depend on the number of database connections from the server. Also, these sessions may contain the type of information, to which users want quick access. With some applications, it is possible for there to be thousands of clients working simultaneously, resulting in thousands of concurrent sessions. Some servers are expected to host many different applications, which further increases the number of sessions that may need to be hosted.
It is desirable to improve the speed and efficiency of such a system so that these tens of thousands of users may use the system effectively. One way to avoid such a bottleneck is to assume that the servers will be up and running 99.9% of the time and simply neglect to backup any information. This may be the solution providing the fastest user experience, but even 0.1% downtime resulting in data loss is unacceptable to many users.
Systems in accordance with the present invention can utilize a primary server to serve requests from a network client such as a web browser. The primary server can be selected from a pool of servers or server cluster. Once a primary server is selected, a client request can be served on that primary. A secondary server can then be chosen, such as by the primary server. Once the primary server responds to the request, the information relating to the session is sent from the primary server to the secondary server. This can be a full set of information for a first request on a session, or can simply be an update to existing information in a session in response to subsequent requests. Information identifying the primary and secondary servers can be stored on the client, such as a “token” that is stored as a cookie or passed on top of standard RMI in a manner similar to a transaction or security context. This identification information or “token” can accompany each request.
A system can take advantage of load balancing using either hardware or software. In a process useful with software load balancing, a request may be received on a session for which a primary and secondary server have already been selected. An attempt is made to serve the request on the primary server. If the primary is unable to receive or respond to the request, the request can be served on the secondary server. If the secondary server receives the request, the secondary server becomes the new primary server. A new secondary server can be selected and sent the session information from the new primary server in order to maintain redundancy.
In a process useful with a hardware load balancer, a request is received on a session for which a primary and secondary server have been selected. An attempt is then made to serve the request on the primary server. If the primary server is unable to receive or respond to the request, the hardware load balancer can select a new primary server and attempt to serve the request on the new primary server, instead of using the secondary server. The session information can be sent from the secondary server to the new primary server, such as in response to a request from the new primary. The new primary server can then respond to the request, and send updated session information to the secondary server, so that the servers are in sync with regard to that session.
The present invention overcomes many of the deficiencies of prior art replication systems. In one system in accordance with an embodiment of the present invention, a session is created when a client makes a request to a server on a network, such as a local area network (LAN), ethernet, or Internet. The server receiving the request can consist of any server that may be used to store information for a session and/or generate a response to a session request, such as for example an application server, a web server, an object server, or a servlet engine. The server ultimately receiving the request becomes the “primary” server, or the server to which the client will send future requests. The system can then choose a “secondary” server for that session, which acts as a source for redundancy.
Each time an update is made in that session, the change may not only be stored on the primary server, but can also be sent, such as by remote call, to the secondary server. The entire set of session data need not be forwarded to the secondary server each time a change is made, but only that data or information that has changed, such as can be sent in a delta or packet of information. Sending the minimum amount of necessary information in the delta can improve overall system efficiency. The replication acts like a mirroring system, except for the fact that it acts on session data. This mirroring can be done, in one example, for web applications using a servlet engine.
When a client connects to a server, a session object can be created that is associated with the client or user. The session object can be maintained on the primary server for the duration of the session or can be timed out after a specified period of time. Each session object can be given a unique identifier or identification number to identify the client and/or object to the server. The server chosen to serve the request can act as the primary server for the duration of the session. The primary server can choose a secondary server for the session object, such that each time the object is updated the update is also stored on the secondary server. The secondary server can be optimized to receive only minimum information or to batch updates, in order to improve the efficiency of the system.
One web-based system 100 in accordance with an embodiment of the present invention is shown in
The object server in this example then chooses a secondary server for the session. In an alternative embodiment, the plug-in can be used to choose a secondary server. The plug-in can also use load balancing for the decision.
The object server passes the data to the secondary server, and lets the secondary server know it will be the backup. The object server then creates a cookie to be sent to, and stored by, the client. The cookie contains an identification of the primary and secondary servers used for the session.
When a subsequent request is sent by the client on the same session, it does not matter which web server receives the request. The web server will look at the cookie to determine the primary server for that session, and will then route the request to that primary server.
Assume an example, as shown in
In some embodiments, the secondary server or web server actively monitors the primary server in order to determine the status of the primary server. This monitoring can be done by any appropriate manner, such as by continually or periodically “pinging” the primary server to determine whether it is connected to the network. If it is determined that the primary server is unable to accept requests, the secondary server can become the new primary server. A new secondary server can then be selected. One advantage of such a design is that the window of time in which a dual server failure could result in session state loss is narrowed. While some embodiments allow the window to be defined by the client request rate, this approach would allow the window to be defined by the rate of server pinging.
The new primary and secondary servers are similarly responsible for information pertaining to the session. The server that was previously the primary server may no longer have any responsibility or information for that session, even if that server becomes able to accept and process requests while the session is still current. The secondary server may automatically change its state, such that it becomes the new primary server for the session, but it may choose not to assign a new secondary server until the new primary server receives a request.
It may be undesirable to actively create a new secondary server or backup the session information on a secondary server, as it may not be known whether the new primary server will receive another request. Creating a new secondary server or backing up information that will not get used can unnecessarily waste resources. The session may alternatively be short lived, and may not “live” long enough to receive a subsequent request. Each session typically has a time-out value, such that if the session is inactive for a specified period of time it will “time-out” or “die”, the session will be ended, and all data stored for the session may be deleted to conserve memory. In such a case, not only might the creation of a secondary server waste resources, but it may also require unnecessary “clean-up” work to remove the session information from the new secondary server.
The primary and/or secondary server can be chosen according to an algorithm, which may have as options, for example, any server in a specified server cluster. While it may be efficient for an algorithm to choose the primary and secondary server for each session, there are cases where the input of an administrator may be desirable. For instance, it may be possible that multiple servers are located on one machine. If an algorithm is choosing servers, such as an algorithm based on load, the algorithm may select two servers on the same machine. In the event of a machine failure, both servers may be unavailable and the session data may be unavailable and/or lost. An administrator, however, can choose to specify primary and secondary servers that are on different machines. This can provide for redundancy not only across servers, but across machines as well.
Alternatively, it may be possible to build a parameter in the algorithm itself that, when doing a load-balancing analysis, takes into account the machine on which a server is located. If the server with the lowest current load is on the same machine as the primary server, the algorithm can choose to go to the server with the lowest load that is on a different machine. This approach can be expanded to any level of separation, such as servers in different rooms, different buildings, or different cities.
In order to allow servers in a cluster to function independently, the servers can be loosely coupled. To achieve this loose coupling, each server in the cluster can be configured to detect the status of other cluster servers such that action can be taken when a server leaves the cluster, either voluntarily or involuntarily. In one embodiment, servers can rely on the underlying operating system to monitor the status of the cluster servers. Other embodiments can require the servers to do monitoring. Embodiments where cluster servers do not have to participate in cluster monitoring may be preferred, as the servers' resources are available to improve the overall throughput of the system.
In
At the first virtual barrier, it is possible that a browser may not be able to get to a specific web server. This may not be a problem in a system in accordance with the present invention, however, because the information relating to the primary and secondary servers may already be stored in a cookie in the browser. The browser can contact any web server on the network, because the browser can indicate to the web server, through the cookie, which server should receive the request. While the system may be most efficient on a local area network (LAN), a similar approach can be used on any capable network. For instance, it may be possible for the browser to contact a second web server and/or backend servers over the Internet that might be located in separate buildings from the first web server.
Depending on the application, the primary and secondary servers can be of several different server types, such as web servers, servlet engines, or Enterprise Java bean (“ejb”) engines. It may still be possible for each server in a cluster to be separate and specialized, such as being of different server types, but still be capable of acting as a primary and/or secondary server.
If clustering is enabled on a system in accordance with the present invention, it may be possible to transparently add new servers to the system to act as additional primary and secondary servers. Clustering is, generally, an approach to server management that allows management of a set of servers by establishing a “managing” server in that set of servers. This approach can simplify the deployment and synchronization of potentially diverse components among the servers in the cluster. Clustering can substantially improve system reliability and scalability.
When clustering with a system in accordance with the present invention, each server in a cluster can be configured to detect a new server entering the cluster and designate that new server as a secondary server to any existing primary server. The method used for load balancing may can immediately designate the new server as a primary or secondary server.
Systems in accordance with the present invention may alternatively utilize a hardware load balancer to direct incoming requests. In an Internet setting, for example, a hardware load balancer can sit on the network with an IP (Internet Protocol) address. Incoming requests from browsers or clients can be directed to that IP address. The hardware load balancer can then redirect those requests to other IP addresses, or other servers each assigned an IP address, located in the system but “behind” the hardware load balancer. In this way, it appears to the browser as if the request is always going to the same IP address, when in fact it may be going to multiple servers behind that IP address. The hardware load balancer can be aware of all servers located behind it in the network, such as may be the result of hardwiring the servers to the hardware load balancer, instead of utilizing another method such as software clustering.
There may be advantages to using a hardware load balancer. A hardware load balancer can utilize better algorithms for load balancing than other approaches. The hardware load balancer may be able to detect node failures, so that those nodes may be pulled out of the list of servers available to the algorithm. This node removal can prevent the algorithm from trying to go to servers that may not be reachable, even though those particular servers might not yet have been sent a request.
A system in accordance with the present invention can also use a Domain Name System (DNS) protocol, such as DNS Round Robin, instead of using a hardware load balancer to map a domain name to several IP addresses, or to redirect requests sent to a web server to several object servers. DNS, however, does not typically determine or detect whether those IP addresses are actually “live”.
A hardware load balancer can be used to proxy certain types of requests to specific servers or server clusters, depending on whether the request requires dynamic page generation or whether the request is for a static page. In
While it may be desirable to optimize a hardware load balancer 414 for use with the present invention, it may be undesirable to require physical changes to the load balancer itself. It may also be undesirable for a hardware load balancer to have to read cookies and figure out that if the first primary 404 fails, the request needs to be redirected to the secondary server 408 indicated in the cookie stored on the browser 402. It may, however, be desirable to have the load balancer direct the request where the load balancer wants, and then make sure the system recovers appropriately.
In one such approach, the hardware load balancer 414 tends to send requests from a browser 402 or client to one server, based on some arbitrary information stored in the cookie on the web browser. For example, a cookie can have an initial string of information, followed by a segment of information related to the primary and secondary servers, as well as a session identifier used for replication. The hardware load balancer 414 can be configured to look only at this segment of information. If this segment of information does not change between successive cookies, the load balancer may keep redirecting the requests back to the primary server 404. Such “session stickiness” can also be based on other appropriate schemes, such as may utilize the IP address of the client.
The segment of information in the cookie can remain the same as long as the requests can go back to the primary server. If the primary server fails for any reason, the second server can assign itself to be the new primary server. The new primary server can then insert new information in the segment for a new secondary server, which can be selected by the new primary server or load-balancing mechanism. Alternatively, the hardware load balancer can choose a new primary server and redirect the request to the new primary server.
The first request of a session that is received by the load balancer may not be hard coded to go to one server primarily. The decision to “stick” to one server can be made after the first request is made and comes back from the object server or other backend server. Hardware load balancers can be smart enough to do this “simple stickiness”, or to return primarily to the assigned primary server with which the load balancer has a connection.
If no cookie exists, the hardware load balancer can be configured to use any of a number of load balancing methods, such as may be based on load or response time. The load balancer can then select a server, such as in the appropriate cluster, and direct the request to that server. When that primary server responds to the request, the server can send the browser a cookie containing the segment of information relating to the primary and secondary servers. Each subsequent request from that browser that is received by the hardware load balancer can have that cookie associated with it, such that the load balancer can associate that request with the primary server.
The system still may not be able to guarantee that a request will go to the primary server. As shown in
If the server 412 chosen by the load balancer 414 is not the second server 408, the chosen server 412 may realize that the request is a request on a session that it is not hosting. In this case, the chosen server 412 can look to the cookie in order to determine the secondary server 408.
Once the chosen server 412 has located the secondary server 408, the chosen server 412 can request session state information 410 from the secondary server 408. The chosen server 412 can then transform itself into the new primary server for the session. The secondary server 408 in this case can remain the same. The cookie is updated so the load balancer 414 will keep directing the requests to the new primary server 412.
In the event that the load balancer 414 chooses to direct the request to a new server that happens to be the secondary server 408, the secondary server can set itself as the new primary server and a new secondary server can be selected.
A system with a hardware load balancer having a servlet cluster behind it can provide a fast data path. If a web server does the routing, it may be necessary for the request to come up into the software where some code is executed, and then be sent back out onto the network. The load balancer/servlet cluster system does everything at a low, protocol level so it may be comparatively very fast.
It may be advantageous to have the load balancing algorithms as localized as possible. In the hardware load balancer case, it may only be necessary to ensure that the software on the server is operating properly, such as software that may be written in Java. In systems without a hardware load balancer, it may be necessary to make sure that each special plug-in of every web server in the system works as well.
It may also be necessary to support plug-ins for different platforms. A hardware load balancer can work equally well with systems based on differing platforms, such as a Netscape Application Server (NAS), WebLogic Server™ (WLS), Microsoft® Internet Information Server (IIS), or Apache HTTP Server. With the hardware load balancer, the system can be reduced by one level of complexity, as one of the banks may be removed. This is shown in
Some of the systems described above can utilize servlets for web access. A similar mechanism can be used for accessing stateful session beans, a type of Enterprise Java bean (“ejb”). While servlets can be used to service requests from browser clients, ejb servers can be used to support requests from Java clients.
With Java clients, there can be a single, persistent connection for the entire duration of a session. There may then be no need (or support) for cookies. Also, since a persistent connection exists, there may no longer be a need for a load balancer. A Java client can connect to one of the backend servers using, for example, DNS or a load balancer. The Java client can then look up a “handle” to a stateful session. A handle in Java is similar to a pointer, which may be used to locate the appropriate session.
Referring to the system 500 of
The stateful session bean 508 can pass this information back to the Java client 502, similar to sending a cookie, using an RMI (Java Remote Method Invocation) protocol. Extra information can be placed “on top” of the standard RMI in order to make this cookie simulation work, similar to the way transaction context propagation works. The primary/secondary server identifier pair can be passed back to the Java client 502 with every response. Each time the Java client 502 makes a call, it can call through an interface 504 into special RMI code adapted to continue to make calls to the primary server 506 for that session. If the primary fails, the Java client 502 can look at the information regarding the location of the secondary server 510, and can instead make a request to the secondary server. It may be preferable, for efficiency, if only the essential information regarding server identification is passed back on top of the RMI.
The Java client 502 may always know which server is the secondary server 510. The Java client can have much of the same logic that a proxy might have, such as always knowing to go to the secondary server 510 if the primary server 506 is unavailable. The Java client can monitor server health in order to avoid sending requests to unavailable servers. In the event that the secondary server 510 becomes the new primary server, a new secondary server 514 can still be chosen. The logic for selecting a new primary and/or secondary server can be similar to that described above. The Java client can immediately update to the new primary/secondary servers.
In accordance with the above discussion, systems in accordance with the present invention can generally follow one of two branching paths, although variations including those mentioned above may be utilized. A common part of such a path is shown in
From this point, the process branches off into a path that can be useful for software load balancing, and a process that can be useful for hardware balancing.
Another path, useful for systems with hardware load balancers, is shown in
To maintain the consistency in any embodiment of the present invention, a change in session data can be associated with a version number. The primary and secondary servers may each know which version of the session it is storing. A server can be instructed to modify the data only if it receives a request that has a version number later, or higher, than the one it is currently storing. The primary and secondary servers can periodically check each other to ensure that they are both on the same version number. The version number can use a method as simple as incrementing a number to guarantee ordering. For session information to remain consistent, it may be desirable for the primary and secondary servers to be in synchronization. When a version number is out of synchronization, the primary server can choose to send the entire session information to the secondary server to bring the session back into synchronization. The synchronization also facilitates the ability of the servers to switch roles between primary and secondary if the need arises.
If the primary server is unable to update information on the secondary server, for reasons such as a bad connection, it may be possible that the primary server will keep updating and the secondary server will be unaware of any updates. It may then be possible for the primary server to be several versions ahead of the secondary server. Once the primary server is again able to send information to the secondary server, a delta between two successive versions may not work. In such a case, the primary server can send an entirely new set of session data to the secondary server in order to make the data session consistent across both servers. In this case, the secondary server either gets a delta between successive versions, or it gets all the data for the entire session. In other embodiments, it may be possible to generate a delta between arbitrary versions in order to bring the secondary server up to the current version.
In simulating cookies for tracking Java states, large random numbers can be used for server identification. The numbers can be large enough that it is highly unlikely that the sum of two different pairs of identification numbers will be the same. It may be possible to only send these two numbers back to the Java client, and the server pair can be identified by adding the two numbers together to get a new number. This allows the passing of only one number to identify two servers, which can improve efficiency. Since the Java client can have a persistent connection with a specific server, the client can identify the second server by subtracting the identification number of the primary server from the summed number being passed, in order to arrive at the identification number of the second server.
Java objects such as session beans may, however, be stateless or have a transient state, as opposed to the persistent state discussed above. If a Java session bean is stateless, the bean may not be able to maintain session information between invokes, or successive requests. If the session information is stored elsewhere, the stateless beans can load the session information temporarily in order to serve the request. Failover, or turning session control over to a new primary server having replicated session information, can only occur where there was a clean invoke failure, such as where the primary server never received the request, the request was transactional and was aborted, or was a onetime-only request. If the session bean is transient, on the other hand, instances can be created by a stateless factory with stateless load balancing and failover. The bean in a transient state may either not be backed up or may be backed up in-memory using primary/secondary replication as discussed above.
Batch updates can be used to improve the throughput of the system with an increased failure window. When batching, or “boxcarring,” several requests are sent together as one large request in order to improve efficiency and scalability. The batching of requests can be based upon any of a number of criteria, such as time intervals or numbers of requests. For instance, a system can send a batch of requests every 10 seconds or for every 100 individual session update messages. The system can also accommodate both criteria, sending a batch when either 10 seconds has passed since the last batch or when 100 requests are received, whichever comes first. Batching may cause the system to no longer be as reliable as synchronous updates, but can improve the overall system scalability.
The criteria may also be configurable, such as by a user or an administrator. Configurable criteria can be appropriate in situations where a system encounters a lot of traffic at certain times, but little to no traffic at other times. Configurable criteria can, for example, allow batching of every 100 messages at peak time, but no batching at all during off time, so that every request is sent in a reasonable amount of time.
A system administrator can also choose to pair two servers in a cluster as primary and secondary servers. The input of the administrator can be desirable in order to improve the overall fault tolerance of the system. For example, multiple servers can be located on one physical machine and an algorithm might choose to place both primary and secondary servers on the same machine. The session information could then be lost entirely if that machine fails. In order to prevent the loss of session information from machine failures, an administrator can choose to specify a primary server and a secondary server each on a separate physical machine. The administrator can also choose a primary server based on various load-balancing schemes. Examples of possible schemes are based on server load, connection number, and physical proximity.
The foregoing description of preferred embodiments of the present invention has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Obviously, many modifications and variations will be apparent to the practitioner skilled in the art. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications that are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalence.
This application claims priority to U.S. Provisional patent application No. 60/305,969, filed Jul. 16, 2001, entitled HARDWARE LOAD-BALANCING APPARATUS FOR SERVLET SESSION REPLICATION, incorporated herein by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4714996 | Gladney et al. | Dec 1987 | A |
| 5163148 | Walls | Nov 1992 | A |
| 5509070 | Schull | Apr 1996 | A |
| 5612865 | Dasgupta | Mar 1997 | A |
| 5613060 | Britton et al. | Mar 1997 | A |
| 5634052 | Morris | May 1997 | A |
| 5644629 | Chow | Jul 1997 | A |
| 5742668 | Pepe et al. | Apr 1998 | A |
| 5748882 | Huang | May 1998 | A |
| 5765171 | Gehani et al. | Jun 1998 | A |
| 5774689 | Curtis et al. | Jun 1998 | A |
| 5796934 | Bhanot et al. | Aug 1998 | A |
| 5805798 | Kearns et al. | Sep 1998 | A |
| 5813017 | Morris | Sep 1998 | A |
| 5819107 | Lichtman et al. | Oct 1998 | A |
| 5828847 | Gehr et al. | Oct 1998 | A |
| 5909689 | VanRyzin | Jun 1999 | A |
| 5910180 | Flory et al. | Jun 1999 | A |
| 5926775 | Brumley et al. | Jul 1999 | A |
| 5933606 | Mayhew | Aug 1999 | A |
| 5951694 | Choquier et al. | Sep 1999 | A |
| 5991804 | Bolosky | Nov 1999 | A |
| 6018805 | Ma et al. | Jan 2000 | A |
| 6055243 | Vincent et al. | Apr 2000 | A |
| 6065046 | Feinberg et al. | May 2000 | A |
| 6122629 | Walker et al. | Sep 2000 | A |
| 6134673 | Chrabaszcz | Oct 2000 | A |
| 6163801 | O'Donnell et al. | Dec 2000 | A |
| 6173293 | Thekkath et al. | Jan 2001 | B1 |
| 6173327 | De Borst et al. | Jan 2001 | B1 |
| 6185601 | Wolff | Feb 2001 | B1 |
| 6189046 | Moore et al. | Feb 2001 | B1 |
| 6195680 | Goldszmidt et al. | Feb 2001 | B1 |
| 6199110 | Rizvi et al. | Mar 2001 | B1 |
| 6212521 | Minami et al. | Apr 2001 | B1 |
| 6212556 | Arunachalam | Apr 2001 | B1 |
| 6243753 | Machin et al. | Jun 2001 | B1 |
| 6256634 | Moshaiov et al. | Jul 2001 | B1 |
| 6269373 | Apte et al. | Jul 2001 | B1 |
| 6301589 | Hirashima et al. | Oct 2001 | B1 |
| 6304879 | Sobeski et al. | Oct 2001 | B1 |
| 6363411 | Dugan et al. | Mar 2002 | B1 |
| 6366930 | Parker et al. | Apr 2002 | B1 |
| 6389462 | Cohen et al. | May 2002 | B1 |
| 6393459 | Lurndal | May 2002 | B1 |
| 6393481 | Deo et al. | May 2002 | B1 |
| 6401239 | Miron | Jun 2002 | B1 |
| 6405219 | Saether et al. | Jun 2002 | B2 |
| 6411956 | Ng | Jun 2002 | B1 |
| 6425005 | Dugan et al. | Jul 2002 | B1 |
| 6430564 | Judge et al. | Aug 2002 | B1 |
| 6438705 | Chao et al. | Aug 2002 | B1 |
| 6453321 | Hill et al. | Sep 2002 | B1 |
| 6453356 | Sheard et al. | Sep 2002 | B1 |
| 6463454 | Lumelsky et al. | Oct 2002 | B1 |
| 6466972 | Paul et al. | Oct 2002 | B1 |
| 6484204 | Rabinovich | Nov 2002 | B1 |
| 6490610 | Rizvi | Dec 2002 | B1 |
| 6505200 | Ims et al. | Jan 2003 | B1 |
| 6505241 | Pitts | Jan 2003 | B2 |
| 6523130 | Hickman et al. | Feb 2003 | B1 |
| 6526521 | Lim | Feb 2003 | B1 |
| 6539381 | Prasad et al. | Mar 2003 | B1 |
| 6539494 | Abramson et al. | Mar 2003 | B1 |
| 6542845 | Grucci et al. | Apr 2003 | B1 |
| 6578160 | MacHardy et al. | Jun 2003 | B1 |
| 6609213 | Nguyen et al. | Aug 2003 | B1 |
| 6651140 | Kumar | Nov 2003 | B1 |
| 6721777 | Sharma et al. | Apr 2004 | B1 |
| 6757708 | Craig et al. | Jun 2004 | B1 |
| 6766361 | Venigalla | Jul 2004 | B1 |
| 6775703 | Burns et al. | Aug 2004 | B1 |
| 6779017 | Lamberton et al. | Aug 2004 | B1 |
| 6826601 | Jacobs et al. | Nov 2004 | B2 |
| 6832238 | Sharma et al. | Dec 2004 | B1 |
| 6836889 | Chan et al. | Dec 2004 | B1 |
| 6877111 | Sharma et al. | Apr 2005 | B2 |
| 6898587 | Messinger | May 2005 | B2 |
| 6918013 | Jacobs et al. | Jul 2005 | B2 |
| 6944785 | Gadir et al. | Sep 2005 | B2 |
| 6957254 | Iterum et al. | Oct 2005 | B1 |
| 6963857 | Johnson | Nov 2005 | B1 |
| 7000019 | Low et al. | Feb 2006 | B2 |
| 7028030 | Jacobs et al. | Apr 2006 | B2 |
| 7065616 | Gajjar et al. | Jun 2006 | B2 |
| 7085834 | Delany et al. | Aug 2006 | B2 |
| 7089317 | Jeyaraman et al. | Aug 2006 | B2 |
| 7089584 | Sharma | Aug 2006 | B1 |
| 7100195 | Underwood | Aug 2006 | B1 |
| 7107543 | Berry et al. | Sep 2006 | B2 |
| 7127713 | Davis et al. | Oct 2006 | B2 |
| 7171692 | DeMello et al. | Jan 2007 | B1 |
| 7240101 | Rich et al. | Jul 2007 | B2 |
| 7254634 | Davis et al. | Aug 2007 | B1 |
| 7392317 | Halpern | Jun 2008 | B2 |
| 20010042073 | Saether et al. | Nov 2001 | A1 |
| 20010049717 | Freeman et al. | Dec 2001 | A1 |
| 20010054062 | Ismael et al. | Dec 2001 | A1 |
| 20020004850 | Sudarshan et al. | Jan 2002 | A1 |
| 20020073188 | Rawson, III | Jun 2002 | A1 |
| 20020073354 | Schoroiff et al. | Jun 2002 | A1 |
| 20020091832 | Low et al. | Jul 2002 | A1 |
| 20020107934 | Lowery et al. | Aug 2002 | A1 |
| 20020111922 | Young et al. | Aug 2002 | A1 |
| 20020116474 | Copeland et al. | Aug 2002 | A1 |
| 20020138551 | Erickson | Sep 2002 | A1 |
| 20020147652 | Gheith et al. | Oct 2002 | A1 |
| 20020147961 | Charters et al. | Oct 2002 | A1 |
| 20020161839 | Colasurdo et al. | Oct 2002 | A1 |
| 20020184444 | Shandony | Dec 2002 | A1 |
| 20020188591 | Santosuosso | Dec 2002 | A1 |
| 20030014480 | Pullara et al. | Jan 2003 | A1 |
| 20030018732 | Jacobs et al. | Jan 2003 | A1 |
| 20030041135 | Keyes et al. | Feb 2003 | A1 |
| 20030046442 | Maryka et al. | Mar 2003 | A1 |
| 20030060214 | Hendrey et al. | Mar 2003 | A1 |
| 20030065686 | Callahan et al. | Apr 2003 | A1 |
| 20030065826 | Skufca et al. | Apr 2003 | A1 |
| 20030074580 | Knouse et al. | Apr 2003 | A1 |
| 20030088713 | Mandal et al. | May 2003 | A1 |
| 20030105837 | Kamen et al. | Jun 2003 | A1 |
| 20030110467 | Balakrishnan | Jun 2003 | A1 |
| 20030115366 | Robinson | Jun 2003 | A1 |
| 20030233433 | Halpern | Dec 2003 | A1 |
| 20040059735 | Gold et al. | Mar 2004 | A1 |
| 20040139125 | Strassburg et al. | Jul 2004 | A1 |
| 20040153558 | Gunduc et al. | Aug 2004 | A1 |
| 20040230747 | Ims et al. | Nov 2004 | A1 |
| 20060143239 | Battat et al. | Jun 2006 | A1 |
| 20060168118 | Godlin et al. | Jul 2006 | A1 |
| 20060212453 | Eshel et al. | Sep 2006 | A1 |
| Number | Date | Country |
|---|---|---|
| WO 0045303 | Aug 2000 | WO |
| WO 0144894 | Jun 2001 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20030014526 A1 | Jan 2003 | US |
| Number | Date | Country | |
|---|---|---|---|
| 60305969 | Jul 2001 | US |
